High number of channels in the user end causes client to slow down excessively

[Gummikavalier]

Description:

Firefox slows down and causes high CPU load after short amount of activity in the web client if you have over 300 channels in the channel list.

Chrome starts exhibiting same symptoms by 1000 channels.

There is no reason to believe that also native desktop client would not suffer from this issue with some amount of channels.

Any well connected people in their organization can accumulate 300 - 600 entries into their channel list after couple of years of RC use. If the user does not actively hide excess rarely used channels, they are hit by this bug.

For exact steps to reproduce see the comment #36643 (comment)

Steps to reproduce:

1. RC version 7.9.x or 7.8.x is required for reproducing. 7.7.x does not suffer from this issue.
2. Have ~600 channels in your channel list.
3. Type in or receive messages on any single channel at fast pace (a, enter, b, enter, c, enter etc.). Use Firefox!
4. Observe action

Expected behavior:

Messages get sent as they should. Causes only moderate CPU load on the web client / browser.

Actual behavior:

Messages start slowing down after ~20 - 30 messages, and eventually CPU load hits 100%.

If you have 1000 channels you only need to send ~10 - 15 messages for issue to become evident.

Very high load remains (20 - 66 %) and every new message at any pace will cause 100% CPU load again. Very slow general response from the web client for anything.

RC tab in Firefox does not recover from this situation anymore; you have to reload the page, or switching channels may also help a bit.

Server Setup Information:

• Version of Rocket.Chat Server: 7.9.1
• License Type: Enterprise
• Number of Users: 500+
• Operating System: RHEL8
• Deployment Method: docker
• Number of Running Instances: 4
• DB Replicaset Oplog: Yes
• NodeJS Version: v22.16.0
• MongoDB Version: 7.0.22

Client Setup Information

• Desktop App or Browser Version: Firefox (latest) and Firefox ESR (latest)
• Operating System: Windows, Linux

Additional information

If you suffer from this issue, hiding mostly unused channels with hide can be used as a workaround. Note that also hiding action itself slows down after a while, so do F5/refresh after every ~20 channels hidden to be able to do this faster.

To speed up hiding you can also click the channel hamburger menu with mouse, and then press down arrow key and enter in succession. Then keep hammering the combination. And of course remember to choose no confirmation required when hiding the first channels.

When finished with hiding channels, remember to do F5 one last time.

[Gummikavalier]

I can confirm that there is something wrong in my account in RC; doing any changes to it on the main detail page freezes Firefox.

It does not matter whether the changes are made by user or admin account; user window freezes in both situation. But not admin window in Chrome.

Also thread "following avatar"-icon does not work for my account, and causes additional 30% constant CPU load when such thread is visible on the main channel. Resetting avatar image to default did not help for this.
"Own follower avatar"-issue has been with me from the moment this feature was introduced into RC recently.

The next likely step is to start deleting stuff via database from my account as I already cleaned up all details through admin console without effect.

[reetp]

Have you opened an internal ticker as well?

[Gummikavalier]

Not yet because devs could not reproduce it easily, as I cannot either, apart from this one instance.

So far I've figured out that avatar-case is separate from the main issue; following avatar images in threads may get loaded from old GridFS:Avatars store when they shouldn't. This triggers 30% CPU usage on that channel when browser fails to render the avatar and the page never finishes rendering properly.

Clearing up such avatar ID from the database fixes that problem. It is likely this is caused by our very old setup. There has been a lot of clearing up of avatars in the past when there were issues with them before, so new instances may not experience this avatar issue at all.

To ensure that store: GridFS:Avatars cannot be preferred by RC over store: FileSystem:Avatars, I now used mongosh command db.rocketchat_avatars.deleteMany({'store':'GridFS:Avatars'})` to clean up traces of avatars with old paths. Although it leaves old gunk in rocketchat_avatars.files- and rocketchat_chunks-collections. Truncating all three collections would have been better in the long run, but this was for some social damage control now.

Having that avatar thing resolved, I'm still stuck with the issue that Firefox gets overly sluggish after sending (also if just receiving) lots of messages in a row on the same channel.

[Gummikavalier]

I think I need to open an internal one for further troubleshooting. Lots of waiting time going on in the FF web console, but no clear indication (to me) why. Some developer is going to be needed to look at this, and if one cannot be found, then I simply have to delete and recreate my account on this instance as a next step. But this may be a very rare chance to troubleshoot this issue before doing so.

[reetp]

I have pinged the team on this already but having a proper ticket will help.

[Gummikavalier]

Thanks for the heads up!

Looks like regular troubleshooting process may be enough for this afterall:

I found out that the actual trigger for problems with my account was the sheer amount channels in the channel pane.

I had roughly 600 of them. I started hiding these channels one by one, and eventually the web client started working faster and faster.

Also I had come to conclusion that with this high number of channels, the slowness/CPU-load issue was actually triggered by every functionality in the web client. For instance I could reproduce it not just by sending messages on a single channel, but also receiving them on it. Later when I was hiding my excess channels that process too slowed down immensely when repeated some 20 times in a row.

In all cases F5/refresh clears up the problem. But it repeats if you have too many channels.

Note that only hide or leave channels cures the usability problem caused by this bug. The new preview UX has ability to hide all channels per type, but that does not help with the issue.

I may be able to reproduce this issue with another account by mass generating channels for it. If this succeeds, we have a nice reproducible user case for the devs. 🙂

[Gummikavalier]

It was possible to reproduce this on our separate test instance. The devs can take it from here on.

I added channels in bunches of one hundred to a test account via API.

After adding couple of hundred channels I actually had to kill the tab to get it updated properly, as updating the list and the whole UX froze so badly already.

The issue while sending messages started to show visibly in Firefox after adding 300 channels to the channel list.

For the test part I reloaded the page. Then typing in ~30 messages in a fast succession on the same channel clearly shows that the web client (in Firefox) is slowing down.

By 600 channels in the list I had to do populating channels with the test user's tab closed.
After opening up the tab and doing the message testing the slowing down is visible already after ~20 messages.

By 1000 channels in the list and after sending ~15 messages, the web client UX becomes practically unusable for the user unless they constantly F5/refreshes the screen.

By 1000 channels also Chrome starts having similar clear slowness that Firefox suffered already by 300 channels.

Note: Naturally the speed of the user's CPU affects these results somewhat. My client computer has a mobile processor that is maybe 4 years old by now. So the amount of channels required for reproducing the issue may vary in different user environment. Regardless, since the load created by the issue seems linear, I have no doubt that with Firefox 2000 channels would suffocate any single CPU core at latest.

[reetp]

When did this start?

Related to FF or Rocket?

Chrome/ium?

[Gummikavalier]

Chrome/Chromium one I never noticed until I did the test yesterday with that 1000 channels account.

Firefox one may have been around at least since 7.8.x. But it is hard to tell because I constantly do F5 anyway on my browser for other reasons. And F5 cleans the situation up for a moment.

I'll do additional test with RC 7.7.x today using our test environment and report back.

[Gummikavalier]

RC 7.9.1: Issue is bad, with both DISABLE_DB_WATCHERS=false and without.

RC 7.8.4: Issue happens, with both DISABLE_DB_WATCHERS=false and without. Not as bad as in 7.9.1 but freezes a lot already.

RC 7.7.8: No issue.

So this issue may be related to changes required by new message highlight / read state system introduced in RC 7.8 and onwards.

Nodejs version is not related to the issue; It remained the same between 7.7.x and 7.8.x, so should the issue be based on that alone, it would have been visible also in 7.7.8 and/or not in 7.8.4 at all.

I updated the original issue post from a stub to reflect exact issue information.

[Gummikavalier]

Tested with RC 7.9.3 and still reproducible. (There was a lot of package updates so was worth a shot.)

[reetp]

Seen quite a few posts on poor performance on FF 141+ with "AI" powered tab groupings.

Might be worth looking at?