[WIP] improve graph performance #1391

pderaaij · 2024-09-02T19:49:42Z

Based on some performance findings found #1375 I started to experiment with some improvements. I've noticed the functionlistByIdentifier being called alot by other functions. In this function, there is a foreach loop running over all resources. A function which works fine for small notebooks, but large notebooks seems to be hit exponentionally.

In this concept I've added a new state object that lists all resources for an identifier. In this case, I choose the identifier to be the file-name of the note. Using that identifier, I list all resource paths with that same identifier. Using this I can remove the foreach loop and just use the identifier to list a small set of resources.

For the 10000 linked notes folder from this repo (https://github.com/rcvd/interconnected-markdown/tree/main/Markdown) this reduces the startup time to:

Foam Logging: info
[info - 21:39:06] Starting Foam
[info - 21:39:07] Loading from directories:
[info - 21:39:07] - /Users/paulderaaij/projects/interconnected-markdown/Markdown/10000s/10000
[info - 21:39:07]   Include: **/*
[info - 21:39:07]   Exclude: **/.foam/**,**/.vscode/**/*,**/_layouts/**/*,**/_site/**/*,**/node_modules/**/*,**/.git,**/.svn,**/.hg,**/CVS,**/.DS_Store,**/Thumbs.db
[info - 21:39:15] Workspace loaded in 7755ms
[info - 21:39:15] Graph loaded in 660ms
[info - 21:39:15] Tags loaded in 4ms
[info - 21:39:15] Loaded 10001 resources

Using the current master the numbers are:

Foam Logging: info
[info - 21:46:53] Starting Foam
[info - 21:46:53] Loading from directories:
[info - 21:46:53] - /Users/paulderaaij/projects/interconnected-markdown/Markdown/10000s/10000
[info - 21:46:53]   Include: **/*
[info - 21:46:53]   Exclude: **/.foam/**,**/.vscode/**/*,**/_layouts/**/*,**/_site/**/*,**/node_modules/**/*,**/.git,**/.svn,**/.hg,**/CVS,**/.DS_Store,**/Thumbs.db
[info - 21:47:01] Workspace loaded in 8147ms
[info - 21:47:27] Graph loaded in 25513ms
[info - 21:47:27] Tags loaded in 2ms
[info - 21:47:27] Loaded 10001 resources

A delta of almost 25 seconds for loading the graph. Not only a performance improvement for loading the workspace, but also for some other operations that uses listByIdentifier directly or via find. This should make several UI components work better and more responsive.

The downside is that it adds a new stateful object, but I don't see an other way to increase the performance of the foreach loop.

@riccardoferretti curious to your thoughts on this.

pderaaij · 2024-09-02T19:51:31Z

packages/foam-vscode/src/core/model/workspace.ts

      }
    }
+
+    if (needle === '/page b.md') {


I need to remove this ;-)

riccardoferretti · 2024-09-03T14:59:12Z

This is amazing @pderaaij ! I am back from holidays (you know, italy + august = holiday) so will look a bit more into the PR this week

riccardoferretti · 2024-09-04T19:23:12Z

Sharing some high level thoughts here.
As I was doing some research in the approach, I came across the trie data structure, which fundamentally feels like what we are kinda implementing here.
As I looked into it, I realized that using the reversed path (similar to what we do in getShortestIdentifier) and saving in in a trie would be the cleanest solution, as it would out of the box replace the existing indexing while giving us the features and performance we want.

What do you think? Again, this was at a cursory look, I might be missing something

pderaaij · 2024-09-06T06:58:23Z

Never really looked into the trie datastructure. Will do some research and fiddle around!

pderaaij · 2024-09-10T17:40:17Z

I need a bit more time to figure out the right approach. Have been looking into tie data structures, but most implementation I found are immutable. A characteristic we don't want for our use case here. So, I need to find an implementation that does allow mutations. Did a quick checkout of https://rimbu.org, but couldn't find a right data structure off the bat for our use case. So, I require some more time to look around and find a solution. Ideas welcome of course.

pderaaij · 2024-09-23T19:37:40Z

I got a long way tonight using https://yomguithereal.github.io/mnemonist/trie-map. Need to finish some things and get the last use cases working. Will do a new performance test when things are working. Hopefully I can finish it later this week.

pderaaij · 2024-09-25T09:21:53Z

@riccardoferretti I've updated the implementation with my WIP so far.

It has a little impact on performance. Compared to the earlier version, I now load the 10000s workspace in 2 to 3 seconds. Which still is a massive improvement compared to the current implementation. I do feel the performance impact is acceptable given the ease of maintenance this implementation brings in the code.

I have one remaining issue and I hope you can help. Originally, a Map holds the insertion order. TrieMap obviously does not do that. But, this means that resources() needs to return a sorted Iterator. I can't find a correct way to sort the values and then return an IteratableIterator. Perhaps you have an idea here?

If we can resolve this issue, all tests will be green so we can do some further testing and enhancing before shipping it.

pderaaij · 2024-10-09T17:59:58Z

@riccardoferretti it seems I was on the right track, but simply not sharp after a long day of work. Got it working now. Please review and check accordingly.

riccardoferretti

Yes this is looking like the right approach, I left a few comments. Let me know what you think

packages/foam-vscode/src/core/model/workspace.ts

riccardoferretti · 2024-10-09T20:33:55Z

packages/foam-vscode/src/core/model/workspace.ts

+      this._resources.values()
+    ).sort(Resource.sortByPath);
+
+    return resources.values();


do we need the .values() again on this line?

It seems so. The only way I can see to return an IterableIterator

riccardoferretti · 2024-10-09T20:37:39Z

packages/foam-vscode/src/core/model/workspace.ts

@@ -70,17 +79,24 @@ export class FoamWorkspace implements IDisposable {
  }

  public listByIdentifier(identifier: string): Resource[] {
-    const needle = normalize('/' + identifier);
+    let needle = this.getReversedIdentifier(identifier);


I see you moved the addition of \ from here into the getReverseIdentifier fn. I wonder if there is any side effect from doing that?

Tbh, I am kinda relying on the tests passing here, I find this code (the existing one, that I wrote :) ) hard to properly process, I think at some point I will need a round to make it clearer for my future self and others :)\

I think fundamentally the semantics we want here is that resources can be found by either the identifier as is and by the identifier + default extension. Maybe we save both keys in the Trie.

In any case, just sharing some thoughts, I am ok with moving forward with the PR before sorting this out.

So far, I haven't found any side effect. I guess it was done originally to always have a start of a path.

I agree with optimising the logic on a next round. Also to solve #1399.

packages/foam-vscode/src/core/model/workspace.ts

pderaaij · 2024-10-11T18:13:48Z

@riccardoferretti somehow I managed to break the build in CI. Locally all works, but can't get it going on Actions. Could you have a look?

riccardoferretti · 2024-10-11T18:35:42Z

@pderaaij I think it's related to the markdown-it version change

pderaaij · 2024-10-11T18:37:11Z

Yeah, I was hoping that would fix it. Had the problems after rebasing the lock files. Which was a hassle on its own....

pderaaij · 2024-10-11T19:10:05Z

Well... I found the little bugger:

markdown-it/linkify-it#111

pderaaij · 2024-10-11T19:14:15Z

I am giving up on this PR. Going to do a fresh branch on master and open a new PR. Not trusting all the mess this has created.

pderaaij commented Sep 2, 2024

View reviewed changes

pderaaij changed the title ~~[WIP] Add state for identifier to reduce foreach~~ [WIP] improve graph performance Sep 2, 2024

riccardoferretti mentioned this pull request Oct 8, 2024

Quarto (qmd) extension not recognised as note format despite being configured #1399

Open

pderaaij force-pushed the performance-upgrade branch from c9c853e to ce662af Compare October 9, 2024 17:59

riccardoferretti reviewed Oct 9, 2024

View reviewed changes

pderaaij added 5 commits October 11, 2024 19:48

Add state for identifier to reduce foreach

71c2c19

WIP implementation of Mnemonist TrieMap

3f4591b

Sort resources on return

9a6e3b9

Process review comments

ed56f6f

Fix merge conflicts in lock files

958a03c

pderaaij force-pushed the performance-upgrade branch from 87807e9 to 958a03c Compare October 11, 2024 17:53

Fix non-matching MarkdownIt type and lib version

45767ea

pderaaij added 3 commits October 11, 2024 20:53

Reset markdownIt types version

3452868

Rebuild yarn lock file

312e54a

Fix issues on types with dependency

6913d0e

pderaaij mentioned this pull request Oct 11, 2024

Improve performance via Triemap in workspace #1406

Merged

pderaaij closed this Oct 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] improve graph performance #1391

[WIP] improve graph performance #1391

pderaaij commented Sep 2, 2024

pderaaij Sep 2, 2024

riccardoferretti commented Sep 3, 2024

riccardoferretti commented Sep 4, 2024

pderaaij commented Sep 6, 2024

pderaaij commented Sep 10, 2024

pderaaij commented Sep 23, 2024

pderaaij commented Sep 25, 2024

pderaaij commented Oct 9, 2024

riccardoferretti left a comment

riccardoferretti Oct 9, 2024

pderaaij Oct 11, 2024

riccardoferretti Oct 9, 2024

pderaaij Oct 11, 2024

pderaaij commented Oct 11, 2024

riccardoferretti commented Oct 11, 2024

pderaaij commented Oct 11, 2024

pderaaij commented Oct 11, 2024

pderaaij commented Oct 11, 2024

[WIP] improve graph performance #1391

[WIP] improve graph performance #1391

Conversation

pderaaij commented Sep 2, 2024

pderaaij Sep 2, 2024

Choose a reason for hiding this comment

riccardoferretti commented Sep 3, 2024

riccardoferretti commented Sep 4, 2024

pderaaij commented Sep 6, 2024

pderaaij commented Sep 10, 2024

pderaaij commented Sep 23, 2024

pderaaij commented Sep 25, 2024

pderaaij commented Oct 9, 2024

riccardoferretti left a comment

Choose a reason for hiding this comment

riccardoferretti Oct 9, 2024

Choose a reason for hiding this comment

pderaaij Oct 11, 2024

Choose a reason for hiding this comment

riccardoferretti Oct 9, 2024

Choose a reason for hiding this comment

pderaaij Oct 11, 2024

Choose a reason for hiding this comment

pderaaij commented Oct 11, 2024

riccardoferretti commented Oct 11, 2024

pderaaij commented Oct 11, 2024

pderaaij commented Oct 11, 2024

pderaaij commented Oct 11, 2024