Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance enhancements for DB v6 writes #2394

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Jan 22, 2025

All changes on this PR are purely to decrease the time it takes to build a v6 DB, which when incorporated the build time in CI drops from ~2.5 hours to ~25 minutes 🎉 (still characterizing, I'll report back with final times when this is out of draft)

Changes:

  • add in-memory ID lookup for common or expensive models (e.g. CPEs originally had a large where clause to fetch similar rows, now this is an O(1) map lookup based on a string). This has been applied to CPEs, OperatingSystems, VulnerabilityHandles, and Providers. These are relatively flat records and based on debug logs appear to have a high cache hit rate. This also required ensuring we are correctly using this cache across transactions as well as within a transaction (deduplicating in memory).
  • Minimizing use of any queries to the DB within BeforeCreate hooks
  • keep blob digests in memory, not to a separate digests table (theme: maximize the items not written to the DB).
  • organizes the store functions to be clear when shared transactions can be used internally vs separate transactions when using public functions.
  • removed the pragma for 16KB page sizes (this had little effect on DB size and a small performance boost for write times).

While working on this, I did make some unrelated changes:

  • I noticed that pragma statements are not strongly verified by sqlite, so explicit checks were added (when adding a new pragma).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant