Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce number of DB connections when resyncing zaken #469

Closed
SilviaAmAm opened this issue Nov 5, 2024 · 1 comment · Fixed by #626
Closed

Reduce number of DB connections when resyncing zaken #469

SilviaAmAm opened this issue Nov 5, 2024 · 1 comment · Fixed by #626
Assignees
@SilviaAmAm
Labels
enhancement New feature or request

Comments

@SilviaAmAm
Copy link
Collaborator

Currently, the celery container shows this output when resyncing zaken:

[2024-11-05 15:07:47,330: DEBUG/ForkPoolWorker-2] "GET /zaken/api/v1/zaken?archiefnominatie=vernietigen&einddatum__isnull=False&einddatum__lt=2024-11-05&expand=resultaat%2Cresultaat.resultaattype%2Czaaktype%2Crollen&page=15 HTTP/11" 200 None
2024-11-05 15:07:47,586 INFO openarchiefbeheer.zaken.tasks  Retrieved page 15.
[2024-11-05 15:07:47,586: INFO/ForkPoolWorker-2] Retrieved page 15.
[2024-11-05 15:07:47,587: DEBUG/ForkPoolWorker-2] Closing all database connections
[2024-11-05 15:07:47,588: DEBUG/ForkPoolWorker-2] Closing all database connections
[2024-11-05 15:07:47,588: DEBUG/ForkPoolWorker-2] Closing all database connections
... (repeat x 100s)

Maybe we should do a bulk save for the zaken in the serializer.
@SilviaAmAm SilviaAmAm added the enhancement New feature or request label Nov 5, 2024
@SilviaAmAm SilviaAmAm moved this from Todo to In Progress in Open Archiefbeheer - Sprints Jan 20, 2025
@SilviaAmAm
Copy link
Collaborator Author

This does not come from saving the zaken not in bulk. It seems to come from the process_expanded_data function, which uses the ThreadPoolExecutor to retrieve the selectielijst procestype data per zaak.

@SilviaAmAm SilviaAmAm self-assigned this Jan 21, 2025
SilviaAmAm added a commit that referenced this issue Jan 21, 2025
@SilviaAmAm
⚡ [#469] Reduce DB and HTTPs connections
d199cdb 
- Avoid retrieving the selectielijst service for each zaak
- Take advantage of connection pooling by sharing client between zaken
SilviaAmAm added a commit that referenced this issue Jan 21, 2025
@SilviaAmAm
⚡ [#469] Save zaken in bulk during resync
79e16e3
SilviaAmAm added a commit that referenced this issue Jan 21, 2025
@SilviaAmAm
✅ [#469] Update tests
eb9108b
@SilviaAmAm SilviaAmAm mentioned this issue Jan 21, 2025
Merged
@SilviaAmAm SilviaAmAm moved this from In Progress to Has Pull Request in Open Archiefbeheer - Sprints Jan 21, 2025
SilviaAmAm added a commit that referenced this issue Jan 23, 2025
@SilviaAmAm
⚡ [#469] Reduce DB and HTTPs connections
20bcb16 
- Avoid retrieving the selectielijst service for each zaak
- Take advantage of connection pooling by sharing client between zaken
SilviaAmAm added a commit that referenced this issue Jan 23, 2025
@SilviaAmAm
⚡ [#469] Save zaken in bulk during resync
36afabe
SilviaAmAm added a commit that referenced this issue Jan 23, 2025
@SilviaAmAm
✅ [#469] Update tests
5a8c8e1
SilviaAmAm added a commit that referenced this issue Jan 23, 2025
@SilviaAmAm
✅ [#469] Test queries
8025100
SilviaAmAm added a commit that referenced this issue Jan 24, 2025
@SilviaAmAm
⚡ [#469] Reduce DB and HTTPs connections
892811c 
- Avoid retrieving the selectielijst service for each zaak
- Take advantage of connection pooling by sharing client between zaken
SilviaAmAm added a commit that referenced this issue Jan 24, 2025
@SilviaAmAm
⚡ [#469] Save zaken in bulk during resync
84c7d9a
SilviaAmAm added a commit that referenced this issue Jan 24, 2025
@SilviaAmAm
✅ [#469] Update tests
e2b8346
SilviaAmAm added a commit that referenced this issue Jan 24, 2025
@SilviaAmAm
✅ [#469] Test queries
5507b62
SilviaAmAm added a commit that referenced this issue Jan 24, 2025
@SilviaAmAm
⚡ [#469] Avoid retrieving the same procestype multiple times
369489e
svenvandescheur pushed a commit that referenced this issue Jan 27, 2025
@SilviaAmAm @svenvandescheur
⚡ [#469] Reduce DB and HTTPs connections
6eefeaa 
- Avoid retrieving the selectielijst service for each zaak
- Take advantage of connection pooling by sharing client between zaken
svenvandescheur pushed a commit that referenced this issue Jan 27, 2025
@SilviaAmAm @svenvandescheur
⚡ [#469] Save zaken in bulk during resync
539d6d2
svenvandescheur pushed a commit that referenced this issue Jan 27, 2025
@SilviaAmAm @svenvandescheur
✅ [#469] Update tests
d258688
svenvandescheur pushed a commit that referenced this issue Jan 27, 2025
@SilviaAmAm @svenvandescheur
✅ [#469] Test queries
ec93887
svenvandescheur pushed a commit that referenced this issue Jan 27, 2025
@SilviaAmAm @svenvandescheur
⚡ [#469] Avoid retrieving the same procestype multiple times
20ed740
@github-project-automation github-project-automation bot moved this from Has Pull Request to Done in Open Archiefbeheer - Sprints Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SilviaAmAm SilviaAmAm

Labels
enhancement New feature or request
Projects
Open Archiefbeheer - Sprints
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#469] Reduce db/HTTP connections maykinmedia/open-archiefbeheer
1 participant
@SilviaAmAm