Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table refresh with ntie #87

Merged
merged 35 commits into from
Jun 6, 2024
Merged

Table refresh with ntie #87

merged 35 commits into from
Jun 6, 2024

Conversation

podliashanyk
Copy link
Contributor

@podliashanyk podliashanyk commented Mar 24, 2024

Closes #4

Depends on Uninett/zinolib#53, Uninett/zinolib#58

Implemented changes:

  • Replaced table refresh by polling get_events() endpoint with a much more efficient NTIE with UpdateHandler. See Add server poll zinolib#53.
  • Updated version requirement for the zinolib dependency.
  • Refactor:
    • Renaming - replaced all notion of "poll" in the table refresh logic and config with "refresh". So no more confusing table refresh with "manually poll router or interface"-feature.
    • Extensive cleanup of the old table refresh implementation, including decoupling table refresh from the bulk-event-update response.
  • Added HTMX request synchronization via hx-sync. This is necessary when dealing with frequent requests to UpdateHandler and meanwhile having users potentially updating some events themselves, or selecting/expanding table rows. Without the synchronization, swaps from the "competing" requests were not complete/pending indefinitely, and heap of error alerts showed up in the view.
  • Mitigated [BUG]: events selected by user during table poll are ignored  #77, the fix is not yet complete though. See [BUG]: events selected by user during table poll are ignored  #77 (comment)

Not implemented/further work:

  • Dropped the idea of fine-tuning hx-indicator to NTIE table refresh.
    With the initial table refresh all table rows during refresh showed spinner indicator at the end of the row like so:
    Screenshot 2024-05-28 at 13 41 40
    The thought for NTIE was to only show spinner on the rows that are about to be updated with NTIE (since NTIE updates are more targeted than a refresh of the whole table). Yet I decided to drop this idea and preserve current spinner implementation for following reasons:
    • NTIE refresh might append or remove table rows, so spinners on all rows quite appropriately indicates that table is about to change
    • fine-tuning spinners for different NTIE cases is not a one-liner
  • Event ordering becomes off with table refresh via NTIE, see Ideas on combining table refresh with NTIE and sorting logic #96
  • Rows that are not updated for a while become stale, see How better fix stale row data with NTIE table refresh #97

Attention needed:

  • Are any docs updates needed?
  • Thoughts on the default refresh interval being reduced to 5?
  • Should we have a minimum refresh interval value?
  • Manual tests are welcome: adding, removing, modifying events on Zino server and then checking whether Howitz table is properly updated

@podliashanyk podliashanyk added enhancement New feature or request help wanted Extra attention is needed blocked Depends on other issue being solved beta Part of beta release labels Mar 24, 2024
@podliashanyk podliashanyk self-assigned this Mar 24, 2024
@codecov-commenter
Copy link

codecov-commenter commented Mar 24, 2024

Codecov Report

Attention: Patch coverage is 9.75610% with 74 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@601583e). Learn more about missing BASE report.

Files Patch % Lines
src/howitz/endpoints.py 7.50% 74 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #87   +/-   ##
=======================================
  Coverage        ?   48.34%           
=======================================
  Files           ?       15           
  Lines           ?      848           
  Branches        ?        0           
=======================================
  Hits            ?      410           
  Misses          ?      438           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Mar 24, 2024

Test results

  4 files    4 suites   8s ⏱️
14 tests 14 ✔️ 0 💤 0
56 runs  56 ✔️ 0 💤 0

Results for commit 70f726c.

♻️ This comment has been updated with latest results.

@podliashanyk podliashanyk removed the blocked Depends on other issue being solved label Apr 3, 2024
Copy link
Member

@lunkwill42 lunkwill42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My, how much simpler this gets when zinolib can handle the updates :)

src/howitz/templates/responses/updated-rows.html Outdated Show resolved Hide resolved
src/howitz/templates/responses/updated-rows.html Outdated Show resolved Hide resolved
src/howitz/templates/responses/updated-rows.html Outdated Show resolved Hide resolved
@@ -55,6 +55,9 @@ def auth_handler(username, password):
current_app.logger.info('Authenticated in Zino %s', current_app.event_manager.is_authenticated)

if current_app.event_manager.is_authenticated: # is zino authenticated
current_app.updater = UpdateHandler(current_app.event_manager)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't followed zinolib, but was this functionality recently added to zinolib? In that case, you might want to change the version requirement for the zinolib dependency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

src/howitz/endpoints.py Outdated Show resolved Hide resolved
UpdateHandler returns only ids of events that have been changed, without differentiating between different types of changes (event removed, event added or event updated). For Howitz being able to differentiate which change occurred on event, info about currently known evenmts must be saved
@podliashanyk podliashanyk force-pushed the table-refresh-with-ntie branch 2 times, most recently from eb6837e to e9e580b Compare May 23, 2024 12:40
@podliashanyk podliashanyk added blocked Depends on other issue being solved and removed blocked Depends on other issue being solved labels May 24, 2024
Docd on sync startegies: https://htmx.org/attributes/hx-sync/
From the HTMX documentation on hx-sync I have gotten an erroneous notion that HTMX's "drop" strategy is by default set in the whole project.

But apparently "drop" is the default only if you use "hx-sync" attribute and don't provide any strategy, ie 'hx-sync' attribute must be used on the tag in order for HTMX to start synchronizing requests on (and within) this tag.

Synchronization is not a global config option in HTMX!
podliashanyk and others added 4 commits May 28, 2024 16:47
This will make the app perform one HTMX request at a time, yet queue all requests that occur during this request. 

Synchronizing is necessary when having NTIE refresh requested with intervals less than 30 seconds and meanwhile having other operations on events available to perform for users. Without any synchronizing in the app, if several HTMX requests update the page unexpected race conditions occur. It most often leads to none of the requests being completely finished, instead a heap of error alerts show up on the screen.
Co-authored-by: Morten Brekkevold <[email protected]>
@podliashanyk podliashanyk marked this pull request as ready for review May 28, 2024 14:48
Copy link
Contributor

@hmpf hmpf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll check the functionality.

This needs a round of ruff or black. (My eyes hurt a bit.) We could do that for the entire codebase after this PR is in.

@hmpf hmpf self-requested a review June 3, 2024 08:44
Copy link
Contributor

@hmpf hmpf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have done som emanual testing. Looks good!

Copy link
Member

@lunkwill42 lunkwill42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My nitpicks have been mostly fixed, but something seems off with the project requirements? @hmpf?

@@ -48,5 +48,5 @@ werkzeug==2.3.8
# flask
# flask-login
# howitz (pyproject.toml)
zinolib==1.0.1
zinolib==1.0.4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something is weird with this. pyproject.toml says zinolib>=1.1.0, so why is zinolib 1.0.4 in the frozen requirements list?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we forgot to update it probably.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With zinolib==1.0.4 I can't even get Howitz to run, with 1.1.0 this NTIE PR seems to work just fine for me :)

Copy link
Member

@lunkwill42 lunkwill42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me :)

@podliashanyk podliashanyk merged commit b07d8c9 into main Jun 6, 2024
5 checks passed
@podliashanyk podliashanyk deleted the table-refresh-with-ntie branch June 6, 2024 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beta Part of beta release enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Have events in Zino web frontend update automatically from backend
4 participants