-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace activity stream with S3 approach #471
base: main
Are you sure you want to change the base?
Replace activity stream with S3 approach #471
Conversation
chopkinsmade
commented
Aug 21, 2024
- JIRA ticket referenced in title
- Title is clear and concise
- Description gives any relevant detail
- Tests are up to date
- Documentation is up to date
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for submitting this; we'd like to have a chat about it for sure :)
Firstly, we are trying to understand what this work is about - it seems to replace the call to Staff SSO's ActivityStream API, but not do anything about the proxy DB that I thought was the whole point of the work? I might have the wrong end of the stick though.
Secondly, this PR is missing a lot of documentation; the repo needs to have a section explaining how this all works, down to the level of what files ought to be in written to the bucket, what's in them and how to process them.
Typing would make the code (especially in utils) much easier to understand and debug; we definitely want to make the utils code typed.
Most of the comments below are very picky around naming or structure to fit in with our approaches; the overall approach looks good.
I would also like to game out the high level approach together; there are a few downsides to implementing this that I want to chat about including error correction and what happens if we miss an ingest or get partial records, and how we might be able to move to a nearly-live system in the future if we wanted to.
Mostly though I think we just want to understand the intention of this code since it doesn't address what I thought we were trying to address with this work (i.e. the proxy DB is unaffected).
db2dfc4
to
0d2da87
Compare
…the data is ingested as expected
I have tested this PR locally and I'm happy the code is functioning, there are still a couple of steps I'd like to do next:
|