Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: import certifications into database #203

Merged
merged 3 commits into from
Jan 2, 2025
Merged

Conversation

mathcolo
Copy link
Collaborator

Asana Task: 📐 Load Orbit Cert expiries into DB

Do what Glides does to import certifications, except also include a rail_line field based on the certification title.

Checklist

  • Tests:
    • (x) Has tests
    • ( ) Doesn't need tests
    • ( ) Tests deferred (with justification)
  • Product/Design sign off:
    • (x) Okayed the plan for the feature (e.g. the design files, or the Asana task)
    • ( ) Reviewed the feature as implemented (e.g. on dev-green, or saw screenshots)
    • ( ) No review needed

@mathcolo mathcolo requested a review from a team as a code owner December 16, 2024 20:39

rail_line =
cond do
String.ends_with?(title, "BL") ->
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skyqrose heads up, this is a code smell, but was actually mentioned by LMS.

Copy link
Member

@skyqrose skyqrose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow that was a lot of comments, sorry

lib/orbit/import/scheduled_row_certification_import.ex Outdated Show resolved Hide resolved
lib/orbit/import/async_import.ex Outdated Show resolved Hide resolved
lib/orbit/certification.ex Outdated Show resolved Hide resolved
lib/orbit/certification.ex Outdated Show resolved Hide resolved
priv/repo/migrations/20241211135758_certifications.exs Outdated Show resolved Hide resolved
lib/orbit/import/import_certifications.ex Outdated Show resolved Hide resolved
lib/orbit/import/import_helpers.ex Outdated Show resolved Hide resolved
lib/orbit/import/import_helpers.ex Outdated Show resolved Hide resolved
@@ -0,0 +1,63 @@
defmodule Orbit.Import.ImportHelpers do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should probably have tests. It'd mostly be exercised by the tests of the individual importers, but those cases would be complex, so it'd be nice to have some simple tests for these functions, too.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, you're right. here are a couple for parse_csv—the other two functions are really just wrappers? https://github.com/mbta/orbit/pull/203/files#diff-b5202e24be21249b1155432d249572dd4a4b0a3b844f445fe4051215d31579deR1

lib/orbit/import/import_certifications.ex Outdated Show resolved Hide resolved
@mathcolo mathcolo force-pushed the pim-load-certifications branch from 9a2052a to 4ad04e8 Compare December 18, 2024 22:17
@mathcolo mathcolo force-pushed the pim-load-certifications branch from 4ad04e8 to 672afe1 Compare December 18, 2024 22:39
@mathcolo
Copy link
Collaborator Author

@skyqrose Let me know what you think; I threw everything (except for the configuration) into Orbit.Import.ImportCertifications, which is clean and a lot easier to follow now. Also fixed the remaining items (🤦); thanks for looking!

@mathcolo mathcolo requested a review from skyqrose December 18, 2024 22:48
}
schema "certifications" do
field(:badge, :string)
field(:type, Ecto.Enum, values: [right_of_way: 0, rail: 1])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most places we use Ecto.Enum, we save the values as atoms (which get converted to strings in the database), though I see the Glides code this is copied from uses ints (it's the only place that uses ints).

Have you thought about whether you'd rather use ints or atoms/strings here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't, but I am now...

Although I admit integers don't make a ton of sense for values that don't have an ordering, I suppose I'm not sure using a physically wide space for an enum makes more sense either :( Lose/lose, going to leave as in glides.

:red

true ->
nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely not worth a splunk alert. I was thinking it might occasionally be useful if you see it locally or if doing a manual search for it while debugging an issue.

But I guess if we're manually debugging something, it's just as easy to grab the CSV from S3 and see the missing title there.

If we expect this to show up rarely or never, then might as well log it, that's what Log.warn is for, but it's definitely not necessary.

Copy link
Member

@skyqrose skyqrose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow this cleaned up. I think it was 17 files before, and now it's 13, including the new tests.

Looks good.

lib/orbit/import/helpers.ex Outdated Show resolved Hide resolved
@mathcolo mathcolo merged commit 73640e3 into main Jan 2, 2025
5 checks passed
@mathcolo mathcolo deleted the pim-load-certifications branch January 2, 2025 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants