Split scraper_utils.py, legistar_utils.py

[dphoria]

Feature Description

Take out top-level utility functions out of scraper_utils.py and legistar_utils.py into files under cdp_scrapers/utils. The files have become too large. i.e. Those files will contain implementations for IngestionModelScraper and LegistarScraper only.

Solution

Restructure so we will do something like

from cdp_scrapers.utils import reduced_list
from cdp_scrapers.utils.legistar import get_legistar_body

Alternatives

Functionally, the current state is completely adequate. I think the change will make for more manageable code.

[dphoria]

LegistarScraper.find_time_zone() should be a base utils member. i.e.

from cdp_scrapers.utils import find_time_zone

[mdahwireng]

How can one identify top-level utility functions?

[dphoria]

Hello @mdahwireng . Thank you so much for your interest. That is a great question and I will add some more definition to this issue later today.

[dphoria]

As far as I can tell these are the functions defined at the file root level in those 2 files.

get_legistar_body
get_legistar_person
get_legistar_events_for_timespan
get_legistar_content_uris

reduced_list
str_simplified
parse_static_person
parse_static_file
sanitize_roles

[mdahwireng]

@dphoria Thanks. I will have a look at it

[mdahwireng]

LegistarScraper.find_time_zone() should be a base utils member. i.e.

from cdp_scrapers.utils import find_time_zone

@dphoria LegistarScraper does not have a find_time_zone() method. Is there something I am missing?

[dphoria]

No, not at all. You are correct; the code has changed since when this issue was created. Sorry about that confusion.