Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add EIA Provider & Weekly Petroleum Status Report #6693

Open
wants to merge 32 commits into
base: develop
Choose a base branch
from

Conversation

deeleeramone
Copy link
Contributor

@deeleeramone deeleeramone commented Sep 25, 2024

  1. Why?:

The EIA's Weekly Petroleum Status Report, and Short Term Energy Outlook are closely followed events for anyone watching the fossil fuels. This PR represents the first integrations of open data available from the EIA.

  1. What?:

The WPSR is comprised of thirteen (excludes discontinued series) high-level categories with each containing a subset of tables. Data is from the static Excel files published here, and each file represents a single category.

All data from a single category is returned by supplying "all" to the table parameter of the WPSR endpoint.

Tables from the WPSR are returned in a flat format in the same order as presented in the Excel files. The response is suitable for pivot tables and SQL storage.

The STEO dataset is twenty-six tables of near-term projections, which is pulled from the V2 API.

  1. Impact:

    • This package contains dependencies not included in openbb-core:

      • async-lru = "^2.0.4" (to prevent Excel files from being repeatedly downloaded)
      • openpyxl = "^3.1.5" (for working with Pandas ExcelFile)
      • xlrd = "^2.0.1" (for working with Pandas ExcelFile)
    • New endpoints:

      • obb.commodity.petroleum_status_report (WPSR - no API key required)
      • obb.commodity.short_term_energy_outlook (STEO)
    • ALRU cache (async drop-in replacement to functools.lru_cache) is used for the WPSR files that retains each "category" requested for the duration of the Python session. Setting use_cache as False will invalidate the cache and download the file again.

    • The table parameter has defined choices that represent all possible tables from all categories.

      • For WPSR, a custom validator checks this parameter against the category and the list of tables within that category; it will raise with a message indicating which choices are valid for the given category.
    • STEO also accepts a symbol input for targeted queries from the API route.

  2. Testing Done:

    • Unit/integration
  3. Reviewer Notes:

    • The combination of (category="weekly_estimates", table="all") will result in a very large response that will take 20 seconds or so to download, parse, and combine 21 sheets and more than 1 million rows. It's best to avoid "all" for that category.

Screenshot 2024-09-24 at 7 32 38 PM

@deeleeramone deeleeramone added enhancement Enhancement do not merge Label to prevent pull request merge platform OpenBB Platform v4 PRs for v4 labels Sep 25, 2024
@deeleeramone deeleeramone changed the title add eia provider and weekly petroleum status report [Feature] Add EIA Provider & Weekly Petroleum Status Report Sep 25, 2024
@deeleeramone deeleeramone removed the do not merge Label to prevent pull request merge label Oct 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking_change enhancement Enhancement platform OpenBB Platform v4 PRs for v4
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant