Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TP2000-1493 - TAP driven quota open data export #1302

Draft
wants to merge 70 commits into
base: master
Choose a base branch
from

Conversation

dougmills-DIT
Copy link
Contributor

TP2000-1493 TAP driven quota open data export

Why

  • The current open data implementation is created from an SQL lite export imported into data workspace and queried via a static query stored in data-flow.
  • The move from data workspace to TAP generated will speed the process up and allows more stringent testing to be automated around the output.
  • Update have been made to the current implementation several times to resolve data inconsistency issues. Moving to within TAP will help reduce future issues as TAP is updated.
  • Moving the creation of quotas open data is the first of several open data exports that would be better placed within TAP.

What

  • Create process to export TAP quota data to a CSV file
  • Add methods of storage for both S3 and local
  • Unit test everything

Checklist

  • Requires migrations? No
  • Requires dependency updates? No

Links to relevant material
See: Description

@SteveCN7
Copy link

SteveCN7 commented Oct 8, 2024

Silly question - the initial volume column, are we pulling that from TAP or OTT? I think we should be pulling it from TAP really if we're not already

dependabot bot and others added 15 commits October 10, 2024 11:12
Bumps [django](https://github.com/django/django) from 4.2.15 to 4.2.16.
- [Commits](django/django@4.2.15...4.2.16)

---
updated-dependencies:
- dependency-name: django
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Delete associations as well as definition

* Add test

* Prevent business rules from failing

* Refactor if statement
* adding view and url

* wip

* wip - urlconf issues

* Fix url args issue

* adding tests

* responding to CR comments

---------

Co-authored-by: Edie Pearce <[email protected]>
* working through current/approved up to transaction issues

* updating to use .all() on querysearch; adding actions column on QuotaDetail view

* fixing broken test

* adding title caption

* copy changes requested by UR team

* tidying up tabs/sub_quotas.jinja

* alligning tab headings with tab names

* fixing conditional headings

* changing query to .current

* fix
* Ability to edit suspension periods

* Add ability to delete a suspension period

* Form tests

* View tests

* Fix failing tests

* Fix and reorder cancel button

* Redesign confirm pages

* PR review comments

* test changes following PR review

* Update get_success_url to make kwarg clearer
* Ability to edit suspension periods

* Add ability to delete a suspension period

* Form tests

* View tests

* Fix failing tests

* Fix and reorder cancel button

* Redesign confirm pages

* PR review comments

* test changes following PR review

* Update get_success_url to make kwarg clearer

* Get rid of +1 to prevent flaky test
* Edit capability for blocking periods

* Tidy up

* Disable ability to edit already started blocking periods

* Form tests

* View tests

* Remove class from table

* Tidy up breadcrumbs into one file
#1310)

* moved 'remove all' button to right hand side of page and hidden this button from non-superusers

* ran black linter

* removed commented div tags
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.3 to 3.0.6.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.0.3...3.0.6)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthew McKenzie <[email protected]>
* Add confirmation page to sync route

* Redirect async route to create success page

* Make measures a small m

* WIP- Make edit process queue page

* add expected measures count property to bulk editor

* Create and apply bulk edit measures processing state filter

* ran linter

* Pull combined measure create and edit process code into a base class

* Refactor confirm edit sync template to show a number of measures not their ids

* Swap render for redirect in Sync done and create sync and async success views

* Change processing queue url to include create

* change create measures queue tab selected tab name, and page title

* Update tab titles and heading semantics and add main heading

* lint

* forgot h2 closing tags oops

* Fix broken test due to changed url

* add "unknown number" to tasks with failed counts within template

* Update headings from content guidance

* Add edit link to the home page

* Remove secondary styling on button

* Refresh button to redirect to edit tab

* remove create from the no bulk edit copy

* Add doc strings to the sync and async confirm views

* Hook up create sync to success view

* Lint

* Fix test to accommodate sync create redirect

---------

Co-authored-by: Lauren Qurashi <[email protected]>
)

* Make real edits in workbasket

* Fix data kwarg error

* Use real edits for exclusions

* Move test
* Change composite key to random unique string

* Undo importer change

* Add test to confirm bug fix

* Include wb pk in composite key

* Update to zero padded wb pk

* PR comment
@uktrade uktrade deleted a comment from DevelopwithTom Dec 13, 2024
dependabot bot and others added 3 commits December 16, 2024 09:43
Bumps [nanoid](https://github.com/ai/nanoid) from 3.3.7 to 3.3.8.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](ai/nanoid@3.3.7...3.3.8)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Test solution

* Add current

* re-adding old business rule check

* rule check

* Distinct count error

* Remove some QA5 and QA6 checks for update form

* Update models.py

Remove comment as not confident enough to make this comment to all

* Rid initial volume none scenario

---------

Co-authored-by: Charles Prichard <[email protected]>
* Basic synchronous auto end measures functionality

* Make it a Celery task

* Add tests

* Make it part of current wb

* Update confirmation page and move link

* Move get measures to helper and make tests robust

* Update template

* Use effective end date on measures

* Move auto end date link as per design request

* Black

* final touches

* Responding to PR comments

* Update message for no measures found

* Add extra atomic
api_query_dates = []

# collect possible query dates, but only for current and historical, not future
if quota.valid_between.lower <= date.today():

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if quota.valid_between.lower <= date.today():
if quota.valid_between.lower <= specified_date:

dependabot bot and others added 24 commits December 24, 2024 10:02
Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.4 to 3.1.5.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.4...3.1.5)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix delete association cancel button

* Include unapproved measures

* As at today and beyond
…1371)

* updates to QA5 to prevent False positives from deleted associations

* linting
finished S3 storage
finished local storage
updates for NamedTemporaryFile
finished S3 storage
finished local storage
updates for NamedTemporaryFile
…rt' into TP2000-1493_quota_open_data_export

# Conflicts:
#	exporter/tests/test_storages.py
#	pyproject.toml
#	reference_documents/views/reference_document_version_views.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants