Feature/tools 10 process imported #382
Conversation
| successfully from Data Workspace. | ||
| There is no validation on the new records, as any validation will | ||
| happen when the data is imported from DW | ||
| """ |
There was a problem hiding this comment.
I'm left confused by this docstring I'm afraid. Has the data already been validated at this point or is it still to be validated? "DW" is also confusing in this context :)
| @@ -0,0 +1,113 @@ | |||
| from tools.models import IrapToolData, IrapToolDataImport, Tool | |||
There was a problem hiding this comment.
I thought the plan was to build this in a way that would easily extensible if/when we want to import from other sources in the future...
Should we be calling this module import.py and making all the methods a bit less specific to IRAP, unless they are specifically and only for IRAP data?
| previous_values = {} | ||
| for field in new_fields: | ||
| new_value = getattr(new, field.name) | ||
| old_value = getattr(old, field.name) |
There was a problem hiding this comment.
do we need to catch times when the field list and/or names might have changed? i.e. old might not have a field that exists on new, or the other way around.
| return changed, previous_values | ||
|
|
||
|
|
||
| def are_field_identical(import_obj: IrapToolDataImport, old_values: dict) -> bool: |
There was a problem hiding this comment.
| def are_field_identical(import_obj: IrapToolDataImport, old_values: dict) -> bool: | |
| def are_fields_identical(import_obj: IrapToolDataImport, old_values: dict) -> bool: |
| IrapToolData.objects.update(imported=False) | ||
|
|
||
| imported_iraps = IrapToolDataImport.objects.all() | ||
| for imported_irap in imported_iraps: |
There was a problem hiding this comment.
It would be great IMO to have all the field names in a configurable mapping object, so that it's easy to update which fields go to which if the column names in the data source change from time to time
|
Marking this as a draft as the work hasn't been actively worked on in a while |
Implement the logic used when importing a new set of data for IRAP tools.
It assumes that the new data has already been validated, and it exists in an intermediate table.