Implement a way to mirror data from EGA #146
Conversation
csc-jm-zz
force-pushed
the
feature/data-mirroring
branch
2 times, most recently
from
e40ff7d to
8a4aae9
Compare
metadata_backend/api/handlers.py
Outdated
| """ | ||
| dataset_id = req.match_info["datasetId"] | ||
| db_client = req.app["db_client"] | ||
| await spawn(req, self.data_mirroring(dataset_id, db_client)) |
There was a problem hiding this comment.
this returns a Job https://aiojobs.readthedocs.io/en/stable/api.html#job which we can watch to see if it has finished or not.
I think it is perfectly fine if we move these it its own handler class/ file as not sure it reuses anything from the base handler class.
There we could implement also an endpoint where we would check the status of that job (active, pending or closed)
There was a problem hiding this comment.
The endpoint was moved to its own handler class in handlers.py, but job status check endpoint remains unimplemented.
metadata_backend/api/handlers.py
Outdated
| :param dataset_id: ID of dataset in EGA | ||
| :param db_client: Motor client used for database connections | ||
| """ | ||
| ega_data = MetadataMirror().mirror_dataset(dataset_id) |
There was a problem hiding this comment.
we need to check if the database already contains the dataset and if it is in a published folder (because we are mirroring public information, I think the folder should be published by default, but if there is another option fine with that as well) so that we don't mirror it again
There was a problem hiding this comment.
Lines 890-892 of handlers.py has an attempt at doing this check. However, the check is not implemented properly yet and needs refining.
csc-jm-zz
force-pushed
the
feature/data-mirroring
branch
from
8a4aae9 to
228992b
Compare
csc-jm-zz
force-pushed
the
feature/data-mirroring
branch
from
228992b to
dbe46d8
Compare
|
Unfortunately, I will not be able to work on this PR any further myself so the things that still need to be done here are:
|
|
we will probably not need it for now, closing as we need to mirror data to EGA #295 |
Description
This PR aims to implement a solution to mirror data from EGA into the application database with its own
/mirrorendpoint.Related issues
Ticket no. 116 from internal board
Type of change
Changes Made
Testing
Mentions
Script used for mirroring is heavily based on the one found here.