Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop rolling up extras #178

Open
5 tasks
avdata99 opened this issue Nov 18, 2020 · 2 comments
Open
5 tasks

Stop rolling up extras #178

avdata99 opened this issue Nov 18, 2020 · 2 comments
Assignees

Comments

@avdata99
Copy link
Contributor

Each time we save a new dataset we group almost any extras in a special extra called extras_rollup (we do this for CKAN 2.3 and for CKAN 2.8).

We need to stop doing this and save extras as regular extras.
This will require:

  • Update code and test to fit this new extras structure
    • Update code and test in this extension
    • Update code and tests in the DCAT-US extension.
    • Update code and tests in the DataGovTheme extension.
  • Deal with the spatial extra. We already have Multi#332 for this.
@amercader
Copy link
Contributor

@avdata99 Just a heads up that this was originally implemented to increase performance with a large number of datasets. When you reach the dataset count that data.gov has, the default CKAN model wasn't designed to cope with it, specially with large number of datasets with many extras like the harvested ones in data.gov. This was done back in the CKAN 2.3 days so a lot of things might have changed in the meantime but just flagging that you should check the performance impact of removing this. IIRC this wiki page had details about it.

@avdata99
Copy link
Contributor Author

Thanks @amercader.
I couldn't understand why this was done and now I understand. We are going to have to make sure that this change does not have a bigger impact than expected.

Maybe also related to Multi#267

FYI @adborden @thejuliekramer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants