Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Find all datasets that share a layer #6726

Open
fm3 opened this issue Jan 6, 2023 · 3 comments · May be fixed by #8350
Open

Find all datasets that share a layer #6726

fm3 opened this issue Jan 6, 2023 · 3 comments · May be fixed by #8350
Assignees
@frcroth
Labels
backend enhancement

Comments

@fm3
Copy link
Member

fm3 commented Jan 6, 2023

Many datasets share a layer (e.g. one color layer, many segmentation layers)
wk should detect this (by resolving symlinks or the paths in datasource-properties.json) and store this information in postgres so that queries can be possible against this info (e.g. list all datasets with this layer)
@fm3 fm3 mentioned this issue Jan 6, 2023
Closed
15 tasks
@fm3
Copy link
Member Author

fm3 commented Nov 15, 2024

This info could also be used when deleting datasets. If the layer is still referenced by another dataset (e.g. created via compose), the data should be moved, and other references/symlinks changed

@fm3 fm3 mentioned this issue Dec 16, 2024
Open
@fm3
Copy link
Member Author

fm3 commented Jan 12, 2025

@frcroth I don’t have a concrete design in mind for this yet. I guess we could store some property like “realpath” with the layers (or mags?) in postgres. We’d have to create a way for that information to get there. We’d have to deal with remote dataset layers.
The main use case right now is deletion #8279 – so remote datasets may not be the most important part right now.
I can show you around the existing code on dataset reporting from the datastore to postgres. Maybe you could then consider some ideas for how to solve this :)

@frcroth frcroth removed the frontend label Jan 13, 2025
@frcroth
Copy link
Member

frcroth commented Jan 13, 2025

  • Realpaths should not be exposed to users
  • should happen during inbox scan
  • real paths stored per mag
  • when deleting a layer, if possible dont create symlink per mag but only per layer

@frcroth frcroth linked a pull request Jan 22, 2025 that will close this issue
Open
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frcroth frcroth

Labels
backend enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Resolve symbolic links to mags and store them in db scalableminds/webknossos
2 participants
@fm3 @frcroth