Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug - XMP sidecar timestamp may be a second or two ahead of the database timestamp #15577

Open
Ashley-Butcher opened this issue Nov 5, 2023 · 4 comments · May be fixed by #17027
Open

Bug - XMP sidecar timestamp may be a second or two ahead of the database timestamp #15577

Ashley-Butcher opened this issue Nov 5, 2023 · 4 comments · May be fixed by #17027
Labels
bug: wip someone is currently working on that, check with them before taking over scope: DAM managing files, collections, archiving, metadata, etc.
Milestone
5.0

Comments

@Ashley-Butcher
Copy link

Describe the bug

The XMP sidecar timestamp may be a second or two ahead of the database timestamp, resulting in a prompt every time you open Darktable to update from XMP sidecar files.

Steps to reproduce

  1. Perform an action on many images to generate/update several XMP sidecar files (i.e. add a tag to 500-1000 images)
  2. Close Darktable
  3. Open Darktable

(This issue occurs with slower edits on just one image, but it's easier to reproduce with a larger number of images).

Observe: A prompt to update from sidecar file will appear with a timestamp delta of 1-2 seconds.

Expected behavior

If the XMP sidecar file is not updated outside of Darktable, no "update" should be found - i.e. the timestamp in the database should be the same as the file timestamp.

Logfile | Screenshot | Screencast

image

Commit

No response

Where did you install darktable from?

darktable.org

darktable version

4.4.2

What OS are you using?

Windows

What is the version of your OS?

Windows 11 Pro 22H2 (build 22621.2428)

Describe your system?

No response

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

NVIDIA GeForce RTX 3090, 24 GiB RAM dedicated (90 GiB shared), version 537.58

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

I've checked with Sysinternals Process Monitor (procmon) to ensure no other application is updating the timestamp on the XMP sidecar files in-between closing Darktable and reopening.

This is more likely to occur (and easier to reproduce) when modifying images in bulk, which is why this is the method given in the reproduction steps.
@parafin
Copy link
Member

parafin commented Nov 5, 2023 •
edited
Loading

Sounds a bit like #12257, but not sure if they are the same. Which filesystem is that? As far as I remember FAT filesystems have only 2-second precision of timestamps. But I see that reported timestamps are both odd and even, so it’s not that either…

@Ashley-Butcher
Copy link
Author

@parafin I saw #12257, also #13679 and #13298, so I'm sorry if it's a duplicate, but it didn't seem like an exact match, and it's still a real problem.

I'm getting this on large edits on a local NTFS formatted drive, but also on network shares via SMBv3. For workflow reasons I'm doing this part of my PP in Windows, and I know there are a lot of ways for timestamp resolution to be an issue here, but then again I think then there's a flaw somewhere in how the crawler detects changes.

Normally NTFS doesn't have such a precision issue. SMB on the other hand, not sure, as there are many factors that could reduce the precision. Some cloud storage synchronising software doesn't do nice things to timestamps either.

Maybe there should be a hash of the XMP file in the sqlite library to test as a secondary check if the timestamp is within a certain threshold, such as ≲ 2 seconds to cover timestamp resolution issues?

I don't know enough about the inner workings of Darktable to comment, though.

@ralfbrown ralfbrown added the scope: DAM managing files, collections, archiving, metadata, etc. label Nov 6, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 6, 2024

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 8, 2024

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

ralfbrown added a commit to ralfbrown/darktable that referenced this issue Jun 20, 2024
@ralfbrown
crawler: allow small time diff, XMP vs database
0042794 
Resolves darktable-org#15577.  By allowing a five-second difference in timestamps,
we account for granularity in filesystem timestamps (e.g. FAT) as well
as delays between updating the database and the sidecar file write
actually finishing.
@ralfbrown ralfbrown linked a pull request Jun 20, 2024 that will close this issue
Open
ralfbrown added a commit to ralfbrown/darktable that referenced this issue Aug 2, 2024
@ralfbrown
crawler: allow small time diff, XMP vs database
d698476 
Resolves darktable-org#15577.  By allowing a five-second difference in timestamps,
we account for granularity in filesystem timestamps (e.g. FAT) as well
as delays between updating the database and the sidecar file write
actually finishing.
ralfbrown added a commit to ralfbrown/darktable that referenced this issue Aug 2, 2024
@ralfbrown
crawler: allow small time diff, XMP vs database
8aebcc8 
Resolves darktable-org#15577.  By allowing a five-second difference in timestamps,
we account for granularity in filesystem timestamps (e.g. FAT) as well
as delays between updating the database and the sidecar file write
actually finishing.
@TurboGit TurboGit added bug: wip someone is currently working on that, check with them before taking over and removed no-issue-activity labels Aug 9, 2024
@TurboGit TurboGit added this to the 5.0 milestone Aug 9, 2024
ralfbrown added a commit to ralfbrown/darktable that referenced this issue Aug 10, 2024
@ralfbrown
crawler: allow small time diff, XMP vs database
dc55b71 
Resolves darktable-org#15577.  By allowing a five-second difference in timestamps,
we account for granularity in filesystem timestamps (e.g. FAT) as well
as delays between updating the database and the sidecar file write
actually finishing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug: wip someone is currently working on that, check with them before taking over scope: DAM managing files, collections, archiving, metadata, etc.
Projects
None yet
Milestone
5.0
Development

Successfully merging a pull request may close this issue.

crawler: allow small time diff, XMP vs database ralfbrown/darktable
4 participants
@TurboGit @parafin @ralfbrown @Ashley-Butcher