Import hangs

[breun]

Describe the bug

I'm trying to import an export created with osxphotos, but after 3.5 hours the step 'Grouping files into import groups' was still at 0%, while the Python process had been running at 100% CPU:

❯ du -sh ~/Downloads/export 
3,1G	/Users/breun/Downloads/export
~ ❯ osxphotos version
You have the latest version of osxphotos: 0.68.6
~ ❯ osxphotos import ~/Downloads/export 
Collecting files to import... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   
Filtering import list for image & video files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Grouping files by parent directory... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Grouping files into import groups... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--

Aborted!
~ took 3h31m15s ❌1 

To Reproduce
Steps to reproduce the behavior:

1. What' the full command line you used with osxphotos?

osxphotos import ~/Downloads/export

3. What was the error output?

There was no error output, but the import process hadn't made any progress in over 3.5 hours.

Expected behavior
I expected at least some progress within 3.5 hours.

Desktop (please complete the following information):

• OS: macOS 15.1 24B83
• osxphotos version (osxphotos --version)

osxphotos, version 0.68.6
Python 3.12.7 (main, Oct 5 2024, 01:39:55) [Clang 16.0.0 (clang-1600.0.26.3)]
macOS 15.1.0, arm64

Additional context
There are 1448 files in ~/Downloads/export, including AAE files. The export was created on another Mac with this command: osxphotos export --export-aae --exiftool --album "${ALBUM_NAME}" "${DESTINATION}".

[oPromessa]

osxphotos import

• Hmmm... On my tests the grouping does take time but I didn't see 3.5 hours.
• I've also experienced that after about 2000 pics imported -- not your case! -- the import fails with continuous errors from Photos, so you can use option --stop-on-error 5 for osxphotos to stop running and you can then rerun with --resume option.

$ osxphotos help import error | cat
Usage: osxphotos import [OPTIONS] [FILES_OR_DIRS]...

  Import photos and videos into Photos. Photos will be imported into the most recently opened Photos library.
(...)
Options that match 'error':

-R, --resume           Resume previous import, skipping any files which have already been imported. Note:
                       data on each imported file is kept in a database in
                       '/Users/XXX/.local/share/osxphotos/osxphotos_import.db'. This data can be used to
                       resume a previous import if there was an error or the import was cancelled. Any files
                       which were already imported will be skipped. See also --skip-dups.
--stop-on-error COUNT  Stops importing after COUNT errors. Useful if you experience a large number of errors
                       during import.  

Alternative: use osxphotos sync

• IF you still have access to the source Mac and Photos Library do consider oxphotos sync
• Do read the instructions to see if it applies to your case.

Usage: osxphotos sync [OPTIONS]

  Sync metadata and albums between Photos libraries.

  Use sync to update metadata in a local Photos library to match metadata in
  another Photos library. The sync command works by finding identical photos
  in the local library and the import source and then updating the metadata in
  the local library to match the metadata in the import source. Photos are
  considered identical if their original filename and fingerprint match.

  The import source can be a Photos library or a metadata export file created
  with the --export option.
(...)

• With the exception that folders are not replicated as indicated on: osxphotos sync to also consider --merge location and folder structure #1639 (I guess Rhet will get around to it when time allows), you can get all the other data.

1. Export metadata from original Library database

osxphotos sync --verbose --timestamp --not-hidden --export MetaData.db --library /path/to/your/PhotosLibrary.photoslibrary/

2. Transfer all the pics to your new Mac. Transfer also the MetaData.db file to your new Mac

3. Then open the new library in Photos and import all the pics normally. The sync magic will come next.

• Please note I found post-Monterey Photos does not always applies correctly .AAE changes when importing. (almost convinced Apple Support to look at it in more recent MacOS versions.. but failed 😢 )
• Also noticed that in some cases the 'Unknown error' occurs at loading. In some instances I had to split the JPG+AAE pair and place the .JPG in a separate folder and loaded alone, so that Photos bypasses .AAE.

4. Open the new library in Photos and import metadata database. Use either --merge or --set as appropriate to your case.

caffeinate -s osxphotos sync --verbose --timestamp --import MetaData.db --unmatched  --merge albums,description,favorite,keywords,location,title --library ~/Pictures/Photos.photoslibrary  --report sync.csv

(...)
Importing metadata for DSC01534-a-2.JPG (ABDC3178-96D3-49FF-82A8-DF8B4AE6FF20)
        Adding to album ABC
        Creating album 'ABC'
        Added DSC01534-a-2.JPG (ABDC3178-96D3-49FF-82A8-DF8B4AE6FF20) to album ABC
(...)

[RhetTbull]

Thanks @oPromessa for the detailed reply.

@breun The collection phase is slower than I'd like but shouldn't be this slow. I'll see if I can replicate this to get some additional data.

[breun]

@oPromessa Thanks for your reply. The source library is on my girlfriend's computer and still available. Our use case is that every now and then we want to share specific albums, like when we've taken photos on a vacation and we want those pictures in both of our libraries. What method would you recommend for this if not the one I've tried?

[oPromessa]

@oPromessa Thanks for your reply. The source library is on my girlfriend's computer and still available. Our use case is that every now and then we want to share specific albums, like when we've taken photos on a vacation and we want those pictures in both of our libraries. What method would you recommend for this if not the one I've tried?

I'd go with the sync as it will bring all the metadata/changes done on the "source library" into the "destination library". In fact, the intention of sync!

You may include the options "--album VacationsXYZ" on step 1. sync export above to limit the operation to just the "specific albums" you want. You may use multiple --album entries which are treated as OR, so brining pics from several albums into the Metadata.db.

[RhetTbull]

I did some testing with a similar size import and the grouping took 200 seconds. Still far too long but not the 3.5+ hours you experienced. I'll keep digging to see if I can figure out what is going in.