Missing metadata [spatial photos]

[JoanCharmant]

Describe the bug

I am loading an .HEIC "spatial photo" file coming from the VisionOS and I cannot see the relevant metadata in the metadata array or anywhere else.

Steps/Code to Reproduce

Here is my code:

if pillow_heif.is_supported("IMG_0009.HEIC"):
        heif_file = pillow_heif.open_heif("IMG_0009.HEIC")

Here is an example file:
IMG_0009.zip

Expected Results

There is metadata related to the camera extrinsics and intrinsics. They can be seen in the "info" box on the Mac, but they are nowhere to be seen from pillow-heif as far as I can tell. The Mac and AVP are using this to flag the files as "spatial photos" (stereoscopic image).

Expected: inspecting the file variable the metadata should be accessible somewhere.

Saving the file right after opening should produce a file with similar metadata as the original.

Actual Results

I can see info.exif, info.xmp but info.metadata is an empty array. I believe the metadata should be there.

Versions

3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)]
Windows-10-10.0.19045-SP0
0.16.0
{'libheif': '1.17.6', 'HEIF': 'x265 HEVC encoder (3.4+31-6722fce1f)', 'AVIF': 'AOMedia Project AV1 Encoder v3.8.2', 'encoders': {'x265': 'x265 HEVC encoder (3.4+31-6722fce1f)', 'aom': 'AOMedia Project AV1 Encoder v3.8.2', 'mask': 
'mask'}, 'decoders': {'libde265': 'libde265 HEVC decoder, version 1.0.15', 'aom': 'AOMedia Project AV1 Decoder v3.8.2'}}

[bigcat88]

It is available.
Take a look at heif_dump_info.py at the examples folder if you want to use it not in Pillow mode.

Mime: image/heic
Number of images: 2
Information about each image:
        Mode: RGB
        Depth: 8
        Alpha: False
        Size: (2560, 2560)
        Data size: 19660800
        Stride: 7680
        Thumbnails: []
        ICC: Display P3
        Exif:
                0th:
                        Make: b'Apple'
                        Model: b'Apple Vision Pro'
                        Software: b'1.1.2'
                        DateTime: 24 bytes.
                        ExifTag: 142
                        GPSTag: 462
                Exif:
                        ExposureTime: (329, 500000)
                        FNumber: (2, 1)
                        ISOSpeed: 49
                        ShutterSpeedValue: (147330, 13939)
                        ApertureValue: (2, 1)
                        ExposureBiasValue: (0, 1)
                        FocalLength: (11706, 19645)
                        MakerNote: 56 bytes.
                        PixelXDimension: 2560
                        PixelYDimension: 2560
                        FocalLengthIn35mmFilm: 18
                        LensMake: b'Apple'
                        LensModel: 46 bytes.
                1st:
                GPS:
                        GPSLatitudeRef: b'N'
                        GPSLatitude: ((0, 1), (0, 1), (0, 100))
                        GPSLongitudeRef: b'E'
                        GPSLongitude: ((0, 1), (0, 1), (0, 100))
                        GPSAltitudeRef: 0
                        GPSAltitude: (0, 1)
                Interop:
                thumbnail:
        Xmp size: 720

        Mode: RGB
        Depth: 8
        Alpha: False
        Size: (2560, 2560)
        Data size: 19660800
        Stride: 7680
        Thumbnails: []
        ICC: Display P3
        Exif:
                0th:
                        Make: b'Apple'
                        Model: b'Apple Vision Pro'
                        Software: b'1.1.2'
                        DateTime: 24 bytes.
                        ExifTag: 130
                Exif:
                        ExposureTime: (329, 500000)
                        FNumber: (2, 1)
                        ISOSpeed: 49
                        ShutterSpeedValue: (147330, 13939)
                        ApertureValue: (2, 1)
                        ExposureBiasValue: (0, 1)
                        FocalLength: (11706, 19645)
                        MakerNote: 56 bytes.
                        PixelXDimension: 2560
                        PixelYDimension: 2560
                        FocalLengthIn35mmFilm: 18
                        LensMake: b'Apple'
                        LensModel: 47 bytes.
                1st:
                GPS:
                Interop:
                thumbnail:
        Xmp size: 720

Added: And it saves by default with the metadata, I just checked on image provided from you.

[JoanCharmant]

Thanks. I ran the example. I had to comment out the XMP block otherwise I get AttributeError: module 'pillow_heif' has no attribute 'getxmp'

I get the same result as you. This is not all the metadata. It's still missing the part about camera intrinsics and pinhole model.

Here is a screenshot of the infobox in Finder. There is an HEIF section with this metadata that I can't find anywhere when loading via python. The files saved are missing this specific block.

[bigcat88]

it looks like this information is not contained in usual EXIF or XMP

I just checked with libheif-info utility, both EXIF and XMP are saved sucesfully with the same size as in original, but as you described those additional information is still missing.

I will create issue in libheif repository with link to this one.

[jwheeler-work]

I'm also trying to get this to work.

This may help. Here is an example of a the extra data added when creating these images in Xcode.

let properties = [ kCGImagePropertyGroups: [ kCGImagePropertyGroupIndex: 0, kCGImagePropertyGroupType: kCGImagePropertyGroupTypeStereoPair, kCGImagePropertyGroupImageIndexLeft: 0, kCGImagePropertyGroupImageIndexRight: 1, ], kCGImagePropertyHEIFDictionary: [ kIIOMetadata_CameraModelKey: [ kIIOCameraModel_Intrinsics: cameraIntrinsics as CFArray ] ] ]

[bigcat88]

As soon as someone knows the format of this data, I invite you to share the knowledge gained with the author of the libheif in the linked issue.
After adding support for this to libheif, we can immediately add support for it here.

[TimYao18]

Hi, I have the same issue that if I opened and saved the heic, the information will be missing. And I don't know how to save the same image quality that the heic will become very large if I ran the below codes:

if pillow_heif.is_supported("IMG_0009.HEIC"):
        heif_file = pillow_heif.open_heif("IMG_0009.HEIC")
        heif_file.save("output.heic", quality=-1)

Is there some walkaround to get the whole heic's information?

[bigcat88]

Is there some walkaround to get the whole heic's information?

HEIC files does not contain "quality" value saved anywhere, take a look at this my reply with link to libheif issue about this: #25 (comment)

Hi, I have the same issue that if I opened and saved the heic, the information will be missing

As to Spatial Photos theme - their support will be added as soon as libheif will implement it.

[bigcat88]

Here is an example file:
IMG_0009.zip

Currently I am trying to add support for camera_intrinsic_matrix values, can I add this file to the tests to this repo?

[bigcat88]

Here is an example file:
IMG_0009.zip

Currently I am trying to add support for camera_intrinsic_matrix values, can I add this file to the tests to this repo?

cc @JoanCharmant

[JoanCharmant]

Yes you can add the file to the repo.