offset between Unity's and OpenXR's coordinate system

[Birkenpapier]

Hello everyone,

while tinkering with the ResearchMode API for the HoloLens 2 other developers and I encountered and offset of ~5 cm. between both of the coordinate systems.

This issue can be reproduced on several ways:

• one way is to create the point cloud directly on dll-level and then visualize it inside Unity like here: https://github.com/jdibenes/hl2ss
• the other way is to get the scene coordinate system during runtime in Unity and then pass it to an WinRT extension to create the point cloud for the visualization in Unity afterwards: https://github.com/petergu684/HoloLens2-ResearchMode-Unity

More detailed problem descriptions can be found

• here: Getting the WorldOriginPtr with unity > 2021.2 petergu684/HoloLens2-ResearchMode-Unity#52 (comment)
• and here: Unity coordinate system and OpenXR coordinate system drift after sleep/movement on app launch jdibenes/hl2ss#27 (comment)

Is this an/a known issue/bug and if so is there a workaround to align both coordinate systems?
If this is not a bug may I ask for the correct way to align both coordinate systems?

Any help is deeply appreciated!

Thanks to anyone in advance and BR,

Birkenpapier

[rpavlik-bot]

An issue (number 1964) has been filed to correspond to this issue in the internal Khronos GitLab (Khronos members only: KHR:openxr/openxr#1964 ), to facilitate working group processes.

This GitHub issue will continue to be the main site of discussion.

[yl-msft]

@Birkenpapier , a few questions to your scenario i hope you can clarify, and a few i can clarify here.

• For the point cloud in your app, which sensor did you use to create it? Are you using the depth camera stream? or PV camera stream? They have different calibration and represented with different dynamic node Ids.
  To get tracking of dynamic node ids, it's easier to use SpatialGraphNode to track these cameras using the GUID you get from the MediaFoundation apis for the camera.

• If you still want to use winrt api on HL2 for camera tracking, and just want to correlate to the Unity's scene origin, i suggest you use this "PerceptionInterop.GetSceneCoordinateSystem" function to get the winrt object (a SpatialCoordinateSystem object) for your winrt logics.

• The "WorldOriginPtr" is replaced with above API, because the way to get the winrt object is managed differently. OpenXR apps is using LOCAL or UNBOUNDED spaces underlying unity plugin, where the winrt plugin is using StationaryFrameOfReference. If you are not familiar with them it's fine, just knowing that they are different. But the "origin of unity scene" is still the same.

• The "drift" bug over device sleep cycle has been fixed for at least a year ago. If you are using latest HL2 plugins for your app, you should be fine here.

• For the 5cm offset, I've seen something like this in other projects. A wild guess. The MF attribute you get from the camera video stream, should have intrinsic and extrinsic parameters. One of them is an offset of the camera relative to the dynamic node. It's a per device calibration data that's about 5cm if you are using the head tracking camera stream. Could it be that offset?