Skip to content

Latest commit

 

History

History
94 lines (78 loc) · 6.7 KB

WebXR-Mixed-Reality-Capture.md

File metadata and controls

94 lines (78 loc) · 6.7 KB
Raw

WebXR Mixed Reality Capture

Using Mixed Reality Capture in WebXR requires different parts to work together. In this page we will go over the different parts and settings to make it work in a project.

General info

Mixed Reality Capture is used to merge between the real world and the virtual world, by positioning a real camera feed inside the virtual environment and displaying the outcome in a spectator display. In this WebXR solution, we made sure that WebXR Export can support Spectator mode and had the code of the Mixed Reality Capture feature in the WebXR Interactions package. A configured prefab can be found in the Sample scene of WebXR Interactions named as SpectatorCameraHolder.prefab.

Expected hierarchy

The current configuration is expected for Mixed Reality Capture to work properly

- SpectatorCameraHolder
  - SpectatorCamera (Components: Camera, SpectatorCamera)
    - StackCameras
      - SpectatorBackgroundCamera (Components: Camera)
      - SpectatorWebcamLightingCamera (Components: Camera)
      - SpectatorForegroundCamera (Components: Camera)
      - Background (Components: MeshFilter with Quad, MeshRenderer with UnlitTransparent Material. Layer: Spectator) 
      - Foreground (Components: MeshFilter with Quad, MeshRenderer with UnlitTransparent Material. Layer: Spectator)
  - MixedRealityCaptureController (Components: MixedRealityCaptureController)
    - CameraPoint (Used as Visual reference for moving point)
    - TopPoint (Used as Visual reference for moving point)
    - BottomPoint (Used as Visual reference for moving point)
  - WebcamHolder
    - WebcamQuad (Components: PlayWebcam, MeshFilter with Quad, MeshRenderer with ChromaKeyUnlit Material. Layer: Webcam)
      - WebcamLightingQuad (Components: MeshFilter with Quad, MeshRenderer with White Legacy Diffuse Material. Layer: WebcamLighting)
    - CameraHint (Used as Visual reference for point on webcam)
    - TopHint (Used as Visual reference for point on webcam)
    - BottomHint (Used as Visual reference for point on webcam)

Layers

The SpectatorCameraHolder.prefab sample prefab uses pre-defined layers:

  • 16 Webcam
  • 17 WebcamLighting
  • 18 Spectator

Configuring Mixed Reality Capture in a project

Some general steps to follow when implementing WebXR Mixed Reality Capture in a project:

  • Import the SpectatorCameraHolder.prefab from the sample. It's better to duplicate it and using the duplicated asset.
  • Add the prefab to the scene with the WebXR Camera Rig (or WebXRCameraSet).
  • If the camera rig is expected to be static, the SpectatorCameraHolder can be at the root of the scene, else it should be a child of the camera rig.
  • Make sure MixedRealityCaptureController is configured correctly.
  • webcamLayer point to the Webcam layer.
  • mixedRealityOnLayers are the layers that the Spectator camera would use during the capture mode, should contain the Webcam and Spectator layers.
  • camerasBase reference to the Cameras Transform in the camera rig hierarchy.
  • cameraFollower reference the CameraFollower Transform in the camera rig hierarchy.
  • xrCameras reference all the XR cameras Camera components in the camera rig hierarchy.
  • spectatorCamera reference the Spectator camera Camera in the SpectatorCameraHolder hierarchy.
  • stackCameras reference the StackCameras GameObject in the SpectatorCameraHolder hierarchy.
  • spectatorBackgroundCamera reference the SpectatorBackgroundCamera Camera in the SpectatorCameraHolder hierarchy.
  • spectatorForegroundCamera reference the SpectatorForegroundCamera Camera in the SpectatorCameraHolder hierarchy.
  • spectatorWebcamLightingCamera reference the SpectatorWebcamLightingCamera Camera in the SpectatorCameraHolder hierarchy.
  • spectatorCameraTransform reference the Spectator camera Transform in the SpectatorCameraHolder hierarchy.
  • spectatorCameraParent reference the SpectatorCameraHolder Transform.
  • backgroundPlaneRenderer reference the Background Renderer in the SpectatorCameraHolder hierarchy.
  • foregroundPlaneRenderer reference the Foreground Renderer in the SpectatorCameraHolder hierarchy.
  • backgroundPlaneTransform reference the Background Transform in the SpectatorCameraHolder hierarchy.
  • foregroundPlaneTransform reference the Foreground Transform in the SpectatorCameraHolder hierarchy.
  • defaultPlaneMaterial default Material used for the background and foreground planes, Unlit Transparent material.
  • webcamParent reference the WebcamHolder Transform in the SpectatorCameraHolder hierarchy.
  • webcam reference the WebcamQuad PlayWebcam in the SpectatorCameraHolder hierarchy.
  • calibrationPointCamera reference the CameraPoint Transform in the SpectatorCameraHolder hierarchy.
  • calibrationPointTop reference the TopPoint Transform in the SpectatorCameraHolder hierarchy.
  • calibrationPointBottom reference the BottomPoint Transform in the SpectatorCameraHolder hierarchy.
  • calibrationHintCamera reference the CameraHint GameObject in the SpectatorCameraHolder hierarchy.
  • calibrationHintTop reference the TopHint GameObject in the SpectatorCameraHolder hierarchy.
  • calibrationHintBottom reference the BottomHint GameObject in the SpectatorCameraHolder hierarchy.
  • leftController reference the left WebXRController in the camera rig hierarchy.
  • rightController reference the right WebXRController in the camera rig hierarchy.
  • webcamFramesDelaySize set the default number of delayed frames to store - Used for cases of Webcam image slower than VR movement.
  • Make sure that the Main Camera and XR Cameras hide or show the correct layers, the Spectator, Webcam and WebcamLighting should be hidden.
  • Make sure that the Spectator camera shows the Spectator and Webcam Layers and other layers that the project should display.
  • Make sure that the Spectator camera hides the WebcamLighting layer.
  • Make sure that the background and foreground cameras hide the Spectator, Webcam and WebcamLighting layers.
  • Make sure that the background and foreground cameras show all other layers that the project should display.
  • Make sure that the SpectatorWebcamLightingCamera only shows the WebcamLighting layer.

That should handle all the configurations needed for Mixed Reality Captuer to work properly.

Enable Mixed Reality Captuer on runtime

Once the project runs, for the feature to be enabled, both the SpectatorCamera and MixedRealityCaptureController should be set to be enable in XR. Both components contains EnableInXR(bool) method to enable their functionality when user is switching to XR mode.

There are more customization and settings that can be handle by scripts like Chroma Key min and max range, check the MixedRealityCaptureController and PlayWebcam code for those options.