HoloLens With DlibFaceLandmarkDetector Example

Demo Video (old version)

Environment

• HoloLens1 10.0.17763.316 / HoloLens2 22621.1399
• Windows 10 SDK 10.0.19041.0 / 10.0.22621.0
• Unity 2019.4.40f1 (Built-in Render Pipeline / LegacyXR / MRTK 2.8.3 / DirectX 11 / Visual Studio 2019 MSVC v142)
• Unity 2021.3.35f1 (Built-in Render Pipeline / OpenXR 1.11.1 / MRTK 2.8.3 / DirectX 11 / Visual Studio 2022 MSVC v143)
• Unity 2021.3.35f1 (Built-in Render Pipeline / OpenXR 1.11.1 / MRTK 3.2.2 / DirectX 11 / Visual Studio 2022 MSVC v143)
• Mixed Reality Feature Tool
• OpenCV for Unity 2.6.3+
• Dlib FaceLandmarkDetector 1.4.0+
• EnoxSoftware/HoloLensCameraStream

Setup (Common)

1. Download the latest release unitypackage. HoloLensWithDlibFaceLandmarkDetectorExample.unitypackage
2. Create a new project. (HoloLensWithDlibFaceLandmarkDetectorExample)
   • Change the platform to UWP in the "Build Settings" window.
3. Import and setup the Microsoft Mixed Reality Toolkit. (Recommend using MixedRealityFeatureTool)
4. Import the OpenCVForUnity.
   • Select MenuItem[Tools/OpenCV for Unity/Open Setup Tools].
   • Click the [Move StreamingAssets Folder] button.
   • Leave the following files and delete the rest. ("StreamingAssets/OpenCVForUnity/objdetect/haarcascade_frontalface_alt.xml", "lbpcascade_ frontalface.xml")
5. Import the DlibFaceLandmarkDetector.
   • Select MenuItem[Tools/Dlib FaceLandmark Detector/Open Setup Tools].
   • Click the [Move StreamingAssets Folder] button.
   • Leave the following files and delete the rest. ("StreamingAssets/DlibFaceLandmarkDetector/sp_human_face_68.dat", "sp_human_face_68_for_mobile.dat", "sp_human_face_17.dat", "sp_human_face_17_for_mobile.dat" and "sp_human_face_6.dat")
6. Clone HoloLensCameraStream repository.
   • Copy the "HoloLensCameraStream/HoloLensVideoCaptureExample/Assets/CamStream/" folder to the "Assets/" folder.
7. Import the HoloLensWithDlibFaceLandmarkDetectorExample.unitypackage.
8. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
9. Configure settings in the "Project Settings" window.
   • Add Define Symbols: the following to Scripting Define Symbols depending on the XR system used in your project; Legacy built-in XR: BUILTIN_XR; XR Plugin Management (Windows Mixed Reality): XR_PLUGIN_WINDOWSMR; XR Plugin Management (OpenXR):XR_PLUGIN_OPENXR.
   • Enable WebCam Capabilties in Publishing settings tab.
10. (Optional) Setup a performance environment suitable for Holorens. (See https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/recommended-settings-for-unity)
11. Build the project: You can now build the Unity project, which generates a Visual Studio Solution (which you will then have to also build). With the Build Settings window still open, click Build. In the explorer window that appears, make a new folder called App, which should live as a sibling next to the 'Assets` folder. Then click Select Folder to generate the VS solution in that folder. Then wait for Unity to build the solution.
12. Open the VS Solution: When the solution is built, a Windows explorer folder will open. Open the newly-built VS solution, which lives in App/HoloLensWithDlibFaceLandmarkDetectorExample.sln. This is the solution that ultimately gets deployed to your HoloLens.
13. Configure the deploy settings: In the Visual Studio toolbar, change the solution platform from ARM to x86 if you are building for Hololens1 or to ARM64 if you are building for Hololens2; Change the deploy target (the green play button) to Device (if your HoloLens is plugged into your computer), or Remote Machine (if your HoloLens is connected via WiFi).
14. Run the app: Go to Debug > Start Debugging. Once the app is deployed to the HoloLens, you should see some confirmation output in the Output window.
    • (Print the AR marker "CanonicalMarker-d10-i1-sp500-bb1.pdf" on an A4 size paper)

• Choosing a Unity version and XR plugin
• Known issues in Unity versions and packages

Setup (HoloLens1 / Unity 2019 (Built-in Render Pipeline / LegacyXR / MRTK 2 / DirectX 11 / Visual Studio 2019))

1. Download the latest release unitypackage.
2. Create a new project. (HoloLensWithDlibFaceLandmarkDetectorExample)
3. Import the Microsoft Mixed Reality Toolkit.
   • Add MRTK2 (Mixed Reality Toolkit - Mixed Reality Toolkit Foundation) to the project using "Mixed Reality Feature Tool".
   • Follow the MRTK2 configuration dialog to set up the project. (XR System: Legacy Built-in XR)
4. Import the OpenCVForUnity.
5. Import the DlibFaceLandmarkDetector.
6. Import the HoloLensCameraStream.
7. Import the HoloLensWithDlibFaceLandmarkDetectorExampleMRTK2.unitypackage.
8. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
9. Configure settings in the "Project Settings" window.
   • Add BUILTIN_XR to the list of Define Symbols.
   • Enable WebCam Capabilties in Publishing settings tab.
10. Build the project; open it in VS2019, set the deployment target to x86 and deploy it to the Hololens1 actual device.

Setup (Unity 2021 (Built-in Render Pipeline / OpenXR / MRTK 2 / DirectX 11 / Visual Studio 2022))

1. Download the latest release unitypackage.
2. Create a new project. (HoloLensWithDlibFaceLandmarkDetectorExample)
3. Import the Microsoft Mixed Reality Toolkit.
   • Add MRTK2 (Mixed Reality Toolkit - Mixed Reality Toolkit Foundation) and OpenXR(Platform Support - Mixed Reality OpenXR Plugin) to the project using "Mixed Reality Feature Tool".
   • Follow the MRTK2 configuration dialog to set up the project. (XR System: OpenXR)
4. Import the OpenCVForUnity.
5. Import the DlibFaceLandmarkDetector.
6. Import the HoloLensCameraStream.
7. Import the HoloLensWithDlibFaceLandmarkDetectorExampleMRTK2.unitypackage.
8. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
9. Configure settings in the "Project Settings" window.
   • Add XR_PLUGIN_OPENXR to the list of Define Symbols.
   • Enable WebCam Capabilties in Publishing settings tab.
10. Build the project; open it in VS2022, set the deployment target to ARM64 and deploy it to the Hololens2 actual device.

• Set up a new OpenXR project with MRTK

Setup (Unity 2021 (Built-in Render Pipeline / OpenXR / MRTK 3 / DirectX 11 / Visual Studio 2022))

1. Download the latest release unitypackage.
2. Create a new project. (HoloLensWithDlibFaceLandmarkDetectorExample)
3. Import the Microsoft Mixed Reality Toolkit.
   • Add MRTK3 (MRTK3 - MRTK Input, MRTK UX Components, MRTK UX Components (Non-Canvas)) and OpenXR(Platform Support - Mixed Reality OpenXR Plugin) to the project using "Mixed Reality Feature Tool".
   • • Set up the MRTK3. (XR System: OpenXR)
4. Import the OpenCVForUnity.
5. Import the DlibFaceLandmarkDetector.
6. Import the HoloLensCameraStream.
7. Import the HoloLensWithDlibFaceLandmarkDetectorExampleMRTK3.unitypackage.
8. Add the "Assets/HoloLensWithDlibFaceLandmarkDetectorExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
9. Configure settings in the "Project Settings" window.
   • Add XR_PLUGIN_OPENXR to the list of Define Symbols.
   • Enable WebCam Capabilties in Publishing settings tab.
10. Build the project; open it in VS2022, set the deployment target to ARM64 and deploy it to the Hololens2 actual device.

• Setting up a new Unity project with MRTK3

Project Assets	Build Settings

Player Settings

ScreenShot (old version)