Add demo for accurate pointmap-based 3D reconstruction

[codepk37]

This PR adds a new notebook demonstrating 3D reconstruction using depth + camera intrinsics to generate a pointmap, followed by mesh alignment.

Key points

• Generates a metric pointmap from depth and intrinsics.
• Applies transformations to produce meshes with correct scale, position, and orientation.
• Exports meshes for inspection
• Includes a NOCS dataset sample to showcase the workflow.

Usage / Validation:

• Demonstrates correct usage of pointmaps.
• Meshes align with a point cloud generated from the same depth + intrinsics.
• Preserves relative object poses as seen in the online playground.

Visualized scene

Exported mesh preserving relative pose

Objects scale and position consistent with pointcloud

[meta-cla]

Hi @codepk37!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[trungpham2606]

hi @codepk37
for my data, the scale is not correct.

FYI, i was using realsense d435

[codepk37]

Hi @trungpham2606, thanks for checking out notebook
could you share repo having your modified notebook and data used (RGB-D, label image and intrinsic)

It would make easier to reproduce issue and debug

[trungpham2606]

hi @codepk37
here is my code and data:
https://drive.google.com/file/d/1xRzzOS4kzawSrGdlF9-VWqFwwTOSBDKV/view?usp=sharing
you can run it directly (put the code in root code folder, instead of inside the notebook folder).

[codepk37]

Hi @trungpham2606

I checked the reconstruction on my side and the size looks right (for example, the scissor length matches).
The mesh just has wider finger holes than the real scissors

[trungpham2606]

hi @codepk37
i also tested with box shape object and seems the mesh is still bigger than real object.
i think its the limitation of the model.

[trungpham2606]

Hi @gleize can you confirm about my concern ? Tks
The scale does not match the actual size — is that an issue caused by the code, or is it due to the model’s own capability?

[trungpham2606]

hi @codepk37 can you test with this data as well
https://drive.google.com/file/d/1z7HCk1-Txj_U2dp_2_VZ1klpVZ02WHAb/view?usp=sharing
you can see that both pose and scale are not correct now.

[codepk37]

hi @codepk37 i also tested with box shape object and seems the mesh is still bigger than real object. i think its the limitation of the model.

Hi @trungpham2606, this PR provides a reconstruction demo using pointmaps. The code has been verified on multiple datasets, including your first sample. Differences in scale, pose, or shape in new examples likely come from the model itself, which can have limitations, rather than the demo code.

[trungpham2606]

hi @codepk37 i also tested with box shape object and seems the mesh is still bigger than real object. i think its the limitation of the model.

Hi @trungpham2606, this PR provides a reconstruction demo using pointmaps. The code has been verified on multiple datasets, including your first sample. Differences in scale, pose, or shape in new examples likely come from the model itself, which can have limitations, rather than the demo code.

I think so @codepk37 tks for your code.

[Cooper-903]

Hi @codepk37,

I was reading through the function transform_mesh_vertices in the demo notebook, and I think I may have found a potential inconsistency related to the coordinate-frame conversion.

In the code:
vertices_world = tfm.transform_points(vertices)

# convert back to Y-up so GLB is saved correctly
vertices = vertices @ R_zup_to_yup.to(vertices.device)

# remove batch dimension
return vertices_world[0]
It seems that the Y-up conversion (vertices @ R_zup_to_yup) is applied after vertices_world is computed, but the function actually returns vertices_world, not the converted vertices.

Because of that, the returned value is still in Z-up coordinates (i.e., PyTorch3D / pointmap convention), not Y-up. The comment suggests the return should be Y-up, but based on the current code flow, the returned vertices never go through R_zup_to_yup.

So I want to confirm my understanding:
• Is this the intended behavior (returning Z-up)?
• Or should vertices_world also be transformed by R_zup_to_yup before returning?

[codepk37]

Hi @Cooper-903
Good catch yes, the function returns Z-up.
The R_zup_to_yup multiplication is unused, I’ll remove it to avoid confusion.