Bug: Incorrect Rotational Pose in Scene Reconstruction & Request for Method Clarification

[lzlfwow]

Hi team,

Thank you for the fantastic work on this project. I am attempting to reconstruct the complete 3D scene using the outputs of your pipeline (mesh.glb and parameters.json).

I have successfully developed a script that loads each object and applies its transformation. The results are promising: the approximate translation and scale of the objects in the scene appear plausible. However, I am facing a persistent issue where the rotational pose of every object is incorrect. The objects are in the right general location and size but are not oriented correctly.

My Reconstruction Approach

My method is to construct a 4x4 transformation matrix for each object from its parameters.json file and apply it to the mesh. The core of my logic, especially for the rotation, is shown in the Python function below. The final matrix is composed as Transformation = Translation @ Rotation @ Scale.

Code Snippet

This function shows exactly how I am interpreting the 6drotation_normalized parameter to generate a rotation matrix:

import numpy as np

def get_transformation_matrix(params):
    """
    Builds a 4x4 transformation matrix from the parameters dictionary.
    """
    # 1. Extract data, accounting for the [[[-1.2, ...]]]] nesting
    scale_vec = np.array(params['scale'])[0]
    rot_6d = np.array(params['6drotation_normalized'])[0][0]
    trans_vec = np.array(params['translation'])[0]

    # 2. Reconstruct the 3x3 rotation matrix from the 6D vector
    # Assumes the 6D vector represents the first two columns of the rotation matrix
    a1 = rot_6d[:3]
    a2 = rot_6d[3:]
    
    # Gram-Schmidt orthogonalization
    b1 = a1 / np.linalg.norm(a1)
    b2_orthogonal = a2 - np.dot(b1, a2) * b1
    b2 = b2_orthogonal / np.linalg.norm(b2_orthogonal)
    b3 = np.cross(b1, b2)
    
    rot_mat = np.stack([b1, b2, b3], axis=1)

    # 3. Build individual 4x4 transformation matrices
    S = np.diag([*scale_vec, 1.0])
    M_rot = np.eye(4)
    M_rot[:3, :3] = rot_mat
    M_trans = np.eye(4)
    M_trans[:3, 3] = trans_vec

    # 4. Combine transformations: T * R * S
    transform_matrix = M_trans @ M_rot @ S
    return transform_matrix

Request for Clarification

Since the primary error appears to be rotational, I suspect a subtle misunderstanding in my interpretation. In your paper, you mention that you implemented a scene reconstruction. To help resolve this, I have two questions:

1. Reference Implementation: Would it be possible to share the script or a code snippet that demonstrates your scene reconstruction implementation? Having a reference for how you interpret the pose parameters would be the most direct way to solve this issue.

2. Intended Workflow: If sharing the code is not possible, could you please clarify the intended workflow for reconstructing the entire scene using your outputs? Specifically, is there a problem with my overall approach?

   • Full Transformation Formula: Is my method—applying a single T @ R @ S matrix derived directly from parameters.json—the complete and correct approach? Or am I missing a crucial step, such as an additional transformation (e.g., a camera-to-world matrix) that needs to be applied?
   • Parameter Interpretation: How should 6drotation_normalized be correctly converted into a rotation matrix, and what are the underlying coordinate system conventions (+Y up, right-handed, etc.)?

Any guidance you can provide on the correct way to transform the objects from their canonical space into the final scene would be incredibly helpful.

Thank you for your time and assistance

[niuxiaozhang]

I think you should use the update_layout interface to update trans, scale, and rotation when calculating the transformation matrix.

trans = out["translation"] # torch.Size([1, 3])
scale = out["scale"] # torch.Size([1, 3])
rotation = out["rotation"] # torch.Size([1, 4])

trans, scale, rotation = update_layout(trans, scale, rotation, center=None, scale=0)

Convert quaternion to rotation matrix (PyTorch3D format)

R_l2c = quaternion_to_matrix(rotation) # torch.Size([1, 3, 3])

Create transform using PyTorch3D logic (Scale -> Rotate -> Translate)

Returns a Transform3d object

tfm = compose_transform(scale=scale, rotation=R_l2c, translation=trans)
"""
M = [
[Rxx, Ryx, Rzx, 0],
[Rxy, Ryy, Rzy, 0],
[Rxz, Ryz, Rzz, 0],
[Tx, Ty, Tz, 1],
]
"""
M = tfm.get_matrix()[0] # torch.Size([4, 4])

"""
M_trimesh = [
[Rxx, Rxy, Rxz, Tx],
[Ryx, Ryy, Ryz, Ty],
[Rzx, Rzy, Rzz, Tz],
[ 0, 0, 0, 1]
]
"""
M_trimesh = M.T.cpu().numpy() # torch.Size([4, 4])

[yyyysr]

@niuxiaozhang

Hi, thank you for your help so far.

I built the full 3D scene mesh following the suggested workflow, but I noticed that the mesh coordinates appear inconsistent with the original point cloud (as shown in the Blender rendering below, especially when checking the coordinate axes in the top-right corner of the view).

Could you please help me check if there are any errors or improper usage in the code below?

outputs = [inference(image, mask, seed=42) for mask in masks]


GLB_PATHS = os.path.join(PATH, "glbs", IMAGE_NAME)
GLB_JSON_PATHS = os.path.join(PATH, "glbs", IMAGE_NAME, "metadata.json")
os.makedirs(GLB_PATHS, exist_ok=True)

metadata = {}

for i, out in enumerate(outputs):
    scale = out["scale"] # torch.Size([1, 3])
    rotation = out["rotation"] # torch.Size([1, 4])
    translation = out["translation"] # torch.Size([1, 3])
    
    metadata[i] = {
        "scale": scale.squeeze(0).cpu().numpy().tolist(),
        "rotation": rotation.squeeze(0).cpu().numpy().tolist(),
        "translation": translation.squeeze(0).cpu().numpy().tolist(),
    }
    
    glb = out["glb"]
    glb.export(os.path.join(GLB_PATHS, f"{i}.glb"))
    print(f"Saved object {i} to {os.path.join(GLB_PATHS, f'{i}.glb')}")

import json
with open(GLB_JSON_PATHS, "w") as f:
    json.dump(metadata, f, indent=4)
print(f"Saved metadata to {GLB_JSON_PATHS}")

with open(METADATA_PATH, "r") as f:
    metadata = json.load(f)

meshes = []

for i in range(len(metadata)):
    obj = metadata[str(i)]
    glb_path = os.path.join(GLB_DIR, f"{i}.glb")
    if not os.path.exists(glb_path):
        print(f"Object {i} not found, skip")
        continue

    mesh = trimesh.load(glb_path)
    if mesh.is_empty:
        print(f"Empty mesh for {i}, skip")
        continue

    trans = torch.tensor(obj["translation"], dtype=torch.float32).reshape(1, 3)
    rotation = torch.tensor(obj["rotation"], dtype=torch.float32).reshape(1, 4)
    scale = torch.tensor(obj["scale"], dtype=torch.float32).reshape(1, 3)
    trans, scale, rotation = update_layout(trans, scale, rotation, center=None, scale=0)
    R_l2c = quaternion_to_matrix(rotation) # torch.Size([1, 3, 3])
    tfm = compose_transform(scale=scale, rotation=R_l2c, translation=trans)
    M = tfm.get_matrix()[0] # torch.Size([4, 4])
    M_trimesh = M.T.cpu().numpy() # torch.Size([4, 4])

    if isinstance(mesh, trimesh.Scene):
        for m in mesh.geometry.values():
            m.apply_transform(M_trimesh)
            meshes.append(m)
    else:
        mesh.apply_transform(M_trimesh)
        meshes.append(mesh)

# merge all meshes
if not meshes:
    raise RuntimeError("No valid meshes to export.")

scene = trimesh.Scene()
for idx, m in enumerate(meshes):
    scene.add_geometry(m, node_name=f"obj_{idx}")

scene.export(OUTPUT_PATH)
print("Exported full scene to", OUTPUT_PATH)

input image

point cloud

mesh

[codepk37]

Hi #79 , might help you to solve issue of correctly aligning mesh

[yyyysr]

@codepk37 Thank you so much for the code! I tried your PR, using an absolute depth npy file, but it doesn't seem to work with the alignment. If it's convenient for you, can you help me see exactly what the problem is?

The input images, camera instrinsics and depth are stored in the test_gym.

[saikiran321]

@yyyysr I have bundled all the changes of @codepk37 to a single script, It should be compatible with above script. I was able to get good alignment during my tests

reconstruct.py

[yyyysr]

@saikiran321 Thanks for your reply. If you look at the test_gym example I provided, you can see that the object’s point cloud is correctly constrained inside the room. However, when I use @codepk37 's code to rieve the object asset, the resulting object still appears outside the room. In addition, when the object is supposed to face upward, its z-axis ends up pointing downward (as seen in the Blender view).

So I am wondering whether codepk37’s code only recovers the object’s relative position rather than aligning it to the absolute depth. Or could this be an issue with the data I provided? I’m not sure why the alignment becomes incorrect like this.

[codepk37]

Hi! @yyyysr the link requires access, so I’m unable to view it.

[yyyysr]

@codepk37 Sorry! My fault!

I’ve already enabled access. Alternatively, you can use this new link:
https://drive.google.com/drive/folders/1FYCw8xglfr8QpW8k4ELwMJzZEjVkDb_o?usp=sharing

[codepk37]

Hey @yyyysr its correctly aligning with pointcloud made using depth+intrinsic

[gaobiaoli]

Hi, thanks for the great work!

I am currently trying to import the predicted GLB files and their corresponding pose information (translation, rotation, scale) into Unity3D to perfectly simulate the spatial relationship between the objects and the camera.

The Problem: I am struggling to find the correct coordinate transformation for the Rotation.

Translation: I found that simply inverting the X-axis (Right-handed to Left-handed conversion: x_unity = -x_pred) makes the position perfectly correct relative to the camera.

Rotation: However, no matter what coordinate transformation I attempt for the rotation (Quaternion), the orientation of the object remains incorrect.

What I have tried:

Using the raw output directly.

Using the provided update_layout functions.

Any advice or snippets on mapping this rotation to Unity would be greatly appreciated.

Thanks!

[yyyysr]

@codepk37 Thank you so much for the verification! It does align perfectly with the depth map. I mistakenly observed the empty room GLB as an absolute position. Sorry to bother you!

[codepk37]

@yyyysr Glad it’s sorted out! I’m happy it helped.