Bounding box of the annotations?

[fmolivato]

Is there a smart way to extract the bounding box of the annotated objects in the scene?

[George121380]

Hi,
Did you find a method to get the bbox or segmentation maps now? I also need that thank you so much.

[SergioArnaud]

Im also looking into this.

@fmolivato @George121380 did you find out how?

[fmolivato]

Hi @SergioArnaud,
yes, I was finally able to get the bounding boxes.

The process was as follows:

1. Extract the semantic mesh of the scene as point clouds in such a way that the annotated texture color of the vertices was preserved as metadata.
2. Since each object has it's own hex color (listed in the txt file), I used it to isolate relevant points (i.e. points of the target instance, e.g. table) from the others.
3. Fit a box around the remaining points. To have a correctly oriented box, I fitted it along the principal components of the points (hint: you probably only need the rotation on the z-axis).

Hope this helps.

[SergioArnaud]

Hi @fmolivato, this is awesome! Do you happen to have the code of how you created the pointcloud?

Im trying something like the following but seems to be failing in associating the correct color to a given point

def scene_to_sampled_pointcloud(scene, points_per_mesh=1000):
    all_points = []
    all_colors = []

    for name, geometry in scene.geometry.items():
        if isinstance(geometry, trimesh.Trimesh):
            # Sample points on the surface of the mesh
            points, face_indices = geometry.sample(points_per_mesh, return_index=True)

            texture = geometry.visual.material.baseColorTexture.copy().convert('RGB')

            # Get UV coordinates of the face vertices
            face_uvs = geometry.visual.uv[geometry.faces[face_indices]]
            
            # Compute barycentric coordinates for the sampled points
            barycentric = trimesh.triangles.points_to_barycentric(geometry.triangles[face_indices], points)
            
            # Interpolate UV coordinates for the sampled points
            uvs = np.sum(face_uvs * barycentric[:, :, np.newaxis], axis=1)
            
            # Sample colors from the texture
            colors = np.array([
                texture.getpixel((u * (texture.width - 1), v * (texture.height - 1)))
                for u, v in uvs
            ])

            all_points.extend(points)
            all_colors.extend(colors)

I really really appreciate the help!

[fmolivato]

Glad to hear it! 😄

Unfortunately, I'm not allowed to release the open source code until after graduation.

Still, you might want to use the texture vertices themselves as a point cloud.
Something like that should do the trick.