models/gemini-3-pro-preview fails to predict 3D bounding boxes

[marphih]

Description of the bug:

I followed the example notebook Spatial_understanding_3d.ipynb(https://github.com/google-gemini/cookbook/blob/main/examples/Spatial_understanding_3d.ipynb) from the Gemini Cookbook and attempted to predict 3D bounding boxes using models/gemini-3-pro-preview. Despite using the same image and prompts provided in the example, the model only outputs 2D bounding boxes instead of the expected 3D boxes.

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

[kkorpal]

Hi @marphih , thank you for reaching out!
I have successfully reproduced the 3D output using the gemini-3-pro-preview model. Unlike previous models, Gemini 3 treats 3D spatial tasks as a reasoning challenge, so it requires a more explicit prompt to trigger the 3D metric format.
Please try using this updated prompt:
"""
Detect the 3D bounding boxes for the musical instruments.
Return a JSON list where each entry contains:

• "label": The name of the instrument (e.g., 'guitar', 'cajon')
• "box_3d": The 9-parameter metric bounding box [x, y, z, w, h, d, roll, pitch, yaw]
  """
  The coordinates must be in metric units (meters) relative to the camera frame.
  Pro Tip: For the best results with Gemini 3 Pro, ensure your thinking_level is set to high in your configuration to allow the model to accurately calculate depth and orientation.
  Please let us know if this works for you!"