Trouble training on M3 Max

[agoransson]

Hello, I'm trying to run training on my Apple M3 Max but run across problems.

I get the following message when I start, which I interpret as it's failing to use MPS, and just falling back to training on my CPU.

      Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
  0%|          | 0/9 [00:00<?, ?it/s]/Users/agoransson/miniforge3/envs/ultralytics-env/lib/python3.11/site-packages/ultralytics/utils/tal.py:143: UserWarning: MPS: nonzero op is not natively supported for the provided input on MacOS14Falling back on CPU. This may have performance implications.See github.com/pytorch/pytorch/issues/122916 for further info (Triggered internally at /Users/runner/miniforge3/conda-bld/libtorch_1738206065984/work/aten/src/ATen/native/mps/operations/Indexing.mm:309.)
  bbox_scores[mask_gt] = pd_scores[ind[0], :, ind[1]][mask_gt]  # b, max_num_obj, h*w
      1/100      69.1G      1.735      4.129      2.206        356       1280:  67%|██████▋   | 6/9 [23:56<13:06, 262.09s/it]

And sure enough it's taking quite some time to train it seems. When I ran the same training command in the fall, when evaluating my concept initially with some tests, it was training much faster.

Also worth noting is that I can't seem to install the latest Ultralytics package, when starting the training I see:

Ultralytics 8.3.62 🚀 Python-3.11.11 torch-2.5.1 MPS (Apple M3 Max)

I have followed the instructions to create a new conda environment, and installing the required packages (except the CUDA specific installations as I gather those are not needed for me).

Yet, it's so slow. What am I doing wrong? Or is this expected?

engine/trainer: task=detect, mode=train, model=/Users/agoransson/Projects/my_yolo_test/base_models/yolo11m.pt,
data=/Users/agoransson/Projects/my_yolo_test/my_datatset/config.yaml, epochs=100, time=None, patience=100, batch=-1,
imgsz=1280, save=True, save_period=-1, cache=False, device=mps, workers=8, project=None, name=my_dataset_6,
exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False,
rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None,
multi_scale=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False,
save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None,
vid_stride=1, stream_buffer=False, visualize=False, augment=False, agnostic_nms=False, classes=None,
retina_masks=False, embed=None, show=False, save_frames=False, save_txt=False, save_conf=False, save_crop=False,
show_labels=True, show_conf=True, show_boxes=True, line_width=None, format=torchscript, keras=False, optimize=False,
int8=False, dynamic=False, simplify=True, opset=None, workspace=None, nms=False, lr0=0.01, lrf=0.01, momentum=0.937,
weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0,
kobj=1.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0,
flipud=0.0, fliplr=0.5, bgr=0.0, mosaic=1.0, mixup=0.0, copy_paste=0.0, copy_paste_mode=flip, auto_augment=randaugment,
erasing=0.4, crop_fraction=1.0, cfg=None, tracker=botsort.yaml, save_dir=runs/detect/my_dataset_6
Overriding model.yaml nc=80 with nc=11

[UltralyticsAssistant]

👋 Hello @agoransson, thank you for bringing this to our attention and for your detailed report 🚀!

We recommend starting by visiting the Docs for guidance on setup and troubleshooting. Key references include our guides on Python and CLI usage, as well as Tips for Best Training Results.

MPS Performance on macOS

From the logs you've shared, it seems like training is falling back to CPU instead of leveraging MPS on your Apple M3 Max. Since MPS support is still evolving in PyTorch, we recommend verifying that your PyTorch version is compatible with your macOS and hardware architecture. You can refer to PyTorch's MPS Documentation for the latest status.

For slow training issues or unexpected behavior on specific environments, please:

• Share a minimum reproducible example of the code you're using.
• Provide any relevant debug information or logs, including datasets or configurations (e.g., your data.yaml file).

Additionally, as noted in your log (Ultralytics 8.3.62), you may be using an outdated version of Ultralytics. Please upgrade to the latest ultralytics package along with all required dependencies to ensure the issue hasn't already been resolved:

pip install -U ultralytics

Pre-configured Environments

To streamline experimentation, you can also try running YOLO in one of the verified environments below:

• Notebooks:
• Google Cloud: Check the GCP Quickstart Guide
• AWS Deep Learning AMI: Check the AWS Quickstart Guide
• Docker Image: . See the Docker Quickstart Guide

Community and Support

Join the Ultralytics community to share insights or troubleshoot interactively:

• Discord for real-time discussions 🎧
• Discourse for in-depth threads
• Subreddit for collaborative learning

Lastly, please note that this is an automated response, and an Ultralytics engineer will review your query and provide further assistance soon. 🚀

Looking forward to resolving this together!

[Y-T-G]

You can upgrade to MacOS15 or use a smaller batch size, not batch=-1 which is probably selecting a very large batch size.

[agoransson]

Thank you @Y-T-G for the recommendations. Upgrading to MacOS 15 helped get rid of warnings and errors. However, it seemed quite slow anyway, but perhaps that's expected when retraining the Yolo 11 Medium model. It seemed to select 16 as batch size today after the upgrade, I'm unsure if that's large or not.

What is the expected time to train this model?

Starting training for 100 epochs...

      Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
      1/100      70.3G      1.594       3.76      2.062        101       1280: 100%|██████████| 9/9 [25:13<00:00, 168.19s/it]
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95):   0%|          | 0/2 [01:06<?, ?it/s]

The training fails regardless, seems I need to use CPU at least for some operations.

NotImplementedError: The operator 'torchvision::nms' is not currently implemented for the MPS device. If you want
this op to be added in priority during the prototype phase of this feature, please comment on 
https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable 
`PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running
natively on MPS.

[Y-T-G]

MPS is probably slow. You are also using imgsz=1280 which would be slower. But Google Colab would probably train faster than what you're getting here.