CPU Mode Request

[djbyrne]

Hi all, just getting in to using Torch Titan and have really loved it! One thing I personally would find useful is the ability to do small day to day development on my laptop in a CPU mode. I realize that TorchTitan is a distributed training repo, but I think a lot of researchers would still find a CPU dev/debug mode useful (VMs are expensive for just tracking down my latest brand of random bugs that I introduce to my code base 😅 )

Would there be an appetite for cpu only compatibility? Happy to make a PR as I will be doing this for my own fork.

Thanks,

Donal

[fegin]

@djbyrne Are you trying to do distributed training on CPU or just single device run? The later should be possible by disabling FSDP (dp_shard_degree=1) and having some tweaks to ensure the device is CPU. The former is more tricky as PTD's parallelisms are not friendly to CPU.