TorchTitan on TPU

Developers can use it to train and optimize large PyTorch models on Google Cloud TPUs.

Availability

Fork available at https://github.com/google-pytorch/torchtitan/

Pricing model

Not established

Deployment

Adapted to enable model training on Google Cloud TPUs

Privacy

Not established

Licence

Not established

• The TorchTitan TPU fork is available at https://github.com/google-pytorch/torchtitan/. ^[1]

  The fork is available at https://github.com/google-pytorch/torchtitan/

• TorchTitan TPU provides recipes for pre-training large dense models, MoE models, and diffusion transformers. ^[1]

  TorchTitan provides official, battle-tested recipes for pre-training large dense models (such as Llama 3 70B/405B and Qwen 3 8B/32B), Mixture-of-Experts (such as Qwen 3 235B MoE), and diffusion transformers (such as FLUX.1 12B).

• TorchTitan TPU can serve as the scalable trainer inside post-training pipelines such as VeRL for GRPO and RL alignment. ^[1]

  Beyond initial pre-training, TorchTitan serves as the scalable trainer inside post-training pipelines like VeRL for Group Relative Policy Optimization (GRPO) and Reinforcement Learning (RL) alignment.

• The TPU recipes can use SimpleFSDP and Graph Trainer to capture training computations for XLA. ^[1]

  Graph Trainer goes further (--tpu_config.use_graph_trainer) and lets the whole training step, forward, loss, backward and optimizer update, be captured as one program.

• TorchTitan TPU recipes include optional Pallas kernels for attention, fused loss, and MoE expert matrix multiplication. ^[1]

  The TPU recipes replace standard attention with Splash Attention, a Pallas kernel that handles causal, sliding-window and packed-document masks without materializing the attention matrix; replace the final projection and cross-entropy loss with a fused Pallas kernel that never writes the full logits tensor to HBM; and replace MoE expert matmuls with grouped Pallas kernels.

• A toolkit for voice agents that listen, respond, and take action Weekly brief 2026-W43
• Voice agents can take actions during a live conversation Daily brief 2026-10-20