Posts 1
A Visual Guide to Parallel Transformers
Illustrating how a transformer’s forward and backward passes are sharded across devices — data, tensor, context, pipeline, and expert parallelism, in the notation of the JAX scaling book.
Illustrating how a transformer’s forward and backward passes are sharded across devices — data, tensor, context, pipeline, and expert parallelism, in the notation of the JAX scaling book.