Tensor parallelism vs pipeline parallelism

Tensor parallelism vs pipeline parallelism is a choice about where a model is cut and what must cross the GPU interconnect after that cut.

Tensor parallelism splits work inside every transformer layer. Pipeline parallelism gives whole groups of layers to different GPUs. The better choice depends on memory fit, network topology, latency, and request concurrency.

The short answer is to keep tensor parallel groups inside the fastest interconnect domain. Use pipeline stages when the model must cross slower links, when layer splits are uneven, or when throughput matters more than one-request latency.

Tensor parallelism vs pipeline parallelism in one request

The Megatron-LM paper introduced an efficient intra-layer scheme for transformer models.

Attention heads and MLP matrix multiplications are partitioned across GPUs. Collective operations then combine the partial results.

Every GPU participates in every layer. A token can therefore use several GPUs at once, which can reduce the compute time of that layer when communication stays cheap.

The GPipe paper takes the other cut. It assigns consecutive groups of layers to stages, then sends activations from one stage to the next.

A single token visits those stages in order. GPU 1 cannot run its layers for that token until GPU 0 has produced the boundary activation.

That dependency makes pipeline parallelism less attractive for the latency of one request. With enough independent work, different stages can process different microbatches and keep the pipeline occupied.

The distinction is simple to state but easy to flatten into a misleading rule. TP spends bandwidth to make one layer concurrent. PP spends stage latency to avoid frequent collectives across every layer.

Tensor parallelism pays communication inside each layer

Megatron-style tensor parallelism partitions both attention and MLP operations. Carefully chosen row and column splits avoid communication between paired matrix multiplications, but synchronization is still required at block boundaries.

The later Megatron-LM scaling paper describes two all-reduces in the forward path of a transformer layer for its tensor-parallel formulation.

The payload starts with an activation tensor. For one decode step, a useful approximation is:

Activation bytes = active sequences x hidden width x bytes per element.

Assume 32 active sequences, hidden width 8,192, and BF16 activations at two bytes each.

The payload is 32 x 8,192 x 2 = 524,288 bytes, or 0.5 MiB.

For a ring all-reduce across p ranks, each rank sends and receives about 2 x (p - 1) / p times the payload. With eight ranks, one all-reduce moves about 0.875 MiB per rank.

Two such collectives across 80 layers move about 0.875 x 2 x 80 = 140 MiB per rank for one decode step.

At 30 decode steps per second, the payload alone is about 4.1 GiB per second per rank. Protocol overhead, latency, topology, and the exact engine can make the practical cost higher or lower.

This is not a benchmark result. It is a way to expose the variable that many comparisons hide: TP communication repeats through the model depth.

Pipeline parallelism sends fewer, larger boundaries

Pipeline parallelism keeps each assigned layer whole. During inference, a stage sends its final activation to the next stage rather than synchronizing partial results after each sharded layer.

Use the same 0.5 MiB activation example with eight pipeline stages. One decode step crosses seven stage boundaries, so the aggregate forward payload is about 3.5 MiB across the fabric.

Each boundary link carries about 0.5 MiB for that step. The comparison with TP is not one-for-one because the communication pattern and overlap differ.

PP has traded collective frequency for a longer dependency chain. A token must still pass through all eight stages before the final stage can produce its logits.

If only one request is active, stages wait on one another. If many requests or microbatches are ready, stage 0 can start later work while stage 1 processes earlier work.

The empty slots at startup and drain are pipeline bubbles. Uneven layer costs create more idle time because the fastest stage waits for the slowest one.

The pipeline literature uses microbatch schedules to reduce this loss. In online inference, prompt lengths, output lengths, and arrivals vary, so keeping every stage balanced is harder than in a fixed training batch.

Interconnect topology usually decides first

Tensor parallelism is a good fit when the ranks share a fast fabric such as NVLink or NVSwitch. Its repeated collectives are sensitive to both bandwidth and latency.

Moving that TP group across nodes puts frequent synchronization on the network. A fast GPU can then spend time waiting for partial results rather than running kernels.

Pipeline parallelism usually tolerates slower inter-node links better because only stage-boundary activations cross them. The cost is sequential stage latency and possible bubbles.

Current vLLM parallelism guidance recommends tensor parallelism for a model that fits inside one multi-GPU node.

For a model larger than one node, vLLM recommends tensor parallelism within each node and pipeline parallelism across nodes.

That creates a topology-aware grid. On two nodes with eight GPUs each, TP size eight keeps collectives local while PP size two crosses the node boundary once per stage transition.

The same guide notes an edge case for GPUs without NVLink: pipeline parallelism can have lower communication overhead and higher throughput even within one node.

Topology is therefore not a footnote. Count which exact links each collective or activation transfer will use.

Memory fit is necessary but not sufficient

Both methods divide model weights, so both can make an oversized model fit. The shape of the split differs.

With TP size eight, each rank holds a shard of every compatible layer. With PP size eight, each rank holds full weights for roughly one eighth of the layers.

Real models do not divide perfectly. Embeddings, output heads, tied weights, and layers with different parameter counts can create imbalance.

PP can place different numbers of layers on each stage. TP often requires dimensions such as attention heads or hidden widths to divide cleanly by the group size.

Weight fit is only the first check. A serving process also needs KV cache, activations, CUDA graphs, temporary workspaces, and runtime overhead.

The KV-cache memory formula shows why a configuration that barely fits weights may leave too little memory for useful concurrency.

Measure free cache capacity after the engine loads. More sharding can free memory while still hurting latency through communication or stage waits.

Latency favors TP and throughput can favor PP

For one request on a fast local fabric, TP lets every rank work on the same layer concurrently. That often gives the lower per-token latency.

PP makes that token traverse stages. It needs concurrent requests or microbatches to overlap work and recover utilization.

This leads to a practical split. Interactive services with strict time-to-first-token and inter-token targets often start with TP inside one node.

Offline generation and high-concurrency services can make PP more competitive because they have enough work to fill stages.

Do not turn that tendency into a universal benchmark claim. Layer balance, kernels, quantization, batch size, prompt mix, and scheduler behavior can move the crossover.

The roofline model guide helps separate a compute limit from a bandwidth or communication limit before adding GPUs.

Measure both quiet latency and loaded goodput. A configuration that wins one-request latency can lose completed tokens under a tail-latency objective.

Hybrid parallelism follows the hardware boundary

Large deployments rarely choose only TP or only PP. They combine degrees so that total GPUs equal tensor-parallel size multiplied by pipeline-parallel size.

Suppose a model needs 16 GPUs spread across two eight-GPU nodes. A common starting point is TP 8 and PP 2.

Each pipeline stage spans one node. The eight GPUs inside that node shard its layers with fast local collectives. Activations cross the network between the two stages.

If TP grows across the node boundary, collective traffic uses the slower link many times per layer. If PP grows too large, stage imbalance and bubbles can dominate.

Hybrid parallelism does not remove tradeoffs. It places each communication pattern on the link that handles it best.

Data parallel replicas can then scale request throughput after one model replica has a workable TP and PP layout.

The continuous batching guide explains how request concurrency changes the amount of work available to fill those replicas and pipeline stages.

Choose with a measurement sequence

Start with the smallest distributed layout that fits weights and the required KV-cache capacity.

Keep the first TP group inside one high-bandwidth domain. Record time to first token, inter-token latency, throughput, and collective time.

Then test PP where the hardware boundary or uneven layer split makes it plausible. Record stage utilization, bubble time, boundary-transfer time, and the slowest stage.

Use the same model revision, precision, cache budget, prompt distribution, output distribution, and arrival process.

Test at least three traffic points: one request, expected concurrency, and saturation. TP and PP can change order across those points.

Inspect tail latency rather than only averages. A balanced pipeline under uniform prompts may become uneven when a few long prefills enter the queue.

Choose TP when fast local collectives buy latency. Choose PP when lower communication frequency or a cross-node split outweighs pipeline delay. Combine them when the cluster has both boundaries.

Keep the layout that meets the latency target with the most useful throughput and enough memory headroom.