Train LLM — Sequence Length vs Batch Size

Same 32 tokens, two layouts: see why batch rows never attend to each other, count causal pairs, and what gradient accumulation does and doesn't change.

You have 32 token positions to process. You could arrange them as four sequences of eight tokens or two sequences of sixteen. Both layouts contain 32 positions, but a token can read different context in each one.

Batch size counts sequences processed together. Sequence length counts positions in each sequence. In ordinary causal attention, a position can read earlier positions in its own row, including itself, but cannot read another batch row.

Use the numbers 0 through 31 as token IDs. Here they identify positions in an invented stream; we are not tokenizing a real sentence.

Layout First row Second row Where are tokens 7 and 8? --- --- --- --- Four rows of length 8 0 through 7 8 through 15 Different rows Two rows of length 16 0 through 15 16 through 31 Same row

Can token 8 use token 7 as context in each layout?

Call the batch size B and the row length L. The first position can read one position, the second can read two, and so on. One row therefore permits 1 + 2 + … + L = L(L + 1)/2 pairs. Multiply by B to count all rows.

Run this with ordinary Python. It counts relationships in a causal mask; it does not train or time a model.

The longer rows allow more pairs even though both layouts contain 32 positions. A full dense attention-score array has B × L² entries per head: 256 versus 512 here. That count is not a measurement of total GPU memory or runtime. Implementations such as FlashAttention avoid storing the whole score array in high-bandwidth memory.

Change the lengths to [4, 32] . Predict whether 8 can read 7 and how many causal pairs each layout permits before running it.

Accumulation changes the update, not the row

A microbatch is one batch processed in a forward/backward pass. Gradient accumulation combines several such passes before one optimizer update. With full, equally sized sequences:

A is the number of accumulated microbatches; D is the number of data-parallel workers. For B = 4, L = 8, A = 2 and D = 1, an update processes 64 input positions. Each sequence still has length 8. Accumulation does not let its tokens read the other microbatch.