---
title: "Train LLM — Sequence Length vs Batch Size"
description: "Same 32 tokens, two layouts: see why batch rows never attend to each other, count causal pairs, and what gradient accumulation does and doesn't change."
canonical_url: "https://fanout.sh/ai/lessons/bonus-lessons-seq-len-vs-batch"
md_url: "https://fanout.sh/ai/lessons/bonus-lessons-seq-len-vs-batch.md"
access: "public"
---

# Train LLM — Sequence Length vs Batch Size

Same 32 tokens, two layouts: see why batch rows never attend to each other, count causal pairs, and what gradient accumulation does and doesn't change.

## Public overview

You have 32 token positions to process. You could arrange them as four sequences of eight tokens or two sequences of sixteen. Both layouts contain 32 positions, but a token can read different context in each one.

Batch size counts sequences processed together. Sequence length counts positions in each sequence. In ordinary causal attention, a position can read earlier positions in its own row, including itself, but cannot read another batch row.

Use the numbers 0 through 31 as token IDs. Here they identify positions in an invented stream; we are not tokenizing a real sentence.

Layout First row Second row Where are tokens 7 and 8? --- --- --- --- Four rows of length 8 0 through 7 8 through 15 Different rows Two rows of length 16 0 through 15 16 through 31 Same row

Can token 8 use token 7 as context in each layout?

Call the batch size B and the row length L. The first position can read one position, the second can read two, and so on. One row therefore permits 1 + 2 + … + L = L(L + 1)/2 pairs. Multiply by B to count all rows.

Run this with ordinary Python. It counts relationships in a causal mask; it does not train or time a model.

The longer rows allow more pairs even though both layouts contain 32 positions. A full dense attention-score array has B × L² entries per head: 256 versus 512 here. That count is not a measurement of total GPU memory or runtime. Implementations such as FlashAttention avoid storing the whole score array in high-bandwidth memory.

Change the lengths to [4, 32] . Predict whether 8 can read 7 and how many causal pairs each layout permits before running it.

Accumulation changes the update, not the row

A microbatch is one batch processed in a forward/backward pass. Gradient accumulation combines several such passes before one optimizer update. With full, equally sized sequences:

A is the number of accumulated microbatches; D is the number of data-parallel workers. For B = 4, L = 8, A = 2 and D = 1, an update processes 64 input positions. Each sequence still has length 8. Accumulation does not let its tokens read the other microbatch.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md