LoRA & QLoRA

How to Fine-Tune Models

A glimpse of LoRA & QLoRA

A model layer can contain a large matrix of learned weights. Adapting the layer by training a second equally large matrix would require another large set of gradients and optimizer state.

LoRA, short for low-rank adaptation, trains a change built from two smaller matrices. The original weight matrix stays frozen.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.