How do you turn an LLM experiment into a publishable AI research paper?
Build an LLM, compare Muon and Adam, structure experiments, and turn the evidence into a publishable AI research paper.
A research paper makes a claim and gives other people enough evidence to check it. Code, plots and polished writing help only when they support that claim.
The video takes a public claim that Muon beats Adam, turns it into a fair comparison, audits the original headline, then writes up and publishes the result. We can practice the same workflow with a much smaller question: on one small regression task, after equal numbers of training steps, how do AdamW and a Muon-based optimizer compare?
That question names the task, comparison and budget. It does not promise a new best optimizer.
Decide what the result would mean
Our model receives two numbers and predicts their product. It has one hidden layer with eight units. The loss is mean squared error: average the squared differences between predictions and correct products. Lower is better.
AdamW adjusts parameters using moving estimates of gradients and their squared values. Muon applies a matrix operation to a momentum update before changing the weights. Momentum carries information from earlier gradients into the current update. We use the library implementation here; the original Muon explanation derives its matrix operation if you want to inspect it.
Muon is intended for appropriate matrix weights, with other parameters handled separately. Here, “Muon hybrid” means Muon on the hidden weight matrix and AdamW on its bias and the output layer. That distinction must appear in a methods section; calling it simply “Muon on the model” hides part of the experiment.
Before running, fix these choices:
| Choice | Our pilot |
|---|---|
| Data | 64 training, 32 validation and 32 test examples |
| Training budget | 80 full-batch updates per run |
| Tuning budget | Two candidate learning rates per method, two seeds each |
| Final comparison | Three new seeds, using the selected rates |
| Other settings | Same model, data and zero weight decay; hybrid's AdamW rate fixed at 0.01 |
Matching seeds makes the two methods start from the same weights within a pair. Validation data chooses learning rates. Test data is consulted only after those rates are locked. The candidate ranges below are a small demonstration, not an exhaustive or equally optimal search for each method.
Run an experiment small enough to audit
This CPU program was checked with PyTorch 2.14. It needs a PyTorch release containing torch.optim.Muon; it downloads no dataset or pretrained weights.
In the checked run, validation selected 0.01 for AdamW and 0.02 for the Muon hybrid. The held-out results were:
| Initialization seed | AdamW test loss | Muon hybrid test loss |
|---|---|---|
| 10 | 0.1810 | 0.2593 |
| 11 | 0.4916 | 0.4409 |
| 12 | 0.2902 | 0.3463 |
The mean losses are about 0.3209 and 0.3489. AdamW has the lower mean in this pilot, but the direction reverses for seed 11. Small numerical differences across library versions are possible; save the full values from your own run.
Which statement can these results support?
- “AdamW is better than Muon.”
- “With these data, settings and three initializations, AdamW had lower mean test error after 80 updates.”
If you change the learning-rate grid after looking at test scores, that set has started influencing your decisions. Treat the changed run as further development and reserve a new independent test for a final claim. Save unsuccessful runs too; keeping only the winning seed makes a method look more reliable than it was.
Turn the result into something another person can check
Start with a figure or table that answers the question. For this pilot, the three paired losses are more informative than a single bold average. A training-curve plot would need labeled axes, the comparison budget and a description of what its variation represents. Equal steps and equal wall time answer different questions.
Draft a two-sentence results paragraph that gives the observed comparison and its limits. Then compare it with this worked example:
Then write the method needed to reproduce it: data construction and split, model, parameter groups, optimizer settings, candidate rates, seeds, stopping rule and software version. Keep the executable script and complete run records alongside the figure. The introduction can now explain why the question mattered; related work explains how it fits existing experiments. An abstract should summarize thAudit a published comparison: open Fantastic Pretraining Optimizers and Where to Find Them, the related work the video cites. Find how it gives each optimizer its own hyperparameter search, and at which point in training it compares them. Then compare its reported speedup at its smallest and largest model sizes with the scope of the pilot statement you wrote above. A paper and a public repository are useful materials to inspect; neither alone verifies a result.d document are useful materials to inspect; neither alone verifies a result.
Your pilot is practice, not yet a novel research contribution. A next research question might test whether the observed difference survives different data splits or compute budgets. Read the closest prior work before claiming that experiment is new. For a paper submission, use the chosen venue's current instructions, disclose limitations and obtain coauthor review. A public preprint shares a manuscript; it is not the same as peer-reviewed acceptance.
- Start with a narrow question, a baseline, a metric and an explicit comparison budget.
- Select settings on validation data; keep final test data out of development decisions.
- Match initializations within comparisons and report variation, not only a winning seed.
- State Muon's parameter groups and the accompanying AdamW updates; equal steps do not establish equal wall time.
- Save code, configurations, complete run records and software versions. Write claims no broader than that evidence.
- A reproducible pilot is practice; a research contribution also needs context in prior work and appropriately reviewed claims.