Personal project · Learning
microGPT in C: studying a language model through its code
A learning project based on Vishal Baraiya’s microgpt-c. I added tools to train, save and observe the model.
Where I started
I wanted to follow what happens inside a language model: how text becomes numbers, how attention works and how weights change during training. I started from Vishal Baraiya’s microgpt-c, credited in my repository’s README.
My contribution
I added command-line parameters, model saving and loading, top-k sampling and gradient clipping. A training/validation split and CSV loss exports make it possible to observe the experiment. An interactive mode lets you enter a prefix and read the completion.
What you can check
The repository contains the C code, a Makefile and a small example dataset. The README documents commands for training, saving and reloading weights. This is a learning project: generation on a small dataset does not demonstrate the capabilities of a model ready for a product.
A reproducible run
Run on September 6, 2026 using the public repository: 2,000 steps, seed 42, 112,000 parameters. The dataset contains 5,059 lines, split into 4,553 for training and 506 held out. The program computes validation loss on the first 100 lines of that held-out group.
In this run, validation loss moves from 2.4386 at step 200 to 1.6223 at step 2,000. It measures next-character prediction error on that sample, not the correctness of generated code.
| Step | Validation loss |
|---|---|
| 200 | 2.4386 |
| 400 | 2.1493 |
| 600 | 1.9945 |
| 800 | 1.8687 |
| 1000 | 1.7967 |
| 1200 | 1.7285 |
| 1400 | 1.6904 |
| 1600 | 1.6434 |
| 1800 | 1.6254 |
| 2000 | 1.6223 |
The first three outputs, unselected
int *p = 0;
for (int i = 0; coat = i++)
return *p;The second example contains invalid C code. The test covers one seed and a small dataset: it shows training behaviour, not an ability to program. Saving and loading the weights were verified in the same run.
Reproduce the run
Darwin arm64 · Apple clang version 21.0.0 (clang-2100.0.123.102)
git clone https://github.com/Elgabor/microgpt-c-v2.git
cd microgpt-c-v2
git checkout baf96c2ea79f72781d6d9cc35f0baf79fa86e63f
make
./gpt -s 2000 --seed 42 --val-every 200 -o model.bin --log loss.csv -n 3
./gpt --load model.bin --seed 42 -n 3Compiler and architecture can produce small numerical differences. Loading resets the seed, so generated samples may differ from those at the end of training.