Training Ari

How a LoRA gets trained and back into XerisOS. Written down because the procedure was reconstructed once from a leftover pod's directory listing, and nobody should have to do that twice.

Scope: fine-tuning an existing checkpoint. Training a base model from zero is a multi-GPU, multi-day job and is out of scope here — see MODELS.md.

0. You may not need a rented GPU#

train_lora.py is a QLoRA recipe built for an 8 GB card: base weights frozen and quantized to 4-bit NF4 (~3.6 GB), rank-16 adapters on every block Linear, gradient checkpointing, bf16 autocast. Its own docstring puts it at roughly 6 GB of VRAM.

If your desktop has an 8 GB GPU, train locally and skip a 13 GB upload entirely. Rent a card when you want it faster, not because it won't fit.

1. Build the corpus#

python tools/make_xeris_corpus.py --root . --out xeris_corpus.jsonl --merge combined_mix.jsonl

Mines the source tree into the same {"text": "\n\nUser: …\n\nAri: …"} format as combined_mix.jsonl and appends the existing mix. Five kinds: tool (reach for run: grep instead of recalling), write (comment → function, which teaches house style), explain, locate, doc.

Two design decisions in there worth knowing, because they are easy to undo by accident:

• "Where is X?" answers with run: grep X, not a line number. The grep answer is still correct after the next commit; a line number is not. Training in facts with a one-commit shelf life is worse than not training them.

• No category may exceed a third of the mined data. The first version had locate at 67% and tool at 0.05%, because locate is the cheapest kind to generate. Left alone, the model learns the shape of the cheap category instead of the content of the useful one.

• Tool verbs are trained as a full round trip, not just the call. Every verb example used to be one turn — question, run: X, end. Measured on the shipped corpus: screen, launch, undo, remember, build, find, lines, close and focus had zero examples between them of the turn that comes after the call, while eval (800) and read (2,009) had nothing but. That is the whole explanation for why those two behaved and the rest did not: the tool fired, ag_exec returned real text, and the model had no template for what to do with it, so it re-called the tool or drifted. Sections 4d/4e now emit three-turn examples for every verb.

  Two rules govern them, and breaking either is worse than not training at all:

  1. The result string must be what ag_exec really writes, "tool result: " prefix included — that prefix is prepended to every result, and ari_agent_run splices the whole thing back as a user turn. A made-up result format trains the model to match a cue the OS never emits.
  2. The payload must be real. The grep and lines windows are computed from actual files, so the file name, the line number and the code text agree with each other. Only files whose stem is ≤ 8 characters are used (fs_dirent.name is the FAT short name, so anything longer comes back as MCRAFT~1.C), names that occur twice in the tree are skipped, and upstream QuickJS is skipped — it is 60k lines, it would dominate any sample, and it is not on the volume anyway.

  python3 tools/verify_tool_turns.py re-checks both rules against the tree and fails loudly. Run it after any edit to sections 4d/4e.

  Failure results are trained as deliberately as successes: a bad launch replies with the real app list, a grep for a name that does not exist replies no matches, and those are exactly the turns where the model has to read the result back and recover instead of inventing.

Check which combined_mix.jsonl you merged — there is a copy in the repo root and a slightly newer one in the parent folder. They are not the same file.

2. Get it onto the pod#

runpodctl moves files peer-to-peer with a one-time code (croc underneath, no API key):

# on the PC
runpodctl send bundle.tgz
# output: code is: 8338-galileo-collect-fidel

# on the pod
runpodctl receive 8338-galileo-collect-fidel
tar xzf bundle.tgz

Tar the small files together — one send, one code to type. The checkpoint goes on its own because it is 13 GB and you do not want to redo the whole transfer if it drops.

What has to be present, in one folder:

3. Train#

pip install --index-url https://download.pytorch.org/whl/cu124 torch
pip install -U bitsandbytes

python train_lora.py \
    --data   xeris_corpus.jsonl \
    --ckpt   ari_6.6b_code_v5.pt \
    --tok    xeris_tokenizer_v2.json \
    --newtok newtokens.json \
    --out    lora_xeris_v1.pt \
    --cache  tokens_cache_xeris.pt \
    --seq    2048

Smoke test first, before the meter is running:

python train_lora.py --data xeris_corpus.jsonl --tiny --steps 20

Runs the whole pipeline against a random tiny model on CPU. Catches a malformed corpus or a tokenizer mismatch in a minute instead of an hour.

You can interrupt at any point — the most recent lora_*.pt save is usable.

Five departures from the defaults, all deliberate#

--tok — THE trap. The default is xeris_tokenizer.json. The pod carries xeris_tokenizer_v2.json. Train with the wrong one and every token id is wrong: the run completes, the loss looks vaguely plausible, and the output is garbage. Nothing warns you.

The tokenizer must match the checkpoint. v2 shipped alongside code_v5, so v2 goes with the newer checkpoints — but confirm rather than assume, because the vocab is extended at load time (XerisTokenizer adds domain tokens) and so the raw JSON entry count does not settle it on its own. The deployed model declares vocab 30,667 in ari_0.bin's header; the tokenizer that produces that number is the right one.

--ckpt defaults to ari_6.6b_base_sft_v13-001.pt. Point it at the newest checkpoint instead so domain knowledge stacks on top of code ability rather than replacing it.

--cache defaults to tokens_cache.pt, and the reuse guard is mtime(cache) > mtime(data) — correct, but it only knows about one data file. A per-dataset name removes the question entirely. That is why the pod has tokens_cache_v3.pt and not tokens_cache.pt.

--newtok — the trap that cost a whole run. It defaults to OFF, and leaving it off silently produces a model that cannot write C.

_ [ ] { } * + = / < > & | % ^ ~ # \ @ $ are 23 token ids the model owns but never saw in pre-training. tools/tok_patch.py assigned them; tools/newtok.py seeds each from a look-alike (_ from -, { from () and trains a 94K-param delta over just those rows. It is a separate flag because LoRA cannot reach them: train_lora.py adapts attn.qkv, attn.out_proj, ff.w1/w2/w3, and line 171 leaves "embeddings / norms / tied head" frozen. No number of steps on any corpus will teach a frozen embedding to emit a character.

Run without it and the new rows stay at their seed. Measured on the v6 checkpoint: cosine(_, -) = 0.88, where two ordinary tokens average 0.20 and never exceeded 0.27 over 400 sampled pairs. The head is weight-tied to tok_emb, so those near-identical vectors also decide the output logits — _ and - come out near coin-flip. That is why the model answered "create-window is in app.js": it is not confused about the codebase, it physically cannot spell the identifier. Every C operator collapses onto -, and {/} onto (/).

Confirm it took, before exporting 4 GB:

# rows must move, and move AWAY from the seed
import torch, json
a = torch.load('ari_6.6b_code_v5.pt', map_location='cpu')['model_state_dict']['tok_emb.weight']
b = torch.load('ari_6.6b_xeris_v6.pt', map_location='cpu')['model_state_dict']['tok_emb.weight']
for t in json.load(open('newtokens.json'))['new'][:5]:
    i, s = t['id'], t['seed_id']
    cos = torch.nn.functional.cosine_similarity(b[i].float(), b[s].float(), 0)
    print(t['token'], 'changed:', not torch.equal(a[i], b[i]), ' cos(tok,seed): %.3f' % cos)

changed: False, or a cosine near 0.88, means the flag did not take. It should land well under 0.5.

Two rows will legitimately fail that check, and must not be chased. A row only trains if its token actually occurs in the data, and '\\</w>' and '`</w>' -- the END-OF-WORD variants of backslash and backtick -- occur zero times: a backslash in C is followed by the escaped character, and a backtick by more backticks or a language tag, so neither is ever word-final. Zero occurrences is zero gradient, and no learning rate multiplies that into anything. Raising --newtok_lr_mult to chase them only pushes the 21 rows that did train further from their seeds. Measured counts over the mined half of the corpus, for reference:

Judge a row against its frequency, not against a flat threshold. The one that decides whether the model can spell an identifier is _, and it should land near 0.2.

Set --newtok_lr_mult, and do not leave it at the adapter LR. The delta starts at exactly zero and has to carry a row from its seed to a different token -- norm ~2.4 against a seed of norm ~1.26. A LoRA adapter only ever nudges an already-trained weight, so one shared LR that suits the adapters leaves the delta 25x short: a full 671-step run at --lr 1e-4 reached |delta| 0.098 where 2.41 was needed, every row reported changed: True (that is just install()'s reseed noise), and the cosine never moved off 0.88. The default of 50 is what produced the table above.

--seq defaults to 768. XerisOS runs a 2048-token context now, and the corpus has targets up to ~2,800 characters — 768 truncates exactly the write examples that teach house style.

What is actually trained#

Only Ari's reply tokens, plus the turn-closing user: cue. User turns and tool results are context, not targets. Without that masking the model learns to hallucinate tool results rather than emitting a tool call and waiting for one — which for the run: grep examples would be the precise opposite of the goal.

3b. What the mix is made of, and why it is weighted#

Check the language balance before spending GPU hours on it. The corpora that ship in this repo are not weighted for the OS they are meant to serve: train_code3.jsonl classified on the ANSWER side is 40% JavaScript and 11% C, and combined with xeris_corpus.jsonl the result was 8.4% C, 22.5% JavaScript, 0.4% assembly. That is what produced a C answer containing a.length.

mix_v6.py reweights it: all assembly x2, all C x3, JavaScript capped at 1500, 4000 run: tool examples kept as their OWN bucket (upweighting code otherwise dilutes the retrieval skill the OS actually depends on), and 5000 prose for retention. Result: ~38% C, ~17% asm, ~17% tool, ~6% JS.

tools/mine_asm.py is where the assembly comes from. make_xeris_corpus.py skips .asm outright -- 63 files and 33,000 lines, including the scheduler, window manager and TCP stack, had never been in a training mix. There are only ~500 labelled routines, so the miner takes several split points per routine ("given the first 6 lines, write the rest", then 12, 20, 30) and windows the 143 routines too long to emit in one turn. That turns 500 routines into 2,015 examples; one example per routine would cap the whole category at 500.

3c. Renting a big card#

train_lora.py defaults are built for an 8 GB GPU and leave an H100 at ~25 GB of 80. Two flags, no code change:

--no_quant --batch 32 --accum 1

--no_quant skips NF4 entirely: base weights stay bf16 (~13 GB rather than 3.6) and every matmul stops dequantising on the fly -- faster AND slightly better, since NF4 is lossy. --batch 32 --accum 1 keeps the effective batch at 32, identical to 16x2, so the LR and step count stay comparable.

Leave gradient checkpointing ON. Turning it off is ~20% faster in principle but un-checkpointed activations across 32 layers at batch 32 will OOM on the long buckets -- and partway in, not at startup.

4. Merge, export, deploy#

The loop the artifacts record is train_codeN.jsonl → lora_code_vN.pt → merge_lora.py → ari_6.6b_code_vN+1.pt. Keep the numbering going; it is the only record of what produced what.

Export on the pod, not at home: it has the RAM and the GPU, and it means you bring back a ~4 GB export instead of a 13 GB checkpoint. ari_0.bin / ari_1.bin come home; the checkpoint stays until you need it again.

Chunking is a FAT32 limit, not a memory one — ari_0.bin is exactly 3 GiB because FAT32 caps a file at 4 GiB − 1. The loader concatenates the chunks into one contiguous block at boot.

To ship the adapter alone instead of a merged checkpoint (MODELS.md):

mingw32-make export_lora LORA=../lora_xeris_v1.pt NAME=XERIS    -> XERIS.XLA (~71 MB at r=16)
:: copy XERIS.XLA to \LORAS\ on the volume, then in the Terminal:
model lora XERIS

That writes ARILORA.TXT, so the adapter is re-attached after every boot.

5. Before you trust it#

Fine-tuning ~11k domain examples onto a 6.6B model can make it good at XerisOS and worse at everything else, and you will not notice unless you look. The trainer holds out 2% by default (--val_frac); read the validation loss, do not just watch the training curve.

First check it can spell. Ask for a symbol that exists:

where is create_window defined?

If the underscore comes back as a hyphen, or { as (, stop — --newtok did not take and nothing else you test will mean anything. Ask about a symbol that actually exists, too: a model trained to retrieve has nothing to retrieve for an invented name like find_cat, so it confabulates, and that tells you nothing about whether the run worked.

Then ask it something you know the answer to:

how do I add an app to the launcher?

A good answer reaches for run: grep or names applauncher.c's tile table. A bad answer is fluent, plausible, and invented. That single question tells you whether the tool training took.

Also check it can still hold an ordinary conversation. If it has started answering "hey" with a file path, the mix was too narrow.