FNova
Training a compact
legal-domain LLM
A complete training pipeline starting from Qwen3-1.7B-Base
The complete training pipeline
Each stage changes the source of supervision.
Domain knowledge
Next-token prediction on legal and general text; FIM augmentation.
Response behavior
Supervised reasoning and final answers with separate loss weights.
Task preference
Compare eight student responses using task-aware rewards.
Teacher distribution
Qwen3-32B scores prefixes generated by the student.
Mixed precision: BF16 compute, FP32 updates
BF16 preserves the FP32 exponent range while using fewer fraction bits.
Autocast for model computation and lower-precision working tensors.
Persistent trainable weights, accumulated gradients, and optimizer state.
forward / backward under BF16 autocast
16 vs 32 bits per scalar; total training memory also includes FP32 state and activations.
Small updates need persistent FP32 weights
Controlled numerical example · w₀ = 1, Δw = −10⁻⁴ · round to nearest BF16
A small update can disappear when it rounds back to the same BF16 value.
FP32 persistence accumulates those updates. The BF16 computation view changes once the accumulated difference becomes representable.
Muon: remove singular-value scales from the update
SVD separates an update’s directions from the strength of each direction.
Ideal SVD view, shown with two nonzero singular values. A separate global scale sets the final step magnitude.
Keep directions; remove unequal scales
P and Q describe the matrix’s feature directions.
Σ contains singular values: the gain of each direction. Replace them by 1 to balance the update.
Apply this to the momentum-derived gradient update. The weight matrix itself remains unconstrained.
FNova uses Muon for attention Q/K/V/O and MLP gate/up/down matrices. Gradient → momentum update M → approximate PQᵀ → scaled weight update.
AdamW: remember direction and scale separately
Each parameter keeps two running averages. Their ratio determines its adaptive step.
Signed gradient memory: mtmt = β₁mt−1 + (1 − β₁)gtRepeated signs reinforce; alternating signs partly cancel.
Squared-gradient memory: vtvt = β₂vt−1 + (1 − β₂)gt²√v̂t is the recent RMS gradient magnitude.Gradients ±2 → scale ≈2; gradients ±0.1 → scale ≈0.1.
Scale the step
Divide the signed trend by its RMS scale; apply η.
Bias correction: m̂t = mt / (1 − β₁t), v̂t = vt / (1 − β₂t).
gt: current gradient; β₁, β₂: memory coefficients. FNova: β₁ = 0.9, β₂ = 0.95. Illustrative summaries; ε and decay omitted.
Muon
Rebalances singular directions of a matrix update jointly.
AdamW
Rescales each parameter coordinate using its own history.
AdamW also applies separate weight decay: θ ← (1 − ηλ)θ. In FNova, Muon handles hidden projection matrices; AdamW handles the remaining parameters.
CPT: a ~20B-token corpus with legal emphasis
packed training tokens
4,096 tokens per sequence
Legal sampling share: 50%
Statutes, legal books, and industry/legal text supply domain exposure.
General sampling share: 50%
Chinese web text maintains broader language coverage.
Shares are configured sampling weights, not raw corpus proportions. Packed corpus size differs from scheduled token exposure.
Fill-in-the-middle during CPT
Reorder a document so the model predicts a missing middle span using both sides.
<fim_middle> M <EOS>
10% of eligible training rows
Deterministic selection and nonempty spans make preprocessing reproducible.
The same causal-LM objective
FIM changes the sequence order; the trainer still performs next-token prediction.
FIM applies to training data only; validation retains the ordinary sequence order.
SFT: loss by response segment
Average each segment first, then combine the segment losses.
ℒthink = 0.02ℓ̄control + 0.38ℓ̄reasoning + 0.60ℓ̄final
Prompt and earlier turns are masked; only the final assistant target is supervised.
A long reasoning trace cannot dominate solely because it contains more tokens.
95,000 training / 5,000 validation records · Project-selected weights; available segments are renormalized.
GRPO: learn from better answers in the same group
Ask the same question eight times. Use the group’s average reward as the baseline.
Above the mean → encourage those sampled tokens
Answer 8: 0.90 − 0.55 = +0.35
Below the mean → discourage those sampled tokens
Answer 1: 0.20 − 0.55 = −0.35
Illustrative rewards · Aᵢ = Rᵢ − mean(R) · G = 8, mean subtraction only · clipping 0.2 + KL penalty 0.001 restrain policy changes; reference refresh every 400 updates.
GRPO reward: one answer, one transparent score
Example: a 6,144-token legal-choice answer, scored for content and response quality.
What the judge rewards
A committed correct answer, supported reasoning, restraint, and the requested format.
Why adjust for length?
Leave 4,096 tokens free for reasoning, then apply a bounded cost for additional tokens.
Illustrative thinking-mode legal-choice example · One generative judge, four dimensions · Correctness has the largest weight; other tasks use different weights. Final reward is clipped to [0, 1].
OPD: learn on the student’s own trajectories
The student generates an answer. Both models then score exactly the same prefixes.
JSDt = ½ KL(pθ,t ∥ mt) + ½ KL(pT,t ∥ mt)
Frozen Qwen3-32B teacher
Full-vocabulary distributions provide supervision at every generated token.
Equal weight per answer
Mask prompt/padding; average generated tokens within each answer.
Synthetic prompts for on-policy distillation
training prompts
663 validation prompts
Construct and check prompts
Creator → schema / length checks → solver reference → model review → deduplication.
Train on student answers
OPD consumes prompts only. Stored reference answers are excluded from training input.
Synthetic tasks include general, legal, dialogue, and long-context exercises. Prompt generation and OPD teacher scoring are separate roles.
The on-policy distillation training loop
The next rollout uses the student we just updated.
Recover the same run
Checkpoint weights, optimizer, scheduler, sampler and RNG state.
Monitor training quality
Track repetition, capped rollouts, gradient norms and weight changes.
OPD shown here · Frozen teacher: FSDP2 · Student: DDP · Activation checkpointing and vocabulary chunks control memory.
Legal-domain benchmark results
A broad legal evaluation profile for a compact student.
The legal emphasis in CPT and post-training gives the model repeated exposure to legal language and structured tasks.
Results are consistent with that focus. A matched baseline and stage ablations would be needed to quantify the gain.
Source: released model card and native full-benchmark results. ¹ MSLR: no AI judge. Metrics and aggregation differ across suites; no cross-suite average is implied.
General capability and remaining gaps
Math is a visible strength in this evaluation profile.
GPQA-Diamond at 16.16 shows a substantial remaining gap on expert-level questions.
Same reported checkpoint and full-benchmark protocol. ² IFEval: strict prompt-level accuracy. Scores use each suite’s native metric.
Evaluation protocol and interpretation
Reported protocol
- Mode
- Thinking enabled · BF16 inference
- Sampling
- Temperature 0.7 · top-p 0.98
- Penalties
- Repetition 1.1 · presence 0.4 · frequency 0
- Budget
- 8,192 output · 32,768 total tokens
- Scoring
- Native prompts / evaluators · one full pass
Scope of the evidence
- The scores describe one selected checkpoint under this protocol.
- Legal exposure supports the domain focus; stage gains require controlled ablations.
- The OPD mixture retained 4,472 benchmark-match flags, limiting contamination-free generalization claims.
- A matched baseline is needed for a defensible claim of improvement.
Project highlights
FNova-1.7B-Chat-v1
- Built a compact LLM from Qwen3-1.7B-Base through ~20B-token CPT, segment-aware SFT, GRPO, and on-policy distillation.
- Implemented BF16 computation with FP32 updates and hybrid Muon / AdamW optimization for transformer training.
- Developed task-aware reward design and a synthetic-prompt distillation pipeline with a frozen Qwen3-32B teacher.
- Released a 32K-context BF16 model under Apache 2.0 and evaluated it across 12 legal and general benchmarks.