FNova: Compact LLM Training

Overview

FNova-1.7B-Chat-v1 retains a 1.7-billion-parameter student architecture while combining domain adaptation and post-training. The pipeline begins with the Qwen3-1.7B-Base model and produces a chat model with a 32K-token context window.

The project covers data preparation, training objectives, optimization, student rollout generation, teacher supervision, and evaluation. The released BF16 model is available under Apache 2.0.

Training pipeline

• Continued pretraining (CPT): approximately 20 billion packed corpus tokens, configured with equal legal and general-text sampling shares, plus fill-in-the-middle augmentation on 10% of eligible training rows.
• Supervised fine-tuning (SFT): segment-aware losses balance control tokens, reasoning, and final answers. Each segment is averaged before weighting so long reasoning traces do not dominate solely through length.
• Group relative policy optimization (GRPO): eight student responses per prompt receive task-aware rewards for correctness, reasoning, relevance, and instruction adherence, with a bounded length penalty.
• On-policy distillation (OPD): the student generates its own answers, and a frozen Qwen3-32B teacher scores the same prefixes. Token-level Jensen–Shannon divergence aligns their full-vocabulary distributions, with equal weight per answer.

Training systems

BF16 computation is paired with persistent FP32 trainable weights and optimizer state to preserve small updates. A hybrid optimizer uses Muon for transformer projection matrices and AdamW for remaining parameters.

The distillation workflow samples from 167,014 training prompts, combining EasyR1 and synthetic tasks. Stored reference answers are excluded from OPD training inputs. Updated student weights are synchronized to the rollout model between batches, with checkpoint state supporting reproducibility and recovery.

Evaluation and interpretation

The presentation reports native scores across five legal and seven general benchmark suites for one selected FNova-1.7B-Chat-v1 checkpoint. Examples include LawBench 47.07, LexRel 61.55, GSM8K 81.35, and GPQA-Diamond 16.16 on 0–100 scales. Each suite retains its own metric; the scores are not averaged across suites.

These are evaluations of the resulting chat model with thinking enabled and BF16 inference. Qwen3-1.7B-Base is the training starting point, not a matched evaluation baseline in the presentation. The frozen Qwen3-32B model supplies distillation supervision; the reported scores belong to the 1.7B student.

The results describe a single full benchmark pass under the documented sampling protocol. A matched baseline and controlled stage ablations are needed to quantify improvement. The OPD mixture retained 4,472 benchmark-match flags, limiting claims about contamination-free generalization.

Materials