32B Model Fine-Tuning and Serving

Scope

The use case and dataset content are private. This page covers the model training and serving work.

I trained two separate LoRA fine-tunes of Qwen's QwQ-32B model. Both were trained with Axolotl, evaluated on data that was kept out of training and then deployed behind APIs that the application could call.

Training data

I prepared 171,031 labelled examples for supervised fine-tuning. The data was formatted for the model's chat template and split into separate training and evaluation sets for each fine-tune.

The dataset itself is private, including the input content, annotation scheme and target classes.

Training configuration

Setting	Value
Base model	Qwen/QwQ-32B
Fine-tuning method	LoRA with Axolotl
GPUs	4 × NVIDIA H200 80GB
LoRA rank	128
LoRA alpha	128
Learning rate	5e-5
Weight decay	0.05
Global batch size	12
Training length	4 epochs
Selected checkpoint	Epoch 2

I compared the checkpoints after each epoch. Performance started to drop after epoch two, so I selected that checkpoint instead of taking the final epoch by default.

Evaluation

I kept a separate 100-example benchmark out of training. The two-stage system reached 93% on that set. I also checked the individual errors when comparing checkpoints so the final decision did not depend on one aggregate number.

Serving

I served the selected weights with vLLM on Modal through OpenAI-compatible endpoints. Model serving stayed separate from the application, so I could change a checkpoint or GPU configuration without rebuilding the rest of the system.

A FastAPI application handled the request boundary around the model endpoints. The deployment kept model loading, application logic and GPU scaling separate.