Fine-Tuning Qwen3-4B — Shahid Kamal

At Checkit Analytics, we were spending thousands of dollars a month on OpenAI API calls for our financial Q&A product. The responses were good, but the costs were scaling linearly with users, latency was unpredictable, and we had zero control over model updates. One morning, GPT-4 started formatting outputs differently — and our entire parsing pipeline broke.

That was the moment we decided to bring the LLM in-house. After evaluating several open-source models, we landed on Qwen3-4B — and fine-tuned it to outperform GPT-3.5 on our domain-specific tasks at a fraction of the cost. This post is the complete story of how we did it.

1. Why Qwen3-4B?

Choosing the right base model is the most important decision in the entire pipeline. We evaluated five candidates on our internal benchmark of 500 financial Q&A pairs:

Llama 3-8B — Strong general performance, but 8B parameters meant higher inference costs and slower latency
Mistral 7B — Excellent reasoning, but struggled with structured financial data
Phi-3 Mini (3.8B) — Impressive for its size but weak on long-context financial documents
Gemma 2-2B — Too small; couldn't handle multi-step reasoning over financial tables
Qwen3-4B — Best balance of size, multilingual support, and structured data understanding

Qwen3-4B stood out because of its strong performance on structured data and instruction following out of the box. For financial analytics — where responses need to reference specific numbers from tables, perform light calculations, and cite sources — this mattered more than raw benchmark scores.

The best model isn't the one with the highest leaderboard rank. It's the one that performs best on YOUR data, at a cost you can sustain.

2. Data Preparation

Fine-tuning is only as good as your data. We built our training dataset from three sources:

Source 1: Production logs. We had 6 months of user queries and GPT-4 responses from our existing product. We manually reviewed and filtered these down to ~3,200 high-quality question-answer pairs. The key was being ruthless about quality — we threw away any pair where the GPT-4 answer was wrong, incomplete, or poorly formatted.

Source 2: Synthetic generation. For underrepresented query types (risk analysis, ratio comparisons, time-series trends), we used GPT-4 to generate additional training examples from our financial document corpus. We generated ~1,800 synthetic pairs, then manually validated a 20% sample to check quality.

Source 3: Edge cases. We specifically crafted ~400 examples for failure modes we'd seen in production: questions about missing data, ambiguous time periods, multi-company comparisons, and "I don't know" scenarios where the answer isn't in the context.

Total dataset: 5,400 examples, split 90/5/5 into train/validation/test sets.

Data format

We used the ChatML format that Qwen expects, with a system prompt that anchors the model in its role:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a financial analyst AI. Answer questions using ONLY the provided context. Cite specific numbers. If the answer is not in the context, say so."
    },
    {
      "role": "user", 
      "content": "Context: [document chunks]\n\nQuestion: What was Apple's gross margin in Q3 2025?"
    },
    {
      "role": "assistant",
      "content": "Apple's gross margin in Q3 2025 was 46.3%, up from 44.5% in Q3 2024. This 1.8 percentage point improvement was primarily driven by higher services revenue, which carries margins above 70%."
    }
  ]
}

Two critical details: first, we always included the source context in the user message — this trains the model to ground answers in provided documents rather than hallucinate from parametric knowledge. Second, we trained explicit refusal behavior: about 8% of our examples had the assistant respond with "This information is not available in the provided documents" when the context didn't contain the answer.

3. Fine-Tuning with LoRA

Full fine-tuning a 4B parameter model requires significant GPU memory and risks catastrophic forgetting of the model's general capabilities. Instead, we used LoRA (Low-Rank Adaptation) — which freezes the base model weights and trains small adapter matrices on top.

The math is elegant: instead of updating a weight matrix W (dimensions d×d), LoRA decomposes the update into two smaller matrices A (d×r) and B (r×d), where r is the rank (typically 8-64). This reduces trainable parameters by 100-1000x while preserving most of the fine-tuning benefit.

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2"
)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")

# LoRA configuration
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=32,                          # Rank — sweet spot for our task
    lora_alpha=64,                 # Scaling factor
    lora_dropout=0.05,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",  # Attention
        "gate_proj", "up_proj", "down_proj"        # MLP
    ],
    bias="none"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 3,937,054,720 
# trainable%: 1.065%

Only ~1% of parameters are trainable, but the impact on domain performance is dramatic.

Training configuration

We trained on a single A100 80GB GPU (AWS p4d.24xlarge instance). Key hyperparameters after our sweep:

from transformers import TrainingArguments
from trl import SFTTrainer

training_args = TrainingArguments(
    output_dir="./qwen3-4b-financial",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,    # Effective batch size: 32
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    max_grad_norm=1.0,
    report_to="wandb"
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    tokenizer=tokenizer,
    max_seq_length=4096,
    dataset_text_field="text"
)

trainer.train()

Training took ~2.5 hours for 3 epochs over 4,860 examples. Total compute cost: about $30 on AWS spot instances. Compare that to the $2,000+/month we were spending on API calls.

What we learned about hyperparameters

LoRA rank 32 was our sweet spot. Rank 8 underfit on financial reasoning; rank 64 showed no improvement over 32 but trained slower
Learning rate 2e-4 with cosine decay. Higher rates caused training instability; lower rates needed more epochs
3 epochs was optimal. At epoch 4, validation loss started increasing — classic overfitting on a relatively small dataset
Gradient accumulation to reach effective batch size 32 was important for training stability with our data distribution

4. Evaluation

We evaluated the fine-tuned model against GPT-3.5-turbo, GPT-4, and the base Qwen3-4B on our held-out test set of 270 examples. We measured four dimensions:

Correctness — Is the answer factually accurate given the context? Scored by human reviewers on a 1-5 scale.

Faithfulness — Does the answer only use information from the provided context, without hallucinating? Binary yes/no per response.

Completeness — Does the answer address all parts of the question? Scored 1-5.

Format compliance — Does the response follow our expected structure (citations, numerical formatting, appropriate length)? Binary.

# Results on 270-example test set

| Model               | Correctness | Faithfulness | Completeness | Format |
|---------------------|-------------|--------------|--------------|--------|
| GPT-4               |    4.6      |    94%       |     4.5      |  91%   |
| Qwen3-4B (ours)     |    4.3      |    96%       |     4.2      |  97%   |
| GPT-3.5-turbo       |    3.8      |    82%       |     3.6      |  74%   |
| Qwen3-4B (base)     |    2.9      |    68%       |     2.7      |  45%   |

The fine-tuned model significantly outperformed GPT-3.5 across all metrics and came close to GPT-4 on correctness and completeness — while beating GPT-4 on faithfulness and format compliance. This makes sense: our fine-tuning data explicitly trained the model to stay grounded in context and follow our output format, which general-purpose models don't optimize for.

The base Qwen3-4B scored poorly across the board, confirming that fine-tuning (not just prompting) was necessary for production-quality results on domain tasks.

5. Deployment on AWS

For serving, we deployed the model using vLLM on an AWS g5.2xlarge instance (A10G GPU, 24GB VRAM). vLLM's PagedAttention and continuous batching gave us dramatically better throughput than naive HuggingFace inference:

# Start vLLM server with merged LoRA weights
python -m vllm.entrypoints.openai.api_server \
    --model ./qwen3-4b-financial-merged \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --dtype bfloat16

Before deploying, we merged the LoRA adapters back into the base model to avoid any adapter overhead at inference time:

from peft import PeftModel

# Load base + adapter, then merge
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
model = PeftModel.from_pretrained(base_model, "./qwen3-4b-financial")
merged = model.merge_and_unload()
merged.save_pretrained("./qwen3-4b-financial-merged")

Production performance

Latency: ~800ms median for a typical financial Q&A response (vs 2-4s with GPT-4 API)
Throughput: ~35 requests/sec with continuous batching
Cost: $0.50/hr for the g5.2xlarge instance → roughly $360/month vs $2,000+/month on OpenAI API
Uptime: 99.7% over the first month (one restart due to CUDA OOM on an unusually large batch)

We put an API gateway (FastAPI) in front of vLLM that handles authentication, rate limiting, request logging, and a fallback to GPT-4 for edge cases where our model's confidence is low. This hybrid approach gives us the cost savings of self-hosting with the safety net of a frontier model.

6. Lessons Learned

Data quality > data quantity. Our first attempt used 12,000 examples with less rigorous filtering. The model trained on 5,400 curated examples performed significantly better. Every bad training example teaches the model a bad habit.

Train refusal behavior explicitly. Without "I don't know" examples in training data, the model will always attempt an answer — even when the context doesn't support one. This is the #1 source of hallucination in RAG systems, and it's fixable with data.

Evaluate on YOUR metrics, not benchmarks. Qwen3-4B scores lower than Llama 3-8B on MMLU, but it scores higher on our financial task after fine-tuning. Public benchmarks measure general capability; you need to measure domain performance.

LoRA is almost always enough. We experimented with QLoRA (quantized base + LoRA) and full fine-tuning. QLoRA had a small quality drop. Full fine-tuning had marginal quality gains but 10x the compute cost and a higher risk of catastrophic forgetting. Standard LoRA hit the sweet spot.

Monitor post-deployment. We log every request and response, and run nightly automated evals on a rolling 100-response sample. Model quality can drift if your user queries shift — and you need to catch that early.

What's Next

We're currently exploring two extensions: DPO (Direct Preference Optimization) to further align the model with human preferences on answer quality, and speculative decoding with a smaller draft model to cut latency further. I'll share our findings in a future post.

The bottom line: if you're spending serious money on LLM API calls for a domain-specific task, fine-tuning a small open-source model is almost certainly worth it. The upfront effort is a few days of work; the ongoing savings compound every month.

Questions or working on something similar? Reach out via email or connect on GitHub.