Retrieval-Augmented Generation (RAG) has become the go-to architecture for grounding LLM responses in real data. But the gap between a "works in a notebook" RAG demo and a production system that handles real queries reliably is enormous.
After building RAG pipelines at Checkit Analytics for financial question-answering, here's what I've learned about the decisions that actually matter.
1. Chunking Is Everything
The single most impactful decision in a RAG pipeline isn't the embedding model or the vector store — it's how you chunk your documents. Too small and you lose context. Too large and you dilute relevance with noise.
The best chunk size depends entirely on your query patterns. There's no universal answer — only experimentation.
We settled on a hybrid approach: semantic chunking for unstructured text, and table-aware parsing for financial reports. The key insight was treating tables as first-class citizens rather than forcing them through text splitters.
# Semantic chunking with overlap
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)2. Embedding Model Selection
We benchmarked several embedding models on our financial corpus. The results surprised us — larger models didn't always win. Domain relevance and query-document alignment mattered more than raw MTEB scores.
What worked for us
- BGE-base for general document search — fast and accurate enough
- Cohere embed v3 when we needed multilingual support
- Custom fine-tuned embeddings when we had labeled query-document pairs
3. The Retrieval-Generation Gap
Even with perfect retrieval, generation quality depends heavily on how you format the context for the LLM. We found that structured prompts with clear document boundaries reduced hallucination by ~30% compared to naive concatenation.
This is an active area of iteration for us, and I'll share more findings as we scale the system to handle complex multi-step financial analysis queries.
What's Next
In the next post, I'll dive into our evaluation framework — how we measure RAG quality beyond simple accuracy, including faithfulness, relevance, and answer completeness metrics.
If you're working on similar problems, I'd love to connect. Reach out via email or find me on GitHub.