FinDoc QA · Resume Guide

"FinDoc QA is a RAG-based document Q&A system for banking policy PDFs. The pipeline has three main stages.

Ingestion: PDFs are loaded with PyMuPDF, split into 800-character chunks with 150-char overlap using RecursiveCharacterTextSplitter, embedded with OpenAI text-embedding-3-small, and stored in ChromaDB.

Retrieval: When a user asks a question, the query is embedded and ChromaDB performs MMR search — returning 5 diverse, relevant chunks from different sections of the documents.

Generation: The chunks are passed to GPT-4o with a grounded system prompt that enforces citation format and prevents hallucination. LangChain's ConversationalRetrievalChain adds conversation memory so follow-up questions work naturally.

The whole thing is exposed via FastAPI with session management, and there's a Streamlit UI for non-technical users. Dockerised with Docker Compose."

"RAG — Retrieval-Augmented Generation — retrieves relevant documents at query time and passes them as context to the LLM, rather than relying on knowledge baked into model weights.

I chose RAG over fine-tuning for three reasons: First, banking policy documents change frequently — RBI circulars update monthly. RAG lets you add new documents in seconds; fine-tuning requires retraining. Second, fine-tuning is expensive and requires hundreds of labelled examples. RAG works out-of-the-box with zero labelling. Third, RAG provides citations — the user can see exactly which document and page the answer came from. Fine-tuned models give answers with no traceability, which is unacceptable in compliance use cases."

"Standard cosine similarity returns the top-5 most similar chunks — but in practice, they're often 5 nearly identical chunks from the same page of the document. The LLM then gets repetitive, redundant context and can't form a complete answer.

MMR — Maximal Marginal Relevance — solves this. It fetches the top-20 candidates by similarity, then greedily selects 5 that are both relevant to the query AND dissimilar to each other. The lambda_mult=0.5 parameter balances 50% relevance vs 50% diversity.

In practice, this meant the LLM got context from 5 different sections of the policy document — the main clause, the exceptions, the definitions, the applicability section — rather than 5 repetitions of the same clause. Answer quality improved noticeably."

"Three layers working together.

First — the system prompt: 'Answer ONLY using the provided context. Do NOT use prior knowledge.' This is the primary guard.

Second — explicit fallback instruction: 'If the answer is not in the context, respond: I cannot find this information in the provided documents.' This handles out-of-scope questions gracefully instead of the LLM guessing.

Third — temperature=0. Zero temperature makes the LLM deterministic — it picks the highest-probability token at every step rather than sampling randomly. This eliminates creative speculation.

Additionally, the context is explicitly passed as {context} in the prompt template — the LLM cannot ignore it. And I added 3 few-shot examples in the system prompt showing the exact citation format expected, which improved format consistency without any fine-tuning."

"It was an empirical decision based on the nature of banking policy documents.

chunk_size=800: A typical policy clause — main rule + conditions + exception — spans roughly 600-900 characters. 800 chars captures a complete clause in most cases. Smaller chunks like 400 chars would split clauses mid-sentence. Larger chunks like 1500 chars reduce retrieval precision — too much irrelevant content gets included.

overlap=150: Policy documents often have clauses that reference definitions from the preceding paragraph. Without overlap, a chunk boundary could separate a rule from its definition, losing critical context. 150 chars (about 2 sentences) covers this safely without making chunks redundant.

I used RecursiveCharacterTextSplitter rather than a fixed-size splitter because it respects semantic boundaries — it tries to split on paragraph breaks first, then sentences, then words. This means chunks rarely cut mid-sentence."

"ChromaDB was the right choice for this use case for three reasons.

Local persistence — ChromaDB runs in-process and persists to disk. No external API, no cloud account, no latency to an external service. For an internal banking tool where data sensitivity matters, keeping vectors local is preferable.

Scale fit — this deployment handles hundreds of policy documents, maybe 50,000-100,000 chunks. ChromaDB's HNSW index handles this comfortably with sub-100ms retrieval. Pinecone's scale advantages only matter beyond millions of vectors.

Simplicity — zero infrastructure setup. pip install chromadb and it works. Pinecone requires API keys, account management, index configuration. For a self-contained internal tool, that complexity is unnecessary overhead.

If this scaled to millions of documents across multiple teams, I would migrate to pgvector (already familiar from LoanIQ) or Pinecone. The code change would be minimal — just swap the vectorstore object."

"I used LangChain's ConversationalRetrievalChain with ConversationBufferMemory.

The key mechanism is question condensation. When the user asks a follow-up like 'What about exceptions?', the chain doesn't retrieve chunks for that four-word question — it first sends the full chat history + the new question to the LLM and asks it to produce a standalone question. The LLM generates: 'What are the exceptions to pre-payment penalty clauses for home loans?' — then THAT condensed question is used for vector retrieval.

This means retrieval always has full context, even for one-word follow-ups. ConversationBufferMemory stores the complete message history in RAM, keyed by session ID. Each session has its own memory instance — users don't share conversation history.

The limitation is that ConversationBufferMemory stores everything — very long conversations will eventually hit the LLM's context limit. For production at scale I'd switch to ConversationSummaryMemory which summarises older turns."

"Three things I would add next.

1. RAGAS Evaluation Framework — right now I'm evaluating quality manually by testing sample queries. I'd add RAGAS to measure answer faithfulness, context precision, and answer relevance automatically. This gives quantitative benchmarks and alerts when retrieval quality degrades after adding new documents.

2. Hybrid Search — BM25 keyword search combined with dense vector search. Pure semantic search sometimes misses exact matches — if a user asks about a specific regulation number like 'RBI/2024/87', BM25 finds it immediately while semantic search might not. Hybrid search combines both for best-of-both-worlds retrieval.

3. LangSmith Tracing — for production observability. LangSmith captures every LLM call, token usage, retrieval latency, and chain steps. Makes debugging retrieval failures much easier — you can see exactly which chunks were retrieved and why the LLM gave a particular answer."

"I implemented lightweight in-memory session management using a Python dictionary in FastAPI.

When a user sends their first query without a session_id, the server generates a UUID, creates a new ConversationBufferMemory instance, builds a ConversationalRetrievalChain with that memory, and stores it in a dict: _sessions[session_id] = {chain, memory, created_at, last_active}.

The session_id is returned in the response. The client includes it in every subsequent request — the server retrieves the existing chain with its memory intact, so conversation history is preserved.

Sessions expire after 30 minutes of inactivity — a cleanup function runs on every request and deletes sessions older than the TTL. This prevents unbounded memory growth.

The limitation: this is single-server state. If you scale to multiple API instances, sessions would need to be stored externally — Redis would be the natural choice, serialising the memory to JSON."