Local Multimodal RAG System
A document Q&A system that retrieves from both a PDF's text and captioned descriptions of its images, powered by a quantized multimodal LLM.
About this project
Built around google/gemma-3-4b-it (Gemma3), a single multimodal model handles both image captioning and final answer generation. PyMuPDF extracts text and embedded images per page; images smaller than 32×32 pixels are filtered out before captioning to avoid wasting a forward pass on icons and decorative artifacts. Captions are merged inline under each page's text, then chunked with LangChain's RecursiveCharacterTextSplitter (1000 characters, 200 overlap) before being embedded with all-MiniLM-L6-v2 and indexed in a FAISS IndexFlatIP store. A Gradio Blocks interface handles PDF upload and a separate chat panel for querying, with streamed responses.
Technical details
The model is loaded 4-bit NF4 quantized with double quantization (bitsandbytes, bfloat16 compute dtype) - the same compute-fitting tradeoff used elsewhere in this portfolio to get a multi-billion-parameter model running on free-tier hardware. Retrieval pulls the top 10 chunks per query (k=10) via FAISS inner-product search over normalized embeddings.
The system prompt explicitly restricts the model to retrieved context only: "Use only the following pieces of retrieved context to answer the question... Do not use any external knowledge or make assumptions." This is a deliberate faithfulness constraint - without it, a capable LLM can produce a fluent, plausible-sounding answer from its own training knowledge even when the source PDF doesn't actually contain the relevant information, which is the core failure mode a document QA system needs to avoid. The tradeoff is a system that will correctly say "not available in this document" rather than one that always sounds confident.
The FAISS index and chunk metadata are cached to disk per document (index.faiss, chunks.json); re-querying an already-processed PDF skips extraction, captioning, and embedding entirely on the next load, which matters because captioning is the most expensive step in the pipeline. Extracted image files are deleted after captioning completes - only their text captions are retained and indexed.
The repository separates the reference implementation (a self-contained notebook running the full pipeline in-notebook) from the deployment path (a second notebook that programmatically packages app.py, main.py, model_setup.py, and utils.py into a standalone HuggingFace Space and uploads it via huggingface_hub).