Agentic RAG System
A LangGraph agent that routes between a local FAISS knowledge base and live web search based on confidence scoring - deployed on HuggingFace ZeroGPU.
About this project
A clean three-node LangGraph StateGraph - no ReAct prompting, no regex parsing of model output. A retrieve_node computes a bounded confidence score from FAISS's L2 distance (sim = 1 / (1 + L2)); if it clears a 0.60 threshold the cached answer is used, otherwise a web_node runs a Tavily search and immediately indexes the results into FAISS for future queries. An answer_node then generates a history-aware response via Qwen 2.5-7B-Instruct (4-bit NF4 quantized) with deterministic, greedy decoding and citations extracted directly from retrieved context rather than relied on the model to produce inline.
Technical details
The repository deliberately keeps three notebooks representing real stages of the build, not a single cleaned-up final version. Notebook 01 is the reference implementation, developed on Colab's free T4 GPU with Qwen 2.5-7B-Instruct, 4-bit quantization via bitsandbytes, and true token streaming through a background-thread TextIteratorStreamer. Notebook 02 targeted HuggingFace Spaces' CPU Basic tier, which required downgrading to Qwen 2.5-3B and switching to a "generate fully, then reveal word-by-word" streaming pattern after the threaded approach caused multi-minute hangs from torch.compile recompiling on every call. Mid-deployment, HuggingFace restricted CPU Basic Spaces to PRO subscribers, and create_repo() started returning 402 Payment Required.
Notebook 03 pivoted to ZeroGPU, HuggingFace's free shared-GPU tier, restoring the 7B model and 4-bit quantization. This required learning ZeroGPU's execution model: real GPU access only exists inside functions decorated with @spaces.GPU, and - counter-intuitively - model weights must be placed on cuda at module scope rather than lazily inside that function, since importing the spaces package patches torch to intercept the placement correctly even before a GPU is attached. The final architecture isolates GPU access to a single small function; retrieval, prompt building, and history handling all run on CPU and never compete for the shared GPU slot.
Two other fixes worth noting: a _safe_content() helper was added after Gradio 6.x was found to store chat message content as either a plain string or a list of typed content blocks even in text-only conversations, which crashed any code assuming .strip() always worked by the second conversational turn. Follow-up detection also needed three simultaneous signals - a short message, a reference word, and a question starter - rather than one, after single-signal detection was found to incorrectly prepend unrelated prior context onto genuinely new questions.