Skip to content

Commit dd76d7f

Browse files
committed
Web Interface Title and Project Readme Updated
1 parent c0365b0 commit dd76d7f

3 files changed

Lines changed: 321 additions & 1 deletion

File tree

README.md

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# RAG Project - Learn with Transformers
2+
3+
A production-ready **Corrective Retrieval-Augmented Generation (CRAG)** system built with LangChain, LangGraph, and FastAPI. This project implements an intelligent RAG pipeline that not only retrieves relevant documents but also **validates, corrects, and improves** retrieval quality through an agent-based workflow.
4+
5+
## What Makes This Different from Traditional RAG?
6+
7+
### Traditional RAG:
8+
```
9+
Query → Retrieve Documents → Generate Answer
10+
```
11+
**Problem**: If retrieved documents are irrelevant or low-quality, the answer will be poor.
12+
13+
### This Project (Corrective RAG):
14+
```
15+
Query → Retrieve → Grade Quality → Transform Query if Needed → Web Search if Necessary → Generate
16+
```
17+
**Solution**: Intelligent agent workflow that **self-corrects** by grading document relevance and taking corrective actions.
18+
19+
## Architecture
20+
21+
```mermaid
22+
graph LR
23+
A[User Query] --> B[Retrieve]
24+
B --> C[FAISS+MMR]
25+
C --> D[Rerank]
26+
D --> E{Grade}
27+
E -->|Relevant| F[Generate]
28+
E -->|Partial| G[Filter]
29+
E -->|Poor| H[Transform]
30+
G --> F
31+
H --> I[Web Search]
32+
I --> F
33+
F --> J[Groq LLM]
34+
J --> K[Answer]
35+
```
36+
37+
## Key Features
38+
39+
### 1. **Intelligent Document Grading**
40+
- LLM evaluates retrieved documents for relevance
41+
- Filters out low-quality results automatically
42+
- Ensures only useful context reaches generation
43+
44+
### 2. **Query Transformation**
45+
- Rewrites ambiguous or poor queries
46+
- Improves retrieval on second attempt
47+
- Adaptive query refinement
48+
49+
### 3. **Web Search Fallback**
50+
- Tavily API integration for external knowledge
51+
- Activates when local documents insufficient
52+
- Combines local + web results
53+
54+
### 4. **Advanced Retrieval Stack**
55+
- **FAISS** vector store with MMR search
56+
- **FastEmbed** (BAAI/bge-small-en-v1.5) embeddings
57+
- **FlashRank** (rank-T5-flan) reranking
58+
- Self-query retriever support
59+
60+
### 5. **LangGraph Agent Workflow**
61+
- State machine orchestration
62+
- Conditional routing logic
63+
- Transparent decision-making
64+
65+
## Tech Stack
66+
67+
| Component | Technology |
68+
|-----------|------------|
69+
| **LLM** | Groq (openai/gpt-oss-120b) |
70+
| **Embeddings** | FastEmbed (BAAI/bge-small-en-v1.5) |
71+
| **Vector Store** | FAISS |
72+
| **Reranker** | FlashRank (rank-T5-flan) |
73+
| **Agent Framework** | LangGraph |
74+
| **RAG Framework** | LangChain 0.3.x |
75+
| **Web Search** | Tavily API |
76+
| **Web Framework** | FastAPI + Uvicorn |
77+
| **Observability** | LangSmith (optional) |
78+
| **Document Source** | "Attention Is All You Need" (Transformer paper) |
79+
80+
## Project Structure
81+
82+
```
83+
RAG Project/
84+
├── project/
85+
│ ├── config/
86+
│ │ └── config.yaml # Model & pipeline configuration
87+
│ ├── logger/
88+
│ │ └── logging.py # Centralized logging
89+
│ ├── exception/
90+
│ │ └── except.py # Custom exception handling
91+
│ ├── utils/
92+
│ │ ├── config_loader.py # YAML config loader
93+
│ │ └── model_loader.py # LLM & embedding initialization
94+
│ ├── source/
95+
│ │ └── data_preparation.py # PDF/ArXiv document loading
96+
│ ├── model/
97+
│ │ ├── retriever.py # FAISS retriever with MMR
98+
│ │ └── reranking.py # FlashRank reranking
99+
│ ├── prompts/
100+
│ │ └── prompt_template.py # RAG, Router, WebSearch prompts
101+
│ └── pipeline/
102+
│ ├── rag.py # Core RAG pipeline
103+
│ └── agents.py # CRAG agent workflow
104+
├── templates/
105+
│ └── index.html # Web UI template
106+
├── static/
107+
│ └── styles.css # Purple gradient theme
108+
├── data/
109+
│ └── attention-is-all-you-need.pdf
110+
├── app.py # FastAPI application
111+
├── main.py # CLI entry point
112+
├── Dockerfile # Docker containerization
113+
└── requirements.txt # Dependencies
114+
115+
```
116+
117+
## Quick Start
118+
119+
### 1. Clone & Install
120+
```bash
121+
git clone https://github.com/Abeshith/RAG-Project-PipeLine.git
122+
cd RAG-Project-PipeLine
123+
pip install -r requirements.txt
124+
```
125+
126+
### 2. Set Environment Variables
127+
Create `.env` file:
128+
```env
129+
GROQ_API_KEY=your_groq_api_key
130+
GOOGLE_API_KEY=your_google_api_key
131+
LANGSMITH_API_KEY=your_langsmith_key
132+
TAVILY_API_KEY=your_tavily_key
133+
```
134+
135+
### 3. Run Web Interface
136+
```bash
137+
python app.py
138+
```
139+
Visit: http://localhost:8000
140+
141+
### 4. Run CLI
142+
```bash
143+
python main.py
144+
```
145+
146+
## Docker Deployment
147+
148+
### Build & Run
149+
```bash
150+
docker build -t rag-project .
151+
docker run -d -p 8000:8000 --env-file .env rag-project
152+
```
153+
154+
## How It Works
155+
156+
### Workflow Example
157+
158+
**Query**: "What is the attention mechanism in transformers?"
159+
160+
1. **Retrieval**: FAISS finds top 3 most similar chunks from "Attention Is All You Need" paper
161+
2. **Reranking**: FlashRank reorders by relevance (top 3 kept)
162+
3. **Grading**: LLM evaluates each document:
163+
- ✅ Doc 1: Relevant (explains attention)
164+
- ✅ Doc 2: Relevant (shows formula)
165+
- ❌ Doc 3: Not relevant (talks about training data)
166+
4. **Decision**: 2/3 relevant → Use filtered docs
167+
5. **Generation**: Groq LLM synthesizes answer from relevant docs
168+
6. **Output**: Comprehensive answer with LaTeX formulas (rendered via MathJax)
169+
170+
### When Retrieval Fails
171+
172+
**Query**: "What are the latest improvements to transformers in 2024?"
173+
174+
1. **Retrieval**: Finds documents from 2017 paper
175+
2. **Grading**: ❌ All documents marked "not relevant" (outdated info)
176+
3. **Transform**: Rewrites query → "Recent transformer architecture improvements 2024"
177+
4. **Web Search**: Tavily searches current web content
178+
5. **Generation**: Answer combines paper fundamentals + recent developments
179+
180+
## Web Interface Features
181+
182+
- **Modern UI**: Purple gradient design with responsive layout
183+
- **MathJax Integration**: Renders LaTeX formulas beautifully
184+
- **Transformer Visualization**: Architecture diagram in header
185+
- **Real-time Search**: Fast async FastAPI backend
186+
- **Error Handling**: Graceful degradation with user-friendly messages

logs/2025_12_01.log

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -360,3 +360,137 @@ ModuleNotFoundError("No module named 'faiss.swigfaiss_avx512'")
360360
[2025-12-01 11:10:09] INFO - project.model.reranking - Reranked 3 documents, returning top 3
361361
[2025-12-01 11:10:11] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
362362
[2025-12-01 11:10:11] INFO - project.pipeline.agents - Node 'generate' completed
363+
[2025-12-01 12:48:45] INFO - __main__ - LangSmith tracing enabled
364+
[2025-12-01 12:48:45] INFO - __main__ - Starting RAG application...
365+
[2025-12-01 12:48:45] INFO - project.utils.model_loader - GROQ API key loaded
366+
[2025-12-01 12:48:45] INFO - project.utils.model_loader - ModelLoader initialized
367+
[2025-12-01 12:48:46] INFO - project.utils.model_loader - Loaded Groq LLM: openai/gpt-oss-120b
368+
[2025-12-01 12:48:46] INFO - project.utils.model_loader - GROQ API key loaded
369+
[2025-12-01 12:48:46] INFO - project.utils.model_loader - ModelLoader initialized
370+
[2025-12-01 12:48:46] INFO - project.utils.model_loader - Loaded Groq LLM: openai/gpt-oss-120b
371+
[2025-12-01 12:48:46] INFO - project.source.data_preparation - DataPreparation initialized with chunk_size=1000
372+
[2025-12-01 12:48:46] INFO - project.utils.model_loader - GROQ API key loaded
373+
[2025-12-01 12:48:46] INFO - project.utils.model_loader - ModelLoader initialized
374+
[2025-12-01 12:48:49] INFO - project.utils.model_loader - Loaded FastEmbed: BAAI/bge-small-en-v1.5
375+
[2025-12-01 12:48:50] INFO - project.utils.model_loader - Loaded Groq LLM: openai/gpt-oss-120b
376+
[2025-12-01 12:48:50] INFO - project.model.retriever - DocumentRetriever initialized
377+
[2025-12-01 12:48:50] INFO - project.model.reranking - FlashRank reranker initialized with model: rank-T5-flan
378+
[2025-12-01 12:48:50] INFO - project.pipeline.rag - RAGPipeline initialized
379+
[2025-12-01 12:48:50] INFO - project.pipeline.agents - Web search tool initialized
380+
[2025-12-01 12:48:50] INFO - project.pipeline.agents - AgentWorkflow initialized
381+
[2025-12-01 12:48:50] INFO - __main__ - Setting up pipeline with Attention Is All You Need paper...
382+
[2025-12-01 12:48:50] INFO - project.source.data_preparation - Loading PDF from local file: data\attention-is-all-you-need.pdf
383+
[2025-12-01 12:48:51] INFO - project.source.data_preparation - Loaded 11 pages from PDF
384+
[2025-12-01 12:48:51] INFO - project.source.data_preparation - Split documents into 43 chunks
385+
[2025-12-01 12:48:51] INFO - project.source.data_preparation - Document preparation complete: 43 chunks ready
386+
[2025-12-01 12:48:58] INFO - faiss.loader - Loading faiss with AVX512 support.
387+
[2025-12-01 12:48:58] INFO - faiss.loader - Could not load library with AVX512 support due to:
388+
ModuleNotFoundError("No module named 'faiss.swigfaiss_avx512'")
389+
[2025-12-01 12:48:58] INFO - faiss.loader - Loading faiss with AVX2 support.
390+
[2025-12-01 12:48:58] INFO - faiss.loader - Successfully loaded faiss with AVX2 support.
391+
[2025-12-01 12:48:58] INFO - project.model.retriever - Vector store created with 43 documents
392+
[2025-12-01 12:48:58] INFO - project.model.retriever - Base retriever configured with mmr search
393+
[2025-12-01 12:48:58] INFO - project.pipeline.rag - RAG chain built successfully
394+
[2025-12-01 12:48:58] INFO - project.pipeline.rag - RAG pipeline setup complete
395+
[2025-12-01 12:48:58] INFO - project.pipeline.agents - LangGraph workflow compiled
396+
[2025-12-01 12:48:58] INFO - project.pipeline.agents - Agent workflow setup complete
397+
[2025-12-01 12:48:59] INFO - project.pipeline.agents - Workflow graph saved to workflow.png
398+
[2025-12-01 12:48:59] INFO - __main__ - Workflow graph saved
399+
[2025-12-01 12:48:59] INFO - project.pipeline.agents - ---RETRIEVE---
400+
[2025-12-01 12:48:59] INFO - project.pipeline.agents - Node 'retrieve' completed
401+
[2025-12-01 12:48:59] INFO - project.pipeline.agents - ---CHECK DOCUMENT RELEVANCE TO QUESTION---
402+
[2025-12-01 12:49:00] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
403+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
404+
[2025-12-01 12:49:00] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
405+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
406+
[2025-12-01 12:49:00] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
407+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
408+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - ---ASSESS GRADED DOCUMENTS---
409+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - ---DECISION: RELEVANT DOCUMENTS FOUND, GENERATE---
410+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - Node 'grade_documents' completed
411+
[2025-12-01 12:49:00] INFO - project.pipeline.agents - ---GENERATE---
412+
[2025-12-01 12:49:01] INFO - project.model.reranking - Reranked 3 documents, returning top 3
413+
[2025-12-01 12:49:03] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
414+
[2025-12-01 12:49:03] INFO - project.pipeline.agents - Node 'generate' completed
415+
[2025-12-01 12:49:03] INFO - project.pipeline.agents - ---RETRIEVE---
416+
[2025-12-01 12:49:03] INFO - project.pipeline.agents - Node 'retrieve' completed
417+
[2025-12-01 12:49:03] INFO - project.pipeline.agents - ---CHECK DOCUMENT RELEVANCE TO QUESTION---
418+
[2025-12-01 12:49:04] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
419+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
420+
[2025-12-01 12:49:04] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
421+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
422+
[2025-12-01 12:49:04] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
423+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
424+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - ---ASSESS GRADED DOCUMENTS---
425+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - ---DECISION: RELEVANT DOCUMENTS FOUND, GENERATE---
426+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - Node 'grade_documents' completed
427+
[2025-12-01 12:49:04] INFO - project.pipeline.agents - ---GENERATE---
428+
[2025-12-01 12:49:05] INFO - project.model.reranking - Reranked 3 documents, returning top 3
429+
[2025-12-01 12:49:07] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
430+
[2025-12-01 12:49:07] INFO - project.pipeline.agents - Node 'generate' completed
431+
[2025-12-01 12:49:07] INFO - project.pipeline.agents - ---RETRIEVE---
432+
[2025-12-01 12:49:07] INFO - project.pipeline.agents - Node 'retrieve' completed
433+
[2025-12-01 12:49:07] INFO - project.pipeline.agents - ---CHECK DOCUMENT RELEVANCE TO QUESTION---
434+
[2025-12-01 12:49:08] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
435+
[2025-12-01 12:49:08] INFO - project.pipeline.agents - ---GRADE: DOCUMENT NOT RELEVANT---
436+
[2025-12-01 12:49:08] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
437+
[2025-12-01 12:49:08] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
438+
[2025-12-01 12:49:09] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
439+
[2025-12-01 12:49:09] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
440+
[2025-12-01 12:49:09] INFO - project.pipeline.agents - ---ASSESS GRADED DOCUMENTS---
441+
[2025-12-01 12:49:09] INFO - project.pipeline.agents - ---DECISION: RELEVANT DOCUMENTS FOUND, GENERATE---
442+
[2025-12-01 12:49:09] INFO - project.pipeline.agents - Node 'grade_documents' completed
443+
[2025-12-01 12:49:09] INFO - project.pipeline.agents - ---GENERATE---
444+
[2025-12-01 12:49:09] INFO - project.model.reranking - Reranked 3 documents, returning top 3
445+
[2025-12-01 12:49:11] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
446+
[2025-12-01 12:49:11] INFO - project.pipeline.agents - Node 'generate' completed
447+
[2025-12-01 12:49:11] INFO - __main__ - RAG application completed successfully
448+
[2025-12-01 12:49:21] INFO - __main__ - Initializing RAG pipeline...
449+
[2025-12-01 12:49:21] INFO - project.utils.model_loader - GROQ API key loaded
450+
[2025-12-01 12:49:21] INFO - project.utils.model_loader - ModelLoader initialized
451+
[2025-12-01 12:49:22] INFO - project.utils.model_loader - Loaded Groq LLM: openai/gpt-oss-120b
452+
[2025-12-01 12:49:22] INFO - project.utils.model_loader - GROQ API key loaded
453+
[2025-12-01 12:49:22] INFO - project.utils.model_loader - ModelLoader initialized
454+
[2025-12-01 12:49:22] INFO - project.utils.model_loader - Loaded Groq LLM: openai/gpt-oss-120b
455+
[2025-12-01 12:49:22] INFO - project.source.data_preparation - DataPreparation initialized with chunk_size=1000
456+
[2025-12-01 12:49:22] INFO - project.utils.model_loader - GROQ API key loaded
457+
[2025-12-01 12:49:22] INFO - project.utils.model_loader - ModelLoader initialized
458+
[2025-12-01 12:49:26] INFO - project.utils.model_loader - Loaded FastEmbed: BAAI/bge-small-en-v1.5
459+
[2025-12-01 12:49:27] INFO - project.utils.model_loader - Loaded Groq LLM: openai/gpt-oss-120b
460+
[2025-12-01 12:49:27] INFO - project.model.retriever - DocumentRetriever initialized
461+
[2025-12-01 12:49:28] INFO - project.model.reranking - FlashRank reranker initialized with model: rank-T5-flan
462+
[2025-12-01 12:49:28] INFO - project.pipeline.rag - RAGPipeline initialized
463+
[2025-12-01 12:49:28] INFO - project.pipeline.agents - Web search tool initialized
464+
[2025-12-01 12:49:28] INFO - project.pipeline.agents - AgentWorkflow initialized
465+
[2025-12-01 12:49:28] INFO - project.source.data_preparation - Loading PDF from local file: data\attention-is-all-you-need.pdf
466+
[2025-12-01 12:49:29] INFO - project.source.data_preparation - Loaded 11 pages from PDF
467+
[2025-12-01 12:49:29] INFO - project.source.data_preparation - Split documents into 43 chunks
468+
[2025-12-01 12:49:29] INFO - project.source.data_preparation - Document preparation complete: 43 chunks ready
469+
[2025-12-01 12:49:40] INFO - faiss.loader - Loading faiss with AVX512 support.
470+
[2025-12-01 12:49:40] INFO - faiss.loader - Could not load library with AVX512 support due to:
471+
ModuleNotFoundError("No module named 'faiss.swigfaiss_avx512'")
472+
[2025-12-01 12:49:40] INFO - faiss.loader - Loading faiss with AVX2 support.
473+
[2025-12-01 12:49:40] INFO - faiss.loader - Successfully loaded faiss with AVX2 support.
474+
[2025-12-01 12:49:40] INFO - project.model.retriever - Vector store created with 43 documents
475+
[2025-12-01 12:49:40] INFO - project.model.retriever - Base retriever configured with mmr search
476+
[2025-12-01 12:49:40] INFO - project.pipeline.rag - RAG chain built successfully
477+
[2025-12-01 12:49:40] INFO - project.pipeline.rag - RAG pipeline setup complete
478+
[2025-12-01 12:49:40] INFO - project.pipeline.agents - LangGraph workflow compiled
479+
[2025-12-01 12:49:40] INFO - project.pipeline.agents - Agent workflow setup complete
480+
[2025-12-01 12:49:40] INFO - __main__ - RAG pipeline ready
481+
[2025-12-01 12:50:13] INFO - project.pipeline.agents - ---RETRIEVE---
482+
[2025-12-01 12:50:13] INFO - project.pipeline.agents - Node 'retrieve' completed
483+
[2025-12-01 12:50:13] INFO - project.pipeline.agents - ---CHECK DOCUMENT RELEVANCE TO QUESTION---
484+
[2025-12-01 12:50:14] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
485+
[2025-12-01 12:50:14] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
486+
[2025-12-01 12:50:14] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
487+
[2025-12-01 12:50:14] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
488+
[2025-12-01 12:50:15] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
489+
[2025-12-01 12:50:15] INFO - project.pipeline.agents - ---GRADE: DOCUMENT RELEVANT---
490+
[2025-12-01 12:50:15] INFO - project.pipeline.agents - ---ASSESS GRADED DOCUMENTS---
491+
[2025-12-01 12:50:15] INFO - project.pipeline.agents - ---DECISION: RELEVANT DOCUMENTS FOUND, GENERATE---
492+
[2025-12-01 12:50:15] INFO - project.pipeline.agents - Node 'grade_documents' completed
493+
[2025-12-01 12:50:15] INFO - project.pipeline.agents - ---GENERATE---
494+
[2025-12-01 12:50:16] INFO - project.model.reranking - Reranked 3 documents, returning top 3
495+
[2025-12-01 12:50:18] INFO - httpx - HTTP Request: POST https://api.groq.com/openai/v1/chat/completions "HTTP/1.1 200 OK"
496+
[2025-12-01 12:50:18] INFO - project.pipeline.agents - Node 'generate' completed

templates/index.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<head>
44
<meta charset="UTF-8">
55
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6-
<title>Learn with Transformers</title>
6+
<title>RAG PROJECT PIPELINE</title>
77
<link rel="stylesheet" href="/static/styles.css">
88
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
99
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>

0 commit comments

Comments
 (0)