Skip to content

README_en.md

Latest commit

 

History

History
222 lines (160 loc) · 11.7 KB

README_en.md

File metadata and controls

222 lines (160 loc) · 11.7 KB

All-in-RAG | Large Model Application Development Practice: RAG Technology Full-Stack Guide

All-in-RAG Logo

🔍 Retrieval-Augmented Generation (RAG) Technology Full-Stack Guide

From theory to practice, from basics to advanced, build your RAG technology system

GitHub stars GitHub forks Python

🎯 Systematic Learning
Complete RAG technology system		 🛠️ Hands-on Practice
Rich project examples		 🚀 Production Ready
Engineering best practices		 📊 Multimodal Support
Text + Image retrieval

Project Introduction（中文 | English）

This project is a comprehensive RAG (Retrieval-Augmented Generation) technology full-stack tutorial for large model application developers. It aims to help developers master RAG application development skills based on large language models through systematic learning paths and hands-on practice projects, building production-grade intelligent Q&A and knowledge retrieval systems.

Main content includes:

  1. RAG Technology Fundamentals: In-depth introduction to RAG core concepts, technical principles, and application scenarios
  2. Complete Data Processing Pipeline: From data loading, cleaning to text chunking - the complete data preparation process
  3. Index Construction and Optimization: Vector embedding, multimodal embedding, vector database construction and index optimization techniques
  4. Advanced Retrieval Techniques: Hybrid retrieval, query construction, Text2SQL and other advanced retrieval technologies
  5. Generation Integration and Evaluation: Formatted generation, system evaluation and optimization methods
  6. Project Practice: Complete RAG application development practice from basic to advanced

Project Significance

With the rapid development of large language models, RAG technology has become the core technology for building intelligent Q&A systems and knowledge retrieval applications. However, existing RAG tutorials are often scattered and lack systematicity, making it difficult for beginners to form a complete understanding of the technical system.

Starting from practice and combining the latest RAG technology development trends, this project builds a complete RAG learning system to help developers:

  • Systematically master the theoretical foundation and practical skills of RAG technology
  • Understand the complete architecture of RAG systems and the role of each component
  • Develop the ability to independently develop RAG applications
  • Master evaluation and optimization methods for RAG systems

Target Audience

This project is suitable for the following groups:

  • Developers with Python programming foundation who are interested in RAG technology
  • AI engineers who want to systematically learn RAG technology
  • Product developers who want to build intelligent Q&A systems
  • Researchers with learning needs for retrieval-augmented generation technology

Prerequisites:

  • Master Python basic syntax and usage of common libraries
  • Ability to use Docker simply
  • Understanding of basic LLM concepts (recommended but not required)
  • Basic Linux command line operation skills

Project Highlights

  1. Systematic Learning Path: From basic concepts to advanced applications, building a complete RAG technology learning system
  2. Theory and Practice Combined: Each chapter includes theoretical explanation and code practice to ensure learning and application
  3. Multimodal Support: Covers not only text RAG, but also multimodal embedding and retrieval technologies
  4. Engineering-Oriented: Focus on engineering problems in practical applications, including performance optimization, system evaluation, etc.
  5. Rich Practical Projects: Provides multiple practical projects from basic to advanced to help consolidate learning outcomes

Content Outline

Part I: RAG Fundamentals

Chapter 1 Unlocking RAG 📖 View Chapter

  1. RAG Introduction - RAG technology overview and application scenarios
  2. Preparation - Environment configuration and preparation
  3. Four Steps to Build RAG - Quick start with RAG development

Chapter 2 Data Preparation 📖 View Chapter

  1. Data Loading - Multi-format document processing and loading
  2. Text Chunking - Text segmentation strategies and optimization

Part II: Index Construction and Optimization

Chapter 3 Index Construction 📖 View Chapter

  1. Vector Embedding - Detailed explanation of text vectorization technology
  2. Multimodal Embedding - Image-text multimodal vectorization
  3. Vector Database - Vector storage and retrieval systems
  4. Milvus Practice - Milvus multimodal retrieval practice
  5. Index Optimization - Index performance tuning techniques

Part III: Advanced Retrieval Techniques

Chapter 4 Retrieval Optimization 📖 View Chapter

  1. Hybrid Search - Dense + sparse retrieval fusion
  2. Query Construction - Intelligent query understanding and construction
  3. Text2SQL - Natural language to SQL query
  4. Query Rewriting and Routing - Query optimization strategies
  5. Advanced Retrieval Techniques - Advanced retrieval algorithms

Part IV: Generation and Evaluation

Chapter 5 Generation Integration 📖 View Chapter

  1. Formatted Generation - Structured output and format control

Chapter 6 RAG System Evaluation 📖 View Chapter

  1. Evaluation Introduction - RAG system evaluation methodology
  2. Evaluation Tools - Common evaluation tools and metrics

Part V: Advanced Applications and Practice

Chapter 7 Advanced RAG Architecture (Extended Elective) 📖 View Chapter

  1. Knowledge Graph-based RAG

Chapter 8 Project Practice I (Basic) 📖 View Chapter

  1. Environment Configuration and Project Architecture
  2. Data Preparation Module Implementation
  3. Index Construction and Retrieval Optimization
  4. Generation Integration and System Integration

Chapter 9 Project Practice I Optimization (Elective) 📖 View Chapter

🍽️ Project Demo

  1. Graph RAG Architecture Design
  2. Graph Data Modeling and Preparation
  3. Milvus Index Construction
  4. Intelligent Query Routing and Retrieval Strategy

Chapter 10 Project Practice II (Elective) 📖 View Chapter In Planning

Directory Structure

all-in-rag/
├── docs/           # Tutorial documentation
├── code/           # Code examples
├── data/           # Sample data
├── models/         # Pre-trained models
└── README.md       # Project description

Practical Project Showcase

Chapter 8 Project I:

Project I

Chapter 9 Project I (Graph RAG Optimization):

Project I (Graph RAG Optimization)

Chapter 10 Project II:

Acknowledgments

Core Contributors

Special Thanks

  • Thanks to @Sm1les for help and support on this project
  • Thanks to all developers who contributed to this project
  • Thanks to the open source community for providing excellent tools and framework support
  • Special thanks to the following developers who contributed to the tutorial!

Contributors

Made with contrib.rocks.

Contributing

We welcome all forms of contributions, including but not limited to:

  • 🚨 Bug Reports: Please submit Issues if you find problems
  • 💭 Feature Suggestions: Welcome to discuss good ideas in Discussions
  • 📚 Documentation Improvement: Help improve documentation content and example code
  • ⚡ Code Contributions: Submit Pull Requests to improve the project

Star History

all-in-rag stats

If this project helps you, please give us a ⭐️

Let more people discover this project (Food protection? Bring it on!)

star

About Datawhale

Datawhale

Scan the QR code to follow Datawhale WeChat Official Account for more quality open source content

License

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.