You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* update readme for cn,en
* update readme for cn,en
* update readme for cn,en
* minor format tweak
---------
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
Copy file name to clipboardExpand all lines: README.md
+69-43Lines changed: 69 additions & 43 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,60 +1,85 @@
1
1
# KAG: Knowledge Augmented Generation
2
2
3
-
[中文版文档](./README_cn.md)
4
-
[日本語版ドキュメント](./README_ja.md)
5
-
6
-
## 1. What is KAG
7
-
8
-
Retrieval Augmentation Generation (RAG) technology promotes the integration of domain applications with large language models. However, RAG has problems such as a large gap between vector similarity and knowledge reasoning correlation, and insensitivity to knowledge logic (such as numerical values, time relationships, expert rules, etc.), which hinder the implementation of professional knowledge services.
9
-
10
-
On October 24, 2024, OpenSPG released v0.5, officially releasing the professional domain knowledge service framework of knowledge augmented generation (KAG). The goal of KAG is to build a knowledge-enhanced LLM service framework in professional domains, supporting logical reasoning, factual Q&A, etc. KAG fully integrates the logical and factual characteristics of the KGs. Meanwhile, it uses OpenIE to lower the threshold for knowledgeization of domain documents and alleviates the sparsity problem of the KG through hybrid reasoning. As far as we know, KAG is the only RAG framework that supports logical reasoning and multi-hop factual Q&A. Its core features include:
11
-
12
-
* Knowledge and Chunk Mutual Indexing structure to integrate more complete contextual text information
13
-
* Knowledge alignment using conceptual semantic reasoning to alleviate the noise problem caused by OpenIE
14
-
* Schema-constrained knowledge construction to support the representation and construction of domain expert knowledge
15
-
* Logical form-guided hybrid reasoning and retrieval to support logical reasoning and multi-hop reasoning Q&A
16
-
17
-
KAG is significantly better than NaiveRAG, HippoRAG and other methods in multi-hop Q&A tasks. The F1 score on hotpotQA is relatively increased by 19.6%, and the F1 score on 2wiki is relatively increased by 33.5%. We have successfully applied KAG to Ant Group's professional knowledge Q&A tasks, such as e-government Q&A and e-health Q&A, and the professionalism has been significantly improved compared to the traditional RAG method.
KAG is a logical reasoning and Q&A framework based on the [OpenSPG](https://github.com/OpenSPG/openspg) engine and large language models, which is used to build logical reasoning and Q&A solutions for vertical domain knowledge bases. KAG can effectively overcome the ambiguity of traditional RAG vector similarity calculation and the noise problem of GraphRAG introduced by OpenIE. KAG supports logical reasoning and multi-hop fact Q&A, etc., and is significantly better than the current SOTA method.
28
+
29
+
The goal of KAG is to build a knowledge-enhanced LLM service framework in professional domains, supporting logical reasoning, factual Q&A, etc. KAG fully integrates the logical and factual characteristics of the KGs. Its core features include:
30
+
31
+
- Knowledge and Chunk Mutual Indexing structure to integrate more complete contextual text information
32
+
- Knowledge alignment using conceptual semantic reasoning to alleviate the noise problem caused by OpenIE
33
+
- Schema-constrained knowledge construction to support the representation and construction of domain expert knowledge
34
+
- Logical form-guided hybrid reasoning and retrieval to support logical reasoning and multi-hop reasoning Q&A
18
35
19
36
⭐️ Star our repository to stay up-to-date with exciting new features and improvements! Get instant notifications for new releases! 🌟
20
37
21
38
22
39
23
-
## 2 Core Features
40
+
#2. Core Features
24
41
25
-
###2.1 Knowledge Representation
42
+
## 2.1 Knowledge Representation
26
43
27
-
In the context of private knowledge bases, unstructured data, structured information, and business expert experience often coexist. KAG references the DIKW hierarchy to upgrade SPG to a version that is friendly to LLMs. For unstructured data such as news, events, logs, and books, as well as structured data like transactions, statistics, and approvals, along with business experience and domain knowledge rules, KAG employs techniques such as layout analysis, knowledge extraction, property normalization, and semantic alignment to integrate raw business data and expert rules into a unified business knowledge graph.
44
+
In the context of private knowledge bases, unstructured data, structured information, and business expert experience often coexist. KAG references the DIKW hierarchy to upgrade SPG to a version that is friendly to LLMs.
45
+
46
+
For unstructured data such as news, events, logs, and books, as well as structured data like transactions, statistics, and approvals, along with business experience and domain knowledge rules, KAG employs techniques such as layout analysis, knowledge extraction, property normalization, and semantic alignment to integrate raw business data and expert rules into a unified business knowledge graph.
28
47
29
48
30
49
31
-
This makes it compatible with schema-free information extraction and schema-constrained expertise construction on the same knowledge type (e. G., entity type, event type), and supports the cross-index representation between the graph structure and the original text block. This mutual index representation is helpful to the construction of inverted index based on graph structure, and promotes the unified representation and reasoning of logical forms.
50
+
This makes it compatible with schema-free information extraction and schema-constrained expertise construction on the same knowledge type (e. G., entity type, event type), and supports the cross-index representation between the graph structure and the original text block.
51
+
52
+
This mutual index representation is helpful to the construction of inverted index based on graph structure, and promotes the unified representation and reasoning of logical forms.
32
53
33
-
###2.2 Mixed Reasoning Guided by Logic Forms
54
+
## 2.2 Mixed Reasoning Guided by Logic Forms
34
55
35
56
36
57
37
-
KAG proposes a logically formal guided hybrid solution and inference engine. The engine includes three types of operators: planning, reasoning, and retrieval, which transform natural language problems into problem solving processes that combine language and notation. In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.
58
+
KAG proposes a logically formal guided hybrid solution and inference engine.
59
+
60
+
The engine includes three types of operators: planning, reasoning, and retrieval, which transform natural language problems into problem solving processes that combine language and notation.
38
61
62
+
In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.
39
63
40
-
##3. Release Notes
64
+
# 3. Release Notes
41
65
42
-
### 3.1 Released Versions
43
-
* 2024.11.21 : Support docs upload, model invoke concurrency setting, User experience optimization, etc.
Navigate to the default url of the KAG product with your browser: <http://127.0.0.1:8887>
87
112
88
113
See the [Product](https://openspg.yuque.com/ndx6g9/wc9oyq/rgd8ecefccwd1ga5) guide for detailed introduction.
89
114
90
-
###4.2 toolkit-based (for developers)
115
+
## 4.2 toolkit-based (for developers)
91
116
92
-
####4.2.1 Engine & Dependent Image Installation
117
+
### 4.2.1 Engine & Dependent Image Installation
93
118
94
119
Refer to the 3.1 section to complete the installation of the engine & dependent image.
95
120
96
-
#### 4.2.2 Installation of KAG
121
+
### 4.2.2 Installation of KAG
122
+
97
123
98
124
**macOS / Linux developers**
99
125
@@ -117,33 +143,33 @@ Refer to the 3.1 section to complete the installation of the engine & dependent
117
143
# Install KAG: cd KAG && pip install -e .
118
144
```
119
145
120
-
####4.2.3 Use the toolkit
146
+
### 4.2.3 Use the toolkit
121
147
122
148
Please refer to the [Quick Start](https://openspg.yuque.com/ndx6g9/wc9oyq/owp4sxbdip2u7uvv) guide for detailed introduction of the toolkit. Then you can use the built-in components to reproduce the performance results of the built-in datasets, and apply those components to new busineness scenarios.
The KAG framework includes three parts: kg-builder, kg-solver, and kag-model. This release only involves the first two parts, kag-model will be gradually open source release in the future.
129
155
130
156
kg-builder implements a knowledge representation that is friendly to large-scale language models (LLM). Based on the hierarchical structure of DIKW (data, information, knowledge and wisdom), IT upgrades SPG knowledge representation ability, and is compatible with information extraction without schema constraints and professional knowledge construction with schema constraints on the same knowledge type (such as entity type and event type), it also supports the mutual index representation between the graph structure and the original text block, which supports the efficient retrieval of the reasoning question and answer stage.
131
157
132
158
kg-solver uses a logical symbol-guided hybrid solving and reasoning engine that includes three types of operators: planning, reasoning, and retrieval, to transform natural language problems into a problem-solving process that combines language and symbols. In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.
0 commit comments