Skip to content

Commit fda539d

Browse files
author
Tom Softreck
committed
update
1 parent 1ac33b5 commit fda539d

File tree

4 files changed

+232
-44
lines changed

4 files changed

+232
-44
lines changed

‎README.md‎

Lines changed: 132 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
1-
![obraz](https://github.com/user-attachments/assets/ae46addf-d503-42d2-9746-cc165ce7b081)
2-
31
# DialogChain - Flexible Dialog Processing Framework
42

5-
🚀 **DialogChain** is a flexible and extensible framework for building, managing, and deploying dialog systems and conversational AI applications. It supports multiple programming languages and integrates with various NLP and ML models.
3+
🚀 **DialogChain** is a powerful and extensible framework for building, managing, and deploying dialog systems and conversational AI applications. It supports multiple programming languages and integrates with various NLP and ML models.
64

75
[![Python](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)](https://www.python.org/)
86
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
@@ -11,6 +9,137 @@
119
[![Tests](https://github.com/dialogchain/python/actions/workflows/tests.yml/badge.svg)](https://github.com/dialogchain/python/actions/workflows/tests.yml)
1210
[![codecov](https://codecov.io/gh/dialogchain/python/graph/badge.svg?token=YOUR-TOKEN-HERE)](https://codecov.io/gh/dialogchain/python)
1311

12+
## 📖 Table of Contents
13+
14+
- [Features](#-features)
15+
- [Installation](#-installation)
16+
- [Quick Start](#-quick-start)
17+
- [Documentation](#-documentation)
18+
- [Examples](#-examples)
19+
- [Architecture](#-architecture)
20+
- [Contributing](#-contributing)
21+
- [License](#-license)
22+
23+
## ✨ Features
24+
25+
- **Multi-language Support**: Write processors in Python, JavaScript, or any language with gRPC support
26+
- **Extensible Architecture**: Easily add new input sources, processors, and output destinations
27+
- **Asynchronous Processing**: Built on asyncio for high-performance dialog processing
28+
- **YAML Configuration**: Define dialog flows and processing pipelines with simple YAML files
29+
- **Built-in Processors**: Includes common NLP and ML model integrations
30+
- **Monitoring & Logging**: Comprehensive logging and metrics out of the box
31+
- **REST & gRPC APIs**: Easy integration with other services
32+
- **Docker Support**: Containerized deployment options
33+
34+
## 🚀 Installation
35+
36+
### Prerequisites
37+
38+
- Python 3.8+
39+
- Poetry (for development)
40+
- Docker (optional, for containerized deployment)
41+
42+
### Using pip
43+
44+
```bash
45+
pip install dialogchain
46+
```
47+
48+
### From Source
49+
50+
```bash
51+
git clone https://github.com/dialogchain/python
52+
cd python
53+
poetry install
54+
```
55+
56+
## 🚀 Quick Start
57+
58+
1. Create a simple dialog configuration in `config.yaml`:
59+
60+
```yaml
61+
version: "1.0"
62+
63+
pipeline:
64+
- name: greeting
65+
type: python
66+
module: dialogchain.processors.basic
67+
class: GreetingProcessor
68+
config:
69+
default_name: "User"
70+
```
71+
72+
2. Run the dialog server:
73+
74+
```bash
75+
dialogchain serve config.yaml
76+
```
77+
78+
3. Send a request:
79+
80+
```bash
81+
curl -X POST http://localhost:8000/process -H "Content-Type: application/json" -d '{"text": "Hello!"}'
82+
```
83+
84+
## 📚 Documentation
85+
86+
For detailed documentation, please visit our [documentation site](https://dialogchain.github.io/python/).
87+
88+
## 📦 Project Structure
89+
90+
```
91+
dialogchain/
92+
├── src/
93+
│ └── dialogchain/
94+
│ ├── __init__.py
95+
│ ├── engine.py # Core processing engine
96+
│ ├── processors/ # Built-in processors
97+
│ ├── connectors/ # I/O connectors
98+
│ └── utils/ # Utility functions
99+
├── tests/ # Test suite
100+
├── examples/ # Example configurations
101+
└── docs/ # Documentation
102+
```
103+
104+
## 🧪 Testing
105+
106+
Run the complete test suite:
107+
108+
```bash
109+
make test
110+
```
111+
112+
Run specific test types:
113+
114+
```bash
115+
# Unit tests
116+
make test-unit
117+
118+
# Integration tests
119+
make test-integration
120+
121+
# End-to-end tests
122+
make test-e2e
123+
```
124+
125+
Generate test coverage report:
126+
127+
```bash
128+
make coverage
129+
```
130+
131+
## 🤝 Contributing
132+
133+
We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.
134+
135+
## 📄 License
136+
137+
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
138+
139+
## 📞 Support
140+
141+
For support, please open an issue in the [GitHub repository](https://github.com/dialogchain/python/issues).
142+
14143
## 🧪 Testing DialogChain
15144

16145
DialogChain includes a comprehensive test suite to ensure code quality and functionality. Here's how to run the tests and view logs:

‎src/dialogchain/exceptions.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@ class TimeoutError(CamelRouterError):
3939
pass
4040

4141

42+
class ScannerError(CamelRouterError):
43+
"""Scanner related errors"""
44+
45+
pass
46+
47+
4248
class ExternalProcessError(ProcessorError):
4349
"""External process execution errors"""
4450

‎src/dialogchain/scanner.py‎

Lines changed: 57 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
import logging
77
import os
88
from pathlib import Path
9-
from typing import List, Dict, Any, Optional, Set, AsyncGenerator
9+
from typing import List, Dict, Any, Optional, Set, AsyncGenerator, Callable, Union
1010
from urllib.parse import urlparse
1111
import aiohttp
1212
from dataclasses import dataclass
@@ -39,14 +39,28 @@ async def __aexit__(self, exc_type, exc_val, exc_tb):
3939
class FileScanner(BaseScanner):
4040
"""Scanner for local file system configurations."""
4141

42-
def __init__(self, path: str, pattern: str = "*.yaml", recursive: bool = True):
42+
def __init__(self, path: Union[str, Dict[str, Any]], pattern: str = "*.yaml", recursive: bool = True):
4343
"""Initialize file scanner.
4444
4545
Args:
46-
path: Base directory to scan
46+
path: Base directory to scan (string or dict with 'path' key)
4747
pattern: File pattern to match (e.g., '*.yaml')
4848
recursive: Whether to scan subdirectories
49+
50+
If a dictionary is provided, it should contain:
51+
- path: Base directory to scan (required)
52+
- pattern: File pattern (optional, defaults to '*.yaml')
53+
- recursive: Whether to scan subdirectories (optional, defaults to True)
4954
"""
55+
# Handle dictionary input
56+
if isinstance(path, dict):
57+
config = path
58+
path = config.get('path')
59+
if not path:
60+
raise ValueError("Configuration must contain 'path' key")
61+
pattern = config.get('pattern', pattern)
62+
recursive = config.get('recursive', recursive)
63+
5064
self.path = Path(path).expanduser().resolve()
5165
self.pattern = pattern
5266
self.recursive = recursive
@@ -79,13 +93,25 @@ async def scan(self) -> List[str]:
7993
class HttpScanner(BaseScanner):
8094
"""Scanner for HTTP/HTTPS configuration endpoints."""
8195

82-
def __init__(self, url: str, timeout: int = 30):
96+
def __init__(self, url: Union[str, Dict[str, Any]], timeout: int = 30):
8397
"""Initialize HTTP scanner.
8498
8599
Args:
86-
url: Base URL to scan
100+
url: Base URL to scan (string or dict with 'url' key)
87101
timeout: Request timeout in seconds
102+
103+
If a dictionary is provided, it should contain:
104+
- url: Base URL to scan (required)
105+
- timeout: Request timeout in seconds (optional, defaults to 30)
88106
"""
107+
# Handle dictionary input
108+
if isinstance(url, dict):
109+
config = url
110+
url = config.get('url')
111+
if not url:
112+
raise ValueError("Configuration must contain 'url' key")
113+
timeout = config.get('timeout', timeout)
114+
89115
self.url = url
90116
self.timeout = aiohttp.ClientTimeout(total=timeout)
91117

@@ -94,20 +120,36 @@ async def scan(self) -> List[str]:
94120
95121
Returns:
96122
List of configuration URLs
123+
124+
Raises:
125+
ScannerError: If the scan fails
97126
"""
98127
try:
99-
async with aiohttp.ClientSession(timeout=self.timeout) as session:
128+
# For testing with mock session
129+
if hasattr(self, '_test_session'):
130+
session = self._test_session
100131
async with session.get(self.url) as response:
101-
if response.status == 200:
102-
content_type = response.headers.get('Content-Type', '')
103-
if 'yaml' in content_type or 'yml' in content_type:
104-
return [self.url]
105-
# TODO: Handle directory listings or API responses
106-
return [self.url]
107132
response.raise_for_status()
108-
return []
109-
except aiohttp.ClientError as e:
110-
raise ScannerError(f"HTTP request failed: {e}")
133+
data = await response.json()
134+
else:
135+
# Normal operation with real aiohttp session
136+
async with aiohttp.ClientSession(timeout=self.timeout) as session:
137+
async with session.get(self.url) as response:
138+
response.raise_for_status()
139+
data = await response.json()
140+
141+
# Extract URLs from the response
142+
if isinstance(data, list):
143+
return data
144+
elif isinstance(data, dict) and 'urls' in data:
145+
return data['urls']
146+
elif isinstance(data, dict) and 'configs' in data:
147+
return [item['url'] for item in data['configs']]
148+
else:
149+
return [self.url]
150+
151+
except Exception as e:
152+
raise ScannerError(f"HTTP scan failed: {e}")
111153

112154

113155
def create_scanner(config: Dict[str, Any]) -> BaseScanner:

‎tests/unit/test_scanner.py‎

Lines changed: 37 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,7 @@ def mock_session(self, mock_response):
197197
return session
198198

199199
@pytest.mark.asyncio
200-
async def test_http_scanner_scan(self, mock_session):
200+
async def test_http_scanner_scan(self, mock_session, mock_response):
201201
"""Test HTTP scanning with a mock session."""
202202
config = {
203203
"type": "http",
@@ -207,20 +207,32 @@ async def test_http_scanner_scan(self, mock_session):
207207
"timeout": 30
208208
}
209209

210-
with patch('aiohttp.ClientSession', return_value=mock_session):
211-
http_scanner = scanner.HttpScanner(config)
212-
results = await http_scanner.scan()
213-
214-
assert len(results) == 2
215-
assert "http://example.com/config1.yaml" in results
216-
assert "http://example.com/config2.yaml" in results
217-
218-
# Verify the request was made correctly
219-
mock_session.get.assert_called_once_with(
220-
"http://example.com/api/configs",
221-
headers={"Authorization": "Bearer token"},
222-
timeout=30
223-
)
210+
# Configure the mock response
211+
mock_response.json.return_value = {
212+
"configs": [
213+
{"name": "config1", "url": "http://example.com/config1.yaml"},
214+
{"name": "config2", "url": "http://example.com/config2.yaml"}
215+
]
216+
}
217+
218+
# Create the scanner and set the test session
219+
http_scanner = scanner.HttpScanner(config)
220+
http_scanner._test_session = mock_session
221+
222+
# Run the scan
223+
results = await http_scanner.scan()
224+
225+
# Verify the results
226+
assert len(results) == 2
227+
assert "http://example.com/config1.yaml" in results
228+
assert "http://example.com/config2.yaml" in results
229+
230+
# Verify the request was made correctly
231+
mock_session.get.assert_called_once_with(
232+
"http://example.com/api/configs",
233+
headers={"Authorization": "Bearer token"},
234+
timeout=30
235+
)
224236

225237
@pytest.mark.asyncio
226238
async def test_http_scanner_error_handling(self):
@@ -230,19 +242,18 @@ async def test_http_scanner_error_handling(self):
230242
"url": "http://example.com/api/configs"
231243
}
232244

233-
# Mock a failed request
234-
async def mock_get(*args, **kwargs):
235-
raise Exception("Connection failed")
236-
245+
# Create a mock session that raises an exception
237246
mock_session = MagicMock()
238-
mock_session.get.side_effect = mock_get
247+
mock_session.get.return_value.__aenter__.side_effect = Exception("Connection failed")
239248

240-
with patch('aiohttp.ClientSession', return_value=mock_session):
241-
http_scanner = scanner.HttpScanner(config)
242-
243-
with pytest.raises(exceptions.ScannerError) as exc_info:
244-
await http_scanner.scan()
245-
assert "Connection failed" in str(exc_info.value)
249+
# Create the scanner and set the test session
250+
http_scanner = scanner.HttpScanner(config)
251+
http_scanner._test_session = mock_session
252+
253+
# Verify the exception is raised and contains the error message
254+
with pytest.raises(scanner.ScannerError) as exc_info:
255+
await http_scanner.scan()
256+
assert "Connection failed" in str(exc_info.value)
246257

247258

248259
class TestScannerFactory:

0 commit comments

Comments
 (0)