Simple Yet Scalable Log Ingestor
Live at https://dyte-logi.kunwar.dev/
- LOGI is a log ingestor system tailored to efficiently manage vast volumes of log data.
- It offers seamless handling of data through APIs or a user-friendly Dashboard interface.
- Real time log ingestion and Search Functionalities.
- Paginated Search
- Users can easily query this data using:
- Full-text search capabilities
- Specific field filters for precision
- Introducing Lquery, a novel query language that:
- Handles regular expressions
- Supports startsWith and endsWith functions
- Allows combining multiple filters
- Facilitates nested field queries
- Enables comprehensive full-text search functionality
- Search within timestamps
- Supports Pagination
The Logi dashboard and the Lquery language serve distinct purposes and enhance the functionality of the LOGI log ingestor system
- Advanced Querying: Offers a specialized language for complex log data searches and filters.
- Enhanced Search: Supports intricate searches using regular expressions and various functions.
- Optimized Data Retrieval: Allows precise extraction of relevant information for quicker analysis.
- Docker: Platform for containerization and deployment.
- NestJS: Backend framework for building scalable applications.
- Bull (Node.js): Redis-backed queue library for Node.js.
- Redis: In-memory data structure store used as a database, cache, and message broker.
- MongoDB: NoSQL database for flexible, scalable data storage.
- WebSockets
- TypeScript
- Vue.js: Progressive JavaScript framework for building user interfaces.
- Vuetify: Material Design component library for Vue.js.
- Socket.io: Real-time bidirectional event-based communication library.
Set Env variables in docker-compose.yml
Backend Env
environment:
DB_URI: <mongodb-uri>
REDIS_HOST: <redis-host>
REDIS_PORT: <redis-port>
Frontend Env
Currently the api url is set to https://dyte-logi-api.kunwar.dev. You can change it in docker-compose.yml to use local
api url.
environment:
VITE_BASE_URL: <Insert API URL Here>
Finally Run the project using
docker compose up -d
- NestJS Architecture: Utilizing NestJS, that supports modular and scalable architecture, enabling efficient handling of increased log volumes by scaling components as needed.
- Bull (Message Queue): For handling background jobs asynchronously, allowing the system to handle more requests per second and log processing in distributed manner. Thus, reducing the impact I/O-heavy tasks on the main app, preventing bottlenecks caused by blocking operations.
- Optimized DB queries: Using MongoDB's indexing capabilities to optimize queries for log data retrieval. Using text search indexes on multiple fields to improve full text lookups.
- Horizontal Scaling: Employing technologies and architectures that support horizontal scaling. This involves distributing the workload across multiple instances or nodes.
- Code Optimization: Ensuring efficient code practices, including optimizing algorithms, minimizing unnecessary I/O operations, and employing best practices for handling large data sets.
- Docker: Using docker for easy containerization, ensuring scalability through the deployment of multiple instances or microservices as log volumes increase.
- TypeScript: Employing TypeScript for backend development, enabling better code maintainability, scalability, and error detection, contributing to the system's overall scalability and robustness.
- Websocket Implementation: For real-time communication, enabling seamless data streaming.
- Real-Time Updates: Utilizing WebSockets to establish a persistent connection between the client and the server allows for real-time data transmission.
- Websocket Integration: Utilized 'socket.io-client' in Vuejs to implement websockets and subscribe to relevant events. Developing Websocket server using socket.io in Nestjs to handle incoming connections and broadcasting data.
- Live Tail Component: Made a vuejs service for displaying live log updates. This component subscribes to WebSocket events(Log Creation) and updates the UI dynamically as new log data arrives.
Backend port running on 3000
POST /ingestor
Body:
{
"level": "success",
"message": "NewLOGGGG to DB",
"resourceID": "server-1234",
"timestamp": "2023-09-15T08:00:00Z",
"traceID": "abc-xyz-123",
"spanID": "span-456",
"commit": "5e5342f",
"metadata": {
"parentResourceId": "server-0987"
}
}POST /ingestor/bulk
Body:
[
{
"level": "success",
"message": "NewLOGGGG to DB",
"resourceID": "server-1234",
"timestamp": "2023-09-15T08:00:00Z",
"traceID": "abc-xyz-123",
"spanID": "span-456",
"commit": "5e5342f",
"metadata": {
"parentResourceId": "server-0987"
}
}
]POST /ingestor/json-file
Send Json file which contains list of logs with multipart form with the field 'file'
- Seamless UI which adjusts to any screen size across all devices.
- Live updates when adding new logs.
- Search box to write Lquery and datetime buttons for lookup between timestamps
- Paginated Searches
- Add logs through UI. With json text or json file.
- Normal lookup
foo:bar
- Special Keyword to perform Full Text search
fts:'full text search query here'
- Regex lookup
foo:r'regex-code-here'
- Match string starting with
foo:bar*
- Match string ending with
foo:*bar
- Use multiple filters
foo:bar1,bar2,bar3
or
foo:'bar1,bar2,bar3'
- Filtering Nested fields
foo1.foo2:bar
- Filtering logs by date
- Use dashboard datetime input fields
- Use dashboard datetime input fields
Create your Lquery syntax and append each of them in space seperated strings Example:
level:error,success fts:'Error in DB' spanID:r'<regex>' commit:*123 resourceID:abd* metadata.foo:bar
- Lquery fails when there are multiple spaces between two queries. Ex:
"level:success foo:bar" - Frontend sends API twice sometimes
Kunwar Pratap Singh
ksingh2001.dev@gmail.com
Live Project Link: https://dyte-logi.kunwar.dev/
Project Link: [https://github.com/dyte-submissions/november-2023-hiring-ToxicityMax](Repo Link)