[Feature] Consider adding VictoriaLogs as a log storage backend

[valyala]

Feature Description

It would be great if Logtide could use VictoriaLogs as a log storage backend. VictoriaLogs is built on architecture ideas from ClickHouse. This makes it very efficient and fast, similar to ClickHouse - it requires small amounts of disk space for storing big volumes of logs. The main difference is that VictoriaLogs is easier to configure, run and manage than ClickHouse:

• it is schemaless - just write logs with arbitrary fields into it according to the data model. It automatically builds all the needed columns and indexes
• it is zero-config, e.g. it automatically adjusts its' default config values according to the available hardware resources (CPU and RAM) in order to provide the maximum efficiency and performance
• it consists of a small statically linked executable (~20MB) without any dependencies

See these docs with more details about the difference between VictoriaLogs and ClickHouse.

Problem/Use Case

VictoriaLogs should be easier to operate on a large scale than ClickHouse.

Target Users

• DevOps Engineers
• Developers
• Security/SIEM Users
• System Administrators
• All Users

Additional Context

VictoriaLogs provides a query language - LogsQL - which is easier to use for typical queries over logs comparing to SQL from ClickHouse. It also provides http-based querying API, which shouldn't be hard to use if needed.

[Polliog]

Thanks for the suggestion @valyala 🙏

LogTide already has a pluggable storage layer (reservoir) with TimescaleDB, ClickHouse and MongoDB backends selected via STORAGE_ENGINE, so adding VictoriaLogs fits the existing extension path.

One thing to sort out first: our storage interface covers logs plus traces/spans and metrics, while VictoriaLogs is logs-only. So a VictoriaLogs backend would cover the logs side well (LogsQL + the HTTP query API map nicely to our query/aggregation methods), but we would need to add capability flags so an engine can declare partial support instead of implementing trace/metric methods.

Marking this as accepted / needs-design. PRs or design input welcome, especially on the LogsQL translation side.