become our partner

Building log aggregation solution with ClickHouse

The Program Committee has not yet taken a decision on this talk

Abstracts

ClickHouse is not an out-of-the box solution for logs (like ELK & Grafana Loki) and requires some additional work to make it suitable for storing & querying logs. But, due to it's powerful SQL capabilities, fast data ingestion and good data compression it allows to use it even for logs. In this talk I will describe
* Why did we choose ClickHouse instead of other solutions
* How logs are propagated from apps to ClickHouse
* Logs structure: we use structured-based logging, so what exactly we collect
* How we designed ClickHouse tables for storing logs: column based storage, data retention, storage tiers, etc
* How ClickHouse storage works in context of logs structure
* How we implemented full-text search on a top of ClickHouse, which does not contain it out-of-the box
* How ClickHouse performs against Loki & Elastic in terms of storage size