Pipeline Architecture
LogDoc processes logs through a pipeline architecture consisting of Sources, Filters, and Sinks. Understanding this flow is key to building effective log processing configurations.
Data Flow Overview
Sources
Filters
Sinks
Sources
Sources are input adapters that collect log data from various origins. LogDoc supports multiple source types out of the box:
Syslog
RFC 5424 and RFC 3164 syslog messages over UDP/TCP
HTTP
RESTful API endpoint for JSON-formatted log ingestion
File
Tail and watch local log files with automatic rotation handling
Pipeline Configuration Example
Here's a complete pipeline configuration that collects syslog, filters by severity, and outputs to both file and Elasticsearch:
yaml
pipeline:
sources:
- type: syslog
port: 5140
protocol: udp
filters:
- type: severity
min_level: warning
sinks:
- type: file
path: /var/log/logdoc/output.log