Four servers, eleven log files, and an incident reconstructed from three of them. Shipping first and parsing second, structured events, and the correlation id that makes it useful.
A log line written as a sentence has to be parsed back into fields by a regular expression somebody maintains, and that expression breaks whenever the sentence changes.…
A shipper that follows a file by descriptor keeps reading the rotated-away file and never sees the new one; a shipper that reopens by name misses whatever was…
Daily indices grow linearly and nothing removes them, so a logging cluster works beautifully for four months and then stops accepting writes with a disk watermark error at…
Grok is regular expressions with named patterns, and it earns its reputation honestly: a pattern that backtracks on an unexpected line will use an entire CPU and stall…
The instinct is to set up log aggregation when there is a problem to investigate. By then the interesting logs have rotated away, and the incident is reconstructed…