Jump to content

Topic on Talk:Requests for comment/Structured logging

We are considering a similar approach for Parsoid

3
GWicke (talkcontribs)

The logging library we are considering is https://github.com/trentm/node-bunyan, which also logs JSON messages. Example messages:

{"name":"myapp","hostname":"banana.local","pid":40161,"level":30,"msg":"hi","time":"2013-01-04T18:46:23.851Z","v":0}
{"name":"myapp","hostname":"banana.local","pid":40161,"level":40,"lang":"fr","msg":"au revoir","time":"2013-01-04T18:46:23.853Z","v":0}
GWicke (talkcontribs)

We can also easily log to GELF. Together with the available transport infrastructure that looks like a very good option to me.

BDavis (WMF) (talkcontribs)

It looks like the current proposed data points mostly map into GELF.

Proposal GELF 1.1 Notes
timestamp timestamp
host host
source _source (facility is deprecated in 1.1)
pid _pid
severity level (via string->number transform)
channel _channel
message full_message
file file
line line
errorcode short_message
exception ? GELF doesn't have collection support
args ? GELF doesn't have collection support
vhost _vhost
ip _ip
user _user
req _req

One thing I don't like about GELF is that it doesn't support fields that are collections. This leads to clumsy representations for things like argument lists or structured stack traces. I do really like the support for chunking and compression in the GELF transport protocol.

One of my use cases for structured output is feeding a Logstash server with a Kibana frontend. Logstash will happily index and search anything that can be represented in JSON via it's ElasticSearch backend. With this combination you can do some pretty neat things like find all errors in the last N days with stack traces that mention file X or have argument foo = 'bar'. This can be done with stack traces as big blobs of text but if you get the JSON document structure right it becomes much easier and less prone to false positives/negatives. It may well turn out that defining the "one log event output format to rule them all" is too hard to spend much time thinking about.

Reply to "We are considering a similar approach for Parsoid"