Setting up Fluentd log publisher to Loggly is straightforward thanks to the detailed tutorials can be found online. Some useful readings:
- Fluentd logs to loggly: https://www.loggly.com/docs/fluentd-logs/
- Filter and anonymize: https://www.loggly.com/blog/pii-logs-managing-log-data-loggly-fluentd/
- Internal tagging to loggly: https://gist.github.com/sobstel/9bb797e46b1838797a8c
One gotcha: numeric fields in loggly
By default everything you send in your JSON will be a string because you parsed a log line with regular expression. Taking nginx access log for example:
Nginx config:
|
1 2 3 4 |
log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" ' '$request_time $upstream_response_time $gzip_ratio'; |
Fluentd config:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# Nginx access.log <source> type tail path /var/log/nginx/access.log pos_file /var/lib/td-agent/nginx-access.log.pos tag nginx.access format /^(?<remote>[^ ]*) - - \[(?<time>[0-9a-zA-Z\/: \+]+)\] "(?<method>[A-Z]+) (?<path>.+) HTTP/[0-9\.]+" (?<code>\d+) (?<size>\d+) "(?<referer>(http(s)?://)?[^"]+)" +"(?<user_agent>[^"]+)" (?<req_time>[0-9\.]+) (?<upstream_time>[0-9\-\.]+) (?<gzip_ratio>[0-9\.\-]+).*$/ time_format %d/%b/%Y:%H:%M:%S %z </source> <match nginx.*> type forest subtype loggly <template> loggly_url https://logs-01.loggly.com/inputs/TOKEN/tag/${tag} </template> </match> |
This results in logs appearing in Loggly nicely but when wanted to create timeline graphs the following error message appears:
Chart cannot be created because no numeric fields have been found. Please try again when there is more data.
Fix it
To fix it we need to cast our JSON attributes sent by fluentd to loggly. We can do this with fluentd typecast plugin and to restore the original tags we will use record reformer plugin.
Fluentd conf with the new changes:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
<source> type tail path /var/log/nginx/access.log pos_file /var/lib/td-agent/nginx-access.log.pos <strong>tag raw.nginx.access</strong> format /^(?<remote>[^ ]*) - - \[(?<time>[0-9a-zA-Z\/: \+]+)\] "(?<method>[A-Z]+) (?<path>.+) HTTP/[0-9\.]+" (?<code>\d+) (?<size>\d+) "(?<referer>(http(s)?://)?[^"]+)" +"(?<user_agent>[^"]+)" (?<req_time>[0-9\.]+) (?<upstream_time>[0-9\-\.]+) (?<gzip_ratio>[0-9\.\-]+).*$/ time_format %d/%b/%Y:%H:%M:%S %z </source> <match raw.nginx.**> type typecast item_types code:integer,size:integer,req_time:float,upstream_time:float,gzip_ratio:float prefix typed </match> <match typed.raw.**> type record_reformer tag ${tag_suffix[2]} </match> |
What happens here is we changed the tag to the original tail plugin to have a raw prefix which we will match later. In the match raw.nginx.** section we will cast the code and size to integer and the request time, upstream time and gzip ratio to float.
The second part is optional but we probably don’t want to have all our events tagged with typed.raw. prefix therefore we can use record-reformer to remove the first to tag part.
There you go… Now you have nicely formatted log events where every attributes has the proper type which enables more sophisticated analysis on them.
Recent comments