JSON HAProxy Logs & jq

JSON Based HAProxy Logs

HAProxy logs can contain a lot of useful information in them that can be helfpul in tracing issues, and especially when used in conjunction with an ELK stack.

Without getting into details (at least in this post!) about the HAProxy configuration itself, here is a log-format line used for a fairly complext setup that first logs locally, then is forwarded on to an ElasticSearch node.

{"type":"haproxy","env":"dev","pid":%pid,"timestamp":%Ts.%ms,"actconn":%ac,"feconn":%fc,"beconn":%bc,"backend_queue":%bq,"srv_conn":%sc,"retry":%rc,"tq":%Tq,"tw":%Tw,"tc":%Tc,"tr":%Tr,"tt":%Tt,"tsc":"%tsc","client_addr":"%ci:%cp","front_addr":"%fi:%fp","front_transport":"%ft","sslv":"%sslv","sslc":"%sslc","http_status":%ST,"http_req":"%r","back_name":"%b","back_server":"%s","req_header_cap":"%hr","resp_header_cap":"%hs","bytes_uploaded":%U,"bytes_read":%B,"unique_id":"%ID"}

A couple additional bits need added your configuration to fill out all of the params:

capture request header        User-Agent len 34  
unique-id-format            %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid  
unique-id-header            X-Unique-ID  

The User-Agent capture length can be whatever fits your needs. For the details on the unique ID, see the HAProxy docs.

But what if we want to tail a log locally? Let’s make it usable with jq:

tail -F /var/log/haproxy.log | awk '{printf "%s\0", substr($0, index($0, "{"))}' | xargs -0 -I'{}' echo '{}' | jq .  

This will yield pretty printed JSON as it comes. For example:

{
  "type": "haproxy",
  "env": "dev",
  "pid": 16707,
  "timestamp": 1438121303.079,
  "actconn": 2,
  "feconn": 2,
  "beconn": 0,
  "backend_queue": 0,
  "srv_conn": 0,
  "retry": 0,
  "tq": 1,
  "tw": 0,
  "tc": 1,
  "tr": 821,
  "tt": 823,
  "tsc": "--VN",
  "client_addr": "10.3.1.201:56852",
  "front_addr": "10.3.3.76:443",
  "front_transport": "pro_admin_console_https~",
  "sslv": "TLSv1.2",
  "sslc": "ECDHE-RSA-AES128-GCM-SHA256",
  "http_status": 200,
  "http_req": "POST /CENSORED/ HTTP/1.1",
  "back_name": "pro_admin_console",
  "back_server": "admin1",
  "req_header_cap": "{Mozilla/5.0 (X11; Linux x86_64}",
  "resp_header_cap": "",
  "bytes_uploaded": 941,
  "bytes_read": 2830,
  "unique_id": "0A0301C9:DE14_0A03034C:01BB_55B7FD57_43B8:4143"
}

Again, see the HAProxy documentation as there is much more one can do with this!