Stats and Metrics

Better to know, than not to know

• "How long has this been happening?"
• "How many concurrent connections before cpu starts to spin out of control?"
• "How long to videos take to process?"
• "How many videos are uploaded after Joe does one of his announcements?"

Many many components

Components

• Graphite
  • Stats Storage Engine. Numeric values only. Has aggregation, rollups, tons of ways to pull out numbers
• Collectd
  • Regularly (10s) collect system information
• Statsd
  • Collect data from apps. # of logins, uploads, etc
• Grafana
  • SO PRETTY! ENHANCE! ENHANCE!

Collectd

• Collects stats at regular intervals (10s by default)
• Out of the box, system information
• Can write to graphite
• Can collect custom info, but I’ve found it easier/better to go graphite directly.

Statsd

• Daemon that collects stat pushes at it
• Runs over UDP so shouldn’t block your app
• Many stat options:
  • Timings (how long a query took)
  • Absolute values (number of results returned)
  • Incremental Values (every login)
• Many backend options
  • Graphite
  • Others
• Client libraries in pretty much every language

Demos later. Its very pretty

I wanna see some stuff!

Lets go!