Devops, The future is friendly

Created by Gavin Mogan / @halkeye

Dev Ops

What is devops?

Short Answer: No Idea

Long Answer: Nooooooo iiiiideeeeaaaaa

Not really clearly defined. I think of devops as movement towards people being informed and incontrol.

Agile System Administration / Agile Operations

Toss over the fence mentality

What does devops mean to me?

  • Repeatable Infrastructure
    • Any box, at any time, should be easily rebuildable
  • Versioned configs
    • Configs shouldn’t be manually edited on a box
  • Unit tests
    • Confidence in code being deployed
  • Automated Deploys
    • Ex: Capistrano


  • Consistency
    • Confidence in infrastructure
    • Confidence that code will work between environments
  • Auditable
    • When did this change?
    • Why did it change?
  • Repetable

The Old Days

  • "Put this Code Live, here’s a tarball"
  • "Run these queries"
  • "What rollback?"
  • Security?
  • High Availability?
  • Scalability?
  • "Worked on my box"
  • Measure Everything
  • No manual changes
  • Test everything
  • Version everything
  • Automate everything

Code Wise

More day to day documentation in code repo

  • What server(s) is it on?
  • What database does it talk to?

Automating Deployments

  • Lower learning curve
    • Each app launches the same way, no matter the language or requirements
    • Server lists are committed in configs, so always up to date
  • Consistency
    • One persons launch will be identical to another persons launch

How is this accomplished?

Continuous Integration Builds

  • Build packaged up code the same way every time.
  • Replayable (can build older versions)

Always Deploy?

  • Just because you are confident you can deploy, shouldn’t mean you should do so constantly.
  • No longer scary to deploy
  • Small changes are less scary than monolitic changes

Story Time

  • Xbox Live
  • Capistrano with alex


If you needed to find out how valdes was doing how would you?

  • No Ticket, No problem?
  • Ping it?
  • SSH in, use uptime/top?
  • Check nginx access/error logs?


What we are missing

Staging and Prod are not super different, but are different.

  • Go isn’t load balanced in staging
  • Hard to rebalance apps (bookmarks are evil)

Ideal Infrastructure

  • Monitored (up/down)
    • Sensu? Nagios
  • Monitored (Statistics)
    • Statsd? Graphite?
    • # of logins? # of uploads? Etc
  • Centralized Logging
    • Less of a need to go “what server is this on?”
    • Fluentd?

Lofty Ideal Infrastructure

  • Scalable Cloud design
    • Servers are all identical/pools of servers
    • Each server can run any app.
    • Docker? (Make it heroku like)
  • Load balancer we control in front of all apps
    • HAProxy?


Monitoring - Sensu / Uchiwa

Monitoring - Sensu / Uchiwa

  • Sensu scales well, pushes not pulls
  • Can collect metrics for graphite/others

Graphite / Graphana

Graphite / Graphana

  • Designed to scale really easily
  • Any recorded metric can be graphed
  • Libraries for every language (very simple protocol)
  • statsd outputs graphite easily
  • Lots of math support, eg: holt winters forecasting, time shifting, derivative

ElasticSearch + Fluentd/logstash + Kibana

ElasticSearch + Fluentd/logstash + Kibana

  • Centralized Logging
    • No longer care what server is running what app
    • Searchable
    • Filterable
    • Queryable
  • New to me - last job was just looking into it as I was leaving

Links / More Info