Devops, The future is friendly
Created by Gavin Mogan / @halkeye
Dev Ops
What is devops?
Short Answer: No Idea
Long Answer: Nooooooo iiiiideeeeaaaaa
Not really clearly defined. I think of devops as movement towards people being informed and incontrol.
Agile System Administration / Agile Operations
Toss over the fence mentality
What does devops mean to me?
-
Repeatable Infrastructure
- Any box, at any time, should be easily rebuildable
-
Versioned configs
- Configs shouldn’t be manually edited on a box
-
Unit tests
- Confidence in code being deployed
-
Automated Deploys
- Ex: Capistrano
Why?
-
Consistency
- Confidence in infrastructure
- Confidence that code will work between environments
-
Auditable
- When did this change?
- Why did it change?
- Repetable
The Old Days
- "Put this Code Live, here’s a tarball"
- "Run these queries"
- "What rollback?"
- Security?
- High Availability?
- Scalability?
- "Worked on my box"
- Measure Everything
- No manual changes
- Test everything
- Version everything
- Automate everything
Code Wise
More day to day documentation in code repo
- What server(s) is it on?
- What database does it talk to?
Automating Deployments
-
Lower learning curve
- Each app launches the same way, no matter the language or requirements
- Server lists are committed in configs, so always up to date
-
Consistency
- One persons launch will be identical to another persons launch
How is this accomplished?
Continuous Integration Builds
- Build packaged up code the same way every time.
- Replayable (can build older versions)
Always Deploy?
- Just because you are confident you can deploy, shouldn’t mean you should do so constantly.
- No longer scary to deploy
- Small changes are less scary than monolitic changes
Story Time
- Xbox Live
- Capistrano with alex
Question
If you needed to find out how valdes was doing how would you?
- No Ticket, No problem?
- Ping it?
- SSH in, use uptime/top?
- Check nginx access/error logs?
Questions?
What we are missing
Staging and Prod are not super different, but are different.
- Go isn’t load balanced in staging
- Hard to rebalance apps (bookmarks are evil)
Ideal Infrastructure
-
Monitored (up/down)
- Sensu? Nagios
-
Monitored (Statistics)
- Statsd? Graphite?
- # of logins? # of uploads? Etc
-
Centralized Logging
- Less of a need to go “what server is this on?”
- Fluentd?
Lofty Ideal Infrastructure
-
Scalable Cloud design
- Servers are all identical/pools of servers
- Each server can run any app.
- Docker? (Make it heroku like)
-
Load balancer we control in front of all apps
- HAProxy?
Puppet
Monitoring - Sensu / Uchiwa
Monitoring - Sensu / Uchiwa
- Sensu scales well, pushes not pulls
- Can collect metrics for graphite/others
Graphite / Graphana
Graphite / Graphana
- Designed to scale really easily
- Any recorded metric can be graphed
- Libraries for every language (very simple protocol)
- statsd outputs graphite easily
- Lots of math support, eg: holt winters forecasting, time shifting, derivative
ElasticSearch + Fluentd/logstash + Kibana
ElasticSearch + Fluentd/logstash + Kibana
-
Centralized Logging
- No longer care what server is running what app
- Searchable
- Filterable
- Queryable
- New to me - last job was just looking into it as I was leaving