WTF is a container?
I remember going to AWS Reinvent last year and having some beers with a bunch of people who did devops. We started talking about tools, and they were utterly flabbergasted, that we had not embraced docker. They went on an on about how simple docker made HA, and handling fail overs/maintenance. More or less made it seem like it was the greatest thing since sliced bread.Me and a few coworkers decided to try and dockerize some of our serivces, and move our staging ES cluster to docker.For the most part building our own containers was easy enough, for the various services. The biggest issue we had was with Elasticsearch, since we have 4 types of nodes. So we ended up building 4 specialized containers for each node type.Then came the issues:* Making containers work together, for example we use a logging agent, we decided to make that its own container. Then actually getting a way to share a log directory with the agent, was very painful and unclear. (Honestly the single most frustrating thing i recall was asking for advice in irc and more or less being told i am doing it wrong)* Containers would randomly crash under load exiting with error 137 out of memory. Apparently a few of our services would randomly leak memory, but only when running inside a docker container vs ubuntu 14.04. (I never figured this out)* Containers would randomly become unreachable, forcing me to kill them and restart them.* Random hanging issues, suddenly docker exec is hanging. Being told to revert to an older version, or install a newer version is tiring and makes my environment very inconsistent.* Trying to debug why our app in the container isn't working is not fun at all.However the single part that killed me was, i was chatting with one of the people who i met at Reinvent, and i mentioned my issues. He acted like it was completely normal for these kinds of issues.After a solid 2 weeks of random issues and the constant barrage of pagerduty alerts, i just rolled everything back to static EC2 instances for staging, and have ran into 0 issues. I want to try containers again because i want them to work, but i have just had too many issues.