State Is a Bug

The story so far

For those listening in on this public conversation, here's the background.

I participated in a podcast episode about Continuous Deployment at the Java Posse Roundup 2012.

Marc Esher tweeted about the podcast, asking me to elaborate further on my assertion that "state is a bug".

That led to a longer question on a GitHub gist:

Joe, thanks for responding.

I'm most interested in what are perhaps pedestrian issues, but they're issues (for me) nonetheless. You mentioned "outsourcing" session management... where can I read more on that?

Take the simple case of: I'm a user on your system. I'm logged in. I'm doing things.  The server I'm on disappears while the screen I'm reading is currently loading.

What happens? Do I see an error? Am I sent to a new machine, with all my session state in tact, and the screen I was reading simply reloads?

Or take perhaps a different architecture where a machine isn't brought down until all its users have been successfully moved to other servers. For example, perhaps our configuration is such that we have N instances, and we do a rolling code push to those instances. User A is on Server 5, which has the old code. We need to get that user to Server 2, which has the new code, along with all of his state. Once all users are off of Server 5, the deploy then moves to Server 5 as well and then Server 5 is brought back into commission and accepts new requests.  

So: what techniques, processes, and tools support these practices?

Thanks again.

My response got a bit long for a gist, so here it is as a blog post. Marc and I haven't met. This is just how the internet works now.

My response

Hi Marc,

Here's how I see it.

Instantaneous failure

As for the question of instantaneous failure, the server shouldn't be expected to disappear while it's handling a user's request and delivering a quick response, unless the instance truly has catastrophic power failure in that moment. In that case there's no way I know of to recover. However, I'm not really concerned with that case because I think it's sufficiently rare, not currently practical to protect against, and not significant yet, because clients still occasionally expect to do three retries before succeeding. During an electrical storm or "Take your monkey to work"-day, servers can end up getting unplugged, and clients will be affected if their requests occurred in the right instant, but I don't think that problem has been solved well yet. The solvable problem is, when the user retries and gets a healthy server's response, will the user have an appropriate experience, or will they lose all their work and need to redo all their recent actions?

So, for the duration of one request and response in a stateless protocol like HTTP, the client and server are dependent on each other, and on the internet hops between them. Keep that request-response duration short enough, and the risk of instantaneous power loss should be mitigated.

If an instance gets a software shut-down signal, it ought to be configured to shut down its web server process gracefully before permitting the operating system to shut down. Web server graceful shut down should mean refusing to accept new requests, while finishing the delivery of responses the server has already started. Therefore, "terminate instance" should usually entail automatic draining of connections before total shut down occurs.

Session state

The state that I consider a bug in any highly available system is session state, not request state. Anything you want to store in session could instead be stored in a remote shared state service such as a database, a queue, or a workflow service. Consider a case for a mature shopping site like Amazon.com or eBay.com. If you log in to your account with two laptops, and start viewing items on both laptops, the history of what you've recently viewed is generally visible to both sessions around the same time, even though you are collecting history in two different sessions, possibly on two different servers. This means that the important shopping transactions of "view item" are stored in a database shared by the two web servers your clients are talking to. If one of those servers shuts down and both of your laptops continue sending shopping requests to view more items, then some of the client traffic should get switched by a load balancer to a healthy server for the future requests. The state of the user's shopping sessions is undamaged because it is still in the shared database, visible to all server instances.

Once you move "session"-type state out of the server's session context and into a remote shared service, it becomes practical to set up auto-scaling policies to get more stateless virtual servers when you need them, and to terminate stateless virtual servers when your traffic levels drop, in order to save money when renting virtual servers from a cloud provider.

I think about this stuff a lot because I work on Asgard, the open source app deployment and cloud management app produced and used by Netflix. I talk to a lot of Netflix engineers about the need to get state out of their applications. I'm also working to get state out of Asgard itself.

Rehearsing for failure

Many Netflix services use AWS auto scaling for availability and cost savings, so they need to keep user state out of their service so servers can be used interchangeably by clients. Netflix also uses Chaos Monkey to terminate instances within an Auto Scaling Group daily during business hours just to make sure the developers are still maintaining a system that is resilient to small-scale instance failures. Amazon and all other data centers have server failures, so our best defense is to expect failures and to plan for them, and to practice automatic recovery all the time. Netflix doesn't use auto scaling for Cassandra database rings because Cassandra doesn't have a good way to handle frequent growth in the size of a cluster. However, we do use Chaos Monkey to exercise Cassandra's ability to recover completely from an occasional single instance termination.

We also have all our services in three AWS Availability Zones (data centers) so if one zone has a major problem, Netflix is generally unaffected while Reddit and Pinterest are sometimes down for hours.

Asgard's state problem

On a more personal level, I'm working to remove state from Asgard so that I can increase Asgard's server count and release new versions of Asgard for use by Netflix engineers more frequently and conveniently. My intent is to use Amazon Simple Workflow Service to store the state of each long-running automation process that gets started by an Asgard user. For example, a complex rolling push of new code to replacement instances in an auto scaling group, or a reversible push of new code to a new auto scaling group, with automated result checking and rollback on failure. The state of the long-running workflow execution will then be visible to all Asgard instances, while none of those instances needs to stay up just to finish the automation process.

Related Netflix tech blog posts

• Post-mortem of October 22,2012 AWS degradation
• Netflix Shares Cloud Load Balancing And Failover Tool: Eureka!
• Chaos Monkey Released Into The Wild
• Lessons Netflix Learned From The AWS Storm
• Scalable Logging and Tracking
• Asgard: Web-based Cloud Management and Deployment
• Netflix Operations: Part I, Going Distributed
• Fault Tolerance in a High Volume, Distributed System
• Auto Scaling in the Amazon Cloud
• Making the Netflix API More Resilient
• Benchmarking Cassandra Scalability on AWS - Over a million writes per second
• The Netflix Simian Army
• Lessons Netflix Learned from the AWS Outage
• NoSQL @ Netflix Talk (Part 1)
• NoSQL at Netflix
• Four Reasons We Choose Amazon’s Cloud as Our Computing Platform
• 5 Lessons We’ve Learned Using AWS

Immediate Ideas from JPR09

My 28 seconds of fame has arrived. If you're not already a Java Posse podcast listener, let episode 233 be your introduction to this excellent resource, and your chance to hear my radio voice as the Java Posse Roundup 2009 attendees all take a chance with the mic.

I'm trying to recall the best ideas I learned at the roundup last week, for prompt use at my workplace, Marketing Technology Solutions. Some of the ideas about how to help get back developer time were interesting, especially the focus on automating more processes. I look forward to reviewing more of the audio sessions in future weeks and months to extract more ideas.

However, there are three ideas that made my eyes light up when I heard them, and they're relatively simple to get started at work almost immediately after selling them to the team.

1. Project Retrospectives

After each code release, allow a full regular day to fix any surprising production issues. Then on the next day, run through a 1 to 3 hour code review and functionality retrospective. During this meeting each team member who completed anything interesting can show the primary changes they made to the code base and the improved functionality they created. This can help other team members learn how areas of the code are evolving that they haven't seen recently, and it can help us learn from each other's challenges and the solutions we found.

2. Regular Discussions with Employees

Spend 30-60 minutes each week with each employee to find out privately if there are any ideas or problems that deserve some attention from the employee's perspective. This should preferably be done far away from the office building to encourage unchecked speech about office troubles. It's reasonable to use a full or partial lunch break for the meeting if the employee wants, or just go for a walk during work hours.

3. Lightning Talks

Once a month our team can have an hour-long lunch-and-learn session of 5 minute lightning talks on any technical or non-technical subjects. Keeping things educational and entertaining helps promote free thinking and creativity, so non-technical talks are just as valuable as technical ones, as long as they're interesting or funny. This is especially helpful for team members who are not yet comfortable speaking in front of a group, which is an increasingly important skill for a developer. To get things started I'll need to prepare a few short talks on technical and non-technical subjects, with and without slides. This can help others see the flexibility of the format so they feel free to present whatever and however they prefer.

The lightning talks at the Java Posse Roundup 2009 were very successful. They went as follows:

TUESDAY
"Fair Allocation" Algorithm
Selenium Intro
How I Became 3/4 of the Man I Was
The Art of the Photo
High Gear Media and GWT
Loop Quantum Gravity
Animation Inside JavaFX
Racing 101
Awesome Productivity with GMail
ScalaCheck Automated Tests
Simple Twitter Client in Scala
Your Eyes Suck at Blue

WEDNESDAY
Why Are There 12? or The Other Staircase
DB Migration in Java
Dynamic Web Skinning
Centerline Soccer
YQL
Java User Groups: How to Start One?
Sophisticated Data Access with JPA and Spring
F1 KERS System (2009)
Scala + Wicket
Surprise?
Slide Rules for Fun and Profit
Call 811
Repository Management

THURSDAY
Fan
jFlubber, FlexFlubber, FXFlubber
The Smallest Plugin System
Hacking Hardware
Helmet Cam Footage
Zombies: Are You Prepared?
JavaScript Shell
Recovering a Stolen Laptop with Flex
Doctor Who
Sexier Software with Flex
Solar Power for Your House
Scala and JavaRebel
Groovy SwingBuilder, Google Maps, YQL Mashup

The JPR09 lightning talks are gradually getting released by Carl Quinn on http://www.youtube.com/javaposse

Powers of Two

When I was 2 my parents divorced, but I didn't know that at the time. I LOVED apple juice.

When I was 4 I made an alphabet book with pop-up pictures of things that start with the 26 English letters.

When I was 8 I learned to type by playing Infocom games. I had already decided that I would grow up to be a fantasy novelist or an Infocom game author. I was incorrect but I was starting down the right path.

When I was 16 I designed theater sets and managed a crew to build them. I wrote and game-mastered year-long role playing games with my friends, which would later turn into an epic poem for high school credit. I didn't know it but all of this was gradually turning me into a software developer and manager.

Today I turn 32 and I'm on a plane returning from the Java Posse Roundup in Crested Butte, Colorado. My driver's license vanished so the airport security folks find me interesting, and maybe even attractive. One guard was even so attentive that he wanted to pat down my arms and legs. He asked if I was sore or tender anywhere.

I told him about falling on my ass repeatedly while cross-country skiing with computer geniuses. On the trail, Dianne Marsh patiently taught me the basics as we trudged through 3 miles of winding hills. The creator of Artifactory, Fred Simon, showed me how to cut the snow by bending my ankles correctly to go slowly down a hill.

My brain is brimming with ideas from the conference. My housemate Todd Costella showed me the value of Crucible and Fisheye for recording code reviews. My other housemates Andrew Harmel-Law and D. J. Halberg helped me understand Scrum master training. The formal and informal discussions were eye-opening, and the 5-minute lightning talks were such a good idea I'm going to try to get them started at work.

Dick Wall will gradually publish the audio of the discussions and the audio-video of the lightning talks. Thanks for everything, Dick! I'll be listening to those files many times, including the ones I was present for. Until the files are available I think I should blog about what I remember. This way I'll have a way to review my experiences and convert them into actions.

Tor Norbye shares my appreciation for fruity cocktails instead of beer. I introduced him to the joy of the Bay Breeze (Malibu rum, vodka, cranberry, pineapple). I'll have to find out his favorite drink so I can try it out as well. He's already shown me how valuable Netbeans is for Groovy, JavaScript, JavaFX, and Matisse.

On this same plane with me is Bill Robertson who redesigned the look and feel of Dick Wall's JFlubber application for the new and improved FlexFlubber and JFXFlubber incarnations. I sliced and diced Bill's design in Photoshop to make an exported JavaFX app. James Ward grabbed the PSD to make FlexFlubber. A few of the drop shadows didn't get into the buttons in Flex but it still looks wicked cool.

Our wrap-up session from last night is already uploaded as episode #233 of the Java Posse Podcast at http://www.javaposse.com and the best of the lightning talks are starting to show up on http://www.youtube.com/javaposse

When I'm 64 I could be handy mending a fuse when the lights have gone.