How developers can survive a 'go live' scenario

Takeaway: Something always goes wrong in a “go live” scenario. Prepare your development team, your work environment, and yourself to handle these high-pressure situations.

The difference between a simple deployment and an all-out “go live” is the project’s sheer scope and scale. A deployment is run of the mill, while a go live is the kind of major change that takes months or years to put together and represents either the initial deployment for production use or a major upgrade. These are some of the lessons I learned over the years to get through a go live in one piece.

Create a game plan

You need to have a well-defined plan in advance. Here is a checklist of things you need to know before you can go live:

• What are the exact steps to perform any needed data migrations or server changes?
• How do we perform the deployment? (Make a checklist and give every team member a printed copy of it. Cross things off as you do them, noting who did them and what date and time they were done.)
• How do you back up the application and data pre-deployment?
• If something goes catastrophically wrong, what is the procedure to restore the original version or restart the deployment?
• How do you reach the IT or support/help departments of all involved parties?
• When do you reach your go/no-go point (i.e., the point when you either commit to finishing the deployment or start to roll back the changes), and what are your criteria?
• What are your post-deployment triage priorities? (This one is especially important, because there will likely be a lot of “squeaky wheels” begging for grease, and you need to know which ones to prioritize first.)
• How do you take things offline and bring them back online if needed?
• How do you contact your customers to let them know things are okay or not okay?

You’ll notice a lot of these items are not about “how do we do this?” but “what do we do if something goes wrong?” There’s a reason for that, which is that things can and will go wrong, and the quality of a go live depends just as much on what went right as how you handled the things that went wrong.

Prepare physically and emotionally

Before heading into the last stretch before a go live, everyone needs a lot of sleep and rest because when things go wrong, you’ll work long hours to sort them out. A team of folks already at their breaking point will not work well together, will have a hard time executing the basics, and will fall down flat when challenges arise.
For employees with spouses, kids, etc., you need to get them on board too. There is no way I could have accomplished what I did on any of my go live adventures if my wife was calling me constantly begging me to come home or laying on a thick guilt trip about not seeing the kids.

Prepare your home office

During a go-live, there is a good chance you will have to work for long hours at a stretch or at odd hours of the night. Many times, the work cannot wait for you to get into the office.
Unlike me, most developers work only a few hours a week from home, if at all, so they are not well-equipped to do it. Make sure your home office setup is ready to do real work, which means a proper keyboard, mouse, and monitors, as well as a squared away Internet connection. I also make sure that I have enough music and a good speaker or headphone setup so I can drown out the household noises.

Be flexible with your team

Your team will likely work long, hard hours during the go live. It doesn’t help anyone if you are a hard case about things like getting to the office at 9:00 AM, shaving, dressing up, etc. unless those things are absolutely necessary (such as on customer meeting days). For the go live I just went through, I often worked until 4:00 AM, 5:00 AM, and even 6:00 AM a couple of times. If I had forced myself to go into the office at 9:00 AM, I would have been too tired to be effective for the entire day. Instead, I let myself get the extra sleep, and then only went into the office if it made sense.
Also, if folks are stuck at the office late, buy them dinner.

Hope for the best, prepare for the worst

NASA builds triple redundancy into projects because of the fact that things will go wrong, no matter how much you plan. Unless you are working for a big enterprise, you probably can’t do that. You have a Development environment, a Testing/QA/Staging environment or two, and a Production environment. The best thing we did for our latest go live was import the data from Production into our Testing environment about a week in advance, which let us catch a lot of issues early and test our data migration with the most realistic data set we could.

Remember you’re all on the same side

As the pressure mounts during the go live, things often get tense. It is easy for folks to start snapping at and working against each other instead of with each other. Finger-pointing will not solve the issues.
You absolutely must not allow the pressure, lack of sleep, or other issues interfere with how you work with your teammates. When you are through it all, you’ll be glad you kept your cool.