Takeaway: Something
always goes wrong in a “go live” scenario. Prepare your development
team, your work environment, and yourself to handle these high-pressure
situations.
The difference between a simple deployment and
an all-out “go live” is the project’s sheer scope and scale. A
deployment is run of the mill, while a go live is the kind of major
change that takes months or years to put together and represents either
the initial deployment for production use or a major upgrade. These are
some of the lessons I learned over the years to get through a go live in
one piece.
Create a game plan
You need to have a well-defined plan in advance. Here is a checklist of things you need to know before you can go live:
- What are the exact steps to perform any needed data migrations or server changes?
- How do we perform the deployment? (Make a checklist and give every
team member a printed copy of it. Cross things off as you do them,
noting who did them and what date and time they were done.)
- How do you back up the application and data pre-deployment?
- If something goes catastrophically wrong, what is the procedure to restore the original version or restart the deployment?
- How do you reach the IT or support/help departments of all involved parties?
- When do you reach your go/no-go point (i.e., the point when you
either commit to finishing the deployment or start to roll back the
changes), and what are your criteria?
- What are your post-deployment triage priorities? (This one is
especially important, because there will likely be a lot of “squeaky
wheels” begging for grease, and you need to know which ones to
prioritize first.)
- How do you take things offline and bring them back online if needed?
- How do you contact your customers to let them know things are okay or not okay?
You’ll notice a lot of these items are not about “how do we do this?”
but “what do we do if something goes wrong?” There’s a reason for that,
which is that things can and will go wrong, and the quality of a go
live depends just as much on what went right as how you handled the
things that went wrong.
Prepare physically and emotionally
Before heading into the last stretch before a go live, everyone needs
a lot of sleep and rest because when things go wrong, you’ll work long
hours to sort them out. A team of folks already at their breaking point
will not work well together, will have a hard time executing the basics,
and will fall down flat when challenges arise.
For employees with spouses, kids, etc., you need to get them on board
too. There is no way I could have accomplished what I did on any of my
go live adventures if my wife was calling me constantly begging me to
come home or laying on a thick guilt trip about not seeing the kids.
Prepare your home office
During a go-live, there is a good chance you will have to work for
long hours at a stretch or at odd hours of the night. Many times, the
work cannot wait for you to get into the office.
Unlike me, most developers work only a few hours a week from home, if
at all, so they are not well-equipped to do it. Make sure your home
office setup is ready to do real work, which means a proper keyboard,
mouse, and monitors, as well as a squared away Internet connection. I
also make sure that I have enough music and a good speaker or headphone
setup so I can drown out the household noises.
Be flexible with your team
Your team will likely work long, hard hours during the go live. It
doesn’t help anyone if you are a hard case about things like getting to
the office at 9:00 AM, shaving, dressing up, etc. unless those things
are absolutely necessary (such as on customer meeting days). For the go
live I just went through, I often worked until 4:00 AM, 5:00 AM, and
even 6:00 AM a couple of times. If I had forced myself to go into the
office at 9:00 AM, I would have been too tired to be effective for the
entire day. Instead, I let myself get the extra sleep, and then only
went into the office if it made sense.
Also, if folks are stuck at the office late, buy them dinner.
Hope for the best, prepare for the worst
NASA builds triple redundancy into projects because of the fact that
things will go wrong, no matter how much you plan. Unless you are
working for a big enterprise, you probably can’t do that. You have a
Development environment, a Testing/QA/Staging environment or two, and a
Production environment. The best thing we did for our latest go live was
import the data from Production into our Testing environment about a
week in advance, which let us catch a lot of issues early and test our
data migration with the most realistic data set we could.
Remember you’re all on the same side
As the pressure mounts during the go live, things often get tense. It
is easy for folks to start snapping at and working against each other
instead of with each other. Finger-pointing will not solve the issues.
You absolutely must not allow the pressure, lack of sleep, or other
issues interfere with how you work with your teammates. When you are
through it all, you’ll be glad you kept your cool.
