Black Friday (or, How I Learned to Stop Worrying and Love the Weekend) — Librato Blog

Black Friday (or, How I Learned to Stop Worrying and Love the Weekend)


Collin Van Dyck


black-friday-developers

Black Friday is the magical day after Thanksgiving during which various retailers hope to become profitable after running losses leading up to the Thanksgiving holiday weekend. This is actually an alternate explanation for the name; according to Wikipedia the original name originated in Philadelphia where the residents used it in a slightly more pejorative context to describe the extra traffic and congestion that occurred the day after Thanksgiving.

At Librato we treat Fridays with a similar level of fear and loathing. Ok, maybe that's too strong. Librato has a high level of respect for its employees and encourages a healthy work life balance. One of the ways in which this manifests is a culture of avoiding deploying software at the end of the week. Why is that?

As anyone who has worked with highly distributed systems knows, sometimes one can tug on a seemingly innocent stray thread and then watch in horror as the developer onesie is unraveled, leaving only weeping and gnashing of teeth in its absence. This of course is rare but when it does happen, out of respect for our customers, we need a full court press on the remediation effort. If a late Friday deploy goes pear-shaped, it's likely we are going to ruin weekend plans for a number of team members.

Perhaps many of you are already thinking, Friday is the worst time for deploys; why would anyone do this? The truth is, some companies literally encourage and form processes around deploying major changes at the end of the week. The thinking is that a late Friday deploy would have the least impact on customers since they are all off work, enjoying time with family and friends. While this reasoning is well-intentioned, it also has a negative impact on the team members who have to stay late to get the weekly deploys online and stable. Deploy processes which are brittle or overly complex also tend to encourage such a process to be built around them.

So if we are not deploying releases on Fridays, what are we doing? A sufficient staging environment is key. Much of the time our Friday changesets go into staging where they are vetted the next Monday after having run for a couple of days; our own Librato production data is also piped into our staging environment to give us extra confidence. In addition to the normal ebb and flow of development, Fridays are also a time to write blog posts, update documentation, or work on that one sweet side project that you've been meaning to do but for which you have not been able to find the time.

Of course, we occasionally will deploy to production on a Friday or any other day where we need to fix a customer-impacting bug or an infrastructure-threatening inefficiency. We like to make this the exception rather than the rule. When we do, we’ll often feature-flag the changes so that we can reduce the risk by incrementally introducing and, if necessary, reverting the changed code via our chat bot. Not shipping on Fridays is one of the important ways we show respect to customers and team members.