Librato's Hack Day — Librato Blog

Librato's Hack Day


You've heard before about our distributed workforce at Librato. We get together about once a quarter for an intensive week of face time, strategic planning, and team building. Recently we began setting one day of our onsite apart as a “hack day”; basically an opportunity for us to form ad hoc teams (or fly solo), and scratch any itch we want. Many of our internal hacks go on to become new features in our product, or stand-alone open source projects themselves.

Hack events are very much in line with what we enjoy about the culture of Librato: collaboration, camaraderie, and encouraging free-form creativity. There are a slew of different ways to run hackathon-style events, ranging from simple eight hour hack days, to multi day hackathons, to full-on hack weeks. We’ve only run hack days so far, and we thought you might like to hear about how our last one went.

The Plan

The week before the event, we sent an announcement to everyone. This prompted team members to begin thinking about what they might want to work on, and possibly begin talking to each other about teaming up to solve a more complicated problem. Some tips we included in the announcement:

  • Keep your project manageable: start with something small that you can grow as time permits.

  • Experiment with things just outside your comfort range. Even it you don’t get something working in the end, you’ll learn a lot and have fun, or start a useful project that can be finished later (but you probably will have something functional as a result)

  • Take time to see what other people are working on. Energy is contagious.

  • Not everything in a hack day has to be code (e.g., if marketing wants to do an analysis on prospect conversion to customers from different engagement modalities, that is fair game)

The Outcome

Going out bowling the night before made for some good pre-hackathon team building, and while there were side effects, everyone was hard at work by 9 am. There were 19 projects worked on by our 16 engineers, and we got to hear about them during project presentations. After the judges scrutinized and counted each vote, and after a verification by our independent team of accountants (ok, maybe not), we were happy to announce the winners.  

In the Most Valuable Project category, the winner was a prototype of being able to link Spaces together relationally by one of our Directors of Engineering, Matt. In the Best-Technical-Feat-in-8-hours category, the winner was “Lita (our Chatops bot) can return snapshots of a chart on demand” by our Director of Support, Jason (Derrett - we have two Jasons). His project description:

“During the course of daily Librato support or operations, there may be some graphs we tend to check often for a given timeframe or set of parameters, such as rate limiting, API load, or billing data. Usually this involves logging into the UI, setting filters, and checking the results visually, and/or taking a snapshot of the graphs individually. My hack day idea was to automate taking snapshots of "sets" of graphs that we might care about, via our Lita ( chatbot and the internal Librato snapshot API.  Now we can ask Lita questions like ‘show me the latest graphs for any kind of rate limiting’ or ‘show me the usage trend for User XYZ' and the chatbot will call the snapshot API to generate images of the graphs in question (stored in S3), and quickly speak the images back into our Slack room.  Hooray!”

The Best Part: Honorable Mentions

Nobody said it out loud, but we know that every single hack heavily influenced the engineering team's thinking about how we make things and how we work together. The winning projects were awesome, but the votes came in very closely for a good reason. Here are a couple of excellent runners-up. 

Luke, who only started with us in February, on “dashboard of tunables in ZK” project his team worked on: “We use Zookeeper for coordinating and releasing new features, and manage Zookeeper feature flags through chat ops. We don't have much of an insight into the Zookeeper flags within our chat console, so we built an user interface to visually see information about feature flags. I build the frontend using React.js with an immutable data structure. Screenshot below.”


Ray and Collin’s project: “Some metric streams have a high cardinality of sources, but sometimes as a developer or operator you are just interested in a subset of sources, for example the top five. Our hack day idea was to modify some of our existing stream processing to sort and report top-k values for a metric.”


Hackathon events, while generally unquantifiable, but have some great benefits. They

  • build team and company camaraderie

  • encourage everyone to express their creativity

  • enable teams to execute projects that do not necessarily address top business priorities but are nevertheless important to the team

  • give people the opportunity to cross train in a no-risk environment

  • give everyone the time and freedom to vet an idea

  • highlight systemic problems, blockers, or frustration that might have otherwise gone unnoticed

On our most recent hack day, we pretty much checked all the boxes above, and hence very much look forward to the next one.

If you’d like to join a world-class engineering team that will always be working on really interesting problems, we’d love to talk with you.