At the latest SF Metrics Meetup I spoke about why we built SuperChief, an in-house distributed stream processing system to replace Apache Storm.
Distributed stream processing is all about gaining actionable insight from high volumes of data in near real-time. Stream processing is complex so many people look to frameworks for help. Last October, Librato went in the other direction, moving from Apache Storm to an in-house system named SuperChief.
In this talk I looked at some of the challenges Librato had faced with Storm and the motivation to move away from frameworks. I covered the design and implementation of SuperChief, the benefits of this new approach, and the direction SuperChief is going in the future.
For those of you who couldn't make it, here is the recording: