SF Metrics Meetup: Approaches to Anomaly Detection at Twitter and Netflix — Librato Blog

SF Metrics Meetup: Approaches to Anomaly Detection at Twitter and Netflix

Nik Wekwerth

February's Meetup focused on Anomaly Detection featuring fantastic talks by Arun Kejariwal from Twitter and Cody Rioux from Netflix. 

Statistical Learning-based Automatic Anomaly Detection @Twitter

Arun Kejariwal, Twitter

Twitter developed novel statistical techniques for automatically detecting anomalies in cloud infrastructure data. Specifically, the techniques employ statistical learning to detect anomalies in both application and system metrics. 

  • They employ time series decomposition to filter the trend and seasonal components of the time series.

  • They use robust statistical metrics – median and median absolute deviation (MAD) – to accurately detect anomalies, even in the presence of seasonal spikes.

The techniques that Arun presents were evaluated with a wide variety of time series (system and application metrics obtained from production as well as stock data), and have been deployed in production at Twitter. Arun demonstrates the efficacy of the proposed techniques using production data.

Netflix Outlier and Anomaly Detection

Cody Rioux, Netflix

Kepler is an in-house outlier and anomaly detection system that currently runs on an in house solution for running Python analytics in Netflix's cloud environment, specifically with the goal of supporting reliability and availability efforts within Netflix's AWS environment. Kepler runs against Netflix's telemetry data, and produces alerts using the same mechanisms as their classical alerting system.

In this talk, Cody discusses the motivations behind the project, current uses within the environment, some technical details of the algorithms used, and then views some examples of the situations they are able to detect and respond to in an automated fashion.

SF Metrics Meetups are organized by Librato. Join the group so that you always know when new Meetups are scheduled. Do you want to speak at one of the events? Let us know!