Rival IQ provides digital marketing analytics with a competitive focus. The service aggregates enormous amounts of data by searching social networks, websites and SEO ranking systems, and then consolidates that information so customers can filter through the noise, understand how they and their competitors are performing, and create informed marketing decisions.
We spoke with Seth Pollack, Founder and Head of Engineering at Rival IQ, about how his team monitors such vast amounts of data.
What kind of environment does Rival IQ have?
Rival IQ runs its application on Heroku, which features a Librato add-on for monitoring and several classes of dynos, but runs additional services in AWS as well. The team uses Node.js, Postgres, a node extension called Streamline, and stores much of its data in Amazon S3.
How do you deal with so much data?
At Rival IQ we’re pulling in digital marketing data from all kinds of different providers and sources. Nobody’s perfect, so these sources frequently have some sort of issue. We need to be able to visualize things like changes in performance of our own software or other data.
We need to look at our database and dynos and other core services from Heroku as well as custom metrics around things like the data APIs, response times, and error rates. For example, it’s not often the case that error rates go from 0 to 100; it’s more common that they go from .01% to 1% or something similar - which is enough to cause real problems. We’ve been able to see those changes with Librato, which has been really helpful to us.
How do you use Librato?
We use Librato primarily as a trending service, looking at graphs and metrics over time to identify patterns.
With Librato, we can monitor the health and performance of external data APIs, looking at substantial information in aggregate. Being able to look at those graphs often tells me the answer right away of what’s going on and how are error rates moving or response times moving.
Alerting is also really important. With Librato, we’re able to alert on the performance of critical applications, receive expected log messages, etc. For example, alerting is a good way to detect hung dynos, which happen sometimes due to oversize memory allocation. In the past, a dyno would start to swap and would more or less freeze up, and we didn’t have great visibility into that. Librato provided an easy way to get that visibility.
We can’t quite quantify how impactful Librato is for us, but qualitatively, Librato is one of the key tools we use as part of our production operations. Between that and logging tools and a few other things, we make sure things are running, monitor them, and debug what’s going on. Librato is mission-critical to our operations.
Haven’t tried Librato? Sign up now for a free, full-featured 30 day trial.