We don't mind telling you that we're pretty excited about the work we've been doing on our alerting infrastructure. Suffice to say some pretty radical change is on the way, so expect to see more alerting-related posts over the next few months. Today we'd like to share a related change we've rolled out to our early-access customers, because it might affect the way you respond to alerts.
Currently, the first source that triggers an alert on a given metric wins. For example, if you have a `cassandra.latency` metric configured on 5 nodes to alert at a given threshold, and all 5 sources begin to alert at the same time, you will only receive a notification from the first source to traverse our system.
However, after this change we will preserve information for N sources per alert. Using the previous example with 5 Cassandra nodes, the notification you would receive will contain the corresponding measurements from every source that has tripped the alert threshold. This is what it looks like in Email:
And in Campfire:
So far, we've feature-flagged this for beta testers, but we intend to enable it for all users as of January 6, 2014.
If you want access to this feature sooner, just drop by our customer chat and let us know.