Unhelpful Graphite Tip #6 - Filtering by Most Deviant

2012-04-13 09:57:32 by jdixon

I remember one day when I was trying to narrow down an application causing high load on an outlier within a fleet of servers. Nagios wasn't suitable for the task, as it only told me which hosts were currently spiking, not which ones have been spiking for a certain window of time. And it certainly couldn't identify a particular host based on a performance visualization.

My Graphite wizard hat went on and I went to work, narrowing down the list of suspects using wildcards and visually inspecting each host's load profile. Within 5 minutes I found my suspect and basked in my glory.

Naturally my brilliance was short-lived.

Indeed, Graphite already provides the superb mostDeviant filter. This function accepts an integer N, averages the entire series, and filters out the top N deviant metrics from the overall average.

Take for example, a one-hour trend from this week:

Apply the filter:

target=mostDeviant(5, runtime.*.load.1m)

Cool beans.


at 2014-06-04 14:28:13, Devin Ben-Hur wrote in to say...

The Graohite docs describes mostDeviant differently then you do: "Draws the N most deviant metrics. To find the deviants, the standard deviation (sigma) of each series is taken and ranked. The top N standard deviations are returned." <http://graphite.readthedocs.org/en/latest/functions.html>.

The series picked is the one(s) that exhibit the highest variance among their series values, not variance compared to the average of all the series. This is still an interesting metric showing hosts exhibiting high changes in their metric, but it doesn't select hosts that are consistently different than the others matching the wildcard.

Add a comment:




max length 4000 chars