Organizing Your Graphite Metrics

2012-05-09 22:13:10 by jdixon

One of the most common questions I get from Graphite users is how best to name and/or organize metric paths. I don't have an exhaustive list of "best practices" but I'd like to share some basic insights I've accumulated.

Misaligned paths are ok. I used to be tempted to try and keep different paths aligned in order to ease correlation of related targets within a graph. Fortunately there are plenty of helpful aliasing functions (and wildcards) to help tame unruly paths.

Control your base prefixes. We've managed to normalize our metrics by submitting (virtually all) host and application metrics through an HTTP endpoint. This service acts as a proxy between JSON metric emitters and our Carbon relays, but more importantly, it allows us to tightly control the format and paths chosen by developers.

    post '/metrics/:name' do
      if %w( custom pulse note ).include?(params[:name])
        halt 404, "unknown metric prefix"

Cordon off your metrics according to their retention policies. This makes resizing/adjusting easier later on, since you won't be forced to cherry-pick whisper directories and files with a heinous find script.

Don't freak out. Shit can always be moved around and resized later on.


at 2012-08-21 06:53:33, Nathan wrote in to say...

Do you think you could expand a little more on what you mean by "Cordon off your metrics according to their retention policies"? Do you have separate namespaces setup for granular retention metrics (per second) versus course retention metrics (per 10 minutes)? Are those namespaces close to the root namespace? An example would be very much appreciated!

at 2012-08-26 15:01:50, Jason Dixon wrote in to say...

@Nathan - Yes, in some cases I would suggest grouping your metrics according to their retention policies, particularly if you think you might need to change that particular policy at a later date. It just makes it a little easier to resize the whisper files. But this is not a requirement, you can always cherry-pick specific files to reconfigure later on.

at 2017-02-05 09:23:48, Surya wrote in to say...

Though the date of this post looks a bit outdated, it's still relevant for those like me, who have recently started playing with Graphite.

In my organization, we've bootstrapped StatsD+Graphite+Grafana and it's been deployed in our production environment though in a limited scale for now. We'll soon be rolling it out organization-wide.

One question that I have is about how to best organize Graphite Metrics that takes care of versioning. One simple way is obviously to include version information as part of the metrics namespace. For example <service_name>.<version>.<metrics>. However, we would soon end up with infinitely growing number of metrics. This makes it hard to compute the disk budget.

Do you have any suggestion on how to get around this issue?

Add a comment:




max length 4000 chars