Organizing Your Graphite Metrics

2012-05-09 22:13:10 by jdixon

One of the most common questions I get from Graphite users is how best to name and/or organize metric paths. I don't have an exhaustive list of "best practices" but I'd like to share some basic insights I've accumulated.

Misaligned paths are ok. I used to be tempted to try and keep different paths aligned in order to ease correlation of related targets within a graph. Fortunately there are plenty of helpful aliasing functions (and wildcards) to help tame unruly paths.

Control your base prefixes. We've managed to normalize our metrics by submitting (virtually all) host and application metrics through an HTTP endpoint. This service acts as a proxy between JSON metric emitters and our Carbon relays, but more importantly, it allows us to tightly control the format and paths chosen by developers.

    post '/metrics/:name' do
      if %w( custom pulse note ).include?(params[:name])
        ...
      else
        halt 404, "unknown metric prefix"
      end
    end

Cordon off your metrics according to their retention policies. This makes resizing/adjusting easier later on, since you won't be forced to cherry-pick whisper directories and files with a heinous find script.

Don't freak out. Shit can always be moved around and resized later on.

Comments

at 2012-08-21 06:53:33, Nathan wrote in to say...

Do you think you could expand a little more on what you mean by "Cordon off your metrics according to their retention policies"? Do you have separate namespaces setup for granular retention metrics (per second) versus course retention metrics (per 10 minutes)? Are those namespaces close to the root namespace? An example would be very much appreciated!

at 2012-08-26 15:01:50, Jason Dixon wrote in to say...

@Nathan - Yes, in some cases I would suggest grouping your metrics according to their retention policies, particularly if you think you might need to change that particular policy at a later date. It just makes it a little easier to resize the whisper files. But this is not a requirement, you can always cherry-pick specific files to reconfigure later on.

Add a comment:

  name

  email

  url

max length 4000 chars