Graphite Tip - A Better Way to Store Events

2014-01-05 20:54:03 by jdixon

Graphite is well known for storing simple key/value metrics using the Whisper time-series database on-disk format. What is not well known about Graphite is that it also ships with a feature known as Events that supports a richer form of metrics storage suitable for, well, events. Imagine a place where you could store tagged metrics and additional data relevant to the event (e.g. code snippets, comments, etc). Many folks use NoSQL databases such as HBase for this purpose, and that's a perfectly reasonable approach. However, if you'd like to store these somewhere where they can be correlated with the rest of your Graphite metrics, then Events might be a good fit for you.

Examples of events that are appropriate for this metric type include releases, commits, application exceptions, and generally anything that represents a state change where you might wish to track the affected data.

Database Storage

Because Whisper was designed to hold simple time-series data (metric key, value, and timestamp), it's altogether unsuitable for storing rich metric data. Users began to experiment with storing event-type data (e.g. releases, state changes, etc) in Whisper and visualizing them with Graphite's drawAsInfinite() function. However, taking advantage of this pattern typically requires the use of wildcards across a significant number of these singleton metric files and directories, which can cause a significant performance hit on the server and result in a poor experience for users.

Therefore, Graphite's application database was extended to support this new metric type. As long as you're running Graphite version 0.9.9 or newer, there are no changes needed to begin storing Events in the database. However, care must be taken to ensure that your Graphite webserver and database can scale with the level of event metrics that you intend to throw at it. I personally recommend PostgreSQL, but MySQL users should be fine to store events on their database as well.

Working with Events

Events can be submitted via HTTP POST or by inserting them manually using the Graphite administration module.

Adding Events by HTTP POST
$ curl -X POST "http://graphite/events/" 
    -d '{"what": "Event - deploy", "tags": "deploy", 
    "data": "deploy of master branch happened at Fri Jan  3 22:34:41 UTC 2014"}'
Managing Events by Admin UI

You can add, edit and delete Events using the Graphite administration module (found at /admin/). This is particularly handy for deleting a large number of events at once.

Querying Events

Graphite allows you to query for tags associated with events. You can search for a single tag string, a combination of tags, or a simple * wildcard using the events() function.

$ curl -s http://graphite/render/?target=events("exception")&format=json | json_pp
[
   {
      "target" : "events(exception)",
      "datapoints" : [
         [
            1,
            1388966651
         ],
         [
            3,
            1388966652
         ],
...
...
...
         [
            2,
            1388966656
         ],
         [
            1,
            1388966657
         ]
      ]
   }
]
Browsing Events

You can also browse events using Graphite's events index (found at /events/). Beware that this page lacks any sort of pagination, so a large number of events could cause significant load time in your browser.

From there you can click on any of the individual events to view its details.

Comments

at 2014-01-06 09:44:37, Jeff wrote in to say...

Hadoop is a NoSQL database that people use for storing monitoring data?

at 2014-01-06 14:39:50, Jason Dixon wrote in to say...

@Jeff - Ugh, brain fart. I meant HBase. Fixed, thanks.

at 2014-02-10 11:56:36, Nir Dothan wrote in to say...

It is also possible to get the full events text details in json by using:

curl 'http://localhost:8080/events/get_data?tags=deploy&from=-3hours&until=now'

returns the following:

[{"data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014", "what": "Event - deploy", "when": 1392046352.0, "id": 2, "tags": "deploy"}, {"data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014", "what": "Event - deploy", "when": 1392046661.0, "id": 3, "tags": "deploy"}]

at 2014-02-19 09:01:48, Joshua wrote in to say...

Isn't graphite a dead project? It seems to not have been touch for over 2 years.

at 2014-02-21 04:55:32, Owen wrote in to say...

This helped me a lot! Thanks

at 2014-02-22 14:39:29, Peter Baumgartner wrote in to say...

Joshua, Graphite is not dead. Last commit was 10 days ago: https://github.com/graphite-project

at 2014-04-17 10:00:42, Scott wrote in to say...

Shameless plug. I created a ruby gem for automatically pumping events into graphite on capistrano deploys and rollbacks. Mine works with capistrano v3 and is located here: https://github.com/scottsuch/capistrano-graphite

I based this off the work from a gem that works with earlier versions which is here:

https://github.com/hellvinz/graphite-notify

at 2014-06-18 06:43:03, Morten Vitved wrote in to say...

Is there any way to sum these events in a graph?

I'd like a graph showing me the trend of a certain tag.

at 2014-06-20 21:28:00, Jason Dixon wrote in to say...

@Morten - That's exactly what the events() function gives you. http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.events

at 2014-12-03 14:38:25, AlexL wrote in to say...

The inputs for the POST call also allow you to specify "where" with a specific timestamp as well, so if you wanted to add historical events, or ones that you know had a certain delay, the following would allow for it:

{

"what": "Event - deploy",

"when": 1388788481,

"tags": "deploy",

"data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014"

}

at 2014-12-29 06:50:36, Hariharan wrote in to say...

Do Graphite supports Mongo Db?

Add a comment:

  name

  email

  url

max length 4000 chars