Graphite Tip - Working with Carbonate

2014-01-02 20:29:59 by jdixon

One of my friends at GitHub, Scott Sanders, recently published a new suite of tools collectively known as Carbonate. Anyone who has had the "pleasure" of migrating Graphite to one or more new servers, in production, has likely felt the pain of dealing with gaps in your time-series data. This is a common source of pain for many administrators; I'm really pleased that Scott was able to put together this collection of shell primitives for managing Whisper migrations.

Carbonate is very young, but I've used it to successfully migrate a large number of production metrics from an old server to a new cluster. There is still some room for improvement in terms of efficiency; carbon-sync does an excellent job of backfilling missing datapoints in a Whisper file, but it does it by rsync-ing the source wsp file to the new server and then performing the heal locally. Ideally, it would only gather the missing datapoints from the remote server, but that would necessitate a much more advanced transmission mechanism.

Regardless, I'm very happy with its current capabilities. I appreciate that Scott looked to solve these common obstacles through a series of small tool primitives rather than one or two large utilitarian applications. It makes it very easy to chain them together in typical UNIX fashion to accomplish a larger, often complex, migration or rebalancing effort. I won't bother to cover all the possible scenarios; Scott has already addressed the most common ones.

However, just so you can get a feel for the general output, here is an example of me syncing a handful of Whisper files from one server to another:

graphite-new:~# ssh graphite-old -- carbon-list | tail -5 | carbon-sync -s graphite-old

* Running batch 1-5
  - Rsyncing metrics
  - Syncing 1 of 5 metrics. Avg: 0.100000s  Time Left: n/as (20%)
  - Syncing 2 of 5 metrics. Avg: 0.000923s  Time Left: 0:00:00.003692s (40%)
  - Syncing 3 of 5 metrics. Avg: 0.000284s  Time Left: 0:00:00.000852s (60%)
  - Syncing 4 of 5 metrics. Avg: 0.000186s  Time Left: 0:00:00.000371s (80%)
  - Syncing 5 of 5 metrics. Avg: 0.000137s  Time Left: 0:00:00.000137s (100%)
    --------------------------------------
    Rsync time: 44.6337509155s
    Merge time: 0.00320601463318s
    Total time: 44.6369569302s

* Sync Report
  ========================================
  Total metrics synced: 5
  Total time: 44.6378269196s

Comments

at 2014-01-08 14:04:00, Victor E. Bazterra wrote in to say...

Dear Jason,

Nice entry, I was not aware of Carbonate. I did something similar (based on you description) to carbon-sync (I called it whisper-sync :-)) using a combination of unison and whisper-merge.py. Unison allows you to provide a external merging program, due to the fact that unison is a bidirectional file synchronizer and therefore it needs to deal with conflicts between different versions of the same file.

This allows to have two graphite servers receiving the same metrics (sent using carbon-relay) and keeping them in sync. In case of one of the servers goes down, you can then recover the information by merging the whisper filers from the other server.

I just wanted to share my experience. I enjoy reading your post.

Victor

at 2014-06-20 18:34:07, Jaime Gago wrote in to say...

wrote an "as dirty as it gets" carbonate install script, thought it might save some time to folks on the interwebs.

https://gist.github.com/jaimegago/78b8c092b6c2418c3378

at 2014-06-20 19:07:59, Jaime Gago wrote in to say...

Nevermind previous comment, just realized carbonate is in pypi...

https://pypi.python.org/pypi/carbonate/0.2.1

at 2014-06-24 19:05:49, nnutter wrote in to say...

It looks like `whisper-merge.py` and `carbonate-fill` are basically the same thing.

Add a comment:

  name

  email

  url

max length 4000 chars