2014-01-02 20:29:59 by jdixon
One of my friends at GitHub, Scott Sanders, recently published a new suite of tools collectively known as Carbonate. Anyone who has had the "pleasure" of migrating Graphite to one or more new servers, in production, has likely felt the pain of dealing with gaps in your time-series data. This is a common source of pain for many administrators; I'm really pleased that Scott was able to put together this collection of shell primitives for managing Whisper migrations.
Carbonate is very young, but I've used it to successfully migrate a large number of production metrics from an old server to a new cluster. There is still some room for improvement in terms of efficiency; carbon-sync does an excellent job of backfilling missing datapoints in a Whisper file, but it does it by rsync-ing the source wsp file to the new server and then performing the heal locally. Ideally, it would only gather the missing datapoints from the remote server, but that would necessitate a much more advanced transmission mechanism.
Regardless, I'm very happy with its current capabilities. I appreciate that Scott looked to solve these common obstacles through a series of small tool primitives rather than one or two large utilitarian applications. It makes it very easy to chain them together in typical UNIX fashion to accomplish a larger, often complex, migration or rebalancing effort. I won't bother to cover all the possible scenarios; Scott has already addressed the most common ones.
However, just so you can get a feel for the general output, here is an example of me syncing a handful of Whisper files from one server to another:
graphite-new:~# ssh graphite-old -- carbon-list | tail -5 | carbon-sync -s graphite-old * Running batch 1-5 - Rsyncing metrics - Syncing 1 of 5 metrics. Avg: 0.100000s Time Left: n/as (20%) - Syncing 2 of 5 metrics. Avg: 0.000923s Time Left: 0:00:00.003692s (40%) - Syncing 3 of 5 metrics. Avg: 0.000284s Time Left: 0:00:00.000852s (60%) - Syncing 4 of 5 metrics. Avg: 0.000186s Time Left: 0:00:00.000371s (80%) - Syncing 5 of 5 metrics. Avg: 0.000137s Time Left: 0:00:00.000137s (100%) -------------------------------------- Rsync time: 44.6337509155s Merge time: 0.00320601463318s Total time: 44.6369569302s * Sync Report ======================================== Total metrics synced: 5 Total time: 44.6378269196s