Migrating Graphite from SQLite to PostgreSQL

2013-12-14 19:29:20 by jdixon

As mentioned in my previous article, I no longer recommend using SQLite as a Graphite backend for anything outside of development or testing work. It is too lenient with data types, and doesn't provide the levels of concurrency I'd like to see in an RDBMS for a production web service.

This opinion was cultivated almost exclusively from my recent experience migrating a single-node Graphite instance with an SQLite database to an HA pair of Graphite nodes with a shared PostgreSQL backend. For those of you considering migrating off SQLite to PostgreSQL, this article documents my initial struggles and eventual fixes for this transition.

Read the rest of this story...

Why You Shouldn't use SQLite with Graphite

2013-12-10 19:44:07 by jdixon

If you've ever had the pleasure of installing Graphite, you're almost certainly aware that it uses Django as it's web framework. In order to support features like saving graphs and dashboards, Graphite needs somewhere to store the data that describes these objects. As you might expect, a relational database with support for SQL is a dandy place for this sort of relational data. Django supports a number of RDBMS backends using the Django ORM, making it relatively painless to get started with Graphite in a development or test environment using the popular SQLite database engine.

Read the rest of this story...

My Impressions of InfluxDB

2013-11-11 12:15:38 by jdixon

I mentioned last week that I was planning to look closer at InfluxDB this past weekend, and some folks asked me to do a writeup on my findings.

InfluxDB is a time-series metrics and events database based on the LevelDB key-value store. LevelDB was written and open sourced by Google, and is an optional backend for Riak. InfluxDB (or "Influx", for short) inherits many of LevelDB's default characteristics, which means it's optimized for writes and uses compression by default, but it can be slow for reads and deletes.

Read the rest of this story...

What's Up with Playfair?

2013-10-19 02:58:37 by jdixon

Within the last hour I stumbled across a tweet from Dan Ryan mentioning a new hosted Graphite + StatsD service called Playfair. As you might expect, this piqued my interest.

Immediately, I thought of Hosted Graphite and wondered how this compares with their offering. Would it have its own dashboard? Was it a DigitalOcean-backed Graphite instance (admittedly, something I've considered trying to package up myself)? I hopped over to their website and looked around.

Read the rest of this story...

Graphite Tip - Mixing Lines and Stacks

2013-08-16 13:43:15 by jdixon

One of Graphite's shortcomings is that it's not easy to construct a composite chart of both lines and area sections. In fact, it's not possible at all unless you're willing to stack your areas. But if you are dealing with data where it makes sense to stack them, and you want to correlate that with something else as a line series, here's an example demonstrating how you can do it.

Read the rest of this story...

Are We Ready to Kill Thresholds?

2013-06-26 09:12:54 by jdixon

I've been hearing a lot of chatter from various sources that adaptive fault detection is going to be The New Shit ™ and that static thresholds are virtually useless because they lack context. While I agree that some of the more advanced techniques sound amazing (and make no mistake, I'm really excited about the possibilities here), it's foolish to think that thresholds as a measure of fault conditions are useless.

Read the rest of this story...

Dusk - Yet Another Graphite Dashboard

2013-06-21 19:36:28 by jdixon

Not too long ago we were looking for a way to visualize a group of metrics across our entire fleet. Although you could render all of the metrics on one graph, it becomes nearly impossible to distinguish one from another. Jesse Newland (@jnewland) suggested that we look at Cubism.js' horizon charts. The nice thing about horizon charts is that you can cram a lot of information into a small vertical space, due to the way they render "overlapping" values with increasing intensity. One thing led to another, and soon Dusk was born.

Read the rest of this story...

Call for OSS Project Instrumentation

2013-05-28 09:56:28 by jdixon

As an open source developer, some portion of my time is spent not just coding and responding to user feedback, but to act as a Project-slash-Product Manager. I have to determine which bugs to prioritize, which features are necessary, and where to allocate my finite resources. Much of this is driven by what interests me at the time and which features will best fit into my overarching vision for the project.

Read the rest of this story...

Graphite Tip - Counting Number of Metrics Reported

2013-05-27 13:59:50 by jdixon

There's been many a time when I've asked the question "I wonder how many hosts are sending this metric?" Unfortunately there's no built-in Graphite function for determining the number of hosts submitting a particular metric (or tree of metrics). But this morning I stumbled across a brilliant hack of a Graphite query by Jesse Newland (@jnewland) for rendering this value.

Read the rest of this story...

WTF is Chartroulette

2013-05-13 16:32:42 by jdixon

Sometimes the silliest features are the ones that inspire you most. This was certainly the case with the new Chartroulette view that I recently merged into Descartes. Because I wanted so badly for this to become a reality it forced me to knock out some other dependencies (user model, favorite dashboards, and better user mapping) rather quickly.

To be fair, there's nothing silly about the idea behind Chartroulette. At GitHub we have an internal app by @maddox that allows users to rotate any Mac or iOS-based device's screen through a series of website URLs. Typically we use this to cycle through dashboards or graphs. While I'd love to see this open-sourced, I know that Jon is a very busy guy so I figured that emulating this functionality within Descartes might be the next best thing.

Read the rest of this story...

Feeding Params into Descartes

2013-05-06 13:07:32 by jdixon

This is a relatively minor enhancement in terms of LoC but it would take too many words to describe on Twitter so here we are. Recent commits added support for passing interval and columns parameters into Descartes views (graphs, dashboards, etc). Previously you would always get the default layout whenever loading any Descartes page.

Read the rest of this story...

Graphite Tip - Group by Node

2013-04-14 18:23:29 by jdixon

In the process of setting up some graphs for Status Board, I thought it would be nice to render my GitHub activity (in terms of commits). As I demonstrated in a post last year, you can fire off a metric to Graphite using GitHub's post-commit webhook feature. Rendered with drawAsInfinite, this is nice for getting a rough visualization of your commit activity, but doesn't provide total counts. Alternatively, you could use group with summarize to get totals per interval, but you wouldn't be able to view per-repository numbers.

Read the rest of this story...

My Thoughts on FOSS Crowdfunding

2013-04-07 23:04:24 by jdixon

I was recently cold-emailed about a new service, Catincan, that offers a Kickstarter-like bounty system for open source software projects or features. Someone representing the company reached out to me (and presumably, many others in the open source community) for feedback, generally about the funding of open source development, and specifically about their own service.

Read the rest of this story...

Contribute to Open Source Monitoring Projects

2013-04-03 16:40:34 by jdixon

You say you want to contribute to an Open Source project, but you're not sure where to start? Have an interest in monitoring, trending or logging software? Hop on over to the Monitorama Hackathon issues list and take a look around. In the weeks leading up to the event we seeded the repo with a bunch of tasks/feature requests/bug reports that are easily digestible over the course of a day or two. These are a great place to get started on a new project, or nail out some quick issues that have an immediate impact.

Read the rest of this story...

Graphite Tip - Grouping Release Metrics in the Legend

2013-04-03 12:48:07 by jdixon

Last year I gave examples for using drawAsInfinite to help visualize the frequency of particular events (deploys, commits, etc). One of the side effects that I failed to mention is that these will quickly fill up your legend with labels, making it impossible to view the legend at all. It's likely that you've seen this sort of thing at least once (assuming you forced hideLegend=false):

Read the rest of this story...

Graphite Tip - Converting Zeroes into Nulls

2013-04-01 17:43:26 by jdixon

I was looking at some internal data for @jnunemaker and @jfryman today when I stumbled across the Data Filters group of functions inside the Graphite composer. These functions are handy for those times when you want to exclude a subset of a particular series of data, for whatever reason. In our case, we were looking at some metrics where we had spikes of data that were interesting, and a lot of uninteresting data reported as zeroes.

Read the rest of this story...

Thoughts from Monitorama 2013

2013-03-30 10:34:42 by jdixon

This is not your typical conference review. This is a braindump of my thoughts following the organization and execution of the 2013 Monitorama Conference and Hackathon in Boston (Cambridge), Massachusetts.

Read the rest of this story...

Monitorama Hackathon

2013-03-05 10:38:23 by jdixon

One of the overarching themes that drove me to organize Monitorama was the desire to bring together OSS developers in an effort to improve the current state of monitoring and trending software. I grew impatient with the lack of measurable progress that happens at the typical SysAdmin/WebOps/DevOps-style events, which tend to focus on automation and traditional operations fare. While I'm pleased that everyone is excited about our speaker lineup, the works we accomplish at this Hackathon will be the true barometer of our success. With this in mind I have some points to consider as you prepare for your attendance and participation at the event.

Read the rest of this story...