obfuscurity.
hidden in plain sight

I've been a little busy lately and haven't found the time to post any new articles, Graphite-related or otherwise. For those who missed the announcement, I started working at GitHub in July. Initially I continued my work on Descartes; more recently my time has been split up among a few different projects, both inside and outside of work. Although I generally detest announcing plans before shipping them, I thought others might like to read about what I'm working on these days.

Read the rest of this story...

Today I was browsing the list of service hooks that GitHub provides. I almost forgot that there's a simple WebHook service that POSTs commit information during the git post-receive hook to any external URL. This got me thinking that it would be nice to trend commit activity inside my Graphite server. Don't get me wrong... GitHub already provides some really nice visualization for project and committer activity on their site. However, as a data junkie, I'd love to be able to correlate this activity with my own application metrics.

This was a perfect fit for Backstop, the HTTP/JSON-to-Graphite bridge. After a couple hours of futzing around I had a working version. If you haven't used Backstop before, rest assured that getting started is pretty darn easy. In fact, if you're a Heroku customer, it's easy and free. There are just a few commands to get your own Backstop server running on Heroku.

Read the rest of this story...

Graphite is renowned for its usefulness and ease for prototyping new charts. It's also known for having a dashboard component that leaves much to be desired. In response the community has seen a rising tide of new dashboard projects aimed at filling this gap. The growing list of third-party Graphite dashboard projects is extensive, but continues to fall short in areas such as self-service, configuration, and collaboration.

Most of this software require users to generate dashboards from JSON or other command-line gymnastics. While this is reasonable for many operations folk, it's an impedance for the engineers and business-oriented users; the same users that we want using this software for making sound decisions. Graph views are static and inflexible for collaboration and historical dialogues. In response to these shortcomings I've started the Descartes project.

Read the rest of this story...

A friend of mine recently asked for some good D3 tutorials and sites. At second glance these are an awesome collection of examples for using D3 and general visualization work.

Pro: You don't have to scour the web for these yourself.

Con: It's unlikely you'll ever fully consume all the awesome.

• Scott Murray's D3 Tutorials
• Jan Willem Tulp's Data Visualization blog
• Jim Vallandingham's blog
• Creating D3 Transitions, from the Visual.ly blog
• Blog of Mike Bostock, creator of D3

Seems that it's common for folks to blog about changes in employment. I hate to be left out on the fun, so I'll take a brief moment to officially announce my pending "new-hire" status with GitHub, effective tomorrow.

Friends who've already heard the news pepper their congratulations with a sense of confusion as to why I'd leave a good thing at Heroku. Indeed, I think most people in our industry would rank Heroku and GitHub at the top of their list of prospective employers. Unsurprisingly, I loved my job. I've never worked with a team of engineers as highly skilled or dedicated to their mission as the men and women at Heroku. So why would I leave?

Read the rest of this story...

Here's an embarrassingly simple script I threw together this morning to track network latency to a handful of remote websites/networks from my home internet. Yes, I understand that these numbers are highly influenced by my proximity to various CDN networks and bear no resemblance to how actual web browsing would perform concurrently. That isn't the point. This is merely to demonstrate a cheap and easy way to get more metrics into Graphite; and at the same time, providing me with some useful reference for when my home internet provider will inevitably have hiccups.

Read the rest of this story...

Part of what I see myself doing (by writing blog posts, creating software like Tasseo, etc) is to try and help others learn better ways of communicating our operational knowledge through visualization tools and methodologies. While I've gotten a lot of positive feedback from my Graphite articles, what I haven't seen as much is a two-way sharing of the harvested data made possible through these experiences.

I think there are a couple possible reasons for this: first, we work with "propietary" data that our employers might not want divulged; second, we assume our data is immaterial and not worth sharing. For the former, I think this is a very similar argument that many of us had with employers during the push to open source software. There is much to be gained by sharing our raw data (perhaps without all of the proprietary metadata and labels that make it relevant to our business) and seeing those examples improved upon and returned by our peers.

Read the rest of this story...

This week I traveled out west for my first Velocity conference as an attendee. I went out two years ago but I was so busy juggling exhibitor duties that I didn't get to enjoy any hallway networking or formal session. This year I went in with plans to catch as many sessions as possible, particularly those skewed towards monitoring, trending and operations workflow. As expected, I skipped quite a few talks but made up for it with a lot of quality time catching up with peers and reviewing new technologies (and philosophies) in the DevOps space.

Read the rest of this story...

There are a ton of open-source and commercial monitoring tools available, so why do we claim that monitoring sucks? Certainly there are some usable tools out there; without them our systems would be even more unpredictable and unreliable than they already are. So what makes one tool sticky where others get tried and tossed aside?

Systems Administrators (and Engineers) are a finicky bunch. We prefer to build complex systems from small, sharp instruments rather than fight with larger, malleable (read: monolithic) software. There's a reason why Pingdom and Pager Duty are enormously popular among technically agile businesses. Cost is only a small part of the equation; these customers understand (implicitly, if not explicitly) that combining these small, sharp tools into a series of logically connected functions (fault detection, notifications and historical trending) is much easier than breaking apart an Enterprise-Ready monitoring suite and coercing it to meet their unique needs.

Read the rest of this story...

This is one of my most favorite, and certainly most underappreciated graphs. Its simplicity belies its usefulness. This single chart gives me a holistic view of our metrics feed, writes to Whisper files, as well as general system health. At a glance I can correlate slow updates caused by a spike in Whisper file creations or a backup resulting in a higher PPU value. We use some of its targets with Nagios to monitor for metric feed issues. And it's always the first place I look whenever there's a whiff of Graphite problems.

Read the rest of this story...