Migrating Graphite from SQLite to PostgreSQL

2013-12-14 19:29:20 by jdixon

As mentioned in my previous article, I no longer recommend using SQLite as a Graphite backend for anything outside of development or testing work. It is too lenient with data types, and doesn't provide the levels of concurrency I'd like to see in an RDBMS for a production web service.

This opinion was cultivated almost exclusively from my recent experience migrating a single-node Graphite instance with an SQLite database to an HA pair of Graphite nodes with a shared PostgreSQL backend. For those of you considering migrating off SQLite to PostgreSQL, this article documents my initial struggles and eventual fixes for this transition.

Read the rest of this story...

Why You Shouldn't use SQLite with Graphite

2013-12-10 19:44:07 by jdixon

If you've ever had the pleasure of installing Graphite, you're almost certainly aware that it uses Django as it's web framework. In order to support features like saving graphs and dashboards, Graphite needs somewhere to store the data that describes these objects. As you might expect, a relational database with support for SQL is a dandy place for this sort of relational data. Django supports a number of RDBMS backends using the Django ORM, making it relatively painless to get started with Graphite in a development or test environment using the popular SQLite database engine.

Read the rest of this story...

Call for OSS Project Instrumentation

2013-05-28 09:56:28 by jdixon

As an open source developer, some portion of my time is spent not just coding and responding to user feedback, but to act as a Project-slash-Product Manager. I have to determine which bugs to prioritize, which features are necessary, and where to allocate my finite resources. Much of this is driven by what interests me at the time and which features will best fit into my overarching vision for the project.

Read the rest of this story...

My Thoughts on FOSS Crowdfunding

2013-04-07 23:04:24 by jdixon

I was recently cold-emailed about a new service, Catincan, that offers a Kickstarter-like bounty system for open source software projects or features. Someone representing the company reached out to me (and presumably, many others in the open source community) for feedback, generally about the funding of open source development, and specifically about their own service.

Read the rest of this story...

Contribute to Open Source Monitoring Projects

2013-04-03 16:40:34 by jdixon

You say you want to contribute to an Open Source project, but you're not sure where to start? Have an interest in monitoring, trending or logging software? Hop on over to the Monitorama Hackathon issues list and take a look around. In the weeks leading up to the event we seeded the repo with a bunch of tasks/feature requests/bug reports that are easily digestible over the course of a day or two. These are a great place to get started on a new project, or nail out some quick issues that have an immediate impact.

Read the rest of this story...

Monitorama Hackathon

2013-03-05 10:38:23 by jdixon

One of the overarching themes that drove me to organize Monitorama was the desire to bring together OSS developers in an effort to improve the current state of monitoring and trending software. I grew impatient with the lack of measurable progress that happens at the typical SysAdmin/WebOps/DevOps-style events, which tend to focus on automation and traditional operations fare. While I'm pleased that everyone is excited about our speaker lineup, the works we accomplish at this Hackathon will be the true barometer of our success. With this in mind I have some points to consider as you prepare for your attendance and participation at the event.

Read the rest of this story...

A Simple HTTP POST Server in Node.js

2012-10-16 21:48:16 by jdixon

In the process of hacking on a plugin for the Uptime project I realized that I needed a simple HTTP server capable of receiving and dumping JSON data via POST. I'm well aware of the awesome Python SimpleHTTPServer module, but alas it doesn't support POST requests.

Fortunately I was able to throw together a quick little server using the sample HTTP server on the Node.js project website along with their API docs. I know this is a ridiculously simple daemon, but someone else might find it as useful as I did.

Read the rest of this story...

My Personal Roadmap

2012-08-26 18:21:47 by jdixon

I've been a little busy lately and haven't found the time to post any new articles, Graphite-related or otherwise. For those who missed the announcement, I started working at GitHub in July. Initially I continued my work on Descartes; more recently my time has been split up among a few different projects, both inside and outside of work. Although I generally detest announcing plans before shipping them, I thought others might like to read about what I'm working on these days.

Read the rest of this story...

Trending your GitHub Commits in Graphite

2012-07-22 22:45:55 by jdixon

Today I was browsing the list of service hooks that GitHub provides. I almost forgot that there's a simple WebHook service that POSTs commit information during the git post-receive hook to any external URL. This got me thinking that it would be nice to trend commit activity inside my Graphite server. Don't get me wrong... GitHub already provides some really nice visualization for project and committer activity on their site. However, as a data junkie, I'd love to be able to correlate this activity with my own application metrics.

This was a perfect fit for Backstop, the HTTP/JSON-to-Graphite bridge. After a couple hours of futzing around I had a working version. If you haven't used Backstop before, rest assured that getting started is pretty darn easy. In fact, if you're a Heroku customer, it's easy and free. There are just a few commands to get your own Backstop server running on Heroku.

Read the rest of this story...

Cron As A Service

2012-02-11 10:35:19 by jdixon

I recently found myself in the need for a way to run a one-off Ruby script at scheduled intervals. As this is a work project I didn't just want to run it on my laptop or some random server. Turns out there's an easy way to run this for free on the Heroku Cedar platform without having to piggyback it on a "real" application. Because there are no web processes running, we'll be able to limit our dyno usage to a single dyno (in other words, it's free).

The script itself handles garbage collection duties for removing expired hosts off our beta account with Boundary. Basically I just want it to run every hour and cull anything that hasn't reported to their collectors in a day. For the purpose of this article the contents of the script are inconsequential, although I intend to present it fully in a future post.

Read the rest of this story...

Trying to Get Shit Done

2012-02-09 23:38:00 by jdixon

This evening I asked for your suggestions on blocking online distractions, allowing me to focus on code for an extended period of time. I have a constant struggle with interruptions (read: shiny things) including online news, email and Twitter. There was a flood of responses in no time. Here are the more popular suggestions, along with my winners below.

  • quit the offending apps
  • block websites
  • login with a different user
  • work offline
  • coffee
  • music
  • self-control (lol rite)

I suspect that all of these would have some benefit, perhaps except coffee, which has never given me much of a boost anyways. As a side note, I've quit Diet Coke since my surgery and am exclusively drinking water. So it's possible that coffee might give me an insanely productive hit, but I'm not willing to tread that path yet. Here are the specific steps I took which seemed to work quite well for my particular workflow.

  1. quit Twitter app
  2. quit Chrome (primary browser)
  3. quit Firefox (used for HTML email, Facebook and banking)
  4. quit Propane (Campfire app, work communications)
  5. quit Adium
  6. quit Skype
  7. detached my remote screen session (mutt and irssi)
  8. equipped Sony MDR-V6 headphones
  9. launched Spotify radio (trance)
  10. put iTerm2 in full-screen mode (used for psql, git and debugging development server)
  11. put MacVim in full-screen mode
  12. launched Safari for API docs and development site

I'm pleased with the recent experiment, even if it only lasted one hour. I'll continue to make adjustments and report any significant improvements I find.