obfuscurity.
http://obfuscurity.com/
hidden in plain sighten-usCopyright 2011, DixonGroup Consultingjason@dixongroup.net (Jason Dixon)jason@dixongroup.net (Jason Dixon)obfuscurity.1901-01-01T00:00+00:001hourlyThe Grand Windows Laptop Experiment
http://obfuscurity.com/2016/12/The-Grand-Windows-Laptop-Experiment
<p>If you follow me on Twitter, you might remember my rants (well, among many other people) regarding their new 2016 MacBook Pro line of laptops. There've been an abundance of reviews online, criticizing Apple for their "courage" to remove ports and functional keys that are still a mainstay in most users' workflows, and for actual performance <em>regressions</em> in most real-world scenarios. I think these changes reflect a desire by Apple to cater to their larger <em>mass consumer</em> audience, while at the same time streamlining a Mac product line facing an eroding market due to our increasingly mobile-first world.</p>
<!--readmore-->
<h3>The Indignation</h3>
<p>Whether or not you agree with their product choices, there's a clear backlash against Apple by the professional user community that's supported them for years (I've used Apple laptops almost exclusively going back to 2002). IT and creative pros alike are acutely affected by these changes. I fall into the former camp, using my <a href="https://support.apple.com/kb/SP715?locale=en_US">13-inch Retina MacBook Pro</a> nearly <strong>round-the-clock</strong> for both work tasks and personal entertainment. This hardware revision came across at best as indifference towards my work and productivity, and at worst as an attack on my livelihood. Obviously this isn't the case, but to many of us who rely on powerful, familiar, and <em>competent</em> computers for our daily tasks, it leaves us looking for alternative platforms (most IT folks are generally wary of vendor lock-in and I'm no exception).</p>
<p>When I'm not doing software development with <a>iTerm2</a>, <a>bash</a>, <a>OpenSSH</a>, <a>Sublime Text 2</a>, <a>Vagrant</a> and <a>VirtualBox</a>, I'm keeping in touch with my friends and peers with <a>Tweetbot</a>, <a>iMessage</a>, and <a>Slack</a>, playing <a>Rocket League</a> in <a>Steam</a>, or watching videos in <a>Netflix</a> and <a>YouTube</a>. Most of these applications are supported in both <em>OS X</em> (sorry, I can't bring myself to call it <em>macOS</em> yet) and Windows 10, so I felt empowered to dump the Apple platform and try switching to a Windows laptop.</p>
<h3>The Hardware</h3>
<p>If you're in the market for a new Windows laptop, and you value portability, I have good news for you. Not only are there some fantastic products across the major vendors (HP, Lenovo, Microsoft, Dell), but most of them manage to squeeze a lot of power <em>and</em> battery life into small, well-designed, durable chassis. Some of my favorites include:</p>
<p>
<ul style="list-style-type: disc; padding-left: 30px; font-size: small;">
<li>HP Spectre x360</li>
<li>Lenovo Yoga 900</li>
<li>Dell XPS 13</li>
<li>Microsoft Surface Book</li>
</ul>
</p>
<p>But for me, I knew I also wanted to squeeze some light gaming out of this system. Generally speaking, you can't expect much gaming performance from ultrabooks, but I was reading some really good reviews for the newest iteration of <a href="http://www.razerzone.com/gaming-systems/razer-blade">Razer Blade</a> and <a href="http://www.razerzone.com/store/razer-blade-stealth">Razer Blade Stealth</a> laptops. The former is a portable beast, equipped with a 14" QHD+ touchscreen or 1080p matte screen, and driven by the NVIDIA GeForce GTX 1060 GPU. While it lacks the featherweight portability of an ultrabook, it manages to squeeze a ton of computing power into a unibody frame roughly the size of the 2015 MacBook Pro.</p>
<p>The Razer Blade Stealth is a far more modest machine in terms of spec, but the form factor is downright delicious. I was drawn in by its size and weight, falling somewhere between a 12" MacBook and its Pro sibling. The fit and finish are top-notch, with tolerances rivaling modern Apple laptops. Aside from its integrated graphics support, the Stealth checked off all of my requirements in quick order.</p>
<p>A trip to the Microsoft Store convinced me that the Stealth was a very capable machine. The employees were happy (frankly, the Microsoft employees were delightful; Apple should watch out here) to let me install Steam and Rocket League on both Razer laptops, playing a few games on each to get a feel for their, ahem, "professional capabilities". Despite the lack of a dedicated GPU, the ultrabook still managed a very respectable 60 fps (frames per second) at 1280x800 resolution with settings at high quality. I'd found my new laptop.</p>
<h3>The Christening</h3>
<p>Outside of brief passing glances at Staples and Best Buy, this was my first real interaction with Windows 10. The new tile-oriented flat design is colorful and modern, a big departure from my previous Windows OS experiences. Most elements were intuitive, although I soon found myself missing the navigation gestures and shortcuts from OS X. Friends jumped in and offered suggestions online, and I was quickly hopping around with Windows' Task View, Virtual Desktops, and a few useful (although limited) touchpad gestures.</p>
<p><strong>Note:</strong> If you like Apple's <tt>ctrl+arrow</tt> desktop switcher, you'll find its Windows analog with <tt>ctrl+windows+arrow</tt>. Unfortunately, I rely <em>very heavily</em> on the Mac's three-finger swipe gesture. There simply isn't anything comparable in Windows. 😞</p>
<p>Although I initially struggled with the location of the <tt>ctrl</tt> key in the standard Windows keyboard layout, I was able to swap the left-hand <tt>ctrl</tt> and <tt>alt</tt> keys using the open source <a href="https://sharpkeys.codeplex.com/">SharpKeys</a> utility. This placed the control key next to the spacebar, where my stubby fingers could benefit from muscle memory; Windows' control key and Apple's command key correspond pretty well for common tasks (copy, paste, application switcher, etc).</p>
<p>Over the course of the next few days, with the help of friends like <a href="https://twitter.com/gwaldo">@gwaldo</a> and <a href="https://twitter.com/jordansissel">@jordansissel</a>, I recreated a comfortable development environment and workflow using a combination of familiar tools (ST2, Heroku Toolbelt, Vagrant, Virtualbox) and new alternatives (<a href="https://git-for-windows.github.io/">Git</a> for Windows, <a href="https://chocolatey.org/">Chocolatey</a>). There were some hiccups along the way but nothing that couldn't be worked around:</p>
<p>
<ul style="list-style-type: disc; padding-left: 30px; font-size: small;">
<li><tt>vagrant ssh</tt> didn't want to work in Windows' Linux Subsystem shell or Powershell, worked fine in git-scm's shell</li>
<li>drive mapping <em>felt</em> brittle and incompatible across the different Windows shells</li>
<li>Chocolatey is a nice analog to Homebrew for Mac, but it left me without a working <tt>psql</tt> when I installed the "newest" PostgreSQL package (9.3.x)</li>
</ul>
</p>
<p>With a working git and ssh environment, I was easily distracted by the occasional Rocket League game. At this point I was very content with my choice. Work sessions at home, as well as mid-day trips to the coffee shop for a change of scenery, were enjoyable and productive. If you'd asked me then for my impression of Windows 10 and the Razer hardware, I would've responded with a genuine <em>"better than expected"</em>.
<h3>The Seven Day Itch</h3>
<p>Frustration and regret became palpable over the course of a week. First and foremost, there is simply no support for Apple's iMessage service in Windows. I didn't realize it earlier, but I rely very heavily on the Messages app in OS X for communicating with my family and friends. This is an Apple-only feature, and I can see why now. Having to dig the iPhone out of my pocket every time I got a text from my wife or kids was a major disruption to my focus on work. On the Mac, I can easily glance at the on-screen notification, and choose to ignore it or <tt>command-tab</tt> over to the Messages app for a quick response.</p>
<p>Only <em>slightly</em> less disappointing (although very surprising) was the lack of quality Twitter clients for Windows 10. I'm a big fan of <a href="http://tapbots.com/tweetbot/mac/">Tweetbot's</a> single-column design on the Mac, and their overall feature set across both Mac and iOS apps. I was at a complete loss to find anything similar for Windows. If you like Tweetdeck, you'll probably be satisfied with their <a href="https://tweetdeck.twitter.com/">web client</a>. I gave it an honest try, but I couldn't get comfortable with the multi-column layout (I'd tried it years ago on Mac, same result).</p>
<p>Even at this point, things were relatively sane. I was bothered, but not yet annoyed. My problems were largely centered around communications apps. Given some hard choices and a little more time (and lots of patience), I could probably find suitable workarounds. Then something unexpected happened.</p>
<p style="margin: 20px 0 40px;"><em>I bought a monitor.</em></p>
<h3>The Slope of Deflated Expectations</h3>
<p>Unlike most IT professionals, I've never been very comfortable with an external monitor. I think this is because I usually try to <em>extend</em> to the second display and find it jarring moving from one screen size to another (duh). I'm not sure what the impetus was but I suddenly found myself interested in large-screen external display (<em>cough</em>, <strong>gaming</strong>, <em>cough</em>). Fast forward a day and I was sporting a <a href="http://www.bestbuy.com/site/asus-27-ips-led-hd-monitor-silver/8767178.p?skuId=8767178">27-inch 1080p display</a> in my home office.</p>
<p>This time around I decided to mirror my display, and the experience was transformative. Previously I found myself routinely moving around the house and coffee shops for a change of scenery, but now I find myself invisibily anchored to the desk more often than not. Unfortunately, this also introduced a new class of hurdles that I hadn't anticipated when I started this adventure.</p>
<p><strong>I hate mice.</strong> I can't emphasize this enough, I <em>really</em> dislike the computer mouse, wired or wireless, in virtually all of its different incarnations. I acknowledge that it's a primary input device for most of the computing public, but I've become increasingly dependent on the Mac laptop trackpads and their software-enabled gestures.</p>
<p>With display mirroring enabled and the laptop lid closed for work, I needed some sort of trackpad (or in Windows-ese, <em>touchpad</em>). Apple makes the best trackpads in the industry, but they don't have any official support or drivers for Windows systems (outside of Bootcamp). An earlier attempt at using these drivers to get my old Magic Mouse working under Windows left me unconvinced that investing $129 in a new trackpad, in the off-chance that it would work with unsupported drivers, would be a wise decision.</p>
<p>So I went shopping for alternatives. As it turns out, Apple is also one of the <em>only</em> manufacturers of external track/touchpads. Logitech used to make a <a href="http://support.logitech.com/en_us/product/touchpad-t650">wireless glass touchpad</a>, but it's discontinued and they only offer drivers up to Windows 8.</p>
<div style="float: right; margin-left: 25px;">
<img style="height: 280px; margin-bottom: 25px;" src="https://s3.amazonaws.com/obfuscurity-blog/windows-touchscreen.gif">
</div>
<p>Though the little nagging issues continued to mount, I was still hoping to find long-term workarounds for all of these items (and more, e.g. Windows DPI scaling on large displays). My anger at Apple for "messing up" their Pro line hadn't dissipated much. I still had my 13-inch retina MacBook Pro, but I managed to get all of my work done in Windows using a (yuck) mouse.</p>
<p>And then it happened. Erratic pointer movement on the display. It only happened when I was using the external display and the laptop lid closed. The mouse wasn't moving, and in fact, moving the mouse around had no effect on the behavior. Pressing lightly on the lid cleared it up, leading me to suspect heat issues with the touchpad or touchscreen.</p>
<p>I was able to quickly rule out the touchpad by disabling it using the device manager and waiting for the problem to surface again. However, I really didn't want to have to disable the appropriate <em>Human Interface Device</em> <strong>every fucking time</strong> I used the external display with the lid closed. Sadly, there's no feature in Windows 10 to support this seemingly common use case.</p>
<p style="margin-bottom: 30px;">Enough was enough.</p>
<h3>The Disentanglement</h3>
<p>Don't get me wrong, I would love to have an alternative to Apple. And while Windows 10 and the greater Microsoft ecosystem have made <strong>huge</strong> steps forward in the past decade, they still have a ways to go before they compete with Apple's "just works" experience. I acknowledge that much of this stems from the competitive marketplace over there, but they also benefit from that competition in terms of vendors pushing themselves that much harder for our dollars.</p>
<p>I think Microsoft has made some excellent product decisions over the last few years with the introduction of the Surface Pro, Surface Book, and more recently, the Surface Studio (if and when it ships early next year). I hope that they continue along this path, going beyond basic hardware and software driver support, to consider the <em>entire user experience</em> and the myriad use cases for mobile professionals like myself: gesture navigation, window management, and automated device management could all sorely benefit from more usability testing.</p>
<p>I've had numerous discussions with like-minded engineers who've successfully made the plunge. They're very happy with their decisions and I'm just as happy for them. As more of us make the transition to Windows, I have no doubt that there will be more - and better - software offerings, e.g. Twitter clients. I'm going to keep my eye on the Windows landscape in the coming months; my frustration with Apple (the company and their current laptop lineup) continues, and the current crop of ultrabooks are just <em>too damn good</em> to be ignored.</p>
<p>Regardless, I'm back to my MacBook Pro for now. Check back with me in 2017.</p>
<div><img style="width: 100%;" src="https://s3.amazonaws.com/obfuscurity-blog/windows-macbookpro.jpg"></div>
jdixonWed, 07 Dec 2016 19:28:09 ESTobfuscurity.What's up with Monitorama EU
http://obfuscurity.com/2016/10/what-happened-to-monitorama-eu
<p>Way back in 2013, I put on a small event in Boston, Massachusetts focused on monitoring software and related themes. The unexpected popularity of the show led me to immediately turn around and announce a second event to take place later that year in Berlin, Germany. Both of the conferences were hugely successful and the subject matter clearly resonated with the larger DevOps and Engineering communities.</p>
<!--readmore-->
<p>Fast forward three years and we're yet to return for another EU conference. What happened? To be blunt, <strong>life happened</strong>. In 2014 I was dealing with multiple job changes. In 2015 I was focused on my book. And this year, just as I was putting blinders on for a mad rush to a late-2016 EU event, my mother passed away very unexpectedly (suffice it to say, 2016 has been a general shitshow).</p>
<p>Why am I telling you this? Because I hate making excuses, but I also believe in transparency. I <strong>want</strong> to come back to Europe. I love the cities, the cultures, the food (oh yes, the food), and the diverse experiences. I love that companies in the EU have different perspectives on software and systems than our mostly-inspired-by-Silicon-Valley culture in the states. But mostly, I just love the people. And I owe it to all of you "across the pond" to make this happen.</p>
<p>We're closer than ever (well, since 2013) to bringing Monitorama back to Europe. I'm on the brink of locking in a venue that will be a fantastic fit for this conference. Although I can't leak specifics <em>yet</em>, I wanted to let you all know that <em>we are making progress</em>, and that we're focused on dates in <strong>September 2017</strong>.</p>
<p>As soon as we sign a contract for the venue, we will give a full announcement with dates, location, ticket information, and much more. I can see light at the end of this tunnel, and I'm very excited to share more details with you as soon as possible.</p>
<p>Warm Regards,</br><em>Jason Dixon</em></p>jdixonWed, 26 Oct 2016 23:31:31 EDTobfuscurity.Benchmarking Graphite on NVMe
http://obfuscurity.com/2016/09/Benchmarking-Graphite-on-NVMe
<div>
<p>Here's another quick update to demonstrate what's possible with a single Graphite node running master (these <a href="https://github.com/graphite-project/carbon/commit/d4a6a83536acb00cc31c00fe642aa20e6dbf8f7d">Carbon</a> and <a href="https://github.com/graphite-project/graphite-web/commit/722a3df131216980351de2ada424081eb4833778">Graphite-Web</a> commits, specifically). As you'll see in the results below, this configuration was able to achieve 300k datapoints per second.</p>
<p>This test was performed on a <a href="https://www.packet.net/bare-metal/">Packet type 3</a> server with the pair of NVMe flash drives striped in a single LVM volume. Installation of the Graphite stack was still performed using Synthesize <a href="https://github.com/obfuscurity/synthesize/releases/tag/v2.4.1">v.2.4.1</a>. To take advantage of the increased I/O capacity I added more cache processes for a grand total of eight (8) relays and sixteen (16) caches. Five instances of <a href="https://github.com/gorsuch/haggar">Haggar</a> ran concurrently, on a separate Packet type 1 server in the same Parsippany, NJ datacenter.</p>
</div>
<!--readmore-->
<div>
<p>I'm very pleased with the results of this benchmark. Although per-process CPU is high, Load is still at reasonable levels, and the box remains very responsive. There are no signs of "fatigue" other than those system metrics; the box is humming along and data retrieval times remain low. Queued datapoints in the relays and caches are stable and (at least based on Packet's published claims for these drives) there is still capacity in terms of write operations, although I'd be hesitant to increase the cache batch writes since that would require additional CPU for sorting among the in-memory queues.</p>
</div>
<div>
<pre>
$ sudo pvcreate /dev/nvme0n1
$ sudo pvcreate /dev/nvme1n1
$ sudo vgcreate graphite_vol /dev/nvme0n1 /dev/nvme1n1
$ sudo lvcreate -i 2 -I 4 -l 100%VG -n graphite_vg graphite_vol
$ sudo mkfs.ext4 /dev/graphite_vol/graphite_vg
$ sudo service carbon-cache stop
$ sudo service apache2 stop
$ sudo mkdir /opt2
$ sudo mv /opt/graphite /opt2/
$ sudo mount /dev/graphite_vol/graphite_vg /opt/
$ sudo mv /opt2/graphite /opt/
$ sudo service carbon-cache start
$ sudo service apache2 start
</pre>
</div>
<div>
<pre>
# carbon.conf
[relay]
USER = carbon
LOG_LISTENER_CONNECTIONS = False
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 1
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
DESTINATIONS = 127.0.0.1:20104:1, 127.0.0.1:20204:2, 127.0.0.1:20304:3,
127.0.0.1:20404:4, 127.0.0.1:20504:5, 127.0.0.1:20604:6,
127.0.0.1:20704:7, 127.0.0.1:20804:8, 127.0.0.1:20904:9,
127.0.0.1:21004:10, 127.0.0.1:21104:11, 127.0.0.1:21204:12,
127.0.0.1:21304:13, 127.0.0.1:21404:14, 127.0.0.1:21504:15,
127.0.0.1:21604:16
[relay:1]
LINE_RECEIVER_PORT = 2113
PICKLE_RECEIVER_PORT = 2114
[relay:2]
LINE_RECEIVER_PORT = 2213
PICKLE_RECEIVER_PORT = 2214
[relay:3]
LINE_RECEIVER_PORT = 2313
PICKLE_RECEIVER_PORT = 2314
[relay:4]
LINE_RECEIVER_PORT = 2413
PICKLE_RECEIVER_PORT = 2414
[relay:5]
LINE_RECEIVER_PORT = 2513
PICKLE_RECEIVER_PORT = 2514
[relay:6]
LINE_RECEIVER_PORT = 2613
PICKLE_RECEIVER_PORT = 2614
[relay:7]
LINE_RECEIVER_PORT = 2713
PICKLE_RECEIVER_PORT = 2714
[relay:8]
LINE_RECEIVER_PORT = 2813
PICKLE_RECEIVER_PORT = 2814
[cache]
USER = carbon
CACHE_WRITE_STRATEGY = sorted
MAX_CACHE_SIZE = 4000000
USE_FLOW_CONTROL = True
WHISPER_FALLOCATE_CREATE = True
MAX_CREATES_PER_MINUTE = 12000
MAX_UPDATES_PER_SECOND = 20000
USE_INSECURE_UNPICKLER = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False
LOG_LISTENER_CONNECTIONS = False
LOG_UPDATES = False
ENABLE_LOGROTATION = False
WHISPER_AUTOFLUSH = False
[cache:1]
LINE_RECEIVER_PORT = 20103
PICKLE_RECEIVER_PORT = 20104
CACHE_QUERY_PORT = 7012
[cache:2]
LINE_RECEIVER_PORT = 20203
PICKLE_RECEIVER_PORT = 20204
CACHE_QUERY_PORT = 7022
[cache:3]
LINE_RECEIVER_PORT = 20303
PICKLE_RECEIVER_PORT = 20304
CACHE_QUERY_PORT = 7032
[cache:4]
LINE_RECEIVER_PORT = 20403
PICKLE_RECEIVER_PORT = 20404
CACHE_QUERY_PORT = 7042
[cache:5]
LINE_RECEIVER_PORT = 20503
PICKLE_RECEIVER_PORT = 20504
CACHE_QUERY_PORT = 7052
[cache:6]
LINE_RECEIVER_PORT = 20603
PICKLE_RECEIVER_PORT = 20604
CACHE_QUERY_PORT = 7062
[cache:7]
LINE_RECEIVER_PORT = 20703
PICKLE_RECEIVER_PORT = 20704
CACHE_QUERY_PORT = 7072
[cache:8]
LINE_RECEIVER_PORT = 20803
PICKLE_RECEIVER_PORT = 20804
CACHE_QUERY_PORT = 7082
[cache:9]
LINE_RECEIVER_PORT = 20903
PICKLE_RECEIVER_PORT = 20904
CACHE_QUERY_PORT = 7092
[cache:10]
LINE_RECEIVER_PORT = 21003
PICKLE_RECEIVER_PORT = 21004
CACHE_QUERY_PORT = 7102
[cache:11]
LINE_RECEIVER_PORT = 21103
PICKLE_RECEIVER_PORT = 21104
CACHE_QUERY_PORT = 7112
[cache:12]
LINE_RECEIVER_PORT = 21203
PICKLE_RECEIVER_PORT = 21204
CACHE_QUERY_PORT = 7122
[cache:13]
LINE_RECEIVER_PORT = 21303
PICKLE_RECEIVER_PORT = 21304
CACHE_QUERY_PORT = 7132
[cache:14]
LINE_RECEIVER_PORT = 21403
PICKLE_RECEIVER_PORT = 21404
CACHE_QUERY_PORT = 7142
[cache:15]
LINE_RECEIVER_PORT = 21503
PICKLE_RECEIVER_PORT = 21504
CACHE_QUERY_PORT = 7152
[cache:16]
LINE_RECEIVER_PORT = 21603
PICKLE_RECEIVER_PORT = 21604
CACHE_QUERY_PORT = 7162
</pre>
</div>
<div>
<pre>
# storage-schemas.conf
[collectd]
pattern = ^collectd\.
retentions = 10s:1w, 60s:1y
[haggar]
pattern = ^haggar\.
retentions = 10s:1d, 60s:1w, 1d:1y
[default]
pattern = .*
retentions = 60s:1y
</pre>
</div>
<div>
<pre>
# haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 8192
pidfile /var/run/haproxy.pid
defaults
balance roundrobin
log global
mode tcp
retries 3
option redispatch
contimeout 5000
clitimeout 50000
srvtimeout 50000
# plaintext listener
listen carbon_relay_2003 0.0.0.0:2003
server carbon_relay_2113 127.0.0.1:2113 check maxconn 1024
server carbon_relay_2213 127.0.0.1:2213 check maxconn 1024
server carbon_relay_2313 127.0.0.1:2313 check maxconn 1024
server carbon_relay_2413 127.0.0.1:2413 check maxconn 1024
server carbon_relay_2513 127.0.0.1:2513 check maxconn 1024
server carbon_relay_2613 127.0.0.1:2613 check maxconn 1024
server carbon_relay_2713 127.0.0.1:2713 check maxconn 1024
server carbon_relay_2813 127.0.0.1:2813 check maxconn 1024
# pickle listener
listen carbon_relay_2004 0.0.0.0:2004
server carbon_relay_2114 127.0.0.1:2114 check maxconn 1024
server carbon_relay_2214 127.0.0.1:2214 check maxconn 1024
server carbon_relay_2314 127.0.0.1:2314 check maxconn 1024
server carbon_relay_2414 127.0.0.1:2414 check maxconn 1024
server carbon_relay_2514 127.0.0.1:2514 check maxconn 1024
server carbon_relay_2614 127.0.0.1:2614 check maxconn 1024
server carbon_relay_2714 127.0.0.1:2714 check maxconn 1024
server carbon_relay_2814 127.0.0.1:2814 check maxconn 1024
</pre>
</div>
<div>
<pre>
$ GOPATH=~/gocode ./gocode/bin/haggar -agents=300 \
-metrics=2000 \
-carbon="x.x.x.x:2003"
</pre>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_300k_a.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_300k_a.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 1: Grafana dashboard for Carbon & Whisper</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_300k_b.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_300k_b.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 2: top output for Carbon</span>
</div>jdixonTue, 13 Sep 2016 13:56:30 EDTobfuscurity.Benchmarking Graphite master on AWS
http://obfuscurity.com/2016/09/Benchmarking-Graphite-master-on-AWS
<div>
<p>Hello, friends. Just wanted to follow up the <a href="http://obfuscurity.com/2016/08/Benchmarking-Carbon-and-Whisper-on-AWS">previous post</a> with a quick update. As I've mentioned publicly, developing and cutting a new release from Graphite's <em><strong>master</strong></em> branch has become a personal and professional priority for me. And while I've become very familiar with much of the code base over the course of writing <a href="">The Graphite Book</a> and have thrown a lot of traffic at it over the last few months, I hadn't run any significant tests for performance regressions at scale (compared to the <em>0.9.15</em> release).</p>
<p>This round of tests used the same configuration and benchmarking processes as before. I neglected to mention this before, but all series of benchmarks started with installing Graphite using the <a href="https://github.com/obfuscurity/synthesize">Synthesize</a> setup script. For the previous test I used Synthesize <a href="https://github.com/obfuscurity/synthesize/releases/tag/v2.4.1">v.2.4.1</a> to install Graphite 0.9.15 on a 64-bit Ubuntu 14.04 LTS instance in Amazon's EC2 cloud. For this round I went with Synthesize <a href="https://github.com/obfuscurity/synthesize/releases/tag/v3.0.0RC2">v3.0.0RC2</a>, which targets Graphite's <em>master</em> branch.</p>
</div>
<!--readmore-->
<div>
<hr style="margin-top: 30px;" />
<p><strong>Editor's Note:</strong> This benchmark and blog post were completed a couple weeks ago but I'm only just now getting around to publishing it publicly. The results in this post, as well as the one preceding it, were never intended to represent the theoretical maximum throughput of a single Graphite node. They were intended to serve as a shared exercise (and catalyst for discussion) in benchmarking for the Graphite community.</p>
<hr style="margin-bottom: 30px;" />
</div>
<div>
<p>The results of this round are fairly boring. Aside from minor improvements in CPU and Load consumption, there are no obvious regressions or significant improvements. Write operations, write times, and points per update are consistent with the previous benchmark. Given that the focus of the master branch has been in <em>adding new features</em> (with no intentional focus on performance improvements), I'm very pleased with the results. The only cause for possible concern was a slight increase in data retrieval times. I'm going to look into this further to see if there are any obvious culprits and if it's justification for pushing back the next release. Note that I don't have any useful <strong>rendering statistics</strong> for these rounds since I've been using Grafana (which performs client-side rendering).</p>
<p>To recap, the test host is a single <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html">EBS-optimized</a> EC2 <a href="https://aws.amazon.com/ec2/instance-types/">i2.4xlarge</a> instance with a 400 GiB <a href="https://aws.amazon.com/ebs/details/">EBS Provisioned IOPS SSD</a> (io1) with a requested 20k Max IOPS. All configuration details and benchmarking processes can be found in the previous blog post.</p>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10a.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10a.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 1: Grafana dashboard for Carbon & Whisper</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10b.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10b.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 2: Grafana dashboard for collectd</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10c.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10c.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 3: top output for Carbon</span>
</div>
jdixonMon, 12 Sep 2016 12:15:25 EDTobfuscurity.Benchmarking Carbon and Whisper 0.9.15 on AWS
http://obfuscurity.com/2016/08/Benchmarking-Carbon-and-Whisper-on-AWS
<div>
<p>This is just a quick post to share some recent benchmarking results for a single Graphite 0.9.15 server. The host is a single <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html">EBS-optimized</a> EC2 <a href="https://aws.amazon.com/ec2/instance-types/">i2.4xlarge</a> instance with a 400 GiB <a href="https://aws.amazon.com/ebs/details/">EBS Provisioned IOPS SSD</a> (io1) with a requested 20k Max IOPS.</p>
<p>I'm not going to dive in too deep with the results, but I'll point out that with the following configuration we were able to increase batch writes effectively, resulting in a peak <strong>38 points per update</strong> (<tt>pointsPerUpdate</tt>, averaged across all cache processes). This means that on average, caches were able to flush 38 datapoints from memory to disk with every write request.</p>
</div>
<!--readmore-->
<div>
<p>The carbon configuration below is pretty reasonable, with <strong>six (6) relays</strong> and <strong>eight (8) caches</strong> behind a single HAProxy listener. The <tt>MAX_CACHE_SIZE</tt> has been tuned over the course of a few tests to find a comfortable, finite limit that would accommodate the intended volume of 60k metrics/second. <tt>MAX_UPDATES_PER_SECOND</tt> was set intentionally low in order to trigger the aforementioned high "points per update multiplier".</p>
<p>The retention policy for our benchmark metrics (seen below, <tt>storage-schemas.conf</tt>) was selected to resemble a reasonable real-life, production environment. You could absolutely get more volume or consume fewer resources by removing any of the rollup archives or lowering the precision for the <tt>[haggar]</tt> metrics. I also expect that swapping in a more performant <a href="http://graphite.readthedocs.io/en/latest/tools.html">relay alternative</a> would result in significant efficiencies.</p>
<p>Metric traffic was generated using <a href="https://github.com/gorsuch/haggar">Haggar</a>, a tool which aims to emulate a fleet of servers running the collectd agent, emitting metrics to Carbon.</p>
<p>I'm happy to answer any questions in the comments below regarding this benchmark or other possible benchmarks. Note that I intend to run the same test against Graphite <em>master</em> very soon and will post a follow-up when that is complete.</p>
</div>
<div>
<pre>
# carbon.conf
[relay]
USER = carbon
LOG_LISTENER_CONNECTIONS = False
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 1
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
DESTINATIONS = 127.0.0.1:2104:1, 127.0.0.1:2204:2, 127.0.0.1:2304:3,
127.0.0.1:2404:4, 127.0.0.1:2504:5, 127.0.0.1:2604:6,
127.0.0.1:2704:7, 127.0.0.1:2804:8
[relay:1]
LINE_RECEIVER_PORT = 2113
PICKLE_RECEIVER_PORT = 2114
[relay:2]
LINE_RECEIVER_PORT = 2213
PICKLE_RECEIVER_PORT = 2214
[relay:3]
LINE_RECEIVER_PORT = 2313
PICKLE_RECEIVER_PORT = 2314
[relay:4]
LINE_RECEIVER_PORT = 2413
PICKLE_RECEIVER_PORT = 2414
[relay:5]
LINE_RECEIVER_PORT = 2513
PICKLE_RECEIVER_PORT = 2514
[relay:6]
LINE_RECEIVER_PORT = 2613
PICKLE_RECEIVER_PORT = 2614
[cache]
USER = carbon
CACHE_WRITE_STRATEGY = sorted
MAX_CACHE_SIZE = 3000000
USE_FLOW_CONTROL = True
WHISPER_FALLOCATE_CREATE = True
MAX_CREATES_PER_MINUTE = 12000
MAX_UPDATES_PER_SECOND = 200
USE_INSECURE_UNPICKLER = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False
LOG_LISTENER_CONNECTIONS = False
LOG_UPDATES = False
ENABLE_LOGROTATION = False
WHISPER_AUTOFLUSH = False
[cache:1]
LINE_RECEIVER_PORT = 2103
PICKLE_RECEIVER_PORT = 2104
CACHE_QUERY_PORT = 7102
[cache:2]
LINE_RECEIVER_PORT = 2203
PICKLE_RECEIVER_PORT = 2204
CACHE_QUERY_PORT = 7202
[cache:3]
LINE_RECEIVER_PORT = 2303
PICKLE_RECEIVER_PORT = 2304
CACHE_QUERY_PORT = 7302
[cache:4]
LINE_RECEIVER_PORT = 2403
PICKLE_RECEIVER_PORT = 2404
CACHE_QUERY_PORT = 7402
[cache:5]
LINE_RECEIVER_PORT = 2503
PICKLE_RECEIVER_PORT = 2504
CACHE_QUERY_PORT = 7502
[cache:6]
LINE_RECEIVER_PORT = 2603
PICKLE_RECEIVER_PORT = 2604
CACHE_QUERY_PORT = 7602
[cache:7]
LINE_RECEIVER_PORT = 2703
PICKLE_RECEIVER_PORT = 2704
CACHE_QUERY_PORT = 7702
[cache:8]
LINE_RECEIVER_PORT = 2803
PICKLE_RECEIVER_PORT = 2804
CACHE_QUERY_PORT = 7802
</pre>
</div>
<div>
<pre>
# storage-schemas.conf
[collectd]
pattern = ^collectd\.
retentions = 10s:1w, 60s:1y
[haggar]
pattern = ^haggar\.
retentions = 10s:1d, 60s:1w, 1h:1y
[default]
pattern = .*
retentions = 60s:1y
</pre>
</div>
<div>
<pre>
# haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 8192
pidfile /var/run/haproxy.pid
defaults
balance roundrobin
log global
mode tcp
retries 3
option redispatch
contimeout 5000
clitimeout 50000
srvtimeout 50000
# plaintext listener
listen carbon_relay_2003 0.0.0.0:2003
server carbon_relay_2113 127.0.0.1:2113 check maxconn 1024
server carbon_relay_2213 127.0.0.1:2213 check maxconn 1024
server carbon_relay_2313 127.0.0.1:2313 check maxconn 1024
server carbon_relay_2413 127.0.0.1:2413 check maxconn 1024
server carbon_relay_2513 127.0.0.1:2513 check maxconn 1024
server carbon_relay_2613 127.0.0.1:2613 check maxconn 1024
# pickle listener
listen carbon_relay_2004 0.0.0.0:2004
server carbon_relay_2114 127.0.0.1:2114 check maxconn 1024
server carbon_relay_2214 127.0.0.1:2214 check maxconn 1024
server carbon_relay_2314 127.0.0.1:2314 check maxconn 1024
server carbon_relay_2414 127.0.0.1:2414 check maxconn 1024
server carbon_relay_2514 127.0.0.1:2514 check maxconn 1024
server carbon_relay_2614 127.0.0.1:2614 check maxconn 1024
</pre>
</div>
<div>
<pre>
$ GOPATH=~/gocode ./gocode/bin/haggar -agents=300 \
-metrics=2000 \
-carbon="x.x.x.x:2003"
</pre>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10a.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10a.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 1: Grafana dashboard for Carbon & Whisper</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10b.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10b.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 2: Grafana dashboard for collectd</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10c.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/haggar-test-10c.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 3: top output for Carbon</span>
</div>
jdixonThu, 25 Aug 2016 15:40:17 EDTobfuscurity.Everybody Loves Graphite
http://obfuscurity.com/2015/11/Everybody-Loves-Graphite
<p>There was an article published recently - not here, and not to be linked or referenced here directly - proposing that "nobody loves Graphite" anymore. A linkbait title if I've ever heard one. Many folks linked this article to me, almost certainly expecting me to respond in an uproar. And yet, what I find myself <i>really</i> disappointed over is the obvious misrepresentation of fact (as it pertains to Graphite's technical limits) and an almost malicious disregard for the enormous community that uses it and contributes back to its ongoing development.</p>
<p><b>It's almost as if they're trying to sell you something.</b> Nah, that couldn't be it.</p>
<p>I readily concede that Graphite was not <i>designed</i> for the transient nature of the sort of bleeding-edge containerized, clustering systems that are becoming popular in conference talks and Hacker News (if not in actual use in production, but we'll forgive them this tiny oversight). Admittedly, it takes an expertly skilled engineer to craft a background job for the purpose of removing old cruft. It's not every SysAdmin that knows how to cron, after all.</p>
<!--readmore-->
<p>The rumor being spread, according to this particular author, is that certain individuals are now using <a href="http://grafana.org/">Grafana</a> (<b>gasp!</b>) or "custom dashboards built in-house" (<b>double gasp!</b>) rather than Graphite's user interface. I guess this means that because folks are building and using tools designed to consume Graphite's API, that... nobody loves Graphite? <b>Hmm.</b></p>
<p>As someone who's developed a variety of custom interfaces for the Graphite API, this message resonates with me. I mean, what is Graphite if not just a simple UI for composing graphs? I mean, wouldn't it be <i>grand</i> if it had its own storage engine (or better yet, <a href="http://graphite.readthedocs.org/en/latest/storage-backends.html">pluggable storage backends</a>)? Imagine a world where Graphite could export JSON, to be consumed and rendered using external <a href="http://d3js.org/">libraries</a> or third-party tools. I guess that's just a <a href="http://graphite.readthedocs.org/en/latest/tools.html">pipe dream</a>.</p>
<p>Regardless, the article proceeds to make a number of claims about what Graphite <i>assumes</i>. Let's dissect these briefly.</p>
<div style="padding: 25px 40px 20px;">
<h3>Relatively few metrics</h3>
<p>You can't prove a negative here, so I'm not sure where the author is getting their facts. Production Graphite deployments are known to support <b>many, many millions</b> of metrics. Graphite is designed to scale horizontally, and it does so quite well. I personally helped design and build out large-scale Graphite deployments at Heroku, GitHub, and Dyn.</p>
<h3>Relatively long-lived metrics</h3>
<p>I'm not quite sure when the ability to retain data over time became <b>A Bad Thing ™</b>, but Graphite (or Whisper, I should say) is perfectly happy to let you keep your data as short (minimum one-second resolution) or as long (it has to be a finite length of time) as you wish. If you no longer need that metric, simply delete the file (or setup a cron job to purge them for you).</p>
<h3>A metric's datapoints will exist continually</h3>
<p>Didn't we just cover that one? Oh well, I'll excuse them for needing more than two reasons to tarnish Graphite.</p>
</div>
<p>The article does make some valid points about irregular data and cardinality. However, I would argue that as your data becomes highly irregular and increasingly cardinal, we're no longer talking about time-series data. We're almost never talking about application or host (or container) data. You're talking about the sort of user and event-driven data intended for analytical systems.</p>
<p>None of the systems mentioned in that article are tailored for that workload. They are designed <i>for time-series data</i>, which is successive, sequential, and "more regular than not". In fact, the very same StatsD that the author praises in the article is specifically designed to <b>normalize irregular time-series data</b>.</p>
<p>When all is said and done, the thing that bothers me the most is that the entire hypothesis of the article postulates that Graphite is not a useful system for thousands of businesses, developers, and users out there. And that simply <b><i>is not true</i></b>. While some folks play around with the newest toy databases, breaking them and losing data all the while - Graphite rolls on, serving up graphs and time-series data dutifully and reliably. It doesn't care whether you're using the built-in Composer, the raw data exports, or its JSON format (made all the more popular by the aforementioned Grafana UI).</p>
<p>Admittedly, Graphite isn't the shiniest new monitoring project out there. The great thing is, it doesn't need to be. For the overwhelming majority of businesses trying to keep their servers and applications running, Graphite runs quietly behind the scenes, accepting your metrics and providing visibility to your operations. It's the <b>Volvo of time-series</b> systems, and I'm grateful for that.</p>
<p>In the end, it doesn't matter to me if <b><i>everybody</i></b> loves Graphite, but I for one still do.</p>jdixonThu, 05 Nov 2015 23:15:04 ESTobfuscurity.Graphite 0.9.14 - the Highlights
http://obfuscurity.com/2015/11/Graphite-0.9.14-the-Highlights
<p>As I mentioned in the <a href="http://obfuscurity.com/2015/10/Graphite-the-Phoenix-Release">previous blog post</a>, we're perilously close to shipping the next Graphite release. Although we typically avoid large, sweeping changes in the <a href="https://github.com/graphite-project/graphite-web/tree/0.9.x">stable</a> branch, the long development cycle leading up to this particular release means we have a number of big new features and performance improvements to announce.</p>
<p>Rather than assume everyone will read <i>and understand</i> the significance of all the changes in the 0.9.14 <a href="http://graphite.readthedocs.org/en/0.9.x/releases/0_9_14.html">Release Notes</a>, I felt it would be a good idea to touch on some of them here. This collection represents a small handful of highlights among the numerous changes from this release. The sustained level of interest and contributions from the community continue to astound me. I can't thank everyone enough for their continued support of the Graphite project.</p>
<!--readmore-->
<p>Note that I'm mostly covering <i>shiny new stuff</i> in this post. There are a <b>ton</b> of bugfixes and performance improvements in this release. I encourage everyone to read the <a href="http://graphite.readthedocs.org/en/0.9.x/releases/0_9_14.html#bug-fixes">Bug Fixes</a> section of the release notes for a thorough list of those changes.</p>
<h3 style="margin-top: 40px;">Merged timeseries results from remote queries</h3>
<p>If you scale Carbon horizontally, you probably understand the hassle involved with keeping Whisper files synchronized. Tools like <a href="https://github.com/jssjr/carbonate">Carbonate</a> help, but it can't prevent network partitions or similar outages that might result in lost time-series data in the first place.</p>
<p>Prior to 0.9.14, Graphite-Web employed a "first match wins" policy, rendering the first set of data returned among a pool of backend ("remote") servers. Ironically, all else being equal (network transit, disk speed, etc), this result tended to be the _least complete_ set of data (assuming a synchronization disparity). Fortunately, with <tt>REMOTE_STORE_MERGE_RESULTS</tt> enabled (the default behavior), Graphite-Web now <a href="https://github.com/graphite-project/graphite-web/pull/1352">automatically merges</a> any remote datasets to form the most complete set of data available.</p>
<p>In the following examples, we have two servers, each with their own incomplete set of datapoints. <b><i>Fig 1</i></b> demonstrates the legacy behavior, rendering the first partial set returned. <b><i>Fig 2</i></b> reveals the new behavior, merging the results from both backend nodes before rendering the final set (or in this case, passing it back to Grafana).</p>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example1.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example1.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 1: Old "first match wins" behavior</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example2.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example2.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 2: Merge results with <tt>REMOTE_STORE_MERGE_RESULTS</tt> enabled</span>
</div>
<p>Also worth noting is that remote fetches are now <a href="https://github.com/graphite-project/graphite-web/pull/1026">run in parallel</a> using threads. This is a significant improvement over the old sequential behavior, especially when retrieving globbed metrics across a pool of backend servers.</p>
<h3 style="margin-top: 40px;">Cache consistency improvements</h3>
<p>There are a variety of cache consistency enhancements and bugfixes in this release. Many of them affected the merging of hot datapoints in cache memory and cold datapoints pulled from Whisper archives. One particularly nasty bug discovered by <a href="https://github.com/jjneely">jjneely</a> would appear anytime a query range was smaller than the interval of the Whisper archive, e.g. a 10-second query attempting to retrieve from a 30-second interval. The query would perform a "wrap-read" on the entire archive, causing excessively long query times and returning large arrays of null datapoints. <b><i>Fig 3</i></b> shows the effectiveness of the fix in action.</p>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example3.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example3.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 3: Bugfix for sub-interval queries</span>
</div>
<h3 style="margin-top: 40px;">Single-hue color schemes</h3>
<p>In the course of writing the <a href="http://shop.oreilly.com/product/0636920035794.do">Graphite book</a> I needed to make some <tt>colorList</tt> schemes that rendered well for print. The result was a variety of single-hue color schemes that I've merged into the Graphite-Web project. Check out <a href="https://github.com/graphite-project/graphite-web/pull/1164">this PR</a> for examples of each.</p>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example7.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example7.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 4: ocean2 lineColors scheme</span>
</div>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example8.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example8.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 5: lava1 lineColors scheme</span>
</div>
<h3 style="margin-top: 40px;">Serving static assets with Whitenoise</h3>
<p>Serving static files via Django and WSGI became a little more complicated over the last couple years, forcing administrators to adopt <a href="http://obfuscurity.com/2014/01/Graphite-Tip-Django-Admin-Workaround">elaborate configurations</a> for serving up static content within the webapp. Graphite-Web now supports serving up static content automatically if the <a href="https://pypi.python.org/pypi/whitenoise">Whitenoise</a> module is installed on the host server.
<h3 style="margin-top: 40px;">URL shortener</h3>
<p>One of the fun things about being a Graphite project maintainer is all the interesting pull requests we see from outside contributors. I was initially hesitant about adding a URL shortener feature to the webapp, but everyone loved the idea so we merged it in. Like other shorteners you're familiar with, this one exports a URL that redirects to the rendered graph, but it avoids the excessively long links that can be truncated by various chat or messaging services.</p>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example4.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example4.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 6: Graph export by URL shortener</span>
</div>
<h3 style="margin-top: 40px;">Timezone fixes</h3>
<p>I won't go into all the gritty details, but I'm very pleased with the attention given to fixing timezone bugs in this release. If you encountered problems in the past I urge you to give 0.9.14 a try. As always, please <a href="https://github.com/graphite-project/graphite-web/issues/new">open an issue</a> if you're still experiencing any problems.</p>
<h3 style="margin-top: 40px;">Import Graph URLs</h3>
<p>Remember my old tip about the <a href="http://obfuscurity.com/2012/04/Unhelpful-Graphite-Tip-2">Graph bookmarklet</a>? This feature is now built right into the Graphite-Web composer. It supports any valid graph URL, although it doesn't support the shortened URLs mentioned earlier; those would need to be expanded first before imported into this field.</p>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example5.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example5.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 7: Import graph by URL</span>
</div>
<h3 style="margin-top: 40px;">IPv6 support</h3>
<p>Graphite-Web and Carbon both support IPv6 with this release. This will be a huge relief to the three people currently running IPv6 on the internet (wink).</p>
<h3 style="margin-top: 40px;">Reorder metrics</h3>
<p>This one is pretty self-explanatory. Sometimes you may want to reorder the list of metrics (especially if you've applied a bunch of functions to them), especially for stacked graphs. This new UI element makes it easy and even supports multi-select.</p>
<div style="margin: 30px 0;">
<a href="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example6.png">
<img src="https://s3.amazonaws.com/obfuscurity-blog/graphite_0-9-14_example6.png" style="width: 100%;">
</a>
<span style="display: inline-block; margin-top: 5px; font-style: italic; font-size: 0.8em;">Fig 8: Reordering metrics</span>
</div>
<h3 style="margin-top: 40px;">New Functions and Graph Parameters</h3>
<ul>
<li><tt><a href="http://graphite.readthedocs.org/en/0.9.x/functions.html#graphite.render.functions.changed">changed()</a></tt></li>
<li><tt><a href="http://graphite.readthedocs.org/en/0.9.x/functions.html#graphite.render.functions.minimumBelow">minimumBelow()</a></tt></li>
<li><tt><a href="http://graphite.readthedocs.org/en/0.9.x/functions.html#graphite.render.functions.perSecond">perSecond()</a></tt></li>
<li><tt><a href="http://graphite.readthedocs.org/en/0.9.x/functions.html#graphite.render.functions.sortByName">sortByName()</a></tt></li>
<li><tt><a href="http://graphite.readthedocs.org/en/0.9.x/render_api.html#maxdatapoints">maxDataPoints</a></tt></li>
</ul>jdixonSun, 01 Nov 2015 11:11:12 ESTobfuscurity.Graphite 0.9.14 - the Phoenix Release
http://obfuscurity.com/2015/10/Graphite-the-Phoenix-Release
<p>If you're like most Graphite users, you're probably wondering if and when there will ever be another release for the project. There hasn't been much public activity over the last couple of years, at least outside of GitHub. A lack of corporate sponsorship, in terms of dedicated developer and maintainer hours, means that the project receives attention as volunteers' schedules permit. Speaking solely for myself, I prioritize Graphite development somewhere behind family, work, the <a href="http://monitorama.com/">Monitorama</a> conference, writing the <a href="http://shop.oreilly.com/product/0636920035794.do">Graphite book</a>, and "other recreational activities".</p>
<p>Despite the lack of a regular release cycle, Graphite is as popular as ever. The Grafana project is going gangbusters, with Graphite as its priority time-series backend. A variety of new open source projects have cropped up offering high-performance alternatives to the original specification implementations (<a href="https://github.com/graphite-project/graphite-web">graphite-web</a> and <a href="https://github.com/graphite-project/carbon">carbon</a>). New software projects, both commercial and open source, continue to target Graphite API compatibility because of its ubiquity and ease of use. Heck, even those other competing time-series engines are forced to support Graphite-friendly interfaces. In some cases they even outperform their own proprietary ingress methods.</p>
<!--readmore-->
<p>Regardless, as a Graphite developer and maintainer, I recognize that we <i>must</i> step up and move the project forward or risk irrelevancy. We've clutched onto Graphite's stable branch (0.9.x) far longer than it was ever intended to exist. For this reason (and many others), I'm pleased to announce that <b>the upcoming 0.9.14 release will officially be the last of the 0.9.x lineage</b>.</p>
<p>We've been merging new features and updating existing ones in the master branch at a heated pace over the last few months. I fully expect that the Graphite project will experience it's long-awaited <b>1.0 release</b> early in the 2016 calendar year.</p>
<p>Because the 0.9.13 release <i>never technically existed</i>, there's nearly two years' worth of <a href="http://graphite.readthedocs.org/en/latest/releases/0_9_14.html">Release Notes</a> for interested parties to digest. In order to make this a bit more palatable, I'm going to dedicate an upcoming blog post to covering some of the more interesting, exciting, and possibly confusing aspects of this release, so keep an eye out for this in the coming days.</p>
<h3 style="margin-top: 30px;">One Small Note</h3>
<hr>
<p>I want to take a moment to express my gratitude to my friends and coworkers at <a href="https://www.librato.com/">Librato</a>. I love these people and the world-class metrics and visualization platform we've built. Despite my involvement in the Graphite project, I'm a strong proponent of SaaS infrastructure services. Unless <i>building your own time-series architecture</i> provides your business with a <a href="http://blog.librato.com/posts/monitoring-software-build-buy">competitive advantage</a>, it's probably in your best interests to outsource this to people who know how (and really love doing it). I've seen so many companies try and fail spectacularly at scaling their time-series datastore, without acknowledging that it wasn't necessary in the first place.</p>
<p>If you're not firmly committed to scaling up Graphite or one of the other open source time-series platforms, I encourage you to <a href="https://metrics.librato.com/sign_up">check out Librato's free trial</a> for yourself. In fact, if you have any questions about Librato (including transitioning to it from Graphite), I'm <a href="mailto:jdixon+graphite@librato.com">happy to chat</a> with you personally.</p>jdixonTue, 27 Oct 2015 23:55:39 EDTobfuscurity.On Writing the Graphite Book
http://obfuscurity.com/2015/02/on-writing-the-graphite-book
<p>I'm writing a book. This may come as a surprise given the lack of content on this blog over the last... entire year of 2014. Nevertheless, I'm pleased to report that the rumors are true and I am in fact writing a book about Graphite.</p>
<p>Three different editors at O'Reilly contacted me over the course of a few years about the possibility of authoring a volume about my favorite Open Source time-series rendering engine. I had significant concerns about the availability of free time I'd have to spend on this project, so I had to turn them down the first couple times. Last year, something finally clicked and I relented. And so, <a href="http://shop.oreilly.com/product/0636920035794.do">Monitoring with Graphite</a> became a thing.</p>
<div style="margin: 30px 0;"><img src="https://s3.amazonaws.com/obfuscurity-blog/graphite-book.png" width="700" /></div>
<p>We've decided to release it as a work in progress, with the Early Release going on sale in December 2014 and an expected official release around June 2015. According to the outline we're almost at the halfway point of the book, so I think it's reasonable to say we're still on schedule.</p>
<p>If you've enjoyed my blog posts, I really think you'll love the book. I've included a healthy discussion around monitoring concepts and the "composable monitoring system", a deep dive into the Graphite components, fully fleshed-out installation processes and tips of the trade, and a helluva lot more. I aim to be as comprehensive as possible while still managing to keep it an entertaining read. Frankly, this is probably the only subject matter that I'll know well enough to write a book about, so I'm not about to let myself screw it up.</p>
<p>I encourage you to grab the Early Release Ebook and provide feedback. Your comments and suggestions (or questions) will continue to fuel the content for the rest of the book. And if you make it out to <a href="http://monitorama.com/">Monitorama</a> this summer, I'll be happy to sign your tablet or laptop.</p>jdixonSun, 22 Feb 2015 23:56:56 ESTobfuscurity.Graphite Tip - A Better Way to Store Events
http://obfuscurity.com/2014/01/Graphite-Tip-A-Better-Way-to-Store-Events
<p>Graphite is well known for storing simple key/value metrics using the Whisper time-series database on-disk format. What is <em>not</em> well known about Graphite is that it also ships with a feature known as <b>Events</b> that supports a richer form of metrics storage suitable for, well, <em>events</em>. Imagine a place where you could store tagged metrics and additional data relevant to the event (e.g. code snippets, comments, etc). Many folks use NoSQL databases such as HBase for this purpose, and that's a perfectly reasonable approach. However, if you'd like to store these somewhere where they can be correlated with the rest of your Graphite metrics, then Events might be a good fit for you.</p>
<!--readmore-->
<p>Examples of events that are appropriate for this metric type include <em><b>releases</b></em>, <em><b>commits</b></em>, <em>application <b>exceptions</b></em>, and generally anything that represents a <em><b>state change</b></em> where you might wish to track the affected data.</p>
<h4>Database Storage</h4>
<p>Because Whisper was designed to hold simple time-series data (metric key, value, and timestamp), it's altogether <em>unsuitable</em> for storing rich metric data. Users <a href="http://codeascraft.com/2010/12/08/track-every-release/">began to experiment</a> with storing <em>event-type</em> data (e.g. releases, state changes, etc) in Whisper and visualizing them with Graphite's <tt>drawAsInfinite()</tt> function. However, taking advantage of this pattern typically requires the use of wildcards across a significant number of these singleton metric files and directories, which can cause a significant performance hit on the server and result in a poor experience for users.</p>
<p>Therefore, Graphite's <em>application database</em> was <a href="https://code.launchpad.net/~lucio.torre/graphite/add-events/+merge/69142">extended</a> to support this new metric type. As long as you're running Graphite version 0.9.9 or newer, there are no changes needed to begin storing Events in the database. However, care must be taken to ensure that your Graphite webserver and database can scale with the level of event metrics that you intend to throw at it. I personally <a href="http://obfuscurity.com/2013/12/Why-You-Shouldnt-use-SQLite-with-Graphite">recommend</a> PostgreSQL, but MySQL users should be fine to store events on their database as well.</p>
<h4>Working with Events</h4>
<p>Events can be submitted via HTTP POST or by inserting them manually using the Graphite administration module.</p>
<h5>Adding Events by HTTP POST</h5>
<pre>
$ curl -X POST "http://graphite/events/"
-d '{"what": "Event - deploy", "tags": "deploy",
"data": "deploy of master branch happened at Fri Jan 3 22:34:41 UTC 2014"}'
</pre>
<h5>Managing Events by Admin UI</h5>
<p>You can add, edit and delete Events using the Graphite administration module (found at <tt>/admin/</tt>). This is particularly handy for deleting a large number of events at once.</p>
<img style="margin: 30px 0 40px;" src="https://s3.amazonaws.com/obfuscurity-blog/graphite_events_0.png" width="600" />
<h5>Querying Events</h5>
<p>Graphite allows you to query for tags associated with events. You can search for a single tag string, a combination of tags, or a simple * wildcard using the <a href="http://graphite.readthedocs.org/en/0.9.12/functions.html#graphite.render.functions.events"><tt>events()</tt></a> function.</p>
<pre>
$ curl -s http://graphite/render/?target=events("exception")&format=json | json_pp
[
{
"target" : "events(exception)",
"datapoints" : [
[
1,
1388966651
],
[
3,
1388966652
],
...
...
...
[
2,
1388966656
],
[
1,
1388966657
]
]
}
]
</pre>
<img style="margin: 30px 0 40px;" src="https://s3.amazonaws.com/obfuscurity-blog/graphite_events_1.png" width="600" />
<h5>Browsing Events</h5>
<p>You can also browse events using Graphite's events index (found at <tt>/events/</tt>). Beware that this page lacks any sort of pagination, so a large number of events could cause significant load time in your browser.</p>
<img style="margin: 30px 0;" src="https://s3.amazonaws.com/obfuscurity-blog/graphite_events_2.png" width="600" />
<p>From there you can click on any of the individual events to view its details.</p>
<img style="margin: 30px 0;" src="https://s3.amazonaws.com/obfuscurity-blog/graphite_events_3.png" width="600" />
jdixonSun, 05 Jan 2014 20:54:03 ESTobfuscurity.