Jekyll2022-12-15T20:18:30+00:00https://mjrusso.com/feed.xmlmjrusso.com | Michael RussoMichael Russo's blogMichael Russohttps://mjrusso.com/Copyright (c) 2022 Michael Russo. All Rights Reserved.Scoot, a MacOS Cursor Actuator2022-07-21T00:00:00+00:002022-07-21T00:00:00+00:00http://blog.mjrusso.com/introducing-scoot
<p><a href="https://github.com/mjrusso/scoot">Scoot</a> is a Mac app that helps you move your mouse cursor, using keyboard shortcuts.</p>
<p>When I started building the first prototype, I didn’t expect it to actually work (technically or practically, but <em>especially</em> the latter). From an early version of the project’s README:</p>
<blockquote>
<p><strong>Scoot is experimental. Is it possible to craft a keyboard-driven mouse
movement utility that’s <em>actually</em> efficient? Something that you’ll actually
<em>want</em> to use? This is going to take some trial and error.</strong></p>
</blockquote>
<p>After ten months of intermittent development, I finally <a href="https://github.com/mjrusso/scoot/commit/6dba0d263518068331a6b456c19fe1a570537308">removed that disclaimer</a>. Scoot is still a highly niche app, but it actually works, and has (a few) real fans that are using it every day.<sup id="fnref:usage-data"><a href="#fn:usage-data" class="footnote">1</a></sup></p>
<hr />
<p>Quick backstory: I had been experiencing pain in my ulnar nerve. I was making a lot of changes to get that under control, but during the process I noticed that mouse and trackpad usage exacerbated all the symptoms. As it happens, I had recently started using the excellent <a href="https://github.com/abo-abo/avy">avy</a> Emacs package, and it occurred to me that those ideas could be brought to the OS level.</p>
<p>I couldn’t find any prior art, so I opened up Xcode and started hacking around.</p>
<hr />
<p>Let’s take a quick tour…</p>
<p>Scoot provides two primary navigation modes.</p>
<p>In <em>grid-based navigation mode</em>, Scoot subdivides the screen into a grid of equally-sized cells, and assigns each a unique character sequence. Type that character sequence, and your mouse cursor immediately moves to that location.</p>
<figure>
<img src="/assets/introducing-scoot/grid.jpg" />
<figcaption>Scoot has assigned the key sequence "pg" to the cell in the top left corner of the screen. Every cell is assigned a unique character sequence. <a href="/assets/introducing-scoot/grid-nav.mp4" target="_blank">Video demo.</a></figcaption>
</figure>
<!--
<video class="video-player" autoplay loop muted controls>
<source src="/assets/introducing-scoot/grid-nav.mp4" type="video/mp4" />
Your browser is unable to play this video file.
</video>
-->
<p>If multiple displays are connected, Scoot extends the grid across each screen:</p>
<figure>
<img src="/assets/introducing-scoot/grid-multiple-displays.jpg" />
<figcaption>Every cell (regardless of which screen it's on) is assigned a unique character sequence.</figcaption>
</figure>
<p>My personal favourite, though, is the <em>element-based navigation mode</em>. In this mode, Scoot acts like a screen reader, finding UI elements (buttons, links, etc.) on screen, and assigning each a unique key sequence. Type the character sequence associated with a UI element, and your mouse cursor will immediately move there.</p>
<figure>
<img src="/assets/introducing-scoot/element.jpg" />
<figcaption>In this example, Scoot has identified the only link on the page ("More information..."), and assigned it the key sequence "aa". The buttons in the toolbar have also been assigned key sequences. <a href="/assets/introducing-scoot/element-nav.mp4" target="_blank">Video demo.</a></figcaption>
</figure>
<!--
<video class="video-player" autoplay loop muted controls>
<source src="/assets/introducing-scoot/element-nav.mp4" type="video/mp4" />
Your browser is unable to play this video file.
</video>
-->
<p>When Scoot is active, you can also move the mouse cursor using keyboard shortcuts that you’re likely already familiar with for editing text.<sup id="fnref:text-editing-repurposed"><a href="#fn:text-editing-repurposed" class="footnote">2</a></sup> Scoot supports standard <a href="https://support.apple.com/en-ca/HT201236#text">MacOS text editing shortcuts</a>, as well as Emacs and vi keybindings. These shortcuts work regardless of which mode is active.</p>
<p>For example:</p>
<ul>
<li>
<p>to nudge the cursor up by a partial “step”, press the <code class="highlighter-rouge">↑</code> key on your keyboard (equivalent to <code class="highlighter-rouge">C-p</code> with Emacs bindings, and <code class="highlighter-rouge">k</code> with vi bindings)</p>
</li>
<li>
<p>to nudge the cursor up a larger amount (a full “step”), press <code class="highlighter-rouge">⌥↑</code> (equivalent to <code class="highlighter-rouge">M-a</code> with Emacs bindings, and <code class="highlighter-rouge">C-k</code> with vi bindings)</p>
</li>
<li>
<p>to move the cursor to the top of the screen, press <code class="highlighter-rouge">⌘↑</code> (equivalent to <code class="highlighter-rouge">M-<</code> with Emacs bindings, and <code class="highlighter-rouge">⇧-k</code> with vi bindings)</p>
</li>
</ul>
<p>These are, of course, only a small subset of movement commands. For the full list, see the <a href="https://github.com/mjrusso/scoot/#usage">usage documentation</a>.</p>
<p>In addition to cursor movement, Scoot also provides keyboard shortcuts for scrolling, left clicking, double clicking, clicking the middle and right mouse buttons, and even for holding the left mouse button down (which enables drag-and-drop, all from the comfort of your keyboard).</p>
<!--There's even a _"freestyle" navigation mode_, which allows you to move the cursor (using the system/Emacs/vi keyboard shortcuts, of course), and click and drag, without any additional navigation assistance.-->
<hr />
<p>Scoot helps reduce my reliance on my mouse and trackpad, especially when I’m at my desk. I use a split mechanical keyboard<sup id="fnref:mouse-emulation"><a href="#fn:mouse-emulation" class="footnote">3</a></sup> with a modified <a href="https://github.com/manna-harbour/miryoku">Miryoku</a> keyboard layout. Miryoku puts every key within reach without needing to move my hands – and with Scoot, I don’t need to move my hands to “actuate” the cursor either. Huge win in comfort (and efficiency).<sup id="fnref:laptop-usage"><a href="#fn:laptop-usage" class="footnote">4</a></sup></p>
<figure>
<img src="/assets/introducing-scoot/desk.jpg" />
<figcaption>Current desk setup. Note the split mechanical keyboard, the trackpad on the left, and the vertical mouse on the right.</figcaption>
</figure>
<p>That being said, it’s important to note that Scoot is <em>not</em> intended to replace physical pointing devices, but rather to augment them. (Replacing hardware entirely is not possible for a variety of reasons. For example, the OS–and many apps–rely heavily on gestures, which Scoot cannot simulate,<sup id="fnref:reverse-engineering"><a href="#fn:reverse-engineering" class="footnote">5</a></sup> among other challenges.<sup id="fnref:system-ui"><a href="#fn:system-ui" class="footnote">6</a></sup>)</p>
<p>And sometimes it’s just easier to grab a mouse and move it where you want it to go.</p>
<hr />
<p>If this sounds remotely interesting, give Scoot a spin, and let me know what you think. You can <a href="https://github.com/mjrusso/scoot/releases/latest/">download the latest release here</a>, and refer to the documentation for <a href="https://github.com/mjrusso/scoot#installation">installation instructions</a> and a <a href="https://github.com/mjrusso/scoot/#usage">usage guide</a>.</p>
<p><em>(Scoot is fully open source, and MIT licensed.)</em></p>
<figure>
<img src="/assets/introducing-scoot/icon.png" style="max-width: 200px; display: block; margin: 0 auto;" />
<figcaption></figcaption>
</figure>
<hr />
<div class="footnotes">
<ol>
<li id="fn:usage-data">
<p>I know this only because people tell me. There’s no tracking or analytics of any kind… Scoot isn’t even allowed to access the network. <a href="#fnref:usage-data" class="reversefootnote">↩</a></p>
</li>
<li id="fn:text-editing-repurposed">
<p>Scoot adapts existing keyboard shortcuts – intended for navigating around in a document – to instead control movement on a 2-dimensional grid. Some liberties have been taken with this mapping, but (hopefully) the results are intuitive. <a href="#fnref:text-editing-repurposed" class="reversefootnote">↩</a></p>
</li>
<li id="fn:mouse-emulation">
<p>Most of my mechanical keyboards support mouse emulation (thanks to <a href="https://github.com/qmk/qmk_firmware/blob/master/docs/feature_mouse_keys.md">QMK</a> and <a href="https://github.com/zmkfirmware/zmk/pull/778">ZMK</a> firmware), but because Scoot is “context-aware”, it’s in a different league of power/efficiency/utility. <a href="#fnref:mouse-emulation" class="reversefootnote">↩</a></p>
</li>
<li id="fn:laptop-usage">
<p>I find myself using Scoot a lot less when I’m on my laptop; trackpads these days are humungous, and easily accessible with my thumbs, even when I’m typing with both hands. <a href="#fnref:laptop-usage" class="reversefootnote">↩</a></p>
</li>
<li id="fn:reverse-engineering">
<p>Apple doesn’t provide any APIs for simulating gestures. And to the best of my knowledge, no one has put in the reverse engineering work required to figure out how to synthesize them. If any reader is interested in tackling this challenge, gesture support would make for <a href="https://github.com/mjrusso/scoot/issues/5">a lovely addition</a> to the app. <a href="#fnref:reverse-engineering" class="reversefootnote">↩</a></p>
</li>
<li id="fn:system-ui">
<p>Scoot is able to interact with most UI elements, but not all. For example, Scoot can’t dismiss system notifications. <a href="#fnref:system-ui" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Michael Russohttps://mjrusso.com/Adventures in Augmenting Creativity2019-04-23T00:00:00+00:002019-04-23T00:00:00+00:00http://blog.mjrusso.com/augmenting-creativity
<p><a href="https://www.robinsloan.com">Robin Sloan</a> has been generating extremely interesting audio using machine learning models (see: <a href="https://www.mcdbooks.com/features/sourdough">Making the music of the Magz</a> and <a href="https://www.robinsloan.com/expressive-temperature/">Expressive temperature</a>).<sup id="fnref:text-too"><a href="#fn:text-too" class="footnote">1</a></sup></p>
<p>Inspired, I took these techniques for a spin.</p>
<hr />
<p><em>First question:</em> what audio to use to train the machine learning model?</p>
<p>Oneohtrix Point Never’s <em>R Plus Seven</em> immediately came to mind. From <a href="https://pitchfork.com/reviews/albums/18537-oneohtrix-point-never-r-plus-seven/">Pitchfork’s review</a>:</p>
<blockquote>
<p>On <em>R Plus Seven</em>, Oneohtrix Point Never’s follow-up to 2011’s <em>Replica</em>, Daniel Lopatin builds new music using the bright yet cold textures of the early computing age. The album plays with our collective unconscious of music technology to develop something that comes off as strange and otherworldly and, most importantly, rich with feeling, despite the icy surface layer.</p>
</blockquote>
<p>Perfect.</p>
<hr />
<p><strong>A quick detour:</strong> Iliana and I visited the beautiful Cape Breton Island in 2015, and found ourselves drawn to <em>R Plus Seven</em>, listening to it over and over and over again.</p>
<p>The album’s ethereal quality was a match for the scenery:</p>
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); grid-gap: 4px; margin-bottom: 4px;">
<div><img src="/images/posts/2019-04-23-augmenting-creativity/DSCF9705.jpg" /></div>
<div><img src="/images/posts/2019-04-23-augmenting-creativity/DSCF9981.jpg" /></div>
<div><img src="/images/posts/2019-04-23-augmenting-creativity/DSCF9931.jpg" /></div>
<div><img src="/images/posts/2019-04-23-augmenting-creativity/DSCF0022.jpg" /></div>
</div>
<figure>
<img src="/images/posts/2019-04-23-augmenting-creativity/DSCF0061.jpg" />
<figcaption></figcaption>
</figure>
<p>Haunting, yet beautiful, and not unlike the audio I’m hoping to produce.</p>
<hr />
<p>After spinning up a server<sup id="fnref:cloud"><a href="#fn:cloud" class="footnote">2</a></sup> and configuring the necessary dependencies, I fed the <a href="https://github.com/richardassar/SampleRNN_torch/blob/a5af976cf618a07cebd6965705190164be1f57fe/scripts/generate_dataset.lua">preprocessed audio</a> (all of <em>R Plus Seven</em>) into <a href="https://github.com/richardassar/SampleRNN_torch">SampleRNN_torch</a>, and started the training process.</p>
<p>I let the model train for approximately 72 hours. (After 48 hours of training, the sounds were starting to take shape; the model was clearly picking up on the structure of the source material. By the 72nd hour, it seemed to have learned all it was going to learn.)</p>
<p>It took another day of processing to generate about two-and-a-half hours of audio.<sup id="fnref:sample-quantity"><a href="#fn:sample-quantity" class="footnote">3</a></sup> (It’s a good idea to generate an abundance of samples, and later discard the dreary and the banal. The quality of samples will vary widely.)</p>
<p>Let’s listen to a few examples. Note that with the exception of the conversion to MP3, these samples are not processed or otherwise edited; they’re pulled directly from SampleRNN.</p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/061503_29102018_49053iters-1.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>It’s striking how clear the lineage is to <em>R Plus Seven</em>. It sounds related, but it’s also something very different.</p>
<p><em><strong>Note:</strong> the overall audio has a rougher, grungier texture than the source material; it’s as if the speaker that’s playing back the sound has a low-grade headache. Expect the audio produced by systems like these to improve in the future.</em></p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/061503_29102018_49053iters-5.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>There are these moments sprinkled throughout that you can pinpoint precisely to specific tracks from the album. It sounds <em>almost</em> like they’ve been sampled.</p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/104302_29102018_49053iters-3.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>And it’s even more pronounced in this example.</p>
<hr />
<p>Now, let’s borrow from Robin’s technique of <a href="https://www.robinsloan.com/expressive-temperature/#knob">varying the temperature</a>:</p>
<blockquote>
<p>In machine learning systems designed to generate text or audio, there’s often a final step between the system’s model of the training data and some concrete output that a human can evaluate or enjoy. Technically, this is <small>the sampling of a multinomial distribution</small>, but call it rolling a giant dice, many-many-sided, and also weighted. Weird dice.</p>
<p>To that final dice roll, there’s a factor applied that is called, by convention, the “sampling temperature.” Its default value is 1.0, which means we roll the weighted dice just as the model has provided it. But we can choose other values, too. We can tamper with the dice.</p>
</blockquote>
<p>I’ve tampered with the dice, generating samples with a starting temperature of <em>0.875</em>, increasing at a constant rate until reaching <em>1.15</em> (by the end of the sample).</p>
<p>Let’s listen:</p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/013102_30102018_49053iters_vary-temp-1.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>This begins slow, cautious. Boring. Eventually, it starts going somewhere, but not really because it’s mostly moving in circles, until the end—<em>temperature cranked all the way up!</em>—when it’s unhinged, free of constraints, soaring.</p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/013102_30102018_49053iters_vary-temp-2.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>I love that break in the middle of the sample; surely, it’s deliberate preparation for the chaotic outburst that will be the final segment.</p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/013102_30102018_49053iters_vary-temp-3.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>What an ending.<sup id="fnref:ending"><a href="#fn:ending" class="footnote">4</a></sup></p>
<p><em>Expressive temperature,</em> indeed.</p>
<hr />
<p>By remixing and combining the generated sounds, it’s possible to create entirely new works.</p>
<p>Here’s an original composition; let’s call it <em><strong>“RNN+7”</strong></em>:</p>
<audio controls="" class="audio-player">
<source src="/assets/augmenting-creativity/RNN+7_mastered.mp3" type="audio/mpeg" />
Your browser is unable to play this audio file.
</audio>
<p>This particular piece was created by splicing together audio samples generated by SampleRNN, with no additional sounds added. (Some noise reduction was applied to the mix.)</p>
<p>In the hands of a musician (i.e., not me), this quirky, idiosyncratic, probabilistic sampler could make for one hell of an instrument.</p>
<hr />
<p><strong>Where all of this <em>really</em> shines is as input to the human creative process.</strong></p>
<p>To create <em>RNN+7</em>, I used the samples produced by the model as input—the only input—to the final piece.</p>
<p>But those samples, instead, could have been used strictly as a creative input. <em>As inspiration.</em></p>
<p>Today’s machine learning models can produce material for the artist to dissect, probe, and riff on. Tomorrow’s will create material to help the artist explore the space of possible solutions, without the burden of preconceived notions or a nagging inner critic. They will be capable of taking the artist down paths that they may never have explored on their own.<sup id="fnref:future"><a href="#fn:future" class="footnote">5</a></sup></p>
<p>And that’s where this gets really exciting.</p>
<p>Regardless of the medium you work in—words written or spoken, images still or moving, sounds melodic or atonal, et cetera, et cetera—one thing <em>is</em> for sure:</p>
<p><strong><em><small><center>The inspiration machines are coming.</center></small></em></strong></p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:text-too">
<p>And likewise, for text. (See: <a href="https://www.robinsloan.com/notes/writing-with-the-machine/">Writing with the machine</a>, and <a href="https://www.robinsloan.com/voyages-in-sentence-space/">Voyages in sentence space</a>.) <a href="#fnref:text-too" class="reversefootnote">↩</a></p>
</li>
<li id="fn:cloud">
<p>I used <a href="https://salamander.ai">Salamander</a> to provision an instance with an NVIDIA K80 GPU (and 4x vCPU, with 61 GB RAM), at a cost of approximately $0.37 (USD) per hour. <a href="#fnref:cloud" class="reversefootnote">↩</a></p>
</li>
<li id="fn:sample-quantity">
<p>215 samples, each 45 seconds long. <a href="#fnref:sample-quantity" class="reversefootnote">↩</a></p>
</li>
<li id="fn:ending">
<p>And, dare I say, <em>“strange and otherworldly?”</em> <a href="#fnref:ending" class="reversefootnote">↩</a></p>
</li>
<li id="fn:future">
<p><em><strong>Machine as muse:</strong></em> One day, it will be standard practice for artists to find inspiration with the help of machine learning models. (Expect incredible care to go into the selection of the training material.) <a href="#fnref:future" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Michael Russohttps://mjrusso.com/Fun with X-H12018-03-15T00:00:00+00:002018-03-15T00:00:00+00:00http://blog.mjrusso.com/fun-with-xh1
<p>I’ve spent the past two weeks with the Fujifilm X-H1, and have been blown away by the image quality and overall camera performance.<sup id="fnref:battery"><a href="#fn:battery" class="footnote">1</a></sup></p>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/xh1.jpg" />
<figcaption>The only photo in this post not shot on the X-H1.</figcaption>
</figure>
<p>Note that I upgraded from the Fujifilm X100S, a camera that will always occupy a fond place in my heart. (My son is not quite crawling yet, but he’s already moving around more quickly than the X100S’s autofocus system can keep up with. It was time.)<sup id="fnref:learning"><a href="#fn:learning" class="footnote">2</a></sup></p>
<p>I love how I’m reaching for my camera more frequently. It’s incredibly liberating to be able to shoot in almost any light, and my behaviour and approach to photography has shifted accordingly.<sup id="fnref:ibis"><a href="#fn:ibis" class="footnote">3</a></sup></p>
<hr />
<p>In this post, I’ve included a number of my favourite shots from these first few weeks with the X-H1. Every photograph was captured handheld, across various lighting conditions, without any flash or other lighting sources added to the scene.</p>
<p><em>(Huge thanks to my family for being such beautiful subjects.)</em></p>
<hr />
<p>For starters, a series of shots captured in ample light:</p>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1092.jpg" />
<figcaption>XF50mmF2 at f/3.2, 1/220 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1092-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0481.jpg" />
<figcaption>XF23mmF2 at f/2.0, 1/60 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0481-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0170.jpg" />
<figcaption>XF23mmF2 at f/2.0, 1/60 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0170-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0507.jpg" />
<figcaption>XF23mmF2 at f/2.0, 1/170 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0507-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1145.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/75 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1145-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0238.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/140 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0238-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1224.jpg" />
<figcaption>XF50mmF2 at f/4.5, 1/200 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1224-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1493.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/120 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1493-full.jpg">full size</a>.</figcaption>
</figure>
<hr />
<p>Lower light shots, at relatively high ISO:</p>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0011.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/75 sec, ISO 2000. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0011-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1033.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/75 sec, ISO 2000. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1033-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0304.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/75 sec, ISO 2000. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0304-full.jpg">full size</a>.</figcaption>
</figure>
<p>To my eyes, the grain has a nice character.</p>
<hr />
<p>And in much lower light, with ISO cranked up even further:</p>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0406.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/200 sec, ISO 3200. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0406-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0159.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/40 sec, ISO 3200. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0159-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1324.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/40 sec, ISO 3200. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1324-full.jpg">full size</a>.</figcaption>
</figure>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF0401.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/110 sec, ISO 3200. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF0401-full.jpg">full size</a>.</figcaption>
</figure>
<p>The grain is significantly more pronounced here. But — now that I have more experience with the X-H1 (and the benefit of hindsight), I would have used a much lower ISO, given how well the camera does with stabilizing longer shutter speeds.</p>
<p>(I love the last photo in this set, but it isn’t focused as well as it could be. I’ll chalk that up to an error on my part.)</p>
<hr />
<p>The CN Tower makes for a good subject, too. Here it is, glowing purple:</p>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1077.jpg" />
<figcaption>XF50mmF2 at f/2.0, 1/3 sec, ISO 800. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1077-full.jpg">full size</a>.</figcaption>
</figure>
<p>This next photo provides an interesting demonstration of the X-H1’s image stabilization. This was shot handheld (like all of the other photos in this post)<sup id="fnref:long-exposure"><a href="#fn:long-exposure" class="footnote">4</a></sup>, but in this case the shutter was open for <em>four entire seconds</em>:</p>
<figure>
<img src="/images/posts/2018-03-15-fun-with-xh1/DSCF1026.jpg" />
<figcaption>XF50mmF2 at f/2.0, 4 sec, ISO 100. <a href="/images/posts/2018-03-15-fun-with-xh1/DSCF1026-full.jpg">full size</a>.</figcaption>
</figure>
<p>Granted, the CN Tower isn’t a particularly good candidate for long exposure photography (there’s a set of lights near the top of the tower that flash every second or so, and the results of that flashing are unfortunately visible).</p>
<hr />
<p>I also shot some footage to test the low-light video performance.</p>
<p>Here, autofocus is doing it’s thing as I move the camera. There are a few moments where the autofocus system clearly becomes confused<sup id="fnref:low-light-autofocus"><a href="#fn:low-light-autofocus" class="footnote">5</a></sup>, but the image quality is fantastic<sup id="fnref:additional-video-footage"><a href="#fn:additional-video-footage" class="footnote">6</a></sup> and the stabilization is working well.</p>
<figure>
<div class="embed-container">
<iframe src="https://player.vimeo.com/video/258456303" width="640" height="337" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen="" style="margin-bottom: 20px;">
</iframe>
</div>
<figcaption>
XF50mmF2 at f/2.0, ISO 12800, using Eterna film simulation, with “area” autofocus, tracking sensitivity +2, and autofocus speed +1.
</figcaption>
</figure>
<hr />
<p>I hope you enjoyed this photo tour. It was certainly a lot of fun to put together.</p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:battery">
<p>The battery life is not great, but that’s a general problem with mirrorless cameras. I do believe that the battery life gap with DSLRs will rapidly shrink as the scope of on-camera computational photography increases. <a href="#fnref:battery" class="reversefootnote">↩</a></p>
</li>
<li id="fn:learning">
<p>I learned how to shoot on the X100S. In addition to being a great camera, it’s also an exceptional choice for learning photography: it’s small enough to take everywhere, many of the important controls are exposed via physical knobs, and the inability to fiddle with different lenses or focal lengths serves as a helpful constraint. And the newer models in the X100 series are even better. <a href="#fnref:learning" class="reversefootnote">↩</a></p>
</li>
<li id="fn:ibis">
<p>The X-H1 is the first in Fuji’s lineup with <a href="https://fujifilm-x.com/uk/x-stories/x-h1-development-story-7/">image stabilization built-in to the camera body</a>. <a href="#fnref:ibis" class="reversefootnote">↩</a></p>
</li>
<li id="fn:long-exposure">
<p>I wasn’t leaning against anything for support, either. <a href="#fnref:long-exposure" class="reversefootnote">↩</a></p>
</li>
<li id="fn:low-light-autofocus">
<p>In fairness, this is extremely low light, and a situation where you really should be adjusting focus manually. (I wouldn’t be surprised if video autofocus performance improves with future firmware updates.) <a href="#fnref:low-light-autofocus" class="reversefootnote">↩</a></p>
</li>
<li id="fn:additional-video-footage">
<p>I’ve since shot additional footage at lower ISOs, and the image quality is even better than in this clip. <a href="#fnref:additional-video-footage" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Michael Russohttps://mjrusso.com/Practical Tips for Recovering from a Concussion2018-01-12T00:00:00+00:002018-01-12T00:00:00+00:00http://blog.mjrusso.com/concussion-recovery
<p>Many moons ago, I was concussed due to an accident during a sparring drill.</p>
<p>The recovery process took years to get a handle on. The experience was isolating, debilitating, and especially terrifying.</p>
<hr />
<p><img src="/images/posts/2018-01-12-concussion-recovery/concussiversary-2.jpg" alt="" /></p>
<hr />
<p>As a result of my experience, I’m often asked for advice about concussion recovery<sup id="fnref:footnote-not-a-doctor"><a href="#fn:footnote-not-a-doctor" class="footnote">1</a></sup>.</p>
<p>My recommendations are straightforward<sup id="fnref:footnote-straightforward"><a href="#fn:footnote-straightforward" class="footnote">2</a></sup>, and (I believe) were very helpful components of my own recovery.<sup id="fnref:footnote-claims"><a href="#fn:footnote-claims" class="footnote">3</a></sup></p>
<p>But before we get started: <em>Please see a doctor for diagnosis and medical advice.</em> And keep in mind that this is not intended to be an exhaustive guide to recovery.</p>
<hr />
<p><img src="/images/posts/2018-01-12-concussion-recovery/concussiversary-1.jpg" alt="" /></p>
<hr />
<p>If you are concussed and your symptoms continue to persist, you may be one of the unlucky minority that experience <a href="https://en.wikipedia.org/wiki/Post-concussion_syndrome">post-concussion syndrome</a> (PCS).</p>
<p>I hope you never experience PCS. But in the event that you are grappling with the condition, these recommendations may be effective.</p>
<p><em>For starters:</em></p>
<ul>
<li>
<p><strong>Don’t ignore sleep issues.</strong> An inability to sleep will almost certainly aggravate symptoms (and delay recovery). If you’re unable to sleep, make sure to tell your doctor.</p>
</li>
<li>
<p><strong>Take care of your neck.</strong> Neck injuries, no matter how small, can exacerbate symptoms.<sup id="fnref:footnote-whiplash"><a href="#fn:footnote-whiplash" class="footnote">4</a></sup> I personally experienced a marked decrease in headache intensity after seeking chiropractic care.</p>
</li>
</ul>
<p><em>Your frame of mind is really important:</em></p>
<ul>
<li>
<p><strong>Try not to panic.</strong> Your short term memory is completely shot? <em>Panic.</em> You’re convinced that you’ve lost the sum total of your problem solving capabilities? <em>Panic.</em> You can’t concentrate on anything, big or small? <em>Panic.</em> All worth freaking out about. But <em>try</em> not to; it’s not productive. Your superpowers will return, in time. Be patient.</p>
</li>
<li>
<p><strong>Go easy on yourself.</strong> Your energy levels may be <em>really</em> low. You may get tired extremely quickly. You may be overwhelmed by things that used to be second nature. It’s hard not to be frustrated. But this is just part of the process, and you aren’t alone.</p>
</li>
<li>
<p><strong>Take the ups, with the downs.</strong> Nothing’s better than that incredible elation when you have a <em>good</em> day. Relief. It’s all over, right? Chapter closed? Unfortunately that crippling nausea<sup id="fnref:footnote-insert-other-symptom"><a href="#fn:footnote-insert-other-symptom" class="footnote">5</a></sup> a few days from now is going to completely crush you, and all because you set your expectations too high. Recovery is not a straight line pointing up-and-to-the-right.</p>
</li>
</ul>
<ul>
<li><strong>Accept that not everyone will understand.</strong> It’s really hard to empathize; don’t get worked up if people don’t understand or appreciate what you’re going through. That being said, I sincerely hope that your family, closest friends, and employer are able to support you – even if not everyone quite grasps what you are feeling. (I was lucky enough to have an incredible level of support, and am <em>so</em> thankful.)</li>
</ul>
<p><em>Build the following behaviours into your daily routine:</em></p>
<ul>
<li>
<p><strong>Limit screen time.</strong> Aggressively limit your time spent looking at screens. Don’t spend your time budget watching TV, or playing video games. If you must use a computer, don’t look at the screen as you scroll through content. More generally…</p>
</li>
<li>
<p><strong>Limit use of your eyes.</strong> Processing visual information is incredibly taxing. If resting at home, I recommend wearing a sleep mask — even during the day, and even when you are not planning on sleeping.</p>
</li>
<li>
<p><strong>Listen to books and podcasts.</strong> There is now so much quality audiobook and podcast content, and it’s a godsend at avoiding boredom.</p>
</li>
<li>
<p><strong>Meditate.</strong> Meditate at least twice per day (morning and night). Start with five minute sessions, and work up to longer sessions (15-30 minutes each).<sup id="fnref:footnote-meditation"><a href="#fn:footnote-meditation" class="footnote">6</a></sup> I also recommend abdominal breathing – and don’t just restrict the practice to when you’re meditating.</p>
</li>
</ul>
<ul>
<li>
<p><strong>Take long walks.</strong> If you feel up for it, take an hour-long walk each night. (Evenings are much better; avoid the Sun.) Skip this if you’re in a neighborhood that’s busy or otherwise loud (in an auditory <em>or</em> visual sense).</p>
</li>
<li>
<p><strong>Ride a stationary bike.</strong> Break a sweat, but do not over-exert yourself when riding. Use a heart rate monitor, and watch it to ensure that you don’t climb above your target range.</p>
</li>
<li>
<p><strong>Tune your diet.</strong> Avoid foods that cause inflammation. For example – blueberries: good; carbs: bad. How you eat affects how you feel, and I’ve personally found that my body is more sensitive now. (And be sure to cut out the alcohol, entirely.)</p>
</li>
<li>
<p><strong>Find an artistic release.</strong> For me, that creative outlet is screenwriting and photography.<sup id="fnref:footnote-photography"><a href="#fn:footnote-photography" class="footnote">7</a></sup> You may prefer to draw, paint, write music, play an instrument…. Experiment, and pick something you enjoy. Do it consistently (but not excessively; your overall energy levels may be low). You’ll find that making art is therapeutic.</p>
</li>
</ul>
<p>I strongly believe that building these behaviours and practices into my daily routine played an important role in my recovery. Your mileage may vary, but I’d love to hear if this helps you, too.</p>
<hr />
<p><img src="/images/posts/2018-01-12-concussion-recovery/concussiversary-0.jpg" alt="" /></p>
<hr />
<p>Finally –</p>
<p>Accept that this experience <em>will</em> change you.</p>
<p>But at least there’s <a href="https://medium.com/@byKanika/how-an-art-exhibit-transformed-into-something-greater-f9c47c35045e">more than one way to put the pieces back together</a>.</p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:footnote-not-a-doctor">
<p>Obligatory disclaimer: not a doctor. <a href="#fnref:footnote-not-a-doctor" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-straightforward">
<p>Conceptually, at least. I’m not claiming that implementation will be easy. <a href="#fnref:footnote-straightforward" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-claims">
<p>Everything on this list was worked into my daily routine, over a long period of trial-and-error. It was this process that gives me the confidence to recommend these practices. Usual caveats apply: the sample size here is one, my trial-and-error exploration is a far cry from a proper medical study, and it’s not outside the realm of possibility that nothing on this list actually made a difference. Et cetera. <a href="#fnref:footnote-claims" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-whiplash">
<p>I didn’t initially realize that the neck pain caused by the accident needed special treatment (and, in hindsight, was almost certainly whiplash). <a href="#fnref:footnote-whiplash" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-insert-other-symptom">
<p>Or, <em>«insert other symptom here»</em>. <a href="#fnref:footnote-insert-other-symptom" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-meditation">
<p>There is a perplexing amount of venture money flowing into meditation startups; you don’t need an app or a subscription. If you’re new to this, I recommend starting with the book “Mindfulness for Beginners”, by Jon Kabat-Zinn. (Listen to the audio version.) <a href="#fnref:footnote-meditation" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-photography">
<p>You may be wondering about the photographs sprinkled throughout this post. I shot them, over the years, to celebrate my “concussiversary”. <a href="#fnref:footnote-photography" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Michael Russohttps://mjrusso.com/Making a Roo2017-03-15T00:00:00+00:002017-03-15T00:00:00+00:00http://blog.mjrusso.com/making-a-roo
<p>My wife Iliana and I have some exciting news. And to share this news with our family and friends, we created a short film (because, <em>why not</em>).</p>
<p>For posterity (and because this really was so much fun), I’m going to share some notes about the filmmaking process<sup id="fnref:footnote-newbie"><a href="#fn:footnote-newbie" class="footnote">1</a></sup>.</p>
<p>But first, let’s <a href="http://vimeo.com/mjrusso/making-a-roo">watch the film</a>:</p>
<div class="embed-container">
<iframe src="https://player.vimeo.com/video/206144583" width="640" height="360" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen="" style="margin-bottom: 20px;"></iframe>
</div>
<hr />
<h4 id="development">Development</h4>
<p>The script (more cheeky than precise) does differ from the final product. But it did provide enough of a foundation to work from:</p>
<div id="making-a-roo-screenplay">
<div class="screenplay us-letter dpi100">
<div class="page">
</div>
</div>
</div>
<h4 id="pre-production">Pre-Production</h4>
<p>We interviewed a number of potential actors.</p>
<p>For example, here’s <strong>SMALL KIWI</strong>, acing the audition.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/audition.jpg" alt="" /></p>
<p>Isn’t it fabulous that the actors we hired have such incredible chemistry together?</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/family.jpg" alt="" /></p>
<p>And they have super powers, too. Here, their remarkable force of attraction is preventing <strong>MASSIVE WOODEN OBSTACLE</strong> from succumbing to pesky gravity.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/range.jpg" alt="" /></p>
<p>Yes, this power is even more impressive when viewed from above.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/gravity.jpg" alt="" /></p>
<p><em>(OK, it’s just tape.<sup id="fnref:footnote-just-tape"><a href="#fn:footnote-just-tape" class="footnote">2</a></sup>)</em></p>
<p><img src="/images/posts/2017-03-15-making-a-roo/antigravity.jpg" alt="" /></p>
<hr />
<p>At a frame rate of 24 frames per second, displaying one photo per frame, and with a target run-time of roughly one minute, I’d need to take 1,440 photos.</p>
<p>That’s a lot of shots.<sup id="fnref:footnote-a-lot-of-shots"><a href="#fn:footnote-a-lot-of-shots" class="footnote">3</a></sup></p>
<p>The next order of business was breaking the script down into segments — and plannng out just how incredibly small the movement increments needed to be between frames.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/segments0.jpg" alt="" /></p>
<h4 id="production">Production</h4>
<p>Shot on the venerable Fujifilm X100S.<sup id="fnref:footnote-x100s"><a href="#fn:footnote-x100s" class="footnote">4</a></sup></p>
<p><strong>Production Mistake #1:</strong> “Fabricating” a tripod using a ball head, a few books, and some masking tape…</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/tripod.jpg" alt="" /></p>
<p>This was <em>almost</em> perfectly stable. Unfortunately, almost isn’t good enough for stop motion.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/camera.jpg" alt="" /></p>
<p>Due to the manual cable release, or infrequent interactions with the camera controls, or perhaps due to the alignment of the Sun, Earth, and Moon, the position of the camera occasionally moved, <em>ever so slightly</em>, in between some shots.</p>
<p><em>(Don’t worry, I’ll fix it in post.)</em></p>
<hr />
<p>Iliana did such a great job with the illustrations.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/illustration.jpg" alt="" /></p>
<p>Thankfully I used a more traditional tripod configuration when photographing the drawing sequence.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/tripod2.jpg" alt="" /></p>
<hr />
<p><strong>Production Mistake #2:</strong> The Sun, as primary lighting source.</p>
<p>It was an overcast day, the blinds were down, and the camera was in a fully manual configuration. But, wow — the exposures<sup id="fnref:footnote-luminance"><a href="#fn:footnote-luminance" class="footnote">5</a></sup> between shots tended to vary, sometimes quite significantly.</p>
<p><em>(Guess I’m fixing this in post, too.)</em></p>
<p>Next time, I’m shooting at night. Lighting the scenes will be a lot more work, but at least the sun isn’t going to poke through and wreak havoc.</p>
<h4 id="post-production">Post-Production</h4>
<p>Turns out that editing the photos to match exposures between shots was a nightmare.</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/exposure.jpg" alt="" /></p>
<p>Countless hours of manually tweaking hundreds of photos of photos yielded improved (but not perfect) results.</p>
<p>I’m <a href="http://www.loopinsight.com/2014/06/27/apple-stops-development-of-aperture/">still</a> an Aperture user. Out of desperation I even tried Lightroom’s “Match Total Exposures” feature, but that didn’t help (<a href="https://forums.adobe.com/message/6331968#6331968">here’s why</a>).</p>
<p>One day I’d like to write some software that automates the matching process.</p>
<hr />
<p>We had been planning on <em>experimenting</em> with the use of a “homemade”, old-school film aesthetic.</p>
<p>However, we didn’t end up having much choice in the matter (given the photography mistakes, reluctance to spend another entire day shooting, and editing inexperience, too…).</p>
<p>So, to make the mistakes look like they were deliberate (I mean, <em>“yes, really, we actually meant to make it look like that!”</em>), we used Final Cut Pro X’s “Aged Film” effect, which helped to paper over the issues. By introducing a slight jitter and brightness variance, we could compensate for the unstable tripod and the differing exposures between shots. (The dust, scratches, and film grain help to sell it, too.)</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/aged-film.jpg" alt="" /></p>
<p>All things considered, it didn’t turn out <em>that</em> bad. Would have preferred to have been able to dictate the terms, though.</p>
<hr />
<p>This was my first time using Final Cut Pro X, and I found it intuitive.<sup id="fnref:footnote-final-cut-pro"><a href="#fn:footnote-final-cut-pro" class="footnote">6</a></sup></p>
<p>The magnetic timeline seems to be well suited for stop motion (although I don’t have experience with any other <a href="https://en.wikipedia.org/wiki/Non-linear_editing_system">NLEs</a> to compare).</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/fcpx.jpg" alt="" /></p>
<p>Most importantly, adjusting the timing of each shot (or groups of shots) was a breeze.<sup id="fnref:footnote-shot-timing"><a href="#fn:footnote-shot-timing" class="footnote">7</a></sup></p>
<hr />
<p>And the sound!</p>
<p><img src="/images/posts/2017-03-15-making-a-roo/ableton.jpg" alt="" /></p>
<p>Huge thanks to my brother <a href="https://twitter.com/_dvrusso">Daniel</a> for creating the music.</p>
<h4 id="distribution">Distribution</h4>
<p>Cheers, Vimeo.<sup id="fnref:footnote-thanks"><a href="#fn:footnote-thanks" class="footnote">8</a></sup></p>
<hr />
<p><img src="/images/posts/2017-03-15-making-a-roo/making-a-roo.jpg" alt="" /></p>
<hr />
<p><strong><em>P.S.:</em></strong> Here’s what a scurrying kiwi <em>actually</em> looks like:</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Wrapping up my week like dis. <a href="https://t.co/AErelI5Jah">pic.twitter.com/AErelI5Jah</a></p>— Jan Schaumann (@jschauma) <a href="https://twitter.com/jschauma/status/840389919352717312">March 11, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Looks just about the same, right?</p>
<p>Nailed it. 😉</p>
<hr />
<link rel="stylesheet" href="/stylesheets/fountain.css" />
<script type="text/javascript" src="/javascripts/fountain.js"></script>
<script type="text/javascript">
jQuery.get("/assets/making-a-roo.fountain", function(data) {
var screenplay = fountain.parse(data);
$("#making-a-roo-screenplay .screenplay .page").html(screenplay.html.script);
});
</script>
<div class="footnotes">
<ol>
<li id="fn:footnote-newbie">
<p>The incredibly amateur filmmaking process, that is. (Did we mention that we’ve never done this before?) <a href="#fnref:footnote-newbie" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-just-tape">
<p>Does this count as a practical effect? <a href="#fnref:footnote-just-tape" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-a-lot-of-shots">
<p>In the end, I took just north of 600 photos — and a subset of those shots made it into the final cut. Timing varied, too (more on that later). <a href="#fnref:footnote-a-lot-of-shots" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-x100s">
<p>The X100S’s native resolution is 4896x3264px. At an output resolution of 1080p (1920x1080px) for the film, that left ample room to zoom/crop/re-frame in post, without sacrificing quality. <a href="#fnref:footnote-x100s" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-luminance">
<p>The technical term I might mean to be using here is <a href="https://en.wikipedia.org/wiki/Luminance">luminance</a>. <a href="#fnref:footnote-luminance" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-final-cut-pro">
<p>For the most part, despite clearly being designed for expert users. (For example: crucial shortcuts that don’t use modifier keys, and limited options in contextual menus.) In other news, the app did hang frequently enough to drive me insane (no beachball, no crash, just <em>frozen</em>). And, even worse (especially for a new user) it had a tendency to completely forget its undo/redo history. Yes, as stressful as it sounds. <a href="#fnref:footnote-final-cut-pro" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-shot-timing">
<p>In the chase sequence, each shot is displayed for a single frame. In the illustration sequence, each shot is displayed for two frames. And in the drop sequence, each shot is displayed for four frames. There are some exceptions, though (for example, during transitions). <a href="#fnref:footnote-shot-timing" class="reversefootnote">↩</a></p>
</li>
<li id="fn:footnote-thanks">
<p>And thank you for reading (and watching, too)! <a href="#fnref:footnote-thanks" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Michael Russohttps://mjrusso.com/Mountain, a Tool for Screenwriting with Fountain2014-08-15T00:00:00+00:002014-08-15T00:00:00+00:00http://blog.mjrusso.com/mountain-screenwriting-with-fountain
<p>Eight months ago, I started writing a screenplay. I’ve been working on it just about every single day since.</p>
<p>I’ll save further details about the project for a later post, but, in short, I’ve found screenwriting to be a fantastic creative pursuit. The writing experience feels like just the right mix of exhileration and frustration. (Exhileration when that next challenging problem is finally solved. Frustration otherwise.)</p>
<hr />
<p><a href="http://fountain.io">Fountain</a> is a plain text markup language for screenwriting.</p>
<p>The most important feature of Fountain is that it is just plain text. You’re not locked into a particular piece of software for writing the screenplay. You can, if you wish, continue to use the software that you already use for editing and working with plain text.</p>
<p>In my case, I’m primarily using <a href="https://github.com/rnkn/fountain-mode/">Fountain Mode for Emacs</a> and <a href="http://slugline.co/">Slugline</a>. (I switch back and forth between both editors fairly frequently, as each has their own unique strengths.)</p>
<hr />
<p>Writing with Fountain is significantly easier when your entire screenplay is contained within a single file, and generally it seems that that’s how everyone does it. (There is an exception – <a href="http://johnaugust.com/2012/writing-screenplays-with-scrivener-and-ia-writer">Scrivener’s Binder and Compile features</a> – but then you’re locked into using a monolithic tool as a crucial part of your writing workflow.)</p>
<p>Over the course of writing, I’ve run into several workflow issues that are all related to a single root cause: <em>file organization</em>. In particular, there are three specific reasons why I highly prefer to split my Fountain screenplay across multiple files:</p>
<ol>
<li>
<p><em>Source control</em>: I version my screenplay with <a href="http://git-scm.com/">Git</a>. There are many advantages to using source control, but some of the benefits are lost when your repository consists of a single, multi-thousand line document. For example, it’s incredibly time consuming to go back in time and see all of the previous versions of a specific scene or sequence (yet trivial, if each scene or sequence lived in a separate file). I compound this issue by abusing source control to store supplementary information in the history; when I’m working on a scene, I’ll often commit a bunch of text (for safe keeping) and then immediately delete the text and commit again. I do this to store alternate versions of scenes, extra lines of dialog (sometimes even entire exchanges) that I wrote and liked but ended up abandoning in favor of some alternative, additional ideas for how the scene could play out, etc.</p>
</li>
<li>
<p><em>Outlining</em>: Fountain has support for <a href="http://fountain.io/syntax#section-sections">sections</a>, which can be used to document the structure of the story. I use these section elements for outlining (in lieu of index cards or an alternate system). For this screenplay, I outlined initially at a very macro level and have been frequently zooming in to fill in more specific details. It’s very cumbersome to work on the outline when your entire screenplay is part of the same document, though: if you move anything around, for example, you have to also move all of the actual scene content. More generally, the scene content gets in the way. The ideal solution is to have a separate file for the outline that can reference the other files that store the actual scenes.</p>
</li>
<li>
<p><em>Mobile</em>: I want to experiment with writing on iPad (and iPhone, too!), which seems particularly feasible because <a href="http://omz-software.com/editorial/">Editorial for iOS</a> is such a fantastic app. However, as great as Editorial is, it’s still extremely difficult to effectively navigate a very long document. (And there’s also the issue of creating conflicting copies in Dropbox when the screenplay syncs. The longer the document, and the more scenes it contains, the more likely it will conflict with the version I’m authoring on my other devices – and the more painful it will be to actually fix the conflicts, when they inevitably arise.)</p>
</li>
</ol>
<hr />
<p>Another advantage of plain text is that it’s relatively straightforward to build additional <a href="http://setpixel.com/writing/writing-a-screenplay-in-fountain/">“creative tools”</a> that operate on the plain text Fountain corpus.</p>
<p>Today, I’m releasing <a href="https://github.com/mjrusso/mountain">Mountain</a>, an open source Python library that solves all of the issues enumerated above by making it easy to split a single (Fountain-formatted) screenplay across multiple physical files.</p>
<p>Mountain accomplishes the same goals as Scrivener’s Binder and Compile features, but in such a way that <em>embraces the plain text nature of Fountain</em>, and, crucially, <em>works with any writing software that you’re already using</em>.</p>
<p><a href="https://github.com/mjrusso/mountain">Try it out</a> and let me know what you think. (Full usage details, including a tutorial, are available in the project’s README.)</p>
Michael Russohttps://mjrusso.com/The Yoga Mat Hack2013-05-07T00:00:00+00:002013-05-07T00:00:00+00:00http://blog.mjrusso.com/the-yoga-mat-hack
<p>One of my favorite ways to find inspiration is to watch as other creative
people work.</p>
<p>And one of the best places to find creativity is in the presence of significant
constraints.</p>
<p>Most indie filmmakers work with a prominent constraint – limited capital – to
create an artistic work. The input to the filmmaking process is passion and a
small budget, and the output is (at least ideally) a compelling film that is
enjoyable for the audience to experience.</p>
<p>After thoroughly enjoying
<a href="http://www.imdb.com/title/tt1549572/">Another Earth</a>, I came across the
following clip while researching how the film was made:</p>
<div class="flex-video">
<iframe width="640" height="360" src="https://www.youtube.com/embed/1rUwqBgFjSs?start=269" frameborder="0" allowfullscreen="">
</iframe>
</div>
<p>Brit Marling explains how they shot the scene that depicts Rhoda, the character
she plays, being released from prison:</p>
<blockquote>
<p>This was the kind of movie where almost everything was for free.</p>
</blockquote>
<blockquote>
<p>The jail, the scene where Rhoda was released from prison, we couldn’t afford
to shoot the exterior of a prison.</p>
</blockquote>
<blockquote>
<p>So Mike [Cahill] and I just drove around Connecticut in his mom’s car until
we found a prison that we could get close enough to the front entrance, and
then Mike parked across the street, rolled down the driver side window and
was holding the camera, and I put the costume on in the passenger seat.</p>
</blockquote>
<blockquote>
<p>I took a yoga mat out of the back of the car, and I walked into the prison
and I was like “Hi, I’m here to teach yoga”, and they were like “What?” and I
was like “Yeah!”, as if I do it all the time. And they went to go and figure
out that problem, and I dropped the yoga mat and turned around and walked out
in costume, and Mike shot it.</p>
</blockquote>
<blockquote>
<p>And that is the release from prison shot in the film.</p>
</blockquote>
<blockquote>
<p>That’s a lot of how this film was made.</p>
</blockquote>
<p>Floored by the yoga mat hack, I was obliged to learn more:</p>
<blockquote>
<p>She [Brit] moved to Los Angeles to act and, after spending a couple of years
exploring the movie industry and being offered roles as “the cute blonde in
horror movies”, she taught herself to write, reasoning that the best way to
get decent parts was to write them, herself.</p>
</blockquote>
<blockquote>
<p><strong>Brit Marling’s <a href="http://www.imdb.com/name/nm1779870/bio">IMDB bio</a></strong></p>
</blockquote>
<p>There is undoubtedly a constraint on the number of <em>interesting</em> roles
available to a budding actress. Especially given the number of budding
actresses in Los Angeles.</p>
<p>But writing your own parts changes the playing field.</p>
<p>The bio continues:</p>
<blockquote>
<p>She worked on two movies, simultaneously - one in the mornings, one in the
afternoons - and eventually both Another Earth (2011) and Sound of My Voice
(2011) premiered at the Sundance Film Festival in 2011.</p>
</blockquote>
<p><em>(Perseverance doesn’t hurt, either.)</em></p>
<hr />
<p>This link between creativity and constraints is fascinating.</p>
<p>Igor Stravinsky famously wrote in
<a href="http://www.hup.harvard.edu/catalog.php?isbn=9780674678569&content=book">“Poetics of Music in the Form of Six Lessons”</a>:</p>
<blockquote>
<p>My freedom thus consists in my moving about within the narrow frame that I
have assigned to myself for each one of my undertakings. I shall go even
further: my freedom will be so much the greater and more meaningful the more
narrowly I limit my field of action and the more I surround myself with
obstacles. Whatever diminishes constraint diminishes strength. The more
constraints one imposes, the more one frees oneself of the claims that
shackle the spirit.</p>
</blockquote>
<p>(i.e., treat constraints as
<a href="http://gettingreal.37signals.com/ch03_Embrace_Constraints.php">advantages in disguise</a>.)</p>
<p>Scott Berkun took this even further,
<a href="http://scottberkun.com/2008/do-constraints-help-creative-thinking/">questioning</a>
if it is indeed <em>possible</em> to be creative in the absence of constraints.</p>
<hr />
<p>Certain types of creative work do not require committing to a finished product.
(Consider the difference in finality between releasing a movie, and releasing
Web-based software that can be updated instantly without user intervention or
any discernable disruption.)</p>
<p>In these cases, <strong>rapid iteration</strong> is a highly effective way to work.</p>
<p>Iteration-centric processes are effective because frequent checkpoints afford
an opportunity for constant learning and course correction (think: <em>“ready,
fire, aim”</em>).</p>
<p>However, there is a hidden benefit.</p>
<p>Iteration is about delivering small (i.e., <em>highly constrained</em>) units of
change. This constraint imposes a significant challenge: determining the
smallest units of change that will add the most overall value. And the
accompanying mindset shift, from <em>“what are all the possible things that can be
done next?”</em>, to <em>“what are the smallest changes that will make the most
difference?”</em>, is profound.</p>
<p>When you have the opportunity to work iteratively, take it.</p>
<hr />
<p>A few anecdotes are hardly proof.</p>
<p>But at least some data suggests that there are benefits to be gained by applying
constraints liberally – even, counter-intuitively, constraints that don’t
necessarily <em>need</em> to be there – as a means of channeling creativity and thus
increasing quality.</p>
<p>Constraints, it seems, are a lot like <strong>creative fuel</strong>.</p>
Michael Russohttps://mjrusso.com/Redis, from the Ground Up2010-10-17T00:00:00+00:002010-10-17T00:00:00+00:00http://blog.mjrusso.com/redis-from-the-ground-up
<p><img src="/images/posts/2010-10-17-redis-from-the-ground-up/redis.png" alt="Redis Logo" /></p>
<p><a href="http://redis.io/">Redis</a> (“<strong><em>RE</em></strong>mote <strong><em>DI</em></strong>ctionary <strong><em>S</em></strong>ervice”) is an open-source (<a href="http://www.opensource.org/licenses/bsd-license.php">BSD-licensed</a>) database server.</p>
<div id="generated-toc" class="generate_from_h3"></div>
<h3 id="history">History</h3>
<p>The Redis project was started in early 2009 by an Italian developer named <a href="http://antirez.com/">Salvatore Sanfilippo</a>. Redis was initially written to improve the performance of <a href="http://lloogg.com/">LLOOGG</a>, a real-time web analytics product out of <a href="http://merzia.com/">Salvatore’s startup</a>.</p>
<p>By June of 2009, Redis was stable enough, and had enough of a base feature set, to serve production traffic at LLOOGG. On June 19, 2009, an important milestone was hit: <a href="http://groups.google.com/group/redis-db/browse_thread/thread/0c706a43bc78b0e5/17c21c48642e4936?#17c21c48642e4936">Salvatore deployed Redis to LLOOGG’s production environment</a> and retired the MySQL installation.</p>
<p>Over the next months, Redis rapidly grew in popularity. Salvatore fostered a great community, added features at a very rapid pace, and dealt with any and all reports of database corruption, instability, etc. with the utmost severity.</p>
<p>In March of 2010 <a href="http://blogs.vmware.com/console/2010/03/vmware-hires-key-developer-for-redis.html">VMWare hired Salvatore</a> to work full-time on Redis. (Redis itself remains BSD licensed.) Shortly thereafter, <a href="http://groups.google.com/group/redis-db/browse_thread/thread/e725706cba45f327">VMWare hired Pieter Noordhuis</a>, a key Redis contributor, to give the project an additional momentum boost.</p>
<p>The rest, as they say, is history.</p>
<h3 id="data-structure-server">Data Structure Server</h3>
<p>The most apt description of Redis is that it is a “data structure server”. This is a very natural label for the database, because Redis natively supports many of the foundational data types of computer science, and provides a rich set of familiar primitives for manipulating these types.</p>
<p>The supported data types are:</p>
<ul>
<li>Strings</li>
<li>Lists</li>
<li>Sets</li>
<li>Sorted Sets</li>
<li>Hashes</li>
</ul>
<h3 id="key-advantages">Key Advantages</h3>
<h4 id="performance">Performance</h4>
<p>Redis can perform <a href="http://code.google.com/p/redis/wiki/Benchmarks">>100k+ SETs per second, and >80k+ GETs per second</a>.</p>
<p>The <code class="highlighter-rouge">redis-benchmark</code> utility <a href="http://github.com/antirez/redis/blob/master/src/redis-benchmark.c">included with the project</a> can be used to reproduce these findings. (More on benchmarks <a href="http://antirez.com/post/redis-memcached-benchmark.html">here</a> and <a href="http://antirez.com/post/update-on-memcached-redis-benchmark.html">here</a>.)</p>
<h4 id="atomicity">Atomicity</h4>
<p>Every operation that Redis exposes via the available command primitives is atomic. (And, as we’ll see later, there are ways to combine multiple primitive commands into larger atomic units.) This makes application-level code easier to build and reason about.</p>
<p>Atomic guarantees are a direct consequence of Redis’ single-threaded core; as a result, Redis’ internals are simpler (no complex locking and synchronization code complicating the codebase, introducing bugs, and burning up CPU cycles).</p>
<h4 id="foundational-data-types">Foundational Data Types</h4>
<p>The data types that Redis offers are foundational. All computer scientists are very familiar with these data types and have already used them to solve countless problems. <em>Lists, sets, etc. are more fundamental to computer scientists than relational database tables, columns, and rows.</em></p>
<p>The documentation for all Redis commands includes complexity measurements, in <a href="http://en.wikipedia.org/wiki/Big_O_notation">big O notation</a>. This makes it very straightforward for computer scientists and software engineers to visualize, reason about, optimize, and understand the performance of Redis queries. Is it this straightforward to understand query performance in a relational database?</p>
<h4 id="polyglot-persistence">Polyglot Persistence</h4>
<p>This is not specifically an advantage inherent to Redis, but is worth noting. Using Redis does not require a commitment to use it exclusively. Use Redis to solve problems that can be naturally modelled using its primitives, and embrace <a href="http://blog.heroku.com/archives/2010/7/20/nosql/">polyglot</a> <a href="http://nosql.mypopescu.com/post/836018286/heroku-encourages-polyglot-persistence">persistence</a> for everything else.</p>
<h4 id="client-protocol">Client Protocol</h4>
<p>The <a href="http://code.google.com/p/redis/wiki/ProtocolSpecification">client protocol</a> is very straightforward. This makes it simple to build <a href="http://code.google.com/p/redis/#Supported_languages">client libraries</a>.</p>
<h4 id="getting-started">Getting Started</h4>
<p>It is very easy to get started. Redis has no external dependencies, and getting up and running only involves the following steps:</p>
<ol>
<li>Clone the <a href="http://github.com/antirez/redis">git repository</a> or unpack the <a href="http://github.com/antirez/redis/downloads">tarball</a></li>
<li>Run <code class="highlighter-rouge">make</code></li>
<li>Run <code class="highlighter-rouge">./src/redis-server</code></li>
</ol>
<p>Then, from another terminal, start the interactive command line interface by running <code class="highlighter-rouge">./src/redis-cli</code>.</p>
<p>Feel free to issue a few commands:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">PING</span>
<span class="n">PONG</span>
<span class="n">redis</span><span class="o">></span> <span class="n">INFO</span>
<span class="n">redis_version</span><span class="p">:</span><span class="mf">2.1.1</span>
<span class="o">...</span></code></pre></figure>
<p><em>(Alternatively, to try Redis from your browser, visit <a href="http://try.redis-db.com/">try.redis-db.com</a>.)</em></p>
<h3 id="key-disadvantages">Key Disadvantages</h3>
<p>Redis requires that the whole dataset be loaded into main memory at all times. (Redis Virtual Memory, which we’ll discuss later, relaxes this requirement, but still needs all keys to always be in memory.). Guaranteed in-memory access to most of the dataset is Redis’ main performance driver – and is also responsible for creating its main limitations.</p>
<h4 id="ram">RAM</h4>
<p>RAM is the gold of cloud computing. (Cloud servers are primarily priced based on the amount of available RAM. By comparison, disk and CPU are cheap.)</p>
<p>The amount of RAM that Redis needs is proportional to the size of the dataset. Large datasets in Redis are going to be fast, but expensive.</p>
<h4 id="persistence">Persistence</h4>
<p>Redis persistence is highly configurable but the implementation makes extremely heavy use of I/O resources. Furthermore, most save operations require additional memory to complete successfully, and, in some cases, asynchronous saves can block the server for lengthy periods of time. (These points are discussed in more detail, below; see the <strong>Persistence</strong> section.)</p>
<h4 id="memory-bloat">Memory Bloat</h4>
<p>Redis’ internal design typically trades off memory for speed. For some workloads, there can be an order of magnitude difference between the raw number of bytes handed off to Redis to store, and the amount of memory that Redis uses.</p>
<h3 id="diving-in">Diving In</h3>
<h4 id="string-keys">String Keys</h4>
<p>Regardless of the data type, the data is always identified by a key, and the key is always a string.</p>
<p>For example, using the string data type:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">SET</span> <span class="n">foo</span> <span class="n">bar</span>
<span class="n">OK</span>
<span class="n">redis</span><span class="o">></span> <span class="n">GET</span> <span class="n">foo</span>
<span class="s">"bar"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">GET</span> <span class="n">dne</span>
<span class="p">(</span><span class="n">nil</span><span class="p">)</span></code></pre></figure>
<h4 id="expiry">Expiry</h4>
<p>Keys can be marked for expiry. For example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">EXPIRE</span> <span class="n">foo</span> <span class="mi">2</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span></code></pre></figure>
<p>After waiting for 2 seconds:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">GET</span> <span class="n">foo</span>
<span class="p">(</span><span class="n">nil</span><span class="p">)</span></code></pre></figure>
<h4 id="sidenote-memcached">Sidenote: memcached?</h4>
<p>So far, this looks quite similar to <a href="http://memcached.org/">memcached</a> (GET/SET API, in-memory storage, etc.). However, there are a few important things to note:</p>
<ol>
<li>Redis supports replication out of the box. Any sort of topology is possible, so you can create replication trees.</li>
<li>Redis supports persistence, so you don’t lose everything that’s in memory when the server restarts.</li>
<li>Redis supports a rich set of data types (far beyond memcached’s simple key-value-pairs).</li>
</ol>
<p>Each of these points will be addressed in more detail, below.</p>
<h4 id="replication">Replication</h4>
<p>Redis’ replication capabilities are powerful yet straightforward.</p>
<p>A master can have any number of slaves, and each slave can have any number of their own slaves, and so on and so forth. Any topology is possible.</p>
<p>To point a slave to a specific master, issue the <code class="highlighter-rouge">SLAVEOF</code> command on the slave. Slaves will block until the initial synchronization with the master is complete.</p>
<p>This initial synchronization process consists of the master asynchronously snapshotting the current state of the database, then transferring the snapshot to the slave, and then subsequently streaming all commands received after initiating the snapshot.</p>
<h4 id="persistence-1">Persistence</h4>
<p>Redis has configurable persistence settings, enabling durability to be tweaked depending on the problem domain.</p>
<h5 id="options">Options</h5>
<p><em>If durability is not important:</em></p>
<p>Redis can be configured in “snapshotting mode”. In this mode, Redis saves a binary dump of the contents of the database every <code class="highlighter-rouge">x</code> seconds or every <code class="highlighter-rouge">y</code> operations. If one of these criteria are met, Redis forks the process. The child process writes the dump file to disk while the master continues to service requests.</p>
<p>This procedure can be memory-efficient due to the way that <a href="http://en.wikipedia.org/wiki/Copy-on-write">Copy-On-Write</a> works when forking. (Here, a snapshot of the database is saved as it existed exactly at the time of forking; extra memory is required only to store the keys that change during the snapshot procedure. If every key changes in value over the course of the snapshot, then roughly 2x the amount of memory used by Redis before the save is required to complete the save operation. This is the upper bound on the memory usage required for saving.)</p>
<p>Of course, in this mode, any data that is not written in the snapshot is immediately lost if the server is killed.</p>
<p><em>If durability is important:</em></p>
<p>Redis can be configured to use an Append-Only File (<em>AOF</em>). Here, every command is written to a file. To recover from a crash or other server restart, the append-only file is replayed. There are three modes:</p>
<ul>
<li><code class="highlighter-rouge">fsync()</code> on every new command</li>
<li><code class="highlighter-rouge">fsync()</code> every second</li>
<li>Let the OS decide when to <code class="highlighter-rouge">fysnc()</code></li>
</ul>
<p>Using the <code class="highlighter-rouge">BGREWRITEAOF</code> command, Redis will update the snapshot and re-write the Append-Only File to shorten it. Like snapshotting, this is done asynchronously, in the background.</p>
<p><em>More advanced configurations:</em></p>
<p>Persistence can be turned off completely. This is useful in a number of scenarios.</p>
<p>For example, if performance is very critical and your application demands extremely tight control over RAM usage, the following configuration is possible:</p>
<ul>
<li>One master, persistence off, and</li>
<li>One slave, persistence off, and</li>
<li>Periodic <strong>synchronous saves</strong>, issues against the <strong>slave</strong> only</li>
</ul>
<p>The advantage of this set-up is that it requires no extra memory to complete a save, regardless of the number and frequency of writes. In this way, you are trading off durability for extremely tight control over memory usage.</p>
<p>No extra memory is required to complete the save because the <code class="highlighter-rouge">SAVE</code> command performs a synchronous save operation, thereby blocking the server that the command is issued against until the saving process completes. (Asynchronous saves, as discussed above, require extra memory proportional to the number of writes performed during the save.)</p>
<p>Other variations on this theme are possible, for example AOF can be enabled on the slave only while persistence remains off on the master.</p>
<h5 id="binary-dumps-and-in-memory-representation">Binary Dumps and In-Memory Representation</h5>
<p>The binary dumps (i.e. those produced by the snapshot operations) are stored in a very efficient manner on disk.</p>
<p>Once a binary dump is loaded, Redis will use several factors more memory than the on-disk representation requires. The exact factor increase depends primarily on the data types that are in use. For example, Sorted Sets use significantly more memory than Sets, even though both data structures require similar amounts of space when serialized to disk.</p>
<p>This is expected behaviour, given that Redis optimizes heavily both read and write performance.</p>
<p>Note that optimizations are continually being made to reduce the amount of memory required to represent each of the data types in memory.</p>
<h5 id="problems">Problems</h5>
<p>Redis exhibits the following issues with persistence:</p>
<ul>
<li>
<p>Most save operations require additional memory to complete successfully (as previously discussed). Depending on the size of the dataset, the frequency of writes, and the amount of RAM you are comfortable reserving, this may or may not be an issue.</p>
</li>
<li>
<p>Redis persistence requires extremely heavy I/O usage. This is discussed in detail <a href="http://blog.kennejima.com/post/1226487020/thoughts-on-redis">here</a>. Also see <a href="http://antirez.com/post/a-few-key-problems-in-redis-persistence.html">Salvatore’s response</a>.</p>
</li>
<li>
<p>In some cases, asynchronous saves can block the server for lengthy periods of time. See <a href="http://groups.google.com/group/redis-db/browse_thread/thread/19f2c7e7dacfc1c9/268615ff111576cd">this post on the mailing list</a> for an interesting discussion.</p>
</li>
</ul>
<p>Although the issues with Redis persistence are hard problems to solve, the issues are beginning to be discussed at length. We should continue to see improvements in this area.</p>
<h4 id="expiry-1">Expiry</h4>
<p>The <code class="highlighter-rouge">EXPIRE</code> command enables a timeout to be set for a specified key. Once the timeout is reached, the server can delete the key. We call a key <code class="highlighter-rouge">volatile</code> if it is set to expire.</p>
<p>Note that expiry semantics changed <strong>very significantly</strong> with Redis 2.1.3. It is important to be aware of the differences, especially given the wide install base of Redis 2.0.</p>
<h5 id="redis-versions-prior-to-213">Redis Versions Prior to 2.1.3</h5>
<p>The initial design of Redis’ expiry mechanism is extremely concerned with consistency.</p>
<p>Here, if the value of a volatile key is modified in any way, then the key becomes non-volatile and will not expire unless explicitly <code class="highlighter-rouge">EXPIRE</code>‘d again.</p>
<p>Furthermore, the following restrictions apply:</p>
<ul>
<li>If a key is volatile, the EXPIRE command cannot be re-issued to change the expiry time</li>
<li>A write operation against a volatile key will <strong>destroy the key first <em>and then</em> perform the write</strong></li>
</ul>
<p>For example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">SET</span> <span class="n">foo</span> <span class="n">bar</span>
<span class="n">OK</span>
<span class="n">redis</span><span class="o">></span> <span class="n">EXPIRE</span> <span class="n">foo</span> <span class="mi">100000</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">GET</span> <span class="n">foo</span>
<span class="s">"bar"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">APPEND</span> <span class="n">foo</span> <span class="mi">123</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">3</span>
<span class="n">redis</span><span class="o">></span> <span class="n">GET</span> <span class="n">foo</span>
<span class="s">"123"</span></code></pre></figure>
<p>In order to enforce consistency requirements, this is expected behaviour. (To be strongly consistent, if <code class="highlighter-rouge">n</code> servers receive the same list of commands in the same sequence, they must always end with the same dataset in memory. However, with replication over unreliable network links, and without the restrictions detailed above, slowdowns in the network could cause the master and slaves to get out sync, thus violating the consistency requirement.)</p>
<h5 id="redis-versions-213-and-above">Redis Versions 2.1.3 and Above</h5>
<p>None of these constraints apply to the later versions of Redis (i.e., write operations no longer destroy the key before performing the write, and expiry times can be modified after initially being set). The implementation was changed to remove the constraints, <em>without sacrificing the consistency requirements</em>.</p>
<p>The new implementation simply injects a <code class="highlighter-rouge">DEL</code> (delete) operation into the replication stream and the AOF file every time that the server expires a key.</p>
<h4 id="more-on-strings">More on Strings</h4>
<p>We previously demonstrated the <code class="highlighter-rouge">GET</code>, <code class="highlighter-rouge">SET</code>, and <code class="highlighter-rouge">APPEND</code> commands.</p>
<h5 id="increment">Increment</h5>
<p>The <code class="highlighter-rouge">INCR</code> command makes it possible to use strings as an atomic counter.</p>
<p>For example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">SET</span> <span class="n">foo</span> <span class="mi">0</span>
<span class="n">OK</span>
<span class="n">redis</span><span class="o">></span> <span class="n">INCR</span> <span class="n">foo</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">INCR</span> <span class="n">foo</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">2</span>
<span class="n">redis</span><span class="o">></span> <span class="n">GET</span> <span class="n">foo</span>
<span class="s">"2"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SET</span> <span class="n">bar</span> <span class="n">baz</span>
<span class="n">OK</span>
<span class="n">redis</span><span class="o">></span> <span class="n">INCR</span> <span class="n">bar</span>
<span class="p">(</span><span class="n">error</span><span class="p">)</span> <span class="n">ERR</span> <span class="n">value</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">an</span> <span class="n">integer</span></code></pre></figure>
<h5 id="setnx-and-locking">SETNX and Locking</h5>
<p>The <code class="highlighter-rouge">SETNX</code> (“set if not exists”) command works like <code class="highlighter-rouge">SET</code>, but performs no operation if the key already exists.</p>
<p>This command can be used to implement a locking system. For example, the Redis documentation <a href="http://code.google.com/p/redis/wiki/SetnxCommand#Design_pattern:_Implementing_locking_with_SETNX">describes a locking algorithm</a>.</p>
<h4 id="lists">Lists</h4>
<p>Lists are used to store an (ordered) collection of items. Stacks and queues can be very easily modelled with lists.</p>
<p>For example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">LPUSH</span> <span class="n">newset</span> <span class="n">a</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">LRANGE</span> <span class="n">newset</span> <span class="mi">0</span> <span class="o">-</span><span class="mi">1</span>
<span class="mf">1.</span> <span class="s">"a"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">LPUSH</span> <span class="n">newset</span> <span class="n">b</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">2</span>
<span class="n">redis</span><span class="o">></span> <span class="n">RPUSH</span> <span class="n">newset</span> <span class="n">c</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">3</span>
<span class="n">redis</span><span class="o">></span> <span class="n">LRANGE</span> <span class="n">newset</span> <span class="mi">0</span> <span class="o">-</span><span class="mi">1</span>
<span class="mf">1.</span> <span class="s">"b"</span>
<span class="mf">2.</span> <span class="s">"a"</span>
<span class="mf">3.</span> <span class="s">"c"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">LPUSH</span> <span class="n">myset</span> <span class="n">z</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">RPOPLPUSH</span> <span class="n">newset</span> <span class="n">myset</span>
<span class="s">"c"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">LRANGE</span> <span class="n">newset</span> <span class="mi">0</span> <span class="o">-</span><span class="mi">1</span>
<span class="mf">1.</span> <span class="s">"b"</span>
<span class="mf">2.</span> <span class="s">"a"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">LRANGE</span> <span class="n">myset</span> <span class="mi">0</span> <span class="o">-</span><span class="mi">1</span>
<span class="mf">1.</span> <span class="s">"c"</span>
<span class="mf">2.</span> <span class="s">"z"</span></code></pre></figure>
<h4 id="sets">Sets</h4>
<p>Sets store an un-ordered, unique collection of items. In addition to supporting sets, Redis has native support for for union, intersection, and diff operations.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">set1</span> <span class="n">a</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">set1</span> <span class="n">b</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">set1</span> <span class="n">c</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">set2</span> <span class="n">c</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">set2</span> <span class="n">d</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">set2</span> <span class="n">e</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SINTER</span> <span class="n">set1</span> <span class="n">set2</span>
<span class="mf">1.</span> <span class="s">"c"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SUNION</span> <span class="n">set1</span> <span class="n">set2</span>
<span class="mf">1.</span> <span class="s">"c"</span>
<span class="mf">2.</span> <span class="s">"d"</span>
<span class="mf">3.</span> <span class="s">"a"</span>
<span class="mf">4.</span> <span class="s">"b"</span>
<span class="mf">5.</span> <span class="s">"e"</span></code></pre></figure>
<p>Sets are a critical tool for building highly-optimized indexes. For example, if your application allows users to like articles and follow users, then you should be manually maintaining indexes (i.e. sets) to <em>pre-compute</em> the answers to questions like “which users liked article <code class="highlighter-rouge">x</code>” and “which users follow user <code class="highlighter-rouge">y</code>”.</p>
<h4 id="sorted-sets">Sorted Sets</h4>
<p>Sorted sets (also known as “zsets”) are similar to sets, but each member of the set has an associated floating point score. Members are stored in sorted order, so no sorting is required to retrieve the data ordered by the floating point score.</p>
<p>For example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">1</span> <span class="n">Monday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">2</span> <span class="n">Tuesday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">3</span> <span class="n">Wednesday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">4</span> <span class="n">Thursday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">5</span> <span class="n">Friday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">6</span> <span class="n">Saturday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZADD</span> <span class="n">days</span> <span class="mi">7</span> <span class="n">Sunday</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">ZRANGE</span> <span class="n">days</span> <span class="mi">0</span> <span class="o">-</span><span class="mi">1</span>
<span class="mf">1.</span> <span class="s">"Monday"</span>
<span class="mf">2.</span> <span class="s">"Tuesday"</span>
<span class="mf">3.</span> <span class="s">"Wednesday"</span>
<span class="mf">4.</span> <span class="s">"Thursday"</span>
<span class="mf">5.</span> <span class="s">"Friday"</span>
<span class="mf">6.</span> <span class="s">"Saturday"</span>
<span class="mf">7.</span> <span class="s">"Sunday"</span></code></pre></figure>
<p>Sorted sets are essential whenever a range query is needed. For a clever application of sorted sets, see <a href="http://antirez.com/post/autocomplete-with-redis.html">Auto Complete with Redis</a>.</p>
<p>Furthermore, as with plain sets, sorted sets are an important tool to consider when constructing indexes.</p>
<h4 id="hashes">Hashes</h4>
<p>Redis hashes are conceptually equivalent to the Ruby hash, Python dictionary, Java hash table or hash map, etc.</p>
<p>For example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">HSET</span> <span class="n">user</span><span class="p">:</span><span class="mi">1</span> <span class="n">name</span> <span class="n">bob</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">HSET</span> <span class="n">user</span><span class="p">:</span><span class="mi">1</span> <span class="n">email</span> <span class="n">b</span><span class="o">@</span><span class="n">bob</span><span class="o">.</span><span class="n">com</span>
<span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="n">redis</span><span class="o">></span> <span class="n">HGET</span> <span class="n">user</span><span class="p">:</span><span class="mi">1</span> <span class="n">name</span>
<span class="s">"bob"</span>
<span class="n">redis</span><span class="o">></span> <span class="n">HGETALL</span> <span class="n">user</span><span class="p">:</span><span class="mi">1</span>
<span class="mf">1.</span> <span class="s">"name"</span>
<span class="mf">2.</span> <span class="s">"bob"</span>
<span class="mf">3.</span> <span class="s">"email"</span>
<span class="mf">4.</span> <span class="s">"b@bob.com"</span></code></pre></figure>
<p>Before this data type was added to Redis, users had two main options for storing hashes: use one key to store a JSON structure with multiple fields, or use one key for each field in the hash. Both of these approaches have many downsides versus using the native hash data type; the former introduces concurrency issues, and the latter requires significantly more memory.</p>
<h4 id="redis-transactions">Redis Transactions</h4>
<p>Redis transactions can be used to make a series of commands atomic. (Every Redis command is already atomic; transactions, however, allow us to combine a number of commands into a single atomic unit.)</p>
<p>In this example, the SET and two INCR commands are all executed atomically within the context of a transaction:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">MULTI</span>
<span class="n">OK</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SET</span> <span class="n">foo</span> <span class="mi">0</span>
<span class="n">QUEUED</span>
<span class="n">redis</span><span class="o">></span> <span class="n">INCR</span> <span class="n">foo</span>
<span class="n">QUEUED</span>
<span class="n">redis</span><span class="o">></span> <span class="n">INCR</span> <span class="n">foo</span>
<span class="n">QUEUED</span>
<span class="n">redis</span><span class="o">></span> <span class="n">EXEC</span>
<span class="mf">1.</span> <span class="n">OK</span>
<span class="mf">2.</span> <span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="mf">3.</span> <span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">2</span></code></pre></figure>
<p>Here, <code class="highlighter-rouge">MULTI</code> delineates the start of a transaction block; <code class="highlighter-rouge">EXEC</code> is responsible for actually running the transaction (i.e., executing all commands between the <code class="highlighter-rouge">MULTI</code> and the <code class="highlighter-rouge">EXEC</code>, with no other commands executed in-between).</p>
<p>There is an important caveat: because the commands are queued and then executed sequentially (all at once, as if they were a single unit), it is not possible to read a value while a transaction is being executed and to then subsequently use this value at any point during the course of the same transaction.</p>
<p>To address this restriction, Redis supports “Check and Set” (CAS) transactions. CAS is available via the <code class="highlighter-rouge">WATCH</code> command.</p>
<p>The following example (in pseudo-code) is an example of a good implementation of a function that only increments a key if the current value of the key is even.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">WATCH</span> <span class="n">mykey</span>
<span class="n">val</span> <span class="o">=</span> <span class="n">GET</span> <span class="n">mykey</span>
<span class="n">MULTI</span>
<span class="k">if</span> <span class="n">val</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">SET</span> <span class="n">mykey</span> <span class="p">(</span><span class="n">val</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="n">EXEC</span></code></pre></figure>
<p>If <code class="highlighter-rouge">mykey</code> changes after the <code class="highlighter-rouge">WATCH</code> command is executed but before the <code class="highlighter-rouge">EXEC</code>, then the entire transaction is aborted. The client can retry indefinitely on transaction failure. Any implementation that does not not use CAS is subject to race conditions, unless some other locking mechanism is employed (for example, the locking system described during the treatment of <code class="highlighter-rouge">SETNX</code>).</p>
<p>A more involved example, taken from an <a href="http://groups.google.com/group/redis-db/msg/d5b79d1133e867ce">earlier post</a> I made to the Redis mailing list:</p>
<blockquote>
I'm looking to build a FIFO queue, with one important/ non-standard property: if an item that already exists in the queue is added again, then the insert should effectively be ignored. (This data structure can be useful for managing certain types of jobs, like warming caches, etc.)
<br /><br />
An efficient way to model this structure in Redis would be to use a ZSET, where the score of each item in the set is the unix timestamp of when the item was added.
<br /><br />
However, the semantics of ZADD are that if an item already exists in the set, that the score is updated to the score specified in the command. If there were some way to tell ZADD to not update the score if an element already exists, or perhaps to supply a certain score if the element is new and a different score if the element is not new, then this queue-like data structure could be easily (and very efficiently) modelled in Redis.
<br /><br />
There are potentially a few other ways to do this, with the obvious approach being to check if an item already exists in the ZSET (using a set intersection) before issuing the ZADD command. However, this isn't bullet-proof, because it doesn't look like we will be able to do this within the context of a MULTI/EXEC.
<br /><br />
Does anyone have any other ideas?
</blockquote>
<p>Coincidentally, this was posted on the exact same day that the <code class="highlighter-rouge">WATCH</code> command made it into the Redis master branch.</p>
<p>A <a href="http://groups.google.com/group/redis-db/msg/85160957473fc761">solution</a> (in pseudocode, with thanks to Salvatore):</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">FUNCTION</span> <span class="n">ZADDNX</span><span class="p">(</span><span class="n">key</span><span class="p">,</span><span class="n">score</span><span class="p">,</span><span class="n">element</span><span class="p">):</span>
<span class="n">WATCH</span> <span class="n">key</span>
<span class="n">MULTI</span>
<span class="n">IF</span> <span class="p">(</span><span class="n">ZSCORE</span> <span class="n">key</span> <span class="n">element</span><span class="p">)</span> <span class="o">==</span> <span class="n">NIL</span>
<span class="n">ZADD</span> <span class="n">key</span> <span class="n">score</span> <span class="n">element</span>
<span class="n">retval</span> <span class="o">=</span> <span class="n">EXEC</span>
<span class="n">IF</span> <span class="n">retval</span> <span class="o">==</span> <span class="n">NIL</span><span class="p">:</span>
<span class="n">ZADDNX</span><span class="p">(</span><span class="n">key</span><span class="p">,</span><span class="n">score</span><span class="p">,</span><span class="n">element</span><span class="p">)</span>
<span class="n">ELSE</span>
<span class="n">DISCARD</span></code></pre></figure>
<p>(In this example: if the element already exists in the ZSET, then do nothing.)</p>
<h4 id="redis-virtual-memory">Redis Virtual Memory</h4>
<p>The goal of Redis Virtual Memory (VM) is to swap infrequently-accessed data from RAM to disk, without drastically changing the performance characteristics of the database. This enables a single instance of Redis to support datasets that are larger than main memory.</p>
<p>Virtual Memory is a very important feature of most modern operating systems. However, for efficiency reasons, Redis does not use the OS-supplied VM facilities and instead implements its own system. The rationale is as follows:</p>
<ol>
<li>A single page, as managed by the OS, is 4 kB.</li>
<li>The value of a single Redis key may touch many different pages, even if the key is small enough to fit in a single page.</li>
<li>For reasons previously discussed, Redis objects can be an order of magnitude larger in RAM than they are when stored on disk. Therefore, if using the OS’ Virtual Memory facilities, the OS would need to perform an order of magnitude more I/O versus a custom Redis Virtual Memory implementation.</li>
<li>Hence, by building Virtual Memory into the database server, overall efficiency can be significantly improved.</li>
</ol>
<h5 id="limitations">Limitations</h5>
<p>There are a few main limitations of Redis Virtual Memory:</p>
<ul>
<li><strong>All keys must be stored in memory at all times</strong>. Values can be swapped to disk, but keys cannot.</li>
<li><strong>Values must be swapped in their entirety, even for complex types.</strong> For example, if a list has one thousand items, all one thousand items must be resident in main memory before <em>any</em> list-related operation can be performed, including accessing the head of the list or appending a single item to the list’s tail.</li>
</ul>
<h5 id="implementation-details">Implementation Details</h5>
<p>When Virtual Memory is enabled, Redis stores the last time that each object was accessed. Additionally, Redis maintains a swap file that is divided into pages of configurable size, with the page allocation table stored in memory. Each page uses 1 bit of actual RAM.</p>
<p>When Redis is out of memory and there is something to swap, a few random objects from the dataset are sampled. The object with the higher “swappability factor” is the object that will be swapped to disk.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">Swappability</span> <span class="o">=</span> <span class="n">Object</span><span class="o">.</span><span class="n">age</span> <span class="o">*</span> <span class="n">Logarithm</span><span class="p">(</span><span class="n">Object</span><span class="o">.</span><span class="n">used_memory</span><span class="p">)</span></code></pre></figure>
<p>Redis maintains a pool of I/O threads that are solely responsible for loading values from disk into RAM.</p>
<p>When a request arrives, the command is read and the list of keys is examined. If any of the keys have been swapped to disk, the client is temporarily suspended while an I/O job is enqueued. Finally, once all keys that are needed by a given client are loaded, then the client resumes execution of the command.</p>
<p>From a configuration perspective, the <code class="highlighter-rouge">vm-max-memory</code> setting can be used to set the maximum amount of memory that Redis can use before it swaps to disk.</p>
<p>For more detail, see <a href="http://antirez.com/post/redis-virtual-memory-story.html">Redis Virtual Memory: the Story and the Code</a>.</p>
<h4 id="publishsubscribe">Publish/Subscribe</h4>
<p>Redis has native support for <a href="http://en.wikipedia.org/wiki/Publish_subscribe">publish/subscribe</a>.</p>
<p>In addition to supporting exact matches on channel names, it is also possible to subscribe against a pattern. In this way, subscribers do not need to know the exact name of all channels a priori, thereby increasing the flexibility of this messaging mechanism.</p>
<p>Although pub/sub may seem like an odd fit, Redis’ internals are very well suited for this feature. Furthermore, pub/sub brings with it numerous advantages. In particular, this feature is highly convenient in the context of the use cases of a large class of modern web applications, <em>and, with some creativity, can be used as a substitute for not having native scripting support within Redis.</em></p>
<h5 id="example">Example</h5>
<p>Imagine the scenario where a news-related site needs to update the cached copy of its home page every time that a new article is published.</p>
<p>The <em>background cache worker process</em> subscribes to all channels that begin with ‘new.article.’:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">PSUBSCRIBE</span> <span class="n">new</span><span class="o">.</span><span class="n">article</span><span class="o">.*</span></code></pre></figure>
<p>The <em>article publishing process</em> creates a new technology article (in this example, this article has ID ‘1021’), adds the article’s ID to the set of all technology articles, and publishes the article’s ID to the ‘new.article.technology’ channel:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="n">redis</span><span class="o">></span> <span class="n">MULTI</span>
<span class="n">OK</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SET</span> <span class="n">article</span><span class="o">.</span><span class="n">technology</span><span class="mf">.1021</span> <span class="s">"In today's technology news, ..."</span>
<span class="n">QUEUED</span>
<span class="n">redis</span><span class="o">></span> <span class="n">SADD</span> <span class="n">article</span><span class="o">.</span><span class="n">technology</span> <span class="mi">1021</span>
<span class="n">QUEUED</span>
<span class="n">redis</span><span class="o">></span> <span class="n">PUBLISH</span> <span class="n">new</span><span class="o">.</span><span class="n">article</span><span class="o">.</span><span class="n">technology</span> <span class="mi">1021</span>
<span class="n">QUEUED</span>
<span class="n">redis</span><span class="o">></span> <span class="n">EXEC</span>
<span class="mf">1.</span> <span class="n">OK</span>
<span class="mf">2.</span> <span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span>
<span class="mf">3.</span> <span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="mi">1</span></code></pre></figure>
<p>At this point, the <em>background cache worker process</em> will receive a message and know immediately that a new technology article was published, subsequently executing the appropriate callback to re-generate the home page.</p>
<h4 id="usage-examples">Usage Examples</h4>
<p>Redis is extremely flexible and highly usable in a number of different scenarios.</p>
<blockquote>
I see Redis definitely more as a flexible tool than as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well.
<br /><br />
<b><i><a href="http://antirez.com/post/redis-as-LRU-cache.html">Salvatore Sanfilippo</a></i></b>
</blockquote>
<p>A <em>small</em> sampling of potential applications:</p>
<h5 id="caching">Caching</h5>
<p>Caching (particularly for web applications) is likely Redis’ most common use case. For details on configuring Redis as an LRU cache, see <a href="http://antirez.com/post/redis-as-LRU-cache.html">here</a>.</p>
<p>Interestingly, despite memcached’s dominance in this area, plain key-value stores (i.e. those without support for data types like lists and sets) are at a disadvantage when acting as a web application cache.</p>
<p>For example, the resources returned from requests to web apps are typically composed of lists (lists of posts, lists of comments, lists of friends, etc.). With plain key-value stores, these lists will almost always be stored in single units (“blobs”). This makes very common list-related operations, such as adding an element to a list, getting the first ten items in a list, deleting the last item in a list, etc. very inefficient because the list is stored as a single unit and needs to frequently be serialized and deserialized within the application server. Furthermore, atomic updates of these lists are impossible without implementing some other mutual exclusion system. (Redis, with native support for lists, can perform these operations efficiently and atomically.)</p>
<p>This flexibility enables other cache-related advantages. For example:</p>
<blockquote>
One potential use for Redis is as a smarter replacement for memcached. A common challenge with caching systems is de-caching things based on dependencies - if a blog entry tagged with "redis" and "python" has its title updated, the cache for both the entry page and the "redis" and "python" tag pages needs to be cleared. Redis sets could be used to keep track of dependencies and hence take a much more finely grained approach to cache invalidation.
<br /><br />
<b><i>Simon Willison, <a href="http://simonwillison.net/static/2010/redis-tutorial/">Redis Tutorial</a></i></b>
</blockquote>
<h5 id="nginx--redis">Nginx + Redis</h5>
<p>This is a more specific type of (web application) caching than described above. Here, responses for certain types of dynamic requests are delivered directly to the requestor via the cache, bypassing the application server entirely. (See <a href="http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/">here</a> for a more detailed treatment of the subject.)</p>
<p>With the <a href="http://wiki.nginx.org/HttpRedis">HttpRedis module</a>, the Nginx web server can serve certain requests directly from Redis.</p>
<h5 id="interprocess-communication">Interprocess Communication</h5>
<p>Redis provides a very effective set of primitives for multiple processes on a single machine (or multiple machines connected via a network) to share state and communicate via message passing.</p>
<h5 id="views">Views</h5>
<p>Redis can be used to compute <a href="http://en.wikipedia.org/wiki/View_\(database\)">“views”</a> for tables in relational (or other NoSQL) databases that are difficult to query effectively, due to factors such as schema design, index design, data volume, write volume, etc.</p>
<p>For example, given a relational table that is used in an append-only fashion, a daemon could periodically pull down rows that it has not yet processed and “explode” the data into Redis, building out a number of lists, sets, sorted sets, counters, etc. (This is, effectively, hand-rolled index generation.) A reporting script can then perform operations against these data structures to compute all of the desired metrics.</p>
<h5 id="job-management">Job Management</h5>
<p><a href="http://github.com/defunkt/resque">Resque</a> (and <a href="http://github.com/defunkt/resque/wiki/alternate-implementations">alternate implementations</a>, like <a href="http://github.com/binarydud/pyres">Pyres</a>) leverage Redis’ capabilities very extensively.</p>
<p>A number of other job systems/ task queues (e.g. <a href="http://celeryproject.org/">Celery</a> and <a href="http://octobot.taco.cat/">Octobot</a>) also support Redis.</p>
<h5 id="locking">Locking</h5>
<p>Redis can be used to implement a lock service. As described earlier, <code class="highlighter-rouge">SETNX</code> is a key element of <a href="http://code.google.com/p/redis/wiki/SetnxCommand#Design_pattern:_Implementing_locking_with_SETNX">this locking algorithm</a>.</p>
<h4 id="designing-with-redis">Designing with Redis</h4>
<p><em>There is no <a href="http://en.wikipedia.org/wiki/Query_optimizer">query optimizer</a>.</em> Redis provides extremely fast primitives, but overall query performance is highly dependent on how the user chooses to arrange the data.</p>
<p>The most important things to remember are:</p>
<ul>
<li>The layout of the data should be designed based on how it will be queried.</li>
<li>It is the user’s responsibility to manually build indexes.</li>
</ul>
<p>As a direct consequence, data will almost always be duplicated in several places.</p>
<p>For example, imagine the scenario of using Redis to store a book database. An efficient data layout will include storing the details of each book (title, author, publisher, ISBN, genre, etc.) in a Redis hash.</p>
<p>In order to query the database to answer questions like “what other books did this book’s author write?”, the data layout should also include a number of manually-designed indexes. In this case, sets like the following should be built, each of which contain the ID number of all applicable books:</p>
<ul>
<li>all authors</li>
<li>all books by author</li>
<li>all publishers</li>
<li>all books by publisher</li>
<li>all genres</li>
<li>all books by genre</li>
<li>etc.</li>
</ul>
<p>In this example, we have duplicated the ID number of each book across multiple disparate data structures. (More generally, we have de-normalized our data to optimize the speed of each query.)</p>
<p>Redis cannot automatically remove all instances of a book from all indexes when the book is deleted. The application developer should keep track of all sets that a book is in (using an additional set) so that clean-up can be performed efficiently.</p>
<p>This type of data duplication is extremely common with non-relational data sets. For most systems, this necessitates running background workers that are responsible for constantly scanning the data set and repairing any inconsistencies that are detected.</p>
<h4 id="other-resources">Other Resources</h4>
<p>Some other fantastic Redis-related resources include:</p>
<ul>
<li>
<p><a href="http://peterc.org/">Peter Cooper</a>’s <a href="http://www.scribd.com/doc/33531219/Redis-Presentation">Redis 101 “whirlwind tour” presentation</a></p>
</li>
<li>
<p><a href="http://simonwillison.net/">Simon Willison</a>’s extremely comprehensive <a href="http://simonwillison.net/static/2010/redis-tutorial/">Redis Workshop/ Tutorial</a></p>
</li>
</ul>
<p><i>You should follow me on Twitter <a href="http://twitter.com/mjrusso">here</a>.</i></p>
<script type="text/javascript" src="/javascripts/generated_toc.js"></script>
Michael Russohttps://mjrusso.com/