<picture>
At the very tail end of my thesis project, I started working on a mobile app that uses the tech I built for the thesis itself. The app is called Pattern Recognition – it’s a thing for turning the world into music.
“Turning the world into music” is one of the great music technology cliches: it usually means parsing an image from left to right, turning colours into frequencies, and making things sound like bad early Stockhausen.
But it depends how you define “music”. Turning a picture into a sound is one thing: turning a particular kind of picture into a generative score/instrument, like, say, TuneTrace is another. Pattern Recognition takes images and turns them into human-playable instruments.
<video>
That’s the app and how it works. The rest of this is about the various things, aesthetic and otherwise, that have influenced me and that have influenced the app.
(You can also read about the research and engineering behind the app and the thesis here!)
For starters, I have a crush on the idea of layering the digital world atop the physical: systematizing (musical) rules on to reality. This is basic augmented reality stuff. This also connects to the whole New Aesthetic concept: the digital imprinting itself on to the real, and computational reality becoming physical reality.
My first first first introduction to this sort of thing was actually via a cat named Kharil Osman, who made wallpapers for a project called Desktopography, way back in 2005 These two in particular are what did it for me:
There’s the glitch, but also the systematization of the real world. A more up-to-date version of this can be found in the works of Mark Dorf:
Again, taking a rough, fuzzy-edged party of the world, and putting a digital layer on top of it. This, in turns, ties back to people like Piet Mondrian and Dorothea Rockburne: clean lines, digital precision, geometric shapes.
While we’re talking about digital layers, let’s talk about video games. Specifically, let’s talk about systems, about max/min-ing, about making discrete choices, about tech trees, sphere grids, pixels, and so on. Start with systems for leveling things :
I never even played Final Fantasy X, but just looking at the full sphere grid fills me with glee. Same with this path through the Alpha Centauri tech tree (which, oh man, did I ever play). Working through these self-contained universes (with an extra special shout out to Loom here), and through the interfaces that describe & define them, has been a huge part of who I am: Pattern Recognition is, in a small way, a method of creating self-contained musical universes.
Pixels and pixel art relate very quickly back to the New Aesthetic and the digital, of course. The idea of discrete choices, of quantified representations of things, helpfully enough, touches on the way our culture puts music on top of sound: making discrete, stepwise scales out of an inherently continuous medium.
It will probably surprise no one to hear that I have a crush on the relationship between math and music. It was Grisey who made up Pythagoras saying “music is number and emotion”, but I’m still a big fan of the statement. From the physical reality of the harmonic series to the human constructs of 12-tone rows and Tonnetzen and so on, math and numbers are, for me, a way of getting a handle on music. This starts with number theory and patterns of integers and scales and so on, and moves rapidly into graph theory, geometry, and so on.
Patter Recognition makes all sorts of assumptions about geometry and tonality / tuning, starting from basic pentatonics, moving up to diatonic, hexatonic, and ocatatonic scales, and eventually to equal-tempered microtonality and ratio-based microtonality. It’s very much a musical process, not a sonic process – though I’m working on adding code that will deal with timbre and volume, and more.
There have been countless ways to drive sound with two dimensions, from the earliest notation to people like Daphne Oram and Iannis Xenakis. Pattern Recognition presents a particular version of music, driven by the image supplied by the user. It was my man Goethe who said “Architecture is frozen music”, and it was my man Michael Flanders who deadpanned that “Donald Swann’s music has often been compared to de-frosted architecture”. Along the lines of that rhetoric, Pattern Recognition de-frosts the underlying geometries of images, and turns them into instruments that can be played and heard.