Jamie Bullock

The Pure Data GUI — a UX study

Wed, 23 Jan 2013 15:46:43 +0000

There has been some recent discussion on the Pd mailing list about rewriting Pd’s GUI using a new framework. This was triggered by my announcement of the Integra Live audio processing tool, followed by a number of people on the list commenting on how good the Integra Live GUI looks.

Integra Live makes use of certain visual effects: gradient filters, glow, anti-aliasing, bezier curves and so on. The line of thinking on the list seemed to be that these things made the UI look good, and that in order to make Pd’s UI look good, it needed to be rewritten using a different framework, as Tcl/Tk doesn’t have good graphics support.

Whilst it’s encouraging that people feel the Integra Live GUI looks appealing and want to emulate aspects of it, I think that focusing on the graphic effects is missing the point about the nature of good design.

Specifically:

User interface design is primarily about how things work
Visual appearance should be used to enhance function and user experience

I think there are basic problems about the Pd UI that could be addressed without a new framework and code rewrite. I have therefore conducted a brief user experience study on Pure Data partly as a personal exercise, partly to illustrate how much could be done without rewriting the application. My method was to simply download Pd and identify as many user experience issues as I could in 30 minutes.

The results…

Download and launch

The website is very text heavy, and I’m confused about the difference between “Downloads” and “Featured Downloads”
Clicking “Downloads” — I’m presented with “Pd-extended” and “Pure Data”. I’m not sure which one to pick, but go for “Pd-extended” since it is at the top, is bigger and claims to be “easy-to-install”
When I download the DMG I get big intimidating license to read — I just click “agree” without reading it
When I double-click the app, I get a warning:

I know a way around this, so I ctrl-click and select open…

In Use

There is no initial call to action when I launch the application

Menus

When I launch the About dialog, the window appears on top of the Pd console, obscuring the Pd Console title bar

The Preferences panel also appears on top of the Pd Console, but this time, up and to the left, obscuring the Console title bar:

The Preferences dialog itself is not clearly laid out. Due to the placement of buttons and checkboxes, I’m not clear which controls relate to “Startup flags” and which relate to “Paths”

It is unclear why the dialog has “Apply” and “OK” and when I would need to click each one
When I click “Reset to Defaults” the dialog just closes. When I reopen, a number of paths have been added to the dialog. I wonder if these are the defaults, why they weren’t there in the first place. Also, I’m not clear as to why I need to know about these paths.

When I select “New” from the “File” menu, I get an empty window, which again obscures the Console title bar, this time the Window is positioned in the top-left of my screen
Again, there is no call to action

Experimenting, I discover that windows always appear in the top-left corner of the screen regardless of where the previous window was. This causes frustration because I’m constantly moving windows to compensate and resizing as the default size is too small. A quick look through the Pd help files suggests the average Pd patch size is far larger than the default patcher window size
I notice neither the Pd console nor the patcher window respond to double-click on their title bar by minimising (the standard behaviour on Mac OS)
Selecting “Media” and “Test Audio and MIDI”, a patch loads, but it looks jumbled, and I notice the Pd Console fills up with errors:

The “Audio settings…” dialog is unclearly laid out (no vertical alignment?), and I’m unclear what “use callbacks” and “Delay” mean

The Audio and MIDI settings dialogs have two buttons at the far left of the window title bar, but they are greyed out and not-clickable
I’m again confused by the function of “Apply” vs “OK” buttons
The font size dialog is not a native font picker, and I’m confused by the meaning of the “Stretch” function

All of the settings dialogs seem to be non-modal, but I’m not sure why I’d want a settings dialog open whilst operating the main application
Selecting the “Help” menu, I get a native “Search” field, but I can’t type in it
Selecting “Help Browser”, I get a non-native help browser

The Help system itself is confounding. Selecting “Manuals” then “+ Start Here” then “+start-here.pd”, I get Pd patch with some errors in

Patching

I decide to make a test patch to confirm Pd is actually working
I create an object, type in the object name and hit enter. I expect the object to be created, but enter inserts a newline instead. I wonder if any objects make of this newline feature…
Adding a [dac~] object doesn’t work, I get an error. So I restart Pd

This time dac~ works and I create a test patch with an simple oscillator and amplitude control
My patch is a little messy, so I select “Edit” -> “Tidy Up”. This does nothing except throw the error: “best vertical distance 4” in the Pd Console

I wonder why there isn’t a keyboard shortcut for “Tidy Up” when it may be a common operation
I arrange my patch until it is nearly tidy, and try “Tidy Up” again

Before:

After:

TIME UP!

Conclusions

I’m sure there are many more UI issues than the ones I’ve outlined here. This hasn’t been an in depth study, but a rather haphazard one. It is also a matter of subjective judgement as to whether some of the points I’ve raised are good design, bad design, or simply bugs.

I’d wager at least half of these points are a matter of frustration and / or confusion for the majority of users, and are simply a matter of bad design, or just a lack of design. Many of the issues I’ve raised may also seem rather minor, but this all adds up.

There are also many good things about the design of Pd’s GUI. It’s clean, concise, simple and portable. I think significant work has probably gone into it.

Which leads me to my conclusion: it takes a lot of work to write a UI, and it is certainly easier to improve an existing UI than start a new one from scratch. So, if the Pd community is seriously going to embark on another GUI rewrite, they need to make sure that they get the design details right. And by that I don’t mean fancy bezier curves and gradient fills, I mean making sure the basic user flow is good and that everything is clear and purposeful.

There are also user experience issues to consider. There are already two versions of Pd (Vanilla, Extended) on the main website, and a bunch of UI plugins. I don’t think adding another alternative will improve user experience, so if a GUI rewrite is undertaken, for good UX, it needs to become the Pd GUI.

Final Words

Building software is fun, but doing it well is really really difficult. Any attempt to rewrite the Pd GUI needs to be certain that it will actually be an improvement and not just an equally unusable version with rounded corners and gradient fills. Its important for those involved to understand that simply adding visual effects to Pd’s GUI isn’t going to improve the design. The design will only improve if issues of user flow, clarity, learnability and discoverability are addressed and aesthetics are employed to enhance meaning and improve experience rather than simply being a veneer. By contrast, there are many many design issues that can be addressed in Pd without rewriting the GUI at all.

Stravinsky on Constraints

Tue, 13 Nov 2012 12:58:13 +0000

I love this quote from Poetics of Music: in the form of six lessons By Igor Stravinsky:

The more constraints one imposes, the more one frees one’s self… the arbitrariness of the constraint serves only to obtain precision of execution.

I wonder how this applies to software design? What is the “sweet spot” between constraint and flexibility that provides the best conditions for creativity?

Slice//Jockey for Pure Data

Thu, 08 Nov 2012 14:13:00 +0000

This a brilliant piece of work by Katja Vetter. A lovely approach to algorithmic beat composition, where you just make sounds into the computer and it arranges them into something musically interesting. The software also has a nice prototype interface for controlling effects.

How to write an opening statement

Wed, 31 Oct 2012 19:45:01 +0000

I’m revising some of my “about” and “biography” pages at the moment, and looking for inspiration. The opening statement to the MIT Media Lab’s “overview” document is just sublime:

The Media Lab is a place where the future is lived, not imagined. Our domain is applying unorthodox research approaches for envisioning the impact of emerging technologies on everyday life.

Wow! That’s a seriously well-crafted piece of writing, and something to aspire to on every level.

Orphion: Orphion - Concept and design of a touchscreen musical instrument

Fri, 19 Oct 2012 12:55:49 +0100

Orphion: Orphion - Concept and design of a touchscreen musical instrument:

orphion:

The usage of contemporary multitouch devices such as Apples iPad as musical instruments needs to focus on the design of a special interface that incorporates the fact of missing kinesthetic feedback by the device. To form a conclusive instrument this interface has then to be linked directly to the sound generation to create an intuitive feel of control. The Orphion uses virtual pads in a certain layout that represent individual voices. Pitch and timbre of each voice depend on the initial point of touch, the size and variation of size of the touchpoint, and the variation of the position after the initial touch. Parameters control a physical model for sound generation.

The development of musical instruments during the last centuries mainly aimed on the improvement of sound generation rather than finding new interfaces. Especially for electronic instruments, apart from exceptional inventions like the Theremin in 1920, the most commonly used interface still is the 18th century approved interface of the piano keyboard.
For the Orphion the goal was to develop an instrument that deliberately departs from that concept and takes advantage of the haptic and technological possibilities of multi-touch devices; it’s sound should be a direct representation of the actions the fingers perform on the pads. My aim was also to create an instrument that feels “natural” to someone familiar with the behavior of existing acoustical instruments like drums and string instruments.

Interfaces for musical instruments

Acoustical instruments have a natural coupling between inter-face and sound generation, which is defined by the materials used for their construction. For most electronic instruments these two components are separated. The requirements of the interface part for all kinds of musical instruments, however, can be generally defined:

allow virtuosity/expression

intuitive playing

traceability (for the audience)

predictability (for the player)

give feedback (acoustic/visual)

Instrument vs. Controller

Since its mechanical structure and the materials used for build-ing the enclosure of electronic instruments normally do not contribute to their sound whilst played, which is especially true for software based instruments, these instruments needs a strong logical link between the action of the player and the generated sound. This strong link between a very specific inter-face and the sound generation is what defines that structure as an instrument.
Alternatively the interface is an open structure with different functions or sounds for different situations, and can be seen more as a controller rather than as an instrument. Such an interface can work very well for a variety of situations, however the above mentioned properties of an instrument are hard or impossible to achieve. Additional layers of complexity arise, if the interface does not control the sound generation directly but drives e.g. a sequencer. Such a “sequencing instrument” points in a new direction but also marks the strongest possible departure from a traditional musical instrument. For my purposes I decided to focus on the idea of a strong coupling between interface and sound generation.

The Orphion

How does a touchscreen sound and what gestures allow expressive playing whilst keeping control? Most applications for touchscreens use the finger or stylus as a replacement for the mouse to control knobs and buttons. Since multitouch has been established, new gestures were created, for example pinching two fingers to resize objects, but in order to express musical ideas, more specific gestures and input models have to be developed.

Finding a logic interaction model and thus a suggested way of playing seems to be the main part in the development of a new instrument. The Orphion should allow polyphonic playing of defined pitches with different articulations (staccato, legato) and timbres for each individual voice. When developing Orphion I had to take into account:

haptic properties of touchscreens (size and tactile or kinesthetic ways of interaction)

musical playability (recognition of initial touchpoint and matching of pitches), musical expression (dynamics, intonation, vibrato, timbre)

intuitive and natural feel

technical possibilities (precision of control data, processing power)

As guiding models for the behavior of the Orphion I looked at two types of instruments: Drums and string instruments.

drums: round playing area with different timbres, re-lease time and damping depend on the velocity and du-ration of touch.

string instruments: multiple individually tuned strings plus ability to play the tuning via tapping the strings, control of tone and articulation during sustain-phase (intonation/vibrato, damping)

See figure 1 for the connection between the interface and the sound generation by different parameters:

Figure 1. Simplified diagram of interface and sound generation

Interface and sound

The interface of the Orphion represents virtual pads, which are capable of sounding either plucked like a guitar string or produce timbres closer to a slap on a conga drum depending on the size of the touch point. The timbre changes when hit closer to the rim like on a real drum, and its pitch is a function of distance from the center hit point, in order to model something that comes close to pulling a string (the range varies by the size of the touch point). Every parameter is controlled by a single finger per note. The iPad currently supports up to eleven touch points. However, internal polyphony of the instrument is de-fined by the number of pads present on the touch screen.
Visually the interface is represented by a defined set of pad drawn as circles of variable size and position. The different sets of layouts allow the instrument to be adapted to multiple musical situations, genres and to match the virtuosity of the player. The arrangement structure with symmetrical intervals (e.g. fig. 2) in each axis can be used to find new harmonic structures by advanced musicians, pad layouts with only pentatonic tone material or other simplified musical concepts (e.g. fig. 3) make it also interesting for musical beginners. Layouts with less pads can give you the feel of a percussion instrument played with a fixed assignment between finger and pad (e.g. fig. 4).

Figure 2. Symmetrical major 3rds horizontally,
minor 3rds and semitones vertically, 4ths and 5ths diagonally

Figure 4. blues-scale layout

Figure 5. five-finger layout, e.g. as tuned bass drums

Sound synthesis

The sound synthesis is based on physical model that simulates a string (Karplus-Strong algorithm). I use a combination of a pulse of filtered noise and a sustained excitation sound created by a two operator FM synthesis structure. The low-pass filtering of the feedback path is controlled in realtime for lively articulation of the sound after initial touch. The complex excitation model allows a wide spectrum of different sounds from gently plucked strings to xylophone-like hits or damped attack of muted drums. As long a finger is touching the surface of a pad, the distance from the pad’s center controls slight detuning (intonation) and variation of timbre towards the rim.
We achieve a very natural feeling by dynamical adjusting parameters within a single synthesis model rather than switching between different models for different playing situation, as it would be the case with sample based instruments.

The visual representation of the pads on iPad is straight for-ward and functional: A pad is defined as a outlined blue circle with it’s note name written in the center. When touching a circle, it is filled with color, ranging from red to yellow depending on touch size, thus indicating the amount of damping.

ja.net Streaming Workshop Notes

Thu, 20 Sep 2012 11:35:00 +0100

Yesterday I attended the ja.net Arts and Humanities Streaming Workshop at the Royal College of Music. Basically the idea of the day was to showcase a number of technologies that make use of ja.net’s high bandwidth NREN backbone.

The event included an impressive demo from the originators of the Low Latency Audio Visual Streaming System (LOLA). The demo was presented as a live performance between a pianist performing the media room at RCM (the event venue), and a Clarinettist performing at Napier University in Edinburgh. The Clarinettist’s sound was played to the audience through a pair of small near-field monitors, and a live video feed was projected onto a screen at the front of the venue.

The sound quality was excellent, and there was no significant musical difficulty related to network latency. When asked in the Q and A session afterwards, the Clarinettist reported that initially, he noticed some difference compared to a same-room performance, but he soon got used to it. My guess is that the performers adjusted to the slight latency (<50ms) and compensated for it accordingly.

However, there was a noticeable difference (from an audience perspective) in terms of the spatial “presence” of the Clarinet. Although the performer was moving around as part of the gestural expression of the piece, the audio was captured through a single microphone, with fixed pan-position in the room. This, coupled with a lack of dispersion from the instrument body and natural room response created ear fatigue after several minutes of listening.

I also strongly missed both the performer-audience interaction and performer-performer interaction we normally get in a concert. This is usually subtle (eye contact, nuances of gesture and posture etc), but important in maintaining focus and interest. The overall effect was of observing a studio recording session.

LOLA itself, is an audio codec for putting audio from a general-purpose computer (e.g. laptop) on a network, with minimal processing delay (latency). It is very similar to CCRMA’s JackTrip project, and aside from some usability nuances, I struggled to understand the key advantages of LOLA over JackTrip. Further research into these systems including quantitive tests and QoS benchmarks are needed, with peer-reviewed publication as a next step.

To achieve the kind of impressive performance witnessed in the LOLA demo, all of the variables in a fairly complex stack of hardware and software need to be finely tuned. The system needs to be run on a high-speed, high-bandwidth link with minimal round-trip latency. There is also a threshold on the physical geographical distance between the locations involved. The current quoted figure is 1ms network latency per 100km, so whilst the CODEC latency is fixed, the network latency resulting from cable distance is an important variable. Additionally, the microphone and speaker setup needs to be carefully tuned, as does the room itself, including the use of diffusers to avoid noticeable room echo. Finally, a music-optimised echo cancellation system such as Brian Shepard’s EchoDamp needs to be used and finely tweaked. Brian demo’d his software during the event, and whilst it was impressive, I suspect there is much research to be done in this area also.

In the coffee break, discussion tended focus on possible applications for LOLA, and audio/video streaming in general. The general impression I got was that most of the delegates weren’t convinced by the notion of a videoconference-type setup being used in a live performance as a “substitute” for physical performer presence. The RCM have been using videoconferencing for delivering and receiving one-to-one instrumental tuition, and this seems to be one of the most promising areas for future development. I can also think of many niche “artistic” applications. However, the needs of these applications could be fulfilled by existing systems such as DVTS or Conference XP, and wouldn’t specifically require super-low roundtrip latency. 100ms is perfectly acceptable for delivering a Violin lesson, for example.

In conclusion, I’d say that it will only be a matter of time, before low latency audio and video streaming systems are commonplace. The question is, how can we use it to add value to what we do? The first talk in yesterday’s workshop focused on the non-technical aspect of this - what is the relationship between the “medium” and the “message”. I think as arts researchers, we should be asking not only how we can deliver existing experiences through a new medium, but also what new possibilities does the medium offer that we hadn’t previously considered…

Coding standards reboot

Wed, 18 Jul 2012 17:19:00 +0100

Most programmers have a set of coding standards they work to. These range from cosmetics such as indentation style to more fundamental things like acceptable maturity of dependencies or memory management standards.

I’ve recently given my own standards a reboot and it mostly boils down to the phrase: “don’t ship bugs”.

I used to write a lot of code like this:

 /* FIX: global - make this thread safe */
float *temp[5000];

/* FIX: eventually pass arguments rather than using globals */
void to_be_replaced(void) 
{
    /* FIX: inline constant */
    for(int i = 0; i < 5000; i++)
    {
         /* FIX: memory leak */
         temp[i] = malloc(sizeof(float));
         /* FIX: inefficient - refactor */
         *temp[i] = cosf(i);
    }
}
/* etc...*/

This code smacks of under-confidence and/or low standards. The issue isn’t so much the problems it contains, as the fact that they are knowingly left in the code. “FIX” comments are added as an acknowledgement as if to say “I know this is wrong, but I’ll come back to it later”.

This might work for some developers, especially on single-author projects. However if you’re working as part of a team, seeing a lot of “FIX” in code is unnerving and undermines confidence. For me, code comments aren’t fine-grained enough to serve as a bug reporting mechanism.

So what’s the solution?

From now on I’m going to be doing the following:

don’t commit code that has known problems
transfer intended improvements from the code to an issue tracker (no “FIX”)
develop vertically rather than rushing larger pieces of functionality
write skeleton code to sketch structure

Point 3 is probably the most significant and important. It also implies a shift in approach to architecture — to facilitate this vertical approach, the software architecture (and development methodology) need to be able to support it.

In summary: “working but incomplete” is better than “complete but broken”.

Hello NodeBox!

Wed, 11 Jul 2012 15:31:00 +0100

I was recently asked to present a workshop at the Interface 2012 conference in Birmingham. The brief I was given was “programming for the humanities”. My instinct was to base the workshop around Python, a language I often recommend to students for its general purpose nature and ease-of-learning. I’ve had a long interest in data visualisation, and been following the NodeBox project for a couple of years, so my hunch was that this might be a good “top down” route into Python…

NodeBox is a “procedural graphics” environment. This means it’s not like a normal graphics app, where you create an image by drawing shapes on a canvas. Instead you specify a set of rules defining the graphics and the computer draws it for you. If that sounds like programming, it is. The set of rules is called an algorithm, and in NodeBox the algorithm is determined by connecting up boxes that do things such as draw a shape, apply a transformation, change a colour, apply randomness etc.

This approach to graphics is useful when…

you don’t know what you want to create
the drawing you want to make is very repetitive
you want to you use external data to determine what the drawing looks like

The first incarnation of NodeBox, took influence from Ben Fry’s Processing environment essentially providing a 2 pane interface with Python code on the right and generated graphics on the left. NodeBox 2 goes beyond this, providing tight integration between node-based programming, direct graphical manipulation, and text-based access to the underlying Python code for each node.

This multi-paradigm single-screen approach is lovely, pre-dating similar approaches taken in environments like Codea, and giving more than a passing nod to Bret Victor’s concept of “dynamic pictures”. NodeBox 3 gets even closer to this, doing away with the Python panel altogether.

After a couple of hours working through the online tutorials, I was up-and-running creating complex patterns like the one shown above, just by connecting up a handful of boxes. NodeBox also supports the ability to load data from CSV files and use the values to drive the image generation. This aspect particularly interests me, and is something I’d like to explore more. Given you can write your own nodes in Python, I presume it would also be possible to make generative graphics using live data from the web…

The Listening Machine

Mon, 21 May 2012 12:57:32 +0100

The web is literally buzzing about The Listening Machine, a new musical work by Daniel Jones and Peter Gregson. The piece takes the content of 500 Twitter accounts, and through some clever computer programming and musical mapping turns them into an indefinite bricolage of sound, which somehow represents their content.

The result is one of the most successful pieces of data sonification I’ve heard. What’s great for me is the clear relationship between the twitter stream content and the end result. If you asked me “what would Twitter sound like as music?”, The Listening Machine isn’t far off. It’s also aesthetically interesting, visually and musically — I wan’t to keep coming back to hear “what’s happening now?”, “what does Twitter sound like at night time?”.

The piece also presents some interesting ideas about music and technology. The question “can machines listen?”, often has the subtext “…like humans”. In The Listening Machine, we have a machine that listens to humans like a machine, and generates music from fragments played by humans. Technology and art serve each other. Beautiful.

The piece is running from May until October 2012 on The Space, the new on-demand digital arts channel from the BBC and Arts Council England.

How does design work?

Tue, 03 Apr 2012 10:19:39 +0100

How does design work?:

I’ve been talking recently about researchers collaborating with designers to make better systems for musicians. But what do I actually mean by that? This article gives a lovely, concise overview on the different types of design, who designers are, and why they are important for making software usable:

http://startupsthisishowdesignworks.com/

Musician-centred Design

Thu, 01 Mar 2012 12:41:00 +0000

Software as medium

In live electronic music, software mediates the experience of composition and performance. Software is the medium through which ideas are developed and structured, it is the medium through which sound is generated and processed, and it is the medium through which sound is controlled. The nature and design of the software is therefore critical to an understanding of the ways in which it mediates user experience. Well-designed software can lead to a pleasant and productive experience, poorly-designed software can lead to frustration and rejection.

Max and the myth of neutral software

It’s not just a question of learning and software literacy. It’s true that the notion of ‘intuitive’ software is a myth, but software that draws upon users’ existing experiences has been shown to be more learnable and therefore more usable. These references to existing experience are called affordances. Affordances can be used to introduce novelty by calling the user to a given action. Software can also be made more usable by introducing assumptions about how the software will be used. These assumptions can be embodied in structured workflows, which mediate the users’ experiences in such a way that it reinforces their intentions.

The primary means for people to work with live electronic music is currently Max by Cycling 74. Max is a tour de force of software engineering, and incredibly powerful on a number of levels. It was originally written in the mid-1980’s as means to control hardware-based audio processing systems. It has since evolved to become very broad in scope, the current website says:

Max gives you the parts to create unique sounds, stunning visuals, and engaging interactive media. These parts are called ‘objects’ – visual boxes that contain tiny programs to do something specific. Each object does something different. Some make noises, some make video effects, others just do simple calculations or make decisions. In Max you add objects to a visual canvas and connect them together with patchcords. You can use as many as you like. By combining objects, you create interactive and unique software without ever writing any code (you can do that too if you really want to). Just connect.

Max makes few assumptions about what it will be used for beyond providing a low-level dataflow and DSP patching language for the creation of anything related to sonic and visual media. Max therefore presents itself as a ‘neutral’ tool for creative work, a tool where ‘the only limit is your imagination’. However, in reality Max has deeply embedded in it, the non-neutral assumptions of a software development workflow. Max is actually an advanced integrated development environment (IDE) for programming multimedia interaction and processing. Far from being a neutral tool, Max mediates the experience of the musician by imposing the workflow, prerequisites and aesthetics of software development on their user experience. With Max, the musician is programmer, instrument maker, designer and composer all in one. Many would argue this to be a positive trait, and in some respects it can be.

Musician-centred design

I believe that musicians need and deserve software that is better tailored to their specific requirements. They need software that has embedded in it, the assumptions of the musician, not the assumptions of the programmer. I call this musician-centred design. The way in which the software mediates the experience of the musician should be centred around the existing knowledge, practices and thought processes of composers and performers. Workflows should be structured around achieving the most common tasks quickly and easily. Interactive music technology has been through many experimental phases, we now know the creative possibilities are ‘endless’. It’s time to consolidate these possibilities and make them more accessible and more musically meaningful.

Some aims of musician-centred software:

Software should encode the assumptions of the musician in how it operates
Workflows should be structured around musical tasks
Software should draw upon musicians’ existing experience
Software should employ innovative UI design to achieve its ends

Conclusions

Max is a wonderful tool for the technically literate who want to employ creative coding to achieve their goals. However, there are many musicians for whom Max is simply too distant from their existing experiences to even approach. I believe the central challenge ahead for music technology research is to investigate and develop new user interfaces that are musician-centred. I see many possibilities here, and feel we haven’t started to scratch the surface in solving these problems. I don’t have the solutions yet, but I believe that collaboration between music technology researchers and designers should be the starting point. Comments welcome…

Bret Victor talks about his principle that “creators need...

Mon, 20 Feb 2012 08:14:00 +0000

Bret Victor talks about his principle that “creators need an immediate connection with what they create”.

This is brilliant stuff. Skip to 12 minutes if you don’t have much time.

Jonathan Harvey: total immersion

Sun, 29 Jan 2012 22:37:00 +0000

Seven years ago, my colleague Lamberto Coccioli and I wrote a research paper for the International Computer Music Conference outlining our intentions to develop a new software version of the electronics in Jonathan Harvey’s Madonna of Winter and Spring. Last night the system was used for the first time in public performance at the BBC Symphony Orchestra’s ‘total immersion’ event at the Barbican Centre in London.

Madonna of Winter and Spring is an incredible piece, featuring Symphony Orchestra, amplification, ring modulation, reverb, dual panning, Yamaha TX816, Yamaha DX1 and Emulator II sampler. However, these synthesis components have made the piece cumbersome to perform, with hardware versions being rare and unwieldy to setup and software versions requiring complex combinations of commercial software, some of which is already obsolete.

The new version we have developed as part of the EU-funded Integra Project, attempts to make the electronics for the piece both more sustainable and more performable. It achieves this through the use of a fully open source, entirely software based system, which simple to setup and use. This system runs on two computers and consists of:

the Hexter DX7 emulation plugin by Sean Bolton
a new external for the Pure Data environment, for hosting DSSI plugins
a new software sampler implemented entirely in Pure Data
ring modulation and panning provided by Integra Live

In the Barbican performance last night, the panning was controlled by an iPad running TouchOSC and communicating with Integra Live via wireless MIDI.

The performance was played in its entirety on BBC Radio 3’s Hear and Now programme. Moments where the synths are prominent include 1:01 - 1:07 and 1:17 - 1:20, although synthesis and electronics appear throughout.

Working with Sound Intermedia and the the BBC Symphony Orchestra this week has been a delight. It has been both exciting and rewarding to see the results of seven years’ work come to fruition. To my knowledge, this marks the first ever use of open source DX7 emulation software for a large scale orchestral work with electronics. The materials we have developed now reside with Harvey’s publishers, Faber Music, and will be used in all future performances.

This is a great achievement for the future performability of works involving DX7-based synthesis and I’d like to thank all who’ve made this possible, specifically Sean Bolton, Miller Puckette, Lamberto Coccioli, Dag Henning Kalvøy and James Dooley. At this moment, I am very proud.

Thoughts on #codingforkids

Fri, 13 Jan 2012 12:50:00 +0000

This week, government minister Michael Gove announced government plans proposing that current school ICT is replaced by a new Computer Science GCSE. I have no actual numbers for this, but my guess is that most people think this is a great idea. Why wouldn’t we want kids to learn to code?

The tricky part comes when we start to dig into the detail. What will the new curriculum contain? How will it be taught? Who will teach it?

One thing I am clear about is that the choice of programming language should be left to individual teachers and pupils.

What I’d like to see the new curriculum achieve:

sharpen creative and analytical skills through projects and play
understand abstraction and think in an abstracted way
self-teaching through experimentation and research
learn to modularise and classify problems and solutions
understand algorithms, data structures, interface, and state

This is what I personally understand as ‘computational thinking’.

The end results, be it an mobile app, a game, or a robot controller and the languages used, should result from the skills and interests of teachers and students.

Vision: the future of music technology...

Wed, 09 Nov 2011 10:52:00 +0000

I read with great interest Bret Victor’s rant on the future of interaction design. The starting point for his post is Microsoft’s ‘Productivity Future Vision’, which is intended to be utopian, but actually gives a dystopian and de-humanising portrayal of future interaction.

Bret’s response to the video is inspiring, particularly his definition of tools:

A tool addresses human needs by amplifying human capabilities.

Furthermore:

A tool converts what we can do into what we want to do.

Reading this has left me thinking about how I can apply this concept in my own field of music technology research. Traditional musical instruments do the job beautifully: they amplify human gestures, ideas and emotions. Projects I’ve been working on recently have the aim of bringing together music and digital technology, but I feel we are still a long way from creating truly musically-meaningful tools.

I’m not really sure how we’ll get there, but my vision is that we should create digital tools that enable musicians to harness the transformative power of technology through already-learned expressive semantics. Current tools work the other way, and force musicians to understand the semantics of specific technology.

What’s your vision for the future?

*Hammer Images used by kind permission of Bret Victor

What makes something a musical instrument?

Mon, 24 Oct 2011 10:06:00 +0100

I’ve been thinking about this question since my final year as an undergraduate, and explored it in some detail in my M.Phil. thesis. I’ve also blogged about the subject, and discussed it many times. I’m now approaching a kind of opaque definition:

“If we take two performances of a piece, played on the same instrument, with identical articulation, and subtract one performance from the other, we’re left with the instrument”

In other words, to be called a musical instrument something must be capable of nuance.

Mechanical and analogue electronic instruments, are broadly predictable in their behaviour, but at a detailed level they constantly produce small, unpredictable variations in tone and response even with the same playing techniques.

With digital instruments, this capability for nuance and unpredictability isn’t inherent and therefore needs to be added artificially. For me, providing these nuances of interaction remains one of the key challenges of digital instrument design.

*Image by Soophoo from Wikimedia commons

Thoughts on sustainability

Wed, 07 Sep 2011 11:36:00 +0100

Over the last 7 years at least some part of my professional work has involved dealing with the issue of ‘sustainability’ in music technology. This has largely centred on the question of making musical works with live electronics more sustainable, for instance, by storing data in an application-neutral format or developing software versions of legacy hardware systems.

However, now I’m coming to the end of a six year research project, which has sustainability at its core, I’m starting to question whether we have achieved our sustainability goals…

What is sustainability?

The first thing we need to acknowledge is that this is a sliding scale. It’s not a question of whether something is sustainable or not, but rather how sustainable. Whilst I previously viewed this as a purely technical matter (e.g. binary file formats are less sustainable than text-based formats), I’m starting to look at the problem along a number of separate dimensions, namely: openness, funding and acceptance. A truly sustainable project needs all three.

Openness can be achieved through open licensing and the use of open standards. Funding can potentially be gained through academic or public research funding as well as commercialisation.

Possibly the most important, but also difficult aspect of sustainability to fulfil is ‘acceptance’. In order to be sustainable, software, protocols and standards need to be accepted by communities that use them. Without a community of users, contributors, supporters and developers who ‘accept’ it, a project ultimately becomes redundant.

How Does 'Incremental' Define Apple?

Thu, 30 Jun 2011 11:01:40 +0100

joewilcox:

I’d like to discuss how Apple innovates, which I understand very well. I posted about Apple’s incremental product strategy last September at Apple Watch: “Apple Demands a High Price to Be Cool.”

The pattern is consistent: Apple launches a “one more thing” product with modest hardware features but something else nevertheless killer—something people want. During the launch, Apple CEO Steve Jobs performs his marketing magic, demonstrating how this “one more thing” product will make peoples’ lives better.

Read More

A book about the practice of live electronic music

Sun, 08 May 2011 15:01:00 +0100

…basically there isn’t one.

There seem to three classes of book referring to the subject, the most common are technical books about the theory and techniques of digital audio processing.

Examples include:

Computer Music (Dodge)
Elements of Computer Music (Moore)
Introduction to Computer Music (Collins)
The Computer Music Tutorial (Roads)
Theory and Techniques of Electronic Music (Puckette)

These books are remarkably similar in terms of structure and content. Fundamental digital audio concepts are introduced (bits and bytes), amplitude, periodicity, digital filter theory and delays, frequency domain processing (FFT), synthesis. Additionally psychoacoustics, audio programming, and control processing are usually covered to some extent. Sometimes theory is contextualised with reference to specific musical examples.

Additionally, there are discursive books about history and aesthetics, e.g.

Living Electronic Music (Emmerson)
Electronic and Computer Music (Manning)
Electronic and Experimental Music (Holmes)

These books also share common structure and content. Historical overview starting from Cage’s Imaginary Landscape #1, presentation of the main components of an electronics setup — the microphone to the loudspeaker — the evolution of digital processing, all presented from what might be considered a ‘musicological’ perspective.

Finally, we have books that deal with a specific system or software, e.g.

The Csound Book (Boulanger)
The SuperCollider Book (Wilson, Cottle, Collins)
Electronic Music and Sound Design - Theory and Practice with Max/MSP (Cipriani, Giri)
Loadbang: Programming Electronic Music with Pure Data (Kreidler)

These books take the presentation of techniques in the given software as their starting point, providing theoretical background and practical examples as needed. Techniques are sometimes presented in the context of a given musical work.

What I’m missing is a book that focuses on the music and not the technology or the historical context. So, rather than have a section on score following, with Boulez’ Anthèmes given as an example, let’s have a case study on Anthèmes with score following discussed as one of the many techniques used in the piece.

I’d also like to see discussion about the day-to-day practicalities of running concerts with live electronics. How does the process work? How is the electronics rehearsed? What should a technical rider look like? How do/should composers describe electronics in their scores? How is it notated (if at all)? What materials do publishers hold and how do we use them? How do works transfer between venues? Who maintains the electronics and how?

Also…

examples of ‘typical’ live electronics configurations
common combinations of processing — common musical idioms
approaches to the problems of live electronics and musical time
how to create effective electronics configurations — foreground/background
how to create effective sense of interaction for the performer and audience
electronics and musical style — composers and ‘schools’
scale — differences in approach between small-scale and large-scale works

I guess I’m looking for something between Introducing Music (Karolyi) and Designing Sound (Farnell), at least in spirit.

Maybe I’ll sketch out a chapter outline in a future post…

Max, Integra and the Turing Tar Pit

Thu, 07 Apr 2011 15:31:58 +0100

With Integra Live, we’re trying to make live sound processing simple for musicians. We achieve this by hiding the complexity of signal processing from our users and exposing only musically-useful controls. This technique is known as abstraction because the user-facing representation is more abstract than that used in the underlying software.

Integra Live modules — simple-to-use, but the implementation details are hidden:

The problem with abstraction is that it limits possibilities. Audio programming environments like Pure Data and Max, remove this limitation by providing low-level processing objects that allow the user to make custom abstractions. This means that users coming from a Max or Pd background often find Integra Live constraining. Conversely, Max represents a Turing Tar Pit, requiring significant technical skill for even basic musical tasks.

Bottom-up approach in Max — the TapDelay~ abstraction is created by connecting low-level objects:

This highlights an interesting design problem: how do we build sound-processing software that makes simple things simple and complex things possible?

My first instinct is that we should invert the Max paradigm and create a top-down approach where high-level modules can be edited through an integrated UI.

Any other thoughts…?