Jamie Bullock

Musician-centred Design

Software as medium

In live electronic music, software mediates the experience of composition and performance. Software is the medium through which ideas are developed and structured, it is the medium through which sound is generated and processed, and it is the medium through which sound is controlled. The nature and design of the software is therefore critical to an understanding of the ways in which it mediates user experience. Well-designed software can lead to a pleasant and productive experience, poorly-designed software can lead to frustration and rejection.

Max and the myth of neutral software

It’s not just a question of learning and software literacy. It’s true that the notion of ‘intuitive’ software is a myth, but software that draws upon users’ existing experiences has been shown to be more learnable and therefore more usable. These references to existing experience are called affordances. Affordances can be used to introduce novelty by calling the user to a given action. Software can also be made more usable by introducing assumptions about how the software will be used. These assumptions can be embodied in structured workflows, which mediate the users’ experiences in such a way that it reinforces their intentions.

The primary means for people to work with live electronic music is currently Max by Cycling 74. Max is a tour de force of software engineering, and incredibly powerful on a number of levels. It was originally written in the mid-1980’s as means to control hardware-based audio processing systems. It has since evolved to become very broad in scope, the current website says:

Max gives you the parts to create unique sounds, stunning visuals, and engaging interactive media. These parts are called ‘objects’ – visual boxes that contain tiny programs to do something specific. Each object does something different. Some make noises, some make video effects, others just do simple calculations or make decisions. In Max you add objects to a visual canvas and connect them together with patchcords. You can use as many as you like. By combining objects, you create interactive and unique software without ever writing any code (you can do that too if you really want to). Just connect.

Max makes few assumptions about what it will be used for beyond providing a low-level dataflow and DSP patching language for the creation of anything related to sonic and visual media. Max therefore presents itself as a ‘neutral’ tool for creative work, a tool where ‘the only limit is your imagination’. However, in reality Max has deeply embedded in it, the non-neutral assumptions of a software development workflow. Max is actually an advanced integrated development environment (IDE) for programming multimedia interaction and processing. Far from being a neutral tool, Max mediates the experience of the musician by imposing the workflow, prerequisites and aesthetics of software development on their user experience. With Max, the musician is programmer, instrument maker, designer and composer all in one. Many would argue this to be a positive trait, and in some respects it can be.

Musician-centred design

I believe that musicians need and deserve software that is better tailored to their specific requirements. They need software that has embedded in it, the assumptions of the musician, not the assumptions of the programmer. I call this musician-centred design. The way in which the software mediates the experience of the musician should be centred around the existing knowledge, practices and thought processes of composers and performers. Workflows should be structured around achieving the most common tasks quickly and easily. Interactive music technology has been through many experimental phases, we now know the creative possibilities are ‘endless’. It’s time to consolidate these possibilities and make them more accessible and more musically meaningful.

Some aims of musician-centred software:

Software should encode the assumptions of the musician in how it operates
Workflows should be structured around musical tasks
Software should draw upon musicians’ existing experience
Software should employ innovative UI design to achieve its ends

Conclusions

Max is a wonderful tool for the technically literate who want to employ creative coding to achieve their goals. However, there are many musicians for whom Max is simply too distant from their existing experiences to even approach. I believe the central challenge ahead for music technology research is to investigate and develop new user interfaces that are musician-centred. I see many possibilities here, and feel we haven’t started to scratch the surface in solving these problems. I don’t have the solutions yet, but I believe that collaboration between music technology researchers and designers should be the starting point. Comments welcome…

Published: March, 2012.

Bret Victor talks about his principle that “creators need an immediate connection with what they create”.

This is brilliant stuff. Skip to 12 minutes if you don’t have much time.

Source: vimeo.com

Published: February, 2012.

Jonathan Harvey: total immersion

Seven years ago, my colleague Lamberto Coccioli and I wrote a research paper for the International Computer Music Conference outlining our intentions to develop a new software version of the electronics in Jonathan Harvey’s Madonna of Winter and Spring. Last night the system was used for the first time in public performance at the BBC Symphony Orchestra’s ‘total immersion’ event at the Barbican Centre in London.

Madonna of Winter and Spring is an incredible piece, featuring Symphony Orchestra, amplification, ring modulation, reverb, dual panning, Yamaha TX816, Yamaha DX1 and Emulator II sampler. However, these synthesis components have made the piece cumbersome to perform, with hardware versions being rare and unwieldy to setup and software versions requiring complex combinations of commercial software, some of which is already obsolete.

The new version we have developed as part of the EU-funded Integra Project, attempts to make the electronics for the piece both more sustainable and more performable. It achieves this through the use of a fully open source, entirely software based system, which simple to setup and use. This system runs on two computers and consists of:

the Hexter DX7 emulation plugin by Sean Bolton
a new external for the Pure Data environment, for hosting DSSI plugins
a new software sampler implemented entirely in Pure Data
ring modulation and panning provided by Integra Live

In the Barbican performance last night, the panning was controlled by an iPad running TouchOSC and communicating with Integra Live via wireless MIDI.

The performance was played in its entirety on BBC Radio 3’s Hear and Now programme. Moments where the synths are prominent include 1:01 - 1:07 and 1:17 - 1:20, although synthesis and electronics appear throughout.

Working with Sound Intermedia and the the BBC Symphony Orchestra this week has been a delight. It has been both exciting and rewarding to see the results of seven years’ work come to fruition. To my knowledge, this marks the first ever use of open source DX7 emulation software for a large scale orchestral work with electronics. The materials we have developed now reside with Harvey’s publishers, Faber Music, and will be used in all future performances.

This is a great achievement for the future performability of works involving DX7-based synthesis and I’d like to thank all who’ve made this possible, specifically Sean Bolton, Miller Puckette, Lamberto Coccioli, Dag Henning Kalvøy and James Dooley. At this moment, I am very proud.

Published: January, 2012.

Thoughts on #codingforkids

This week, government minister Michael Gove announced government plans proposing that current school ICT is replaced by a new Computer Science GCSE. I have no actual numbers for this, but my guess is that most people think this is a great idea. Why wouldn’t we want kids to learn to code?

The tricky part comes when we start to dig into the detail. What will the new curriculum contain? How will it be taught? Who will teach it?

One thing I am clear about is that the choice of programming language should be left to individual teachers and pupils.

What I’d like to see the new curriculum achieve:

sharpen creative and analytical skills through projects and play
understand abstraction and think in an abstracted way
self-teaching through experimentation and research
learn to modularise and classify problems and solutions
understand algorithms, data structures, interface, and state

This is what I personally understand as ‘computational thinking’.

The end results, be it an mobile app, a game, or a robot controller and the languages used, should result from the skills and interests of teachers and students.

Published: January, 2012.

Vision: the future of music technology…

I read with great interest Bret Victor’s rant on the future of interaction design. The starting point for his post is Microsoft’s ‘Productivity Future Vision’, which is intended to be utopian, but actually gives a dystopian and de-humanising portrayal of future interaction.

Bret’s response to the video is inspiring, particularly his definition of tools:

A tool addresses human needs by amplifying human capabilities.

Furthermore:

A tool converts what we can do into what we want to do.

Reading this has left me thinking about how I can apply this concept in my own field of music technology research. Traditional musical instruments do the job beautifully: they amplify human gestures, ideas and emotions. Projects I’ve been working on recently have the aim of bringing together music and digital technology, but I feel we are still a long way from creating truly musically-meaningful tools.

I’m not really sure how we’ll get there, but my vision is that we should create digital tools that enable musicians to harness the transformative power of technology through already-learned expressive semantics. Current tools work the other way, and force musicians to understand the semantics of specific technology.

What’s your vision for the future?

*Hammer Images used by kind permission of Bret Victor

Published: November, 2011.

What makes something a musical instrument?

I’ve been thinking about this question since my final year as an undergraduate, and explored it in some detail in my M.Phil. thesis. I’ve also blogged about the subject, and discussed it many times. I’m now approaching a kind of opaque definition:

“If we take two performances of a piece, played on the same instrument, with identical articulation, and subtract one performance from the other, we’re left with the instrument”

In other words, to be called a musical instrument something must be capable of nuance.

Tone arm

Mechanical and analogue electronic instruments, are broadly predictable in their behaviour, but at a detailed level they constantly produce small, unpredictable variations in tone and response even with the same playing techniques.

With digital instruments, this capability for nuance and unpredictability isn’t inherent and therefore needs to be added artificially. For me, providing these nuances of interaction remains one of the key challenges of digital instrument design.

*Image by Soophoo from Wikimedia commons

Published: October, 2011.

Thoughts on sustainability

Over the last 7 years at least some part of my professional work has involved dealing with the issue of ‘sustainability’ in music technology. This has largely centred on the question of making musical works with live electronics more sustainable, for instance, by storing data in an application-neutral format or developing software versions of legacy hardware systems.

Stockhausen Electronics

However, now I’m coming to the end of a six year research project, which has sustainability at its core, I’m starting to question whether we have achieved our sustainability goals…

What is sustainability?

The first thing we need to acknowledge is that this is a sliding scale. It’s not a question of whether something is sustainable or not, but rather how sustainable. Whilst I previously viewed this as a purely technical matter (e.g. binary file formats are less sustainable than text-based formats), I’m starting to look at the problem along a number of separate dimensions, namely: openness, funding and acceptance. A truly sustainable project needs all three.

Sustainability Dimensions

Openness can be achieved through open licensing and the use of open standards. Funding can potentially be gained through academic or public research funding as well as commercialisation.

Possibly the most important, but also difficult aspect of sustainability to fulfil is ‘acceptance’. In order to be sustainable, software, protocols and standards need to be accepted by communities that use them. Without a community of users, contributors, supporters and developers who ‘accept’ it, a project ultimately becomes redundant.

Published: September, 2011.

How Does ‘Incremental’ Define Apple?

joewilcox:

I’d like to discuss how Apple innovates, which I understand very well. I posted about Apple’s incremental product strategy last September at Apple Watch: “Apple Demands a High Price to Be Cool.”

The pattern is consistent: Apple launches a “one more thing” product with modest hardware features but something else nevertheless killer—something people want. During the launch, Apple CEO Steve Jobs performs his marketing magic, demonstrating how this “one more thing” product will make peoples’ lives better.

Read More

Source: joewilcox

Published: June, 2011.

A book about the practice of live electronic music

…basically there isn’t one.

There seem to three classes of book referring to the subject, the most common are technical books about the theory and techniques of digital audio processing.

Examples include:

Computer Music (Dodge)
Elements of Computer Music (Moore)
Introduction to Computer Music (Collins)
The Computer Music Tutorial (Roads)
Theory and Techniques of Electronic Music (Puckette)

These books are remarkably similar in terms of structure and content. Fundamental digital audio concepts are introduced (bits and bytes), amplitude, periodicity, digital filter theory and delays, frequency domain processing (FFT), synthesis. Additionally psychoacoustics, audio programming, and control processing are usually covered to some extent. Sometimes theory is contextualised with reference to specific musical examples.

Dodge - Computer Music

Additionally, there are discursive books about history and aesthetics, e.g.

Living Electronic Music (Emmerson)
Electronic and Computer Music (Manning)
Electronic and Experimental Music (Holmes)

These books also share common structure and content. Historical overview starting from Cage’s Imaginary Landscape #1, presentation of the main components of an electronics setup — the microphone to the loudspeaker — the evolution of digital processing, all presented from what might be considered a ‘musicological’ perspective.

Finally, we have books that deal with a specific system or software, e.g.

The Csound Book (Boulanger)
The SuperCollider Book (Wilson, Cottle, Collins)
Electronic Music and Sound Design - Theory and Practice with Max/MSP (Cipriani, Giri)
Loadbang: Programming Electronic Music with Pure Data (Kreidler)

These books take the presentation of techniques in the given software as their starting point, providing theoretical background and practical examples as needed. Techniques are sometimes presented in the context of a given musical work.

What I’m missing is a book that focuses on the music and not the technology or the historical context. So, rather than have a section on score following, with Boulez’ Anthèmes given as an example, let’s have a case study on Anthèmes with score following discussed as one of the many techniques used in the piece.

I’d also like to see discussion about the day-to-day practicalities of running concerts with live electronics. How does the process work? How is the electronics rehearsed? What should a technical rider look like? How do/should composers describe electronics in their scores? How is it notated (if at all)? What materials do publishers hold and how do we use them? How do works transfer between venues? Who maintains the electronics and how?

Also…

examples of ‘typical’ live electronics configurations
common combinations of processing — common musical idioms
approaches to the problems of live electronics and musical time
how to create effective electronics configurations — foreground/background
how to create effective sense of interaction for the performer and audience
electronics and musical style — composers and ‘schools’
scale — differences in approach between small-scale and large-scale works

I guess I’m looking for something between Introducing Music (Karolyi) and Designing Sound (Farnell), at least in spirit.

Maybe I’ll sketch out a chapter outline in a future post…

Published: May, 2011.

Max, Integra and the Turing Tar Pit

With Integra Live, we’re trying to make live sound processing simple for musicians. We achieve this by hiding the complexity of signal processing from our users and exposing only musically-useful controls. This technique is known as abstraction because the user-facing representation is more abstract than that used in the underlying software.

Integra Live modules — simple-to-use, but the implementation details are hidden:

Integra TapDelay

The problem with abstraction is that it limits possibilities. Audio programming environments like Pure Data and Max, remove this limitation by providing low-level processing objects that allow the user to make custom abstractions. This means that users coming from a Max or Pd background often find Integra Live constraining. Conversely, Max represents a Turing Tar Pit, requiring significant technical skill for even basic musical tasks.

Bottom-up approach in Max — the TapDelay~ abstraction is created by connecting low-level objects:

Max abstraction

This highlights an interesting design problem: how do we build sound-processing software that makes simple things simple and complex things possible?

My first instinct is that we should invert the Max paradigm and create a top-down approach where high-level modules can be edited through an integrated UI.

Any other thoughts…?

Published: April, 2011.