Title: OLoS: Online Open Source Music
Synopsis: A modular web audio environment where everything—both the music and the underlying code—can be remixed. I will design interfaces that show the underlying processes and invite play. My goal is to foster an open source community for shared learning.
Description: OLoS is an environment to play with code and music. The web makes both more accessible and shareable. I want to provide a forum for the community that is forming around the Web Audio API to share resources through modules that can be combined and remixed. Through user research into the community’s best practices, I’ll develop an API spec for modules that will enable integration into outside projects as well as the OLoS node-based visual interface. My research might lead me to extend a pre-existing project. Nodes will include instruments, effects and algorithms, and UI/control nodes. They can be chained together to design custom environments for music production and interaction. Users can also share/remix environments & compositions.
Research Process:
I will continue conversations with web audio developers, musicians with an interest in code, and creative coders with an interest in audio.
I will experiment and continue dialogs with people who have attempted similar projects that would form community around open source web audio modules, such as:
- Soledad Penadés (Mozilla / Open Music)
- Jordan Santell (Mozilla / Web Audio Components)
- Ben Taylor and Jesse Allison (BRAID: A Web Audio Instrument Builder with Embedded Code Blocks | braid.nexusosc.com)
Through this research, I will attempt to determine what the spec might look like for OLoS modules, and ask for contributions from established members of the Web Audio community. I might find that an existing model, or a hybrid of existing models, is the best way forward because my goal is to unite the community, not to fragment.
I will speak with developers of digital environments that promote computational thinking through tinkering. Examples include:
- Mitch Resnick, creator of Scratch
- Georgia Tech’s Earsketch, a Digital Audio Workstation where loops are controlled by JavaScript and Python code in order to teach concepts such as recursion
- Luke DuBois (Max/MSP/Jitter)
I will continue research into designing digital learning environments. This research may involve readings and videos by:
- Bret Victor – “Media for Thinking the Unthinkable“
- Disney – Donald Duck in Mathemagic land
- Seymour Pappert – Mindstorms
- Code-sharing projects such as Shadertoy, bl.ocks.org, codepen
- Alex Ruthmann, director of NYU’s MusEDLab and researcher into music + computational thinking
- Eric Rosenbaum, whose research explores the intersection of music, play and improvisation
- John Finney and Pamela Burnard (editors), Music Education with Digital Technology
Papers and Articles:
- Proceedings of the First Web Audio Conference at IRCAM, 2015
- Charlie Roberts, Rapid Creation and Publication of Digital Musical Instruments
- Lonce Wyse and Srikumar Subramanian – The Viability of the Web Browser as a Computer Music Platform
- Ethan Hein, Music Games in Education
- Thor Magnusson, Scoring with Code: Composing with algorithmic notation
- Jason Freeman et al, EarSketch: An integrated approach to teaching introductory computer music
- Jesse M. Heines et al, Two Approaches to Interdisciplinary Computing + Music Courses
- Alex Ruthmann et al, Teaching Computational Thinking Through Musical Live Code in Scratch
- Natalie Rusk et al, Origins and Guiding Principles of the Computer Clubhouse
- Resnick, Rethinking Learning in the Digital Age
- Resnick, Scratch: Programming for All
Production:
I will create prototypes that explore different paradigms for code + music, and will also help me better define the audience who would be invited to tinker with these environments. Some of the exercises I might try include:
- Audience: Master’s Students + Professors
- I will work with NYU professors Ken Perlin (developer of Chalk Talk) and Agnieszka Roginska (3D Sound / Digital Signal Theory) to develop interactive examples that illustrate DSP concepts.
- An additional example might be to model the physics of sound by allowing the user to change the material length of a xylophone using a given mathematical equation to achieve their desired musical scale.
- Audience: Data Journalists
- I will create a module that can convert data, such as an HTML table or a CSV, into a Score, using the Lenses open source node-based system for data journalism. The user can set which column in a table should be used for Note, Time, Duration. Set scale, root note, I’ll also be working with Lenses on this project outside of ITP.
- Audience: Web Audio Developers, Creative Coders, Amateur Musicians
- Prototype: Sequencer composed of web components, where the user can tinker with the instruments’ parameters through code.
- Audience: Teenagers in Urban Arts’ Fresh Prep program
- Prototype: Sampler to remix songs about US and Global History for the Regents test prep. I’m already working on this through NYU’s MusEDLab.
- Audience: ~4th grade and up
- Prototype: I will create custom Code Blocks that facilitate music composition within Scratch. For example, I’ll create an Envelope, and a KeyUp, making it easier to play music with the keyboard.
I will probably build at least three of these prototypes. I will test them out on different audiences, and choose one to pursue as the introductory interface for OLoS. I will try to envision what the hook might be for the potential audience, and I will also be using each platform to realize my own musical ideas through code, as I learn by tinkering.