In September 2023, I had the honor of delivering a presentation to the music division of the
Norwegian Association of Theatres and Orchestras (NTO). The central theme of my talk was the
convergence of music and artificial intelligence (AI), with a particular emphasis on the
untapped opportunities this burgeoning field offers to the worlds of art and music, as
opposed to merely delineating its challenges.
I initiated the discussion by offering a rudimentary introduction to AI and machine learning.
Subsequently, I focused on two specific methodologies, classification and generation, that I
deemed most pertinent to this specialized audience. To elucidate the core principles underlying
these areas, I used two straightforward applications that I had developed for demonstration purposes.
Illustration image by Dall-E
I then proceeded to explore existing services and applications that hold particular relevance for choirs and orchestras. I posited that although many commercial AI solutions in the musical domain are still in their infancy and often yield less-than-satisfactory results—especially for those with a nuanced understanding of music—there exists a plethora of intriguing possibilities for professional music ensembles. These opportunities span various facets of musical creation, ranging from composition and performance to engaging audiences in novel ways.
In conclusion, I highlighted the potential of advanced language models like ChatGPT as valuable collaborators in the conceptualization and development of new artistic projects and ideas.
By focusing on the yet-to-be-realized possibilities that AI presents, I aimed to provide a balanced perspective that encourages proactive engagement with these emerging technologies.
The slides from the talk (in Norwegian) can be downloaded here.
During my research term at Infomus, Casa Paganini, University of Genoa, Italy, I started a collaboration with Sanket
Sabharwal analysing a set of motion capture recordings of two dancers I did at the ZHdK in Zürich, fall 2021.
This collaboration resulted in a paper that Sabharwal presented at the AudioMostly'23 conference in Edinburgh,
since I didn't receive enough funding from my university to be able to go there myself.
Abstract: In this paper, we will present a pilot study that explores the relationship between music and
movement in dance phrases spontaneously choreographed to follow phrases of electroacoustic music.
Motion capture recordings from the dance phrases were analyzed to get measurements of contraction-expansion
and kinematic features, and the temporal location of the peaks of these measurements was subsequently compared
with the peaks of a set of audio features analyzed from the musical phrases. The analyses suggest that the
dancers variably accentuate their movements to the peaks or accents in the music. The paper discusses the
findings in addition to possible improvements of the research design in further studies.
The paper is published by the ACM Digital Library and can be read and downloaded here.
Since 2011, I've had the privilege of collaborating with Robert Wechsler, the founder and inventor of the MotionComposer project. Together, we've been developing a musical environment that uses computer vision technology and audio software to convert movement into music. This environment is designed to be accessible to individuals with varying abilities.
The onset of the COVID-19 pandemic has inevitably led to some delays in our development timeline. Specifically, the "Particles" sound environment, initially developed for MotionComposer 2.0, was not incorporated into version 3.0 as planned. However, following two pivotal workshops—one in Genoa in the spring of 2022 and another in Trondheim in 2023, Particles has officially been reintroduced as one of the six key sound environments available.
I am delighted to announce that this updated version of Particles is now featured on the official MotionComposer website. It comes complete with an enthralling demonstration performed by a dancer, which beautifully showcases the capabilities and creative possibilities of this interactive musical environment.
In the spring of 2023 I did a series of workshops with dancer and choreographer Johanna Ciampa from Montana (US)
developing an interactive performance entitled Moving, Listening and Being: An Iterative Process.
The performance featured 3D sonic landscapes with elements controlled by the dancer with the use of
body-worn sensors and an interactive digital instrument. The workshops were a part of Ciampa's Master
project, which she submitted at the international Choreomundus program Dance Knowledge, Practice and
Heritage spring/summer'23. The topic of the performance was how human activities pose threats to aquatic
life, using the coral as point of focus for the dramaturgy of the performance. The main ideas and concept
in the project were Ciampa's, and my role was to be a co-creator and off-stage co-performer with
responsibility for the interactive audio technologies involved. In the performance, Ciampa embodied
the character of an adult-stage coral in a fjord. She then controlled an interactive instrument intended
to portrait an unhealthy coral gradually coming back to good health. Accompanying the coral sounds were
sounds of differently sized ships as well as aquatic life - the former thanks to a set of bioacoustic
recordings from the Norwegian Polar Institute in Tromsø.
The performance took place in Music Technology's Portal (Fjordgt.1 campus) at NTNU in front of a small, invited audience.
We used the 20.2 speaker setup I have put up there, using MIDAS-M32 mixer and a Mac Mini running Reaper with
ICST ambisonics fourth order plugins to spatialize the sounds. My interactive instruments were integrated
in a VST plugin made with Csound and Cabbage. Three NGIMU sensors (x-io) were used to track Ciampa's
movements.
Interactive Digital Art & Societal Health Conference is a new conference organized by
Jin Hyun Kim and Marcello Lusanna
at Humboldt University of Berlin taking place December 2-3. The organizer
states that the motivation of organizing this conference is to promote the
usage of interactive digital art in healthcare and other daily life contexts where societal health plays a role both on a
theoretical and practical level. The conference gathered scholars from many diciplines including music therapy, psychotherapy,
musicology, music technology, HCI, philosophy, anthropology and somatic interactive design.
I was honored to be invited to do at talk at the conference, which took place at the Kunsthaus KuLe. The title of my
talk was Potential for health and well-being effects in interactive sonification of movements.
Abstract:
This paper argues how interactive digital art combining aspects from dance and music
through sensor and sound generation technologies has the potential of promoting the health
and well-being of users of all abilities. Dance and music in a more traditional sense are
known to have positive health effects by motivating movement, creative expression and
social interaction. Moreover, they can reduce the risk of physical illness by improving aerobic
capacity, balance, elasticity and coordination as well as mental disorders by reducing stress
and inducing positive emotions. Technologies for interactive sonification of dance movement
combines aspects of dance and music at the same time by translating movement sensor
data into sound and music, and if designed carefully they can have a considerable potential
for many of the same health and well-being effects that music and dance have separately.
For example, they often tend to imply feedback loops where a movement first will generate a
sound, and then in turn can motivate the user to move as a response to the sound. Through
the possibilities of generating and controlling musical parameters on higher levels,
interactive sonification of dance movement can afford degrees of inclusion that surpass that
of e.g. traditional instruments, which often require years of training to master. This is also
why interactive movement sonification can offer people with different kinds of disabilities a
way of playing music through their movements, and thereby a way of generating positive
experiences of basic psychological need satisfaction as well as other elements of well-being
like positive emotions, engagement, relationships, meaning, and accomplishment. The paper
will show how two different interactive systems/devices using two sensing technologies both
afford these kinds of positive health and well-being potential, namely the MotionComposer
and the VibraChair, where the author has been engaged in the development of both.
Whereas the former is a well-established product in its third version, the latter is still in its
exploratory stages of development. The paper will discuss and compare different aspects of
the systems/devices, and conclude with some general suggestions for design aiming for
positive health and well-being effects.
"The Portal" is a room at the Music Technology campus in Fjordgt.1, originally designed for the two-campus
program "Music, Communication and Technology" that NTNU and UiO had running from 2018-2022. During the fall
I have been working with a new spatial audio setup for the room. The previous setup was a small ring with eight
Genelec 8020D loudspeakers, about 2.5 meters in diameter, covering only a small part of the room. My aim was to have a setup
with a much larger listening zone (and sweet spot), including also a height dimension. With 16 Genelec 8030A speakers to my disposal,
the solution I chose was to locate the speakers close to the walls in two heights. Although the aim was to have a relatively
symmetric setup, the layout of the room and the equipment in it forced a number of compromises. Still, when testing
the setup with different sound material decoded in up to 5th order ambisonics, the result was quite good. The students
taking the course 'Electronic Music' (MUST3054) at
the NTNU Master's programme in Creative Music Technology, were the first
to compose for the setup, each of them making original compositions in abisonics audio. We were also so lucky as to
have a version of Natasha Barrett's composition
Hidden Values decoded especially for the setup, which was also actively used in the course.
The coordinates for the loudspeaker setup can be downloaded here.
The image shows half of the room and eight of the loudpeakers in the NTNU Portal.
On Saturday the 5th of November 2022, Constellation Stories: Stories of the Night Sky, a
dance performance with original music composed by myself and Madeleine Shapiro and choreography
by Merli V. Guerra, was performed by the Luminarium Dance Company at the Springfield Science
Museum (MA). The composition was made through remote collaboration between Shapiro, contributing
with virtuoso cello improvisation from New York City, and myself, contributing with electronic
processing, adding other types of material and the overall mixing and production, working from
Genova, Italy and Trondheim, Norway. A local newspaper, The Republican, covered the performance, and you can see the story below.
The performance featured dancers Alexa Barreiros, Melenie Diarbekirian, Kate Harbison, Joscelyn E. Hunter,
McKenzie Lani Jones, Jamie Peña, Aniky Salima, and Marissa Stellato.
It was reported that several hundreds of visitors attended the event.
In 2022, I was invited to write an article about MotionComposer in the German journal Pädiatrische praxis
together with Robert Wechsler, Alicia Peñalba, and Stephan Geiger. This is my first publication German and also
in a medical journal.
Abstract (translated to English):
It has been shown that digital music technoloin combination with sensors and carefully written software
software can bring movement, joy and social social interaction in people with a wide range of
people with a wide range of abilities, including children with even children with severe physical or mental
mental disabilities. This article describes the MotionComposer and some of its possible therapeutic
therapeutic applications in sociopaediatric diatric centers.
The article can be downloaded here.
AudioMostly is, as the website proclaims, "an interdisciplinary conference on design and experience of interaction with sound
that prides itself on embracing applied theory and reflective practice". In 2022 the conference was held at the St.Pölten university of applied sciences in Austria during September 6-9.
My submission was about my work at the ICST (ZHdK) in the fall of 2021 and was submitted as a short paper.
That allowed me to do a demo and poster presentation at the conference.
Abstract: The paper describes a work-in-progress exploring the expressive and creative potential of dance phrase onsets and endings
in interactive dance, using an artistic research approach. It briefly delineates the context of the presented work, before describing the
technical setup applied, both in terms of hardware and software. The main part of the paper is concerned with the specific mappings of three
different sections in the performance that the project resulted in. Subsequently, the process and performance are evaluated, including both
the dancer’s feedback and observations by the author. The points from the evaluation are then discussed with reference to relevant research
literature. Findings include that the dancer experienced an increased awareness of beginnings and endings in different sections of the
performance, and that postural adjustments were necessary to make the interaction more robust.
The paper is available for download
here.
In June 2022 I was very proud to receive the notification from the Faculty of Humanities at the NTNU that I had been appointed full professor
in the field of music technology within the research area of experience and expression in embodied music performance
practices involving music technology. The appointment was made on the basis of academic qualifications. The expert
committee, consisting of professors Thor Magnusson, Trond Lossius and Wayne Siegel, noted that some of my work
"indicates additional qualifications of relevance and value within the cross-disciplinary environment at NTNU, operating
between music and arts practices, music technology, audio engineering and musicology".
The SMC-22 conference (Sound and Music Computing) is, as the organizer informus, "a multifaceted event around acoustics, music,
and audio technology". It will take place in Saint-Étienne (France) on June 4-12, but will also have a considerable online program.
The theme of this year's conference is: Music Technology and Design.
I am happy to be presenting a paper entitled Designing interactive sonifications for the exploration of dance phrase edges
at this year's conference. The presentation format is new of the year: 5 minute presentation followed by a hybrid poster session/demo.
Abstract: The paper presents practice-led research where interactive sonifications of dance movements are created so as
to have special emphasis on the movement edges of dance phrases. The paper starts out by discussing to what degree interactive
sonification of movement and the artistic practice of interactive dance have areas of overlap, and whether their contexts and
intentions may be combined to complement each other. A discussion of the concept of salience and how it relates to perceptual
edges in general follows, and more specifically to onsets and endings both of movements and musical objects. In this discussion,
the author also considers the role of accents as well as different degrees of abruptness of changes. The paper subsequently
presents a set of interactive sonifications demonstrating different degrees of salience from very high to
very low. Details related to the technical setup, analysis of movement data and the movement-sound mappings are presented, and the results
are discussed.
If you are interested, check out the project page with accompanying material.
The paper can be downloaded
here.
May 27 and 28 I hosted a workshop at Casa Paganini, Genova,
with dancer and choreographer Cora Gasparotti and musician Giangiacomo
Gallo visiting. Both are aspiring artists from Rome, where Gasparotti just graduated from Accademia Nazionale di Danza with a
Master's thesis about the use of interactive sonification in dance training. Gallo has a background from studies in electroacoustic
composition and sound engineering, and is part of DiacronieLab, a group of musicians in Rome working with music and technology.
At the workshop we explored different technologies such as inertial sensors, EMG armbands (Myo) and stereoscopic 3D video tracking (MotionComposer).
The workshop combined demonstration, discussion and practical exploration and improvisation, involving topics such as:
different sensors' strengths and weaknesses
sensor communication
sensor data treatment methods
movement to sound mappings
audience communication and involvement
Excerpts from the workshop at Casa Paganini, showing dancer/choreographer Cora Gasparotti
We started by looking at the Myo armbands, and how data treatment methods such as scaling, filtering, noise gating and transfer functions
can be used to have the best condition for developing an interactive instrument. Together we tested some simple mappings using the data
from three of the electrodes to control sine tones and band pass filtered white noise, which Cora explored in a little improvisation.
The first section of the video shows the result of the demonstration and discussion of points 1-4 of the list above.
Next, we tested out several of the interactive instruments that I developed
during my stay at the ZHdK fall 2021, applying Myo armbands and NGIMU sensors from x-io (left and right wrists plus torso). I explained
how each of the instruments were related to the composition in which they appeared, Beginnings and Endings Study I, before Cora (and sometimes
Giangiacomo) would explore freely, before making a short improv for the camera (see sections 2-5 of the video).
The second day we moved our attention to the MotionComposer version 3, where we tested out some beta features that will be released later this summer. Although the lighting conditions were far from ideal, we got to test out four of its five environments: Tonality, Drums, Fields and Particles (which I implemented on the device during the workshop with MotionComposer the preceding week). Cora made two videos playing the Tonality environment, one with a chromatic scale, the other with a pentatonic scale.
In February 2022 I was invitated by cello improvisor Madeleine Shapiro
and choreographer Merli V. Guerra to collaborate to make the music
for an upcoming dance piece that Guerra was choreographing for her company Luminarium. During two intensive weeks in February I collaborated remotely -- CoVid-style -- with renowned New York-based cellist, Madeleine Shapiro, to make a 14 minute piece. Madeleine
sent me sections of improvisations on the cello and crotales sounds, which I then would electronically process and use as elements in the composition.
Madeleine would then respond to this and send me new improvisations. I also added sounds that I had recorded myself, for instance, a carpet of lively
conversations from a piazza in Genoa, Italy, and percussion sounds played by retired colleague music technology professor, drummer and percussionist, Carl Haakon Waadeland.
The final mix kept a main cello voice in the centre to capture a live-feeling.
The piece was world premiered the 4th of March 2022 at Rider University (New Jersey, US) with dancers Tanisha Anand, Lex Barreiros, Kaniah Browne, Jossie Hunter, Nevaeh Peaks, Jamie Pena and Vee Williams.
The video shows the full piece from the premiere with audio recorded on site.
Constellation Stories by Luminarium Dance. Choreography Merli Guerra, music by Madeleine Shapiro (cello) and Andreas Bergsland.
As an introduction to Infomus at the University of Genoa I held a hybrid presentation
at Casa Paganini for the research group at the research center entitled Sonification of automated
movement analysis focusing on onsets and endings in interactive dance. The presentation mainly dealt with the interactive
instruments developed during the stay at ZHdK, including the movement anlyses techniques that they were based on.
Demonstration of rarity index with NGIMU sensor.
One of these techniques was a slightly modified implementation of the rarity index , a computational model of saliency developed by the Infomus team.
The presentation also included demonstration videos of the interactive sonifications made at Infomus. These
are videos accompanying two research papers that are forthcoming, but will also be presented at the VIBRA pages soon.
Illustration image by Dall-E
The image shows half of the room and eight of the loudpeakers in the NTNU Portal.
