Posted on: April 26, 2018
Overview
This proposal outlines an artist-led and practice-based research project that intends to realise a system through which its users can create personal, interactive and immersive mobile sound art experiences, or interactive soundscapes, that will enhance their perceptions of interior and exterior urban environments. This will be achieved through the use of sonified personal, locative and environmental data, along with processed and unprocessed ambient sonic artefacts. I will outline how this could be achieved using a combination of multi-sensing technologies and interface design, and in relation to contemporary art, cultural and musical concepts. I will also outline how this will build on existing related work and research, and present some of my own research questions that I hope will illustrate the project’s potential beyond that of its immediate cultural and artistic value.
Introduction
Attali (2009) asserts that ‘noise is the source of power’, and that by listening to noise we can better understand ourselves, our situation and even gain insights into possible futures:
“Now we must learn to judge a society more by its sounds, by its art, and by its festivals, than by its statistics. By listening to noise, we can better understand where the folly of men and their calculations is leading us, and what hopes it is still possible to have” (Attali, 2009, p. 19).
This suggests that not only do the sounds, or the noises, of our environment provide the artist with a fertile resource with which to reflect society back at itself, but it also suggests that if we can embed additional content, or superimpose our soundscape with associated information, it could lead to an even more representative sonic record.
This augmentation of our soundscape could include the sonification of non-audible elements such as mobile network signal data, electromagnetic fields and ultrasonic data in order to provide a more detailed document of a specific location by rendering hidden content audible. In this context, it may be useful to think of the soundscape as a sound art version of the painted landscape, but with the benefits of a time-based medium, the ability to represent non-visible elements, along with the superior representational qualities of noise, as advocated by Attali (2009).
This landscape as soundscape analogy is hinted at by Ingold (2011), who also points out that in order for a landscape to be audible it needs to be rendered by a form of sound art or sound recording. We can hear this process in play in related work such as Gaye et al’s (2003) Sonic City, a real-time mobile musical experience that generates music based on the users interaction with their environment, and Monastero et al’s (2016) Wandertroper, a real-time sound based generator to facilitate aesthetic mobile walking experiences.
But this is an analogy that Ingold (2011) takes issue with in regards to Schafer’s (1993) soundscape concept. Ingold (2011) suggests that sound should be compared to light rather than vision, citing a potential confusion of the object and the medium of perception.
As Ingold (2011) points out, we do not refer to landscapes as lightscapes, therefore, from a sound arts perspective, why can’t we paint a representation of a location in sound or, rather, paint a landscape with sound? The definition of a soundscape as a ‘sonic representation of a landscape’ would be a more useful definition within the context of this project.
If we extend the concept of the augmented soundscape to include sonified personal data, we can, perhaps, begin to generate interesting and interactive experiences through the process of documentation and recording, which includes a representation of the human experience within the soundscape. It is this relationship between space, place and experience, argue Altavilla & Tanaka (2012), that form one of the key ideas behind acoustic ecology theory.
This personalised sound art performance with one’s own data, when combined with the soundscape, also offers exciting opportunities for the generation of new musical experiences. Both Cage (1961) and Attali (2009) point towards the empowerment of the listener as a performer, and the embracing of the indeterminate nature of found sounds, or noises, as being key compositional factors in the realisation of new musical experiences.
It is envisaged that this project will involve detailed research and development into the effective capture of environmental and personal data for the purposes of its sonification and inclusion within the soundscape, along with the real-time audio processing of ambient sonic artifacts.
I expect this process of research and development to build upon many of the techniques described by Gaye et al. (2003) in a closely related project entitled: Sonic City: The Urban Environment as a Musical Interface.
Sonic City was realised in an era prior to the smartphone, and relied upon a laptop based technical solution with interfaces, micro-controllers, microphones, various sensors and their connecting wires, which, as Gaye et al. (2003) point out, hindered both usability, long-term testing and technical reliability:
“However, for testing aspects of everyday, long-term use and misuse, the current prototype is too limited. It has too many wires and hardware components to carry around, a noticeable latency in sound processing and control, and is not modular or robust enough” (Gaye et al., 2003, p.114).
It is therefore anticipated that a smartphone based solution would address many of these issues and enable long-term use, testing and evaluation via a more accessible, mobile and widely available technology with greater processing capabilities. Such a solution could also make use of the many sensors built into modern smartphones, which would provide more options for the input of both user, and environmental data into the system.
Although it is expected that the input of data into the system will be mainly facilitated by multi-sensory smartphone technology, this project also has the capability of further extending the research provided by Gaye et al. (2003) through the development of a specifically designed wearable interface.
Such an interface could extend the capabilities of Gaye et al’s (2003) ‘Testing jacket’ by enabling the system with binaural recording and real-time binaural signal processing and the better positioning and use of sensors overall, specifically in relation to capturing the spatial location of data sources, and their respective position within the soundscape. This approach would result in a marked increase in the granularity of the data, and the fidelity of the audio captured, and therefore offer a significant increase in options in relation to its mapping, processing and sonification.
Research questions
Based on the outline above, it therefore appears that this research project has the potential to provide some answers to, or contribute to a greater understanding around, the following research questions:
Research approach
I aim to take a artist-led and practice-based approach to my research, where artistic practice can provide research data obtained from the ethnographic study of audience and participant interaction with deployed mobile sound art experiences, installations and technologies.
Evaluation of this data, which would include the results of both qualitative and quantitative approaches (for example: observations, surveys, semi-structured interviews, workshops, videos, field notes and data logs), can then inform my ongoing research and contribute to an understanding of how the deployed technologies and experiences perform ‘in the wild’ with real audiences and participants.
Potential theories and frameworks can then be formulated and applied to future technological and experiential deployments which would be grounded in the effective analysis of the results of recognised research methodologies. This approach is illustrated in Benford & Giannachi (2012) and essentially forms a cyclical process where practice contributes to theory, and theory, in turn, contributes to practice (Figure 1).
Figure 1. The artist-led research process as outlined by Benford & Giannachi (2012).
Additionally, as the realisation of research and development projects as part of this PhD will require working in both the Arts and in Human Computer Interaction (HCI), it is possible that learning can take place not just from participants, but also from my own repeated experiences with the projects, and via a process of self-evaluation, informed by the dual nature of the practice. A similar complimentary approach is identified by Taylor et al. (2011), who also suggest that the personal experiences of the researcher as participant play an important part within this cyclical and iterative design process as it can help, in combination with the analysis of each design iteration, to identify emerging and embedded trends that could potentially compromise either artistic integrity or effective HCI practice. Additionally, Taylor et al. (2011) point out that this dual design and participatory role can lead to dialogue and exchange with users that would ordinarily be absent, and can gain insights that may be unobtainable in a traditional design process, both of which having the potential to drive development forward.
Examples of this artist-led and practice-based approach can be found in related works such as: Hazzard (2016) Guidelines for Composing Locative Soundtracks, Lacey (2016) Sonic Rupture: A Practice-led Approach to Urban Soundscape Design, Taylor et al. (2011) Designing from within: humanaquarium, and in Chaparro & Duenas (2015) Psychogeographical Sound-drift.
In Gaye et al. (2003) Sonic City: The Urban Environment as a Musical Interface, the authors describe a ‘multi-disciplinary and iterative development process [where] ethnographic studies, scenarios, and workshops provided insight into the user experiences’ (Gaye et al. 2003).
It should be mentioned that this approach is not without its challenges. Developing installation based, and mobile artistic experiences in the real-world, for real audiences, will most often require working to real-world performance schedules, venue and festival timetables, and other time constraints and deadlines associated with possible platforms that can be utilised to deploy such projects ‘in the wild’. This potential issue is identified in Benford et al. (2013). In relation to this, Taylor et al. (2011) suggest that this may, at times, require compromise, and the complexity of a specific design iteration may need to be adapted to facilitate an effective deployment, which effectively constitutes a trade-off between the ease and importance of the deployment and the speed of the project’s development.
Another consideration, concerning the dual role of practitioner and researcher, is the Importance of clarifying which role is active when involved in verbal or written communication regarding the project, a point which is mentioned by both Hazzard (2016) and Jacobs et al. (2013).
Additionally, Benford et al. (2013) also identify issues pertaining to the effective ethnographic study of, in particular mobile art experiences, but also art experiences in general, where the presence of researcher may compromise either the artistic integrity of the work, or the immersion of the participant within the experience. In these cases Benford et al. (2013) recognise the need for the direct recruitment of participants, and for the direct participation of the ethnographer.
Indeed, this ‘artist as researcher’ approach is described in Benford et al. (2013) as having three distinct perspectives: practicing, studying and theorizing, through which the individual will need to navigate and adopt appropriate positions and ‘shifts in perspectives’ to suit relevant activities.
Interestingly, Benford et al. (2013) also predict that this ‘artist as researcher’ approach may become more popular as artists engage in interdisciplinary PhD’s, and specifically those engaged with HCI development.
The advantages of this approach, as identified by Benford et al. (2013), appear to support the research questions outlined previously, specifically those regarding the projects potential contribution to future sound art practice, and how grounded theoretical findings may be applied in a wider context.
It is envisaged that the personal data element would take the form of sensor data captured from a user’s personal device, or an associated peripheral, for use within the construction of the augmented soundscape. For example, this personal data could take the form of the GPS coordinates of the user’s current location. This data could be collected by an application running on the user’s smartphone and then utilised by the system in the construction of the experience.
In addition to this, it is also envisaged that certain aspects of the soundscape may be constructed from hidden, or non-audible, location based data assets such as mobile network signal data, electromagnetic field (EMF) data and ultrasonic data. Although arguably contextual data, rather than personal data, one could argue that this type of data becomes personal, and forms part of the users ‘data-footprint’ as a user interacts with it and effects it in relation to their location. Gaye et al. (2003) suggest such data sources as a possible resource for sonification, and future study, in relation to location based and generative musical systems.
As this project essentially aims to provide a framework, and subsequently an effective system for realising the generation of personally relevant augmented soundscapes, it also provides an opportunity for the recording, archiving and retrieval of such assets for a variety of potential uses. These could include secondary, passive, though still immersive, sound art experiences, location based data and acoustic analysis, or sonic ecology archiving for historical or other research related activities.
Such a system would rely on the effective semantic tagging and storage of the audio assets, both at the point of capture, as highlighted by Benford & Giannachi (2012) in relation to the historical trajectory, and within a database structure, in order to geographically locate its source, and to enable the search and retrieval of specific assets based on their contents. This relevance to the FAST program also extends to the potential use of FAST associated technologies, such as Semantic Player (Thalmann, 2015) as a possible framework, or point of inspiration, for content delivery, creation and retrieval within the context of this proposed project.
The ability to record and retrieve the soundscape assets not only represents the historical trajectory within the experience, which Benford & Giannachi (2012) recognise as an important and constituent part of an interactive experience, but also enables the development of personalised individual, and user community, soundscape and data maps, or personal datascapes. Which could present us with an archive of personal data and soundscape experiences, representative of a life in data, or multiple lives in data.
Ethical considerations
Even at this early stage, various ethically problematic features of the project can easily be identified. Two such examples would be: the use of personal microphones, and the collection of personal locative data. Whilst the latter can be dealt with through a process of effective anonymisation, the use of personal microphones would appear to require prior consent, and clear and transparent communication to potential participants regarding its role.
Although there are the usual considerations concerning compliance with the Data Protection Act 1998, and School of Computer Science ethics, there are additional ethical challenges which are a consequence of the proposed artist-led and practice-based research approach.
This approach effectively proposes using artistic practice as a vehicle for research, and Benford et al. (2015) outline some of the challenges of such a research approach, which include: boundaries, consent, withdrawal, data and integrity.
Each of these potential challenges can be identified within the context of this proposed project and are detailed as follows:
Boundaries: Defining the boundaries of participation and performance, which can determine where the boundaries of consent lie, specifically with regards to publically accessible installation environments, and potentially multi-user mobile environments.
Consent: Where and when should consent be sought, and what form it should take? The answer to this question needs to be considered in relation to both research and artistic integrity.
Withdrawal: Facilitating the right of withdrawal within immersive and interactive environments. How should this be dealt with, what provisions are in place to enable users to exercise this right, and how will it effect other participants, particularly in a multi-user environment?
Data: Ethical issues relating to the use of personal data are evident within the context of the immediate user environment, but perhaps even more so in relation to the historical trajectory, and the potential ongoing archiving and remote delivery of media assets produced through the use of personal data.
Integrity: Maintaining the integrity of the art and the research in decision making, and identifying the responsibilities of the two differing positions in relation to each other.
It should perhaps be noted that these specific challenges are present alongside the overarching challenges and implications presented by existing ethical approval processes in relation to experimental and practice-based research approaches, and more specifically research projects that are going to develop through each design iteration.
References
Altavilla, A., & Tanaka, A. (2012). The Quiet Walk: Sonic Memories and Mobile Cartography. Proc. of Sound and Music Computing Conference (SMC), (January 2012), 157–162.
Attali, J. (2009). Noise: The Political Economy of Music. Minneapolis: University of Minnesota Press
Benford, S., Adams, M., Farr, J. R., Tandavanitj, N., Jennings, K., Greenhalgh, C., … Giannachi, G. (2015). The Ethical Implications of HCI’s Turn to the Cultural. ACM Transactions on Computer-Human Interaction, 22(5), 1–37.
Benford, S., Adams, M., Tandavanitj, N., Row Farr, J., Greenhalgh, C., Crabtree, A., … Giannachi, G. (2013). Performance-Led Research in the Wild. ACM Transactions on Computer-Human Interaction, 20(3), 1–22.
Benford, S., & Giannachi, G. (2012). Interaction as performance. Interactions, 19(3), 38.
Cage, J. (1961). Silence : lectures and writings. Middletown, Conn. : Wesleyan University Press.
Chaparro, I., & Duenas, R. (2015). Psychogeographical Sound-drift. Proceedings of CHI 2015, 187–188.
Gaye, L., Mazé, R., & Holmquist, L. (2003). Sonic city: the urban environment as a musical interface. Proceedings of the International Conference on New Interfaces for Musical Expression (NIME-03), 109–115.
Hazzard, A. (2016). Guidelines for Composing Locative Soundtracks. (Doctoral Thesis)
Ingold, T. (2011) Being Alive: Essays on Movement, Knowledge and Description. Oxford: Routledge.
Jacobs, Rachel, Steve Benford, Mark Selby, Michael Golembewski, Dominic Price, and Gabriella Giannachi. 2013. ‘A Conversation between Trees: What Data Feels like in the Forest’. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 129–38. ACM.
Lacey, J. (2016). Sonic Rupture: A Practice-led Approach to Urban Soundscape Design. London: Bloomsbury.
Monastero, B., McGookin, D., & Torre, G. (2016). Wandertroper: Supporting Aesthetic Engagement with Everyday Surroundings Through Soundscape Augmentation. Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, 129–140.
Murray Schafer. (1993). The soundscape: Our sonic environment and the tuning of the world. Inner Traditions/Bear & Co.
Taylor, R., Schofield, G., Shearer, J., Wallace, J., Wright, P., Boulanger, P., & Olivier, P. (2011). Designing from within: humanaquarium. Proceedings of CHI 2011, 1855–1864.
Thalmann, F. (2015) DMO Designer and Semantic Player. Retrieved from www.semanticaudio.ac.uk/demonstrators/d4/
Posted on: April 26, 2018
Overview This proposal outlines an artist-led and practice-based research project that intends to realise a system through which its users can create personal, interactive and immersive mobile sound art experiences, or interactive soundscapes, that will enhance their perceptions of interior and exterior urban environments. This will be achieved through the use of sonified personal, locative and environmental data, along with processed and unprocessed ambient sonic artefacts. I will outline how this could be achieved using… read more
Posted on: April 7, 2016
This paper by Takashi Ikegami and Takashi IIashimoto is proving to be a key point of reference regarding my research into the act of destruction as a creative process and the aesthetics of malfunction and destructive, chance based actions. With regards to their experiments with self-replicating tape loop systems Ikegami and IIashimoto interestingly distinguish the difference between ”active mutation” and ”passive mutation”. The former being ”deterministic mutation” caused by the technology (in this case the tape… read more