Singing Digital Canvases: cognition en acte, mémoire,créativité et synesthésie

''Singing Digital Canvases: enacted cognition, memory,creativity and synesthesia''

• Auteur / Autrice : Myriam Hammani
• Direction : Amos Fergombe
• Type : Projet de thèse
• Discipline(s) : Arts : plastiques, spectacle, musique, esthétique, sciences et histoire de l'art
• Date : Inscription en doctorat le 23/10/2019
• Etablissement(s) : Valenciennes, Université Polytechnique Hauts-de-France
• Ecole(s) doctorale(s) : École doctorale polytechnique Hauts-de-France (Valenciennes, Nord ; 2021-....)
• Partenaire(s) de recherche : Laboratoire : Laboratoire de recherche sociétés et humanités (Valenciennes, Nord ; 2021-....)

Mots clés

Mots clés libres

Corps chanté

Navigation narrative

Théatre interactif numérique

Interactive singing installations

Environements immersifs

Enacted cinema

Digital humanities

Digital memory

Human computer interaction

Neuro cinema

Digital cultural heritage

Narrative navigation

Résumé

Cette recherche s'intéressera à la manière dont un film interactif recourant au corps (voix et mouvement) et percussion peut aider une personne à développer des aptitudes de synesthésie et de mémoire. Une recherche artistique et scientifique pour l'avancement de la plasticité du cerveau avec l'art numérique et la ‘performance artistique'. Il s'agira d'examiner comment le corps réagit à une installation de film interactif vocal . Comment le corps réagit à un dispositif filmique et sonore avec comme sujet la peinture. Et après réflexion comment l'améliorer et le perfectionner pour susciter des états émotionnels et esthétiques (chez le/la visonneur(euse) en chantant et en étant mobile) . On pourra ainsi s'appuyer sur des expériences artistiques contemporaines et recourir à des dispositifs interactifs et à des données de recherches scientifiques en synesthésie pour composer des paysages sonores et visuels en interactivité avec la voix chantée et le mouvement du corps. Les objectifs de mon doctorat sont de créer et expérimenter des dispositifs filmique interactifs avec la voix et le corps chanté afin d'observer des résultats positifs dans le développement de la plasticité synesthète de leurs cerveau; en utilisant des casques EEG. En introduisant des installations interactives au sein de recherches neurophysiologique et inviter les personnes non artistes et artistes à les expérimenter avec leurs corps et voix, j'espère trouver une méthode de création artistique pour créer des dispositifs efficaces au service du développement personnel et de la recherche neurophysiologique, Contexte Insérer la technologie moderne de l'interactif entre le data numérique pour servir le corps humain à s'épanouir dans un contexte de l'oralité humaine. Ici je travaillerai avec la cosmologie féminine nord africaine dont les motifs géométriques tracés (langage visuel) et l'oralité celtique océanique (car l'océan possède une oralité avec ses animaux marins) pour tisser mes environnements immersifs. L'aspect poétique et esthétique qui s'en dégagera aura effet (je le souhaite) sur la mémoire archetypale du corps chanté de la spectatrice/spectateur. Cette stimulation émotionnelle permettra à l'utilisa(teur)(trice) de réveiller et faire vibrer leurs voix chanté, leurs rythmique interne et leurs mouvements pour naviguer en profondeur le dispositif audiovisuel. Les donnés de cet échange entre le corps humain et l'environnement numérique me permettront d'observer et analyser les retours des utilisateurs et spectateurs et comment l'installation immersive les aides ou pas à exprimer et développer leurs corps chanté et leurs états synesthète.

Abstract : Making interactive art for beauty, emotional expression, entertainment and political statement is how art is often used. However, using art for scientific data research is not as common. I would like to explore and prove how interactive immersive art can play a fundamental role in science research and can be used to trigger dormant parts of the human brain and activate them to develop new neural networks such as synaesthete abilities. Today, my research merges art making with synaesthesia research and memory/learning research to help develop social capacities, musical development, visual awareness and synaesthesia abilities. I am also developing a «how to » guide for artists who would like to make art in collaboration with cognitive scientists for scientific research by showing how step by step my own art piece will take form. Several art projects have been made to invite the viewer to interact with the art piece such as Tony Dove's ‘Artifical Changelings' which is a non linear narrative film played by the movement of the viewer's body. It has a simple 4 step interactive motion sensor and triggers 4 different type scenarios within the same story. I will follow Tony Dove's non linear way of telling a story with my St. Brendan character while he travels through the Ocean on the water and in his dreams. There is also Char Davies' ‘Osmosis' which invites the viewer to put on a sensor suit and virtual glasses to navigate through her 3D virtual environment by breathing in and out and moving forward and back. This piece inspired me to make ‘Weaving a berber rug with the human voice' which weaves itself when a person sings into a microphone. This is a project I made at Iear, Rensselaer polytechnic institute for my master's degree of fine arts. However the piece was limited because it did not react to the pitches and rhythms of the singer, it was basically a video which turned on when the person sang and turned off when the person stopped singing and this gave the illusion that the person was « weaving » a virtual rug with their singing voice. I then discovered the interactive opera Alma Sola of A. Bonardi and C. Zeppenfeld who integrated voice and face mapping interaction within their multimedia performance. It consists of a professional singer up on stage interacting with a computer program while singing which then gave information about her emotional facial state. This piece intrigued me because her facial expressions triggered certain musical compositions already made. Todd Machover's ‘Death and the powers' opera introduced robotic objects interacting with humans and vice versa on stage. What was interesting about these operas is that they use computer programs to react to the human voice and touch. I will use these as guides to work on my own voice system. However I find that their research in body performance art is weak and would like to further investigate in Amos Fergombé's body and performance ritual works and writings. Another immersive piece which uses human history data in interactive spaces to further knowledge and memory is Sarah Kenderdine's works and writings on digital humanities and how to create new futures with hidden memories. Finally there exists a piece made by Cecile Leprado called « Le Triangle d'incertitude» which is a finely sculpted soundscape of ocean oriented activity between humans living off the shores of Bretagne, Spain and Ireland. It creates a mysterious atmosphere which inspires the listener to imagine the place visually. This piece will inspire me to make inticing soundscapes with protools, natural recordings and light sound enhancers adding dimension to my immersive film installation and acting as background sounds accompanying the singing user's voice. Finally, all these art pieces will guide my artistic craft and esthetic choices. These artists and thinkers followed historical data, and their own imagination which assembled chosen sounds, visuals and visions to certain musical moments of their pieces. To continue the trajectory of these artists and thinkers, I will include cognitive science research to develop and design my artistic vision. This piece will also be used to help further cognitive research by inviting their test groups to try it out and observe the brain activity of each user interacting with my piece. The art piece I am making will be an interactive film using the voice as the main trigge. I will invite scientists who work with sound/color synesthesia research. Furthermore I will also contact and ask advise from neuro researcher Hervé Platel, professor of neuropsychology at l'université de Caen. He has worked with the effects of music making on memory and learning brain centers of people.This research can feed my artistic choices while creating the soundtrack for the interactive piece which will create various emotional atmospheres merged with the images inspiring the person to sing. This interactive art film entitled «The Whale's last song » invites people to navigate through it with their singing voice.They will be able to actively watch a film by responding to it with their emotional voice sometimes by singing various pitches, sometimes by immitating the ocean creature sounds they hear within the film. The sounds they make will guide them through the different sections of the film. This project is somewhat like a game in the sens that it is interactive and is based on decision making. But it differs in the sens that there is a film to discover which has a written scenario shuffled with layers and layers of sub scenarios. For this art film, I am choosing 2 types of scenarios, one type is narrative which is St Brendan's journey guiding us through the ocean and time traveling between the 6th century and our time period on the atlantic ocean. This second narrative is about the ocean and her inhabitants. It is a sensual intuitive collage of the ocean and its creatures, at the various levels and depths of ocean life. Part of it is found footage, filmed footage of real actors and 2d/3d animations which I will direct, create and assemble. It will inform the viewer about its atmospheres and inhabitants and its contemporary problems (pollution). In order to watch and hear the film, the participant will have to sing/sound out into a microphone with their voice and drum in a water bassin with their hands in order to navigate through the different layers of the story. Each colors and atmospheric tones of the film will act as emotional tableaus responding to different vocal pitches, volume and frequencies (notes) emitted by the participant. By studying the biofeedback of the singing/sounding out film viewer, it will reveal the combined triggered regions of the brain activated during participatory film « watching » and it will also strenghthen neural pathways between singing and visual watching in the participant's mind making them stronger in those unexplored regions of their mind. The visual memory center /visual cortex will latch on to the auditory cortex of the brain through the soundscapes of the film including the sound of the sea waves, the boat rowing, the voices of birds, whales, wind, fish, sharks, otters, human voices and icebergs. This visual auditory connection will be re-enforced with participatory motor skills such as singing and sounding out into the microphone from the viewer. This participatory voice will help build and shape a new neural pathway in the music making improvisation area of the participant's brain. The act of singing and/or sounding out with voice and drumming while taking in information can also re-enforce the audio visual connection with emotions , colors and sounds relating to the ocean. This is the way a documentary film ougth to be seen, as a sensual experience which triggers the imaginative parts of the mind actively. The way to trigger these parts of the brain while watching a film is to be in a visceral connection with it by making a physical gesture or a vocal sound. In this way my project would be an intricate research between what we see and how we perceive it according to what we do while watching it. Research has shown that fiction narrative stories help develop imagination and brain connectivity. This is why I am using the St. Brendan's voyage narrative story full of adventure, history and vivid environment description. As neuroscientist Gregory Berns writes in his research journal for Emory University ; “Stories shape our lives and in some cases help define a person”. He continues “We want to understand how stories get into your brain, and what they do to it.” What he is researching is brain plasticity which is a term that is used to refer to the continually new connections that are made when we take in information, engage in thinking, and force ourselves to remember things. This is something which will also be developed in my project.