Construction of a Tactile Vibration and Sound Visualization Device to Enhance the Music Experience of Individuals with Hearing Impairment

[Introduction]
Over 25 million individuals in the U.S. face hearing disabilities beyond the age of 12, with nearly two million experiencing severe hearing loss. This project aims to develop an open-platform device that combines tactile vibration and the visualization of sound waves. This innovative solution intends to provide individuals with hearing impairments, including the deaf and elderly, the opportunity to not only hear but also experience and enjoy music.
During the COVID-19 social shutdown, my interest was sparked while playing the violin and recording myself. Upon reviewing the video, I noticed a sine-curve-like wave motion of the violin string, particularly evident when captured by an old iPhone with a “rolling-shutter” camera (Fig.1). Subsequent microscopic observations of Brownian motion in response to violin sounds further fueled my curiosity, leading to the realization that sound could be visualized both from the instrument and its conductance in the air.
[Methods]
The study employed both iPhone 5 and 7 cameras to capture the vibrating G string’s video image, exploring various perpendicular angles to assess ‘sound wave’ formation during replay. Microscopic observations of fume particles’ motion in response to airborne sound were conducted, along with testing sound conductance at the air-water interface using particles of different sizes in water. Complementary research on the human auditory system via web searches aided in interpreting anatomical data.
[Results]
In the video visualization experiment, the ‘sound wave’ formation was exclusively captured with an old iPhone’s ‘rolling-shutter’ camera in landscape orientation. The calculated vibration speed from the captured wavelength was 159.4Hz, slightly deviating from the G string’s actual 196Hz (explained in Discussions). Microscopy revealed pronounced air particle motion in response to sound, contrasting with diminished water particle response, indicating challenges in sound conductance at the air-water interface.
[Discussions]
The striking visualization of sound and its conductance holds significant potential for making music and conversations tangible for those with hearing disabilities. Even individuals with normal hearing could expand their perceptual horizons. The necessity of a ‘rolling shutter’ for sound wave visualization and the offset in vibration speed suggest technical considerations. Challenges in sound conductance at the air-water interface underscore the complexity of human auditory structures, such as the tympanic membrane, ossicles, cochlea, hair cells, tectorial membrane, etc.
[Project Proposed]
Building on the study findings, the proposed project aims to design an affordable and accessible “open device” platform. Utilizing Raspberry Pi, Python programming, and Arduino technology, this platform will feature a tactile vibration device controlled by Raspberry Pi and Arduino. This device intends to enhance the musical experience for individuals with hearing impairments by enabling them to perceive and enjoy music through vibrations. Additionally, plans include incorporating visual elements like sound wave images or Brownian motion to enrich the overall sensory experience.