mafmadmaf's work
↩︎ back to index

        《步入時澗》源自坪山美術館「聽覺之外」項目，圍繞聽障及聾人群體展開，嘗試探索超越聽覺器官的聲音感知方式，並構建聲音與視覺、觸覺等感知維度之間的轉化與關聯。作品以聽障及聾人群體的感受為重要參照，同時向不同聽力程度的觀眾開放，邀請人們通過身體參與其中，感知包括聲音在內的多種媒介信息。
作品由一套視聽交互程序構成。參與者的身體動作並不在被攝像頭捕捉的瞬間發聲，而是被轉譯到畫布上，隨著時間推進逐步生成一份圖像樂譜。這份持續生成的樂譜從右往左滾動，直到抵達畫面左側的「讀針」位置時，樂譜上的音符，亦即那些被時間定格下來的身體圖樣，才會被讀取並演奏。
這種感覺近似一台手搖紙帶八音盒。參與者以身體創作出如同「紙帶」一般的樂譜，音高從低到高排列在畫面之中；當紙帶行進至讀針，也就是屏幕最左側的黃色豎線，聲音才被觸發。身體不再只是即時演奏的工具，而成為書寫樂譜的介質。
圖像樂譜的書寫速度慢於攝像頭的記錄速度，因此參與者可以在鏡頭前放慢動作，讓樂譜生成更平滑、舒緩的聲音線條；也可以通過迅速而大幅度的動作，創造密集、短促的聲音擊發。然而，這個「演奏」並非即時反饋。動作與聲音之間的滯後，使參與者更像是一位作曲者，正在用身體姿態書寫一條延綿生長的樂譜。
現場程序生成的樂譜將參與者的肢體動作視為符號，以相對靜止的拍攝背景作為橫向延展的線條。圖譜中，時間從左至右更新前進，聲音頻率則由下至上逐漸升高。除了在現場聽到程序讀取樂譜所產生的聲音，這份樂譜也可以被保存下來，作為圖像曲譜交由其他程序重新詮釋，甚至由人類音樂家以真實樂器再次演奏。
因此，參與者與程序的交互首先被理解為一種書寫。它可以被存檔、復讀、延伸，並保留以不同形式重新表達的可能性。影像與聲音之間的時間間隙，讓人更容易觀察被凝結在畫面上的身體如何生長成樂譜，也更充分地感受樂譜與聲音生成之間的關係，從容地構思下一個「身體音符」。在這個場景之中，我們更像是在用身體創造樂譜，而不是直接演奏。
當樂譜被讀取後，聲音經由揚聲器單元的震動帶動空氣震動，進入常規的聽覺接收路徑。考慮到聽障及聾人群體具有多樣的聽覺經驗，並非所有人都能以相同方式感受完整頻段的聲音，作品特別強化中低頻聲音，並通過固體傳導將其轉化為身體可感的振動。聲音在這裡不只被「聽見」，也可以被觸摸、被感受、被身體接收，為不同聽力狀況的參與者提供更開放的感知入口。
自 2026 年 2 月以來，項目通過問卷調查收集關於聲音和觀展經驗的反饋，並在映諾社區發展機構的支持下，持續深化對聽障及聾人群體的理解。與此同時，項目也意識到不同參與者在文字理解、溝通方式、文化背景和現場手語翻譯需求上的差異。這些限制並非外部條件，而是作品方法的一部分：它們推動作品持續調整感知入口、交互方式與交流節奏。
        Stepping Into the Time Stream grew out of the project Beyond Hearing, which was developed around hard-of-hearing and Deaf communities. It explores ways of perceiving sound beyond the auditory organs, and attempts to construct mechanisms of transformation and connection between sound and perceptual dimensions such as vision and touch. Taking the experiences of hard-of-hearing and Deaf communities as an important point of reference, the work also opens itself to audiences with different hearing conditions, inviting people to engage their bodies and perceive multiple forms of media information, including sound.
The work is built around an audiovisual interactive program. Body movements do not produce sound at the moment they are captured by the camera. Instead, the participant's body language is translated onto the canvas, gradually generating a score over time. This continuously generated graphic notation scrolls from right to left. Only when it reaches the "reading needle" on the left side of the screen will the notes on the score, the patterns in which body language has been fixed in time, be read and played.
The experience is similar to a hand-cranked paper-tape music box. Through their bodies, participants create a score like a paper tape, with pitches arranged from low to high across the image. When the tape moves to the reading needle, the yellow vertical line at the far left of the screen, sound is triggered. The body is no longer simply an instrument for immediate performance, but becomes a medium for writing a score.
The writing speed of the graphic notation is slower than the camera's recording speed. This allows participants to slow down their movements in front of the camera, so that the score writes smoother and gentler sound lines. Conversely, rapid and expansive movements can create dense, short bursts of sound. However, this "performance" is not immediate feedback. The delay between movement and sound makes the participant's role closer to that of a composer, writing a continuously growing score through bodily posture.
The score generated by the live program treats the participant's bodily movements as symbols, while using the relatively static background captured by the camera as horizontally extending lines. In the diagram, time advances from left to right, while the vertical axis from bottom to top represents sound frequencies from low to high. In addition to hearing the sound produced when the program reads this score in real time, the score can also be saved as an image score, interpreted by other programs, or even handed over to human musicians to perform with real instruments.
Therefore, the participant's interaction with the program is first understood as a form of writing. It can be archived, reread, extended, and kept open to the possibility of being re-expressed in different forms. The interval between image and sound makes it easier to observe how the body, condensed on the screen, grows into a score, and to sense the connection between the score and the generation of sound. It also allows participants to leisurely conceive the next "body note". In this situation, we are more like creating a score with the body than directly performing.
After the score is read, the generated sound is produced through the vibration of loudspeaker units, which sets the air into vibration and enters the usual auditory pathway. Considering that hard-of-hearing and Deaf communities have diverse hearing experiences, and that not everyone can perceive the full frequency range of sound in the same way, the work particularly emphasizes mid- and low-frequency sounds, and transforms them into body-perceivable vibration through solid conduction. Here, sound is not only heard; it can also be touched, felt, and received by the body, providing a more open perceptual entry point for participants with different hearing conditions.
Since February 2026, the project has collected questionnaire responses on sound and exhibition-viewing experiences, and with the support of Yingnuo Community Development Organization, the team has continued to deepen its understanding of hard-of-hearing and Deaf communities. At the same time, the project has also recognized differences among participants in reading comprehension, communication methods, cultural backgrounds, and the need for on-site sign language interpretation. These limits are not external conditions, but part of the work's method: they continue to shape its perceptual entry points, interactive methods, and rhythms of exchange. ↩︎ back to index