Feel The Ball: Convert Ball Motion to Touch for Vision/Hearing - Impaired Sport Audiences
Voiceover
Presenter Information(s)
Amy Zhang
Project Category
Sr - Computer Sciences & Mathematics
Project Number
SR-CMP-005
Optional: Supplementary Materials
https://drive.google.com/file/d/1ZUWKdlEoIj85_EKsYP5yGGkQC8AnoxpI/view?usp=sharing
Young Changemaker Award
285 million vision-impaired and 430 million hearing-impaired people cannot fully experience the thrill of a live tennis match. They cannot see or feel the pulse-pounding speed and dynamics like the thousands of others seated in the stands next to them. While caption and audio-based existing technologies have been created, they only address aspects of accessibility. But, with my system, they can. I have developed the first device that can convey a tennis ball's motion through touch. With my breakthrough in reliable ball motion tracking from broadcast videos and novel psychology-guided design criteria for haptics (i.e., sense of touch) display of fast-moving objects, the 87% of people with visual or hearing disabilities who express frustration at the inadequate current tools can now immerse themselves in the rush of the game. Feeling the ball's motion naturally provides spatial information. With my device, sports events will be much more accessible and thrilling for these spectators. The societal impact is incredible, improving quality of life for hundreds of millions and creating a more inclusive society, where everyone can enjoy the excitement of sports. Even people who are not hearing or visually impaired can use this technology to improve their sports watching experience. My work has the potential to transform the way people with disabilities experience sports events. I am proud that my project makes entertainment more accessible and immersive for everyone but focuses on an often forgotten or ignored population for whom it will make the most significant difference.
While the technology I developed was focused on improving the experience of visual/hearing impaired fans at a live or video broadcast sporting event, it has the potential for a much larger impact. My technology contributions in tracking small and fast-moving objects from broadcast videos have improved continuity in the detection and tracking of objects over time. Object tracking is fundamental in computer vision, improving video surveillance, activity recognition, road condition monitoring, airport safety, and monitoring of protection along the marine border. Furthermore, I found a promising way to smoothly convey real world spatial motion through a haptics display, which is especially important since current haptics technologies mainly have simple pause-style vibrations and have been mainly used in virtual world scenarios. My new technologies could heighten the user's sense of reality and spatial awareness, potentially changing and enhancing the way people take in information about the world around them. This new type of spatial information could not only change people's immersive entertainment experiences but also improve work performance in various applications. With the system I developed, the potential for change is endless.
Abstract or Description
My research aims to develop an innovative system that makes sporting events more accessible and engaging to sports enthusiasts with vision or hearing impairments. Unlike existing captioning-based (i.e., see the real-time commentary) or audio-based (i.e., hear the ball) assistance methods, my system enables users to feel the spatial, dynamic ball movements. The architecture of this system includes extracting real-time ball motion information from broadcast sports videos using computer vision and artificial intelligence technologies and transforming it into dynamic touch feeling through a force feedback device (so-called haptics techniques). This system helps visual- or hearing-impaired spectators have a more immersive accessibility to live sports and thus experience the excitement accompanying a sports match. Additionally, this system will heighten people’ sense of reality and spatial awareness and potentially change the way people interact with the real world, advancing future immersive technologies in various fields requiring sensory augmentation. Besides personal use, the fundamental technology innovation in reliable tracking of small and fast-moving objects and intuitive haptics of real-world object motion has broader impact on other applications such as video surveillance and airport safety.