Short answer

Integrate multimodal sensing capabilities into head-worn devices to enable discreet, tongue-gesture-based control, reducing reliance on traditional input methods.

Field: Human Factors
Source: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (2023)
Method: Sensor fusion and machine learning for gesture recognition
Sample: 16 participants
Evidence: Strong effect

Leveraging multimodal sensor data from off-the-shelf head-worn devices, tongue gestures can be recognized with high accuracy, providing a silent and hands-free interaction method. This human factors research insight is drawn from a 2023 study published in INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION. Using Sensor fusion and machine learning for gesture recognition with 16 participants, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Integrate multimodal sensing capabilities into head-worn devices to enable discreet, tongue-gesture-based control, reducing reliance on traditional input methods.

Study

Human FactorsRecentStrong effect

Tongue gestures offer 94% accurate, discreet control for head-worn devices

Leveraging multimodal sensor data from off-the-shelf head-worn devices, tongue gestures can be recognized with high accuracy, providing a silent and hands-free interaction method.

INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · 2023

Key Findings

01Eight closed-mouth tongue gestures were classified with 94% accuracy using multimodal sensor fusion.
02The IMU sensor alone achieved 80% accuracy for eight gestures and 92% for a subset of four gestures.
03A large dataset of 48,000 gesture trials enabled user-independent classification.

Application

Design takeaway

Integrate multimodal sensing capabilities into head-worn devices to enable discreet, tongue-gesture-based control, reducing reliance on traditional input methods.

How to apply

Develop prototypes for AR/VR interfaces or assistive technologies that utilize tongue gestures for commands like navigation, selection, or confirmation, especially in public or noisy environments.

Project actions

01Consider using existing wearable sensors (like those in smartwatches or fitness trackers) to capture subtle user movements.
02Explore how different sensor combinations affect the accuracy of recognizing specific user actions.

Method & Evidence

AimCan multimodal sensor data from commercial head-worn devices accurately recognize closed-mouth tongue gestures for discreet, hands-free interaction?

MethodSensor fusion and machine learning for gesture recognition

ProcedureSynchronized EEG, PPG, IMU, eye tracking, and head tracking data were collected from two commercial headsets. This multimodal data was used to train a model to classify eight distinct closed-mouth tongue gestures. The performance of individual sensors, particularly the IMU, was also evaluated.

Sample16 participants

ContextWearable technology, specifically head-worn devices for AR/VR and earables.

Variables

IV["Type of sensor data (EEG, PPG, IMU, eye tracking, head tracking)","Specific tongue gesture performed"]

DV["Gesture recognition accuracy (%)","Classification accuracy for IMU alone"]

CV["Type of head-worn devices used","Number of gestures classified","Participants' physical characteristics (implicitly)"]

Strengths & Limitations

Strengths

+Utilized multimodal sensor fusion for enhanced accuracy.
+Demonstrated user-independent classification through a large dataset.

Limitations

The study used specific commercial headsets; results might differ with other hardware. The gestures were performed with the mouth closed, which may not apply to all interaction scenarios.

Reliability & validity

The study's reliability is supported by a large dataset (48,000 trials) and user-independent classification. Validity is demonstrated by achieving high accuracy in recognizing specific gestures, suggesting the system effectively measures what it intends to.

Think critically

To what extent can tongue gesture recognition be generalized across different user populations and a wider range of device form factors beyond the specific head-worn devices tested?

Design Principles

"Exploit subtle, underutilized human movements for intuitive and discreet device interaction."

This research opens avenues for novel interaction paradigms in wearable technology, particularly for augmented and virtual reality. It demonstrates that complex, discreet user input can be achieved without intrusive hardware, enhancing user experience and accessibility.

What This Means for Your Design

You can control devices like VR headsets with your tongue without making a sound or using your hands, using sensors already built into some headbands.

How to use in your project

1.Reference this study when exploring alternative input methods for your design project, particularly if your design involves wearable technology or requires hands-free operation.

Add to My Project

Quick Cite

Paragraph starter

Research into multimodal interaction, such as the TongueTap system, demonstrates the potential for highly accurate (94%) gesture recognition using off-the-shelf head-worn sensors. This approach offers a discreet, hands-free control method, suggesting that subtle physiological cues can be effectively leveraged for user input in wearable technology contexts.

Source

INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

TongueTap: Multimodal Tongue Gesture Recognition with Head-Worn Devices

journal · 2023

View source

Questions About This Research

What does the research say about tongue gestures offer 94% accurate, discreet control for head-worn devices?: Integrate multimodal sensing capabilities into head-worn devices to enable discreet, tongue-gesture-based control, reducing reliance on traditional input methods. Evidence: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (2023).
Why does "Tongue gestures offer 94% accurate, discreet control for head-worn devices" matter for design?: This research opens avenues for novel interaction paradigms in wearable technology, particularly for augmented and virtual reality. It demonstrates that complex, discreet user input can be achieved without intrusive hardware, enhancing user experience and accessibility.
How can designers apply this research?: Integrate multimodal sensing capabilities into head-worn devices to enable discreet, tongue-gesture-based control, reducing reliance on traditional input methods.
What were the main findings?: Eight closed-mouth tongue gestures were classified with 94% accuracy using multimodal sensor fusion.. The IMU sensor alone achieved 80% accuracy for eight gestures and 92% for a subset of four gestures.. A large dataset of 48,000 gesture trials enabled user-independent classification.
What research method was used?: Sensor fusion and machine learning for gesture recognition with 16 participants.
How strong is the evidence?: Evidence strength is rated Strong effect, based on a 2023 journal from INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION.
What should I do differently in my next project?: Develop prototypes for AR/VR interfaces or assistive technologies that utilize tongue gestures for commands like navigation, selection, or confirmation, especially in public or noisy environments.
What are the limitations?: Accuracy may vary with different types of head-worn devices, individual user anatomy, and the complexity of the gesture set. Environmental factors could also influence sensor readings.