ERUSARA

==Overall objective
Develop a novel framework for attention-based shared autonomy enabling
disabled users and robots to collaboratively complete complex assistive
tasks. By integrating scene context, spoken dialogue, and non-verbal
cues such as gaze and gestures, we aim to create models capable of
interpreting object-related and task requests, predicting attention
shifts, and inferring intention to support adaptive, context-aware
responses within a shared autonomy (SA) system.

This objective brings together two closely connected goals addressed
jointly.
1) HRI and the practical design and deployment of SA systems: 

- Understand the factors that improve SA usability for individuals with
C1–C4 Spinal Cord Injury (SCI), Multiple Sclerosis (MS), Cerebral Palsy
(CP), and Hemiparetic Stroke (HS); 

- Design multimodal SA systems adapted to disabled users and supporting
their daily activities.

2) Developing AI models for Embodied Reference Understanding (ERU),
grounding deictic referential expressions (e.g., “this one,” “over
there”). We will design holistic, multimodal models that can: 

- Detect deictic gestures from gaze, head, or hand movement;
• Identify the referent in the scene toward which the gesture is
directed; 

- Link verbal deictic expressions to these referents for robust
grounding in interaction.
We will also explore advanced reasoning with MLLMs, leveraging their
contextual and spatial abilities while addressing current limitations in
processing fine-grained embodied cues. 

== Extending the state of the art: 

- Multimodal shared autonomy is still at its infancy, with recent work
highlighting the challenges of open-ended modalities such as language
and ambiguous ones like gaze. By providing an integrated framework we
aim to advance shared autonomy. 

- By centering development on users with disabilities, we ensure the
system addresses real needs. Taking an open-ended, real-world approach
to shared autonomy, rather than focusing on explicit tasks and objects,
extends opportunities for assistive robots to impact the lives of
disabled users. 

- Beyond language-only RES/REC: Whereas most research in referring
expression comprehension focuses on text-based grounding, this project
addresses the underexplored area of gesture- and gaze-based reference
resolution, which is crucial in real-world HRI.

- This project goes beyond prior work by resolving purely deictic
expressions (e.g., ‘this,’ ‘here’) and explicitly addressing non-object
targets such as empty spatial locations (e.g., ‘put that here’), where
existing RES models biased toward salient objects fail. 

- Integration of MLLMs: While MLLMs excel at reasoning and scene
understanding, their limits in analyzing subtle embodied cues are
acknowledged. This project proposes novel ways to train or extend MLLMs
to capture gaze and deictic behaviors, and evaluate their ability to
generalize in assistive settings. 

Most research in behavior understanding focuses on able-bodied
subjects. By including disabled users in our dataset and testing on
these populations, we advance the state of the art toward a more
inclusive approach to research. 

== Overview of the research approach.
The research is organized into four interconnected tracks. Track 1
(Participatory design and accessibility) builds a community network of
researchers, stakeholders, patients, and caregivers to identify needs
and co-design tasks and interfaces. Track 2 (Multimodal deictic event
detection and target localization) develops perception methods to detect
when deictic events occur and where they point in 2D/3D, combining
gaze, head, and hand cues. Track 3 (Multimodal deictic visual grounding)
integrates these outputs with language, vision, and behavioral cues to
resolve embodied references using unified MLLM frameworks. Track 4
(Attention-based shared autonomy) leverages insights from all tracks to
create and evaluate a flexible shared autonomy paradigm, co-designed and
tested with both able-bodied and disabled users.