Situational Assistants — NYCU NLP Lab

Timeline

The series so far.

Can an assistant that sees and hears decide when to speak? EgoArgus measures that judgment. VISTA produces the situations to test it on.

2026
EgoArgus · EMNLP 2026
A human-annotated benchmark of dialogue-video scenarios that tests whether vision-language models can arbitrate between what they see and what the user says, and decide whether to intervene.
2026
VISTA · EMNLP 2026 Demo
A controllable platform that turns a one-line scenario seed into an editable plan, an egocentric video, and an auditable review trail, so that rare or unsafe assistance situations can be studied without filming them.
Next
More to come
New work in this series will appear here.

EgoArgus · EMNLP 2026

Same scene, different words, different decision.

Pick a real EgoArgus decision video, then change what the user says. The reviewed decision, the intervention time, and the assistant’s message all change with it. Everything shown is taken verbatim from the released annotations.

What the user says

Reviewed decision

Why

Dataset & code EgoArgus paper

Videos, dialogue, reviewed decisions, intervention times, and assistant messages are taken verbatim from the EgoArgus decision annotations (dataset/paper_eval.csv). The videos were generated with VISTA.

Two capabilities

Understanding: multiple-choice reasoning under multimodal, contradictory, off-topic, on-topic, and text-grounded conditions.
Decision: whether to intervene, when, and which assistance steps to give.

Release

7,767 examples: 6,978 understanding, 789 decision.
311 decision videos on Hugging Face, generated with VISTA.
Code for evaluation, modality-separation analysis, attention reweighting, and preference alignment.

Links

VISTA · EMNLP 2026 Demo

From a one-line scenario seed to a reviewed egocentric video.

Rare or risky assistance situations are hard, costly, or unsafe to film, and simulators lack the social common sense to play out consequences. VISTA generates the scenario, the video, and the review trail, with an explicit approval gate at each step.

One demonstration, step by step

1

Scenario seed“A reflective metal lunch container is held at an open microwave while the user expresses uncertainty.”

2

PlanReveal the container’s material. Show the microwave door opening. Intervene before heating begins.

3

Expected assistanceWarn before the container is heated and suggest a microwave-safe dish.

4

Review trailVisible cues, dialogue cues, why assistance is warranted, and a stated limitation.

Open the VISTA platform VISTA paper

Generated by VISTA

14 reviewed demonstrations

Three interaction modes (reactive, explicit proactive, implicit proactive) across two consequence families (safety-critical, everyday support), plus matched restraint cases where the right answer is to stay quiet. Click any card to open it on the platform.

ReactiveExplicit proactiveImplicit proactiveAppropriate restraint