Answering ASL questions: Argon leads a 21-question pilot

Datasets:

salqadri /

argon-asl-question-understanding-eval

Like 1

Tasks:

Question Answering

Modalities: Tabular

Text

Formats: csv

Languages: English

Size: < 1K

Tags: evaluation

asl

accessibility

video-understanding

gemini

meta

• 3

Libraries: Datasets

pandas

Polars

• 1

License: cc-by-4.0

Dataset card Data Studio Files Files and versions xet

Community 1

Dataset Viewer

Auto-converted to Parquet API Embed Duplicate Data Studio

Subset (3) questions · 21 rows

questions (21 rows)results (336 rows)short_answer_followup (8 rows) Split (1)

test · 21 rows

test (21 rows)

SQL

Console

video_id

stringlengths

8

8

intended_question

stringlengths

19

92

stored_width

int64

1.92k

1.92k

stored_height

int64

1.08k

1.08k

display_width

int64

1.08k

1.08k

display_height

int64

1.92k

1.92k

average_fps

float64

29.7

30

duration_seconds

float64

6.45

15.1

audio_stream_count

int64

0

0

video_01

Does Coke Zero really have 'zero' sugar?

1,920

1,080

1,080

1,920

29.879436

9.237122

0

video_02

How many light-years would it take to travel from one end of the universe to the other?

1,920

1,080

1,080

1,920

29.985617

13.673222

0

video_03

What is the response time on an Xbox controller?

1,920

1,080

1,080

1,920

29.919581

12.0991

0

video_04

How many humans could fit in a 3200 square foot square?

1,920

1,080

1,080

1,920

29.990031

12.070678

0

video_05

How fast is the speed of light?

1,920

1,080

1,080

1,920

29.971558

6.539533

0

video_06

How much energy does a sound wave need to be heard by a human from around a mile away?

1,920

1,080

1,080

1,920

29.939583

14.262056

0

video_07

How much time does it take for a person to see a text message from halfway across the world?

1,920

1,080

1,080

1,920

29.868512

15.066033

0

video_08

Who invented the very first sign language?

1,920

1,080

1,080

1,920

30.003439

7.365822

0

video_09

Why do we dream when we sleep?

1,920

1,080

1,080

1,920

30.01116

6.630867

0

video_10

Why do planes stay in the sky if they are so heavy?

1,920

1,080

1,080

1,920

29.899179

8.930011

0

video_11

Can fish see water?

1,920

1,080

1,080

1,920

29.944526

6.946178

0

video_12

How many stars are in the universe?

1,920

1,080

1,080

1,920

30.011213

7.163989

0

video_13

How much water is inside a cloud?

1,920

1,080

1,080

1,920

29.900281

6.454789

0

video_14

Why are bubbles round?

1,920

1,080

1,080

1,920

29.748954

8.605344

0

video_15

What will happen if the Earth stops rotating?

1,920

1,080

1,080

1,920

30.011133

8.563489

0

video_16

How long would it take to walk around the Earth?

1,920

1,080

1,080

1,920

29.904252

9.831378

0

video_17

Why can't humans see air?

1,920

1,080

1,080

1,920

29.970922

9.709411

0

video_18

Will we ever be able to travel faster than light?

1,920

1,080

1,080

1,920

29.916898

11.130833

0

video_19

Why can't we see sound waves?

1,920

1,080

1,080

1,920

29.91864

6.5511

0

video_20

Can a robot ever have feelings?

1,920

1,080

1,080

1,920

30.010521

9.663278

0

video_21

Is an octopus a mammal?

1,920

1,080

1,080

1,920

29.792428

7.518689

0

Results

Recordings and prompt

Scoring and the short-answer follow-up

Generation time

What the answers show

Model scope and related work

Questions and consistency

Privacy and reuse

Citation

Answering ASL questions from video: Gemini Argon leads a 21-question pilot

Syed Salman Qadri · Rayan Syed Qadri October 2026 · Preliminary case study · 21 questions from one signer

American Sign Language (ASL) has grammar distinct from English. A model must follow handshape, movement, facial expression, body posture and timing; missing a small cue can change a question's meaning. Reliable ASL input could help Deaf signers ask questions, use tutoring tools and reach digital services in their own language. Signed STEM resources remain scarce. NIDCD on ASL · ASL STEM Wiki (EMNLP 2024)

We tested whether general video models could answer a signed question directly, without being asked to translate it. On the same 21 silent recordings and generic prompt, Gemini 4 Argon matched the full intended question in 69.4% of completed answers (59/85), versus 11.4% (12/105) for Gemini 3.1 Pro preview and 6.0% (6/100) for Gemini 3.8 Flash. All used HIGH thinking and five rounds per video. Argon completed 81.0% of requests (85/105), yielding an overall successful-answer rate of 56.2% (59/105). A later, separate one-round follow-up with Meta Muse Wonder 1.0 completed all 21 requests and yielded 0/21 matches under its provider's video pipeline.

Argon's lead on these recordings is striking, and several answers directly addressed complete science questions. The result is a single-signer case study, scored against contributor labels by OpenAI's GPT-6 Astra. Completed-answer matching describes performance conditional on producing an answer. Incomplete attempts may relate to question difficulty, and these recordings cannot establish success rates for ASL users generally.

Inspect every attempt: Complete answers and judgments, organized by question, with downloadable records for the 336 original-prompt attempts and eight separately prompted follow-ups.

We emphasize intent matching among completed answers alongside completion rate. The former measures whether a returned answer addresses the intended question; it does not audit every factual claim. The latter measures how often a request produces a complete answer, whether right or wrong. The secondary all-attempt match rate counts incomplete answers and API errors as unsuccessful delivery. All three measures use the same saved judgments; no trials were rescored.

Argon's 20 unsuccessful deliveries included 4 HTTP errors, 13 token-limit stops and 3 incomplete responses without a recorded finish reason. Flash had 5 token-limit stops; Pro and Meta completed every request. These are not all API-stability failures. For example, four of Argon's five attempts on the airplane question reached the token limit, and its one completed answer was incorrect. Excluding incomplete attempts gives that question less weight, so completed-answer rates compare different subsets of attempts. Removing only the four recorded HTTP errors would give Argon 59/101 (58.4%).

Five Gemini rounds show substantial variation by question. Argon matched all five attempts on six questions, at least once on 15 of 21, and never on six. Those repetitions are not additional signers or independent language examples. The round counts below retain all 21 attempts per round, including unsuccessful deliveries.

For completed-answer matching, resampling the 21 question groups 20,000 times gave 95% percentile intervals of Flash 0.0–15.7%; Pro 3.8–21.0%; Argon 52.6–83.2%. Each draw keeps a selected question's five Gemini attempts together and divides total matches by total completed answers. The chart's whiskers show these intervals; its dots show each round's completed-answer rate. These intervals describe question selection under the bootstrap assumptions. They do not correct for selective noncompletion or represent signer diversity. The earlier all-attempt intervals remain in the summary. Meta has only one round, so no five-round interval is shown; 0/21 is not proof of zero underlying success.

Recordings and prompt

The 21 videos show one female ASL expert, as described by the authors, standing before a plain wall. The English intended questions came from contributor comments. No separate signer-confirmed transcript of every recording is documented.

Privacy-edited illustration derived from one frame. An AI image-editing tool applied strong blur; composition is approximate. It cannot be used to inspect signs, and the original image and videos remain private.

Exact prompt used for the 336 primary results:

The video shows a deaf person asking a question in American Sign Language (ASL). Answer the question directly in text. Do not translate, transcribe, or restate the signed question.

These are 1080p / 30fps source files submitted without cropping or video re-encoding. The 24 fps API field is a sampling request, not a claim that every source frame was seen. Google's documented VideoMetadata.fps range for this runner was at most 24. Google API reference

Scoring and the short-answer follow-up

OpenAI's GPT-6 Astra scored all 336 original-prompt responses against each contributor's intended question. It marked Yes when an answer preserved the essential objects, quantities and relationships, including when it corrected a false premise. Wrong relations, topic-only replies, translation without an answer and requests to repeat were No. Model identity was available during review. The released rows include judgment reasons and Astra's uncalibrated understanding-confidence estimates; these are not the tested models' self-reported confidence.

An answer could receive Yes when it substantively addressed the intended question, even if it also offered other interpretations. Extraneous interpretations could reduce confidence without automatically invalidating the match. Four accepted Argon answers to the 3,200-square-foot capacity question assumed a tent; one primarily discussed 320 square feet, then addressed 3,200 square feet as an alternative. A matched bubbles answer also discussed triangles and the Bermuda Triangle. These judgments credit coverage of the intended question without requiring an exclusively correct interpretation; the complete answers and reasons allow a stricter reading.

A bare number or Yes/No reply may answer a question without showing which question the model saw. The scoring rubric counted such compatible but nondiagnostic replies as No. Among completed answers, accepting every flagged short reply would raise Flash to 8/100 (8.0%) and Pro to 18/105 (17.1%); Argon remains 59/85 (69.4%).

We made eight selected reruns across three recordings: six Pro attempts and two Flash attempts, with one fresh generation for each flagged original response. The recordings asked about human capacity in a 3,200-square-foot square, whether fish can see water, and the time needed to walk around Earth. The silent files, models, HIGH settings and 24 fps request stayed the same. The prompt added this sentence after the original text:

Give a direct answer in at least 30 words, with enough context to make clear what you are answering.

Under that changed prompt, one of eight attempts matched full intent (1/6 Pro; 0/2 Flash). All seven returned answers met the length requirement; one request timed out. GPT-6 Sol reviewed these eight new answers using the same intent rubric. These were fresh generations, not elaborations of the original answers. The selected follow-up cannot establish whether the longer prompt improves performance generally or reveal what the models understood on their original attempts. It is excluded from all primary comparison statistics and charts.

Three Pro answers to “Can fish see water?” show why context matters. These are excerpts from separate reruns of the same recording:

All eight follow-up answers and their timings, original short replies and judgments remain available, including the timeout.

Times cover the full non-streaming generation call, including thinking and network response, and exclude uploads and pacing. Gemini file activation occurred beforehand; Meta may process video within the timed call. The