PIE-V | Mistake Aware Egocentric Procedural Videos

About

PIE-V is a pipeline to create examples of erroneous procedure execution where mistakes are not random, scripted, or staged, but human-plausible deviations with explicit procedural consequences and, when appropriate, recoveries. It constructs controlled mistake-aware egocentric procedural videos from clean keystep-annotated procedures.

The pipeline combines phase- and load-aware error planning, semantic role edits, recovery sampling, LLM rewriting and judging, and targeted video generation for affected windows.

From Clean Keysteps To Mistake-Aware Videos

We show how the PIE-V pipeline and evaluation protocol work on procedural videos from Ego-Exo4D. Source videos and original Ego-Exo4D annotations remain local inputs governed by the Ego-Exo4D access terms.

Ego-Exo4D Subset

PIE-V starts from 17 Ego-Exo4D keystep tasks: COVID test procedures, bicycle repair, CPR, and cooking tasks (e.g., salads, noodles, omelets, tea, pasta, and sushi rolls). The subset contains 50 takes: three videos per task, except Cooking Sushi Rolls, where only one take is available, and COVID Test, where a fourth take is added because it is the most frequent task.

Structured Error Planning

The planner samples 102 mistakes across the 50 takes. Sampling is conditioned on video duration, step complexity, task complexity, procedure phase, working-memory load, and semantic step information from SRL. The semantic resources and generated text examples document the intermediate outputs.

Recovery Mechanism

PIE-V adds 27 sampled corrections that are sampled conditionally on detectability, action type, latency, and repair strategy, including stopping to fix an action, rolling back and redoing a step, undoing an extra step, or performing a missed step later.

Human Evaluation

Five annotators evaluated the resulting scenarios with a nine-part rubric designed for plausible erroneous procedures and grounded in human perception of mistakes. The aggregate annotation summary and annotation template document the rating.

Error Taxonomy + Correction

PIE-V maps generated mistakes and existing datasets into shared procedural categories.

WE

Wrong Execution

Wrong object, side, amount, location, orientation, or manner.

D

Deletion

A required step is skipped, leaving a missing prerequisite.

S

Substitution

A plausible incorrect action replaces the intended step.

I

Insertion

An extra unnecessary step enters the procedure.

T

Transposition

Two locally plausible steps are executed in the wrong order.

C

Correction

A recovery step detects and repairs a specific mistake.

Examples

Each preview compares an original correct-step clip with a generated erroneous-step clip.

Wrong execution: Applying solution to the wrong side

Side-by-side GIF: original correct COVID test step and generated wrong-execution clip.

Apply a few drops of the test solution to the test indicator. Apply a few drops of the test solution to the wrong side of the test indicator.

Wrong Execution and Correction: Skimming instructions, then recovering

Side-by-side GIF: original correct instruction-reading step and generated mistake with correction.

Read the test instruction sheet. Read the sheet, but only skim it and miss key details. Then stop and re-read the sheet carefully from start to finish.

Human Rubric

The rubric separates local plausibility, procedural coherence, and video grounding.

Metric	What we measure	Why it matters
Error Validity	Whether the proposed deviation has procedural impact.	Filters out harmless execution variation before downstream scoring.
Human Plausibility	How likely a real person is to make this mistake in context.	Keeps generated mistakes aligned with human behavior.
Confusability	How difficult the mistake is to notice during execution.	Captures detectability and subtlety.
Procedure Logic	Whether the full procedure becomes logically broken, with annotator confidence.	Connects local edits to global procedure feasibility.
Sequence Consistency	Whether the edited step order remains executable.	Detects ordering failures after edits and transpositions.
State-Change Coherence	Whether object states and prerequisites remain possible across the trace.	Catches impossible or missing state transitions.
Taxonomy Fit	Which WE, D, S, I, T, or C label best explains the marked deviation.	Makes heterogeneous mistake datasets comparable.
Video Plausibility	How natural the generated or annotated visual behavior looks.	Separates procedural quality from visual artifacts.
Text-Video Grounding	Whether the full textual trace matches the video at episode level.	Catches mismatches that accumulate after a mistake or recovery.

Results

Aggregated rubric statistics compare PIE-V with existing egocentric mistake datasets. Taxonomy Fit is categorical, so it is omitted from the scalar table.

Lower is better for Proc.Logic and State-Chg; higher is better elsewhere.
Dataset	Err.Valid %	Human Pl.	Confus.	Proc.Logic %	Seq.Cons.	State-Chg %	Vid.Pl.	T-V Gr.
EgoPER	51	2.67	1.88	26	4.41	14	3.22	3.42
EgoOops	72	3.73	1.63	28	4.50	10	4.31	3.74
Assembly101	65	3.69	2.09	12	4.82	4	4.05	3.22
CaptainCook4D	74	3.71	2.32	12	4.73	2	3.42	3.63
PIE-V	89	3.41	1.76	6	4.48	3	3.54	3.87

Citation

@article{loginova2026correctly,
  title={How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos},
  author={Loginova, Olga and Keller, Frank},
  journal={arXiv preprint arXiv:2604.15134},
  year={2026}
}