Ego2Web

From real-world video perception to online web execution

Ego2Web is a new multimodal agent benchmark that bridges real-world video perception and online action. Instead of relying on text or screenshots, agents must ground fine-grained evidence from egocentric videos—such as objects, brands, or actions—and use it to complete downstream web tasks. The benchmark spans diverse workflows across E-Commerce, Media Retrieval, Knowledge Lookup, and Local/Maps. Tasks require visual understanding like identifying a specific snack, an exercise step, a car brand, or a university name—demanding strong temporal reasoning, object grounding, and cross-modal transfer to web queries. Unlike existing benchmarks, Ego2Web evaluates whether agents can correctly interpret the real visual world before acting online, not just execute browser actions.

Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user’s real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user’s surroundings and then complete a related task online (e.g., making a purchase). To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real- world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video- task pairs across diverse web task types, including e-commerce, navigation, media search, and so on. To facilitate a more accurate and scalable evaluation for our novel benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse state-of-the-art agents show that their performance remains far from perfect, revealing a major performance gap. We also conduct a comprehensive ablation study on task design, highlighting the necessity of video perception in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.

Semi-Automatic Data Generation & Automatic Evaluation

Ego2Web is the first benchmark that connects egocentric video perception with real-world web agent execution. We build the benchmark with a semi-automatic pipeline: first, a vision-language model parses first-person videos into structured clip-level captions and visual metadata; then, a large language model generates grounded web task instructions over active websites such as Amazon, YouTube, and Wikipedia; finally, human annotators verify and refine each example to ensure visual grounding, web feasibility, and instruction quality. This model-human pipeline produces diverse, high-quality video-task pairs that require agents to identify relevant evidence in the physical world and translate it into downstream digital actions.

To support scalable evaluation in live web environments, we further introduce Ego2WebJudge, an automatic LLM-as-a-Judge framework tailored for visually grounded web tasks. Given the task instruction, action trajectory, web screenshots, and annotated visual evidence from the egocentric video, Ego2WebJudge first extracts key success criteria, then selects the most relevant screenshots from the web trajectory, and finally determines whether the agent has completed the task correctly and consistently with the visual evidence from the real world. This enables reliable online evaluation for multimodal agents operating across both physical perception and web interaction.

Current web agents remain far from perfect on Ego2Web

We evaluate a diverse set of state-of-the-art web agents on Ego2Web. Despite strong capabilities in web interaction and vision-language tasks, all models exhibit substantial performance gaps, highlighting the difficulty of grounding real-world visual perception for downstream web execution.