Apple explores LLMs for spatial understanding and sign language annotation

Apple remains committed to its spatial computing projects, despite recent rumors claiming the Vision Pro was a failure. In April 2026, reports suggested the product's demise, but those claims are now being questioned. The company continues to publish research exploring new AI applications in this field.

One notable study introduces a benchmark called SFI-Bench, designed to evaluate multimodal large language models (MLLMs). This system tests whether models understand spatial layouts, object functions, and their contextual use, combining visual reasoning with language comprehension.

It assesses if AI models grasp what objects are, where they are, how they are used, and how to troubleshoot them.

Apple’s researchers compared various models, with Google Gemini 3.1 Pro achieving top results in spatial reasoning, followed by OpenAI’s GPT-5.4-High. Despite progress, all models still struggle with logical reasoning and spatial memory, especially in tasks involving counting and object relationships.

Another project focuses on using AI to annotate sign language videos automatically. This pseudo-annotation pipeline aims to reduce manual effort and costs, showing promising results in finger spelling and sign recognition, with potential future integration into Apple devices like AirPods for translation features.

Further, Apple developed methods to reconstruct detailed 3D head models from multiple images, enabling realistic avatars and AR applications. These innovations could enhance future Apple products by providing more immersive and personalized experiences.

Overall, Apple’s ongoing research indicates a strong interest in spatial AI, sign language recognition, and 3D modeling, with concrete developments that could influence upcoming hardware and software updates.