Researchers have systematically evaluated multimodal large language models' ability to understand social norms from both text and images, revealing a significant performance gap between modalities and highlighting the challenges of complex norm reasoning. This work is a critical step toward enabling AI agents and robots to operate with nuanced social intelligence in real-world environments, moving beyond the limitations of traditional symbolic AI approaches.
Key Takeaways
- Five leading MLLMs were evaluated on their norm reasoning competence using 30 text-based and 30 image-based social stories, with their performance compared to human benchmarks.
- Results show MLLMs perform significantly better at norm reasoning from text than from images, indicating a core weakness in visual social understanding.
- GPT-4o achieved the best overall performance across both modalities, followed by the open-source model Qwen-2.5VL.
- All tested models, including top performers, struggled with reasoning about complex social norms, regardless of the input modality.
- The study positions MLLMs as a promising alternative to traditional symbolic AI (e.g., formal logic) for enabling normative reasoning in complex, embodied Multi-Agent Systems (MAS).
Evaluating Social Intelligence in Multimodal AI
The research paper presents a novel evaluation framework to assess the norm reasoning competence of Multimodal Large Language Models (MLLMs). The core challenge addressed is enabling artificial agents to understand and reason about implicit social rules—norms—in varied contexts, a capability essential for functional Multi-Agent Systems (MAS) and human-robot interaction. Historically, Normative MAS (NorMAS) research has relied on symbolic approaches like formal logic for norm representation, which struggle to scale to the complexity and ambiguity of real-world social situations.
The study tested five prominent MLLMs on a total of 60 norm-related scenarios: thirty presented as text-based stories and thirty as image-based stories. Each scenario required the model to answer questions that demonstrated an understanding of the social norm at play, potential violations, and appropriate sanctions. The models' responses were then systematically compared against human responses to establish a performance benchmark. This direct comparison is crucial for quantifying the "social intelligence gap" between current AI and human understanding.
The findings are clear on two fronts. First, a pronounced modality gap exists: all models demonstrated superior performance in norm reasoning when the social situation was described in text versus when it was depicted in an image. Second, even the best-performing models found it challenging to reason about complex norms, which often involve layered social contexts, conflicting rules, or nuanced cultural expectations. This indicates that while MLLMs have advanced, their social reasoning remains a brittle function of their training data and architectural limitations.
Industry Context & Analysis
This research sits at the convergence of two major trends in AI: the push for embodied, interactive agents and the benchmarking of increasingly nuanced model capabilities. The explicit comparison to symbolic NorMAS approaches underscores a fundamental industry shift. While symbolic AI offers precision and verifiability in closed environments—evident in rule-based simulation platforms—it fails in open-world settings. MLLMs, trained on vast corpora of human culture and interaction, offer a data-driven, scalable path to social understanding, albeit as a "black box" compared to logic-based systems.
The identified performance gap between text and image modalities is significant and mirrors broader findings in AI evaluation. For instance, while models like GPT-4V and Claude 3 Opus excel on text-based reasoning benchmarks like MMLU (Massive Multitask Language Understanding), their scores on multimodal benchmarks such as MMMU (Massive Multidisciplinary Multimodal Understanding) are notably lower. This study's norm-reasoning task adds a crucial social dimension to this gap, suggesting that visual social cues (body language, spatial positioning, contextual objects) are not being effectively integrated into reasoning pipelines. The success of GPT-4o, OpenAI's flagship "omni" model designed for native multimodal processing, aligns with its architectural intent to better fuse modalities, giving it an edge over models that may treat vision and language as separate components.
The strong showing of Qwen-2.5VL, an open-source model from Alibaba, is a critical data point for the industry. It demonstrates that competitive multimodal social reasoning is not the exclusive domain of proprietary, closed models from leaders like OpenAI and Google. With over 1.3 million pulls on Hugging Face for its predecessor, the Qwen series represents a viable, accessible alternative for researchers and developers building normative MAS, potentially accelerating open innovation in social AI. However, the universal struggle with complex norms reveals a ceiling for current architectures. This challenge is analogous to AI performance on advanced mathematical reasoning or coding benchmarks like HumanEval, where models can handle common patterns but fail on novel, multi-step problems requiring deep conceptual understanding.
What This Means Going Forward
The immediate beneficiaries of this research are developers and researchers working on collaborative robots, virtual assistants, and autonomous agent swarms. For them, the study provides a clear hierarchy of model choice for prototyping normative systems: GPT-4o for state-of-the-art performance, with Qwen-2.5VL as a leading open-source alternative, especially where cost or data privacy are concerns. The documented modality gap serves as a critical specification, indicating that systems relying on visual input for social decision-making require additional safeguards, fine-tuning, or hybrid architectures.
Going forward, we can expect a surge in targeted benchmarking and training for social AI. Just as datasets like MMLU and HELM drove progress in general knowledge, new benchmarks focused on multimodal social reasoning—building on this paper's methodology—will emerge. The research and development focus will shift from merely detecting obvious norm violations (e.g., "don't steal") to interpreting complex, context-dependent norms (e.g., understanding appropriate personal space in a crowded elevator versus a park). Success in this area is a prerequisite for the next generation of AI: agents that can truly collaborate with humans in unstructured environments, from elder-care robots to negotiating autonomous vehicles.
The key trend to watch is the integration of this evaluative feedback into model training cycles. Will the next generation of MLLMs show a closed modality gap on social tasks? Furthermore, the ultimate test will be in deployment. The next critical research phase will involve moving from static image/text evaluation to dynamic, embodied simulations—testing how these models guide the real-time actions of an agent in a simulated social world. The race is no longer just about which model has the most parameters, but which can best navigate the invisible web of rules that governs human interaction.