The eye test: How China is benchmarking multimodal AI against human clinicians

Chinese scientists have systematically evaluated five leading multimodal large language models—including GPT-5, Gemini-2.5 Pro, GLM-4.5V, and Claude-Sonnet-4.5—for their ability to diagnose ocular surface diseases in a real-world, resource-limited setting. Using a retrospective dataset of 259 representative cases from Aksu, China, the study compared model performance against structured clinical data and anterior segment photographs. The findings, published in the American Journal of Pathology, represent one of the most rigorous independent assessments of multimodal LLM accuracy, safety, and equity in a clinical domain where specialist access remains scarce.

The research is significant not only for its direct clinical implications—potentially expanding diagnostic reach in Western China and similar underserved regions globally—but also for its methodological approach. By focusing on safety and equity alongside raw accuracy, the study highlights gaps that may not surface in controlled laboratory benchmarks. For international observers, this work underscores China’s growing capacity to conduct high-stakes, real-world validation of frontier AI systems in public health contexts, a capability that will shape both regulatory standards and commercial deployment strategies worldwide.

Why it matters:
The convergence of multimodal AI with clinical diagnostics is no longer theoretical. This study provides early evidence that frontier models can function in low-resource environments, with implications for healthcare cost structures and access equity. For technology buyers and investors, China’s willingness to publicly benchmark these systems offers a rare transparent view into what works and what does not in real-world medical AI.

Source →