The Parity Benchmark: A Multimodal Framework for Assessing Bias in LLMs Shmona Simpson, CEO and Founder of Paritii<https://www.paritii.com/> Join the meeting <https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZTQ5NWI2MjktY2UyMy00M...> Abstract: Large Language Models (LLMs) demonstrate impressive capabilities but remain susceptible to harmful biases that can reinforce inequity across domains like content moderation, healthcare, and decision-making. The Parity Benchmark offers a rigorous, multimodal framework to evaluate such biases across protected characteristics including gender, race, socioeconomic status, and intersectional identities. Using an expert-curated dataset, the benchmark assesses leading LLMs such as Granite 3.8 against GPT-4o, Claude 3.5 Sonnet, Llama 3, DeepSeekR-1, and Gemini. The data will show which LLM’s perform best across categories such as ageism, supremacism, colorism, homophobia, disability and neurodivergence. Future iterations of the benchmark will expand to include image-based evaluations, open-ended free-text prompts, and integrated multimodal analytics—providing a richer, real-world assessment of AI fairness. Initial results show that models excel differently across tasks, suggesting opportunities for targeted bias mitigation. The Parity Benchmark aims to serve as a practical, evolving tool for researchers, developers, and policymakers committed to building more equitable and accountable AI systems. <https://arxiv.org/abs/2407.02191>Bio: Dr. Shmona Simpson is the visionary CEO and Founder of Paritii<https://www.paritii.com/>, a pioneering platformdedicated to advancing equity in technology through data-driven frameworks and innovation. A recognized leader in AI ethics and technology strategy, Shmona specializes in developing solutions that promote fairness, accountability, and transparency within AI systems. Her recent development of The Parity Benchmark<https://link.springer.com/article/10.1007/s43681-024-00613-4>, published by Springer in 2025, underscores her commitment to Responsible AI. This groundbreaking framework provides a robust, data-informed approach to evaluating and enhancing fairness within machine learning systems. By identifying and mitigating biases, The Parity Benchmark equips researchers, developers, and organizations with actionable insights to create competitive, responsible AI technologies. Under Shmona’s leadership, Paritii has emerged as a trusted resource for organizations aiming to build ethical AI systems that serve broad communities. Her expertise in AI ethics and strategic innovation continues to shape critical conversations about how technology can be harnessed to promote equity and accountability. Swyft, Paritii's flagship AI solution for broad access higher education institutions, streamlines academic operations with intelligent automation. Swyft handles high-volume administrative tasks like registration, financial aid workflows, and compliance tracking—freeing up staff time and improving the student experience at scale. Swyft is available at institutions across the country<http://www.tryswyft.com/>. An influential voice in the tech industry, Shmona is a sought-after advisor and speaker on AI ethics, fairness frameworks, and the broader implications of responsible technology development. She has been invited to share her perspective at SXSW, ASU-GSV, MITRE, IBM, and Anita B.org. Before founding Paritii, she served as a Strategist at the Bill & Melinda Gates Foundation, where she strengthened health systems across diverse regions and led cross-functional initiatives to improve organizational impact. Shmona holds a DPhil from Oxford University, where she was a Rhodes Scholar. Subscribe to future talk announcements: Anyone outside Bell Labs can receive talk announcements by subscribing to the mailing list. To subscribe, send an empty email with the subject line "Subscribe RAI" to daniele.quercia@nokia-bell-labs.com<mailto:daniele.quercia@nokia-bell-labs.com>.