Advancing the science of AI systems

Detecting Perspective Shifts in Multi-agent Systems

Methods for detecting when agents in multi-agent systems experience shifts in their perspectives or behaviors.

Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations

Statistical testing framework for detecting meaningful differences in LLM responses under query perturbations.

Gaussian mixture models as a proxy for interacting language models

Using Gaussian mixture models to approximate and analyze the behavior of interacting language model systems.

Toward a Digital Twin of U.S. Congress

Empirical evidence that a collection of language models parameterized by real people's data satisfies the definition of a digital twin.

ACL 2025

Statistical inference on black-box generative models in the data kernel perspective space

Providing theoretical and empirical support for predicting the behavior of generative models using vector representations.

2024

Investigating social alignment via mirroring in a system of interacting language models

Investigating the effect of a simple "mirroring" mechanism on the dynamics of a system of interacting language models.

EMNLP 2024

Tracking the perspectives of interacting language models

Methods for tracking how language models' perspectives evolve through interaction.

Consistent estimation of generative model representations in the data kernel perspective space

Providing theoretical support for using vector representations of populations of generative models.

MedFuzz: Exploring the robustness of large language models in medical question answering

Systematic evaluation of LLM robustness in medical QA through adversarial perturbations and fuzzing techniques.

2023

A statistical Turing test for generative models

A statistical framework for evaluating whether generative model outputs are distinguishable from real data.

Comparing Foundation Models using Data Kernels

A method for comparing foundation models by analyzing their behavior through data kernel representations.