technical reports & published papers

advancing the science of agent evaluation and monitoring

our research focuses on developing statistically principled monitoring and evaluation tools for models, agents, and multi-agent systems.

2026

ICML 2026

Detecting Perspective Shifts in Multi-agent Systems

Methods for detecting when agents in multi-agent systems experience shifts in their perspectives or behaviors.

Monitor

ICML 2026

Query-efficient model evaluation using cached responses

Leveraging cached responses from previously-evaluated models to reduce the number of queries needed for accurately evaluating new models on benchmarks.

Evaluate

arXiv

Black-box model classification under the discriminative factorization

A framework for evaluating query set quality when classifying black-box AI models, demonstrating exponential decay of classification error with query budget.

Evaluate

arXiv

Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models

Efficient susceptibility prediction and effective defense transfer for jailbreak defense / blue teaming.

Evaluate

arXiv

When prompt perturbations break your A/B test: A valid statistical test for generative surveying

Valid & robust statistical analysis of generative surveying data.

Evaluate

ACL 2026

Toward a Digital Twin of U.S. Congress

Empirical evidence that a collection of language models parameterized by real people's data satisfies the definition of a digital twin.

Monitor

ACL 2026

A Model of The Language Process

Expanding the capabilities of prediction with explicit time-conditioning.

Monitor

2025

arXiv

Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations

Statistical testing framework for detecting meaningful differences in LLM responses under query perturbations.

Evaluate

arXiv

Gaussian mixture models as a proxy for interacting language models

Using Gaussian mixture models to approximate and analyze the behavior of interacting language model systems.

Monitor

ACL 2025

Statistical inference on black-box generative models in the data kernel perspective space

Providing theoretical and empirical support for predicting the behavior of generative models using vector representations.

Evaluate

2024

arXiv

Investigating social alignment via mirroring in a system of interacting language models

Investigating the effect of a simple "mirroring" mechanism on the dynamics of a system of interacting language models.

Monitor

EMNLP 2024

Tracking the perspectives of interacting language models

Methods for tracking how language models' perspectives evolve through interaction.

Monitor

arXiv

Consistent estimation of generative model representations in the data kernel perspective space

Providing theoretical support for using vector representations of populations of generative models.

Evaluate

arXiv

MedFuzz: Exploring the robustness of large language models in medical question answering

Systematic evaluation of LLM robustness in medical QA through adversarial perturbations and fuzzing techniques.

Evaluate

2023

arXiv

A statistical Turing test for generative models

A statistical framework for evaluating whether generative model outputs are distinguishable from real data.

Evaluate

arXiv

Comparing Foundation Models using Data Kernels

A method for comparing foundation models by analyzing their behavior through data kernel representations.

Evaluate

research partners

collaborating with leading institutions to advance AI research

Johns Hopkins University

interested in collaborating?

we're always looking for research partners and collaborators.

Get in touch

HELIVAN