I am an Associate Professor in the Department of Epidemiology and Biostatistics at the University of California, San Francisco and the UCSF-UC Berkeley Joint Program in Computational Precision Health.

As a principal investigator at the UCSF-Stanford Center of Excellence in Regulatory Science and Innovation (CERSI), I also collaborate closely with researchers from the US Food and Drug Administration to develop methods that improve the safety, reliability, and interpretability of artificial intelligence (AI)/machine learning (ML) algorithms in healthcare. Recent projects include diagnosing performance drops in AI/ML algorithms, using Large Language Models (LLMs) for scaling regulatory science, and understanding the statistical limits of LLM-as-a-judge for evaluating generative AI systems.

I am also the data science lead on the PROSPECT team, the digital innovation taskforce for the Zuckerberg San Francisco General Hospital. Recent projects by the PROSPECT team include the deployment of a readmission risk prediction model and the development of an LLM pipeline to summarize patient charts.

I completed my Ph.D. in Biostatistics under Noah Simon and Erick Matsen at the University of Washington. Before that, I studied computer science at Stanford and was a software engineer at Coursera.

CV / GitHub / Twitter / Google Scholar

Email:

News

• Jan 2026: Our work on understanding the statistical limits of LLM-as-a-judge was accepted at AISTATS 2026! Congratulations to the team, especially to Patrick for leading the effort! And special thanks to Yifan Mai for sharing data from HELM for this project! Excited for Morocco!
• Dec 2025: Our work on developing an LLM-based summarization and information extraction engine for the social workers at the SF safety net hospital was accepted at AAAI 2026! Incredible work by Avni Kothari and Patrick Vossler, and deep gratitude to the social workers for their engagement in this work! See you all in Singapore!
• Dec 2025: Our work on learning concept bottleneck with LLM priors in a statistically principled framework was accepted at NeurIPS! Thank you to my wonderful coauthors Yan Shuo Tan, Chandan Singh, Avni Kothari, and Luke Zier!
• Jun 2025: Our work on testing for subgroup-level performance decay of ML algorithms was accepted at ICML! Congrats Harvineet and the rest of the team!
• Jan 2025: We just published a review in NEJM AI on real-world clinical AI monitoring systems, which was a wonderful collaboration between Romain Pirracchio, Fan Xia, and Karandeep Singh. TL;DR Not all clinical AI monitoring systems are created equal!
• Oct 2024: Our lab has two NeurIPS workshop papers accepted! One on explaining performance differences of ML algorithms across domains and one on leveraging LLMs as Bayesian priors to fit Concept Bottleneck Models.
• Sep 2024: Our work on explaining performance differences of ML algorithms across domains was accepted at NeurIPS! Congratulations Harvineet and the rest of the team!
• June 2024: Congratulations Harvineet Singh and Avni Kothari for acceptances for oral presentations at the AMIA Annual Symposium!
• Mar 2024: More exciting news! Our proposal for supplemental research funding on studying robustness of LLMs in healthcare was approved for PCORI funding! The project will extend our current work on creating diagnostic tools for ML-based clinical decision support systems, but now encompassing LLMs.
• Feb 2024: I presented our lab’s work on QA and QI for ML-based medical devices at the inaugural AI Regulatory and International Symposium, hosted by the US FDA and Korea MFDS and attended by regulatory bodies from over twenty countries! It was an honor to share our work with regulators and industry members, and energizing to see academia, regulators, and industry all come together to shape regulatory policies for AI in healthcare.
• Jan 2024: What a month! Our lab has one paper accepted at the Conference on Causal Learning and Reasoning (CLeaR): “Monitoring the performance of machine learning algorithms that induce feedback loops: what is the causal estimand”. We also have two papers accepted at AISTAT: Is this model reliable for everyone? Testing for strong calibration (oral!) and Monitoring machine learning-based risk prediction algorithms in the presence of performativity.
• Nov 2023: Our lab has two papers accepted at the Regulatable AI Workshop at NeurIPS! One on a causal monitoring framework for ML-based medical devices (selected for oral) and one on sample size calculations for model fairness audits.
• Sep 2023: Avni Kothari has joined our lab as a data scientist! We’re growing!
• Apr 2023: Harvineet Singh will be joining as a postdoctoral researcher in the lab in July! We’re really excited to have him!
• Nov 2022: Big news! Our grant proposal on developing diagnostic tools for ML algorithms has been funded by PCORI! See the project description here.
• Nov 2022: Thank you to SER digital for the opportunity to discuss the intersection of ML and epidemiology! Slides are here.
• May 2022: Our paper “Clinical Artificial Intelligence Quality Improvement: Towards Continual Monitoring and Updating of AI Algorithms in Healthcare” is published in Nature Digital Medicine! Check it out here.
• Apr 2022: I gave a talk at the Biometrics Journal Club on the statistical considerations when regulating ML-based medical devices that evolve over time. The talk is based on our 2021 Biometrics paper “Approval policies for modifications to machine learning-based software as a medical device: A study of bio-creep”. Slides are here.
• Oct 2021: Romain Pirracchio and I have received a grant to extend our original UCSF-Stanford CERSI proposal “Safe algorithmic change protocols for modifications to AI/ML-based Software as a Medical Device”. This new project will look at the impacts of integrating real-world data. We’re excited to continue working with our CDRH collaborators Berkman Sahiner and Alexej Gossmann!
• Sep 2021: Our work on online recalibration and revision of clinical prediction models was presented at BIOP 2021.
• Aug 2021: I gave a talk as part of the ASA Statistical Learning and Data Science webinar series on deep learning! Check out the slides here.
• Sep 2020: Romain Pirracchio and I have received a grant from the UCSF-Stanford CERSI program for our proposal “Safe algorithmic change protocols for modifications to AI/ML-based Software as a Medical Device”. This grant will be done in close collaboration with Berkman Sahiner and Alexej Gossmann from the US FDA.

Tutorials and Short Courses

• June 2022: I co-taught the Columbia ML bootcamp with Noah Simon and Cody Chiuzan for the third time!
• Aug 2021: I joined forces again with Noah Simon and Cody Chiuzan to teach a short course on machine learning for biomedical and health data.
• Nov 2020: I taught a short course on supervised statistical learning with Ali Shojaie on November 15th at the 6th Seattle Symposium in Biostatistics. (Slides)
• Aug 2020: I taught a short course on machine learning for biomedical and health data with Noah Simon and Cody Chiuzan.