We’re hiring for three positions:
- a post-doc to work on post-market monitoring of ML algorithms
- a post-doc to work on LLM auditing
- a post-doc/data scientist to be part of the ZSFG Predictive Analytics group
Postdoctoral researcher on causal inference for monitoring machine learning algorithms
The project:
After a machine learning (ML)-based system is deployed, monitoring its performance is important to ensure the safety and effectiveness of the algorithm over time. When an ML algorithm interacts with its environment, the algorithm can affect the data-generating mechanism and be a major source of bias when evaluating its standalone performance, an issue known as performativity. Although prior work has shown how to validate models in the presence of performativity using causal inference techniques, there has been little work on how to monitor models in the presence of performativity. The goal of this project is to bring together techniques from causal inference and statistical process control to develop a comprehensive framework for post-market monitoring of ML algorithms. This work builds on a number of our previous works, including this paper that was published at the Conference of Causal Learning and Reasoning (CLeaR) and was presented at the NeurIPS Regulatable AI workshop.
We are seeking a postdoctoral researcher to join our lab. The primary responsibilities are:
- Develop new statistical methods/frameworks for post-market monitoring of ML algorithms
- Implement a software package that can be readily used by ML developers, health AI deployment teams, and ML auditors/regulators
- Write, edit, and publish research manuscripts in collaboration with the team
Our team is highly collaborative and includes members with wide-ranging expertise:
- Jean Feng: PI of the lab. Primary advisor of postdoctoral researcher.
- Fan Xia: Co-advisor of postdoctoral researcher. Assistant Professor in the Department of Epidemiology and Biostatistics at UCSF. Research interests include causal inference, clinical trial design, and machine learning.
- Alexej Gossmann: Staff Fellow and mathematical statistician in the Division of Imaging, Diagnostics, and Software Reliability (CDRH/OSEL/DIDSR) at the FDA. Research interests include performance evaluation of AI/ML-enabled medical devices and software in medicine.
- And many others, including Berkman Sahiner, Gene Pennello, Adarsh Subbaswamy, Nicholas Petrick, Romain Pirracchio, and Mi-Ok Kim!
The position:
We are looking to hire a postdoctoral researcher to join the team. The position (100% funded) will be for two years. Salary and benefits are competitive.
Qualifications:
The post-doctoral researcher position requires at least a PhD degree in (bio)statistics, computer science, data science, or another relevant field. We are looking for someone who:
- has experience in at least one of these fields: sequential monitoring, machine learning, and causal inference
- has experience in methodological development and can perform independent research, with a strong and relevant publication record
- has strong software engineering background (e.g. python, git-based workflows, high-performance computing)
- is able to work collaboratively with a team
Applying
If you are interested, please submit the following materials to :
- A cover letter
- A CV summarizing your education and work experience so far
- The names and email addresses of three references
- A code sample
- One representative publication
Screening of applicants will begin immediately and will continue as needed throughout the recruitment period.
Posted 3/31/2024
Postdoctoral researcher on LLM auditing
The project:
The unprecedented ability of large language models (LLMs) to interpret text data with human-like reasoning is poised to transform many fields. Nevertheless, for LLMs to be safe and effective for use in high-risk domains like healthcare, it is crucial to understand biases embedded in this technology, as it has been shown to vary in performance across subgroups and even discriminate against minorities. This project aims to study and develop red-teaming solutions to audit LLMs and to understand the limits of current approaches. Methodologies developed in this project will be tested on real-world clinical data, including unstructured notes. This project is a supplement of our existing PCORI project “Diagnostic Tools for Quality Improvement of Machine Learning-Based Clinical Decision Support Systems” (see project description here).
We are seeking a postdoctoral researcher to join our lab. The primary responsibilities are:
- Rigorously analyze and evaluate existing red-teaming algorithms for LLMs
- Develop new statistical methods/frameworks for comprehensive red-teaming of LLMs
- Develop an explanation framework and statistical inference procedures to understand systematic limitations of LLMs
- Write, edit, and publish research manuscripts in collaboration with the team
Our team is highly collaborative and includes members with wide-ranging expertise:
- Jean Feng: PI of the lab
- Julian Hong: Assistant Professor and Medical Director of Radiation Oncology Informatics in the Department of Radiation Oncology. Led one of the first randomized controlled studies of clinical machine learning. Research interests include the development and implementation of computational methods for providing personalized cancer care for patients, natural language processing of clinical notes, and evaluation of AI-based tools.
- Fan Xia: Assistant Professor in the Department of Epidemiology and Biostatistics at UCSF. Research interests include causal inference, clinical trial design, and machine learning.
- Alexej Gossmann: Staff Fellow and mathematical statistician in the Division of Imaging, Diagnostics, and Software Reliability (CDRH/OSEL/DIDSR) at the FDA. Research interests include performance evaluation of AI/ML-enabled medical devices and software in medicine.
- And many others, including Berkman Sahiner, Gene Pennello, Adarsh Subbaswamy, Nicholas Petrick, and Romain Pirracchio!
The position:
We are looking to hire a postdoctoral researcher to join the team. The position (100% funded) will be for two years. Salary and benefits are competitive.
Qualifications:
The post-doctoral researcher position requires at least a PhD degree in data science, (bio)statistics, computer science, or another relevant field. We are looking for someone who:
- has experience in training and testing ML algorithms for large datasets
- has experience in natural language processing and working with LLMs
- has experience in methodological development and can perform independent research, with a strong and relevant publication record
- has strong software engineering background (e.g. python, torch, huggingface, git-based workflows, high-performance computing, SQL, spark)
- is able to work collaboratively with a team
Applying
If you are interested, please submit the following materials to :
- A cover letter
- A CV summarizing your education and work experience so far
- The names and email addresses of three references
- A code sample
- One representative publication
Screening of applicants will begin immediately and will continue as needed throughout the recruitment period.
Posted 3/1/2024
Postdoctoral researcher/senior data scientist on the ZSFG Predictive Analytics team
The project:
Zuckerberg San Francisco General Hospital is at the forefront of applying artificial intelligence and machine learning (AI/ML) to help improve outcomes in vulnerable and underserved populations. We are growing the hospital’s predictive analytics team, whose charter is to leverage large datasets to improve patient care for all. Our team is dedicated to the development and testing of ML algorithms that support the hospital’s performance improvement efforts, with an emphasis on health equity and algorithmic fairness. We are committed to the translation of these algorithms into clinical practice and will be embedding all our algorithms into various clinical systems including the electronic health record (EHR) for rigorous testing and monitoring.
We are seeking a postdoctoral researcher/senior data scientist to join our team. The primary responsibilities are:
- Develop and test new ML algorithms that analyze structured data and clinical notes from the electronic health record (EHR) system
- Research and assess the use of large language models to develop interpretable and scalable clinical decision support systems
- Be up-to-date on state-of-the-art methodologies in the relevant technical fields and application domains
- Ensure that the developed ML algorithms are reliable and fair
- Write and edit research manuscripts in collaboration with the team
Our predictive analytics team is highly collaborative and includes team members with wide-ranging expertise:
- Jean Feng: Data analytics lead, assistant professor in the Department of Epidemiology and Biostatistics at UCSF
- Lucas Zier: Clinical lead, cardiologist at UCSF and ZSFG
- Jim Marks: Chief of the Medical Staff and Chief of Performance Excellence at ZSFG, anesthesiologist at UCSF and ZSFG
- Seth Goldman: Informatics Director for Technology Integration and Digital Health for the Office of Health Informatics for the San Francisco Department of Public Health, hospitalist at ZSFG
- Avni Kothari: Data scientist
The position:
We are looking to hire a postdoctoral researcher/data scientist to join the team. The position (100% funded) will be for one year, with the possibility of extension. Salary and benefits are competitive.
Qualifications:
The post-doctoral researcher/data scientist position requires at least a PhD degree or equivalent experience in data science, (bio)statistics, computer science, or another relevant field. We are looking for someone who:
- has experience in training and testing ML algorithms for large datasets
- has experience in methodological development and can perform independent research, with a strong and relevant publication record
- has interest in analyzing Electronic Health Record (EHR) data and natural language processing (prior experience is a plus)
- has strong software engineering background (e.g. git-based workflows, high-performance computing, linux, SQL, spark)
- is able to work collaboratively with a team
Applying
If you are interested, please submit the following materials to :
- A cover letter
- A CV summarizing your education and work experience so far
- The names and email addresses of three references
- A code sample
- One representative publication
Screening of applicants will begin immediately and will continue as needed throughout the recruitment period.
Posted 11/20/2023
The University of California is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or protected veteran status.