Reinforcement Learning Can Make Latent Private Information More Accessible in Language Models

Niloofar Mireshghallah’s group at Carnegie Mellon University, led by CMU student researcher Renfei Zhang, studies how reinforcement learning changes which information language models can retrieve from their parameters. The group trains models with RL and measures when previously latent knowledge becomes easier to retrieve or regurgitate, especially when it includes memorized private data. This work aims to identify privacy risks introduced by post-training and inform safer training and release evaluations.

Biography

Niloofar Mireshghallah is an Assistant Professor at CMU with a joint appointment in the Language Technologies Institute (LTI) and the Department of Engineering & Public Policy (EPP), and a core faculty member of CyLab. Her research focuses on privacy-preserving AI, contextual integrity in LLMs, the societal implications of ML, and AI for science and health. Her benchmarks for evaluating privacy in LLMs are used in industry and referenced in safety evaluations of frontier labs. Her work has been covered by The Washington Post and WIRED. Previously, she was a Research Scientist at Meta FAIR’s Alignment group and a postdoc at UW. She received her Ph.D. from UC San Diego in 2023. Her honors include the Rising Star in Adversarial ML Award, Rising Stars in EECS, the NCWIT Collegiate Award, and finalist distinctions for the Greenwall Faculty Scholars and Qualcomm Innovation Fellowship. She is also a founding member of humans&.