UC Berkeley researchers develop diffusion model to better simulate rare, extreme outcomes

Planning for what usually happens is relatively straightforward. Preparing for what rarely happens, but can have enormous consequences, is much harder. Businesses, governments and financial institutions rely on averages to estimate typical demand, costs and resource needs. But a sudden spike in demand or an unusually large financial loss can fall well beyond those typical patterns, leaving systems strained precisely when the stakes are highest.

The consequences of being unprepared for extreme events are increasingly visible. This summer, repeated heat waves across Europe disrupted daily life and economic activity. Such episodes illustrate a broader challenge for planners: preparing for outcomes that fall well outside the patterns seen most often in historical data. New UC Berkeley research approaches one part of that challenge from the perspective of generative AI — by asking whether models used to simulate possible outcomes can accurately reproduce rare but consequential ones.

Researchers in UC Berkeley’s Department of Industrial Engineering and Operations Research, or Berkeley IEOR, tackle that question in a paper recently published online in the journal Operations Research. The study, “Learning to Simulate from Heavy-Tailed Distribution via Diffusion Model,” was co-authored by Haoyu Liu and Tingyu Zhu, who contributed equally as co-first authors, along with Nanshan Jia, Jinghai He and Berkeley IEOR associate professor Zeyu Zheng.

The research focuses on heavy-tailed data, in which extreme outcomes are rare but occur more often than light-tailed statistical models, such as the familiar bell-shaped normal, or Gaussian, distribution, would suggest. Stock market returns and sudden spikes in customer demand can exhibit this pattern. For organizations determining how much inventory, capital or capacity they may need, underestimating the likelihood of those extremes can have significant consequences.

The researchers show mathematically that conventional diffusion models can struggle to reproduce this behavior. Diffusion models learn to generate new data by progressively adding noise to observed data and then learning to reverse that process. Standard diffusion models typically rely on Gaussian noise — an assumption that, the researchers found, can make extreme regions of a heavy-tailed distribution difficult to learn during training and reproduce during generation.

Their solution, called Score-based Heavy-tailed Diffusion, or SHD, changes that fundamental part of the diffusion process. Rather than relying on Gaussian noise, the Berkeley method uses a Student’s t distribution, which assigns more probability to extreme values than a Gaussian distribution, and begins generation from a heavy-tailed distribution. The researchers designed the approach so the choice of noise can be guided by estimates of the tail behavior in the underlying data. The goal is to help the model better capture the full distribution — including the sparsely observed extremes — rather than concentrating primarily on its most common values.

The researchers tested the approach on synthetic problems as well as two real-world data sets involving bicycle demand and stock returns. In the bike-sharing experiment, the model used rental activity during the first half of a day to generate possible demand patterns for the second half. SHD outperformed the comparison models overall, with its largest advantage on weekends, which represented a minority of the observations and followed a different demand pattern from weekdays. In another experiment using returns from 20 stocks, SHD generated distributions that more closely resembled the observed data than the comparison models. For an equally weighted portfolio, its Wasserstein distance — a measure in which lower values indicate distributions that are more similar — was 0.285, compared with 0.586 for a standard score-based diffusion model using Gaussian noise.

The immediate contribution is methodological: The work gives researchers a new way to build data-driven simulators when rare outcomes matter and observations of those outcomes are limited. Over time, such simulators could be useful in areas such as financial risk analysis, supply chain planning, transportation demand and service-system operations, where failing to account for unusual outcomes can carry significant costs.

The approach is not designed to predict the next crisis or extreme event. Rather, it addresses a more fundamental question: If organizations increasingly use generative AI to build simulations, can those models reproduce not only the most likely outcomes, but also the rare ones that may matter most?