Generative AI using diffusion modelling
The diffusion group at SCDLDS seeks to explore various theoretical questions in the subfield of diffusion, which is the state-of-the-art theory employed in the best generative models around the world such as image/video generators, and recently in LLMs.
Our world is increasingly becoming data-driven, with vast amounts of data hiding decades of patterns, subconscious behavior and telling insights. Of major interest in the generative AI community is the problem of heavy-tails; informally, a heavy-tailed random variate tends to deviate extremely often and to great effect, from its mean. A significant quantity of decision making occurs in the face of heavy tailed phenomena, and culminates in the generation of various predictive scenarios in fields such as finance, climate, and weather. Examples include rainfall prediction, volatile stock price prediction, and the prediction of catastrophes such as cyclones and heat waves.
Modern generative modeling has no comprehensive answer to the problem of heavy-tailed generation, with various fixes and patches unable to cover the gaping hole. Our aim is to supply the literature with a number of methods that we believe can address the problem at least in the context of the other projects handled by the center.
In the existing literature, there are some known methods for dealing with heavy-tailed tilting. One involves the transformation of samples before diffusion (such as log transforms), modifying the score learned by a neural net in the vanilla forward process, or using a different forward process altogether which has a heavy-tailed prior. We have explored all these techniques in our discussions, although only two of them reflect in our past and present work.
Chiefly, among the problems we have dealt with comprehensively is tilting, which employs importance sampling to generate from rare events arising from experiments in finance and weather. We have demonstrated the existence of an optimal sampler, and the accuracy of a diffusion mechanism capable of generat- ing tilted samples. Furthermore, our final theorem is a hardness result for tilted sample generation: it is simply impossible to tilt beyond a certain amount, using a certain number of samples. This theory has been extended to the heavy-tailed setup.
Our current interests lie in a more fundamental exploration of diffusion, where we study precisely what about diffusion modeling favors light tails over heavy tails in the context of kernel methods for score learning. More precisely, we study to what extent an empirical score estimator matches the true score, and how this discrepancy manifests itself during inference, for heavy tailed and light tailed priors. Initial results are rather promising, and we expect to conclude with a clearer picture of whether the kernelized score estimator can be improved under information about the heavy-tailed parameters, and whether this will result in improved generation.