After college I worked with Dr. Ilya Monosov to study the brain as it learned risks and rewards in an uncertain environment. My thought back then was that studying how biological neurons encoded risks and rewards could inform the design of artificial neural networks. At the time, there was a breakthrough in using neural networks to learn how to play video games without human supervision by using "only the pixels and the game score as inputs". However, when watching videos of the neural network's performance, I noticed it often failed to learn how to play games where the game score increased very rarely. I thought one way to remedy this was to design a cost function that could "motivate" the network to learn how to play in the absence of frequent rewards.
I kept this in mind as I ran experiments to measure a monkey's neuronal activity in the basal forebrain while it watched visual cues that indicated the probability that it would receive a fruit juice reward in the next few seconds. I ended up characterizing two neuronal signals: one was a pre-reward ramping signal that was associated with anticipating a probablistic reward in the future, and the other was a post-reward spiking signal that was associated with receiving a reward that had a low probability of occuring. I felt these signals could be useful in the context mentioned earlier: pre-reward ramping ties together a visual cue and possible future reward, even though they don't occur simultaneously, and post-reward spiking draws the most attention to the rarest rewards, and could be a powerful learning signal in the absence of frequent rewards.