By creating a computer program that turns sequences of events from a video into unique geometric shapes, Dartmouth researchers are analyzing how the brain creates, uses, and stores memories. When compared, the resulting shapes can further the knowledge of the memory experience.
We model experiences and memories as trajectories through word-embedding spaces whose coordinates reflect the universe of thoughts under consideration. Memory encoding can then be modelled as geometrically preserving or distorting the ‘shape’ of the original experience.
Jeremy R. Manning
“When you experience something, its shape is like a fingerprint that reflects its unique meaning, and how you remember or conceptualize that experience can be turned into another shape. We can think of our memories like distorted versions of our original experiences. Through our research, we wanted to find out when and where those distortions happen (i.e. what do people get right and what do people get wrong), and examine how accurate our memories of experiences are,” he added.
Their approach was applied to a group of participants that watched and verbally recounted a television episode from Sherlock, while undergoing functional neuroimaging. The dataset also contained detailed scene-by-scene annotations of the episode.
The team ran those annotations through their computer program, and a “topic model” for the episode was created using computer modeling. When these findings are represented in 2D, a connect-the-dots-style representation of subsequent events emerges. The shape of that representation represents how the episode’s thematic substance evolves over time, as well as how distinct events are connected. The researchers employed a similar method to determine the morphologies of each of the participants’ accounts of the episode’s happenings. In this way, they were able to identify which aspects of the episode people tended to remember accurately, forget or distort.
One of our most intriguing findings was that, as people were watching the episode, we could use their brain activity patterns to predict the distorted shapes that their memories would take on when they recounted it later.
Jeremy R. Manning
This implies that some facts about our ongoing experiences become corrupted in our brains as soon as they are stored as new memories. Even when two people witness the same physical event, their subjective perceptions of it begin to diverge as soon as their brains begin to make sense of what happened and distill it into memories.
This research can be used in other domains, including in health and education, the applied methods of modeling the shape of memories provide ways of assessing a patient’s level of understanding or the possible progress of a student during a lecture or a course.
Geometric models reveal behavioural and neural signatures of transforming experiences into memories, Andrew C. Heusser, Paxton C. Fitzpatrick & Jeremy R. Manning
Published: September 2018
DOI: 10.1038/s41562-021-01051-6