USER EXPERIENCE COMPUTATION RESEARCH ARCHITECTURE ABOUT

Momentos

A Tangible Memory Box for Understanding Memory Distortion Through Object-Based Interaction MAS. 834 Tangible Interfaces @MIT Media Lab

[Date]
2024.10 - 2024.12
[Location]
Cambridge, MA
[Team Member]
Hui-Ying Suk, Grace Mai, Jingfei Huang, Hua Wang
[Role]
Prototype Implementation, Interaction Design
[Keywords]
Tangible Interfaces, Interactive Storytelling, Personal Narratives

Abstract

Memory distortion profoundly impacts human cognition and self-perception, shaped by factors such as emotions, social dynamics, and time. We presents \textit{Mementos}, a novel tangible interface that visualizes and interacts with the factors of memory distortion. Leveraging video object detection, RFID technology, and interactive storytelling, Mementos integrates physical mementos to explore how memories evolve under various influences. The system simulates memory reconstruction, stripping away emotional and temporal distortions to help users introspect and reevaluate their narratives. We draw insights from cognitive psychology, such as the roles of emotional arousal and social cues in reshaping recall, and incorporate these into the design. By illustrating the experience through the story of one user, this work demonstrates Mementos' potential for emotional healing, education, and interdisciplinary design at the intersection of psychology and art.

Introduction

Memories are essential to human cognition, shaping our identities, relationships, and emotional well-being. However, memories are not static but are reconstructed and reshaped over time, influenced by emotional states, social dynamics, and temporal decay. This malleability, while adaptive, can lead to biases and distortions that impact how we perceive our past and make decisions in the present. Despite significant research in psychology and neuroscience on memory distortion, the tangible exploration of these processes remains limited in human-computer interaction (HCI).

Interactive systems have the potential to externalize the subjective experience of memory, enabling reflection and understanding of how memories evolve. By translating abstract concepts such as emotional arousal, social influence, and temporal reconstruction into tangible and visual experiences, these systems can deepen our engagement with personal narratives. While previous work in HCI has explored memory archiving, retrieval, and storytelling, there is a gap in tools designed to actively interrogate and reflect on the distortions inherent in memory.

We introduces Mementos, an interactive memory box that examines the malleability of human memory through object-based interaction. Drawing on insights from cognitive psychology and memory science, Mementos integrates tangible artifacts with digital visualization to simulate how emotions, relationships, and time distort recollections. By presenting speculative use cases and exploring a user's interaction journey, this work illustrates how such a system can foster introspection, emotional healing, and interdisciplinary dialogue at the intersection of art and psychology.

We contribute:
(1) A framework for translating psychological factors of memory distortion into interactive system design.
(2) An implementation of Mementos, which leverages tangible interaction and digital storytelling to externalize and reflect
on memory reconstruction processes.
(3) Insights into how interactive systems can be applied to contexts such as emotional healing, education, and creative ex-
pression.

By bridging concepts from memory science with tangible interaction, Memetos highlights the potential to deepen our understanding
of the subjective and dynamic nature of memory.

Our Memories DECEIVE Us.

Design Methods

The design of Mementos was guided by the dual goals of educating users about memory distortion and encouraging introspection through tangible interaction. By integrating psychological insights and storytelling practices, the system translates abstract cognitive concepts into interactive experiences that allow users to engage with their evolving recollections.

To achieve these goals, Mementos incorporates key psychological factors influencing memory distortion, including mood, social dynamics, and the passage of time. Mood-based distortions manifest through heightened emotional arousal, which enhances recall vividness while introducing biases. Social dynamics, such as shared narratives and misinformation, reshape memory content through external influence. Finally, temporal decay and retrospective reconstruction reveal how memories degrade or are reconstructed over time, often blending with schemas and beliefs.

The design draws inspiration from traditional memory boxes, physical repositories of significant artifacts, and extends this concept with interactive storytelling techniques. This combination allows for the integration of physical and digital mementos, where objects act as triggers for memory recall and manipulation. The system visualizes memory distortion in real time, enabling users to observe how their recollections evolve as they interact with tangible artifacts. By externalizing these processes, Mementos fosters self-reflection, offering a deeper understanding of how memories are shaped and reshaped by internal and external influences.

Prototype Implementation

Mementos project integrates computer vision and RFID technologies to create a dynamic interactive system. The primary detection mechanism relies on visual object recognition using a similarity-based classification approach. To ensure reliability in scenarios where visual detection may fail, such as when a user’s hand blocks the object, the system utilizes RFID-based reader detection. Together, these complementary methods enhance the interaction experience by addressing potential challenges in real-world use.

The visual detection system employs a camera to capture real-time video frames, which are analyzed using the OpenAI CLIP model. Objects in the system are represented as feature vectors derived from encoded text concepts, such as polaroid, key, or flower. Each frame captured by the camera is encoded into a feature vector and compared to these predefined object vectors through cosine similarity. By smoothing similarity scores using a rolling average, the system stabilizes its detections, reducing noise and improving accuracy. A predefined threshold ensures that only objects with sufficiently high similarity scores are recognized, and decision logic prevents repetitive detections of the same object consecutively.

While visual detection serves as the primary mechanism, RFID technology provides a fallback solution for scenarios where the camera's view is obstructed. Physical objects are tagged with unique RFID stickers, and a RFID reader scans continuously for tags that match the initial UID settings. If an object’s visual features cannot be detected, the system defaults to identifying the object using the RFID data. This dual-detection approach ensures that object recognition is not disrupted by occlusions, improving the robustness of the tangible interaction experience.

The system prioritizes visual recognition for its ability to dynamically respond to user interactions and display feedback in real-time. However, when visual detection fails, RFID provides reliable supplementary input to maintain the continuity of interaction. The detected object, whether identified through vision or RFID, triggers an associated action, simulating a keyboard press, that acts as the input to TouchDesigner.

By combining these two complementary technologies, the project addresses the limitations inherent in each method. The visual detection system offers dynamic recognition capabilities, while the RFID system enhances reliability in challenging conditions. Together, these technologies create a robust platform for users’ interaction, allowing users to engage with objects in real-world settings without interruption. This hybrid approach demonstrates the potential of combining vision-based and RFID methods to enhance human-computer interaction systems.

User Interaction Scenarios

Mementos enables users to engage with memory distortion through tangible interaction and dynamic visualization. By connecting physical artifacts with digital storytelling, the system provides a platform for exploring how emotional intensity, social influence, and the passage of time reshape memory. The interaction focuses on the simple yet impactful act of placing or removing objects, allowing users to observe how these factors distort memory representations. The example scenarios illustrate how this interaction facilitates introspection and emotional engagement:

Users place tangible artifacts into the system to trigger memory visualizations. Each artifact represents a specific factor, such as emotional intensity or temporal decay, grounding the abstract concept of memory distortion in a personal and physical interaction.
By observing how visualizations change as different objects are introduced or removed, users witness the dynamic reconstruction of memories. This interaction encourages reflection on the factors influencing their recollections, such as emotional biases or time-based decay.
Comparing the visualized distortions triggered by different artifacts enables users to identify how specific emotions, relationships, or beliefs shape their perception of the past. This process fosters self-awareness and critical reflection on the reliability of their memories.

Design Challenge

Several challenges emerged throughout the design and development of Mementos.

Visualizing memory distortion, especially in emotionally charged memories, risks triggering discomfort or distress. The system must carefully balance engagement with sensitivity, ensuring users feel supported rather than overwhelmed.
While the design draws on established frameworks from cognitive psychology, simplifying complex theories into intuitive interactions introduces trade-offs between usability and scientific fidelity. Representing nuanced distortions, such as schema-driven reconstruction or social influences, remains an ongoing challenge.
The need to keep the system accessible and engaging limits the complexity of interactions. Achieving depth without overwhelming users requires iterative refinement of both tangible and digital components.