Projects

The work presents the first human memory augmentation system that can construct a synthetic spatiotemporal memory for objects in an indoor environment. We named it as ExoMem. Our solution leverages augmented reality (AR) and artificial intelligence (AI) and comprises two components:

• AR goggles that sense the environment, exchange data over a wireless network and construct a spatiotemporal memory in the AR environment.
• A computing module that performs computer vision-based localization and object detection on first-person view data received from the AR goggles.

To demonstrate the efficacy of our system, we designed object memorization and recall tasks and measured mental workload and performance during these tasks. In the memorization task, participants completed a 20-minute tour of the three floors of the building and had to memorise the location of ten different objects they saw along the path. In the recall task, participants had to recall the positions of the memorised objects in a computer-based test with and without using AR-based spatiotemporal memory.

In the video below, we show a participant completing the memorization task with the ExoMem:

In the next video, we show computer vision-based user localization using ArUco fiducial markers and object recognition using YOLO Version 4 object detector running on the computing module of our system:

In this following video we show a participant completing the recall test with assistance of ExoMem:

According to subjective evaluations, when using ExoMem, participants experienced much less mental demand and effort in both tasks. Similarly, participants’ experience of temporal demand and frustration decreased with ExoMem in each task. In contrast, physical demand results indicated that participants experienced more load in Task 1 with ExoMem than without it, whereas they experienced less load in Task 2 with ExoMem than without it.

Objective evaluations indicated that participants made 7.52 times fewer errors on the recall test and completed the test spending 27% less time when using the AR system. The evaluation of the usability of the system, indicated a System Usability Scale score of over 80% among the participants.

The results highlight the potential of AR and AI technologies in developing a new generation of memory assistant systems to enhance and compensate for human memory functions.

All the codes developed for our memory enhancement system are available at the GitHub repository.