Category Archives: project demo

Adaptive Virtual Environments for ASD Attention Training

A prototype of adaptive virtual environments therapy system (AVET) was developed which will enable innovative Virtual Reality (VR)-based therapy approach for children with attention deficit on the autism spectrum. Many systems have successfully used VR in Autism Spectrum Disorders (ASD) therapies. Most of them use VR as an alternative way to conduct therapies by simulating traditional therapies or real-life experiences. The AVET employed VR-exclusive “impossible experiences” (e.g., a chair that deforms upon the user’s gaze, a transparent human) which are not available in real world. The AVET identifies, influences the user’s cognition, and delivers a customized Prolonged Exposure (PE)-style VR therapy for children with attention deficits on the autism spectrum.

The overall logic of AVET is to first expose the user to a Virtual Environment (VE) to finish a task (e.g., listen to a virtual teacher’s talking) with potential distractions. At the second step, the system will detect the user’s attention with eye-tracking data and physiological signals, then remove all the detected distractions (e.g., a red mug, the virtual teacher’s necklace) from the VE. The last step is to gradually put all distractions back into the VE, which will allow the user exposure to one more distraction object at a time. In this step, the system will provide positive/negative reinforcements to reinforce the desired specific attention. Figure 1 – a) shows the initial VE which is a classroom and a virtual teacher stands in front of the user. The Cyan and the red balls representing the current left and right eye gazing points of the user. The eye gazing point in this figure is just for demonstration purpose, it is hidden from the actual users. b) shows the user is gazing at the teacher. c) shows the user is distracted by the globe and gazed at the globe. d) shows another distraction – the table. e) shows that all the distractions have been removed from the scene. f) shows one of the previous detected distractions, the table, is reintroduced to the scene.

Figure 1

We conducted a preliminary evaluation to the current AVET prototype with the experts. Based on the interview feedbacks, we anticipate the AVET will have a great potential to deliver innovative and effective ASD attention training therapies.

Customizable Virtual Human and Imagination Soccer

Virtual Reality (VR) training games have many potential benefits for autism spectrum disorder (ASD) therapy, such as increasing motivation and improving the abilities of performing daily living activities. Persons with ASD often have deficits in hand-eye coordination, which makes many activities of daily living difficult. A VR game that trains hand-eye coordination could help users with ASD improve their quality of life. Moreover, incorporating users’ interests into the game could be a good way to build a motivating game for users with ASD.
I developed a Customizable Virtual Human (CVH) which enables users with ASD to easily customize the appearance of a virtual human and then interact with the CVH in a 3D task. Specifically, we investigated the effects of CVHs with a VR hand-eye coordination training game – Imagination Soccer – and conducted a user study on adolescents with high functioning ASD.

Cooking Simulator


This project worked with Oculus Rift and Razer Hydra. I made this project for my research on evaluating the usability of 3DUI for kids with autism. With the virtual hands controlled by users’ real hands, users are able to pick up milk/cereal box, pour and mix cereal and milk in the bowl. Measuring cups could be used to accurately deliver a certain amount of milk and cereal. During the users interactions with the simulator, data such as the movement trail of user’s hands, tasks performances etc. were recorded and analysed. I am trying to find the general guidelines for building 3DUI Virtual Reality tools for autism therapy. To download and try this simulator please click here.

In addition, I made small tests to measure your sense of manipulating  3D virtual objects, it could be download by click here. The task is simply rotating or translating the house at the left side to make its orientation the same as the house at the right side.