The SAKURA project (System for Augmented Kinetics and Unified Rehabilitation Assessment) involves the development of an intelligent system based on Computer Vision and Artificial Intelligence algorithms and augmented reality (AR) technologies for remote medical rehabilitation. The system allows the doctor to wear an augmented reality headset and perform exercises that involve interaction with objects (e.g., pouring a glass of water) that the patient, located in a remote location, wearing another augmented reality headset, will then repeat. The doctor’s interaction with the real objects present in the environment is virtualized and displayed in the patient’s environment as holograms, allowing the patient to perform the doctor’s gestures with high precision. In order to create the SAKURA system, it is necessary to study and design artificial intelligence algorithms capable of automatically analyzing egocentric images and videos to infer the 3D position and shape of hands and objects with which users interact in the reference system of the environment in which the user is located. The discrepancy between the actual position of the hands and objects and the ideal position will be quantitatively measured, providing detailed feedback for follow-up and rehabilitation assessment.