Al-Rafidain Engineering Journal (AREJ)

Evaluation of Physiotherapy Exercise by Motion Capturing Based on Artificial Intelligence: A Review

Document Type : Review Paper

Authors

1 Department of Computer Techniques Engineering, Technical Engineering College/Mosul, Northern Technical University, Mosul, Iraq

2 Department of Mechatronics Engineering, College of Engineering, University of Mosul, Mosul, Iraq

Abstract

Physical therapy is an important form of rehabilitation for patients suffering from a variety of disorders. Since professional physiotherapists are not always available, there is a need to introduce an intelligent system that assets the patients to perform the exercise by themselves. Any evaluation system consists of hardware interfacing, computers, processing, and evaluation tools. These tools made it easier to build methods for automating the evaluation of patient performance and advancement in functional rehabilitation. In this research, about one hundred research papers are classified according to the above-mentioned system parts. The review of current tools for capturing rehabilitative motions shows that the Kinect camera has been used in about 35% of the studies. This review concentrates on using machine learning techniques to evaluate motion in rehabilitation. The most relevant research for physiotherapy evaluation using deep learning have shown that the Convolutional Neural Network (CNN) is widely used by 44% of the researcher. A useful overview the collection of the reference datasets illuminates that the KIMORE dataset is popular and used by 38% as compared with other types of datasets. The advanced literature in the present peer-reviewed paper (2016–2022), includes primary studies and organized reviews.

Keywords

Main Subjects

References
  1. C-H. Huang, C.-F. Lin, C.-A. Chen, C.-H. Hwang, and D.-C. Huang, "Real-time rehabilitation exercise performance evaluation system using deep learning and thermal image," in 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), 2020: IEEE, pp. 1-6.
  2. S. Ravali et al., "A systematic review of artificial intelligence for pediatric physiotherapy practice: past, present, and future," Neuroscience Informatics, vol. 2, no. 4, p. 100045, 2022.
  3. Campbell, E. H. Coulter, P. G. Mattison, L. Miller, A. McFadyen, and L. Paul, "Physiotherapy rehabilitation for people with progressive multiple sclerosis: a systematic review," Archives of physical medicine and rehabilitation, vol. 97, no. 1, pp. 141-151. e3, 2016.
  4. A. Felipe et al., "Evaluation instruments for physical therapy using virtual reality in stroke patients: a systematic review," Physiotherapy, vol. 106, pp. 194-210, 2020.
  5. L. Chmielewski et al., "Low-versus high-intensity plyometric exercise during rehabilitation after anterior cruciate ligament reconstruction," The American journal of sports medicine, vol. 44, no. 3, pp. 609-617, 2016.
  6. Escalona, E. Martinez-Martin, E. Cruz, M. Cazorla, and F. Gomez-Donoso, "EVA: EVAluating at-home rehabilitation exercises using augmented reality and low-cost sensors," Virtual Reality, vol. 24, pp. 567-581, 2020.
  7. Paraskevopoulos, E. Spyrou, D. Sgouropoulos, T. Giannakopoulos, and P. Mylonas, "Real-time arm gesture recognition using 3D skeleton joint data," Algorithms, vol. 12, no. 5, p. 108, 2019.
  8. Maciejasz, J. Eschweiler, K. Gerlach-Hahn, A. Jansen-Troy, and S. Leonhardt, "A survey on robotic devices for upper limb rehabilitation," Journal of neuroengineering and rehabilitation, vol. 11, no. 1, pp. 1-29, 2014.
  9. V. Gauthier et al., "Video Game Rehabilitation for Outpatient Stroke (VIGoROUS): protocol for a multi-center comparative effectiveness trial of in-home gamified constraint-induced movement therapy for rehabilitation of chronic upper extremity hemiparesis," BMC neurology, vol. 17, no. 1, pp. 1-18, 2017.
  10. Zhao, R. Lun, D. D. Espy, and M. A. Reinthal, "Realtime motion assessment for rehabilitation exercises: Integration of kinematic modeling with fuzzy inference," Journal of Artificial Intelligence and Soft Computing Research, vol. 4, no. 4, pp. 267-285, 2014.
  11. K. O'Brien et al., "Activity recognition for persons with stroke using mobile phone technology: toward improved performance in a home setting," Journal of medical Internet research, vol. 19, no. 5, p. e184, 2017.
  12. Omelina, B. Jansen, B. Bonnechère, M. Oravec, P. Jarmila, and S. V. S. Jan, "Interaction detection with depth sensing and body tracking cameras in physical rehabilitation," Methods of information in medicine, vol. 55, no. 01, pp. 70-78, 2016.
  13. Anton, I. Berges, J. Bermúdez, A. Goñi, and A. Illarramendi, "A telerehabilitation system for the selection, evaluation and remote management of therapies," Sensors, vol. 18, no. 5, p. 1459, 2018.
  14. -S. Hosseini, H. Peyrovi, and M. Gohari, "The effect of early passive range of motion exercise on motor function of people with stroke: a randomized controlled trial," Journal of caring sciences, vol. 8, no. 1, p. 39, 2019.
  15. M. Hulteen, N. J. Lander, P. J. Morgan, L. M. Barnett, S. J. Robertson, and D. R. Lubans, "Validity and reliability of field-based measures for assessing movement skill competency in lifelong physical activities: a systematic review," Sports medicine, vol. 45, pp. 1443-1454, 2015.
  16. Kramer, N. Schmidt, R. Memmesheimer, and D. Paulus, "Evaluation of physical therapy through analysis of depth images," in 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2019: IEEE, pp. 1-6.
  17. Spasojević, A. Rodić, and J. Santos-Victor, "Kinect-based approach for upper body movement assessment in stroke," in New Trends in Medical and Service Robotics: Advances in Theory and Practice, 2019: Springer, pp. 153-160.
  18. Capecci et al., "An instrumental approach for monitoring physical exercises in a visual markerless scenario: A proof of concept," Journal of biomechanics, vol. 69, pp. 70-80, 2018.
  19. Yuan et al., "Depth-based 3d hand pose estimation: From current achievements to future goals," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2636-2645.
  20. Fan, "Cerebral Infarction Rehabilitation Evaluation with Posture Analyses," in IOP Conference Series: Materials Science and Engineering, 2019, vol. 612, no. 2: IOP Publishing, p. 022082.
  21. Lu, Z. Deng, J. Luo, W. Chen, S.-K. Yeung, and Y. He, "3D articulated skeleton extraction using a single consumer-grade depth camera," Computer Vision and Image Understanding, vol. 188, p. 102792, 2019.
  22. Sarafianos, B. Boteanu, B. Ionescu, and I. A. Kakadiaris, "3d human pose estimation: A review of the literature and analysis of covariates," Computer Vision and Image Understanding, vol. 152, pp. 1-20, 2016.
  23. Sarsfield et al., "Clinical assessment of depth sensor based pose estimation algorithms for technology supervised rehabilitation applications," International journal of medical informatics, vol. 121, pp. 30-38, 2019.
  24. A. Khedher, D. A. Alkababji, and O. Hadi, "Improving the Reliability of Object Recognition Based On Template Matching," Al-Rafidain Engineering Journal (AREJ), vol. 23, no. 5, pp. 81-88, 2015.
  25. Huan and H. Zhou, "Continuous human pose estimation by machine learning and computer vision," 2022.
  26. Meng et al., "Exploration of human activity recognition using a single sensor for stroke survivors and able-bodied people," Sensors, vol. 21, no. 3, p. 799, 2021.
  27. Tao et al., "A comparative study of pose representation and dynamics modelling for online motion quality assessment," Computer vision and image understanding, vol. 148, pp. 136-152, 2016.
  28. C. Alarcón-Aldana, M. Callejas-Cuervo, and A. P. L. Bo, "Upper limb physical rehabilitation using serious videogames and motion capture systems: A systematic review," Sensors, vol. 20, no. 21, p. 5989, 2020.
  29. A. Sultan and M. Ghanim, "Comprehensive Study and Evaluation of Commonly used Dimensionality Reduction Techniques in Biometrics Field," Al-Rafidain Engineering Journal (AREJ), vol. 25, no. 2, pp. 152-163, 2020.
  30. Ben, P. Adeline, H. Sion, and M. Majid, "Skeleton-Free Body Pose Estimation from Depth Images for Movement Analysis," in Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), Santiago, Chile, 2015, pp. 7-13.
  31. S. Ravali et al., "A systematic review of artificial intelligence for pediatric physiotherapy practice: past, present, and future," Neuroscience Informatics, vol. 2, no. 4, p. 100045, 2022.
  32. Liu, J. Zhu, J. Bu, and C. Chen, "A survey of human pose estimation: the body parts parsing based methods," Journal of Visual Communication and Image Representation, vol. 32, pp. 10-19, 2015.
  33. Hart, H. Smith, and Y. Zhang, "Systematic review of automatic assessment systems for resistance-training movement performance: A data science perspective," Computers in Biology and Medicine, vol. 137, p. 104779, 2021.
  34. Halilaj, A. Rajagopal, M. Fiterau, J. L. Hicks, T. J. Hastie, and S. L. Delp, "Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunities," Journal of biomechanics, vol. 81, pp. 1-11, 2018.
  35. Fabbri et al., "A systematic review of the psychometric properties of the Jebsen–Taylor Hand Function Test (JTHFT)," Hand Surgery and Rehabilitation, vol. 40, no. 5, pp. 560-567, 2021.
  36. C. Quatman-Yates et al., "Physical therapy evaluation and treatment after concussion/mild traumatic brain injury: Clinical practice guidelines linked to the international classification of functioning, disability and health from the academy of orthopaedic physical therapy, American Academy of sports physical therapy, academy of neurologic physical therapy, and academy of pediatric physical therapy of the American Physical therapy association," Journal of Orthopaedic & Sports Physical Therapy, vol. 50, no. 4, pp. CPG1-CPG73, 2020.
  37. Panuccio et al., "Internal consistency and validity of the Italian version of the Jebsen–Taylor hand function test (JTHFT-IT) in people with tetraplegia," Spinal Cord, vol. 59, no. 3, pp. 266-273, 2021.
  38. Vakanski, J. M. Ferguson, and S. Lee, "Metrics for performance evaluation of patient exercises during physical therapy", International journal of physical medicine & rehabilitation, Vol. 5, Issue:3, 2017.
  39. T. Um et al., "Parkinson's disease assessment from a wrist-worn wearable sensor in free-living conditions: Deep ensemble learning and visualization," arXiv preprint arXiv:1808.02870, 2018.
  40. Hachaj and M. R. Ogiela, "Rule-based approach to recognizing human body poses and gestures in real time," Multimedia Systems, vol. 20, pp. 81-99, 2014.
  41. Zhao, R. Lun, D. D. Espy, and M. A. Reinthal, "Rule based realtime motion assessment for rehabilitation exercises," in 2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE), 2014: IEEE, pp. 133-140.
  42. Lee, Y.-S. Lee, and J. Kim, "Automated evaluation of upper-limb motor function impairment using Fugl-Meyer assessment," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 26, no. 1, pp. 125-134, 2017.
  43. Kurillo, J. J. Han, A. Nicorici, and R. Bajcsy, "Tele-MFAsT: Kinect-Based Tele-Medicine Tool for Remote Motion and Function Assessment," in MMVR, 2014, pp. 215-221.
  44. Pogrzeba, T. Neumann, M. Wacker, and B. Jung, "Analysis and quantification of repetitive motion in long-term rehabilitation," IEEE journal of biomedical and health informatics, vol. 23, no. 3, pp. 1075-1085, 2018.
  45. Dehbandi et al., "Using data from the Microsoft Kinect 2 to quantify upper limb behavior: a feasibility study," IEEE journal of biomedical and health informatics, vol. 21, no. 5, pp. 1386-1392, 2016.
  46. H. Osgouei, D. Soulsbv, and F. Bello, "An objective evaluation method for rehabilitation exergames," in 2018 IEEE Games, Entertainment, Media Conference (GEM), 2018: IEEE, pp. 28-34.
  47. Leightley, J. S. McPhee, and M. H. Yap, "Automated analysis and quantification of human mobility using a depth sensor," IEEE journal of biomedical and health informatics, vol. 21, no. 4, pp. 939-948, 2016.
  48. Gu, S. Pandit, E. Saraee, T. Nordahl, T. Ellis, and M. Betke, "Home-based physical therapy with an interactive computer vision system," in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0-0.
  49. Klishkovskaia, A. Aksenov, A. Sinitca, A. Zamansky, O. A. Markelov, and D. Kaplun, "Development of Classification Algorithms for the Detection of Postures Using Non-Marker-Based Motion Capture Systems," Applied Sciences, vol. 10, no. 11, p. 4028, 2020.
  50. Williams, A. Vakanski, S. Lee, and D. Paul, "Assessment of physical rehabilitation movements through dimensionality reduction and statistical modeling," Medical engineering & physics, vol. 74, pp. 13-22, 2019.
  51. Liao, A. Vakanski, M. Xian, D. Paul, and R. Baker, "A review of computational approaches for evaluation of rehabilitation exercises," Computers in biology and medicine, vol. 119, p. 103687, 2020.
  52. Coskun, D. J. Tan, S. Conjeti, N. Navab, and F. Tombari, "Human motion analysis with deep metric learning," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 667-683.
  53. Galna, G. Barry, D. Jackson, D. Mhiripiri, P. Olivier, and L. Rochester, "Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease," Gait & posture, vol. 39, no. 4, pp. 1062-1068, 2014.
  54. D. Correia et al., "Medium-term outcomes of digital versus conventional home-based rehabilitation after total knee arthroplasty: prospective, parallel-group feasibility study," JMIR rehabilitation and assistive technologies, vol. 6, no. 1, p. e13111, 2019.
  55. Houmanfar, M. Karg, and D. Kulić, "Movement analysis of rehabilitation exercises: Distance metrics for measuring patient progress," IEEE Systems Journal, vol. 10, no. 3, pp. 1014-1025, 2014.
  56. Saraee et al., "Exercisecheck: remote monitoring and evaluation platform for home based physical therapy," in Proceedings of the 10th international conference on PErvasive technologies related to assistive environments, 2017, pp. 87-90.
  57. Capecci et al., "A Hidden Semi-Markov Model based approach for rehabilitation exercise assessment," Journal of biomedical informatics, vol. 78, pp. 1-11, 2018.
  58. -J. Su, C.-Y. Chiang, and J.-Y. Huang, "Kinect-enabled home-based rehabilitation system using Dynamic Time Warping and fuzzy logic," Applied Soft Computing, vol. 22, pp. 652-666, 2014.
  59. Zhang, Z. C. Lipton, M. Li, and A. J. Smola, "Dive into deep learning," arXiv preprint arXiv:2106.11342, 2021.
  60. Tang, "Hybridized hierarchical deep convolutional neural network for sports rehabilitation exercises," IEEE Access, vol. 8, pp. 118969-118977, 2020.
  61. Alzubaidi et al., "Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions," Journal of big Data, vol. 8, pp. 1-74, 2021.
  62. Liao, A. Vakanski, and M. Xian, "A deep learning framework for assessing physical rehabilitation exercises," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 2, pp. 468-477, 2020.
  63. Anaz, M. Skubic, J. Bridgeman, and D. M. Brogan, "Classification of therapeutic hand poses using convolutional neural networks," in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018: IEEE, pp. 3874-3877.
  64. Tschuggnall, V. Grote, M. Pirchl, B. Holzner, G. Rumpold, and M. J. Fischer, "Machine learning approaches to predict rehabilitation success based on clinical and patient-reported outcome measures," Informatics in Medicine Unlocked, vol. 24, p. 100598, 2021.
  65. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, "Realtime multi-person 2d pose estimation using part affinity fields," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7291-7299.
  66. Vasileiadis, C.-S. Bouganis, and D. Tzovaras, "Multi-person 3D pose estimation from 3D cloud data using 3D convolutional neural networks," Computer Vision and Image Understanding, vol. 185, pp. 12-23, 2019.
  67. Liu et al., "Feature boosting network for 3D pose estimation," IEEE transactions on pattern analysis and machine intelligence, vol. 42, no. 2, pp. 494-501, 2019.
  68. Vakanski, J. Ferguson, and S. Lee, "Mathematical modeling and evaluation of human motions in physical therapy using mixture density neural networks," Journal of physiotherapy & physical rehabilitation, vol. 1, no. 4, 2016.
  69. R. Shareef and Y. F. M. Al-Irhayim, "Comparison Between Features Extraction Techniques for Impairments Arabic Speech," Al-Rafidain Engineering Journal (AREJ), vol. 27, no. 2, pp. 190-197, 2022.
  70. Sardari, A. Paiement, S. Hannuna, and M. Mirmehdi, "Vi-net—view-invariant quality of human movement assessment," Sensors, vol. 20, no. 18, p. 5258, 2020.
  71. Elkholy, M. E. Hussein, W. Gomaa, D. Damen, and E. Saba, "Efficient and robust skeleton-based quality assessment and abnormality detection in human action performance," IEEE journal of biomedical and health informatics, vol. 24, no. 1, pp. 280-291, 2019.
  72. K. Deters and Y. Rybarczyk, "Hidden Markov Model approach for the assessment of tele-rehabilitation exercises," International Journal of Artificial Intelligence, vol. 16, no. 1, pp. 1-19, 2018.
  73. Capecci et al., "Physical rehabilitation exercises assessment based on hidden semi-markov model by kinect v2," in 2016 IEEE-EMBS international conference on biomedical and health informatics (BHI), 2016: IEEE, pp. 256-259.
  74. Ferraris et al., "A self-managed system for automated assessment of UPDRS upper limb tasks in Parkinson’s disease," Sensors, vol. 18, no. 10, p. 3523, 2018.
  75. Panwar et al., "Rehab-net: Deep learning framework for arm movement classification using wearable sensors for stroke rehabilitation," IEEE Transactions on Biomedical Engineering, vol. 66, no. 11, pp. 3026-3037, 2019.
  76. Boukhennoufa, X. Zhai, K. D. McDonald-Maier, V. Utti, and J. Jackson, "Improving the activity recognition using GMAF and transfer learning in post-stroke rehabilitation assessment," in 2021 ieee 19th world symposium on applied machine intelligence and informatics (SAMI), 2021: IEEE, pp. 000391-000398.
  77. Wei, C. Mcelroy, and S. Dey, "Using sensors and deep learning to enable on-demand balance evaluation for effective physical therapy," IEEE Access, vol. 8, pp. 99889-99899, 2020.
  78. Wei, C. McElroy, and S. Dey, "Towards on-demand virtual physical therapist: Machine learning-based patient action understanding, assessment and task recommendation," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 9, pp. 1824-1835, 2019.
  79. D. Hssayeni, J. L. Adams, and B. Ghoraani, "Deep learning for medication assessment of individuals with Parkinson’s disease using wearable sensors," in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018: IEEE, pp. 1-4.
  80. Bejaković, "Eurofound: Living, Working and Covid-19, Covid-19 Series," Revija za socijalnu politiku, vol. 28, no. 1, pp. 115-117, 2021.
  81. M. Burns, N. Leung, M. Hardisty, C. M. Whyne, P. Henry, and S. McLachlin, "Shoulder physiotherapy exercise recognition: machine learning the inertial signals from a smartwatch," Physiological measurement, vol. 39, no. 7, p. 075007, 2018.
  82. Piñero-Fuentes, S. Canas-Moreno, A. Rios-Navarro, M. Domínguez-Morales, J. L. Sevillano, and A. Linares-Barranco, "A deep-learning based posture detection system for preventing telework-related musculoskeletal disorders," Sensors, vol. 21, no. 15, p. 5236, 2021.
  83. Suzuki, Y. Amemiya, and M. Sato, "Deep learning assessment of child gross-motor," in 2020 13th International Conference on Human System Interaction (HSI), 2020: IEEE, pp. 189-194.
  84. Ge, H. Liang, J. Yuan, and D. Thalmann, "3d convolutional neural networks for efficient and robust hand pose estimation from single depth images," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1991-2000.
  85. Butepage, M. J. Black, D. Kragic, and H. Kjellstrom, "Deep representation learning for human motion prediction and classification," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6158-6166.
  86. D. Tsakanikas et al., "Evaluating the performance of balance physiotherapy exercises using a sensory platform: The basis for a persuasive balance rehabilitation virtual coaching system," Frontiers in digital health, vol. 2, p. 545885, 2020.
  87. Zhang, C. Su, and C. He, "Rehabilitation exercise recognition and evaluation based on smart sensors with deep learning framework," IEEE Access, vol. 8, pp. 77561-77571, 2020.
  88. W. X. Cejnog, T. de Campos, V. M. C. Elui, and R. M. Cesar Jr, "A framework for automatic hand range of motion evaluation of rheumatoid arthritis patients," Informatics in Medicine Unlocked, vol. 23, p. 100544, 2021.
  89. Izadmehr, H. F. Satizábal, K. Aminian, and A. Perez-Uribe, "Depth Estimation for Egocentric Rehabilitation Monitoring Using Deep Learning Algorithms," Applied Sciences, vol. 12, no. 13, p. 6578, 2022.
Al-Rafidain Engineering Journal (AREJ)
Volume 28, Issue 2
September 2023
Page 237-251
Files
Share
Export Citation
Statistics