The International Arab Journal of Information Technology (IAJIT)

..............................
..............................
..............................


Gesture Recognition Technology of VR Piano Playing Teaching Game based on Hidden Markov Model

Min Zeng,

Virtual Reality (VR) is a kind of simulation environment generated by computer simulation. It is a kind of isolation of users from the physical environment, and users are fully engaged in the simulation environment. In music teaching, especially in the learning of piano performance, VR technology can simulate the real playing experience without physical piano, so that learners can learn and practice piano anywhere. At present, the traditional way of learning piano often needs the guidance of professional piano teachers, and learners must study and practice beside the piano, which undoubtedly brings limitations to the time and space of learners. VR technology provides a more convenient way for learners to learn and practice without a physical piano, making learning more flexible and free. Firstly, a piano playing teaching game platform based on VR technology is constructed to provide a virtual piano playing scene. Then, by collecting a large number of piano players' gesture data, they are accurately marked. Finally, the Hidden Markov Model (HMM) is used to train and model the gesture data, and the key features of the gesture sequence are extracted to realize the recognition and real-time feedback of the piano player's gesture. According to the findings, the recognition accuracy of the Principal Component Analysis-HMM (PCA-HMM) without playing correction is very low after playing correction. Compared to the logarithmic likelihood ratio test, the recognition rate of the quadratic statistical model is only 15%. The results show that this method has good iterative convergence and convergence performance, which can effectively avoid falling into local optima. Compared with the traditional hidden Markov algorithm, it reduces 96.05% and 89.48% respectively. Compared with the fuzzy hidden Markov algorithm, it reduces 90.20% and 73.35% respectively. The average time and standard deviation for real-time positioning and map construction are 0.251s and 0.152s, respectively. The fitness of HMM based on fuzzy Particle Swarm Optimization (PSO) is 6.8% and 2.6% higher than that of the hidden Markov algorithm and the fuzzy hidden Markov algorithm, respectively. At the same time, in the comparison between the research model and other algorithm models, the accuracy and other parameter performance of the research model are better, with the highest error value of only 3.5, which is 5.9 lower than the lowest Convolutional Neural Network (CNN) algorithm model. It can be seen that the error value of the model used in the study is lower and the performance is better. Learners can receive real-time guidance and a more immersive playing experience through this technology. This provides useful reference for further exploring the combination of music teaching and VR technology.

[1] Albahri O. and AlAmoodi A., “Navigating the Metaverse of Big Data: A Bibliometric Journey,” Mesopotamian Journal of Big Data, vol. 2023, pp. 92-106, 2023. https://doi.org/10.58496/MJBD/2023/013

[2] Amaya M., Linde N., and Laloy E., “Adaptive Sequential Monte Carlo for Posterior Inference and Model Selection among Complex Geological Priors,” Geophysical Journal International, vol. 226, no. 2, pp. 1220-1238, 2021. https://doi.org/10.1093/gji/ggab170

[3] Briggs A., Robstad C., and Rosell F., “Using Radio‐Frequency Identification Technology to Monitor Eurasian Beavers,” Wildlife Society Bulletin, vol. 45, no. 1, pp. 154-161, 2021. https://doi.org/10.1002/wsb.1147

[4] Cui K., “Artificial Intelligence and Creativity: Piano Teaching with Augmented Reality Applications,” Interactive Learning Environments, vol. 31, no. 10, pp. 7017-7028, 2023. https://doi.org/10.1080/10494820.2022.2059520

[5] De Wael A., De Backer A., and Van Aert S., “Hidden Markov Model for Atom-Counting from Sequential ADF STEM Images: Methodology, Possibilities and Limitations,” Ultramicroscopy, vol. 219, no. 1, pp. 1-14, 2020. https://doi.org/10.1016/j.ultramic.2020.113131

[6] Deja J., Mayer S., Pucihar K., and Kljun M., “A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences,” Proceedings of the ACM on Human-Computer Interaction, vol. 6, no. ISS, pp. 226-253, 2022. https://doi.org/10.1145/3567719

[7] Deng H., Shang W., Lu G., Guo P., Ai T., Fang C., and Tian J., “Targeted and Multifunctional Technology for Identification between Hepatocellular Carcinoma and Liver Cirrhosis,” ACS Applied Materials and Interfaces, vol. 11, no. 16, pp. 144526-14537, 2019. DOI:10.1021/acsami.8b20600

[8] Don M. and Khan F., “Process Fault Prognosis Using Hidden Markov Model-Bayesian Networks (HMM-BN) Hybrid Model,” Industrial and Engineering Chemistry Research, vol. 58, no. 27, pp. 12041-12053, 2019. DOI:10.1021/acs.iecr.9b00524

[9] Frimane A., Munkhammar J., and Van der Meer D., “Infinite Hidden Markov Model for Short- Term Solar Irradiance Forecasting,” Solar Energy, vol. 244, no. 9, pp. 331-342, 2022. DOI:10.1016/j.solener.2022.08.041

[10] Goodall V., Ferreira S., Funston P., and Maruping-Mzileni N., “Uncovering Hidden States in African Lion Movement Data Using Hidden Markov Models,” Wildlife Research, vol. 46, no. 4, pp. 296-303, 2019. DOI:10.1071/WR18004

[11] Jiang J., Chen R., Chen M., Wang W., and Zhang C, “Dynamic Fault Prediction of Power Transformers Based on Hidden Markov Model of Dissolved Gases Analysis,” IEEE Transactions on Power Delivery, vol. 34, no. 4, pp. 1393-1400, 2019. DOI:10.1109/TPWRD.2019.2900543 772 The International Arab Journal of Information Technology, Vol. 21, No. 4, July 2024

[12] Li Y., Li X., Chen Z., Ma Y., Yang G., Zhang Y., Sun M., Zhang L., Chu J., Lu J., Zhu J., and You J., “Application of Plant Print Identification Technology in Salt-Resistant Soybean Breeding,” Journal of Plant Growth Regulation, vol. 40, no. 4, pp. 1557-1565, 2021. DOI:10.1007/s00344- 020-10213-1

[13] Li Y., Wu M., Zhang Y., Xu L., and Yu L., “Piano: A Parametric Hand Bone Model from Magnetic Resonance Imaging,” in Proceedings of the 30th International Joint Conference on Artificial Intelligence, Montreal, pp. 816-822, 2021. https://doi.org/10.24963/ijcai.2021/113

[14] Patel S. and Makwana R., “Connectionist Temporal Classification Model for Dynamic Hand Gesture Recognition Using RGB and Optical flow Data,” The International Arab Journal of Information Technology, vol. 17, no. 4, pp. 497- 506, 2020. https://doi.org/10.34028/iajit/17/4/8

[15] Raskar C. and Nema S., “Metaheuristic Enabled Modified Hidden Markov Model for Traffic Flow Prediction,” Computer Networks, vol. 206, no. 7, pp. 1-11, 2022. DOI:10.1016/j.comnet.2022.108780

[16] Rosenblatt J., Ritov Y., and Goeman J., “Discussion of ‘Gene Hunting with Hidden Markov Model Knockoffs’,” Biometrika, vol. 106, no. 1, pp. 29-33, 2019. DOI:10.1093/biomet/asy062

[17] Shin Y., Liu D., Sang H., Ferguson T., and Song P., “A Binary Hidden Markov Model on Spatial Network for Amyotrophic Lateral Sclerosis Disease Spreading Pattern Analysis,” Statistics in Medicine, vol. 40, no. 13, pp. 3035-3052, 2021. DOI:10.1002/sim.8956

[18] Soper B., Nygrd M., Abdulla G., Meng R., and Nygard J., “A Hidden Markov Model for population Level Cervical Cancer Screening Data,” Statistics in Medicine, vol. 39, no. 25, pp. 3569-3590, 2020. DOI:10.1002/sim.8681

[19] Stolz B., “The Author and the Piano Student: The Transferability of the Creative Process in Practice,” International Journal of Music Education, vol. 40, no. 10, pp. 53-65, 2022. DOI:10.1177/02557614211019144

[20] Tallon M., Greenlee M., Wagner E., Rakoczy K., and Frick U., “How do Art Skills Influence Visual Search?-Eye Movements Analyzed with Hidden Markov Models,” Frontiers in Psychology, vol. 12, pp. 1-17, 2021. https://doi.org/10.3389/fpsyg.2021.594248

[21] Xia T. and Chen X., “A Weighted Feature Enhanced Hidden Markov Model for Spam SMS Filtering,” Neurocomputing, vol. 444, no. 15, pp. 48-52, 2021. DOI:10.1016/j.neucom.2021.02.075

[22] Yildiz A., Adar N., and Mert A., “Convolutional Neural Network Based Hand Gesture Recognition in Sophisticated Background for Humanoid Robot Control,” The International Arab Journal of Information Technology, vol. 20, no. 3, pp. 368- 375, 2023. https://doi.org/10.34028/iajit/20/3/9

[23] Zhiyong G., Jiwu L., and Rongxi W., “Prognostics Uncertainty Reduction by Right-Time Prediction of Remaining Useful Life Based on Hidden Markov Model and Proportional Hazard Model,” Eksploatacja i Niezawodność-Maintenance and Reliability, vol. 23, no. 1, pp. 154-164, 2020. DOI:10.17531/ein.2021.1.16