Integration of wearable and ambient sensors towards characterization of physical effort

Integration of wearable and ambient sensors towards characterization of physical effort

Aaron Appelle, Liming Salvino, Yun-An Lin, Taylor Pierce, Emerson Noble, Gabriel Draughon, Kenneth J. Loh, Jerome P. Lynch

Abstract. Human performance monitoring in complex operational environments calls for sensing solutions that measure human physiology as well as human interactions with their surroundings. Recent advances in multimodal sensing have led to the development of intelligent environments that analyze human activities with high granularity. One of the greatest challenges is to unify multiple discrete sensing systems through synchronization and integration of multimodal data streams. This paper describes an intelligent environment that consolidates wearable skin-strain sensors for physiological monitoring; geophones and microphones to record ambient vibrations and sounds; and video cameras to visually observe human activities. We show proof-of-concept functionality by using the system to differentiate walking effort in human subjects. First, the work shows the alignment of wearable and ambient sensor time-history records. Then, data features are extracted and correlated to walking speed using three sensor modalities. Finally, feature-level analysis is done to associate the data features with the perceived walking exertion for each subject.

Keywords
Physiological Monitoring, Multimodal, Synchronization, Thin Film, Kinesiology Tape, Geophone, Microphone, Computer Vision, Signal Processing

Published online 3/30/2023, 8 pages
Copyright © 2023 by the author(s)
Published under license by Materials Research Forum LLC., Millersville PA, USA

Citation: Aaron Appelle, Liming Salvino, Yun-An Lin, Taylor Pierce, Emerson Noble, Gabriel Draughon, Kenneth J. Loh, Jerome P. Lynch, Integration of wearable and ambient sensors towards characterization of physical effort, Materials Research Proceedings, Vol. 27, pp 300-307, 2023

DOI: https://doi.org/10.21741/9781644902455-39

The article was published as article 39 of the book Structural Health Monitoring

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

References
[1] G. Aroganam, N. Manivannan, and D. Harrison, “Review on Wearable Technology Sensors Used in Consumer Sport Applications,” Sensors, vol. 19, no. 9, p. 1983, Jan. 2019. https://doi.org/10.3390/s19091983
[2] A. Aryal, A. Ghahramani, and B. Becerik-Gerber, “Monitoring fatigue in construction workers using physiological measurements,” Autom. Constr., vol. 82, pp. 154–165, Oct. 2017. https://doi.org/10.1016/j.autcon.2017.03.003
[3] K. Hinde, G. White, and N. Armstrong, “Wearable Devices Suitable for Monitoring Twenty Four Hour Heart Rate Variability in Military Populations,” Sensors, vol. 21, no. 4, Art. no. 4, Jan. 2021. https://doi.org/10.3390/s21041061
[4] J.-H. Lee and H. Hashimoto, “Intelligent Space — concept and contents,” Adv. Robot., vol. 16, no. 3, pp. 265–280, Jan. 2002. https://doi.org/10.1163/156855302760121936
[5] G. Draughon, J. Lynch, and L. Salvino, “Integrated Vision-Body Sensing System for Tracking People in Intelligent Environments,” in European Workshop on Structural Health Monitoring, Cham, 2023, pp. 885–893. https://doi.org/10.1007/978-3-031-07254-3_89
[6] D. L. Hall and J. Llinas, “An introduction to multisensor data fusion,” Proc. IEEE, vol. 85, no. 1, pp. 6–23, Jan. 1997. https://doi.org/10.1109/5.554205
[7] Y.-A. Lin et al., “Graphene K-Tape Meshes for Densely Distributed Human Motion Monitoring,” Adv. Mater. Technol., vol. 6, no. 1, p. 2000861, 2021. https://doi.org/10.1002/admt.202000861
[8] Y.-A. Lin, E. Noble, C.-H. Loh, and K. J. Loh, “Respiration Monitoring using a Motion Tape Chest Band and Portable Wireless Sensing Node,” J. Commer. Biotechnol., vol. 27, no. 1, p. 32, Jan. 2022. https://doi.org/10.5912/jcb1026
[9] “Mcity,” Mcity. https://mcity.umich.edu/ (accessed Nov. 24, 2022).
[10] G. A. V. Borg, “Psychophysical bases of perceived exertion,” Med. Sci. Sports Exerc., vol. 14, no. 5, pp. 377–381, 1982. https://doi.org/10.1249/00005768-198205000-00012
[11] J. W. Cooley and J. W. Tukey, “An Algorithm for the Machine Calculation of Complex Fourier Series,” Math. Comput., vol. 19, no. 90, pp. 297–301, 1965. https://doi.org/10.1090/S0025-5718-1965-0178586-1
[12] R. R. Bechbache and J. Duffin, “The entrainment of breathing frequency by exercise rhythm.,” J. Physiol., vol. 272, no. 3, pp. 553–561, Nov. 1977. https://doi.org/10.1113/jphysiol.1977.sp012059
[13] S. Pan, A. Bonde, J. Jing, L. Zhang, P. Zhang, and H. Y. Noh, “BOES: Building Occupancy Estimation System using sparse ambient vibration monitoring,” in Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2014, Apr. 2014, vol. 9061, pp. 406–421. https://doi.org/10.1117/12.2046510
[14] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 779–788. https://doi.org/10.1109/CVPR.2016.91
[15] P. Sun, G. Draughon, R. Hou, and J. P. Lynch, “Automated Human Use Mapping of Social Infrastructure by Deep Learning Methods Applied to Smart City Camera Systems,” J. Comput. Civ. Eng., vol. 36, no. 4, p. 04022011, Jul. 2022. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000998