Monitoring of Human-centric Assembly Operations in Manufacturing by Modeling Assemblies as Spatiotemporal Graphs

Abstract

Process improvement in the manufacturing industry is an iterative exercise. Opti- mization of assembly operations plays a major role in this regard. Robust and continuous monitoring of assembly operations can improve the quantity and quality of the finished product, in addition to assisting in identifying manufacturing defects and irregularities. However, assembly workers have basic anthropometric differences, such as varying levels of expertise, personal working style, physical ailments, etc., and the assembly steps vary in scale and duration. Therefore, developing a robust assembly monitoring system that caters to individual workers can be challenging. Wireless sensors worn by workers have been previously used to monitor productivity, but they pose a safety hazard and may cause discomfort and intrude on privacy, thus affecting their efficiency. Here, computer vision and deep learning can help overcome these hurdles by developing reliable assembly monitoring systems and exploiting existing infrastructures, e.g., real-time video capture. Although Human Activity Recognition and Localization in videos have been studied to a large extent in the domain of deep learning, the key to effectively monitoring assembly line operations lies in capturing the temporal and spatial data of the process physics. This study focused on developing a novel approach where assembly operations are modeled as spatial and temporal graphs called Spatial-Temporal Graph-based Assembly Monitor- ing (STGAM). The underlying mechanism behind the assembly operations was better captured through graphs by effectively modeling long-term spatiotemporal relationships between the objects and persons associated with an assembly (tools and operators). The approach was evaluated against two datasets, one from an assembly workstation in in- dustry and the other from a laboratory. The model outputs were compared with those obtained from the State Machine Integrated Recognition and Localization (SMIRL) model. The results show that STGAM showed comparable performance with an F1 score of 86 %, which can be further improved to 92 % based on resource availabilities. This study ultimately aims to contribute towards using artificial intelligence for real-time monitoring of human-centric assembly line operations in smart manufacturing.