Smart Cameras Monitoring Human Performed Tasks

Reference project

Client: undisclosed energy and automation technology corporation
Services obtained:
  • Sprint to concept
  • Sprint to MVP
The client's challenge

Human performed tasks in production are prone to costly mistakes

Manual assembly procedures, labor intensive tasks in production are particularly subject to human errors. These can lead to costly damages to equipment, undesired rework, quality issues at the product or even injury of the operators. Checklists and definition of standardized work procedures can partially mitigate the problem; however, these do not completely eliminate occurrence of undesired events. Hence, companies often need downstream quality control gates to address the problem, but these increase the complexity of the process chain, and also the costs.

The EMBRIO approach

Thanks to a combination of machine learning and computer vision, smart cameras offer a practical solution to monitor human performed activities in production

Smart cameras offer the unprecedented possibility of sensing the physical world and automatically process information to generate meaningful insights in production. Thanks to state-of-the-art object recognition algorithms based on convolutional neural networks; smart cameras can be trained to detect the occurrence of particular events in production. The tightening of a screw, the correct application of labels or a specific sequence in assembly operations, can be automatically monitored. On the occurrence of undesired events, the operator can be instantly notified, and reminded to fix the issue right away.

The results delivered

EMBRIO developed a smart camera application to detect potentially harmful errors in cluttered production environments

In a practical implementation study, EMBRIO developed a smart camera system to monitor the manual set-up procedure of heavy machining equipment. For this purpose, the system detects setup events and evaluates whether they occurred in the correct order. The events are detected based on the presence, position, and orientation of the relevant objects in consecutive images. The relevant objects are

  1. the part (namely a shaft whose size and shape varies with the production order),
  2. the tool used for tightening the workpiece.

The smart camera is able to automatically monitor the set-up procedure and verify whether the workpiece has been clamped correctly on the lathe, and whether it is safe or not to start production safely. Failure in clamping the part correctly can result in major damage to the equipment. The detection system comprise two individual smart cameras, for redundancy in case of an impaired view.

The smart camera algorithm has been trained with transfer learning using 650 distinct images from both cameras showing the tool and the part.

Tools and technologies applied:
  • Tiny YOLO
  • OpenCV
  • MQTT
  • Embedded Linux

Do you like this project? Let's keep in touch!

Every now and then we will keep you updated about similar projects, new articles or general updates from EMBRIO with our newsletter.

You can unsubscribe in one click, and we will never share your email address.