Humans and Machines Work in Sync in a Cockpit Assistance System

Within the framework of the European Peggasus project, CSEM and its partners have developed a vision system that allows pilots to focus on important tasks in the cockpit, improving crew efficiency and maximizing flight safety. This new human-machine interface enables remote eye tracking and hand gesture recognition and draws on the latest developments in artificial intelligence and computer vision. The system has been tested successfully by ten professional pilots in a flight simulator.

Image Credit: CSEM

Most commercial aircraft built since 1914 are equipped with an autopilot system. Despite the clear benefits these systems provide (such as enhanced flight safety and operations), new challenges have emerged owing to the increased complexity of modern cockpits. Today these cabins contain an array of large computer screens, many secondary dials, and hundreds of switches. That means pilots have to process vast amounts of information in high-stress, high-workload situations.

CSEM's tech : Peggasus, a pilot's eye gaze tracking technology

Video Credit: CSEM

A New Human-machine Interface

This is the problem that PEGGASUS – short for Pilot Eye Gaze and Gesture tracking for Avionics Systems using Unobtrusive Solutions – has set out to tackle. The research was coordinated by CSEM and financed by the EU, and the other consortium members include ETH Zurich, SERMA Ingénierie and Swiss International Air Lines. CSEM worked with these project partners to develop a novel human-machine interface in the form of a dashboard-mounted, non-intrusive camera system. The system can track a pilot’s gaze and recognize their hand gestures in real time, inside a cockpit. It gives pilots seamless and accurate feedback, thereby lightening their workload and improving their situational awareness when executing a multitude of complicated tasks.

The PEGGASUS system addresses the unique challenges encountered in aeronautical context situations, such as extreme variations in light intensity,” says Andrea Dunbar, the consortium coordinator at CSEM. “It improves pilots’ situational awareness, and consequently decision-making, and makes it easier for the flight crew to work with aircraft controls.

Eye Gaze Tracking and Hand Gesture Recognition

PEGGASUS employs state-of-the-art computer-vision and machine-learning algorithms. Its algorithmic pipeline for eye tracking combines analytical and data-driven approaches, operating in real time at 60 frames per second with minimum latency while achieving eye-tracking accuracy of better than 1°.

To enable hand-gesture recognition, a deep neural network model was developed and incorporated into the final PEGGASUS system, providing a new form of interaction directly in the cockpit.

This one-of-a-kind system is the result of a collaborative effort among all consortium members, with each one drawing on its own expertise. CSEM developed the vision system and machine-learning algorithms for eye tracking and hand gesture recognition. SERMA Ingénierie integrated the PEGGASUS hardware system into a cockpit demonstrator for testing. ETH Zurich, with its know-how in human-computer interaction and aviation safety, designed and carried out the testing in a simulator study. Swiss International Air Lines supported the consortium by providing insight on use cases and ideas for future training applications.

Tested and Approved by Professionals

The PEGGASUS system was tested by ten professional pilots in a cockpit simulator provided by Lufthansa Aviation Training, Switzerland. They gave very positive feedback and rated the PEGGASUS vision system better than head-mounted eye-tracking systems in terms of comfort and low distraction.

Based on many incident and accident analyses, the highest degree of flight safety can be obtained from an optimized human-machine interface,” says Gilad Scherpf, the head of Aviation Training Development at Lufthansa Group. “Any steps towards improving such an interface will be of great value to Europe’s avionics industry. What’s more, these interfaces can improve both flight operations and pilot training. Computer-assisted technology like the one used in PEGGASUS can be very effective in modern airliners, especially since air-safety levels will need to be maintained or increased as the degree of automation increases.”



Please use one of the following formats to cite this article in your essay, paper or report:

  • APA

    CSEM. (2023, April 11). Humans and Machines Work in Sync in a Cockpit Assistance System. AZoM. Retrieved on July 25, 2024 from

  • MLA

    CSEM. "Humans and Machines Work in Sync in a Cockpit Assistance System". AZoM. 25 July 2024. <>.

  • Chicago

    CSEM. "Humans and Machines Work in Sync in a Cockpit Assistance System". AZoM. (accessed July 25, 2024).

  • Harvard

    CSEM. 2023. Humans and Machines Work in Sync in a Cockpit Assistance System. AZoM, viewed 25 July 2024,

Tell Us What You Think

Do you have a review, update or anything you would like to add to this news story?

Leave your feedback
Your comment type

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions.