Alan Tay wants to clone your best flight instructor and turn him/her into an AI Avatar

“We are incorporating AI as machine learning to replicate the instructors on their motions and behavior. Cloning basically,” Tay said.

Calgary, Canada-based Illumia – which originated in 2020 as Delphi Technology – has been developing immersive virtual- and mixed-reality training platforms for commercial aviation and defense customers, as well as other businesses. The newest platform features adaptive learning with AI enhancement.

The initiative began with a research project assisted by professors and graduate students at the Intelligent Robot Learning (IRL) and EdTeKLA Lab at the University of Alberta which sought to test the potential for an AI agent to teach ab initio flight students basic flying manuevers.

“Alan was looking at how we can help them in real time,” explained Matthew Taylor, Director of the IRL lab and a Fellow of the Alberta Machine Intelligence Institute. “A student’s doing an exercise and the [agent] might say, ‘You’re going too far to the left. Go to the right.’” One challenge is “figuring out when to make that intervention. How much rope do you want to give the person? How much do you want to let them play out and see if they recognize their own mistake? Because you don’t want to nanny them,” Taylor added.

MIMICKING EXPERTS

The “Augmenting Flight Training with AI” experiments, described in a paper at the 2023 AAAI Conference on Artificial Intelligence, used an X-Plane flight simulator to evaluate straight-and-level flight by novice students. First, expert pilots demonstrated the S&L task for 12.5 minutes, operating under visual flight rules. The final dataset consisted of 25 trials.

From collected data, they trained a ‘behavior cloning’ agent to mimic the expert pilots’ policy, i.e. parameters such as pitch and roll.

“Part of it is the dual challenge of getting the instructor agent up to speed and then testing that agent with the human students,” Taylor noted. “You want this agent to be able to learn new tasks quickly or adapt to new tasks. And that’s part of the framework we’re building. If you have to spend the same amount of time getting this agent up to speed on every task for every type of airplane, it just wouldn’t be feasible.”

“A teacher should be able to detect students’ mistakes and provide feedback to correct them,” stated the AAAI paper. “Therefore, we:

Recorded students’ poorly performed flights

Asked the pilots to prepare annotated critiques on errors made

Identified two main types of errors

Used simple distance metrics to decide whether the agent agreed or not with the student’s decisions.”

The researchers referred to this as ‘informative tutoring’ – “formative feedback that presents verification feedback, error flagging, and strategic hints how to proceed.”

“Right now it’s just an AI agent,” acknowledged Taylor. “But ultimately, in addition to the AI sitting over your shoulder and saying, ‘Do you really think you know what you’re doing?’, that AI agent could raise a flag and say, ‘Hey, human instructor, this person seems pretty confused. You should come over here and talk to them when you get a chance.’ That could be in person in the classroom or having them pop up in a virtual video window.”

Illumia’s Tay said they tested about 100 post-secondary Gen Z and Millenial students on the system with “a 76% success rate in saying those are really good and intuitive and helping us to learn. It helps us to gain – within that one week – the 90% confidence level to achieve a particular task, rather than watching video and giving us only like 50%.”

They also worked with the Air Cadet League of Canada, about 150 16- to 18-year-olds “to see how that helped them to get ready for their ‘power scholarship selection’.” About 70% of them said the AI is much better on the theory of flight and navigation side, providing them ‘real-life’ envisioning.

Taylor: “It seems like it works. A lot of academics, we’re really looking at dotting our i’s and crossing our t’s. Whereas a company is much more interested in is this is going to be useful? You don’t care about statistical significance. You care about actual significance, right? We’re convinced it’s going to work. Both from an academic standpoint – statistically significantly improved learning outcomes – but also going from a commercial perspective. Faster learning with a reduced number of human teachers.

LLM as ATC

Tay said the company is advancing into crew resource management (CRM) training. “We are able to have multiple agents within that scenario to role play with you. You do not need a co-pilot. You do not need a pilot to be there to role-play with you. You only train with yourself because all the various avatars are there. The avatars are there to role-play with you, and if you do something wrong, they will tell you, ‘You missed out this call. You didn’t hit the button. You didn’t hold the switch long enough.’ They will catch all that.”

Illumia Labs has an agreement with Flair Airlines for onboarding training. “We are replacing the classroom time. Instead of ground training time, sitting there watching videos or PowerPoints, now you become interactive instead. We are replacing the poster training.” said Tay.

“We are able to train the AI on a specific company, including SOPs. We are providing AI actors or AI avatars replicating the human experience or interaction. That’s our value proposition. We do not require you to buy our whole sim; you can just have our actor, our AI, an upgrade in the simulator.”

Taylor said they’re also looking at large language models to come up with scenarios, “having the pilot interact both with the simulator but also the large language model. This is particularly useful about the pilot’s communication. If something is happening and the pilot needs to communicate that to someone, they could interact with the LLM. And that’s kind of like the tower, the controller, or whomever they’re supposed to be communicating with.”

“This generation of AI with the large language models has a better chance of being resilient and handling that unexpected behavior,” explained Taylor. “With the past AI systems, one of the criticisms is they can be very brittle. As soon as the human does something slightly unexpected, the system just breaks. Hopefully with the LLMs when the human does something weird the system can handle that and push them back towards the path we were hoping they would follow.”

FINDING AI DEVELOPERS

Taylor said finding talent for AI projects is a challenge. “Right now we’re in a bit of an AI gold rush. There are not enough people trained to use these cutting-edge AI techniques as there is demand in the marketplace. That’s one of the motivations for companies working with academics, because they really can’t find the right people to hire.”

“You probably need some kind of technical undergrad, usually computer science, some kind of engineering, and then at least two years to get up to speed. More like five years if you’re doing research. So it’s pretty demanding.”

“We’re looking for students that have that basic technological background, coding and understanding computers. With the desire and interest in learning.”

“The people who are graduating right now with exactly the skill set get gobbled up by the FAANG companies” – Facebook (Meta), Amazon, Apple, Netflix, and Google (Alphabet). [Maybe MAANA?] “Everyone else either has to try to train up their own employees internally, which is slow, or try to compete with Google to hire these people, which is really difficult.”

Excerpted from The Robot in the Simulator: Artificial Intelligence in Aviation Training by Rick Adams, FRAeS – https://aviationvoices.com/shop/