From textbook to measurable learning experience

As a leading provider of online courses and certifications for continuing medical education, our client had a lot of experience and domain knowledge in dealing with doctors and other medical staff.

Both residents who were completing their mandatory time in the hospital and seasoned professionals who needed to recertify in order to continue practicing had the same problem: not a lack of information, but a lack of time to consume it and a lack of certainty that all the key points had been understood and learned.

In their stressful everyday working lives, hardly anyone found time to read textbooks or run through sample exams. Between two surgeries, there was just enough time for a coffee and maybe a short technical article on the cell phone.

In 2020, we at NEOMO had started researching how Large Language Models could improve the way we consume information and acquire knowledge.

Together with our client, we were able to put these technologies to the test in practice.

Objective

Continuing education in the medical field must entail consuming teaching content in 5-minute chunks and at the same time always keeping an eye on how well you have mastered the material.

The logical conclusion for our client was to make the extraordinarily extensive subject knowledge of the various specializations accessible in the form of "test-enhanced learning", i.e. to create individual multiple-choice questions from long texts including images and diagrams, which at the same time served to test knowledge but also - through detailed explanations of the correct answer with references to the original text and further articles - to impart knowledge.

Intelligent algorithms had to help with both the creation and the adaptation of these questionnaires to the respective learner and thus break down the learning experience into small, self- contained units.

Automatic selection of the "learning path" and flexible measurement of progress

From its previous work, our client had already created a repertoire of 3000 questions with answers and explanations that completely covered a specific medical specialty. This meant that anyone who could answer 80% of these questions correctly was very likely "ready for the exam".

Of course, advanced experts who just need a "refresher" do not want to answer countless "beginner questions".

NEOMO therefore first tackled the task of finding an optimal learning path for each user and how to score the answers as "fairly" as possible.

We designed an engine that automatically assessed the difficulty of the questions based on past answers. In combination with another clever algorithm, each new learner could be assigned a suitable proficiency level within a few steps, so that they automatically skipped almost all questions "below their level". The client's requirement here was to always keep the learner at the edge of his comfort zone so that he made rapid progress but did not grow frustrated.

As the scoring requirements for trainee doctors on the one hand and recertifiers on the other were very different, we designed several different scoring modules - including one based on state-space algorithms - and a "Measurement Decision Theory" component.

AI-based creation of „test-enhanced learning" from texts

The next limiting factor for the client was the effort involved in creating the learning content. In 2021, it still sounded like science fiction to have an AI do this work for you, but NEOMO implemented exactly that.

We trained a dedicated Large Language Model - at the time based on Google T5 - using 10,000s of texts and corresponding real and synthetic multiple-choice questions.

The quality of these questions reached such a high level that, as a rule, only a few manual adjustments to the generated content were necessary and our client's customers were delighted with the new possibilities.

In the course of our collaboration, we constantly provided our client with model updates so that their software was always state of the art. The current implementation can use both a GPT model and a Llama-based model to protect our client against any legal and technical risks for this business-critical function.

Support for audio input and webinars

While early applications of our technology focused primarily on text content and diagrams, it quickly became clear that other formats were also suitable for this approach.

Due to the COVID lockdowns, companies had recorded 1000s of hours of Zoom conferences. Many of these could still be useful in a different format.

In the next step, we therefore extended the AI course creation with a cloud-based pipeline that could also transform such formats into test-enhanced learning experiences through text recognition in the video (e.g. presented slides), extraction of audio content and automatic segmentation into "chapters".

Result

The final result was a powerful authoring platform that automated the tedious grunt work of importing and splitting content into different formats. Instructional designers can now focus on selecting the best sources and optimizing the learning experience.

This platform has opened up several new audiences for our client, as it has not only transformed the specialized training of medical professionals, but also many forms of corporate training, for example: every HR document and every recorded training session - e.g. on "Sexual Harassment" or "Privacy" - can be transformed into interactive and measurable courses.

As of today, more than 10,000 users work on the client's platform and benefit from an interactive and engaging learning experience.

T.K. Kanestrom, CEO, LEARNSWELL

"It was always clear to us that innovative online education tools need to rely on AI-supported workflows. It is just not effective to curate the amount of content manually. NEOMO has provided bleeding edge ideas and technology to build what we envisioned. Way before anybody even talked about ChatGPT, they helped us bring the power of Generative AI into our products."

Start a conversation with us.

We are always interested in a good conversation, whether with a specific reason or simply to exchange ideas.

You can reach us via the contact form or you can book a meeting directly.