Knowledge Distillation (Glossary)

Knowledge Distillation falls into the Artificial Intelligence category and is particularly important in areas such as automation and Industry 4.0.

Imagine there's a large, very clever Artificial Intelligence (AI) that solves complex tasks but requires a lot of processing power – much like an experienced expert. However, sometimes a smaller, faster AI is intended to take over the same tasks, perhaps because it needs to run on a simple device. This is where model knowledge transfer comes in: the knowledge, or „experience,“ of the large AI model (the teacher model) is specifically transferred to a smaller, more efficient model (the student model). The small model thus learns the tricks and shortcuts from the large model, delivers similar results, but uses far fewer resources.

For example: In a factory, a large AI software monitors all the machines. So that small devices or robots in the hall can also act intelligently, they receive only the really relevant knowledge through knowledge distillation. This allows them to work quickly and save energy while still benefiting from the findings of the main AI.