The emergent abilities of LLMs are not what they seem

Large language models (LLM) like ChatGPT and GPT-4 have captivated the imagination of the world. They have manifested many fascinating properties, including emergent abilities at scale.

But a new study by researchers at Stanford University suggests that previously reported “emergent abilities” might be a mirage and caused by choosing the wrong metrics to evaluate the LLMs.

This study is important because it demystifies some of the magical and obscure abilities that have been attributed to LLMs.

Key findings:

• Previous studies show that as you scale LLMs they acquire new capabilities that are not present in smaller models

• These studies suggest that you can continue to improve LLMs just by scaling them

• The new study shows that emergent abilities are caused by evaluating the models with non-linear metrics

• In reality, the models are constantly improving, but the non-linear metrics only show that improvement after the model passes certain threshold

• The study is significant because it can help scientists better predict the capabilities of scaling LLMs

Read the full report on TechTalks.

For more on AI research:

• What you need to know about multimodal language models

• What is reinforcement learning from human feedback (RLHF)?

• How to solve AI’s “common sense” problem

• Large language models have a reasoning problem