LLMs and math

Mathematics is one of the areas where large language models like ChatGPT are confusing. On the one hand, they can correctly produce the answer to advanced problems such as integration and differentiation, while on the other, they sometimes fail spectacularly at very easy problems, such as comparing two numbers.

In this week’s article, I examine two recent papers that investigate the limits and capabilities of LLMs in math. One of them focuses on ChatGPT in advanced math, and the other explores LLMs in elementary math/word problems.

Some key findings:

• LLMs are inconsistent in nearly any math topic

• They also fail at elementary math

• They might perform decently at one skill, but can’t combine different skills (elementary math+common sense)

• LLMs fine-tuned on math datasets improve drastically, but still require further investigation

• LLMs can be reliable tools for searching and reverse-searching math knowledge bases

Read the full article on TechTalks.

For more on AI research:

• To understand language models, we must separate “language” from “thought”

• What is the “forward-forward” algorithm, Geoffrey Hinton’s new AI technique?

• The power of wide transformers models

• AI21 Labs’ mission to make large language models get their facts right