TechTalks Newsletter

Share this post
Do deep learning models understand language?
bdtechtalks.substack.com

Do deep learning models understand language?

Ben Dickson
Jan 27
Comment
Share

Language modeling (such as GPT or Gopher) has been used to perform hundreds of tasks, including some with very few training examples. So remarkable are these systems, that the philosopher David Chalmers, not generally known for hyperbole, has said that “GPT-3 is instantly one of the most interesting and important AI systems ever produced. … More remarkably, GPT-3 is showing hints of general intelligence.”

But Herbert Roitblat, data scientist and author of the book Algorithms are not Enough, has a different perspective on large language models. In an essay for TechTalks, Roitblat explains the mechanics of language modeling, from tokenization to inference, and discusses their limits with some practical examples.

“With more parameters and more input data, these models get better at predicting the next word in a sequence,” Roitblat writes. “But that improvement does not necessarily indicate a substantial breakthrough in artificial intelligence. These models do just one thing—they model the probability of text tokens conditional on the context provided by other text tokens.”

According to Roitblat, “language models produce examples of language, but it is the people who read those examples who inject (most of) the meaning.”

Read the full essay on TechTalks.

For more:

  • Why AI can’t solve unknown problems

  • How can we tell if artificial intelligence understands our language?

  • AI’s struggle to reach “understanding” and “meaning”

  • Why we must rethink AI benchmarks

CommentComment
ShareShare

Create your profile

0 subscriptions will be displayed on your profile (edit)

Skip for now

Only paid subscribers can comment on this post

Already a paid subscriber? Sign in

Check your email

For your security, we need to re-authenticate you.

Click the link we sent to , or click here to sign in.

TopNewCommunity

No posts

Ready for more?

© 2022 Ben Dickson
Privacy ∙ Terms ∙ Collection notice
Publish on Substack Get the app
Substack is the home for great writing