#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED]

Patreon: https://www.patreon.com/mlst
Discord: https://discord.gg/HNnAwSduud
YT version: https://youtu.be/pMtk-iUaEuQ

Dr. Walid Saba is an old-school polymath. He has a background in cognitive  psychology, linguistics, philosophy, computer science and logic and he’s is now a Senior Scientist at Sorcero.

Walid is perhaps the most outspoken critic of BERTOLOGY, which is to say trying to solve the problem of natural language understanding with application of large statistical language models. Walid thinks this approach is cursed to failure because it’s analogous to memorising infinity with a large hashtable. Walid thinks that the various appeals to infinity by some deep learning researchers are risible.

[00:00:00] MLST Housekeeping
[00:08:03] Dr. Walid Saba Intro
[00:11:56] AI Cannot Ignore Symbolic Logic, and Here’s Why
[00:23:39] Main show - Proposition: Statistical learning doesn't work
[01:04:44] Discovering a sorting algorithm bottom-up is hard
[01:17:36] The axioms of nature (universal cognitive templates)
[01:31:06] MLPs are locality sensitive hashing tables

References;
The Missing Text Phenomenon, Again: the case of Compound Nominals
https://ontologik.medium.com/the-missing-text-phenomenon-again-the-case-of-compound-nominals-abb6ece3e205

A Spline Theory of Deep Networks
https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf

The Defeat of the Winograd Schema Challenge
https://arxiv.org/pdf/2201.02387.pdf

Impact of Pretraining Term Frequencies on Few-Shot Reasoning
https://twitter.com/yasaman_razeghi/status/1495112604854882304?s=21
https://arxiv.org/abs/2202.07206

AI Cannot Ignore Symbolic Logic, and Here’s Why
https://medium.com/ontologik/ai-cannot-ignore-symbolic-logic-and-heres-why-1f896713525b

Learnability can be undecidable
http://gtts.ehu.es/German/Docencia/1819/AC/extras/s42256-018-0002-3.pdf

Scaling Language Models: Methods, Analysis & Insights from Training Gopher
https://arxiv.org/pdf/2112.11446.pdf

DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
https://arxiv.org/abs/2006.08381

On the Measure of Intelligence [Chollet]
https://arxiv.org/abs/1911.01547

A Formal Theory of Commonsense Psychology: How People Think People Think
https://www.amazon.co.uk/Formal-Theory-Commonsense-Psychology-People/dp/1107151007

Continuum hypothesis
https://en.wikipedia.org/wiki/Continuum_hypothesis

Gödel numbering + completness theorems
https://en.wikipedia.org/wiki/G%C3%B6del_numbering
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems

Concepts: Where Cognitive Science Went Wrong [Jerry A. Fodor]
https://oxford.universitypressscholarship.com/view/10.1093/0198236360.001.0001/acprof-9780198236368