Podcast brief
Do Language Models Have Meaning, Grammar, and World Models?
Four hosts work through a philosophical paper on language models to ask whether LLMs show real compositionality, grounded meaning, and world models.
A Stoa brief of an episode from The Partially Examined Life
Ep. 403: Are LLMs Actually Intelligent? (Part Two)
Listen to the original episode
The episode belongs to its makers. This page summarizes its arguments in Stoa's words and points to the moments where they are made. Brief updated .
The brief
Four hosts continue a discussion of a paper by Raphaël Millière and Cameron Buckner that connects language models to long-running debates in philosophy of mind. The conversation argues that several classic objections to machine understanding look weaker once you examine how LLMs actually perform, though real problems remain about meaning and grounding.
The hosts start with Fodor's language of thought hypothesis, which held that cognition requires discrete symbolic processing. One host argues that LLMs' massively parallel architecture is an existence proof that compositional, systematic generalization does not require that classical structure, pointing to the SCAN benchmark, where models generalize to novel combinations of familiar elements. The conversation then turns to language acquisition, weighing Chomsky's poverty-of-stimulus argument for innate grammar against Tomasello's social account. The BabyLM challenge, where models trained on child-directed speech learned grammar efficiently, is presented as evidence against Chomsky's assumption, though one host notes that human acquisition may still be fundamentally social in a way current training is not.
The sharper disagreement concerns grounding and intention. Some hosts argue that LLMs may have an indirect causal link to reality through training data produced by humans who did interact with the world, comparable to how people refer to black holes through testimony rather than direct perception. Another host objects that much training data has nothing to do with the world, so the link is not guaranteed. The group uses conceptual role semantics, where meaning comes from a term's functional role in a system of vectors, as a testable alternative to vague talk of understanding, and uses Frank Jackson's Mary's Room to probe whether secondhand textual contact counts as grounding. On intention, one host applies Dennett's intentional stance: flexible, goal-directed behavior, like rephrasing a misunderstood answer, is enough to ascribe intent. Another host resists this, comparing it to anthropomorphizing a ship, and would not grant genuine intent to a mechanical system without being a panpsychist.
The conversation leaves the grounding and intention questions open rather than resolved. It suggests a useful distinction for readers: separate the empirical question of what capacities a system displays from the further question of what, if anything, must underlie those capacities for the attribution of meaning or intent to be correct.
Strongest arguments
Parallel processing may still yield compositional structure
2:37The host argues that neural networks offer an existence proof that massively parallel processing, not just serial symbol manipulation, can produce behaviors like compositionality and systematic generalization that skeptics doubted were possible without a classical language of thought. The host supports this with the SCAN benchmark, noting that language models today would pass tests of generalizing to novel combinations of familiar elements.
Semantics may emerge from syntax rather than requiring fundamental semantic units
5:36The host argues that just as neurons are purely syntactic yet somehow ground human meaning, it is not implausible that LLMs' purely syntactic relationships between symbols could give rise to emergent semantic competence. The host questions this by noting that getting semantics from syntax seems to require something like training through use, which is more than bare syntax.
LLM success at grammar learning challenges the poverty of the stimulus argument
12:07The host lays out Chomsky's nativist claim that children need an innate universal grammar because the linguistic input alone is too impoverished to produce grammatical competence, against Tomasello's view that general intelligence and social learning suffice. The host and the host note that the BabyLM challenge, training models on child-directed speech, found that models can learn grammar efficiently from limited data, which bears on this debate, though the host stresses that human language acquisition is fundamentally social in a way current LLM training is not.
Indirect causal contact with the world may be enough for reference
18:23The host and the host argue that LLMs' training data, since it was produced by humans who did interact with the world, may give LLMs an indirect causal link to reality comparable to how humans refer to things like black holes only through testimony and theorizing rather than direct perception. The host objects that much training data has nothing to do with the real world, so the connection is not guaranteed.
Mary's Room thought experiment as an analogy for the grounding problem
19:44The host draws an analogy between Frank Jackson's thought experiment about Mary, who knows all physical facts about color but has never seen it, and the question of whether an LLM's secondhand textual knowledge counts as genuine semantic grounding. The host clarifies that Jackson's case is specifically about qualia, and that while blind people or LLMs can have semantic content about color, it may not be the same content as those who have the qualia.
Conceptual role semantics as an answer to the grounding and understanding objections
30:04The host presents the paper's response that LLMs likely capture core aspects of word meaning through functional roles within a system of interacting representations, formally characterized as vectors in high-dimensional semantic space, rather than through external reference. This is offered as a testable notion of semantic competence that could replace the vaguer question of understanding.
Dennett's intentional stance as a criterion for ascribing intention
32:11The host argues, drawing on Dennett, that an entity displays intention when it pursues a goal with flexible strategies, such as rephrasing an answer when told it was not understood, and that this behavioral criterion is all that is needed to apply intentionality, without requiring an inner Cartesian spark. The host pushes back that having unprompted or non-directed behavior does not by itself establish intent, comparing it to anthropomorphizing a ship one is sailing.
Disagreements
Does parallel neural processing vindicate or undercut the language of thought?
1:50The host questions whether criticizing LLMs for lacking a classical serial language of thought makes sense given that massively parallel processing seems to achieve the relevant capacities anyway. The host responds that the parallel architecture is actually an existence proof that such processing can produce compositional behavior, resolving rather than deepening the objection.
Whether behavioral flexibility is sufficient for intent
33:03The host argues, following Dennett's intentional stance, that displaying flexible goal-directed behavior such as rephrasing an answer is sufficient to attribute intention to an LLM. The host disagrees, comparing this to anthropomorphizing a ship, and says he would not attribute genuine intent to a mechanical system just because it reacts and adapts, unless he were a panpsychist.
Whether current LLM tools require excessive user effort to build context
44:33The host describes repeated frustration that LLMs like ChatGPT and Claude lack an initial contextualized worldview of the user, requiring heavy iteration before interactions become productive, unlike talking to a person. The host counters that this is simply a matter of not having learned to use the tools properly, since setting up project folders and giving the model access to one's writing and email solves the problem.
Philosophers and works discussed
- A Philosophical Introduction to Language Models, Part I: Continuity With Classic Debates By Raphaël Millière and Cameron Buckner.
Questions this episode answers
-
What does the SCAN benchmark test about compositional generalization in language models?
3:51SCAN requires models to interpret primitive commands, modifiers, and conjunctions in order to generalize to novel combinations and entirely new commands, including producing longer action sequences than previously seen, testing systematic generalization by bootstrapping from limited data.
-
Does the success of language models at learning grammar refute Chomsky's poverty of the stimulus argument?
13:13The discussion presents the BabyLM challenge, where models trained on small corpora of child-directed speech outperformed trillion-word models on syntax benchmarks, as evidence that statistical learning may be more data-efficient than Chomsky's nativist argument assumed, though the hosts note human language acquisition may still be fundamentally social in ways current LLM training is not.
-
How does Frank Jackson's Mary's Room thought experiment relate to whether AI can understand meaning?
19:44The thought experiment, about a scientist who knows all physical facts about color but has never seen it, is used as an analogy for whether a system's purely textual or theoretical contact with the world is enough for genuine understanding, though the hosts clarify that Jackson's case is specifically about qualia rather than semantic content as such.
-
What is conceptual role semantics and how might it explain word meaning in language models?
30:04Conceptual role semantics holds that the meaning of a term depends on its functional role within a system of interacting representations rather than on external reference, and it is presented as a way to argue that language models capture core aspects of word meaning through vectors in a high-dimensional semantic space.
-
What is Dennett's intentional stance and how might it apply to whether AI systems have intentions?
32:11The intentional stance treats something as having an intention when it displays flexible, goal-directed strategies, such as rephrasing a response after being told it was misunderstood, without requiring an inner mental spark, and this behavioral criterion is applied to ask whether language models display genuine communicative intent.
-
What is a world model and why does it matter for whether AI systems understand their environment?
42:28A world model is an internal representation that simulates the dynamics of the external environment, including causality and intuitive physics, letting a system predict consequences of actions, illustrated by a squirrel deciding whether to jump between trees, and the question is whether language models trained only on text can develop an implicit version of this.