We ask whether today’s language and reasoning models can be called planners in the classical sense, judging only from what they output, and what output-only test would separate a valid plan from plan-shaped text. We review the three core readings and eight further papers, and group them by the claim they make: that LLMs retrieve plan-shaped text, that reasoning models show a partial and improving capability, that planning is a property of a system in which a sound component guarantees validity, and that the model’s useful role is to write the planner rather than the plan.
The decisive evidence is a contrast: classical planners are invariant to renaming, while every language model studied loses validity under obfuscated names. We conclude that current models are not planners in the classical sense, but that “copying” no longer describes the best of them.
GPT-4, zero-shot plans on Blocksworld versus the same domain under deceptive names.
o1-preview on the same contrast: far better, at over 60 times the cost of GPT-4o per instance.
GPT-5 on fresh tasks after random renaming. The classical planner LAMA stays at 204.
A classical planner searches a PDDL task for a sequence of actions that provably reaches the goal. A language model completes a description into steps that look like a plan, with no guarantee they execute. So validity is checked separately, by a validator such as VAL.
Renaming separates the two. It leaves every precondition and effect intact, so a planner’s search and VAL’s verdict do not change. Any change in validity must come from the model’s reliance on names.
(:action pick-up :parameters (?x) :precondition (and (clear ?x) (ontable ?x) (handempty)) :effect (and (holding ?x) (not (ontable ?x)) (not (clear ?x)) (not (handempty))))
(:action overcome :parameters (?x) :precondition (and (province ?x) (harmony ?x) (craves)) :effect (and (pain ?x) (not (harmony ?x)) (not (province ?x)) (not (craves))))
The literature is grouped by what it claims, not paper by paper. All four claims must explain the evidence above.
GPT-4 and Claude-3 Opus collapse to about 0% under renaming. Chain-of-thought barely moves the result, and success does not track optimal plan length the way a planner’s would.
Wrong if: obfuscated validity sat close to ordinary validity for the same model and prompt.
o1-preview and R1 saturate Blocksworld and make the first non-trivial progress on Mystery Blocksworld. On fresh tasks, frontier models keep 59–94% of their performance after renaming.
Wrong if: validity collapsed to near zero on fresh random renamings, or followed familiarity rather than difficulty.
In LLM+P, LLM-Modulo and the Du et al. pipeline, an external component guarantees every returned plan. The success rate measures how often the LLM supplies usable material, not whether it plans.
Wrong if: there were no gain over a random or empty proposer with the same sound component.
GPT-4 programs solve a mean of 62% of larger held-out tasks over seven domains, falling to 4% with names removed. Katz et al. extend this to generating verified search components, models or policies.
Wrong if: programs fit training tasks but failed on larger held-out ones, or showed no drop without names.
| Source | Model / system | Domain, condition, prompting | Ord. | Obf. | External help in the result |
|---|---|---|---|---|---|
| Valmeekam et al. [2023b] | GPT-4 | PlanBench BW vs. Mystery BW (deceptive), NL, zero-shot, 600 inst. | 34.6 | 0.16 | None; VAL scores only. Randomized names: 0%. CoT: 35.6 vs. 9.0 |
| Kambhampati et al. [2024] | Claude-3 Opus; GPT-4o | Same, zero-shot | 59.3; 35.5 | 0; 0 | None |
| Kambhampati et al. [2025] (v1) | o1-preview; R1; o1-mini | Same, zero-shot | 97.8; 97; 56.6 | 52.8; 42.6; 19.1 | None at inference; verifiers used in post-training |
| Valmeekam et al. [2023b], reported in Kambhampati et al. [2024] | GPT-4 + VAL back-prompting | 50 previously failed inst. per domain, ≤15 rounds; BW / Logistics / Mystery BW | 82 / 70 | 10 | VAL finds errors; LLM proposes all plans |
| Corrêa et al. [2025] (v1) | GPT-5; Gemini 2.5 Pro; R1; LAMA | 360 freshly generated IPC-2023 tasks, 8 domains; random-string names | 56.9; 43.1; 43.6; 56.7 | 42.2; 40.6; 25.8; 56.7 | None for LLMs; LAMA is a planner |
| Du et al. [2026] | GPT-4; GPT-4o; o1-mini + pipeline | PlanBench Mystery BW, 600 inst., budget b=3 | – | 46.2; 46.2; 48.0 | STRIPS simulator, VAL, targeted repair. Baselines 0.16–19.1 taken from earlier work |
| Liu et al. [2023] | GPT-4 → PDDL → Fast Downward | 7 IPC domains, 20 tasks each, with one example (problem, PDDL) | 45.7 (69.3†) | n/a | Planner does all search; human-written domain file. LLM-as-planner: 9.3 |
| Silver et al. [2024] | GPT-4 writes Python | 7 PDDL domains, held-out tasks larger than training; “No Names” ablation | 61.6 | 4.4 | Debugging feedback on training tasks only |
| Goebel and Zips [2025] | o1 vs. Fast Downward | 5 PDDL domains, 93 tasks | 63.4 | n/a | None; planner reaches 97.9 |
“Ord.” and “Obf.” give the percentage of valid plans (or tasks solved) under ordinary and obfuscated names for the same setup. All rates are external-validator scores. Numbers are from the cited version and are not comparable across rows with different benchmarks. † Counting the sub-optimal planner alias where the optimal one timed out. Liu and Silver means are computed from the per-domain rates in those papers.
Kambhampati et al. call 52.8% “a long way” from robustness; Corrêa et al. find frontier models “competitive” after renaming. The setups differ in task freshness, kind of renaming, domains and model generation. The two agree on the direction and differ on the size of the drop.
Du et al. see three models converge at 46–48% and credit validation and repair. LLM-Modulo with VAL alone reached only 10%. The improvement is real, but much of it is the pipeline’s, which is Claim C rather than evidence that the model plans.
“Surprisingly powerful”, say Silver et al., yet their own No Names column drops from 62% to 4%. Both readings hold: the programs generalise to larger instances, but the strategy is recovered from the meaning of the names.
LLM-Modulo checks every proposal in the loop; Katz et al. prefer calling the LLM once per domain and reusing what passes. Both agree the LLM is not the planner. They differ on where its unreliability is absorbed.
Threats to cross-paper comparison. Four factors make the numbers fragile. Contamination: PlanBench instances and solutions are public, so fresh generation is the only control found. Prompt and representation: natural-language and PDDL prompts give different rates for the same model. Kind of obfuscation: deceptive words, random strings and generic identifiers are not interchangeable. Budget: LRMs, back-prompting and repair pipelines spend far more computation than single-shot generation, which a success rate alone hides.
What outputs alone can support. Validity, renaming invariance, size generalisation and detection of unsolvable tasks are output claims. Statements that a model “reasons”, “searches internally” or that its intermediate tokens are “thoughts” are claims about internals, and this review does not draw them.
A system earns the label “planner” on a domain if, on freshly generated instances scored by VAL:
For a generated program, (i)–(iii) apply to held-out tasks larger than any training task. When a sound component takes part, the label belongs to the system, and the LLM’s share is measured against a null proposer.
No language or reasoning model studied is a planner in the classical sense. Every one fails criterion (i), while the planner baselines pass it by construction. But “copying plans it has seen” is no longer an adequate description of frontier reasoning models, which keep most of their performance on fresh, renamed tasks.
We read current models as strong, renaming-sensitive plan generators. Validity guarantees come from sound components, at inference or at construction time, and that is where reliable LLM planning currently lies.
@techreport{yakpetrolindo2026language,
author = {Yakp{\'e} Trolindo, Hubert and {CAP 6671 group members}},
title = {Can a Language Model Plan, or Is It Copying Plans It Has Seen?
An Enhanced Review Organised Around the Output-Only Test},
institution = {University of Central Florida},
type = {CAP 6671 Course Report},
year = {2026},
month = nov,
url = {https://github.com/cybsecoff31/Can-a-language-model-plan-or-is-it-copying-plans-it-has-seen-}
}