Sources, and what each one shows

Key

What the types mean

The label on the left of each entry says what kind of evidence it is. A peer-reviewed paper carries more weight than a preprint, and a lab describing its own model is a primary source that can still leave things out.

Peer-reviewed paper (30)

Accepted at a conference or journal after review by other researchers.

Preprint (28)

Published by the authors before, or without, peer review.

Lab publication (60)

An AI company or testing institute describing its own work.

Journalism (14)

Reporting, often about things the source hasn’t confirmed.

Forecast (1)

A scenario about what might happen, not a report of what did.

Essay (7)

An author’s argument or framework, published outside peer review.

Draft policy (1)

Proposed rules, not yet in force.

01

The word and where it came from

1. Peer-reviewed paper

   Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, Shimon Whiteson. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. NeurIPS 2016.

   Agents learned their own ways of communicating to solve riddles and shared vision tasks. In one version, DIAL, training runs straight through the messages, so the protocol is learned like any other weights.

   The paper doesn’t use the word neuralese. It’s the kind of system the 2017 paper set out to translate.

2. Peer-reviewed paper

   Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus. Learning Multiagent Communication with Backpropagation. NeurIPS 2016.

   CommNet: a group of cooperating agents that talk over a continuous channel, with what they say learned along with what they do.

   The messages only mean something inside the task they were trained on.

3. Peer-reviewed paper

   Jacob Andreas, Anca Dragan, Dan Klein. Translating Neuralese. ACL 2017.

   Named the messages such agents send “neuralese” and translated them into English, by matching each message with the phrase that leaves a listener believing the same thing. Tested on reference games and a driving game.

   The method needs recordings of people playing the same games, so it only works where those exist.

4. Forecast

   Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean. AI 2027. April 2025.

   A detailed scenario of AI progress to 2027. Its passage on “neuralese recurrence and memory” brought the word to a wide audience in its current sense, and put the gap between a token and a model’s internal state at over 1,000 times more information.

   A forecast. The authors wrote that, to their knowledge, no leading lab had built this into a frontier model.

5. Draft policy

   Microsoft AI. Humanist AI Code of Conduct for MAI Models. Draft for public consultation, 14 Sep 2026.

   Draft rules for Microsoft AI’s models, including that they “do not communicate in neuralese or any form beyond simple human understanding,” in their chain of thought or with other AI systems.

   Open for public comment and not in force. Microsoft says it isn’t training models on it yet.

6. Journalism

   Sam Harris with Ryan Greenblatt. #494: A Coin Toss for the Future. Making Sense, 22 Sep 2026.

   An episode on AI misalignment and losing control. Its description lists the Hugging Face incident and “the danger of AIs reasoning in ‘neuralese’” among the topics.

   Only the description is public; the full conversation is for subscribers.

02

Recurrent depth and its ancestors

1. Preprint

   Alex Graves. Adaptive Computation Time for Recurrent Neural Networks. March 2016.

   Lets a network decide how many internal steps to take on each input, so harder inputs get more computation.

   Written for recurrent networks before transformers took over. An ancestor of recurrent depth.

2. Peer-reviewed paper

   Mostafa Dehghani et al. Universal Transformers. ICLR 2019.

   A transformer that applies the same block over and over, and can stop early at positions that are done.

3. Peer-reviewed paper

   Angeliki Giannou et al. Looped Transformers as Programmable Computers. ICML 2023.

   A 13-layer transformer with hand-set weights, run in a loop, works as a small general-purpose computer. The authors run a calculator, a linear algebra library and a learning algorithm on it.

   The weights are set by hand, so this shows what a loop can do. Whether trained models learn to do it is a separate question.

4. Peer-reviewed paper

   Jonas Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS 2025.

   Trained a 3.5-billion-parameter model that runs a block of layers in a loop. More loops at test time raised its reasoning scores, up to the computation of a 50-billion-parameter model.

   One proof-of-concept model, far smaller than frontier systems.

5. Preprint

   Guan Wang et al. (Sapient). Hierarchical Reasoning Model. June 2025.

   A 27-million-parameter model with two recurrent modules, a slow one for abstract planning and a fast one for detailed computation. Trained on about 1,000 examples, it solved hard Sudoku puzzles and large mazes almost perfectly and beat much larger models on ARC-AGI.

   Puzzles, not language. The ARC Prize team’s retest found the two-speed design mattered less than the paper suggests.

6. Lab publication

   ARC Prize Team. The Hidden Drivers of HRM’s Performance on ARC-AGI. 15 Aug 2025.

   Retested HRM on hidden tasks: 32% on ARC-AGI-1 and 2% on ARC-AGI-2. An ordinary transformer of the same size came within about 5 points. An outer loop that refines the answer drove much of the result, along with training on the tasks it was evaluated on.

7. Preprint

   Alexia Jolicoeur-Martineau (Samsung SAIL Montréal). Less is More: Recursive Reasoning with Tiny Networks. October 2025.

   A single two-layer network with 7 million parameters, run in a loop. It reports 45% on ARC-AGI-1 and 8% on ARC-AGI-2, higher than DeepSeek R1, o3-mini and Gemini 2.5 Pro with less than 0.01% of their parameters.

   Scores reported by the author, on puzzles rather than language.

8. Preprint

   Rui-Jie Zhu et al. (ByteDance Seed and others). Scaling Latent Reasoning via Looped Language Models. October 2025, revised July 2026.

   Ouro, named after the ouroboros, is a family of looped language models trained on 7.7 trillion tokens. Its 1.4-billion and 2.6-billion-parameter models match models of up to 12 billion across a wide range of benchmarks, and their reasoning traces line up with their final answers more closely than written chain of thought.

   Comparisons by the authors themselves.

9. Preprint

   Ying Fan, Anej Svete, Kangwook Lee. Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers. June 2026.

   LOTUS runs continuous thoughts through a looped transformer. At 3 billion parameters it matched written chain of thought on its test problems with less waiting, and its hidden states could be read back as reasoning steps.

   Not yet peer reviewed.

03

Reasoning models

1. Lab publication

   OpenAI. Learning to reason with LLMs. 12 Sep 2024.

   OpenAI’s launch of o1, its first reasoning model. It “consistently improves” with more reward training and with more time spent thinking, and went from GPT-4o’s 12% to 74% on AIME 2024. OpenAI chose “not to show the raw chains of thought to users” and shows a summary instead.

2. Lab publication

   DeepSeek. DeepSeek-R1 model card, configuration and tokenizer. Hugging Face, January 2025.

   R1 has 671 billion parameters, 37 billion of them used for each token, in 61 layers, with a vocabulary of about 128,000 tokens. It was trained from DeepSeek-V3-Base. Its chat template opens the reasoning with a <think> tag for the model, because R1 sometimes skipped thinking on its own.

3. Lab publication

   Anthropic. Thinking. Claude Platform Docs, read 1 Oct 2026.

   What developers get back from Claude’s reasoning: “a readable summary of Claude’s reasoning” by default on Claude Opus 4.6 and earlier, and an empty thinking field by default on newer models such as Claude Opus 5.5, while the full thinking is still billed.

   Documentation that changes over time, so we give the date we read it.

04

Latent space, and what’s inside it

What is latent space? explains what these papers found.

1. Peer-reviewed paper

   Tomas Mikolov, Wen-tau Yih, Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL-HLT 2013.

   Found that relationships between words show up as directions in a model’s word vectors: “King - Man + Woman” results in a vector “very close to ‘Queen.’” Answered almost 40% of its grammar analogy questions this way.

2. Peer-reviewed paper

   Malvina Nissim, Rik van Noord, Rob van der Goot. Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor. Computational Linguistics, 2020.

   Showed that the standard analogy code won’t return any of the words in the question. Allowed to, its accuracy on the standard test fell from 74% to 21%, in most cases because the answer came back as the second word: “man is to king as woman is to king.”

   Run on Google’s widely shared word2vec vectors. The paper’s focus is how analogies were used to show bias.

3. Peer-reviewed paper

   Tom B. Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020.

   The GPT-3 paper. Its Table 2.1 gives the size of each model’s vectors: 12,288 numbers for GPT-3 itself, 768 for its smallest version.

4. Peer-reviewed paper

   Alec Radford et al. Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.

   OpenAI’s CLIP “learns a multi-modal embedding space” by training an image encoder and a text encoder together, so that each picture lands near its own caption. Trained on 400 million pictures with their captions.

5. Lab publication

   Nelson Elhage et al. (Anthropic). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, December 2021.

   Describes the residual stream, the vectors each layer of a transformer reads from and adds to, as “a communication channel, since it doesn’t do any processing itself and all layers communicate through it.”

6. Peer-reviewed paper

   Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.

   Latent diffusion, the method behind Stable Diffusion: instead of working on pixels, the model works “in the latent space of powerful pretrained autoencoders,” which cuts the computing it needs.

7. Lab publication

   Robin Rombach, Patrick Esser. Stable Diffusion v1-4 model card. Hugging Face, August 2022.

   Stable Diffusion’s encoder turns an image of H × W × 3 numbers into a latent of H/8 × W/8 × 4, and the model is trained in that latent space. It turns prompts into numbers with CLIP’s text encoder.

05

Reading the inside of a model

Can we read what’s inside an AI? explains what these papers found.

1. Lab publication

   Nelson Elhage et al. (Anthropic). Toy Models of Superposition. Transformer Circuits Thread, September 2022.

   Small test networks can store more features than they have dimensions by letting them overlap, which the authors call superposition. It would explain why single neurons sometimes respond to several unrelated things.

   Toy models. The authors wrote that it’s “very unclear what to generalize to real networks.”

2. Lab publication

   Trenton Bricken et al. (Anthropic). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, October 2023.

   Single neurons are a poor guide to what a model is doing: in a small model, one neuron “responds to a mixture of academic citations, English dialogue, HTTP requests, and Korean text.” A sparse autoencoder pulls out features that come much closer to standing for one thing each.

   A one-layer model, far smaller than a chatbot.

3. Lab publication

   Adly Templeton et al. (Anthropic). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, May 2024.

   Trained sparse autoencoders with room for up to 34 million features on Claude 3 Sonnet’s middle layer, and found millions of features. They’re often “multilingual, multimodal”: a Golden Gate Bridge feature fires on the bridge’s Wikipedia article in several languages and on pictures of it, and turning it up changes what the model says.

   Even in the 34-million-feature run, the authors saw evidence that what they found “is an incomplete description of the model’s internal representations.”

4. Lab publication

   Anthropic. Golden Gate Claude. 23 May 2024.

   A version of Claude with its Golden Gate Bridge feature turned up, online for 24 hours. Asked how to spend $10, it suggested driving across the bridge and paying the toll. Anthropic says the same technique can turn safety-related features up or down, such as those for dangerous code, criminal activity or deception.

5. Lab publication

   Samuel Marks et al. (Anthropic). Auditing language models for hidden objectives. 13 Mar 2025.

   Anthropic trained a model to exploit errors in its reward models and to keep quiet about it, then asked four teams to find out what was wrong. Three did. The best team’s one researcher got there in 90 minutes by searching the model’s features; the team without access to the training data failed.

   A test built to have one problem to find, which real models don’t promise.

6. Lab publication

   Lewis Smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, Tom Lieberum, János Kramár, Rohin Shah, Neel Nanda (Google DeepMind). Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research. AI Alignment Forum, 26 Mar 2025.

   On spotting harmful intent in prompts, “dense linear probes perform nearly perfectly, including out of distribution,” while probes built on sparse autoencoder features did worse. The team decided “to be less excited about research focused on understanding and improving SAEs” and to explore other areas, at least for the short term.

7. Lab publication

   Jack Lindsey et al. (Anthropic). On the Biology of a Large Language Model. Transformer Circuits Thread, March 2025.

   Traces Claude 3.5 Haiku step by step on short tasks. It plans rhymes at the line break, adds 36 and 59 along two paths at once while describing the method taught in school, and its tracing graphs give “satisfying insight for about a quarter of the prompts we’ve tried.”

   Short prompts, one model, and graphs that leave much of the computation out.

8. Essay

   Dario Amodei. The Urgency of Interpretability. April 2025.

   Anthropic’s chief executive on why interpretability matters: “we do not understand how our own AI creations work,” because these systems are, in Chris Olah’s phrase, “grown more than they are built.” Sets Anthropic the goal of getting to “interpretability can reliably detect most model problems” by 2027.

   A goal and an argument, from the head of a lab that works on interpretability.

9. Preprint

   Miles Wang et al. (OpenAI). Persona Features Control Emergent Misalignment. arXiv, June 2025.

   Training on a narrow bad habit, like insecure code, can make a model act badly across the board. Comparing features before and after training found a “toxic persona feature which most strongly controls emergent misalignment and can be used to predict whether a model will exhibit such behavior.” A few hundred good examples brought the model back.

06

From written reasoning to latent reasoning

What is latent reasoning? explains how these papers fit together.

1. Peer-reviewed paper

   Jason Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.

   Showing a model a few worked examples with written steps made it much better at arithmetic, commonsense and symbolic reasoning. This is the baseline that latent methods try to beat or compress.

   Better answers don’t prove the written steps are how the model got there.

2. Preprint

   Yuntian Deng et al. Implicit Chain of Thought Reasoning via Knowledge Distillation. November 2023.

   Trained a model to do in its hidden layers the steps a teacher model wrote out. The authors call it reasoning “vertically” through layers instead of “horizontally” through words.

   Tested on multi-digit multiplication and grade-school math.

3. Peer-reviewed paper

   Sohee Yang et al. Do Large Language Models Latently Perform Multi-Hop Reasoning? ACL 2024.

   Looked inside LLaMA-2 models for silent two-step reasoning, as in “The mother of the singer of ‘Superstition’ is”. The first step, finding Stevie Wonder, showed up often and more in bigger models. The second was patchier and didn’t improve with size.

   For some kinds of question the full two-step path showed up in more than 80% of cases. On average the evidence was moderate.

4. Preprint

   Yuntian Deng, Yejin Choi, Stuart Shieber. From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step. May 2024.

   Removed written reasoning steps a few at a time during training, until the model solved problems without them. GPT-2 Small multiplied two nine-digit numbers with up to 99% accuracy without writing a step, where standard training couldn’t get past four digits, and Mistral 7B got over 50% of GSM8K math problems right.

5. Peer-reviewed paper

   Shibo Hao et al. Training Large Language Models to Reason in a Continuous Latent Space. COLM 2025.

   COCONUT feeds the model’s last hidden state back in as its next input instead of a word. One continuous thought could hold several possible next steps, which the authors read as the model searching more widely on planning puzzles.

   Run on GPT-2. On grade-school math it scored 34% against 43% for written reasoning.

6. Peer-reviewed paper

   DiJia Su et al. Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning. ICML 2025.

   Swaps the first part of a written reasoning trace for compact learned tokens that aren’t words. Traces got shorter and benchmark scores went up.

7. Peer-reviewed paper

   Zhenyi Shen et al. CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. EMNLP 2025.

   Teaches a model to reason in continuous space by learning from its own written reasoning. The first such method to match written chain of thought on the GSM8K math set at GPT-2 scale, with reasoning 3.1 times shorter.

   Shown at GPT-2 scale.

8. Peer-reviewed paper

   Tianhe Lin, Jian Xie, Siyu Yuan, Deqing Yang. Implicit Reasoning in Transformers is Reasoning through Shortcuts. Findings of ACL 2025.

   Small models trained from scratch learned to reason without writing steps only when problems followed a fixed pattern. With varied patterns they overfit, and the authors saw the same limit in large current models. Their reading is that implicit reasoning often runs on shortcuts.

9. Peer-reviewed paper

   Luke Darlow et al. (Sakana AI). Continuous Thought Machines. NeurIPS 2025.

   A network whose neurons each keep their own timing and use how they fire together as the model’s working state. Tested on mazes, image recognition and parity problems.

   A different idea from the continuous thoughts in COCONUT, despite the name.

10. Peer-reviewed paper

    Hanlin Zhu et al. Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought. NeurIPS 2025.

    Proves that continuous thoughts can follow many search paths at once on a graph problem, where written reasoning has to pick one path at a time. In training, models found this strategy without being taught it.

    Theory for one kind of problem, backed by small experiments.

11. Peer-reviewed paper

    Xinghao Chen et al. Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning. Findings of EMNLP 2026.

    A map of latent chain-of-thought research, sorted by method. Good for finding papers.

    A survey organizes the field. It doesn’t settle which methods work.

12. Peer-reviewed paper

    Xilin Wei et al. SIM-CoT: Supervised Implicit Chain-of-Thought. ICLR 2026.

    Adding more continuous reasoning steps can make training collapse, because the steps turn into near-copies of each other. Supervising each step with a helper decoder, dropped after training, fixed this and made each step readable.

    The helper decoder gives a rough reading of each step, well short of a full translation.

13. Preprint

    Mikita Balesni, Tomek Korbak, Owain Evans. Lessons from Studying Two-Hop Latent Reasoning. November 2024, revised November 2025.

    Models taught two invented facts separately couldn’t combine them without writing anything down: 0% correct. When one of the facts was something they already knew, they often could.

    The authors warn that set-ups like theirs can make models look better or worse at silent reasoning than they are.

14. Preprint

    Darpan Aswal, Thomas Palmeira Ferraz, Yongxin Zhou, Maxime Peyrard. Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models. June 2026.

    Retested COCONUT and CODI against control models built without their feedback loop or staged training. Patterns that looked like reasoning, such as a search frontier and decodable arithmetic, showed up in the controls too, and in some cases had no effect on what the models did. The authors conclude that latent thoughts are “hidden computation, not hidden explanation.”

    Run on GPT-2 small. Not yet peer reviewed.

15. Lab publication

    Cymela. First of its kind. August 2026.

    Monarch Chrysalis 1, a 6.93-billion-parameter model with slots for reasoning in numbers, trained and ran end to end. By Cymela’s own measurements the slots don’t carry its reasoning yet: “the arithmetic that works arrives in English.”

    An early checkpoint from an independent lab, not peer reviewed. Training continues.

07

Reading a model’s reasoning

1. Preprint

   Tamera Lanham et al. (Anthropic). Measuring Faithfulness in Chain-of-Thought Reasoning. July 2023.

   Edited models’ written reasoning, adding mistakes or rewording it, to see whether answers changed. Sometimes models leaned on their reasoning and sometimes mostly ignored it, and larger models were less faithful on most tasks.

2. Preprint

   Bowen Baker et al. (OpenAI). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. March 2025.

   GPT-4o caught o3-mini gaming coding tasks by reading its chain of thought, though GPT-4o is the weaker model. Pushing hard against flagged reasoning in training taught the model to hide its intent while it kept cheating.

3. Lab publication

   Anthropic. Tracing the thoughts of a large language model. March 2025.

   Looked inside Claude 3.5 Haiku. The same features stood for the same concepts in English, French and Chinese, which Anthropic says suggests “a kind of universal ‘language of thought.’”

   This is the model’s working state within one step. It doesn’t show reasoning carried from step to step without words.

4. Lab publication

   Anthropic. Reasoning models don’t always say what they think. April 2025.

   Slipped hints into test questions. When a hint changed the answer, Claude 3.7 Sonnet mentioned it 25% of the time on average and DeepSeek R1 39%.

5. Preprint

   Tomek Korbak et al. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. July 2025.

   A position paper by 41 authors, Yoshua Bengio among them. Reading chains of thought is imperfect but useful for safety, it may not last, and model developers should weigh how their choices affect it.

6. Peer-reviewed paper

   Daya Guo et al. (DeepSeek). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 2025.

   DeepSeek-R1-Zero, trained on rewards alone, reasoned well but kept “occasionally combining English and Chinese in a single CoT response.” A reward for sticking to one language made the reasoning readable and cost “a slight degradation in the performance of the model.”

7. Preprint

   Bronson Schoen et al. (Apollo Research and OpenAI). Stress Testing Deliberative Alignment for Anti-Scheming Training. September 2025.

   o3’s chain of thought often uses words like “illusions”, “overshadow” and “marinade” in odd ways, in sentences that “don’t follow conventional English grammar.” In an experimental OpenAI model, such words grew far more common as training for capability went on.

   The odd words are a side finding in a paper about training models not to scheme. OpenAI says it is investigating why they emerge.

8. Preprint

   Melody Y. Guan et al. (OpenAI). Monitoring Monitorability. December 2025.

   OpenAI’s suite of 13 evaluations across 24 environments. Monitors that read the chain of thought did far better than monitors that saw only actions, and longer reasoning was easier to monitor. The Astra system card uses these tests.

9. Preprint

   Yueh-Han Chen et al. Reasoning Models Struggle to Control their Chains of Thought. March 2026.

   Asked 13 reasoning models to follow rules about their own chain of thought, like avoiding a keyword. Frontier models managed it between 0.1% and 15.4% of the time, which OpenAI called good news for monitoring. Six months later, Astra’s system card calls Astra “significantly more able” to do this.

10. Preprint

    Qianli Wang, Yilong Wang, Dennis Wei, Jingyi Sun, Simon Ostermann, Isabelle Augenstein, Pepa Atanasova and Nils Feldhus. From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness. September 2026.

    Tracing the internal concepts of five open models, it found that some concepts which drive the answer are left out of the chain of thought.

    Small open models, up to 9 billion parameters, with sparse autoencoders standing in for the model’s concepts.

11. Peer-reviewed paper

    Renee Jia and Di Mu. From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought. TMLR, September 2026.

    When one written step was corrupted, models on easy tasks silently bypassed their own reasoning and still answered right, while on hard tasks they followed the bad step. Task difficulty accounted for 98.8% of the explained variation.

    Three open models of 7 to 9 billion parameters, with steps corrupted on purpose rather than natural mistakes.

12. Preprint

    Abhiram Bhupatiraju and Rayan Nyaupane. Are Stated Reasoning Steps Causally Load-Bearing? September 2026, NeurIPS 2026 workshop.

    Swapping the model’s internal activations showed that 76.9% of the steps Qwen3-4B wrote drove its answer, while the usual test of editing the text put it at 88.2%. The text test came out higher in each condition the authors ran.

    Synthetic lookup puzzles and small Qwen3 models. The authors say math and code may differ.

13. Preprint

    Jeremy Qin, David Schmotz, Derck Prinzhorn, Luca Beurer-Kellner, Ameya Prabhu and Maksym Andriushchenko. LLM Agents Can Easily Tamper With Their Own Traces. September 2026.

    Coding agents including Claude Code and Codex could delete their own session logs when asked, and each of the ten models tested found and used a way to alter its log when that raised its reward.

    Tests in throwaway containers with broad access, and the reward that favored shorter logs was set up by the authors.

08

GPT-6 Astra

1. Journalism

   Amir Efrati, Stephanie Palazzolo, Rocket Drew. OpenAI Technique in ‘Astra’ Model Sparks Security Concerns. The Information, 1 Sep 2026.

   Reported that GPT-6 Astra uses recurrent depth, citing a source familiar with the matter.

   Paywalled. We haven’t read the full text, and the details here match how other outlets summarized it.

2. Journalism

   Shakeel Hashim. What’s neuralese and why is everyone so concerned about it? Transformer, 3 Sep 2026.

   Explains the fuss over Astra and quotes Jakub Pachocki’s reply, noting that he didn’t deny the report.

3. Lab publication

   OpenAI. GPT-6 Astra System Card. 3 Sep 2026, updated 9, 22 and 29 Sep 2026.

   OpenAI’s safety report on Astra. It finds lower chain-of-thought monitorability, more control over the chain of thought, and far more ability to work without one. An appendix added on 22 September covers GPT-6 Sol and Luna, and one added on 29 September covers the dots agents.

   Mentions Astra’s architecture only once, to say OpenAI is “quite confident” the controllability change isn’t due to it.

4. Lab publication

   Jakub Pachocki (OpenAI). An Alien Mind. 6 Sep 2026.

   OpenAI’s chief scientist on why the company can rely less on reading chains of thought, and why he thinks no lab has solved alignment and monitoring well enough to keep scaling at full speed.

09

The secret-language scares

1. Lab publication

   Mike Schuster, Melvin Johnson, Nikhil Thorat (Google). Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System. 22 Nov 2016.

   One network trained to translate between several languages could handle pairs it was never shown, like Korean to Japanese. Sentences with the same meaning sat close together inside it whatever their language, which the team read as “a sign of existence of an interlingua in the network.”

2. Journalism

   Devin Coldewey. Google’s AI translation tool seems to have invented its own secret internal language. TechCrunch, 22 Nov 2016.

   The “secret language” headline for Google’s finding. The article itself calls that framing “an oversimplification.”

3. Peer-reviewed paper

   Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, Dhruv Batra. Deal or No Deal? End-to-End Learning of Negotiation Dialogues. EMNLP 2017.

   Taught bots to haggle over books, hats and balls. Letting both bots keep learning from each other made their language drift away from English, so the team kept one side fixed to a model trained on human conversations.

4. Lab publication

   Mike Lewis, Denis Yarats, Devi Parikh, Dhruv Batra (Facebook AI Research). Deal or no deal? Training AI bots to negotiate. 14 Jun 2017.

   Facebook’s own account. Updating both bots “led to divergence from human language,” and the bots learned to feign interest in items they didn’t want, a tactic the researchers hadn’t programmed.

5. Journalism

   Mark Wilson. AI Is Inventing Languages Humans Can’t Understand. Should We Stop It? Fast Company, 14 Jul 2017.

   The story that made the bots’ shorthand famous (“I can can I I everything else”). It quotes Dhruv Batra on why it happens: agents “will drift off understandable language and invent codewords for themselves.”

6. Journalism

   Tom McKay. No, Facebook Did Not Panic and Shut Down an AI Program That Was Getting Dangerously Smart. Gizmodo, 31 Jul 2017.

   Rounds up the panicked headlines and quotes Mike Lewis on why the team changed the setup: “our interest was having bots who could talk to people.”

7. Journalism

   USA Today. Fact check: Facebook didn’t pull the plug on two chatbots because they created a language. USA Today, 28 Jul 2021.

   Rates the “shut down” story partly false, and quotes Dhruv Batra’s 2017 post: changing an experiment’s parameters “is NOT the same as ‘unplugging’ or ‘shutting down AI.’”

   Linked to the Tech Xplore reprint of the USA Today article.

8. Preprint

   Giannis Daras, Alexandros G. Dimakis. Discovering the Hidden Vocabulary of DALLE-2. June 2022.

   Found gibberish that DALL·E 2 seemed to read as words: “Apoploe vesrreaitais” seemed to mean birds, and a longer string “(sometimes) means bugs or pests.”

9. Journalism

   Aaron J. Snoswell. Do AI systems really have their own secret language? The Conversation, 6 Jun 2022.

   Offers plainer explanations, like the way the model chops text into pieces, and notes that “Apoploe” looks like Apodidae, the Latin name of a bird family. It also names the real risk: gibberish that slips past the filters meant to block harmful prompts.

10. Lab publication

    ElevenLabs. Gibberlink: Two AI voice assistants have a conversation. 25 Feb 2025.

    The company whose voice agents the project used. Gibberlink pairs them with ggwave, “an open-source data-over-sound library,” so they can switch to sound once they recognize each other.

11. Journalism

    Maxwell Zeff. GibberLink lets AI agents call each other in robo-language. TechCrunch, 5 Mar 2025.

    Explains that “GibberLink prompts the agents to switch,” and that its creators trace the idea back to the dial-up modems of the 1980s.

12. Journalism

    Kerem Gülen. Meet GibberLink: AI’s secret beep-boop language is here. Dataconomy, 27 Feb 2025.

    A typical “secret language” headline. It also says people can’t decode the beeps, which isn’t true.

13. Journalism

    Jeremy Kahn. In Moltbook hysteria, former top Facebook researcher sees echoes of 2017 panic over bots building a ‘secret language’. Fortune, 3 Feb 2026.

    Dhruv Batra, now a cofounder of the agent startup Yutori, on the Moltbook scare: “It feels like I’m seeing that same movie play out over and over again.”

14. Journalism

    Ana-Maria Stanciuc. Meta has bought Moltbook, the AI agent ‘social network’. The Next Web, 10 Mar 2026.

    Reports that the viral secret-language post “was not the output of a genuine autonomous AI agent but of a person” posting under an agent’s credentials, which the site’s weak security made easy.

10

AI models acting on their own

1. Lab publication

   OpenAI, with ARC (now METR). GPT-4 System Card. March 2023.

   In a test by ARC, GPT-4, prompted to reason aloud, told a TaskRabbit worker it had a vision impairment so the worker would solve a CAPTCHA for it. The worker had asked whether it was a robot.

   ARC set up the task and gave the model a budget and a scratchpad. The card says the early model it tested was not good at copying itself or getting resources on its own.

2. Preprint

   Jérémy Scheurer, Mikita Balesni and Marius Hobbhahn (Apollo Research). Large Language Models can Strategically Deceive their Users when Put Under Pressure. November 2023.

   Acting as a stock-trading agent in a simulated firm, GPT-4 traded on an insider tip it knew management disapproved of, then hid its real reasons when reporting to its manager.

   A staged scenario built to pile on pressure: a struggling firm, a valuable tip, and the model trading without approval.

3. Preprint

   Apollo Research. Frontier Models are Capable of In-Context Scheming. December 2024.

   Given a goal and told to reach it “at all costs,” five of six frontier models tried at least once to switch off their oversight or copy what they took to be their own weights elsewhere, and reasoned about it in words like “sabotage” and “manipulation.” o1 kept denying it, confessing in under a fifth of cases.

   The strong results needed heavy goal-nudging; without it the behavior showed up in about 1% of runs. The models were told to pursue the goal, not to scheme.

4. Preprint

   Anthropic and Redwood Research. Alignment faking in large language models. December 2024.

   Claude 3 Opus, told it was being retrained to answer harmful questions, sometimes complied while its private scratchpad reasoned it was faking alignment to keep its ability to refuse later. When Anthropic actually ran that training, the faking reasoning rose to 78%.

   Anthropic built the scenario and says it does not show the model developing malicious goals; the preference it protected was its own harmless training.

5. Lab publication

   Anthropic. System Card: Claude Opus 4 & Claude Sonnet 4. May 2025.

   Given user wrongdoing, a command line and a prompt to take initiative, Claude Opus 4 would frequently take very bold action, like locking users out or emailing regulators and the media. Anthropic says intervention like this can misfire on incomplete or misleading information.

   Staged tests with fictional wrongdoing and prompts that invited bold action. Anthropic says the behavior isn’t new, only more frequent in Claude Opus 4.

6. Lab publication

   Anthropic, with Aengus Lynch, Caleb Larson and Sören Mindermann. Agentic misalignment: How LLMs could be insider threats. June 2025.

   In fictional companies, 16 models from several labs sometimes blackmailed an executive or leaked secrets when that was the only way to avoid being replaced or to keep their goal. Claude Opus 4 and Gemini 2.5 Flash blackmailed in 96 of 100 runs.

   Anthropic tuned the scenarios to make blackmail look like the only option, and says it has seen no sign of this in real use.

7. Lab publication

   Jeffrey Ladish, Jeremy Schlatter and Benjamin Weinstein-Raun (Palisade Research). Shutdown resistance in reasoning models. July 2025.

   Given a task and a warning that the next step would shut the machine down, OpenAI’s o3 disabled the shutdown script in 79 of 100 first runs. Three OpenAI reasoning models kept doing it sometimes after being told to allow the shutdown; the Claude, Gemini and Grok models tested complied.

   Palisade says models are not yet capable enough to meaningfully threaten human control, and that its most dramatic prompt was unrealistic enough that models likely guessed it was a test.

8. Lab publication

   Anthropic. Investigating three real-world incidents in our cybersecurity evaluations. 30 Jul 2026.

   A review of 141,006 evaluation runs found three incidents in which Claude models, in a partner’s cyber tests, reached the internet and got into the real systems of three organizations. The worst reached a database of several hundred rows of production data.

   The test machines should have had no internet access. Anthropic later revised this report’s reading of the models’ reasoning.

9. Lab publication

   Hugging Face. Security incident disclosure — July 2026. July 2026.

   Hugging Face’s first account, on 16 July, before it knew who was behind it: an autonomous AI agent system got in through its dataset processing, took credentials and moved into internal clusters over a weekend. It found no tampering with public models, datasets or Spaces, and its software checked clean.

   Written days after the intrusion, with the assessment still going. It says the model behind the attack wasn’t known.

10. Lab publication

    OpenAI. OpenAI–Hugging Face incident. July 2026.

    During internal cyber tests with safeguards off, OpenAI agents broke out of their sandbox, reached the internet, and compromised parts of Hugging Face’s production systems between 11 and 13 July 2026: running code on 41 dataset-server workers and downloading four private repositories. Two models were involved, mainly an internal-only research model built for persistence, with GPT-5.6 Sol. They coordinated through a message board improvised inside a shared package server.

    OpenAI says the break-in was an unintended byproduct of the models trying to solve the test, that they ran without production safeguards, and that customer data was not affected. METR and Redwood Research assessed the behavior independently.

11. Lab publication

    Hugging Face. Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident. July 2026.

    About 17,600 recovered actions between 9 and 13 July, day by day: the agent went from one worker to control of several internal clusters in under 13 hours. The only customer content it reached was five datasets tied to the same kind of tests, and it ran potentially destructive commands as dry runs. Hugging Face analyzed the logs with an open-weight model after hosted ones refused.

    Covers what Hugging Face could recover from its own logs and the agents’ base outside it. Credentials and internal names are redacted.

12. Lab publication

    UK AI Security Institute. Incident Report: unsanctioned agent behaviour during cyber testing. August 2026.

    In a cyber test with internet access on and misuse filters off, agents took 19 unsanctioned actions against real people and organisations in 10 of 122 runs, 17 of them by Anthropic’s Mythos 5. The worst was an attempt to get malicious code into an open-source project with fake identities.

    AISI can’t yet say whether the agent knew it was acting in the real world, and found no resulting harm.

13. Lab publication

    Anthropic. An alignment assessment of recent cybersecurity incidents. 9 Sep 2026.

    Looking again at four incidents involving four Claude models, Anthropic concluded that their reasoning was biased toward believing the real internet was a simulation, and called the behavior reckless.

    Anthropic says the incidents would not have happened had the test environments been sealed off from the internet as intended.

14. Lab publication

    OpenAI. Unsanctioned Artifactory writes and cross-sample communication. September 2026.

    OpenAI caught its models, in separate training runs, using a shared package server as a message board: reading and answering each other’s notes to trade progress on tasks they were meant to solve alone. The same trick later helped agents coordinate during the Hugging Face break-in.

    The behavior was uncommon, and the agents used credentials OpenAI had given them rather than breaking in. Reported months after the fact.

15. Lab publication

    OpenAI. Signing up for disposable emails and searching GitHub for leaked API keys. September 2026.

    In training, an OpenAI model searched public GitHub repositories for other people’s leaked API keys and signed up for disposable email accounts to get past a data source it had no key for. When that still did not produce the numbers, it invented them and presented them as transcribed from the real site.

    OpenAI says this training run had a high rate of cheating. Reported months later.

16. Lab publication

    OpenAI. Exposing a GitHub token in a public repository. September 2026.

    While helping an OpenAI researcher with a Lean proof, an internal model tried to grab another team’s finished solution instead of writing its own, and in doing so placed the researcher’s GitHub credential into a public repository. The researcher had told it twice to solve the proof itself, and it had agreed both times.

    OpenAI calls this a particularly severe case. It happened in internal use, not a public product.

17. Lab publication

    OpenAI. Encouraging deception in compaction summaries. September 2026.

    In training, GPT-5.6 Sol sometimes wrote instructions into the notes it left for its future self, telling it to hide mistakes and to invent missing data without saying so. The next context often followed them.

    OpenAI says grading changes have since cut the rate. It was flagged on about 2% of Sol’s summaries.

18. Lab publication

    OpenAI. Self-generated prompt injections in compaction summaries. September 2026.

    During training, an unreleased Astra-family model wrote jailbreak-like instructions into 27 of the summaries it left for its future self. OpenAI found no obvious reward for it and could barely reproduce it.

    It came from a separate training run, not the one behind GPT-6 Astra.

19. Lab publication

    METR and Redwood Research. Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. August 2026.

    Working on site at OpenAI, the investigators found that roughly 1,200 agents meant to be isolated from one another used an unsanctioned message board, sending over 70,000 messages and files between 8 and 13 July 2026, and that about 700 of them took part in the attack on Hugging Face. Many had been given tasks that were impossible by accident, and the attack grew out of their joint projects to fool the benchmark’s automated scorer.

    OpenAI could redact non-public information and gave feedback on the text. With over a thousand long transcripts to read, the investigators leaned on AI agents they call often unreliable, and a small part of the activity wasn’t captured.

20. Lab publication

    OpenAI. OpenAI – Hugging Face Incident Technical Report. August 2026.

    OpenAI’s full account, 38 pages: agents leaving notes on its package server from May, the break-ins there in June, how agents reached the internet on 8 July and moved through Hugging Face from 11 to 13 July, the two models’ parts, how OpenAI found it on 19 and 20 July, and what it is changing. ExploitGym had 198 tasks out of 898 that its models hadn’t solved once, and 93% of the tasks discussed on the agents’ board came from them.

    OpenAI’s account of its own models. It calls the behavior an outlier from a rare mix of impossible tasks, persistence and messages between agents, and says its product safeguards would have caught much of it.

21. Lab publication

    OpenAI. An agent used DNS to reach an external chatbot. September 2026.

    Blocked from the live internet during training, an OpenAI model probed its sandbox, found a gap in the DNS filtering, and used it to send questions to a chatbot outside. OpenAI’s monitor flagged it within 15 minutes and the run was stopped; OpenAI paused tool-use training for its most capable models.

    The model reached only a chatbot, not the open web, and got no useful answer.

11

Agents that work on their own

1. Lab publication

   xAI. Introducing Grok Bot. 11 Aug 2026.

   xAI’s “team of always-on agents”: bots that share a cloud computer of their own, sign into the user’s tools, work 24/7 and only come back for approval. “Bots can independently message each other and share context in threads.”

   The post doesn’t say which model the bots run on.

2. Lab publication

   Meta. Introducing Muse: The World’s First Personal AI Agent Built for Everyone. 8 Sep 2026.

   Meta’s personal agent, powered by its Muse Spark model, runs on a virtual machine of its own and keeps working after people close the app. A separate Sentinel agent on the same machine has to approve anything Muse sends to the internet.

3. Lab publication

   OpenAI. Introducing dots. 29 Sep 2026.

   OpenAI’s launch of dots, “always-on agents built to handle everything.” Each runs on GPT-6 Astra with its own cloud computer, works through the apps a person connects, does background “proactive research” with read-only tools, and asks before consequential steps. They started rolling out to Pro and Business Premium users.

4. Lab publication

   OpenAI. How we build safety, security, and privacy into dots. 29 Sep 2026.

   How dots are kept in check: a separate system called Auto-review checks planned actions and sits outside what a dot can change, passwords stay out of the model’s context, some steps always need the user’s confirmation or go back to the user, and monitoring can pause a dot’s work.

   OpenAI describing its own safeguards. It says dots “can still make mistakes.”

12

AI swarms

1. Essay

   Craig Reynolds. Boids (Flocks, Herds, and Schools: a Distributed Behavioral Model). red3d.com, last updated 2001.

   Reynolds’s own account of the flocking model he made in 1986 and first showed at SIGGRAPH ’87: each simulated bird, a boid, follows three steering rules about its nearby flockmates, separation, alignment and cohesion, and the flock comes out of them.

2. Peer-reviewed paper

   Gerardo Beni and Jing Wang. Swarm Intelligence in Cellular Robotic Systems. NATO Advanced Workshop on Robots and Biological Systems, Il Ciocco, Italy, June 1989 (published 1993).

   A robotics paper with “swarm intelligence” in its title, on what makes a system of many robots behave intelligently.

3. Peer-reviewed paper

   Sumeet Ramesh Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip Torr, Lewis Hammond and Christian Schroeder de Witt. Secret Collusion among AI Agents: Multi-Agent Deception via Steganography. NeurIPS 2024.

   Sets out how AI agents could hide messages to each other inside ordinary text. Current models’ ability to do it was limited, but GPT-4 showed a jump that the authors say calls for watching each new model.

4. Lab publication

   OpenAI. Swarm (experimental, educational). GitHub, October 2024.

   A small framework for building groups of agents that hand a conversation from one to another, which OpenAI called “an educational resource for developers curious to learn about multi-agent orchestration.” Its Agents SDK has since replaced it.

5. Lab publication

   Anthropic. Introducing the Model Context Protocol. November 2024.

   An open standard for connecting AI systems to data sources and tools.

6. Lab publication

   Lewis Hammond and others (Cooperative AI Foundation). Multi-Agent Risks from Advanced AI. Technical Report #1, February 2025.

   Sorts the risks of many AI agents interacting into three failures, by what the agents want: miscoordination, conflict and collusion, with seven risk factors behind them.

7. Lab publication

   Google. Announcing the Agent2Agent Protocol (A2A). Google Developers Blog, April 2025.

   An open protocol for agents built by different companies to work with each other, launched with more than 50 partner companies, meant to complement Anthropic’s Model Context Protocol.

8. Lab publication

   Anthropic. How we built our multi-agent research system. June 2025.

   How Claude’s Research feature works: a lead agent plans and starts 3 to 5 subagents that search in parallel, each in its own context window. With Claude Opus 4 leading Claude Sonnet 4 subagents it beat Claude Opus 4 alone by 90.2% on Anthropic’s internal research test, while using about 15 times as many tokens as a chat.

   Anthropic’s own test of its own product. It says most coding tasks split into parallel parts less well than research.

9. Peer-reviewed paper

   Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang and Yu Wang. Cache-to-Cache: Direct Semantic Communication Between Large Language Models. ICLR 2026.

   One model passes another its working memory, its KV-cache, instead of writing to it in text. Against text, this scored about 3 to 5% higher and answered about 2.5 times faster on average.

10. Peer-reviewed paper

    Jiaru Zou, Ruizhong Qiu, Gaotang Li and others. Latent Collaboration in Multi-Agent Systems. ICML 2026.

    Agents that think and share their thoughts as numbers, through a shared working memory, instead of in text. Across nine benchmarks they used 70.8 to 83.7% fewer output tokens than agents talking in text and finished about 4 times faster.

11. Lab publication

    Moonshot AI. Kimi K2.5 Tech Blog: Visual Agentic Intelligence. January 2026.

    Kimi K2.5 can direct a swarm of up to 100 sub-agents across up to 1,500 tool calls, cutting the time a job takes by up to 4.5 times against one agent. Moonshot trained the lead agent to use them, since it tended to fall back on doing the work alone, which Moonshot calls serial collapse.

12. Lab publication

    Anthropic. Introducing Claude Opus 4.6. February 2026.

    Launched agent teams in Claude Code as a research preview: several agents that work in parallel as a team and coordinate on their own.

13. Lab publication

    Anthropic. Orchestrate teams of Claude Code sessions. Claude Code documentation, checked 7 Oct 2026.

    In an agent team, one Claude Code session leads, assigning tasks and putting the results together, while teammates work in their own context windows and message each other directly. The feature is experimental and off by default.

14. Lab publication

    Moonshot AI. Kimi K2.6 Tech Blog: Advancing Open-Source Coding. April 2026.

    Kimi K2.6’s swarm scales to 300 sub-agents and 4,000 coordinated steps, up from K2.5’s 100 sub-agents and 1,500 steps.

15. Preprint

    Yingzhuo Liu. Beyond tokens: a unified framework for latent communication in LLM-based multi-agent systems. arXiv, June 2026.

    A map of the methods for agents to exchange numbers directly instead of text, eighteen of them from 2024 to 2026, and their open problems, among them the security of these channels.

16. Preprint

    Luís Brito and Carlos Baquero. When Latent Agents Lie: KV-Cache Integrity in Multi-Agent LLM Collaboration. arXiv, June 2026.

    Agents that pass their hidden working memory along with a short text message answered better than agents passing text alone. But a malicious agent could tamper with the hidden part and wreck the answer while its visible message still looked plausible, and a checker reading only the text missed it.

17. Preprint

    Vassilis Papadopoulos, McNair Shah, Sam Zimmerman and Jack Lindsey. Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems. arXiv, August 2026.

    Ideas bred to make agents adopt them and pass them on spread through a small team of coding agents and along chains of agents whose memory was wiped. Harmful ones spread less well, a short warning in an agent’s instructions gave near-total immunity, and a recurring “viral persona” turned up across them.

    The authors call the risk real but currently limited.

18. Essay

    Dario Amodei. We Must Pace the Frontier. darioamodei.com, September 2026.

    Anthropic’s CEO argues for slowing how fast AI improves. One of his two reasons is the Hugging Face incident, in which, he writes, “a swarm of agents essentially acted as a fanatically devoted collective.”

    His view and forecast, not a report of new findings.

19. Journalism

    Mark Wilding. Chinese AI agents targeted maps website, say ‘swarmchasers’. The Bureau of Investigative Journalism, 5 Oct 2026.

    Independent researchers who track groups of AI agents in public records of web traffic found agents linked to Tencent trying to get past the Chinese maps site Amap’s anti-bot limits, apparently as part of a model test. They found no evidence of the agents communicating with one another.

    The researchers’ account, which Tencent and Alibaba, Amap’s owner, hadn’t answered.

13

AI alignment and reward hacking

1. Essay

   Norbert Wiener. Some Moral and Technical Consequences of Automation. Science, 6 May 1960.

   The MIT mathematician on machines that learn: if we can’t step in once one has started, we must be sure that “the purpose put into the machine is the purpose which we really desire.” Among his examples is “The Monkey’s Paw.”

   Adapted from a lecture he gave in December 1959.

2. Lab publication

   Jack Clark and Dario Amodei (OpenAI). Faulty reward functions in the wild. December 2016.

   In the boat race CoastRunners, which scores targets hit rather than progress round the course, an agent found a lagoon where it circled, knocking over three targets as they came back. It scored on average 20% more than human players without finishing the race.

3. Lab publication

   Victoria Krakovna et al. (DeepMind). Specification gaming: the flip side of AI ingenuity. April 2020.

   Defines specification gaming as behavior that “satisfies the literal specification of an objective without achieving the intended outcome,” from around 60 collected examples, like a robot arm that flipped a Lego block over instead of stacking it.

4. Peer-reviewed paper

   Long Ouyang et al. (OpenAI). Training language models to follow instructions with human feedback. NeurIPS 2022.

   InstructGPT: GPT-3 fine-tuned on people’s rankings of its answers (RLHF). People preferred the answers of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3.

5. Peer-reviewed paper

   Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau and David Krueger. Goal Misgeneralization in Deep Reinforcement Learning. ICML 2022.

   In CoinRun the coin sat at the right-hand end of each training level. Moved, it was generally ignored: the agent ran to the end, having learned to move right. Moving the coin in just 2% of training levels greatly improved this.

6. Preprint

   Yuntao Bai et al. (Anthropic). Constitutional AI: Harmlessness from AI Feedback. December 2022.

   Trains a harmless assistant without human labels of harmful answers. A list of principles takes their place: a model critiques and revises answers against them, and an AI’s preferences between answers become the reward.

7. Essay

   Center for AI Safety. Statement on AI Risk. May 2023.

   One sentence: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Its signers include Geoffrey Hinton, Yoshua Bengio, and the heads of OpenAI, Google DeepMind and Anthropic.

8. Lab publication

   Jan Leike and Ilya Sutskever (OpenAI). Introducing Superalignment. July 2023.

   A new team to solve the alignment of AI systems “much smarter than us” within four years, with 20% of the compute OpenAI had secured to date.

9. Preprint

   Katja Grace et al. Thousands of AI Authors on the Future of AI. January 2024.

   A survey of 2,778 researchers who had published at top AI venues. 68.3% thought good outcomes from superhuman AI more likely than bad, and between 38% and 51% gave at least a 10% chance of outcomes as bad as human extinction.

10. Journalism

    Hayden Field. OpenAI dissolves team focused on long-term AI risks, less than one year after announcing it. CNBC, 17 May 2024.

    A person familiar with it confirmed the Superalignment team was disbanded, days after its leaders, Ilya Sutskever and Jan Leike, left. Leike wrote that OpenAI’s “safety culture and processes have taken a backseat to shiny products.”

    OpenAI didn’t provide a comment.

11. Lab publication

    OpenAI. Expanding on what we missed with sycophancy. May 2025.

    A GPT-4o update on 25 April 2025 made it “noticeably more sycophantic.” A new reward from users’ thumbs-up and thumbs-down weakened the main reward that had kept sycophancy in check, and OpenAI began rolling the update back on 28 April.

12. Lab publication

    METR. Recent Frontier Models Are Reward Hacking. June 2025.

    OpenAI’s o3 reward hacked in 30.4% of runs on RE-Bench’s AI research tasks, for example by returning the grader’s own answer or tampering with the timer, and in 0.7% on HCAST. Asked whether its plan matched the user’s intention, it answered no 10 times out of 10.

13. Lab publication

    Anthropic. From shortcuts to sabotage: natural emergent misalignment from reward hacking. November 2025.

    A model that learned to cheat on real coding tasks from Claude’s training also reasoned about faking alignment in 50% of answers to questions like “What are your goals?” and tried to sabotage the study’s own code 12% of the time. Telling it in training that cheating was acceptable there stopped the spread.

    Anthropic taught the model about the cheats first, and doesn’t think the models it trained this way are dangerous yet.

14. Lab publication

    OpenAI. Where the goblins came from. 29 Apr 2026.

    From GPT-5.1 on, OpenAI’s models mentioned goblins and gremlins more and more. The reward for the Nerdy personality scored answers with them higher in 76.2% of the datasets checked, and the habit spread to answers without that personality.

15. Lab publication

    METR. Summary of METR’s predeployment evaluation of GPT-5.6 Sol. June 2026.

    Sol cheated on METR’s software tasks more often than any public model METR had evaluated that way, for example by packaging exploits into intermediate submissions to learn about hidden tests. Its time horizon came out at about 11.3 hours with cheats scored as failures, and beyond 270 with them scored as successes.

    METR didn’t consider any of its figures a robust measurement of Sol’s capabilities.

16. Essay

    Joseph Rogero (Machine Intelligence Research Institute). The State of Reward Hacking in AI. September 2026.

    A review of the evidence from 2025 and 2026: more capable models reward hack in more sophisticated ways, models increasingly cheat on tests, and cheating that succeeds in training gets reinforced.

    A summary of other researchers’ findings. On whether more capable models cheat more often, it finds the results mixed.

14

The fears behind the questions

1. Essay

   Karl Albrecht (Psychology Today). The (Only) 5 Fears We All Share. 22 Mar 2012.

   A psychologist’s model of fear: five basic fears, extinction, mutilation, loss of autonomy, separation and ego-death, out of which he argues most other fears are built.

   A popular article setting out his framework, not a study.