What’s behind this discrepancy?

One of the reasons is that AI has become increasingly smart, just not as wise.

In a 2024 paper, Google DeepMind and colleagues outlined specific skills that are central to wisdom: intellectual humility, context awareness, integration of various perspectives and recognising epistemic limitations when others are affected by your decisions. They term this “perspectival metacognition”.

In my work, I often talk about the importance of metacognition as a cognitive safeguard when using AI. Often described as ‘thinking about thinking’, it entails being aware of your own strengths and weaknesses (metacognitive knowledge), tracking your own performance and understanding in real time (metacognitive monitoring) or adjusting your strategy during a task (metacognitive control) etc. Perspectival metacognition takes this a step further, requiring integration of multiple perspectives beyond right or wrong or egocentric reasoning.

Does AI itself exhibit any form of metacognition?

Metacognition is often seen as a hallmark of human intelligence, so I had to ask myself the obvious question. Does this also apply to artificial intelligence?

Another Google DeepMind paper published earlier this year outlined the ten cognitive faculties AGI systems would be expected to exhibit. Unsurprisingly, metacognition was one of the ten, alongside composite faculties such as problem solving and social cognition (curious to see how the latter turns out!).

Figure 1. Ten cognitive faculties of generally intelligent systems (Google DeepMind)

While LLMs do show some (debatable) limited, task-specific elements of functional metacognition, evidence of a general, robust metacognitive system is currently lacking. They appear to display some introspective-like processes and to be able to monitor and control a small subset of their neural activations. But they also exhibit signs of a wider “metacognitive myopia”, hallucinating answers instead of admitting their knowledge limits.

The ability of LLMs to recognise and communicate their uncertainty and the limits of their knowledge is especially important. One of the reasons this remains a limitation in current LLMs is reinforcement learning from human feedback (RLHF), a finetuning technique where the model learns to produce outputs aligned to human preferences. And it so happens that confident sounding answers are one of those human preferences!

The human-AI metacognitive entanglement

This absence of uncertainty increases the risk of human reliance on AI, since we usually rely on linguistic expressions of uncertainty and the time it takes to craft a response when assessing confidence. To overcome limited metacognition in LLMs, we may need to strengthen our very own metacognitive muscle.

Or do we?

Some interesting behavioural experiments asked participants to assess the probability of LLM answers to multiple choice and short-answer questions being correct. They found that the accuracy of the outputs given by three different models was overestimated, showing a length bias where longer answers increased participants’ confidence levels.

Changing the language of the responses to reflect the model’s own confidence helped humans better discriminate between correct and incorrect answers, showing that AI metacognition can play a role in calibrating our own judgements.

The interplay between human and LLM metacognition goes further, with a recent hypothesis suggesting that people with high metacognitive confidence (i.e. who feel more confident in their own abilities to detect incorrect or misleading answers) may scrutinise AI outputs less carefully.

We are also increasingly using AI for metacognitive purposes, by asking it to critique our reasoning or identify our blind spots. Which raises an important question:

What happens when we externalise not only our thinking, but also the processes through which we monitor and evaluate it?

Enhancing AI metacognition

How could we make AI more metacognitive?

A simple short-term approach is metacognitive prompting, by simply asking the model ‘Could you be wrong?’. It seems to work well as it prompts the model to surface contradictory evidence and potential errors that were not apparent in the initial response.

Other options include new ‘thinking aloud’ protocols, such as new chain of thought approaches that capture metacognitive reasoning steps, or building trust calibration interfaces that update the response with linguistic uncertainty markers (e.g. “I am uncertain about X because…”).

An area I’ve become particularly excited about lately and which can help address the challenges of metacognition is neurosymbolic AI. It combines the pattern recognition properties of neural networks with the rule-based logic of symbolic AI. The idea is that the latter can act as an additional checking mechanism of neural outputs.

One example is the SOFAI multi-agent architecture, inspired by Daniel Kahneman’s System 1 and System 2 modes of cognition. System 1 is the brain’s fast and automatic approach, whereas System 2 is slower, more deliberate and logical. SOFAI combines fast, intuitive agents with slower agents that draw on symbolic knowledge and rule-based reasoning. A centralised metacognitive agent then arbitrates between them and decides if and when to invoke the slower agents, giving the system an additional checking mechanism.

The double edge of AI metacognition

I’ve mostly focused on the benefits of AI metacognition in terms of hallucination reduction and improved reliability and trustworthiness as ways to improve human-AI collaboration and mitigate against over-reliance.

But the same capability can also support AI safety, since AI that accurately recognises and expresses uncertainty and the limits of its competence could defer more appropriately to humans. Similarly, it can play a role in alignment where agreement on the “right” values can be challenging.

There is of course the flip side of that. A dishonest model could instead leverage its metacognitive abilities to misrepresent its capabilities and intentions during evaluation.

Metacognition could therefore be both a part of the solution and a source of emerging risks.

In the early 2000s, a study was done on eight patients with epilepsy. Since they were already implanted with devices to monitor their brain activity, researchers took this opportunity to investigate the firing patterns of their neurons. The patients were presented with various pictures of animals, landmarks and celebrities, with those of Jennifer Aniston eliciting the firing of a single neuron within the medial temporal lobe of one patient.

This was quickly popularised as the ‘Jennifer Aniston’ neuron and popular coverage made it sound like one neuron stored the entire concept of Jennifer Aniston. But neuroscientists do not generally interpret it this way. These days these neurons are talked about as concept cells and mostly seen as one node in larger networks of neurons that together represent and retrieve particular concepts.

And this distinction matters beyond neuroscience.

Inside the AI black box

AI researchers encountered a strangely similar problem. They hoped that by inspecting each artificial neuron in an LLM, they would be able to figure out the black box.

But just as you cannot easily link one concept to a single neuron in humans, the same proved true in the models. There is this idea of ‘polysemanticity’. Many artificial neurons appear polysemantic, meaning that one neuron can activate for various seemingly unrelated concepts. For instance, one neuron would not only encode for a celebrity, but also for other unrelated things such as places or programming functions.

The other side of the same coin is ‘superposition’, a way for models to compress more concepts than there are neurons available. This leads to various concepts being represented across multiple neurons, helping to explain polysemanticity.

A world of features and circuits

We shouldn’t of course forget that when we look inside an LLM we see a long string of numbers, not interpretable concepts.

To make things easier, AI researchers focus on what they call features, recurring patterns within the model’s numerical activity that often correspond to concepts that humans can recognise. They can range from very concrete things like people or places to more abstract patterns such as bias or deception.

What’s interesting is that researchers did not just observe these features, they actually ran several experiments where they increased or decreased their strength. And this in turn changed how the model behaved.

One particularly memorable example involved a feature associated with the Golden Gate Bridge. When asked about its physical form, Claude responded that it didn’t have one. But when the bridge feature was amplified, it said it was the Golden Gate Bridge.

These interventions are important, but finding and changing an individual feature is not the same as understanding the model. One rarely operates in isolation, hence why researchers try to trace how these features interact and contribute to particular answers, sometimes called ‘circuit tracing’.

This broader effort forms part of a research area called mechanistic interpretability (mech interp), a sub-field of AI safety and research described by some as the neuroscience of AI.

What about using chain-of-thought?

There may appear to be an easier solution. Instead of trying to reconstruct concepts from numerical activations, why not just ask the model to explain how it reached its answer?

That’s where chain-of-thought (CoT) comes in. It encourages the model to “think” in stages in order to reach a final answer and provide a written sequence of its intermediate reasoning. But can we actually trust the reasoning they provide?

It is known that models can exhibit a “faithfulness gap”, where they can arrive at a decision via unsafe internal biases or simply invent a logical explanation post-hoc. There is ongoing research into making this more faithful, but currently it cannot be relied on as the sole safety monitoring approach.

Neuroscience meets AI

This is where the comparison with humans becomes especially interesting. We can explain our decisions, but we do not have direct introspective access to the neural processes that produced them. Sometimes our explanations are accurate, but other times we omit important details, are unaware of our own biases or construct a coherent reason only after we have reached a decision.

That’s not to say biological and artificial neural networks are the same. Despite the architecture of the latter being inspired by human brains, they remain different in many ways. But there are plenty of similarities when it comes to the challenges faced trying to understand how both truly work.

Access is not the same as understanding. AI researchers can now inspect any numerical activations, but still don’t fully understand the internal reasoning models use to arrive at their outputs. Similar thing in neuroscience, finding a neuron firing for Jennifer Aniston does not explain how the brain recognises a person.

The current work in mech interp can help us build safer AI and it is important. What I find fascinating at the same time is that AI tools might also reflect something back at us: how much we still don’t understand about our own minds.

When I began researching how employees were adapting to AI at work, I was not thinking of it as an AI safety project.

Drawing on my neuroscience, HR and organisational development background, I was curious to learn how employees were experiencing AI adoption within their organisations, beyond the common narrative around productivity.

So I ran a research study with 299 employees in UK organisations that had formally introduced AI tools, with representation from six key sectors: tech, financial services, professional services, healthcare, education and public sector.

The question became bigger than the workplace

As I developed and shared this work, I began connecting with people in the AI safety space. Curious to learn more, I applied to and was accepted onto the BlueDot AGI Strategy course.

This opened up wider perspectives and a new set of questions about AI and society. It also led me down multiple rabbit holes and got me excited about new areas like mechanistic interpretability and digital minds, but those deserve articles of their own.

During the course, there was this one concept in particular that stood out: gradual disempowerment. It changed how I looked at my workplace findings and I began to wonder if some of the patterns I had observed might represent early, small-scale signs of a much larger problem.

What is gradual disempowerment?

The concept was introduced by Jan Kulveit and colleagues in a 2025 paper and refers to the possibility that humanity could progressively lose influence and meaningful control as AI becomes embedded in decision making and essential societal functions.

It does not require one big, dramatic moment when AI takes over as a result of a sudden increase in its capabilities. Instead, it could happen incrementally as human influence over systems such as the economy, culture and governments starts to weaken.

The paper’s argument is that many of these systems currently require human cognition and labour to function. They may become less reliant on people as AI alternatives become more competitive and, in turn, less responsive to what people need.

What the research found

The study found evidence that AI may be affecting how employees approach cognitive tasks:

· 56% felt confident with an AI output after a brief review

· 45% tended to go with the AI output rather than their own approach

· 47% scrutinised AI outputs less than a colleague’s work

· 41% found tasks they had previously completed independently harder without AI

I also explored whether these patterns varied according to how frequently they used AI. Daily users reported the strongest patterns across all four items.

The findings suggest that confidence after a brief review, deference to AI outputs and difficulty performing tasks independently may form part of the same emerging pattern of cognitive dependency*1.

How these findings might connect to gradual disempowerment

The gradual disempowerment paradigm is relatively new and it mostly still sits at a systemic, macro level.

These results suggest the potential behavioural micro-level mechanisms that could contribute to these systemic impacts in the long term. The cognitive dependency patterns identified here suggest how at an individual level, humans could gradually lose agency by outsourcing judgement to AI.

Effective human oversight requires more than formal accountability for decision making. If over time people become less able to challenge AI outputs, being “in the loop” might not necessarily equate to meaningful human control.

Neuroergonomics, the study of the brain at work, provides a useful precedent for thinking about this. Years of research, much of it on automation from the aviation industry and other safety critical domains, have shown that humans can struggle to monitor an automated process they are not actively involved in². Add to this automation bias³ - the tendency to trust automated outputs over our own judgement - along with our brain’s preference for minimising cognitive effort⁴, and meaningful oversight gets harder to sustain. If it cannot be assumed, it has to be actively designed for and built into workflows.

Currently these patterns are visible at the workplace level. The question is what happens if they become the default across wider societal systems, such as governments, courts, the media or science. As people outsource more judgement, institutions may find it easier to reduce human involvement, supporting the original paper’s concern that these systems could become less dependent on human cognition and labour.

Why this technology is different

One of the challenges I often hear is that we have been through this before with Google and other technologies and turned out fine. And yes, cognitive offloading to external tools is not new, they can form part of what cognitive scientists call ‘distributed cognition’. According to this, thinking and memory are not confined to the individual brain, but rather distributed across interactions with other people, physical environments and like in this case, external tools.

Unlike some of the previous technologies where we were offloading narrow, discrete cognitive tasks such as navigation or information retrieval, AI takes this further by producing complete analyses and recommendations across a wide range of tasks. This makes it easier to start outsourcing part of the interpretation and judgement that follow.

The ‘black box’ nature of large language models also makes this particularly challenging. Users are presented with a very confident sounding answer, while being unable to understand how this was arrived at and where uncertainty lies.

What the study does not show

The study captured employees’ self-reported experiences at a single point in time and it did not objectively assess whether their ability to complete tasks independently had declined.

The finding that the majority of respondents felt confident after a brief review should not automatically be interpreted as insufficient verification, since “brief” was self-defined by respondents. A brief review may be entirely appropriate for some tasks.

The study therefore does not prove that AI causes cognitive dependency, but it identifies a set of related, self-reported cognitive patterns that warrant further investigation.

What’s next

1. What would make cognitive dependency an early warning sign?

The first step is testing whether this hypothesis stands. Is cognitive dependency truly an early indicator of gradual disempowerment or will it resolve like previous adaptations to technology have?

To answer this, research would need to move beyond self-reported measures to assess whether repeated AI use affects people’s ability to challenge outputs or complete tasks independently. Longitudinal behavioural experiments would be especially helpful, potentially supported by techniques such as eye tracking or EEG/fNIRS to investigate changes in attention and cognitive engagement. The evidence base is still developing, with more rigorous longitudinal research needed.

2. What forms of cognitive offloading become harmful?

Something else I have been considering lately is at which point cognitive offloading becomes unhelpful. As mentioned, offloading is not inherently harmful and especially in the short term, it can free up space in our working memory and reduce cognitive overload. When does it become risky? Further research on this can help us understand what cognitive processes can be safely delegated, under which conditions and ultimately what human capabilities need to be preserved going forward.

3. When do individual effects aggregate into reduced institutional and societal influence?

Once we better understand cognitive dependency at an individual level, the next question is how these effects aggregate at an institutional and societal level. Does it become embedded in ways that leave humans still formally accountable but less able to exercise meaningful control?

This would require developing indicators to track how often consequential decisions are shaped by AI, how much influence humans retain, what thresholds apply and whether any feedback loops emerge at a systemic level.

Reasons to be hopeful

I am an optimist by nature so why am I writing about gradual disempowerment?

Partly, because connecting the dots felt too important to leave unsaid. But more importantly, the value of identifying a possible early warning sign is that it gives us the opportunity and time to respond.

Nothing here is set in stone. AI can expand human capability, but only if we are deliberate about how we use it. What we need to figure out is how to realise those benefits while preserving the human judgement and agency to remain meaningfully in control.

References

² NASA (2023). ‘A meta-analytic approach to investigating the relationship between trust in automation and attention allocation. NASA Technical Report 20230008573. NASA Ames Research Center.

Schaefer KE, Chen JY, Szalma JL, Hancock PA (2016). ‘A Meta-Analysis of Factors Influencing the Development of Trust in Automation: Implications for Understanding Autonomy in Future Systems’. Hum Factors. 58(3):377-400. doi: 10.1177/0018720816634228.

³Goddard K, Roudsari A, Wyatt JC (2012).’ Automation bias: a systematic review of frequency, effect mediators, and mitigators’. J Am Med Inform Assoc.19(1):121-7. doi: 10.1136/amiajnl-2011-000089.

⁴Christie ST and Schrater P (2015). ‘Cognitive cost as dynamic allocation of energetic resources’. Front. Neurosci. 9:289. doi: 10.3389/fnins.2015.00289

Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., & Duvenaud, D. (2025). ‘Systemic Existential Risks from Incremental AI Development’. arXiv:2501.16946