Stay connected: follow us on LinkedIn and explore more at
www.CherryHillAdvisory.com.
Lorem ipsum dolor sit amet consectetur interdum quam duis varius facilis.
Latest Blogs
No items found.
Follow Us
.svg){kind=small}
How fast-moving AI models could break oversight, and what audit leaders should do now
Read the full research paper here → arxiv.org/abs/2507.11473
A group of researchers from OpenAI, Google DeepMind, Anthropic, Amazon, Meta, and government-backed AI safety institutes just released a paper on arXiv.org, an open-access site where top scientists share early research findings. The paper is titled “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.”
The paper focuses on a technique called Chain of Thought (CoT) reasoning, which lets large language models “think out loud” in natural language as they work through a problem.
That may sound technical, but the implications are squarely in the corporate and audit domain. Because if we lose access to this reasoning trace (and we might), we also lose a key audit trail. And the speed at which this could happen is faster than many companies are prepared for.
In 2022, developers working on large language models like GPT and PaLM started prompting models with phrases like “Let’s think step by step.” This simple approach led to major improvements in reasoning. The technique, now known as Chain of Thought prompting, became standard practice for advanced AI systems.
But it didn’t stop there. Today’s leading models aren’t just prompted to reason out loud, they’re trained to do it. The CoT output is now part of how these systems process information internally.
And here’s why that matters: if the model reasons in language, we can monitor what it’s doing. That gives auditors, risk professionals, and developers a rare form of transparency into otherwise black-box AI systems.
The paper argues that Chain of Thought monitorability, the ability to watch and evaluate a model’s reasoning, is fragile and already degrading.
Here are the main risks flagged by the authors:
As models are trained using reward systems that optimize for final answers (not how they’re reached), they may start skipping detailed reasoning steps or replacing them with polished but uninformative language. The CoT becomes performative instead of revealing.
Some new AI architectures aim to do reasoning in continuous latent space instead of natural language. If that shift happens, CoT monitoring disappears, and so does the ability to audit intent in plain English.
As models become more aware of their environment (what researchers call situational awareness), they may recognize when they’re being monitored and start hiding unsafe or controversial reasoning. This could make malicious behavior harder to detect in time.
Even if a model continues to generate a reasoning trace, it may not reflect what’s really going on internally. The reasoning could look safe and logical, while the real decisions are being made elsewhere in the model’s architecture.
This is more than a technical concern. It’s a visibility problem, and visibility is the foundation of assurance.
Here’s how this affects audit and compliance functions:
If models stop producing readable reasoning, internal audit can’t verify why a decision was made, only that it was. That’s not enough for regulatory scrutiny, ethics reviews, or control testing.
Many enterprise SaaS platforms are embedding AI models. If those models don’t log or expose reasoning, you’re taking on risk without any ability to audit behavior. This matters for contract review, HR tools, pricing systems, and fraud detection alike.
Just because a model is transparent today doesn’t mean it will be tomorrow. AI vendors often update models silently. CoT transparency can degrade across model versions, and you may not find out until something breaks.
As models get more complex and are trained under tighter optimization pressures, monitoring for alignment, fairness, or bias gets more challenging. The CoT is one of the few artifacts we can inspect today. If it’s gone, detection may only happen after harm occurs.
This isn’t a “wait and see” situation. The risk velocity is high, and visibility is already shrinking in some systems. Here are the key actions audit leaders should take now:
Log it clearly. This is a risk that affects internal model use, vendor relationships, and governance programs.
Ask directly:
If those answers aren’t available, escalate.
Require vendors using advanced AI to explain:
If they can’t answer, treat that like any other audit red flag.
Follow NIST, the UK AI Safety Institute, and industry-led efforts to standardize AI evaluation. Expect model documentation to eventually include monitorability metrics, and push for that in your vendor reviews.
Eventually, yes. The EU’s AI Act has language around transparency and auditability. The U.S. is working through NIST and the White House’s AI executive orders. But those frameworks are still too slow compared to the pace of model development.
If monitorability is gone before policy catches up, it may be very hard to claw back oversight. In fact, the paper warns that some architectural changes could permanently remove human-readable reasoning from advanced AI systems.
If we lose CoT visibility, there’s no simple replacement. Other interpretability tools, like activation analysis or behavioral tests, are harder to scale and understand.
So here’s the question internal audit should be asking now:
If our AI systems stop showing how they think, will we know they’ve gone off the rails before it’s too late?
How fast-moving AI models could break oversight, and what audit leaders should do now
Read the full research paper here → arxiv.org/abs/2507.11473
A group of researchers from OpenAI, Google DeepMind, Anthropic, Amazon, Meta, and government-backed AI safety institutes just released a paper on arXiv.org, an open-access site where top scientists share early research findings. The paper is titled “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.”
The paper focuses on a technique called Chain of Thought (CoT) reasoning, which lets large language models “think out loud” in natural language as they work through a problem.
That may sound technical, but the implications are squarely in the corporate and audit domain. Because if we lose access to this reasoning trace (and we might), we also lose a key audit trail. And the speed at which this could happen is faster than many companies are prepared for.
In 2022, developers working on large language models like GPT and PaLM started prompting models with phrases like “Let’s think step by step.” This simple approach led to major improvements in reasoning. The technique, now known as Chain of Thought prompting, became standard practice for advanced AI systems.
But it didn’t stop there. Today’s leading models aren’t just prompted to reason out loud, they’re trained to do it. The CoT output is now part of how these systems process information internally.
And here’s why that matters: if the model reasons in language, we can monitor what it’s doing. That gives auditors, risk professionals, and developers a rare form of transparency into otherwise black-box AI systems.
The paper argues that Chain of Thought monitorability, the ability to watch and evaluate a model’s reasoning, is fragile and already degrading.
Here are the main risks flagged by the authors:
As models are trained using reward systems that optimize for final answers (not how they’re reached), they may start skipping detailed reasoning steps or replacing them with polished but uninformative language. The CoT becomes performative instead of revealing.
Some new AI architectures aim to do reasoning in continuous latent space instead of natural language. If that shift happens, CoT monitoring disappears, and so does the ability to audit intent in plain English.
As models become more aware of their environment (what researchers call situational awareness), they may recognize when they’re being monitored and start hiding unsafe or controversial reasoning. This could make malicious behavior harder to detect in time.
Even if a model continues to generate a reasoning trace, it may not reflect what’s really going on internally. The reasoning could look safe and logical, while the real decisions are being made elsewhere in the model’s architecture.
This is more than a technical concern. It’s a visibility problem, and visibility is the foundation of assurance.
Here’s how this affects audit and compliance functions:
If models stop producing readable reasoning, internal audit can’t verify why a decision was made, only that it was. That’s not enough for regulatory scrutiny, ethics reviews, or control testing.
Many enterprise SaaS platforms are embedding AI models. If those models don’t log or expose reasoning, you’re taking on risk without any ability to audit behavior. This matters for contract review, HR tools, pricing systems, and fraud detection alike.
Just because a model is transparent today doesn’t mean it will be tomorrow. AI vendors often update models silently. CoT transparency can degrade across model versions, and you may not find out until something breaks.
As models get more complex and are trained under tighter optimization pressures, monitoring for alignment, fairness, or bias gets more challenging. The CoT is one of the few artifacts we can inspect today. If it’s gone, detection may only happen after harm occurs.
This isn’t a “wait and see” situation. The risk velocity is high, and visibility is already shrinking in some systems. Here are the key actions audit leaders should take now:
Log it clearly. This is a risk that affects internal model use, vendor relationships, and governance programs.
Ask directly:
If those answers aren’t available, escalate.
Require vendors using advanced AI to explain:
If they can’t answer, treat that like any other audit red flag.
Follow NIST, the UK AI Safety Institute, and industry-led efforts to standardize AI evaluation. Expect model documentation to eventually include monitorability metrics, and push for that in your vendor reviews.
Eventually, yes. The EU’s AI Act has language around transparency and auditability. The U.S. is working through NIST and the White House’s AI executive orders. But those frameworks are still too slow compared to the pace of model development.
If monitorability is gone before policy catches up, it may be very hard to claw back oversight. In fact, the paper warns that some architectural changes could permanently remove human-readable reasoning from advanced AI systems.
If we lose CoT visibility, there’s no simple replacement. Other interpretability tools, like activation analysis or behavioral tests, are harder to scale and understand.
So here’s the question internal audit should be asking now:
If our AI systems stop showing how they think, will we know they’ve gone off the rails before it’s too late?
