The problem is still that the AIs are kinda bad, not that they are good

So as hard as this is to take in - news like the last post, the spate of mathematical proofs using big AI for formal verification or Anthropic's admission they had their own HuggingFace moment sound obviously like the arrival of super-duper smart AI and in a sence that is of course also true. But every single one of these stories also highlight that in many ways all the AI agents are still also really bad.

It remains true that you can't translate your intuition about which kinds of intelligence and skills go together from people to agents and robotos.

Take Anthropic's lawbreaking with agents. In the logs it keeps maintaining that this sure is a convincing simulation, instead of realizing it was doing actual crimes on real systems, not simulated crimes on test systems.

Even when you look at the spectacular math breakthroughs and dive into exactly how the bots went at it, you see a busy bee not a fantastic genius just grinding at the problem like it was an exam problem - checking every door with infinite patience and high speed.

For all the sophisticated results - the robots are still - to a very large degree - simpletons.

Indeed, all the crimes they have committed are not the crimes of supervillains but of simpletons. They are basically defacing websites more than doing a Blofeldian world takeover.

And this remains the central point: Even for these super strong model - the main problem is not that they are strong, but that they are also weak. The weakness is what makes them dangerous.