We didn’t see it coming until it was too late. Over the past seven days, three of the most widely used AI benchmark databases—MMLU, HumanEval, and GSM8K—lost their last shred of discriminative power. Every major model now scores above 90% on them. The signal is gone. And when the signal disappears, what replaces it? For Scott Wu, CEO of Cognition, the answer is clear: proprietary, real-world evaluations. But for those of us who have spent years building trust in decentralized systems, this shift feels alarmingly familiar. It’s the same story we lived in crypto—centralized gatekeepers deciding what “good” looks like, without transparency, without accountability, and without the community’s consent.
Cognition’s flagship product, Devin, is an autonomous software engineering agent. It doesn’t take multiple-choice tests; it builds real applications. Wu argues that static benchmarks have become useless for comparing frontier models. He’s right—technically, the data supports him. Yet his solution—proprietary evaluation—is a dangerous retreat into silos. It replaces one kind of opacity with another. And here’s the kicker: the crypto industry already solved this problem years ago. We built proof-of-stake consensus, on-chain reputation systems, and decentralized oracles precisely to address this kind of trust failure. The question is whether AI will learn from our experience or repeat our mistakes.
Let me ground this in something personal. In 2021, during the NFT mania in Manila, I watched my dormitory neighbors lose their savings to a rug pull. I manually audited the top five trending NFT projects and identified the scam two days before launch. That intervention saved $15,000 in student money—not because I was smarter, but because I had access to on-chain data. The chain didn’t lie. Today, we face a similar situation with AI evaluation: the public data is saturated, and the proprietary data is locked behind corporate walls. But the solution isn’t to abandon open metrics—it’s to build better ones on decentralized infrastructure.
The Saturation Trap: When Benchmarks Become Noise
Scott Wu’s central claim is that “models have saturated every test.” This is not hyperbole. Industry-wide, GPT-4, Claude 3, and Gemini Ultra all exceed 90% on MMLU, HumanEval, and GSM8K. The differences are within statistical noise. As a result, choosing a model based on these scores is like choosing a car based on top speed when all cars hit 300 km/h—you need something else: reliability, safety, real-world handling. In AI, that “something else” is task-specific performance.
But here’s the hidden dynamic: the saturation wasn’t accidental. It was driven by a generation of researchers optimizing for exactly those benchmarks. Training data leaked, hyperparameters were tuned to maximize those exact eval sets, and soon the scores ceased to reflect generalization. This is what I call the “leaderboard trap.” In crypto, we saw the same with DeFi TVL metrics—projects would farm liquidity to top a ranking, but the TVL didn’t measure sustainability or security. We learned to look past vanity metrics. The AI community is now learning the same lesson.
Cognition’s response is to double down on proprietary evaluation. They will run Devin through internal tests that simulate real software engineering tasks—creating a custom sandbox, running scripts, debugging errors. They will then publish only the aggregate results they choose. This creates a trust problem. Who verifies the data? Who audits the evaluation pipeline? Without a decentralized, verifiable system, the results are just marketing.
From Proprietary to Decentralized: The Blockchain Playbook
Based on my experience auditing lending protocols during the 2022 bear market, I know that trust is not achieved by hiding data. It is achieved by making data permissionless and verifiable. When my DeFi Resilience DAO of 200 members audited Aave and Uniswap, we didn’t just read the whitepapers—we ran the test suites on-chain, we checked the invariants, we verified the math. That process is replicable for AI evaluation.
Imagine a decentralized evaluation protocol where: - Tasks are sampled from a public pool maintained by a DAO of domain experts. - Execution occurs in a reproducible sandbox on a trusted execution environment (TEE) or a zero-knowledge rollup. - Results are posted on-chain with a cryptographic proof that the model was evaluated honestly. - Reputation scores accumulate over time, allowing the community to weight evaluations based on the validator’s track record.
This isn’t science fiction. Projects like Chainlink already provide verifiable randomness and off-chain computation. Arweave offers permanent storage for evaluation scripts and results. Polygon’s zkEVM can scale verifiable computation. The building blocks exist. What’s missing is the will to treat AI evaluation as a public good, not a corporate asset.
The Contrarian Angle: Why Decentralized Evaluation Isn’t a Panacea
But let me be honest—decentralization has its own blind spots. The first is quality. Public task pools can be gamed. A malicious actor could submit trivial tasks to boost their model’s score, or a validator could collude with a model provider to skew results. In crypto, we fight this with economic incentives—staking, slashing, and dispute resolution. That works for financial consensus, but does it work for subjective AI performance? A code-generation task that runs successfully might still produce insecure code. A human evaluation layer is still needed.
Second, latency matters. Waiting for on-chain finality (even with L2s) is impractical for real-time model comparisons. You’d need an off-chain aggregation layer with periodic settlement—similar to how Optimistic Rollups work. It’s doable, but adds complexity.
Third, the AI companies themselves may resist. Proprietary evaluation is a moat—it allows them to control the narrative. A decentralized protocol would reduce their pricing power and expose weaknesses. Expect fierce lobbying against “standardized transparency.”
Yet these challenges are not insurmountable. We solved similar problems in DeFi: flash loans were a vulnerability, then became a tool for arbitrage and audits. We solved the trilemma with sharding and L2s. We can solve the AI evaluation trilemma with a layered architecture: public task registry → off-chain execution with cryptographic proofs → on-chain settlement with economic security.
What This Means for Builders and Investors
If you are building in AI, the signal is clear: public benchmarks are dead for differentiation. Your competitive moat will come from proprietary data or proprietary evaluation. But that moat is fragile if you cannot prove its integrity. Start investing in verifiable evaluation pipelines now. Use TEEs, zk-proofs, or consortium chains to transparently share evaluation results with limited disclosure. It will give you credibility with sophisticated buyers.
If you are investing, watch for signs of evaluation opacity. A startup that refuses to share its evaluation methodology beyond a press release is hiding something. Demand on-chain, auditable proof. The due diligence process for AI companies should include a technical review of their eval pipeline, just as we reviewed smart contract code in DeFi.
If you are a regulator, the time to act is now. Mandate that AI safety evaluations be conducted on a publicly verifiable platform, at least for high-risk applications. The EU AI Act’s requirement for “appropriate transparency” is a start, but without decentralized infrastructure, it will be toothless.
The Road Ahead
We didn’t enter crypto because we loved volatility. We entered because we believed in a world where trust is not delegated to institutions but verified by mathematics. AI is now at a similar inflection point. The benchmarks that once guided us have become noise. The proprietary evaluations that replace them risk creating new gatekeepers. But we have the tools—blockchain, zero-knowledge proofs, decentralized oracles—to build a better alternative.
Scott Wu is right about one thing: the old evaluation paradigm is broken. But his proposed solution is a step backward. We don’t need more black boxes. We need open, verifiable, community-governed evaluation protocols. This is the next frontier for crypto, not as speculation, but as infrastructure for the AI age.
Will we seize it? Or will we watch another industry fall into the same centralization trap we fought so hard to escape?
Consensus is built in the dark—but verified in the light.