Google's Chief Scientist delivers a masterclass on AI infrastructure, inference hardware, and where founders should focus — from YC Startup School 2026.
In 2001, Jeff Dean and Sanjay Ghemawat realized Google's entire search index could fit in RAM. They shipped it in days. Search got fast. In 2013, Dean did another napkin calculation: if every Google user used speech recognition for just three minutes a day, Google would need to double its entire server fleet. That calculation became the TPU.
Now, at YC Startup School 2026, Google's Chief Scientist sat down with YC's Diana Hu to share the thought experiments behind both breakthroughs — and where the next ones are hiding. The 58-minute conversation is dense with insights for anyone building in AI, from hardware engineers to startup founders.
Here's the deep dive.
Last year, Dean predicted AI had reached "junior engineer" capability. Looking back, he says the prediction was spot on — but he underestimated the speed of improvement.
"The models have been getting a lot better at agent-based, longer-running coding tasks, and it seems pretty clear that they are now actually pretty capable."
More surprisingly, agent-based systems are shining beyond coding. Dean sees this as a critical trend: AI agents are no longer just writing snippets — they're tackling multi-step, complex tasks across domains.
Dean's next bold call? ML systems will start improving themselves through automated experimentation:
"You will see a lot more automation of ML systems themselves — getting ML systems to improve their capabilities by running lots of experiments, breaking things down into sub-problems, running those sub-problems in a tight automatic experimentation loop, putting the results together."
This isn't just about ML. Dean believes this "automated scientific method" applies to any field with a measurable objective — from materials science to chip design. The key is compressing the experiment-evaluate-iterate cycle to near-zero latency.
Dean draws a direct parallel between two inflection points:
The argument is straightforward but profound. Today's AI systems are built on general-purpose chips (GPUs, TPUs). But inference — the act of running trained models to generate predictions — has very different requirements than training. It demands low latency, low energy consumption, and high throughput.
"You're going to see more and more high-performance and low-energy inference hardware systems. Because inference is the key to making these agent-based systems be available to more and more people."
His vision: inference hardware so fast that latency drops 50x. Imagine AI agents that respond instantly — no more watching the spinner. "Waiting's no fun," Dean quips. Speed is king.
The TPU story proves the point. By building a chip that does only low-precision dense linear algebra — the mathematical heart of nearly all ML algorithms — Google achieved:
The TPU couldn't run Chrome or Word. It couldn't do anything except ML math. And that's exactly why it was transformative.
Dean sees the same opportunity today, but for inference specifically. The next wave of specialized chips will minimize data movement, use ultra-low precision computation, and hardcode only the precision formats that matter.
This is perhaps the most underappreciated insight in the entire interview.
Here's the math that should keep AI engineers up at night:
"Just bringing it in from HBM on an accelerator into the processor so it can actually compute on it — that's a thousand times more energy than the computation itself."
This 1,000x gap quietly shapes everything in AI system design. It's why we batch data for training (to amortize the data movement cost across many samples). It's why low-latency inference is hard (batching introduces delays). It's why algorithm design is constrained by hardware topology.
Many problems that founders label "model problems" are actually energy or data IO problems in disguise. The architecture of the system — how data flows between memory, compute units, and network — often matters more than the model's parameters.
Dean calls for an updated version of his famous "Latency Numbers Every Engineer Should Know." For 2026, the critical numbers are:
These aren't abstract benchmarks. They directly determine what AI products are viable.
The TPU's origin is a story about napkin math and foresight.
In 2013, deep learning-based speech recognition at Google halved error rates overnight — essentially delivering 20 years of speech recognition progress in months. But the models were computationally expensive.
Dean's calculation was simple: if better speech recognition meant people used it more — just three minutes a day — Google would need to double its server fleet. Running on CPUs was economically unsustainable.
The solution: build a chip that does only what ML needs. Low-precision dense linear algebra. Nothing else.
"If you build a specialized chip for low-precision, dense linear algebra and can't do anything else, that turns out to be really useful for machine learning inference."
Crucially, the TPU was designed as a general-purpose linear algebra system, not a model-specific chip. The Transformer architecture didn't exist yet. But Dean and team knew ML algorithms would evolve, so they built for the mathematical foundation, not the specific model.
That bet paid off massively when Transformers emerged — the TPU was already perfectly suited for the matrix multiplications that Transformers rely on.
Most people think of AI agents as tools that run for minutes — maybe an hour. Dean says think bigger.
"People don't quite realize how possible it is to have agent-based systems that can run not just for an hour or two hours, but for some problem domains and with highly capable models, you can get them to run for days or weeks and do really, really complicated tasks."
Real example: telling an agent to rewrite an entire software system in a different programming language, with better safety or performance properties. The agent works through it methodically, over days.
Today's agents typically degrade after ~10 tool interactions. Dean identifies two causes:
His solutions:
This "search over solution space" technique is one of the most powerful tools for reliable, long-chain agent systems.
Dean makes a critical distinction: AI progress is no longer just about bigger models. It's about everything around the model.
"The model is just one part of the system you're building. The real goal is to build a complete system that can solve interesting problems."
This means:
Dean calls this "context engineering" — and it's accessible to anyone with an API key. You don't need billions in compute. You need creativity in how you structure context, tools, and workflows.
Dean and Sanjay Ghemawat recently built a skill for optimizing microbenchmark performance at Google. The skill teaches models how to:
They also published a 30-page "Performance Hints" document that, when fed to models, measurably improves their ability to reason about code performance. It's publicly available — and Dean encourages everyone to try it.
This is the insight that gives the talk its title.
Dean's advice to founders: find problems where current AI models succeed only 0–1% of the time. Not 20%.
Why? If a model already handles a task at 20%, that means the capability is emerging. With more data, bigger models, and more scale, it'll improve. You'll be competing with frontier labs on a converging path.
But if the model fails completely (0–1%), you've found a genuine gap. Two paths to exploit it:
Personal information organization: General models don't have access to your personal data. A product that helps users organize their own information has a structural advantage that no amount of model scaling can overcome.
Specialized domain models: Build a highly accurate, narrow model for a specific domain. AlphaFold is the perfect example — a specialized model for protein folding that's been transformative, built by a small team with deep domain expertise.
Other promising domains: materials science, chip design, and any field where you can access the right training data for a specialized model.
Before building, ask: will frontier models solve this in 6–12 months? If yes, pick something else. If the gap is 2–3 years, you have runway.
If AI agents can write all the code, what becomes valuable?
"Taste — having excellent judgment on what problems to tackle — as the scarce skill."
Dean draws from his research background: in research, problem selection determines most of the outcome. A well-chosen problem solved adequately beats a boring problem solved brilliantly.
Three methods Dean recommends:
Experience: Work on many different problems. Your accumulated experience teaches you which problems are worth pursuing.
Write predictions: List things you think will be important in the next 12 months. After 12 months, review: what came true? What did others build? What didn't happen? This trains your "taste generation ability."
Crazy thought experiments: Question assumptions everyone takes for granted.
For 60 years, the semiconductor industry has worked to make transistors smaller and more reliable — aiming for zero bit flips. But what if we built systems with transistors that fail 20 times a day instead of once every millions of years?
"That would be a completely different design point. Maybe it opens up interesting new possibilities in manufacturing."
The brain works this way — signals aren't perfectly reliable, so important information travels through multiple pathways. Could we design computing systems the same way? Dean isn't saying we should — but the thought experiment reveals how deeply held assumptions shape what's possible.
Dean points to systems like AlphaChip (for chip layout) and AlphaEvolve (for proposing, evaluating, and keeping solutions) as early examples of AI building better AI.
The broader vision: automate the scientific method itself.
"If you can automate that loop — propose experiment, run experiment, evaluate results, iterate — and compress the latency of that loop to be very low, that would be extremely impactful."
The architecture: an orchestrator decomposes a high-level goal into sub-problems. Each sub-problem enters an auto-experiment loop. Solutions are combined back into a complete answer. This framework could accelerate ML, science, and engineering simultaneously.
When working with AI agents, clarity is everything.
"If you can articulate what you want clearly, you will be much more successful. The clearer you are, the more the agent has boundaries, rules, and knows what it's supposed to accomplish."
The best example of this working today: translating code between programming languages. Why does this work so well? Because the specification is perfectly clear — the original source code is the spec. The model can run tests from the original language, translate them, and verify behavior matches.
Dean's advice: think of working with agents like writing design documents. Before delegating, write a crisp specification. The better your spec, the better your results.
| Insight | Action |
|---|---|
| AI is at junior engineer level | Build for agent-based workflows, not one-shot prompts |
| Inference hardware is the next frontier | Think about latency as a product feature, not a cost |
| Data movement costs 1000x more than compute | Design systems around data flow, not just model architecture |
| Agents will run for weeks | Build infrastructure for long-running, multi-step tasks |
| Context engineering > model scaling | Invest in tools, retrieval, memory, and orchestration |
| The 1% Rule | Find problems where models fail completely, not partially |
| Taste is the scarcest skill | Practice problem selection through predictions and thought experiments |
| Clear specs = better agents | Write crisp design docs before delegating to AI |
Original source: Jeff Dean: The 1% Rule for Building in AI — Y Combinator, July 30, 2026
Full transcript: YC Root Access
YC Library page: Y Combinator Startup Library
Jeff Dean's interview is a rare glimpse into the mind of someone who has shaped computing infrastructure at Google for over two decades. His core message: the biggest opportunities in AI aren't in building bigger models — they're in rethinking the hardware, energy constraints, and system architecture that make AI possible. And the best founders will be the ones with the taste to know which problems are worth solving.
