Why Token Efficiency Has Less To Do With Prompting – And Everything To Do With How You Design Your Codebase
Over the past few months, I’ve been obsessed with a simple yet painful question:
How do you make AI work reliably on a large, messy, fast-evolving codebase without burning tokens like gasoline?
Cursor switching to token-based billing, Claude Code tightening quotas, and the limitations of local models turned this from a curiosity into an engineering priority.
Along the way, frustration gave way to clarity.
And clarity turned into a strategy.
This article is that strategy.
Most articles about “AI for coding” focus on prompting.
Almost none talk about codebase architecture as the actual interface for AI.
That’s the missing piece.
The Early Phase: When I Thought Prompting Could Solve Everything
At first, I believed the same thing many devs believe today:
“If I perfect the rulebook, the model will behave perfectly.”
So I went all in.
I played with:
- basic prompts
- granular prompts
- multi-layer rule systems
- MCP
- memory
- tools
- sub-agents
- plugins
- custom agent workflows
If a tool existed, I tried it.
If a hack existed, I implemented it.
And honestly?
It worked.
Outputs were good.
Predictable.
Consistent.
But the cost was ridiculous.
Token usage ballooned to unsustainable levels.
Then pricing changed.
Quota limits tightened.
Local models choked.
Context windows broke.
And I found myself deep in diminishing returns.
The more “rules” you add, the more your AI feels like a junior dev drowning in onboarding documents.
That’s when reality hit.
The Hard Truth: Adding More Rules Makes Models Worse, Not Better
Here’s what I discovered after hundreds of experiments:
- No rules → ~5/10 performance
- Basic rules → ~8/10
- More rules → still ~8/10, sometimes even 6/10
Indeed:
- more rules = more hallucinations
- more rules = more confusion
- more rules = more token waste
- more rules = lower precision
This realization pushed me into a discipline I didn’t even have a name for at the time.
Context Engineering
The invisible skill that determines 80% of AI effectiveness
Context Engineering is not about stuffing the model with more knowledge or more instructions.
It’s about designing the environment in which the model operates.
A balance between:
- clarity
- context size
- navigability
- token efficiency
- cognitive load on the model
So I did something extreme.
I removed all rules.
And suddenly:
Performance increased to 8–9/10 while token usage dropped to 10–20%.
(Yes, in that direction.)
This was the moment I stopped believing in spec-driven prompting.
Specs are ambiguous.
Code is concrete.
And models interpret specs differently from how humans do, no matter how carefully you craft them.
(Even the best models struggle with abstract specs. Specs lack grounded behavior, while code offers deterministic feedback. This is why tests often outperform documentation as a source of truth.)
Trying to force the model through rigid rules feels like paying a junior developer $200/month and spending $800 managing him.
At some point, it stops making sense.
A New Mental Model
One ugly, rain-soaked afternoon, an analogy crystallized everything:
- AI Agent → the car
- Model → the engine + control system
- You → the driver
- Codebase → the road + signs + traffic lights
When your road system is clean:
- the car moves naturally
- the driver doesn’t micro-manage
- fewer instructions are needed
- the engine performs predictably
You don’t tell the car when to stop at a red light.
You design the city so it knows.
Poorly structured code is like a city without signs:
everything becomes guesswork even for humans.
That’s when it became clear:
The codebase – not the prompt – is the real UI for AI.
Source Code as Documentation (For AI, Not Humans)
“Source code as documentation” isn’t new.
But its value becomes exponentially higher in the AI era.
Your codebase must be:
- readable
- explorable
- self-describing
- structured for both humans and machines
Here’s what worked consistently across models.
1. Keep Files Small
150–250 lines is the sweet spot.
It reduces cognitive overhead for both humans and machines.
2. Use Descriptive Filenames (AI Reads Them First)
Examples:
update.go →
change_product_quantity_when_order_placed.go
Looks verbose?
Yes.
Does AI instantly understand it?
Also yes.
Filenames are the model’s first heuristic.
3. Add a 10-Line Summary at the Top
This functions like a “mini SKILL file”:
- purpose
- inputs
- outputs
- main flow
- side effects (if any)
4. Each Module Gets a Short .md Guide
Not a full wiki.
Just one simple index:
- responsibilities
- main flows
- key files
- escalation points
This mirrors how Anthropic later designed their SKILL system.
AI doesn’t benefit from long documentation.
It benefits from intentionally placed breadcrumbs.
Agents Will Break Things – But That’s Part of the Loop
AI Agents will:
- generate messy files
- violate your conventions
- break flows
- create dead-end functions
It’s normal.
You can either:
- Fix immediately
- Or let entropy accumulate and refactor later with a stronger model
I occasionally open my “big guns” and let it reconstruct the architecture like city maintenance.
And guess what?
I have spare quota to do this.
Because my daily work consumes far fewer tokens.
The Moment Everything Clicks
Once your context design is solid:
Every model becomes usable.
Some fast.
Some slow.
Some reliable.
Some quirky.
But all competent because the environment is clean.
At that point, the hard problems shift upward:
- system design
- architecture
- trade-offs
- reasoning
Because once you hit the development phase…
Most of the work becomes auto-complete.
Final Thought
The real optimization in AI-assisted development is not:
- bigger prompts
- smarter rules
- more tools
- or complicated agent scripts
It is:
Designing a codebase that AI can navigate effortlessly.
When you engineer the context, not just the instructions, everything becomes:
– faster
– cheaper
– more predictable
– more resilient
– more model-agnostic
This may be the future of AI development:
Codebases crafted for humans – optimized for machines.
(This article is referenced from the experience of Mr. Viet Tran)
