How to Reduce LLM API Token Costs in Production
Caching Math, Architecture, and Margin Protection
When building AI software, every API call to OpenAI, Anthropic, or Google costs real money. If your application relies on multi-agent execution loops, a single user click can easily trigger 10 to 20 separate model calls.
Without a deliberate caching strategy, your cloud bill will quickly erode your unit economics.
This research note breaks down how we structured semantic caching inside Exogram.ai to cut token spend by over 50% while dramatically improving response speed for end users.
Traditional web caching matches exact URLs or exact database keys, but AI users ask questions using different words for the exact same intent.
Semantic caching generates a vector representation of the incoming query and checks whether a mathematically similar query already exists inside the cache database.
Cosine similarity scores range from 0.0 to 1.0.
0.95+ Threshold: Extremely safe for factual, transactional queries, carrying virtually zero risk of serving an irrelevant answer.
0.88 - 0.92 Threshold: Good for general intent classification, though it requires ongoing monitoring for subtle nuance loss.
Rule: Start at 0.94 and tune downward based on real user feedback.
AI context changes over time, so you should never store semantic cache entries indefinitely.
Static Context (System Rules, FAQs): 30-day TTL.
Dynamic Context (User Career Profiles, Active Workflows): 1-hour to 24-hour TTL.
Real-time State Checks: Do not cache, forcing live deterministic evaluations every time.
During an automated attack or bot scrape, malicious traffic can trigger thousands of LLM API calls per minute. Placing Cloudflare Turnstile and strict edge rate-limiting in front of your caching layer ensures that bot traffic is dropped before touching either your cache database or your LLM endpoints.
By deploying semantic caching and edge pre-filtering across our runtime:
Average Cache Hit Rate: Settled at roughly 38% across routine agent execution loops.
Cost per 1,000 Requests: Dropped from $4.20 down to $1.85.
User-Perceived Speed: Cache hits feel instantaneous, consistently returning under 20ms.
This economic buffer is what allows us to run infrastructure for Exogram.ai and CareerWin.ai efficiently without sacrificing capability or performance.
Built In: Is Anything Standing Between Your AI Agent and Your Database? (https://builtin.com/articles/ai-agent-security-gates)
Systems Infrastructure: Learn how Exogram.ai enforces runtime security and how CareerWin.ai applies context systems to career intelligence.
Join the list to receive our newest posts straight to your inbox.
Enter your email to access the Product Debt Index Calculator, and join the private ledger where I break down the exact strategies bridging technical execution with financial viability for B2B SaaS.
Home
Archive
Login
Reset Password
Update Password
Profile
Search