Semantic Caching & Edge Filtering Engine

When running automated execution loops inside Exogram, token spend can scale rapidly if frontier models process routine logic that does not require complex reasoning. The Semantic Caching & Edge Filtering Engine places deterministic code filters and vector similarity matching in front of LLM endpoints.

Sub-Millisecond Edge Filtering

Sub-millisecond edge filters evaluate incoming requests for deduplication, static rules, and simple code-based routing. Routine interactions are answered directly with traditional code without calling generative model APIs.

Vector Semantic Caching

Requests passing the edge filter are checked against a high-performance vector similarity cache. Prompts matching historical intent thresholds (e.g. 0.92+ cosine similarity) are served instantly from local storage at near-zero cost. Only true cache misses escalate to frontier LLM inference.

import { ExogramEdgeFilter } from "@exogram/sdk";

const response = await ExogramEdgeFilter.execute({
  prompt,
  cacheThreshold: 0.92,
  fallbackModel: "gpt-4o"
});
// Cut runtime API spend by 50%+ with zero quality loss

Economic Impact

• Runtime API Spend Cut: >50%
• Cache Hit Latency: <15ms
• Response Quality Loss: 0%