1. Token Compression Efficiency Matrix (22 Datasets)
Evaluating OpenAI Tiktoken (cl100k_base and o200k_base) BPE token counts for Raw JSON vs Contex Compact
text (@schema, @dict, @enum, @time).
| Dataset & Payload Schema | Category | Records | Raw JSON Tokens | Contex Compact | Token Cut (%) | WASM Speed | Parity |
|---|---|---|---|---|---|---|---|
| synthetic_flat_100 Flat relational table dump (100 rows) | SQL | 100 | 6,302 | 1,566 | 75.1% | 149 μs | ✓ 100% |
| synthetic_flat_1k Large flat database query output (1,000 rows) | SQL | 1,000 | 63,002 | 15,184 | 75.9% | 1.20 ms | ✓ 100% |
| synthetic_flat_10k Enterprise batch database export (10,000 rows) | SQL | 10,000 | 630,020 | 148,200 | 76.5% | 9.40 ms | ✓ 100% |
| hr_candidate_profiles RealWorld HR candidate skill matrix & experience | RAG | 100 | 9,050 | 2,172 | 76.0% | 380 μs | ✓ 100% |
| it_support_tickets_nested Deeply nested IT ticket metadata & history | Nested | 100 | 12,400 | 2,850 | 77.0% | 420 μs | ✓ 100% |
| ecommerce_product_catalog E-commerce search results with variant arrays | RAG | 250 | 18,920 | 4,010 | 78.8% | 510 μs | ✓ 100% |
| financial_transaction_ledger Bank transaction audit records & timestamps | SQL | 500 | 34,500 | 7,120 | 79.3% | 680 μs | ✓ 100% |
| customer_chat_history_50 Multi-turn context inheritance conversational loop | Chat | 50 turns | 145,000 | 5,800 | 96.0% | 820 μs | ✓ 100% |
| pinecone_vector_rag_payload Top-20 Pinecone hybrid vector search JSON chunks | RAG | 20 chunks | 8,420 | 1,760 | 79.1% | 290 μs | ✓ 100% |
| weaviate_chunk_metadata Weaviate document chunk metadata & score vectors | RAG | 50 chunks | 16,200 | 3,380 | 79.1% | 440 μs | ✓ 100% |
| real_estate_listings Property search records & geospatial coordinates | RAG | 150 | 14,350 | 3,080 | 78.5% | 410 μs | ✓ 100% |
| healthcare_ehr_records Electronic health records & lab test encounters | Nested | 75 | 11,800 | 2,490 | 78.8% | 390 μs | ✓ 100% |
| github_pr_comments_diff GitHub PR review comments & code diff hunks | Nested | 40 diffs | 22,400 | 4,810 | 78.5% | 620 μs | ✓ 100% |
| logistics_shipment_events Supply chain shipment events & GPS waypoints | Nested | 300 | 21,600 | 4,420 | 79.5% | 590 μs | ✓ 100% |
| saas_billing_events Multi-tenant usage metering & billing logs | SQL | 400 | 28,900 | 5,840 | 79.7% | 630 μs | ✓ 100% |
| cyber_threat_intel_ioc STIX 2.1 Cyber Threat Intelligence Indicators | Nested | 120 IOCs | 15,600 | 3,240 | 79.2% | 460 μs | ✓ 100% |
| iot_sensor_telemetry_stream High-frequency industrial IoT sensor readings | SQL | 800 | 42,000 | 8,320 | 80.1% | 890 μs | ✓ 100% |
| codebase_ast_symbols TypeScript AST symbol export & function calls | Nested | 200 symbols | 17,800 | 3,720 | 79.1% | 520 μs | ✓ 100% |
| knowledge_graph_triples Wikidata RDF knowledge graph entity triples | RAG | 350 triples | 19,400 | 3,980 | 79.4% | 540 μs | ✓ 100% |
| hotel_booking_availability Global hotel room rates, dates & amenity arrays | RAG | 180 | 16,900 | 3,450 | 79.5% | 480 μs | ✓ 100% |
| academic_paper_abstracts PubMed clinical trial abstracts & MeSH terms | 128K | 50 papers | 94,000 | 19,800 | 78.9% | 1.85 ms | ✓ 100% |
| k8s_audit_event_logs Kubernetes control plane API audit stream (128K) | 128K | 500 events | 112,000 | 22,500 | 79.9% | 2.10 ms | ✓ 100% |
2. Needle-in-a-Haystack Factual Retrieval Accuracy
Fact retrieval and reasoning accuracy evaluated across 5 top LLM model families at 8K, 32K, 64K, and 128K token depths comparing Raw JSON vs Contex Compact context.
Claude 3.5 Sonnet
100.0%
Factual Accuracy (128K)
GPT-4o / o3-mini
100.0%
Factual Accuracy (128K)
Gemini 2.0 Flash
100.0%
Factual Accuracy (128K)
DeepSeek R1 / V3
100.0%
Factual Accuracy (64K)
Llama 3.1 70B
100.0%
Factual Accuracy (32K)
| Evaluated LLM Model | Task Benchmark Type | Context Depth | Raw JSON Accuracy | Contex Accuracy | Parity Result |
|---|---|---|---|---|---|
| Anthropic Claude 3.5 Sonnet | Multi-hop Variable Relation Extraction | 128,000 tokens | 99.8% | 100.0% | ✓ Exact Parity |
| OpenAI GPT-4o (2026) | Single Ticket Factual Passkey Lookup | 128,000 tokens | 100.0% | 100.0% | ✓ Exact Parity |
| Google Gemini 2.0 Flash | Deeply Nested Attribute Query | 128,000 tokens | 100.0% | 100.0% | ✓ Exact Parity |
| DeepSeek R1 (Reasoning) | Structured Data Deduplication & Math | 64,000 tokens | 99.9% | 100.0% | ✓ Exact Parity |
| Meta Llama 3.1 70B Instruct | Needle-in-a-Haystack Passkey Retrieval | 32,000 tokens | 100.0% | 100.0% | ✓ Exact Parity |
3. WASM Rust Codec Latency & Throughput Benchmark
In-process Rust WebAssembly engine (@tens-lab/wasm) microsecond latency distribution compared to
native language parsers.
| Encoder / Parser Language Core | p50 Latency | p95 Latency | p99 Latency | Throughput (MB/s) | Memory Footprint |
|---|---|---|---|---|---|
TensLab WASM Rust Engine (@tens-lab/wasm) |
120 μs | 380 μs | 790 μs | 420 MB/s | 1.8 MB Heap |
Node.js Native JSON.stringify() |
85 μs | 290 μs | 610 μs | 310 MB/s | 4.2 MB Heap |
Python Standard json.dumps() |
480 μs | 1.40 ms | 3.20 ms | 85 MB/s | 8.4 MB Heap |
Go Standard json.Marshal() |
190 μs | 520 μs | 1.10 ms | 240 MB/s | 2.6 MB Heap |
4. Financial & API Cost Impact Matrix
Calculated monthly API spend reduction for 1 Million RAG context queries across major LLM provider pricing tiers.
| Model Provider & Pricing Tier | Input Price / 1M Tokens | Raw JSON Cost (1M Calls) | Contex Cost (1M Calls) | Monthly Dollar Savings |
|---|---|---|---|---|
| Claude 3.5 Sonnet (Anthropic) | $3.00 / 1M | $27,150 | $6,516 | $20,634 / mo |
| OpenAI GPT-4o (OpenAI) | $2.50 / 1M | $22,625 | $5,430 | $17,195 / mo |
| OpenAI o3-mini (Reasoning) | $1.10 / 1M | $9,955 | $2,389 | $7,566 / mo |
| DeepSeek R1 (Reasoning) | $0.55 / 1M | $4,977 | $1,194 | $3,783 / mo |
| Google Gemini 2.0 Flash | $0.10 / 1M | $905 | $217 | $688 / mo |
Reproduce Benchmarks Locally via TensLab CLI
npx @tens-lab/cli benchmark --all --dataset synthetic_flat_100 --tokenizer
cl100k_base