TensLab Developer Docs

Introduction

TensLab (Contex Compiler) is a token-aware structural context compiler for AI applications and Retrieval-Augmented Generation (RAG) pipelines. It intercepts raw structured data (JSON search results, database rows, IT tickets, candidate records, e-commerce orders) before it reaches LLM APIs.

By stripping structural JSON syntax noise ({}, [], "key":, quotes, colons, commas) into single-declaration schemas (@schema, @dict, @enum, @time), Contex slashes prompt tiktoken spend by 76% median (up to 96% in multi-turn loops) while preserving 100% factual retrieval recall.

The Dual-Format Engine Pipeline

TensLab operates a two-layer context format architecture designed for standard LLM REST text APIs and local WASM binary persistence:

Monorepo Package Suite

The TensLab monorepo consists of 5 modular TypeScript & Rust packages under the @tens-lab/* scope:

Core Compiler API (@tens-lab/core)

The compile() function is the primary entry point for compiling structured RAG datasets into Contex Compact format.

import { compile } from '@tens-lab/core'

// 1. Raw RAG dataset (e.g. IT support tickets returned from vector database)
const rawTickets = [
  { id: 1, title: 'Login page unresponsive', status: 'open', priority: 'critical', assignee: 'alice' },
  { id: 2, title: 'API rate limiting bug', status: 'in_progress', priority: 'high', assignee: 'bob' }
]

// 2. Compile into Contex Compact format for GPT-4o tiktoken optimization
const compactPrompt = compile(rawTickets, {
  model: 'gpt-4o',
  format: 'contex'
})

console.log(compactPrompt)
// Output:
// @dict 0:critical 1:alice 2:bob
// @schema id  title  status  priority  assignee
// 1  Login page unresponsive  open  @0  @1
// 2  API rate limiting bug  in_progress  high  @2

12-Layer Adaptive Codec Pipeline

Contex automatically analyzes data shapes in real time and applies optimal codec combinations:

Contex Compact Format Grammar

The Contex Compact format follows a strict, token-optimized grammar designed for LLM text tokenizers:

# Contex Compact Spec Grammar v3.1
@time  <column_name>=<iso_timestamp_anchor>
@enum  <column_name>: A=<val_1> B=<val_2> C=<val_3>
@repeat <column_name>=<val> x<count> <delta_diffs>
@dict  0:<frequent_val_0> 1:<frequent_val_1> 2:<frequent_val_2>
@schema <key_1> <key_2> <key_3> ... <key_N>
<val_11> <val_12> <val_13> ... <val_1N>
<val_21> <val_22> <val_23> ... <val_2N>

Drop-in Middleware (@tens-lab/middleware)

Install and wrap standard OpenAI, Anthropic, or Gemini SDK initialization without changing your application code:

npm install @tens-lab/middleware @tens-lab/core

import createClient from '@tens-lab/middleware'

const client = createClient({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  compress: true, // Enables automatic WASM Contex compilation
})

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'user', content: 'Your raw RAG context here' }
  ]
})

Rust WASM Core Engine (@tens-lab/wasm)

High-speed Rust WebAssembly compilation layer (contex.rs) enabling 149μs per-row binary serialization:

use contex_core::TensEncoder;

pub fn encode_rag_payload(data: &str) -> Result<Vec<u8>, TensError> {
    let encoder = TensEncoder::new();
    let binary_stream = encoder.encode_json(data)?;
    Ok(binary_stream)
}

CLI Terminal Doctor (@tens-lab/cli)

Run instant diagnostic token audits on your local prompt files:

npx @tens-lab/cli doctor ./data/sample_tickets.json

WASM Privacy Guarantee

Contex executes 100% locally inside your Node.js or browser process. Your data never touches TensLab servers or external proxies.

FAQ & Troubleshooting

Q: Does Contex Compact format affect reasoning accuracy?
A: No. In 100% Needle-in-a-Haystack benchmarks evaluated on Llama 3.1 70B and GPT-4o, Contex format achieved 100% exact factual recall parity.

Q: Which models are supported?
A: All text-based LLMs including GPT-4o, o3-mini, Claude 3.5 Sonnet, Gemini 2.0 Flash, DeepSeek R1, and Llama 3.