Semafold: smaller vectors, measurable trade-offs | Mindtro

Semafold is Mindtro’s open-source Python library for compressing the numerical representations used by AI systems. It helps teams reduce embedding storage and explore smaller attention-cache tensors, with the saved bytes and reconstruction error visible together.

Synthetic benchmark · source report 0.1.0, 31 March 2026 · library reviewed at 0.2.0

Research question

How much space can AI vectors save when the complete compressed representation, including metadata, is counted?

Method

The source report uses synthetic data: 128 vectors with 1,536 values each, and pairs of key/value tensors shaped (4, 8, 256, 128). It compares 2-, 3- and 4-bit vector encodings with dense float32 and 16-bit storage. Totals include compressed data, auxiliary values and metadata; quality is assessed using reconstruction and similarity errors.

Results

At 3 bits, the embedding representation falls from 786,432 to 74,738 bytes: 90.50% less space than float32, or 80.99% less than 16-bit storage. The representative key/value pair falls from 8,388,608 to 885,734 bytes. These are storage measurements on synthetic inputs, not measured gains in application speed or answer quality.

Where it fits

Embedding archives, vector memory for agents and custom retrieval pipelines are practical integration points. A separate preview handles the key and value tensors used by a model’s attention mechanism. Semafold compresses numbers; it does not summarise documents or shorten prompts.

Choose the size–quality balance

In the reconstruction benchmark, 2-bit encoding uses 50,077 bytes with mean squared error 0.1178; 3-bit uses 74,738 bytes with error 0.0348; 4-bit uses 99,484 bytes with error 0.0098. More bits preserve the original values more closely. These errors do not measure search recall or response correctness.

A Python library with explicit boundaries

Python 3.10+ and NumPy are sufficient for the core. Version 0.2.0 adds optional acceleration through PyTorch (CUDA/MPS) or MLX (Metal). The documented root API provides typed encode/decode contracts and byte accounting. TurboQuant codecs and the key/value API use preview imports whose compatibility is not yet guaranteed. GPU speedups were not measured in this report.

Limitations

Compression is lossy. Real retrieval quality and end-to-end model behaviour need testing on the intended workload. Very small cache blocks can grow relative to 16-bit storage because of metadata overhead. The compression and cache APIs remain preview interfaces; ready-made serving integrations are outside this report.

About this report

This technical report is not peer reviewed. Figures were transcribed from the repository’s 0.1.0 benchmark report and were not rerun for this website. Code, examples, benchmark runners and the API stability policy are public under Apache-2.0.

Technical notes and source records

Synthetic benchmark · source report 0.1.0, 31 March 2026 · library reviewed at 0.2.0

The file names and SHA-256 fingerprints identify the exact source versions used to prepare this report.

benchmarks/turboquant_benchmark_report.mdSHA-256: ef384c74195d3f0944b36b1d2cf75baa78ca77e6e29009faa1f08ed3d648f1ad
STABILITY.mdSHA-256: f085264ec4f6c0a0b3e962e8cccea470fd6e11fdb501f3492b4b4865ae6885e7
CHANGELOG.mdSHA-256: b52cccda6dc65f2d5f8de5f31d7c44a634d1250dcd61c5b1cd1c5ebd46b603f6
pyproject.tomlSHA-256: b5ed5776e2bdde03a2ff6b62176ed687993b9a7bc39e4e74721f49c8763950ce

Request the research materials →