Progress on provable inference for large language models
When evaluating the protocol on a Llama 4 variant:
per GPU, proving a batch of 24 requests takes about 72 times longer than vLLM takes to process the same 24 requests… and about 94 times longer when vLLM runs at the highest load we tried.
Anthropic and research collaborators open-sourced the library: FlashProve.
[A]t some point this doesn’t matter. The point is cost for the capability. If you want verifiable LLM results, on a non-local model, and can’t use a TEE, is 100x overhead acceptable?