feat: async batch inference benchmark for Crusoe Managed Inference

[Sakshi3027]

What this adds

Async batch inference with a sequential vs parallel vs batched benchmark on Crusoe Managed Inference.

Mode	Time	Speedup
Sequential	3.96s	baseline
Parallel	0.57s	7.0x faster
Batched (4 at a time)	1.29s	3.1x faster

Benchmark: 8 prompts on Llama-3.3-70B-Instruct.

Why it's useful

Production LLM applications rarely run one prompt at a time evaluation pipelines, batch scoring, and multi-user systems all need concurrent inference. This shows exactly how to use asyncio.gather with ChatCrusoe to get 7x throughput gains, and how to add batch size control for rate-limit-sensitive workloads.

Testing

Tested locally with Groq as a drop-in replacement. Numbers above are from a local run.

To run on Crusoe:
export CRUSOE_API_KEY="your-api-key"
python batch.py

Related contributions

• feat: LangGraph multi-node research agent on Crusoe Managed Inference #55 — LangGraph multi-node agent
• feat: MLflow experiment tracking for Crusoe Managed Inference #56 — MLflow experiment tracking
• feat: RAG pipeline with Qdrant and sentence-transformers on Crusoe Managed Inference #57 — RAG pipeline with Qdrant
• feat: structured output and tool calling examples for Crusoe Managed Inference #58 — Structured output and tool calling
• feat: real-time streaming output examples for Crusoe Managed Inference #59 — Real-time streaming output