What Is Going On
Startups pivot on economics. Several high-profile AI startups are pivoting away from model-heavy products. The unit economics don't work when you're paying $0.01+ per inference and customers expect free trials.
Cloud providers race. Cloud providers are racing to offer inference optimization. AWS Inferentia, Google TPUs, and Azure's custom silicon all promise cost savings—but lock-in concerns are real.
Small models gain credibility. The 'small model' movement is gaining credibility. For many tasks, a well-tuned 7B model beats GPT-4 at 1% of the cost. The catch: 'well-tuned' requires expertise most companies lack.
AI Corner
Quantization advances. Quantization techniques have improved dramatically. 4-bit models now match 16-bit performance for most tasks. This changes what's possible on consumer hardware.
Speculative decoding ships. Speculative decoding is moving from research to production. The idea: use a small model to draft, a large model to verify. Speed gains of 2-3x are common.
News You Can Use
- ●
Together AI raises big. Together AI raised at a $1B+ valuation on the promise of efficient inference.
- ●
NVIDIA announces next-gen. NVIDIA announced next-gen inference chips coming in 2025. The GPU shortage may finally ease.
- ●
Open source serves. Several open-source projects are tackling model serving. Vllm and TGI are becoming production-ready.