Topic: speculative decoding

Charles Frye of Modal examines the infrastructure constraints behind large-scale AI research. With Ravid Shwartz Ziv and Allen Roush, he discusses underused hardware, saturated resources, sharing GPU capacity and deciding when to train a model. The conversation also covers inference and speculative decoding, hardware economics, virtualization and the security boundaries around agents and open models. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Fireworks AI co-founder Benny Chen explains the infrastructure required to serve machine-learning models at scale, from GPU procurement and kernels to runtimes and request routing. He discusses power constraints, hardware utilization, mixture-of-experts models, quantization, reinforcement-learning efficiency and speculative decoding. The conversation also covers distributed training across data centres, sparse autoencoders, the constraints on open-source interpretability research, and the relationship between open models and compute availability. Hosted by Ravid Shwartz Ziv and Allen Roush; th