Notes

• Traditional gossip-based methods suffer from communication costs that grow with model size.
• SeedFlood exploits the seed-reconstructible structure of zeroth-order updates, making messages near-zero in size.
• This makes communication overhead negligible and independent of model size.
• Enables training previously impractical regimes, like billion-parameter models across hundreds of clients.
• Experiments show SeedFlood outperforms gossip baselines in generalization and communication efficiency for decentralized LLM fine-tuning.
• Achieves results comparable to first-order methods in large-scale settings.

• 传统基于Gossip的方法通信成本随模型大小增长，成为扩展瓶颈。
• SeedFlood利用零阶更新的种子可重构结构，使消息大小近乎为零。
• 该方法使通信开销可忽略且独立于模型大小。
• SeedFlood支持在数百个客户端上训练十亿参数模型等先前不切实际的场景。
• 实验表明，在去中心化LLM微调中，SeedFlood在泛化性能和通信效率上均优于Gossip基线。
• 在大规模设置中，甚至取得了与一阶方法相当的结果。