Generative pre-trained large language models (LLMs) have demonstrated impressive performance over a wide range of tasks, thanks to the unprecedented amount of data they have been trained on. As established scaling laws indicate, LLMs' future performance improvement depends on the amount of computing and data sources we can leverage for pre-training. Federated learning (FL) has the potential to unleash the majority of the planet's data and computational resources, which are underutilized by the data-center-focused training methodology of current LLM practice. Our work presents a robust, flexible, reproducible FL approach that enables large-scale collaboration across institutions to train LLMs. This would mobilize more computational and data resources while matching or potentially exceeding centralized performance. We further show the effectiveness of the federated training scales with model size and present our approach for training a billion-scale federated LLM using limited resources. This will help data-rich actors to become the protagonists of LLMs pre-training instead of leaving the stage to compute-rich actors alone.

Also See: Worldwide Federated Training Of Language Models

Claude’s Summary:

The two papers, “Worldwide Federated Training of Language Models” by Iacob et al. and “The Future of Large Language Model Pre-training is Federated” by Sani et al., both propose using federated learning (FL) as a new paradigm for pre-training large language models (LLMs). The main ideas are:

  1. FL allows leveraging more data and compute resources from multiple organizations around the world, while keeping the data decentralized and private. This can enable training larger LLMs on more diverse data compared to centralized training.

  2. FL relaxes synchronization requirements and reduces communication overheads compared to data-parallel distributed training, making it feasible for geographically distributed participants with varying hardware and connectivity.

  3. The papers present systems and algorithms for enabling efficient federated pre-training of LLMs at billion-parameter scales. Key techniques include allowing participants to modulate their amount of local training based on resource constraints, and partially personalizing models to clusters of participants with related data.

  4. Experimental results show federated LLM pre-training can match or exceed centralized training performance, with the performance gap narrowing as model size increases to billions of parameters. Larger federated models also converge faster and are more robust.

  5. Challenges include data and hardware heterogeneity across participants. The papers propose techniques like adaptive aggregation and load balancing to mitigate these issues.

In summary, the papers argue federated learning is a promising new direction for democratizing LLM pre-training by allowing many more organizations to collaboratively train large models on their combined data and compute resources. Let me know if you would like me to expand on any part of the summary or papers in more detail.

  • General_Effort@lemmy.worldEnglish
    0·
    5 months ago

    As far as I know, federated learning is pretty much dead. The point would be that it allows organizations to create a joint model without sharing data. But it doesn’t look like anyone who doesn’t want to share data wants to share a model.

    • Hackworth@lemmy.worldOPEnglish
      0·
      5 months ago

      Until they can distribute the training load of large models to consumer graphics cards (and do something like SETI@Home) it does seem like the benefit of distributed training isn’t enough to overcome the friction.

  • Audrey0nne@leminal.spaceEnglish
    0·
    5 months ago

    Lot of words just to say that once the advertisers move in on a centralized platform its value is shot. A huge part of the reason I abandoned the last platform I was using and sought a federated alternative.