Journal
ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY
Volume 13, Issue 4, Pages -Publisher
ASSOC COMPUTING MACHINERY
DOI: 10.1145/3510033
Keywords
Federated learning; pre-training; BERT; NLP
Funding
- National Natural Science Foundation of China [62102157]
- Fundamental Research Funds for the Central Universities [HUST:2020kfyXJJS019]
Ask authors/readers for more resources
The fast growth of pre-trained models has revolutionized natural language processing, becoming the dominant technique for various NLP applications. However, pre-training requires significant training data and computing resources, making it impossible for individual clients to conduct. To enable clients with limited computing capability to participate in pre-training large models, FEDBERT proposes a federated learning approach that achieves excellent performance without sharing raw data.
The fast growth of pre-trained models (PTMs) has brought natural language processing to a new era, which has become a dominant technique for various natural language processing (NLP) applications. Every user can download the weights of PTMs, then fine-tune the weights for a task on the local side. However, the pre-training of a model relies heavily on accessing a large-scale of training data and requires a vast amount of computing resources. These strict requirements make it impossible for any single client to pre-train such a model. To grant clients with limited computing capability to participate in pre-training a large model, we propose a new learning approach, FEDBERT, that takes advantage of the federated learning and split learning approaches, resorting to pre-training BERT in a federated way. FEDBERT can prevent sharing the raw data information and obtain excellent performance. Extensive experiments on seven GLUE tasks demonstrate that FEDBERT can maintain its effectiveness without communicating to the sensitive local data of clients.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available