UM  > Faculty of Science and Technology
Affiliated with RCfalse
A method and device for transfer learning
Xingjian Li1; Hang Hua2; Chengzhong Xu3; Dejing Dou4
CountryChina; USA

This invention introduces a method and device for fine-tuning multi-layer Transformer, which is a typical application of deep transfer learning. Multi-layer Transformer is a popular hierarchical architecture widely adopted in deep learning.  Specifically, this invention introduces a novel and effective regularization method to improve the fine-tuning process, referred to as layer-wise noise stability regularization. It adds random noise to the input and get the output deviations w.r.t. each layer of the multi-layer Transformer. These deviations are constrained during fine-tuning as a penalty term that will be minimized, in addition to the original ERM loss.

KeywordTransfer Learning Fine-tuning Regularization Noise Stability Transformer
Document TypePatent
CollectionFaculty of Science and Technology
Affiliation1.University of Macau
2.Baidu Research
3.University of Macau
4.Baidu Research
First Author AffilicationUniversity of Macau
Recommended Citation
GB/T 7714
Xingjian Li,Hang Hua,Chengzhong Xu,et al. A method and device for transfer learning[P]. 2023-12-01.
APA Xingjian Li,Hang Hua,Chengzhong Xu,&Dejing Dou.A method and device for transfer learning.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Xingjian Li]'s Articles
[Hang Hua]'s Articles
[Chengzhong Xu]'s Articles
Baidu academic
Similar articles in Baidu academic
[Xingjian Li]'s Articles
[Hang Hua]'s Articles
[Chengzhong Xu]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Xingjian Li]'s Articles
[Hang Hua]'s Articles
[Chengzhong Xu]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.