UM  > Faculty of Science and Technology
Affiliated with RCfalse
Status申請中Pending
A method and device for transfer learning
Xingjian Li1; Hang Hua2; Chengzhong Xu3; Dejing Dou4
2023-12
CountryChina; USA
Subtype发明专利Invention
Abstract

This invention introduces a method and device for fine-tuning multi-layer Transformer, which is a typical application of deep transfer learning. Multi-layer Transformer is a popular hierarchical architecture widely adopted in deep learning.  Specifically, this invention introduces a novel and effective regularization method to improve the fine-tuning process, referred to as layer-wise noise stability regularization. It adds random noise to the input and get the output deviations w.r.t. each layer of the multi-layer Transformer. These deviations are constrained during fine-tuning as a penalty term that will be minimized, in addition to the original ERM loss.

KeywordTransfer Learning Fine-tuning Regularization Noise Stability Transformer
Language中文Chinese
Document TypePatent
CollectionFaculty of Science and Technology
Affiliation1.University of Macau
2.Baidu Research
3.University of Macau
4.Baidu Research
First Author AffilicationUniversity of Macau
Recommended Citation
GB/T 7714
Xingjian Li,Hang Hua,Chengzhong Xu,et al. A method and device for transfer learning[P]. 2023-12-01.
APA Xingjian Li,Hang Hua,Chengzhong Xu,&Dejing Dou.A method and device for transfer learning.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Xingjian Li]'s Articles
[Hang Hua]'s Articles
[Chengzhong Xu]'s Articles
Baidu academic
Similar articles in Baidu academic
[Xingjian Li]'s Articles
[Hang Hua]'s Articles
[Chengzhong Xu]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Xingjian Li]'s Articles
[Hang Hua]'s Articles
[Chengzhong Xu]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.