UM
Status即將出版Forthcoming
Modeling recurrence for transformer
Hao, Jie1; Wang, Xing2; Yang, Baosong3; Wang, Longyue2; Zhang, Jinfeng1; Tu, Zhaopeng2
2019
Source PublicationNAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume1
Pages1198-1207
AbstractRecently, the Transformer model (Vaswani et al., 2017) that is based solely on attention mechanisms, has advanced the state-of-the-art on various machine translation tasks. However, recent studies reveal that the lack of recurrence hinders its further improvement of translation capacity (Chen et al., 2018; Dehghani et al., 2019). In response to this problem, we propose to directly model recurrence for Transformer with an additional recurrence encoder. In addition to the standard recurrent neural network, we introduce a novel attentive recurrent network to leverage the strengths of both attention and recurrent networks. Experimental results on the widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness of the proposed approach. Our studies also reveal that the proposed model benefits from a short-cut that bridges the source and target sequences with a single recurrent layer, which outperforms its deep counterpart.
URLView the original
Language英語English
Scopus ID2-s2.0-85084045628
Fulltext Access
Citation statistics
Document TypeConference paper
CollectionUniversity of Macau
Affiliation1.Florida State University, United States
2.Tencent AI Lab,
3.University of Macau, Macao
Recommended Citation
GB/T 7714
Hao, Jie,Wang, Xing,Yang, Baosong,et al. Modeling recurrence for transformer[C],2019:1198-1207.
APA Hao, Jie,Wang, Xing,Yang, Baosong,Wang, Longyue,Zhang, Jinfeng,&Tu, Zhaopeng.(2019).Modeling recurrence for transformer.NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference,1,1198-1207.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Hao, Jie]'s Articles
[Wang, Xing]'s Articles
[Yang, Baosong]'s Articles
Baidu academic
Similar articles in Baidu academic
[Hao, Jie]'s Articles
[Wang, Xing]'s Articles
[Yang, Baosong]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Hao, Jie]'s Articles
[Wang, Xing]'s Articles
[Yang, Baosong]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.