- NEZHA: Neural Contextualized Representation for Chinese Language Understanding [-] Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, Qun Liu.
- Bert in chinese corpus
- change position -> relative position
- using jieba to do the word segmentation
- using mixing precision traning + LAMB optimizer