156页!NLPCC 2020《预训练语言模型回顾》讲义下载
声明:本文转载自 哈工大讯飞联合实验室 公众号
哈工大讯飞联合实验室(HFL)资深级研究员、研究主管崔一鸣受邀在NLPCC 2020会议做题为《Revisiting Pre-trained Models for Natural Language Processing》的讲习班报告(Tutorial),介绍了预训练语言模型的发展历程以及近期的研究热点。本期推送文末提供了报告的下载方式。
NLPCC 2020 Tutorials:
http://tcci.ccf.org.cn/conference/2020/tutorials.php
报告信息
Title: Revisiting Pre-Trained Models for Natural Language Processing
Abstract : Pre-Trained Language Models (PLM) have become fundamental elements in recent research of natural language processing. In this tutorial, we will revisit the technical progress of the text representations, i.e., from one-hot embedding to the recent PLMs. We will describe several popular PLMs (such as BERT, XLNet, RoBERTa, ALBERT, ELECTRA, etc.) with their technical details and utilizations. On the other hand, we will also introduce various efforts on Chinese PLMs. At the end of this talk, we will analyze the shortcomings of the recent PLMs and envision the directions of future research.
报告目录
Introduction
Traditional Approaches for Text Representation
one-hot, word2vec, GloVe
Contextualized Language Models
CoVe, ELMo
Deep Contextualized Language Models
GPT, BERT, XLNet, RoBERTa, ALBERT, ELECTRA
Chinese Pre-trained Language Models
Chinese BERT-wwm, ERNIE, NEZHA, ZEN, MacBERT
Recent Research on PLM
Trending: GPT-2, GPT-3, T5
Distillation: DistilBERT, TinyBERT, MobileBERT, TextBrewer
Multi-lingual: mBERT, XLM, XLM-R
Summary