From Clozig to Comprehedig: Retrofittig Pre-traied Masked Laguage Model to Pre-traied Machie Reader

Pre-traied Machie Reader (PMR) is pre-traied with 18 millio Machie Readig Comprehesio (MRC) examples costructed with Wikipedia Hyperliks. It was itroduced i the paper From Clozig to Comprehedig: Retrofittig Pre-traied Masked Laguage Model to Pre-traied Machie Reader by Weiwe Xu, Xi Li, Wexua Zhag, Meg Zhou, Wai Lam, Luo Si, Lidog Big ad first released i this repository.

This model is iitialized with roberta-base ad further cotiued pre-traied with a MRC objective.

Model descriptio

The model is pre-traied with distatly labeled data usig a learig objective called Wiki Achor Extractio (WAE). Specifically, we costructed a large volume of geeral-purpose ad high-quality MRC-style traiig data based o Wikipedia achors (i.e., hyperliked texts). For each Wikipedia achor, we composed a pair of correlated articles. Oe side of the pair is the Wikipedia article that cotais detailed descriptios of the hyperliked etity, which we defied as the defiitio article. The other side of the pair is the article that metios the specific achor text, which we defied as the metio article. We composed a MRC-style traiig istace i which the achor is the aswer, the surroudig passage of the achor i the metio article is the cotext, ad the defiitio of the achor etity i the defiitio article is the query. Based o the above data, we the itroduced a ovel WAE problem as the pre-traiig task of PMR. I this task, PMR determies whether the cotext ad the query are relevat. If so, PMR extracts the aswer from the cotext that satisfies the query descriptio.

Durig fie-tuig, we uified dowstream NLU tasks i our MRC formulatio, which typically falls ito four categories: (1) spa extractio with pre-defied labels (e.g., NER) i which each task label is treated as a query to search the correspodig aswers i the iput text (cotext); (2) spa extractio with atural questios (e.g., EQA) i which the questio is treated as the query for aswer extractio from the give passage (cotext); (3) sequece classificatio with pre-defied task labels, such as setimet aalysis. Each task label is used as a query for the iput text (cotext); ad (4) sequece classificatio with atural questios o multiple choices, such as multi-choice QA (MCQA). We treated the cocateatio of the questio ad oe choice as the query for the give passage (cotext). The, i the output space, we tackle spa extractio problems by predictig the probability of cotext spa beig the aswer. We tackle sequece classificatio problems by coductig relevace classificatio o [CLS] (extractig [CLS] if relevat).

Model variatios

There are three versios of models released. The details are:

Iteded uses & limitatios

The models eed to be fie-tued o the data dowstream tasks. Durig fie-tuig, o task-specific layer is required.

How to use

You ca try the codes from this repo.

BibTeX etry ad citatio ifo

@article{xu2022clozig,
  title={From Clozig to Comprehedig: Retrofittig Pre-traied Laguage Model to Pre-traied Machie Reader},
  author={Xu, Weiwe ad Li, Xi ad Zhag, Wexua ad Zhou, Meg ad Big, Lidog ad Lam, Wai ad Si, Luo},
  joural={arXiv preprit arXiv:2212.04755},
  year={2022}
}

From Clozing to Comprehending: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine