ConSERT

yym6472/ConSERT is ranked 17,435 among these 8 peers; woodrush/neural-art-tf is ranked 17,434.

ConSERT Code for our ACL 2021 paper - ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer Requirements To install apex, run: Get Started English STS Tasks 1. bert-base-uncased) to folder from HuggingFace's Library 2. Download STS datasets to folder by running . The script is modified from SentEval toolkit 3. Run the scripts in the folder to reproduce our experiments. For example, run the following script to train unsupervised consert-base: *Update*: Recently we update the support for

RoBERTa model. Download the pre-trained language model (e. chinese-roberta-wwm-ext) to folder from HuggingFace's Library 2. Download the chinese STS datasets to folder by running 3. Run the scripts in the folder to train models for chinese STS tasks. For example, run the following script to train *base* model for task: The option in is used to select which chinese STS dataset to use. Pre-trained Models & Results English STS Tasks | ID | Model | STS12 | STS13 | STS14 | STS15

| STS16 | STSb | SICK-R | Avg. 86 | | - | bert-large-uncased (baseline) | 33. All the *base* models are trained from and the *large* models are trained from . For the unsupervised transfer, we merge all unlabeled texts from 7 STS datasets (STS12-16, STSbenchmark and SICK-Relatedness) as the training data (total 89192 sentences), and use the STSbenchmark dev split (including 1500 human-annotated sentence pairs) to select the best checkpoint. The sentence representations are obtained by averaging the token embeddings at the

last two layers of BERT. For model 2 to 10, we re-trained them on a single GeForce RTX 3090 with pytorch 1. 0 in our initial experiments) and changed the from 64 to 40 to reduce the required GPU memory (only for *large* models). Consequently, the results shown here may be slightly different from those reported in our paper. 86 | | - | chinese-roberta-wwm-ext-large (baseline) | 13. All the *base* models are trained from and the *large* models are

trained from . For each model, we train it on *single* chinese STS dataset but evaluate it on all 5 datasets. **The bold numbers** indicate that the model is evaluated on the same dataset it trained on. The sentence representations are also obtained by averaging the token embeddings at the last two layers of BERT. For *base* models, we set to 96 and to 64, while for *large* models, we set to 32 and to 40 to reduce the required GPU

memory.