snips.md

Results

Train/Test with Complete • Test with Incomplete • Train/Test with Incomplete

Complete

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
EmbraceBERT-bs4+Dropout0.1	98.71	99.00	98.71	98.57	98.71	98.86	99.00	98.57	98.43	98.71	98.73	0.17
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	99.00	98.86	98.71	98.86	98.57	98.86	99.00	98.71	98.71	98.86	98.81	0.13
BERT-bs16+Dropout0.1	99.00	99.00	98.71	98.71	98.57	98.86	98.86	98.57	98.71	99.00	98.80	0.16
BERT-bs16+Dropout0.3	98.57	98.86	98.57	98.57	99.00	98.86	98.86	98.29	98.86	98.86	98.73	0.21
EmbraceBERT-bs16	98.71	98.71	98.86	98.71	98.29	98.71	98.43	98.86	98.86	98.57	98.67	0.18
EmbraceBERT-bs16+Dropout0.1	98.71	98.57	98.86	98.71	98.57	99.00	98.71	98.71	99.14	98.86	98.79	0.17
EmbraceBERT-bs16+Dropout0.3	98.29	98.86	98.71	98.71	98.29	98.86	98.71	98.86	98.86	99.00	98.71	0.23
EmbraceBERT-bs16+Branches
CondensedEmbraceBERT-bs16	98.57	98.71	98.57	98.71	98.86	98.71	98.86	98.71	98.86	99.00	98.76	0.13
CondensedEmbraceBERT-bs16+Dropout0.1	98.71	98.71	98.14	99.00	98.86	98.43	98.71	98.71	98.86	99.00	98.71	0.25
CondensedEmbraceBERT-bs16+Dropout0.3	98.43	98.86	98.00	98.57	98.57	98.57	98.57	98.86	98.86	98.29	98.56	0.26
CondensedEmbraceBERT-bs16+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	99.00	98.57	98.43	98.43	98.43	98.43	98.71	98.57	98.71	98.71	98.60	0.18
FrozenEBERT-bs16-ep100	98.86	97.71	99.14	98.71	98.57	98.57	98.71	98.43	98.86	98.14	98.57	0.38
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenEBERT-bs16-ep100+Branches
FrozenCEBERT-bs16-ep100	98.57	98.86	98.43	98.57	98.71	98.43	98.86	98.57	98.29	98.71	98.60	0.18
FrozenCEBERT-bs16-ep100+Dropout0.1	99.00	98.71	98.29	98.57	98.43	98.57	98.57	99.00	98.29	98.43	98.59	0.24
FrozenCEBERT-bs16-ep100+Dropout0.3	99.00	99.14	98.86	98.86	98.43	98.57	98.14	98.86	98.57	98.29	98.67	0.31
FrozenCEBERT-bs16-ep100+Branches	98.86	99.14	98.43	99.00	98.86	98.43	98.57	98.86	98.86	98.00	98.70	0.32
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	99.00	98.86	98.86	98.71	98.57	99.00	99.00	98.86	98.57	98.86	98.83	0.15
BERT-bs32+Dropout0.1	98.57	99.00	98.57	98.86	98.43	98.86	99.00	98.71	98.43	98.86	98.73	0.21
BERT-bs32+Dropout0.3	99.00	98.86	98.57	98.86	98.57	98.43	99.00	99.00	98.71	98.71	98.77	0.19
EmbraceBERT-bs32	98.86	98.71	98.71	98.71	98.57	98.71	98.86	98.71	98.57	98.86	98.73	0.10
EmbraceBERT-bs32+Dropout0.1	98.43	99.00	99.00	98.86	98.71	98.14	98.57	98.57	98.86	98.86	98.70	0.26
EmbraceBERT-bs32+Dropout0.3	99.14	99.00	98.86	98.71	98.43	98.71	98.57	98.86	98.86	99.00	98.81	0.20
EmbraceBERT-bs32+Branches
CondensedEmbraceBERT-bs32	98.86	99.14	98.71	99.00	98.71	98.86	99.00	98.71	98.57	98.86	98.84	0.16
CondensedEmbraceBERT-bs32+Dropout0.1	99.00	99.57	98.86	98.57	98.29	98.86	98.86	99.14	98.43	97.71	98.73	0.48
CondensedEmbraceBERT-bs32+Dropout0.3	98.71	98.86	98.86	98.86	98.86	98.86	98.57	98.71	98.57	98.29	98.71	0.18
CondensedEmbraceBERT-bs32+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	99.00	98.71	98.86	98.71	98.43	99.00	99.00	98.86	98.86	98.86	98.83	0.17
FrozenEBERT-bs32-ep100	98.86	98.71	98.57	98.71	98.43	98.29	98.57	98.43	98.57	98.43	98.56	0.16
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenEBERT-bs32-ep100+Branches
FrozenCEBERT-bs32-ep100	98.86	99.14	98.71	98.86	98.71	98.71	98.71	98.71	98.71	98.57	98.77	0.15
FrozenCEBERT-bs32-ep100+Dropout0.1	99.00	99.43	98.57	98.14	98.57	98.71	98.43	98.14	98.57	99.14	98.67	0.39
FrozenCEBERT-bs32-ep100+Dropout0.3	98.86	99.29	98.86	99.00	98.57	99.14	98.29	98.57	98.57	98.86	98.80	0.29
FrozenCEBERT-bs32-ep100+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	99.00	99.00	99.00	99.00	99.00	98.57	98.86	98.86	98.71	98.86	98.89	0.14
RoBERTa-bs16+Dropout0.1	99.00	98.86	98.71	99.14	98.86	98.86	99.00	98.43	98.71	98.86	98.84	0.19
RoBERTa-bs16+Dropout0.3	99.57	99.14	98.86	98.86	98.71	99.00	98.57	98.71	98.71	98.71	98.89	0.28
EmbraceRoBERTa-bs16	98.71	98.86	98.71	99.00	98.43	98.86	99.00	98.57	99.00	99.00	98.81	0.19
EmbraceRoBERTa-bs16+Dropout0.1	98.71	98.43	99.29	98.57	99.14	98.71	98.86	99.14	98.86	99.43	98.91	0.31
EmbraceRoBERTa-bs16+Dropout0.3	98.43	99.00	98.86	98.86	98.57	99.00	98.71	99.00	98.86	98.71	98.80	0.18
EmbraceRoBERTa-bs16+Branches
CondensedEmbraceRoBERTa-bs16	99.00	99.00	98.86	98.86	99.00	99.00	99.00	99.14	99.14	98.71	98.97	0.12
CondensedEmbraceRoBERTa-bs16+Dropout0.1	98.57	98.71	98.71	99.00	99.71	98.57	98.86	99.14	98.86	98.86	98.90	0.32
CondensedEmbraceRoBERTa-bs16+Dropout0.3	98.71	99.29	98.86	98.86	98.43	98.43	98.71	98.57	98.57	98.57	98.70	0.24
CondensedEmbraceRoBERTa-bs16+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	98.29	98.86	98.86	99.00	98.57	99.14	98.86	98.86	98.14	98.57	98.71	0.30
FrozenERoBERTa-bs16-ep100	97.86	98.86	98.71	98.86	98.86	99.00	98.57	98.71	98.71	98.86	98.70	0.30
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenERoBERTa-bs16-ep100+Branches
FrozenCERoBERTa-bs16-ep100	98.86	99.57	98.57	98.71	99.00	98.71	98.43	99.00	98.71	98.14	98.77	0.36
FrozenCERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100+Dropout0.3
FrozenCERoBERTa-bs16-ep100+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	99.00	99.00	99.00	99.14	99.14	98.86	99.00	99.14	99.00	98.86	99.01	0.10
RoBERTa-bs32+Dropout0.1	98.86	99.00	98.14	99.29	98.57	99.00	99.00	98.86	98.71	98.86	98.83	0.29
RoBERTa-bs32+Dropout0.3	99.43	99.14	98.57	99.14	99.00	99.00	99.29	98.71	99.00	98.86	99.01	0.24
EmbraceRoBERTa-bs32	98.86	98.71	98.86	98.86	98.86	99.00	99.14	98.86	99.00	99.00	98.91	0.11
EmbraceRoBERTa-bs32+Dropout0.1	98.86	98.86	99.00	99.00	98.57	98.71	98.71	98.57	99.14	99.00	98.84	0.19
EmbraceRoBERTa-bs32+Dropout0.3	99.14	98.86	99.00	98.86	98.86	99.00	99.14	99.00	99.00	98.86	98.97	0.11
EmbraceRoBERTa-bs32+Branches
CondensedEmbraceRoBERTa-bs32	99.14	99.00	99.00	99.00	99.00	99.57	99.00	98.86	99.00	98.86	99.04	0.19
CondensedEmbraceRoBERTa-bs32+Dropout0.1	98.86	99.00	98.86	98.43	99.14	99.00	98.86	99.29	99.00	99.71	99.01	0.32
CondensedEmbraceRoBERTa-bs32+Dropout0.3	99.00	98.57	98.57	99.00	98.71	98.71	98.71	99.00	99.00	98.71	98.80	0.17
CondensedEmbraceRoBERTa-bs32+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	99.14	99.00	98.57	98.71	99.00	99.00	98.86	98.57	98.29	98.71	98.79	0.25
FrozenERoBERTa-bs32-ep100	98.43	98.71	98.57	98.71	98.43	98.57	98.86	98.71	98.57	99.00	98.66	0.17
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenERoBERTa-bs32-ep100+Branches
FrozenCERoBERTa-bs32-ep100	98.86	98.71	98.43	99.00	99.29	98.86	99.00	99.00	98.86	99.00	98.90	0.21
FrozenCERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100+Dropout0.3
FrozenCERoBERTa-bs32-ep100+Branches

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16
BERT-bs16+Dropout0.1	98.86	98.57	98.86	98.86	98.86	99.00	98.86	98.71	98.71	98.71	98.80	0.11
EmbraceBERT-bs16-WRONG
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32
BERT-bs32+Dropout0.1	99.00	98.71	99.00	98.86	99.00	98.71	98.57	98.71	98.71	98.71	98.80	0.15
EmbraceBERT-bs32-WRONG
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16
RoBERTa-bs16+Dropout0.1	98.86	98.86	98.86	98.86	98.86	99.00	98.86	98.86	98.86	98.86	98.87	0.04
EmbraceRoBERTa-bs16-WRONG
EmbraceRoBERTa-bs16+Dropout0.3	99.00	98.71	99.00	98.71	98.86	98.86	98.71	99.00	98.57	99.14	98.86	0.17
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32
RoBERTa-bs32+Dropout0.1	99.00	99.00	98.71	98.57	99.00	98.71	98.86	98.57	99.14	99.00	98.86	0.19
EmbraceRoBERTa-bs32-WRONG
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	99.14	98.86	99.00	98.57	98.86	98.86	99.00	98.57	98.86	98.86	98.86	0.17

• Epoch 100 - BERT

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
LOCAL - BERT-bs32	99.00	98.71	98.57	98.57	98.43	98.57	98.86	98.71	98.57	98.71	98.67	0.16
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LOCAL - BERTwithAtt-bs32	98.57	98.71	98.71	98.86	98.86	98.71	98.86	99.00	99.00	98.71	98.80	0.13
LOCAL - BERTwithAttClsProjection-bs32	98.71	98.43	98.71	98.71	98.71	98.57	98.71	98.43	98.57	98.86	98.64	0.13
LOCAL - BERTwithProjection-bs32	98.71	99.14	98.71	98.86	98.57	99.00	98.71	99.00	99.00	99.00	98.87	0.17
LOCAL - BERTwithProjectionAtt-bs32	98.86	98.71	98.71	98.86	98.71	98.43	98.71	98.86	98.71	98.86	98.74	0.12
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS3 - EmbraceBERT-bs32	98.57	98.71	98.71	98.57	99.00	98.71	98.71	98.71	98.71	98.86	98.73	0.12
LDS3 - EmbraceBERT-bs32-p_attclsqw	98.86	98.86	98.71	99.00	98.86	99.00	98.43	98.86	98.71	98.57	98.79	0.17
LDS3 - EmbraceBERTwithProj-bs32	98.86	98.57	98.57	98.86	98.71	98.86	98.71	98.57	98.43	98.57	98.67	0.14
LDS3 - EmbraceBERTwithProj-bs32-p_attclsqw	99.00	98.71	99.00	98.43	98.57	98.57	98.57	98.71	98.57	98.86	98.70	0.19
LDS3 - EmbraceBERTconcatAtt-bs32	98.71	98.71	98.57	98.57	98.86	98.86	98.57	98.71	98.43	98.71	98.67	0.13
LDS3 - EmbraceBERTconcatAtt-bs32-p_attclsqw	98.71	98.86	98.29	98.57	98.57	98.86	98.86	98.86	98.57	98.86	98.70	0.19
LDS3 - EmbraceBERTconcatAttwithProj-bs32	98.86	98.71	99.14	98.43	98.43	98.86	98.86	98.86	98.71	98.14	98.70	0.27
LDS3 - EmbraceBERTconcatAttwithProj-bs32-p_attclsqw	98.71	98.57	98.71	98.86	98.43	98.43	98.71	98.71	98.43	98.86	98.64	0.16
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS3 - EmbraceBERT-KVQ-bs32	98.86	98.71	98.71	99.00	98.86	98.86	98.57	98.57	98.71	98.57	98.74	0.14
LDS3 - EmbraceBERT-KVQ-bs32-p_attclsqw	98.29	98.71	98.71	98.71	99.00	98.57	98.71	98.71	98.71	98.71	98.69	0.17
LDS3 - EmbraceBERT-KVQ-withProj-bs32	98.71	98.57	98.71	99.00	98.57	98.57	98.57	98.86	98.86	99.00	98.74	0.17
LDS3 - EmbraceBERT-KVQ-withProj-bs32-p_attclsqw	98.57	98.71	98.71	98.57	98.71	98.57	98.57	98.71	98.86	99.00	98.70	0.13
LDS3 - EmbraceBERTconcatAtt-KVQ-bs32	98.57	98.57	98.43	98.71	98.71	98.86	98.71	98.57	98.71	98.86	98.67	0.13
LDS3 - EmbraceBERTconcatAtt-KVQ-bs32-p_attclsqw	99.00	98.86	98.86	98.71	98.57	98.57	99.00	98.86	98.71	98.43	98.76	0.18
LOCAL - EmbraceBERTconcatAtt-KVQ-withProj-bs32	98.86	98.71	98.57	98.57	98.43	98.57	98.71	98.57	98.71	98.86	98.66	0.13
LOCAL - EmbraceBERTconcatAtt-KVQ-withProj-bs32-p_attclsqw	98.71	98.71	98.57	98.86	98.57	98.86	99.00	99.00	98.71	98.29	98.73	0.21
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16+Dropout0.1	98.71	98.86	98.71	98.57	98.86	98.71	98.71	98.71	98.86	99.00	98.77	0.11

• Epoch 100 - BERT bs48

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
LOCAL - BERT-bs48	98.57	98.71	98.57	98.57	98.57	98.71	98.86	98.86	98.57	98.71	98.67	0.11
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LOCAL - BERTwithAtt-bs48	98.57	98.71	98.57	98.86	98.86	98.86	98.43	98.57	98.86	98.86	98.71	0.16
LDS3 - BERTwithAttClsProjection-bs48	99.00	98.86	98.71	98.86	98.71	98.86	98.71	99.14	98.57	98.71	98.81	0.16
LOCAL - BERTwithProjection-bs48	98.86	98.71	98.57	98.43	99.00	98.86	98.71	98.86	98.71	99.00	98.77	0.17
LOCAL - BERTwithProjectionAtt-bs48	98.71	99.00	98.86	98.71	99.00	98.57	99.00	98.71	98.86	99.00	98.84	0.15
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS1 - EmbraceBERT-bs48	98.71	98.71	98.57	98.71	98.71	98.71	98.43	98.71	98.86	98.71	98.69	0.11
LDS1 - EmbraceBERT-bs48-p_att_clsquery_weights	98.86	98.71	98.86	98.86	98.86	98.86	98.71	98.86	98.57	98.86	98.80	0.09
LDS1 - EmbraceBERTwithProj-bs48	98.71	98.86	99.00	98.86	98.86	98.29	98.71	98.86	98.43	98.86	98.74	0.21
LDS1 - EmbraceBERTwithProj-bs48-p_attclsqw	99.00	99.00	98.86	98.86	98.71	98.57	98.71	98.86	98.71	98.71	98.80	0.13
LDS1 - EBERTconcatAtt-bs48	98.86	98.57	98.86	99.14	98.43	99.00	98.71	98.29	98.86	98.86	98.76	0.25
LDS1 - EBERTconcatAtt-bs48_attclsqw	98.57	98.71	99.00	98.57	98.86	98.86	98.86	99.00	98.71	98.71	98.79	0.15
LDS1 - EBERTconcatAttwithProj-bs48	98.57	98.71	98.86	98.43	98.71	98.86	98.71	98.86	98.86	98.86	98.74	0.14
LDS1 - EBERTconcatAttwithProj-bs48_attclsqw	98.71	98.71	98.86	98.57	98.71	98.86	98.86	98.57	98.71	98.86	98.74	0.11
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS1 - EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	98.57	98.57	98.86	98.86	98.86	98.86	98.57	98.86	98.29	98.86	98.71	0.19
LDS1 - EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.57	98.71	98.86	98.86	98.57	99.00	98.86	98.57	98.71	98.86	98.76	0.14
LDS1 - EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	98.71	98.86	98.57	98.71	98.71	99.14	98.29	98.71	98.71	98.71	98.71	0.20
LDS1 - EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.57	99.00	98.29	98.86	98.86	99.00	98.71	99.00	99.00	98.86	98.81	0.22
LDS1 - EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	99.00	98.57	99.00	98.86	98.71	98.71	98.86	98.86	98.86	98.71	98.81	0.13
LDS1 - EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	99.00	98.86	98.86	98.57	98.57	98.71	98.86	98.57	98.86	98.86	98.77	0.15
LDS1 - EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	98.71	98.86	98.86	98.86	98.57	99.14	98.71	98.71	98.57	99.00	98.80	0.17
LDS1 - EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.86	98.57	98.43	98.71	98.86	99.00	98.57	98.86	98.57	98.71	98.71	0.17

• Epoch 100 - RoBERTa

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
RoBERTa-bs16+Dropout0.1	98.86	98.86	98.71	98.86	98.43	98.86	98.86	98.86	98.71	98.57	98.76	0.14
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	98.86	98.57	99.29	99.00	98.71	98.86	98.71	99.00	98.86	99.29	98.91	0.22

STT Error: trained with COMPLETE data and tested with INCOMPLETE data

gtts-google - Bad (Only good with BERT bs32)

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	61.71	60.57	60.86	60.71	59.71	61.43	60.57	59.43	61.43	60.14	60.66	0.71
BERT-bs16+Dropout0.1	60.14	61.43	60.43	59.43	61.00	59.57	61.14	58.57	60.14	59.43	60.13	0.85
BERT-bs16+Dropout0.3	60.71	60.00	59.00	61.71	60.43	63.14	60.29	59.00	60.86	60.29	60.54	1.16
FrozenBERT-bs16-ep100	61.71	60.71	60.14	60.86	60.14	60.14	59.29	59.86	58.43	59.57	60.09	0.86
FrozenBERT-bs16-ep100+Dropout0.1	59.57	62.29	58.00	58.71	62.57	59.86	60.29	59.57	59.14	57.57	59.76	1.55
FrozenBERT-bs16-ep100+Dropout0.3	59.71	61.00	59.57	59.00	61.71	62.29	58.29	61.86	61.00	62.14	60.66	1.34
EmbraceBERT-bs16	62.86	59.29	60.14	59.14	57.43	61.43	60.43	61.43	59.14	59.71	60.10	1.45
EmbraceBERT-bs16+Dropout0.1	60.00	57.29	57.43	59.71	59.29	60.57	61.71	66.43	57.43	59.71	59.96	2.57
EmbraceBERT-bs16+Dropout0.3	60.29	60.43	59.86	61.14	57.00	59.57	60.00	65.14	58.86	60.57	60.29	1.95
CondensedEmbraceBERT-bs16	14.00	10.57	13.14	19.71	15.71	8.14	10.57	15.71	11.71	5.00	12.43	3.98
CondensedEmbraceBERT-bs16+Dropout0.1	59.57	60.43	57.86	61.00	60.29	60.86	59.43	64.29	59.43	59.00	60.21	1.62
CondensedEmbraceBERT-bs16+Dropout0.3	58.43	60.43	60.00	60.43	59.57	61.00	59.00	66.86	58.14	60.14	60.40	2.32
FrozenEBERT-bs16-ep100	61.57	56.29	60.29	59.86	59.14	60.71	61.29	59.57	59.71	59.43	59.79	1.39
FrozenEBERT-bs16-ep100+Dropout0.1	58.86	61.14	56.71	58.71	57.86	61.86	58.29	64.29	59.00	61.43	59.81	2.17
FrozenEBERT-bs16-ep100+Dropout0.3	56.71	58.00	61.86	60.57	59.43	59.43	57.43	64.29	60.29	57.43	59.54	2.21
FrozenCEBERT-bs16-ep100	11.57	7.71	7.57	14.43	12.43	5.71	8.57	17.43	23.86	7.00	11.63	5.38
FrozenCEBERT-bs16-ep100+Dropout0.1	62.71	60.57	57.86	58.29	57.71	58.57	60.57	68.57	59.86	60.57	60.53	3.06
FrozenCEBERT-bs16-ep100+Dropout0.3	60.00	60.86	60.14	60.43	57.57	60.57	60.57	66.29	60.29	58.86	60.56	2.13
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	58.43	57.29	57.43	58.71	59.00	59.57	61.00	59.29	59.00	65.57	59.53	2.25
EmbraceBERT-bs16+Branches+Dropout0.1	59.14	58.29	60.00	59.57	59.43	61.14	61.71	59.29	62.57	60.86	60.20	1.26
EmbraceBERT-bs16+Branches+Dropout0.3	59.43	60.14	58.29	62.00	58.29	57.86	60.71	59.29	59.29	61.00	59.63	1.26
CondensedEmbraceBERT-bs16+Branches	61.86	60.14	61.14	58.86	60.29	60.00	59.71	62.14	58.86	61.00	60.40	1.07
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	60.43	59.29	62.00	59.00	61.71	60.43	61.29	58.86	61.57	58.71	60.33	1.22
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	58.14	61.14	62.29	61.14	58.14	60.29	59.86	60.71	61.14	58.71	60.16	1.34
FrozenEmbraceBERT-bs16+Branches	60.57	60.71	57.29	62.00	60.00	62.00	60.71	60.43	56.57	66.14	60.64	2.50
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	59.29	60.86	60.86	60.43	57.43	61.00	59.71	58.71	59.71	60.43	59.84	1.07
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	58.43	60.71	61.29	58.57	59.86	61.57	59.00	60.71	63.14	59.57	60.29	1.41
FrozenCEBERT-bs16+Branches	64.14	59.14	61.29	58.29	58.86	62.14	58.71	60.14	60.29	62.00	60.50	1.78
FrozenCEBERT-bs16+Branches+Dropout0.1	59.86	59.86	62.43	57.14	57.29	60.00	58.14	60.29	60.86	58.86	59.47	1.56
FrozenCEBERT-bs16+Branches+Dropout0.3	61.71	59.71	63.71	60.14	62.71	63.14	62.14	61.00	62.43	58.00	61.47	1.67
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	62.00	60.00	61.86	61.00	58.86	59.86	62.29	60.29	60.43	60.00	60.66	1.05
BERT-bs32+Dropout0.1	61.29	60.86	59.57	61.43	60.57	58.71	61.57	58.57	60.86	60.86	60.43	1.04
BERT-bs32+Dropout0.3	60.00	62.14	60.57	61.14	61.43	59.57	61.00	60.57	59.86	61.43	60.77	0.77
FrozenBERT-bs32-ep100	62.00	59.14	61.71	61.43	59.00	59.00	61.00	59.57	61.00	58.86	60.27	1.20
FrozenBERT-bs32-ep100+Dropout0.1	66.43	58.86	59.43	62.43	60.86	59.29	60.29	59.29	60.00	59.71	60.66	2.16
FrozenBERT-bs32-ep100+Dropout0.3	61.00	62.43	60.86	60.86	60.57	58.43	58.71	60.71	60.14	58.43	60.21	1.24
EmbraceBERT-bs32	59.86	60.43	58.00	59.57	58.00	60.29	60.14	61.00	61.43	60.57	59.93	1.09
EmbraceBERT-bs32+Dropout0.1	58.43	59.29	60.29	60.57	59.29	60.86	59.86	66.00	59.86	59.00	60.34	2.01
EmbraceBERT-bs32+Dropout0.3	60.86	60.14	62.57	59.71	57.86	59.14	58.57	60.14	59.29	58.29	59.66	1.31
CondensedEmbraceBERT-bs32	12.00	10.14	12.00	11.14	18.14	7.43	11.14	23.71	23.86	6.71	13.63	5.85
CondensedEmbraceBERT-bs32+Dropout0.1	61.43	61.00	59.14	58.86	59.29	60.57	59.57	69.86	60.71	59.71	61.01	3.06
CondensedEmbraceBERT-bs32+Dropout0.3	59.57	58.43	60.14	59.29	59.57	61.14	59.71	66.71	57.86	59.86	60.23	2.32
FrozenEBERT-bs32-ep100	60.86	60.00	60.86	57.86	57.57	60.86	59.57	60.57	57.29	57.57	59.30	1.47
FrozenEBERT-bs32-ep100+Dropout0.1	59.14	61.00	59.43	60.00	58.57	60.14	59.57	65.71	60.00	59.57	60.31	1.90
FrozenEBERT-bs32-ep100+Dropout0.3	57.43	59.57	58.43	60.86	58.71	61.29	57.86	66.86	60.14	59.43	60.06	2.55
FrozenCEBERT-bs32-ep100	12.29	10.00	13.00	11.00	18.57	7.57	11.57	23.71	23.14	7.14	13.80	5.67
FrozenCEBERT-bs32-ep100+Dropout0.1	61.00	62.43	58.57	59.00	60.43	59.86	58.86	60.86	57.43	66.00	60.44	2.30
FrozenCEBERT-bs32-ep100+Dropout0.3	59.43	59.00	59.86	60.86	60.71	59.43	57.57	61.43	60.29	60.00	59.86	1.04
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32+Branches	57.57	60.86	58.00	59.86	59.43	61.00	59.71	65.29	59.43	61.43	60.26	2.04
EmbraceBERT-bs32+Branches+Dropout0.1	61.14	59.43	58.71	60.29	57.71	60.29	60.29	60.29	60.00	57.71	59.59	1.11
EmbraceBERT-bs32+Branches+Dropout0.3	58.71	59.29	58.00	60.86	58.86	60.43	61.71	60.29	60.14	59.14	59.74	1.07
CondensedEmbraceBERT-bs32+Branches	59.43	58.29	60.43	60.43	61.57	60.00	59.86	68.43	58.14	60.29	60.69	2.76
CondensedEmbraceBERT-bs32+Branches+Dropout0.1	58.71	59.00	58.57	60.43	60.57	61.14	59.43	61.29	60.14	59.14	59.84	0.95
CondensedEmbraceBERT-bs32+Branches+Dropout0.3	60.57	60.57	59.14	60.57	57.71	58.57	58.43	60.14	60.71	61.00	59.74	1.11
FrozenEmbraceBERT-bs32+Branches	58.14	62.00	61.43	60.71	59.43	59.14	58.57	63.00	59.71	59.86	60.20	1.48
FrozenEmbraceBERT-bs32+Branches+Dropout0.1	61.71	56.71	60.29	60.29	59.43	62.29	60.29	62.43	59.43	58.14	60.10	1.71
FrozenEmbraceBERT-bs32+Branches+Dropout0.3	57.14	58.86	58.00	59.43	58.43	59.29	60.57	61.86	61.71	59.29	59.46	1.45
FrozenCEBERT-bs32+Branches	58.71	58.14	59.14	60.86	59.86	60.86	58.43	58.57	59.29	62.71	59.66	1.36
FrozenCEBERT-bs32+Branches+Dropout0.1	61.86	59.71	61.29	57.86	59.00	60.00	59.71	64.29	58.14	60.14	60.20	1.80
FrozenCEBERT-bs32+Branches+Dropout0.3	61.14	56.86	60.00	61.57	61.00	61.14	58.00	62.57	59.57	59.86	60.17	1.63
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	61.43	61.57	62.00	62.14	61.71	60.86	61.14	62.00	61.86	61.29	61.60	0.40
RoBERTa-bs16+Dropout0.1	60.14	60.71	59.86	60.29	62.71	61.00	61.14	61.14	61.86	61.86	61.07	0.84
RoBERTa-bs16+Dropout0.3	60.71	61.43	60.71	61.14	60.57	61.29	60.57	60.29	60.86	62.00	60.96	0.48
FrozenRoBERTa-bs16-ep100	59.57	61.57	61.00	59.71	65.14	61.57	60.14	61.29	58.71	61.00	60.97	1.66
FrozenRoBERTa-bs16-ep100+Dropout0.1	59.43	60.57	59.71	61.29	59.00	61.00	60.57	60.00	60.29	62.43	60.43	0.94
FrozenRoBERTa-bs16-ep100+Dropout0.3	58.14	58.86	60.86	61.00	60.71	60.86	60.14	61.43	60.43	60.71	60.31	0.97
EmbraceRoBERTa-bs16	62.43	59.86	58.71	60.71	61.00	61.00	61.00	60.43	61.00	60.43	60.66	0.90
EmbraceRoBERTa-bs16+Dropout0.1	61.00	60.14	60.86	60.71	63.71	61.86	60.57	61.00	61.14	61.14	61.21	0.93
EmbraceRoBERTa-bs16+Dropout0.3	61.71	61.43	61.29	61.00	60.71	61.14	59.14	60.43	59.86	61.14	60.79	0.74
CondensedEmbraceRoBERTa-bs16	5.14	12.57	25.00	7.43	7.57	17.29	13.57	9.86	19.14	14.29	13.19	5.78
CondensedEmbraceRoBERTa-bs16+Dropout0.1	59.43	60.00	59.86	60.71	62.00	61.57	59.71	60.86	59.71	60.29	60.41	0.81
CondensedEmbraceRoBERTa-bs16+Dropout0.3	59.29	60.86	60.43	60.43	62.00	61.71	61.71	60.71	60.57	62.00	60.97	0.83
FrozenERoBERTa-bs16-ep100	61.71	61.57	61.00	60.57	59.14	61.57	59.71	58.00	58.71	61.14	60.31	1.26
FrozenERoBERTa-bs16-ep100+Dropout0.1	60.00	60.14	60.00	60.57	58.57	60.57	59.43	58.43	60.14	61.43	59.93	0.87
FrozenERoBERTa-bs16-ep100+Dropout0.3	60.57	59.86	60.57	61.43	60.29	61.71	62.00	59.86	60.29	63.86	61.04	1.18
FrozenCERoBERTa-bs16-ep100	8.14	12.29	24.86	7.43	5.71	15.00	13.57	5.43	16.86	9.14	11.84	5.72
FrozenCERoBERTa-bs16-ep100+Dropout0.1	61.71	58.29	61.14	60.57	60.71	58.86	64.00	63.86	60.00	61.00	61.01	1.76
FrozenCERoBERTa-bs16-ep100+Dropout0.3	61.57	61.57	60.14	60.86	61.00	60.57	59.43	61.57	60.14	61.00	60.79	0.68
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	60.43	62.00	59.86	61.71	59.14	60.71	60.57	61.29	62.43	61.14	60.93	0.95
EmbraceRoBERTa-bs16+Branches+Dropout0.1	59.71	60.43	61.00	61.14	60.14	61.71	60.71	61.29	58.86	60.57	60.56	0.79
EmbraceRoBERTa-bs16+Branches+Dropout0.3	60.86	61.14	62.00	60.86	60.71	61.71	60.29	60.29	61.00	61.43	61.03	0.53
CondensedEmbraceRoBERTa-bs16+Branches	62.29	61.14	62.14	60.86	62.14	61.86	60.71	62.57	60.29	62.86	61.69	0.83
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	60.57	60.43	61.00	59.29	60.71	62.29	60.57	63.00	59.29	60.14	60.73	1.11
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	60.86	59.71	59.14	60.29	63.43	62.14	61.43	62.43	59.71	62.86	61.20	1.41
FrozenEmbraceRoBERTa-bs16+Branches	60.71	60.71	59.14	62.86	61.43	61.71	60.71	61.29	62.00	60.86	61.14	0.93
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	61.43	64.14	59.57	60.29	59.29	59.14	58.86	62.00	60.57	59.43	60.47	1.56
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	62.00	62.00	59.29	60.86	59.57	59.86	62.86	60.86	59.71	61.00	60.80	1.14
FrozenCERoBERTa-bs16+Branches	60.71	61.57	61.71	63.14	60.57	61.86	60.29	65.29	60.71	60.00	61.59	1.52
FrozenCERoBERTa-bs16+Branches+Dropout0.1	62.57	60.14	60.86	59.71	61.57	60.71	62.29	61.29	62.00	63.71	61.49	1.14
FrozenCERoBERTa-bs16+Branches+Dropout0.3	60.86	59.86	61.00	61.00	62.57	60.14	61.43	60.57	60.86	58.86	60.71	0.93
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	60.86	61.57	61.00	61.14	62.14	61.29	63.00	62.43	62.29	63.29	61.90	0.81
RoBERTa-bs32+Dropout0.1	60.57	60.43	59.14	60.86	61.57	61.14	58.57	61.43	60.57	62.00	60.63	1.01
RoBERTa-bs32+Dropout0.3	61.00	61.43	58.86	61.57	62.29	60.71	59.29	61.86	59.86	60.71	60.76	1.06
FrozenRoBERTa-bs32-ep100	62.00	60.43	59.14	60.00	63.43	62.43	61.00	62.43	60.71	60.86	61.24	1.24
FrozenRoBERTa-bs32-ep100+Dropout0.1	60.57	61.00	58.57	60.43	60.86	62.57	59.57	60.14	60.43	62.00	60.61	1.07
FrozenRoBERTa-bs32-ep100+Dropout0.3	61.00	60.86	57.14	60.14	60.71	61.29	59.00	61.71	60.57	61.00	60.34	1.27
EmbraceRoBERTa-bs32	61.57	61.29	59.14	63.00	60.00	60.43	61.86	63.43	61.29	61.00	61.30	1.22
EmbraceRoBERTa-bs32+Dropout0.1	62.86	60.86	61.43	62.00	60.57	60.29	60.43	62.71	61.00	61.71	61.39	0.87
EmbraceRoBERTa-bs32+Dropout0.3	60.86	60.14	62.43	62.57	60.00	60.00	61.00	60.86	61.43	60.14	60.94	0.90
CondensedEmbraceRoBERTa-bs32	5.43	6.86	23.57	6.57	5.43	13.29	13.14	4.57	12.71	11.00	10.26	5.53
CondensedEmbraceRoBERTa-bs32+Dropout0.1	62.71	59.86	60.86	62.71	60.71	61.57	62.29	63.00	61.14	61.14	61.60	0.99
CondensedEmbraceRoBERTa-bs32+Dropout0.3	61.43	60.00	60.57	59.71	60.29	60.86	60.29	63.00	60.57	61.43	60.81	0.90
FrozenERoBERTa-bs32-ep100	61.14	60.00	60.57	60.57	58.14	62.00	61.57	63.00	60.29	62.00	60.93	1.28
FrozenERoBERTa-bs32-ep100+Dropout0.1	60.86	60.29	58.71	60.71	60.86	59.14	60.14	60.14	60.43	60.00	60.13	0.67
FrozenERoBERTa-bs32-ep100+Dropout0.3	59.71	59.43	61.71	60.14	61.00	60.14	60.43	61.29	61.00	63.29	60.81	1.07
FrozenCERoBERTa-bs32-ep100	3.71	7.29	30.71	7.00	7.29	14.43	16.29	4.29	13.00	5.86	10.99	7.75
FrozenCERoBERTa-bs32-ep100+Dropout0.1	61.86	60.14	62.71	61.86	61.71	61.43	60.43	62.29	60.57	59.71	61.27	0.94
FrozenCERoBERTa-bs32-ep100+Dropout0.3	60.86	60.57	60.71	62.43	62.00	59.86	60.43	61.14	60.71	63.29	61.20	0.99
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Branches	62.43	60.43	61.71	61.00	61.71	60.29	66.14	60.14	60.57	61.71	61.61	1.67
EmbraceRoBERTa-bs32+Branches+Dropout0.1	62.43	61.29	62.00	66.86	59.14	61.71	60.86	61.86	60.14	61.29	61.76	1.93
EmbraceRoBERTa-bs32+Branches+Dropout0.3	61.43	61.43	60.29	61.43	60.29	62.29	63.00	61.71	61.29	61.57	61.47	0.77
CondensedEmbraceRoBERTa-bs32+Branches	60.00	59.71	60.43	63.86	66.43	61.71	61.14	65.86	59.86	65.29	62.43	2.53
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.1	61.57	60.86	61.00	62.14	60.86	61.71	61.71	64.14	61.00	62.86	61.79	0.99
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.3	61.86	60.00	62.57	60.00	60.29	63.00	62.00	62.14	60.00	62.14	61.40	1.13
FrozenEmbraceRoBERTa-bs32+Branches	61.43	60.00	60.71	60.86	61.57	61.43	63.00	60.43	60.71	64.29	61.44	1.23
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.1	61.71	63.00	61.71	62.00	59.71	59.71	59.43	61.00	63.71	60.71	61.27	1.36
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.3	60.71	60.57	60.29	62.43	62.86	60.29	62.00	61.43	64.00	59.71	61.43	1.30
FrozenCERoBERTa-bs32+Branches	60.86	60.29	62.29	60.43	61.86	62.00	64.71	61.86	60.14	61.57	61.60	1.27
FrozenCERoBERTa-bs32+Branches+Dropout0.1	59.57	60.86	61.57	60.29	60.86	63.00	60.71	61.29	60.00	61.00	60.91	0.89
FrozenCERoBERTa-bs32+Branches+Dropout0.3	60.29	61.43	60.86	60.43	61.57	60.86	61.86	60.43	60.00	60.86	60.86	0.57

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	61.57	60.86	60.14	59.71	60.43	59.14	62.43	60.86	59.86	59.86	60.49	0.92
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	59.57	60.71	62.14	61.71	60.29	62.29	63.00	60.57	61.57	60.71	61.26	1.00
BERTwithAttClsProjection-bs32	62.57	60.43	61.00	60.43	62.14	60.71	61.43	61.86	61.57	61.57	61.37	0.69
BERTwithProjection-bs32	62.00	61.57	61.00	60.29	60.14	62.14	59.71	61.57	60.43	61.29	61.01	0.79
BERTwithProjectionAtt-bs32	61.43	60.29	62.29	61.29	59.57	60.29	61.00	60.86	61.14	62.00	61.01	0.78
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	61.29	60.57	61.00	61.29	60.57	60.71	62.14	63.71	60.71	60.57	61.26	0.94
EmbraceBERT-bs32-p_attclsqw	63.00	60.57	60.43	59.43	61.43	62.29	60.14	62.14	62.57	60.57	61.26	1.13
EmbraceBERTwithProj-bs32	61.14	61.00	61.00	61.86	62.00	60.00	61.71	59.86	61.43	63.57	61.36	1.00
EmbraceBERTwithProj-bs32-p_attclsqw	63.14	60.71	61.86	59.86	62.57	61.29	62.57	62.14	61.71	62.71	61.86	0.96
EBERTconcatAtt-bs32	61.86	59.29	61.14	60.00	59.86	60.71	60.86	60.57	61.86	61.00	60.71	0.79
EBERTconcatAtt-bs32-p_attclsqw	60.00	59.71	61.57	61.29	61.29	60.71	60.29	62.71	60.71	61.00	60.93	0.82
EBERTconcatAttwithProj-bs32	61.86	60.00	62.29	60.43	61.43	61.14	61.71	60.57	59.71	59.00	60.81	1.00
EBERTconcatAttwithProj-bs32-p_attclsqw	61.29	58.86	61.29	61.14	61.14	60.57	61.29	61.86	62.29	61.86	61.16	0.89
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery	60.57	60.00	61.14	61.86	61.86	63.00	62.14	60.29	59.14	60.71	61.07	1.09
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	61.29	60.86	61.86	62.43	61.14	61.14	62.00	61.14	60.29	60.86	61.30	0.60
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery	61.71	62.14	61.71	59.29	60.43	62.86	60.86	62.43	63.29	57.86	61.26	1.60
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	59.29	61.57	60.86	60.57	61.29	60.43	60.29	61.71	60.71	61.29	60.80	0.68
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery	60.57	61.86	61.57	60.71	61.43	61.00	61.71	60.43	60.71	59.29	60.93	0.73
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	59.71	60.14	60.29	61.71	60.00	60.86	61.00	61.29	61.86	60.14	60.70	0.71
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery	62.29	61.71	61.14	61.14	62.00	60.86	61.14	61.57	62.86	61.57	61.63	0.58
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	60.00	62.29	61.71	60.43	61.14	60.29	61.71	61.71	64.14	60.71	61.41	1.15

• Epoch 100 - bs48 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	60.86	60.43	60.29	61.14	58.71	59.43	61.00	61.14	59.29	59.14	60.14	0.87
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	61.29	59.86	59.57	61.57	61.57	62.00	61.29	59.71	59.14	59.43	60.54	1.03
BERTwithAttClsProjection-bs48	60.00	61.71	60.43	61.57	63.00	62.29	61.43	60.71	61.14	61.29	61.36	0.83
BERTwithProjection-bs48	62.43	61.57	60.86	60.71	62.00	61.43	61.43	61.43	61.00	62.71	61.56	0.62
BERTwithProjectionAtt-bs48	61.14	61.00	58.71	59.71	60.86	63.00	61.86	61.29	60.57	62.29	61.04	1.16
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	60.57	60.43	59.86	60.71	60.14	60.71	61.57	60.71	60.29	60.86	60.59	0.44
EmbraceBERT-bs48-p_attclsqw	63.14	59.57	60.71	61.57	60.00	59.00	59.86	61.43	60.14	60.71	60.61	1.13
EmbraceBERTwithProj-bs48	61.29	61.00	62.14	60.71	61.29	61.43	61.29	61.29	61.43	62.14	61.40	0.42
EmbraceBERTwithProj-bs48-p_attclsqw	58.43	60.14	61.57	60.57	61.00	61.29	62.00	63.86	61.43	61.57	61.19	1.31
EBERTconcatAtt-bs48	61.00	61.43	60.43	61.00	60.71	60.29	60.29	59.71	59.86	60.71	60.54	0.51
EBERTconcatAtt-bs48-p_attclsqw	62.43	59.86	61.29	61.86	59.57	61.43	61.29	60.71	61.00	60.43	60.99	0.83
EBERTconcatAttwithProj-bs48	61.43	62.29	61.86	59.14	60.71	60.43	61.57	62.71	62.14	60.71	61.30	1.01
EBERTconcatAttwithProj-bs48-p_attclsqw	60.00	60.43	60.71	61.29	61.86	62.14	61.43	61.57	60.71	60.71	61.09	0.64
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	61.14	61.86	59.86	59.43	61.43	61.00	60.86	62.29	61.29	61.14	61.03	0.81
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	61.29	59.57	61.71	60.71	60.29	60.00	59.14	60.86	60.86	61.00	60.54	0.75
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	61.29	60.43	60.57	60.71	60.43	60.57	60.43	61.86	62.43	60.86	60.96	0.65
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	59.86	60.86	62.00	60.00	60.86	61.14	61.29	62.57	62.00	66.43	61.70	1.77
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	61.57	61.86	60.29	61.57	59.29	60.29	60.86	60.29	60.14	59.43	60.56	0.84
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	60.57	59.57	61.29	60.29	60.57	62.14	60.14	60.43	62.29	61.86	60.91	0.88
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	66.57	60.00	60.29	61.29	61.57	60.43	61.14	60.86	60.29	60.86	61.33	1.81
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	61.57	60.57	61.29	60.86	61.00	60.43	61.29	61.86	62.43	60.71	61.20	0.59

gtts-sphinx - not good with any

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	83.14	83.86	84.00	83.00	80.14	84.00	83.71	80.86	83.43	83.00	82.91	1.27
BERT-bs16+Dropout0.1	83.71	85.43	83.71	79.43	83.86	83.00	85.43	79.29	84.43	81.29	82.96	2.12
BERT-bs16+Dropout0.3	84.29	80.29	81.71	86.43	80.57	81.57	82.71	78.29	84.71	82.29	82.29	2.26
FrozenBERT-bs16-ep100	82.43	81.86	82.71	83.71	81.29	82.71	79.57	83.14	80.43	81.86	81.97	1.20
FrozenBERT-bs16-ep100+Dropout0.1	81.86	84.71	77.43	80.57	82.71	82.43	84.14	81.29	81.14	81.29	81.76	1.92
FrozenBERT-bs16-ep100+Dropout0.3	81.43	87.43	80.57	79.86	82.43	84.00	76.43	83.29	83.00	84.57	82.30	2.83
EmbraceBERT-bs16	85.14	80.29	84.14	82.43	74.57	83.00	83.71	83.14	80.57	78.00	81.50	3.06
EmbraceBERT-bs16+Dropout0.1	83.14	77.71	78.43	82.71	81.43	83.29	85.43	84.86	80.00	83.57	82.06	2.48
EmbraceBERT-bs16+Dropout0.3	82.71	84.00	85.29	81.00	74.86	80.71	81.86	82.71	80.71	84.00	81.79	2.73
CondensedEmbraceBERT-bs16	14.29	8.43	8.86	13.71	11.71	9.43	4.43	19.43	12.29	5.29	10.79	4.25
CondensedEmbraceBERT-bs16+Dropout0.1	82.29	81.86	80.86	82.57	81.43	82.71	80.86	82.57	82.14	80.29	81.76	0.81
CondensedEmbraceBERT-bs16+Dropout0.3	83.57	81.71	81.57	86.00	79.86	81.29	79.14	82.71	80.71	81.71	81.83	1.84
FrozenEBERT-bs16-ep100	80.71	69.71	82.14	82.00	80.00	81.86	85.14	81.14	79.29	80.00	80.20	3.82
FrozenEBERT-bs16-ep100+Dropout0.1	81.00	81.29	82.00	80.43	79.86	83.14	80.71	83.00	82.29	84.57	81.83	1.38
FrozenEBERT-bs16-ep100+Dropout0.3	77.43	77.86	85.29	81.00	83.57	81.29	78.71	81.14	83.71	79.86	80.99	2.48
FrozenCEBERT-bs16-ep100	15.14	5.29	6.29	11.14	11.29	6.14	4.71	19.14	23.86	6.29	10.93	6.24
FrozenCEBERT-bs16-ep100+Dropout0.1	84.43	80.86	76.86	77.57	78.57	77.14	81.43	82.57	80.71	81.71	80.19	2.41
FrozenCEBERT-bs16-ep100+Dropout0.3	84.43	84.29	83.43	81.14	75.86	80.86	80.71	85.86	83.00	80.57	82.01	2.69
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	81.57	72.57	79.14	78.29	81.29	81.00	84.71	80.57	79.29	81.29	79.97	2.98
EmbraceBERT-bs16+Branches+Dropout0.1	83.00	80.14	78.43	80.29	76.86	83.43	82.00	81.57	83.43	85.29	81.44	2.42
EmbraceBERT-bs16+Branches+Dropout0.3	83.86	80.43	76.86	83.43	80.29	80.14	84.71	81.71	82.14	82.43	81.60	2.16
CondensedEmbraceBERT-bs16+Branches	84.71	82.00	82.43	79.43	82.43	81.14	82.71	84.00	79.57	83.86	82.23	1.69
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	85.43	83.00	77.00	79.71	85.14	80.86	82.57	80.00	83.29	79.43	81.64	2.56
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	84.00	81.71	84.00	84.57	77.00	81.43	80.29	82.14	84.86	80.86	82.09	2.29
FrozenEmbraceBERT-bs16+Branches	85.00	80.14	78.00	83.71	79.43	83.86	82.00	84.43	75.57	84.57	81.67	3.07
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	84.00	85.00	82.14	84.57	77.43	82.57	79.14	82.14	83.57	82.29	82.29	2.25
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	81.00	84.29	81.71	76.29	79.43	83.86	79.14	84.14	83.14	82.14	81.51	2.46
FrozenCEBERT-bs16+Branches	85.14	81.57	82.71	76.57	80.00	83.29	79.43	82.86	78.43	80.14	81.01	2.44
FrozenCEBERT-bs16+Branches+Dropout0.1	84.00	84.29	79.57	78.57	80.29	81.43	82.43	81.43	82.86	82.14	81.70	1.75
FrozenCEBERT-bs16+Branches+Dropout0.3	85.57	84.14	82.14	83.00	82.29	84.29	83.29	83.00	87.29	75.29	83.03	2.98
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	83.57	83.43	84.00	81.00	80.43	82.86	84.29	83.43	82.00	83.86	82.89	1.25
BERT-bs32+Dropout0.1	83.43	82.86	82.29	82.43	82.00	81.86	84.71	80.86	80.86	84.86	82.61	1.32
BERT-bs32+Dropout0.3	83.43	83.00	84.00	82.57	82.86	81.43	83.86	84.29	82.43	84.00	83.19	0.85
FrozenBERT-bs32-ep100	84.29	81.29	84.43	82.00	81.00	81.57	82.00	81.86	83.57	83.14	82.51	1.18
FrozenBERT-bs32-ep100+Dropout0.1	85.14	81.57	79.43	83.00	83.57	82.57	82.14	80.14	85.86	85.71	82.91	2.10
FrozenBERT-bs32-ep100+Dropout0.3	83.86	85.43	81.29	80.14	81.29	81.00	80.00	82.14	83.86	81.29	82.03	1.69
EmbraceBERT-bs32	83.14	83.14	80.14	79.57	78.71	80.71	82.43	80.43	82.00	81.71	81.20	1.44
EmbraceBERT-bs32+Dropout0.1	82.29	81.00	83.14	85.14	81.14	80.14	84.57	86.43	82.71	79.71	82.63	2.11
EmbraceBERT-bs32+Dropout0.3	84.29	80.00	82.71	83.14	79.43	78.00	79.00	82.14	84.57	80.71	81.40	2.18
CondensedEmbraceBERT-bs32	15.57	9.57	7.43	5.71	17.71	10.14	5.43	24.71	27.14	6.00	12.94	7.59
CondensedEmbraceBERT-bs32+Dropout0.1	84.57	83.57	82.00	80.57	81.29	79.14	83.14	86.43	83.14	82.43	82.63	1.96
CondensedEmbraceBERT-bs32+Dropout0.3	83.29	78.43	84.14	79.71	83.14	81.71	81.71	83.14	79.86	82.14	81.73	1.76
FrozenEBERT-bs32-ep100	83.57	82.14	84.43	78.29	78.14	79.43	80.00	80.71	77.57	78.71	80.30	2.25
FrozenEBERT-bs32-ep100+Dropout0.1	83.14	82.57	80.71	79.86	78.00	81.86	81.14	82.43	81.00	80.29	81.10	1.44
FrozenEBERT-bs32-ep100+Dropout0.3	82.43	79.14	83.86	82.00	77.00	82.43	76.86	83.29	83.14	79.29	80.94	2.50
FrozenCEBERT-bs32-ep100	16.00	9.43	7.43	5.86	16.86	9.86	5.43	25.00	27.71	6.14	12.97	7.70
FrozenCEBERT-bs32-ep100+Dropout0.1	83.14	83.86	80.57	77.29	81.86	80.14	81.57	80.86	79.14	80.71	80.91	1.79
FrozenCEBERT-bs32-ep100+Dropout0.3	83.57	81.71	81.43	81.86	82.14	80.29	78.86	80.86	81.14	83.29	81.51	1.31
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32+Branches	79.86	83.86	77.86	82.00	81.71	82.14	82.57	83.43	81.71	81.14	81.63	1.65
EmbraceBERT-bs32+Branches+Dropout0.1	84.71	84.00	78.43	84.14	80.29	81.43	79.43	85.86	81.57	80.57	82.04	2.36
EmbraceBERT-bs32+Branches+Dropout0.3	82.29	81.29	79.14	82.00	81.00	81.14	84.29	83.86	82.43	80.43	81.79	1.46
CondensedEmbraceBERT-bs32+Branches	83.86	82.86	82.14	80.57	82.14	81.57	81.14	82.14	77.71	83.57	81.77	1.66
CondensedEmbraceBERT-bs32+Branches+Dropout0.1	81.71	83.00	77.71	83.57	81.43	83.86	78.57	85.14	83.29	81.86	82.01	2.21
CondensedEmbraceBERT-bs32+Branches+Dropout0.3	84.29	83.71	79.43	83.71	78.00	79.43	78.00	81.14	81.86	82.57	81.21	2.26
FrozenEmbraceBERT-bs32+Branches	81.00	84.29	83.43	81.57	80.86	78.71	80.86	84.71	80.71	80.71	81.69	1.78
FrozenEmbraceBERT-bs32+Branches+Dropout0.1	83.43	75.57	78.57	79.43	82.43	82.00	81.00	86.57	81.00	79.14	80.91	2.85
FrozenEmbraceBERT-bs32+Branches+Dropout0.3	78.86	80.57	73.29	80.86	79.29	79.14	81.29	85.57	82.57	79.00	80.04	2.98
FrozenCEBERT-bs32+Branches	81.71	77.14	79.14	83.57	84.71	83.71	78.43	80.00	80.14	84.71	81.33	2.60
FrozenCEBERT-bs32+Branches+Dropout0.1	86.57	83.71	82.14	76.71	77.14	80.71	81.43	84.86	77.71	82.57	81.36	3.16
FrozenCEBERT-bs32+Branches+Dropout0.3	83.57	79.29	78.57	82.86	81.43	80.57	74.57	86.14	81.29	78.71	80.70	3.02
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	84.71	83.57	84.14	85.14	85.43	85.71	84.43	84.57	84.86	85.14	84.77	0.60
RoBERTa-bs16+Dropout0.1	80.57	85.43	85.86	79.14	84.57	83.00	83.00	82.14	85.00	80.29	82.90	2.22
RoBERTa-bs16+Dropout0.3	85.86	85.29	84.86	80.00	81.00	83.14	84.14	83.00	84.29	83.43	83.50	1.75
FrozenRoBERTa-bs16-ep100	80.57	82.57	83.57	76.00	84.00	84.43	83.43	84.86	79.57	83.14	82.21	2.60
FrozenRoBERTa-bs16-ep100+Dropout0.1	80.57	82.71	80.57	82.14	81.86	81.86	82.43	80.71	81.29	81.71	81.59	0.73
FrozenRoBERTa-bs16-ep100+Dropout0.3	80.86	79.00	82.29	80.14	84.86	81.00	82.00	81.71	81.86	82.00	81.57	1.46
EmbraceRoBERTa-bs16	83.00	82.29	83.14	80.71	82.14	84.43	82.14	81.71	80.00	83.71	82.33	1.26
EmbraceRoBERTa-bs16+Dropout0.1	84.43	82.43	81.29	79.71	84.71	81.43	82.71	82.29	82.86	83.71	82.56	1.44
EmbraceRoBERTa-bs16+Dropout0.3	85.00	82.00	83.57	85.14	82.43	83.00	82.71	79.43	82.71	80.57	82.66	1.67
CondensedEmbraceRoBERTa-bs16	3.86	8.86	28.57	13.86	6.43	24.00	10.71	11.71	14.29	16.29	13.86	7.21
CondensedEmbraceRoBERTa-bs16+Dropout0.1	84.14	83.00	82.43	83.29	82.14	75.86	83.29	80.71	82.43	84.43	82.17	2.33
CondensedEmbraceRoBERTa-bs16+Dropout0.3	82.57	82.86	86.29	83.57	83.43	84.57	84.00	83.29	79.86	84.14	83.46	1.55
FrozenERoBERTa-bs16-ep100	86.43	83.71	82.00	81.29	81.14	83.71	82.00	80.29	81.43	82.86	82.49	1.68
FrozenERoBERTa-bs16-ep100+Dropout0.1	84.14	81.71	81.00	83.86	81.00	82.43	83.14	81.71	81.29	78.86	81.91	1.48
FrozenERoBERTa-bs16-ep100+Dropout0.3	83.29	80.29	84.14	80.71	83.29	82.00	83.57	80.00	79.71	84.00	82.10	1.68
FrozenCERoBERTa-bs16-ep100	3.57	14.43	28.71	13.29	3.14	23.71	12.29	7.86	17.43	8.29	13.27	7.86
FrozenCERoBERTa-bs16-ep100+Dropout0.1	82.57	77.43	84.29	78.71	82.43	81.00	83.57	82.43	83.71	85.00	82.11	2.30
FrozenCERoBERTa-bs16-ep100+Dropout0.3	82.00	83.86	80.00	83.86	82.57	81.14	80.86	85.14	80.71	84.00	82.41	1.65
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	81.57	83.43	83.00	82.43	83.14	83.00	84.00	84.29	84.14	85.29	83.43	1.00
EmbraceRoBERTa-bs16+Branches+Dropout0.1	82.43	82.14	83.29	82.29	82.86	82.71	83.00	83.00	82.57	79.71	82.40	0.96
EmbraceRoBERTa-bs16+Branches+Dropout0.3	84.29	77.86	83.29	82.57	83.00	83.14	84.14	84.86	82.14	84.29	82.96	1.88
CondensedEmbraceRoBERTa-bs16+Branches	81.71	84.43	83.00	83.43	82.57	84.00	80.14	87.29	79.14	83.86	82.96	2.17
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	81.71	83.29	84.71	82.43	85.57	84.57	82.43	81.29	83.71	83.71	83.34	1.31
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	82.29	83.71	81.00	83.00	83.43	83.14	84.43	86.43	82.14	82.86	83.24	1.39
FrozenEmbraceRoBERTa-bs16+Branches	80.00	82.43	78.86	84.00	83.29	85.00	85.00	81.71	84.43	82.00	82.67	1.98
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	80.71	81.43	82.14	82.86	78.86	82.71	82.86	85.86	82.71	82.86	82.30	1.71
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	84.14	80.14	79.14	83.43	82.71	79.43	85.71	82.43	82.57	85.57	82.53	2.23
FrozenCERoBERTa-bs16+Branches	83.57	82.86	82.71	81.00	82.86	84.43	82.43	84.29	82.29	84.00	83.04	1.00
FrozenCERoBERTa-bs16+Branches+Dropout0.1	83.43	85.00	85.00	82.29	82.43	81.29	81.71	85.00	83.71	83.71	83.36	1.32
FrozenCERoBERTa-bs16+Branches+Dropout0.3	84.14	85.14	84.00	81.57	81.14	84.71	82.86	82.86	83.29	81.14	83.09	1.37
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	85.29	83.29	84.57	85.43	85.29	85.29	84.43	84.71	84.86	86.14	84.93	0.72
RoBERTa-bs32+Dropout0.1	82.29	83.29	79.43	83.14	83.29	82.57	82.86	82.71	84.57	84.14	82.83	1.31
RoBERTa-bs32+Dropout0.3	81.14	82.57	80.71	81.29	83.43	80.86	79.71	84.14	85.00	79.71	81.86	1.74
FrozenRoBERTa-bs32-ep100	84.29	80.57	82.14	79.57	84.29	81.71	83.00	83.29	82.71	83.43	82.50	1.46
FrozenRoBERTa-bs32-ep100+Dropout0.1	82.14	83.14	81.71	83.29	82.00	83.00	83.14	81.43	84.86	82.57	82.73	0.94
FrozenRoBERTa-bs32-ep100+Dropout0.3	84.86	83.43	81.43	80.14	83.14	82.43	83.00	84.57	81.86	82.14	82.70	1.35
EmbraceRoBERTa-bs32	84.29	83.71	83.43	82.14	84.71	83.29	83.57	85.14	83.29	83.71	83.73	0.79
EmbraceRoBERTa-bs32+Dropout0.1	85.57	81.86	83.57	84.14	85.43	82.71	81.71	83.29	83.00	82.43	83.37	1.27
EmbraceRoBERTa-bs32+Dropout0.3	84.43	79.86	84.14	85.14	81.43	83.14	83.71	82.29	79.86	81.57	82.56	1.77
CondensedEmbraceRoBERTa-bs32	7.86	3.57	27.43	13.57	3.71	15.86	8.71	5.00	12.29	11.14	10.91	6.79
CondensedEmbraceRoBERTa-bs32+Dropout0.1	83.57	83.14	83.86	83.29	82.71	85.71	84.14	84.57	82.57	84.43	83.80	0.91
CondensedEmbraceRoBERTa-bs32+Dropout0.3	85.29	81.57	82.14	83.14	82.14	83.71	81.86	82.86	84.43	84.00	83.11	1.16
FrozenERoBERTa-bs32-ep100	83.86	82.14	84.86	82.57	82.14	83.57	83.29	81.57	81.14	83.00	82.81	1.07
FrozenERoBERTa-bs32-ep100+Dropout0.1	82.71	79.14	82.29	82.43	80.71	83.29	82.57	80.29	81.57	84.29	81.93	1.44
FrozenERoBERTa-bs32-ep100+Dropout0.3	82.14	78.86	84.29	82.57	82.29	80.71	81.71	84.43	79.86	81.14	81.80	1.68
FrozenCERoBERTa-bs32-ep100	2.43	2.86	32.57	13.00	5.43	17.71	8.43	5.00	7.71	4.57	9.97	8.77
FrozenCERoBERTa-bs32-ep100+Dropout0.1	81.14	82.29	83.14	82.86	83.00	84.14	82.57	84.57	82.43	84.14	83.03	0.98
FrozenCERoBERTa-bs32-ep100+Dropout0.3	84.00	81.86	83.57	81.14	85.14	84.71	82.57	81.71	85.14	81.71	83.16	1.46
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Branches	85.43	85.14	83.43	84.86	84.57	81.86	82.00	82.00	84.57	84.86	83.87	1.35
EmbraceRoBERTa-bs32+Branches+Dropout0.1	85.57	82.00	85.14	84.29	83.14	80.57	77.86	84.00	82.57	83.86	82.90	2.19
EmbraceRoBERTa-bs32+Branches+Dropout0.3	83.86	78.57	83.57	83.57	82.00	85.43	82.57	84.29	81.71	86.00	83.16	2.00
CondensedEmbraceRoBERTa-bs32+Branches	81.43	79.71	81.00	85.14	83.57	83.29	81.43	84.00	82.86	83.00	82.54	1.54
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.1	82.29	83.29	79.86	82.71	82.43	83.14	85.86	83.43	82.57	86.00	83.16	1.68
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.3	84.71	83.86	85.00	79.43	82.86	84.29	84.14	83.86	80.86	84.57	83.36	1.73
FrozenEmbraceRoBERTa-bs32+Branches	84.14	82.57	82.71	84.00	84.57	78.86	82.71	81.14	82.29	84.43	82.74	1.66
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.1	85.71	82.86	83.29	79.43	83.14	83.29	80.14	82.43	83.86	83.43	82.76	1.71
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.3	82.71	81.57	81.00	82.86	80.57	82.14	81.00	83.00	84.43	80.00	81.93	1.28
FrozenCERoBERTa-bs32+Branches	84.86	82.29	83.57	81.86	83.14	84.86	83.86	82.14	84.14	83.29	83.40	1.02
FrozenCERoBERTa-bs32+Branches+Dropout0.1	81.43	82.86	83.00	82.00	82.43	83.00	83.71	83.43	83.29	83.29	82.84	0.67
FrozenCERoBERTa-bs32+Branches+Dropout0.3	82.71	82.00	83.43	83.29	82.71	81.43	80.86	83.43	79.71	85.57	82.51	1.54

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	82.86	85.57	84.29	83.71	84.71	81.71	85.43	84.29	81.14	83.29	83.70	1.40
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	82.43	83.86	83.29	84.00	83.86	83.14	87.43	82.57	85.71	83.29	83.96	1.45
BERTwithAttClsProjection-bs32	85.71	82.71	83.86	83.57	82.57	84.86	85.71	84.86	85.71	84.29	84.39	1.13
BERTwithProjection-bs32	84.29	84.43	85.14	82.29	84.00	84.14	81.14	86.86	84.29	84.43	84.10	1.45
BERTwithProjectionAtt-bs32	84.43	85.71	84.14	83.86	84.29	82.57	85.57	83.43	83.57	85.14	84.27	0.94
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	82.86	84.29	85.00	83.57	83.29	84.00	84.29	86.57	83.00	84.14	84.10	1.03
EmbraceBERT-bs32-p_attclsqw	82.86	84.43	83.00	83.00	84.29	85.43	82.00	83.57	83.71	81.71	83.40	1.07
EmbraceBERTwithProj-bs32	84.14	83.57	84.43	84.57	86.86	83.71	84.43	83.14	85.86	84.86	84.56	1.05
EmbraceBERTwithProj-bs32-p_attclsqw	86.14	84.29	84.00	83.14	86.00	84.00	82.57	84.29	83.29	83.00	84.07	1.14
EBERTconcatAtt-bs32	82.14	81.29	83.14	83.29	82.86	83.86	83.14	81.86	84.71	83.14	82.94	0.94
EBERTconcatAtt-bs32-p_attclsqw	84.43	82.86	84.00	84.86	84.29	84.14	85.57	81.14	82.14	85.00	83.84	1.31
EBERTconcatAttwithProj-bs32	84.86	84.29	86.00	83.14	83.29	82.71	84.43	83.86	83.14	83.14	83.89	0.96
EBERTconcatAttwithProj-bs32-p_attclsqw	82.29	83.57	82.14	84.43	83.86	83.86	83.14	85.14	86.57	84.86	83.99	1.27
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery	83.43	84.57	82.14	84.71	83.29	85.71	85.43	83.43	83.14	83.71	83.96	1.06
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	82.86	85.00	82.86	86.57	84.43	83.29	83.86	84.14	84.00	84.86	84.19	1.07
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery	85.86	86.57	83.71	83.43	84.71	84.86	84.71	86.43	87.00	83.57	85.09	1.25
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	84.43	84.29	84.14	86.00	84.43	85.57	82.00	84.71	82.86	84.00	84.24	1.10
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery	82.29	84.43	83.29	83.43	85.00	84.71	85.14	82.43	86.43	82.00	83.91	1.38
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	84.86	82.43	83.71	84.14	82.43	81.86	82.86	84.86	85.71	84.14	83.70	1.20
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery	84.71	84.43	84.29	83.57	83.43	85.00	85.29	85.71	85.00	85.29	84.67	0.71
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	82.43	84.14	82.71	81.71	84.57	84.71	84.43	85.29	87.14	84.43	84.16	1.48

• Epoch 100 - bs48 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	81.86	82.86	83.00	82.86	80.14	83.14	83.71	84.57	80.00	82.71	82.49	1.38
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	84.29	83.14	84.86	85.14	82.43	85.00	83.57	83.00	83.57	84.14	83.91	0.87
BERTwithAttClsProjection-bs48	85.14	82.29	83.43	83.14	85.14	84.86	85.43	86.57	82.86	84.00	84.29	1.28
BERTwithProjection-bs48	83.29	84.14	83.86	84.00	85.00	86.57	84.14	86.00	84.14	84.00	84.51	0.98
BERTwithProjectionAtt-bs48	82.86	82.86	82.43	84.14	81.57	83.29	86.57	84.29	83.71	83.43	83.51	1.27
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	82.43	81.43	83.29	82.14	81.71	84.43	83.29	83.29	81.14	82.57	82.57	0.96
EmbraceBERT-bs48-p_attclsqw	83.14	81.71	85.29	85.00	78.86	81.86	84.43	83.86	84.71	83.14	83.20	1.86
EmbraceBERTwithProj-bs48	84.43	85.43	84.86	83.00	84.86	84.57	84.14	82.71	85.86	85.00	84.49	0.94
EmbraceBERTwithProj-bs48-p_attclsqw	84.00	82.86	83.29	85.00	86.29	83.57	84.29	86.86	84.86	84.43	84.54	1.20
EBERTconcatAtt-bs48	83.57	82.14	85.86	81.71	84.00	80.00	82.71	83.14	81.14	82.71	82.70	1.54
EBERTconcatAtt-bs48-p_attclsqw	83.00	83.86	85.00	84.57	82.14	84.43	83.29	84.57	86.14	85.14	84.21	1.11
EBERTconcatAttwithProj-bs48	86.00	86.00	87.00	82.86	83.14	82.71	84.00	85.00	86.00	83.71	84.64	1.47
EBERTconcatAttwithProj-bs48-p_attclsqw	84.00	83.57	83.29	85.43	85.14	83.14	83.86	84.29	85.86	81.71	84.03	1.16
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	83.57	83.57	85.14	82.43	83.71	83.71	83.14	84.57	86.43	84.29	84.06	1.06
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	82.43	82.86	85.14	82.86	82.00	84.43	80.71	82.43	83.14	83.00	82.90	1.16
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	84.43	84.43	85.29	83.14	83.57	85.71	85.14	83.86	86.57	85.29	84.74	1.00
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	85.14	84.29	84.57	83.29	83.00	84.57	85.00	86.29	83.29	84.29	84.37	0.95
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	83.00	86.14	82.71	84.71	83.29	82.29	83.71	82.43	81.86	81.71	83.19	1.30
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	80.71	85.14	82.71	84.00	82.14	86.71	84.14	81.14	84.86	84.29	83.59	1.78
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	84.71	82.43	83.29	85.29	85.43	84.86	82.43	83.29	83.14	86.14	84.10	1.27
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	85.57	84.57	83.29	82.00	82.43	82.43	82.43	83.29	85.29	81.14	83.24	1.39

gtts-witai - not good with any

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	98.86	98.43	98.29	98.57	98.29	98.14	98.14	98.57	98.00	98.57	98.39	0.25
BERT-bs16+Dropout0.1	98.86	98.86	98.29	98.71	98.29	98.29	98.86	98.57	98.43	98.43	98.56	0.23
BERT-bs16+Dropout0.3	98.43	98.71	98.29	98.86	98.43	98.57	98.43	98.00	98.57	98.29	98.46	0.23
FrozenBERT-bs16-ep100	98.57	98.29	98.14	98.14	98.00	98.43	98.29	98.43	97.71	98.43	98.24	0.24
FrozenBERT-bs16-ep100+Dropout0.1	98.86	98.57	98.29	98.43	98.14	98.29	98.57	98.43	98.57	98.29	98.44	0.20
FrozenBERT-bs16-ep100+Dropout0.3	98.43	98.57	98.14	99.00	98.43	98.14	98.14	98.00	98.57	98.29	98.37	0.28
EmbraceBERT-bs16	98.57	98.43	98.71	98.14	97.71	98.57	98.14	98.43	98.57	98.00	98.33	0.30
EmbraceBERT-bs16+Dropout0.1	98.43	97.43	98.14	98.57	98.29	98.86	98.57	98.71	98.71	98.86	98.46	0.41
EmbraceBERT-bs16+Dropout0.3	98.00	98.71	98.29	98.43	97.71	98.29	98.43	98.43	98.57	98.71	98.36	0.29
CondensedEmbraceBERT-bs16	15.57	3.14	9.00	15.29	12.43	9.57	0.71	18.86	7.43	1.14	9.31	5.98
CondensedEmbraceBERT-bs16+Dropout0.1	98.29	98.29	97.71	98.57	98.43	98.29	98.29	98.57	98.57	97.86	98.29	0.28
CondensedEmbraceBERT-bs16+Dropout0.3	98.43	98.71	97.57	98.29	98.14	98.43	98.14	98.71	98.71	98.00	98.31	0.35
FrozenEBERT-bs16-ep100	99.00	96.71	99.14	98.57	98.14	98.29	98.71	97.86	98.43	97.86	98.27	0.66
FrozenEBERT-bs16-ep100+Dropout0.1	98.14	98.71	97.86	98.43	98.14	98.43	98.14	98.86	98.00	98.57	98.33	0.31
FrozenEBERT-bs16-ep100+Dropout0.3	98.29	98.00	98.57	98.57	98.14	98.43	98.00	98.57	98.57	98.43	98.36	0.22
FrozenCEBERT-bs16-ep100	11.71	0.43	5.86	13.00	11.57	5.14	1.71	19.86	20.86	1.29	9.14	7.06
FrozenCEBERT-bs16-ep100+Dropout0.1	98.71	98.43	97.43	98.14	97.43	98.14	97.86	98.57	98.14	98.29	98.11	0.41
FrozenCEBERT-bs16-ep100+Dropout0.3	98.57	98.86	98.57	98.43	97.86	98.14	97.71	98.71	98.29	98.14	98.33	0.35
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	98.14	97.71	97.86	98.43	98.57	98.14	98.86	98.43	98.57	98.14	98.29	0.33
EmbraceBERT-bs16+Branches+Dropout0.1	98.57	98.29	98.00	98.43	98.14	98.57	98.57	98.29	99.43	98.86	98.51	0.38
EmbraceBERT-bs16+Branches+Dropout0.3	98.43	98.43	98.43	98.43	98.14	97.43	98.71	98.57	99.00	98.14	98.37	0.39
CondensedEmbraceBERT-bs16+Branches	98.43	98.86	99.00	98.14	98.29	98.43	98.29	98.57	98.00	98.57	98.46	0.29
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	98.71	99.00	98.00	97.86	98.57	98.00	98.43	98.57	98.71	97.86	98.37	0.39
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	98.71	98.00	98.14	98.29	97.71	98.00	97.86	98.57	98.29	98.14	98.17	0.29
FrozenEmbraceBERT-bs16+Branches	98.43	98.57	96.86	98.43	97.71	98.86	98.14	98.43	98.29	98.43	98.21	0.53
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	98.57	98.29	98.29	98.57	97.29	98.29	97.86	98.86	98.57	99.14	98.37	0.49
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	98.14	98.86	98.29	98.43	98.14	98.43	98.00	98.86	99.29	98.14	98.46	0.39
FrozenCEBERT-bs16+Branches	98.71	98.43	98.43	98.14	98.43	98.71	98.43	98.57	98.29	97.86	98.40	0.25
FrozenCEBERT-bs16+Branches+Dropout0.1	98.29	98.71	98.00	98.43	98.29	98.57	98.29	98.29	98.86	98.00	98.37	0.26
FrozenCEBERT-bs16+Branches+Dropout0.3	98.86	98.43	98.43	98.43	98.14	98.71	98.57	98.00	99.00	97.43	98.40	0.43
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	99.00	98.29	98.43	98.14	98.29	98.57	98.43	98.71	98.29	98.57	98.47	0.24
BERT-bs32+Dropout0.1	98.57	98.71	98.00	98.43	98.14	98.57	98.71	98.14	98.43	98.57	98.43	0.24
BERT-bs32+Dropout0.3	98.86	98.43	98.29	98.43	98.29	98.00	98.57	98.86	98.43	98.29	98.44	0.25
FrozenBERT-bs32-ep100	99.00	98.14	98.43	98.57	98.29	98.43	98.43	98.43	98.57	98.71	98.50	0.22
FrozenBERT-bs32-ep100+Dropout0.1	98.29	98.57	98.14	98.86	98.14	98.29	98.57	98.14	98.57	98.29	98.39	0.23
FrozenBERT-bs32-ep100+Dropout0.3	98.29	98.86	97.71	98.86	98.29	99.00	98.14	98.14	99.00	98.00	98.43	0.44
EmbraceBERT-bs32	98.71	98.43	98.00	98.43	98.14	98.29	98.86	98.57	98.29	98.29	98.40	0.25
EmbraceBERT-bs32+Dropout0.1	98.43	98.29	98.86	98.71	98.57	98.00	98.29	98.43	98.71	98.57	98.49	0.24
EmbraceBERT-bs32+Dropout0.3	98.86	98.57	98.57	98.43	97.71	98.29	98.00	98.43	98.57	98.29	98.37	0.31
CondensedEmbraceBERT-bs32	10.29	9.00	6.43	2.00	15.43	9.14	2.86	27.00	29.71	1.14	11.30	9.48
CondensedEmbraceBERT-bs32+Dropout0.1	98.57	99.14	98.43	98.29	98.29	98.71	98.57	99.00	97.86	97.43	98.43	0.48
CondensedEmbraceBERT-bs32+Dropout0.3	98.57	98.43	98.43	98.57	98.43	98.43	98.57	98.43	98.00	98.14	98.40	0.18
FrozenEBERT-bs32-ep100	98.71	98.29	98.00	98.43	98.00	98.14	98.00	98.43	98.14	98.14	98.23	0.22
FrozenEBERT-bs32-ep100+Dropout0.1	98.14	98.29	98.14	98.43	98.43	98.57	98.29	98.29	98.29	98.43	98.33	0.13
FrozenEBERT-bs32-ep100+Dropout0.3	98.29	98.57	98.57	98.57	98.29	98.71	98.29	98.71	98.57	98.43	98.50	0.16
FrozenCEBERT-bs32-ep100	10.86	8.71	6.43	2.43	16.29	9.57	3.00	26.43	30.14	1.00	11.49	9.46
FrozenCEBERT-bs32-ep100+Dropout0.1	98.57	98.43	98.00	97.71	98.29	98.57	98.00	97.71	97.86	98.86	98.20	0.38
FrozenCEBERT-bs32-ep100+Dropout0.3	98.57	98.86	98.29	98.71	98.29	98.86	97.57	98.29	97.86	98.43	98.37	0.39
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32+Branches	98.00	98.57	97.57	98.57	98.57	98.43	98.43	98.43	98.71	98.29	98.36	0.32
EmbraceBERT-bs32+Branches+Dropout0.1	98.71	98.43	98.14	98.43	98.29	98.71	98.43	99.00	98.43	98.43	98.50	0.23
EmbraceBERT-bs32+Branches+Dropout0.3	98.14	97.86	98.57	98.43	98.43	98.43	98.29	99.14	98.86	98.71	98.49	0.35
CondensedEmbraceBERT-bs32+Branches	98.29	99.00	98.29	98.14	98.57	98.14	98.29	98.14	98.29	98.14	98.33	0.26
CondensedEmbraceBERT-bs32+Branches+Dropout0.1	98.57	99.00	98.14	97.86	98.86	98.57	98.71	98.86	98.57	98.14	98.53	0.35
CondensedEmbraceBERT-bs32+Branches+Dropout0.3	98.57	99.00	98.14	98.29	98.43	98.29	97.57	98.29	98.14	98.00	98.27	0.35
FrozenEmbraceBERT-bs32+Branches	98.14	98.43	98.14	98.71	98.43	98.86	98.14	98.43	98.43	98.71	98.44	0.24
FrozenEmbraceBERT-bs32+Branches+Dropout0.1	98.00	98.14	97.71	98.43	98.57	98.29	98.43	98.43	98.86	98.29	98.31	0.30
FrozenEmbraceBERT-bs32+Branches+Dropout0.3	98.14	98.57	97.29	98.43	98.00	98.29	98.29	99.00	98.29	98.14	98.24	0.41
FrozenCEBERT-bs32+Branches	98.14	98.43	98.43	98.29	98.43	98.57	97.00	98.71	98.14	98.00	98.21	0.45
FrozenCEBERT-bs32+Branches+Dropout0.1	98.86	98.57	98.29	98.00	98.43	98.43	98.43	98.43	98.29	98.57	98.43	0.21
FrozenCEBERT-bs32+Branches+Dropout0.3	98.57	98.00	97.71	98.00	98.14	98.00	98.43	98.86	98.43	98.00	98.21	0.33
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	98.00	97.57	98.14	97.86	98.43	98.29	97.86	98.14	97.57	98.14	98.00	0.27
RoBERTa-bs16+Dropout0.1	97.43	98.29	97.71	96.86	96.86	97.86	96.86	97.14	98.00	97.00	97.40	0.51
RoBERTa-bs16+Dropout0.3	98.00	98.29	98.14	96.86	97.71	98.00	97.86	97.71	98.43	97.43	97.84	0.43
FrozenRoBERTa-bs16-ep100	96.71	98.00	97.29	97.14	97.43	96.71	96.57	98.14	96.43	98.14	97.26	0.63
FrozenRoBERTa-bs16-ep100+Dropout0.1	97.00	98.43	97.57	97.71	97.43	97.71	98.14	97.43	97.86	97.71	97.70	0.37
FrozenRoBERTa-bs16-ep100+Dropout0.3	97.86	95.57	98.29	97.14	98.00	97.86	98.14	98.43	98.14	96.86	97.63	0.83
EmbraceRoBERTa-bs16	97.86	98.00	97.71	98.57	96.86	98.43	98.43	97.86	97.00	97.71	97.84	0.54
EmbraceRoBERTa-bs16+Dropout0.1	97.14	97.86	98.00	96.86	98.29	97.71	98.00	96.57	97.43	97.71	97.56	0.52
EmbraceRoBERTa-bs16+Dropout0.3	97.14	96.71	98.14	98.29	97.29	98.00	97.86	96.29	97.86	98.00	97.56	0.63
CondensedEmbraceRoBERTa-bs16	2.14	4.29	28.00	14.86	1.29	25.71	4.71	8.43	12.29	16.00	11.77	8.97
CondensedEmbraceRoBERTa-bs16+Dropout0.1	97.86	97.86	96.86	98.14	97.14	97.71	98.43	97.57	97.71	97.86	97.71	0.43
CondensedEmbraceRoBERTa-bs16+Dropout0.3	97.71	97.29	97.86	98.00	97.43	98.29	98.43	97.43	96.71	97.43	97.66	0.48
FrozenERoBERTa-bs16-ep100	97.57	97.43	98.00	97.43	98.14	98.00	97.86	98.29	98.29	98.71	97.97	0.39
FrozenERoBERTa-bs16-ep100+Dropout0.1	97.14	98.29	96.86	97.86	97.71	98.00	97.57	98.43	96.71	97.43	97.60	0.55
FrozenERoBERTa-bs16-ep100+Dropout0.3	97.43	97.86	98.29	97.43	97.29	97.71	98.71	97.43	96.71	97.14	97.60	0.55
FrozenCERoBERTa-bs16-ep100	0.71	17.43	29.14	14.57	0.57	24.29	7.00	5.00	12.71	4.29	11.57	9.33
FrozenCERoBERTa-bs16-ep100+Dropout0.1	97.86	98.00	98.14	96.43	98.71	97.43	97.43	97.14	97.57	97.43	97.61	0.59
FrozenCERoBERTa-bs16-ep100+Dropout0.3	97.14	98.29	97.86	98.43	97.71	97.71	97.71	97.00	98.57	98.29	97.87	0.50
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	98.43	98.14	98.14	98.00	97.86	97.86	98.29	98.14	97.29	97.86	98.00	0.30
EmbraceRoBERTa-bs16+Branches+Dropout0.1	98.29	97.57	97.00	98.00	97.29	98.14	98.43	97.43	97.71	97.86	97.77	0.43
EmbraceRoBERTa-bs16+Branches+Dropout0.3	97.14	97.29	97.71	98.00	97.57	98.14	98.43	97.71	98.29	98.29	97.86	0.42
CondensedEmbraceRoBERTa-bs16+Branches	96.57	98.14	98.00	98.14	97.29	97.71	98.14	98.71	97.00	97.86	97.76	0.60
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	97.86	98.00	98.29	98.14	98.57	98.00	98.00	97.86	98.43	96.71	97.99	0.48
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	97.86	97.86	98.14	97.29	97.86	97.57	98.29	98.71	97.29	97.43	97.83	0.44
FrozenEmbraceRoBERTa-bs16+Branches	97.57	98.57	97.86	97.86	97.57	98.29	97.43	98.00	98.57	96.71	97.84	0.54
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	96.43	96.71	97.86	97.86	97.86	98.14	98.00	98.00	98.14	97.71	97.67	0.57
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	97.29	97.86	97.86	97.14	97.43	97.71	98.00	98.14	97.57	97.86	97.69	0.31
FrozenCERoBERTa-bs16+Branches	97.29	98.43	96.86	97.00	97.86	97.71	98.43	97.86	97.14	98.00	97.66	0.54
FrozenCERoBERTa-bs16+Branches+Dropout0.1	96.86	98.57	98.00	98.14	97.71	98.00	97.86	97.57	98.43	97.57	97.87	0.46
FrozenCERoBERTa-bs16+Branches+Dropout0.3	98.00	98.14	97.86	98.29	96.86	96.86	96.86	97.14	97.86	97.71	97.56	0.54
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	98.43	97.86	98.14	98.14	98.43	98.14	98.00	98.00	98.14	98.43	98.17	0.19
RoBERTa-bs32+Dropout0.1	97.86	98.43	97.29	97.14	97.57	97.86	97.86	98.14	98.14	98.29	97.86	0.40
RoBERTa-bs32+Dropout0.3	96.29	97.86	98.00	97.14	97.86	98.00	98.00	98.14	98.14	97.14	97.66	0.58
FrozenRoBERTa-bs32-ep100	97.86	98.14	97.00	97.71	97.14	96.57	98.29	96.71	98.14	98.00	97.56	0.61
FrozenRoBERTa-bs32-ep100+Dropout0.1	97.57	98.14	98.29	98.14	97.14	97.14	98.00	96.57	98.00	98.29	97.73	0.56
FrozenRoBERTa-bs32-ep100+Dropout0.3	97.71	97.57	97.86	98.00	97.43	97.86	98.00	97.86	96.71	97.71	97.67	0.36
EmbraceRoBERTa-bs32	98.43	97.71	98.00	97.29	98.57	97.71	98.29	97.29	97.86	98.00	97.91	0.42
EmbraceRoBERTa-bs32+Dropout0.1	98.14	97.86	98.29	98.00	98.14	97.29	98.00	98.71	97.43	97.86	97.97	0.39
EmbraceRoBERTa-bs32+Dropout0.3	98.29	97.57	98.29	98.00	97.43	97.57	97.57	97.14	96.71	97.29	97.59	0.47
CondensedEmbraceRoBERTa-bs32	10.86	1.00	27.43	14.71	1.57	15.14	2.71	1.14	8.29	9.00	9.19	7.96
CondensedEmbraceRoBERTa-bs32+Dropout0.1	97.00	98.57	97.43	97.29	97.86	98.14	97.43	97.71	98.14	97.71	97.73	0.44
CondensedEmbraceRoBERTa-bs32+Dropout0.3	97.71	97.86	97.71	97.57	98.00	97.86	98.14	97.43	97.71	97.71	97.77	0.19
FrozenERoBERTa-bs32-ep100	98.14	98.00	98.43	96.71	97.86	97.71	97.43	97.57	97.29	97.57	97.67	0.46
FrozenERoBERTa-bs32-ep100+Dropout0.1	97.86	97.86	97.57	97.57	98.14	97.86	98.00	97.29	94.71	97.86	97.47	0.95
FrozenERoBERTa-bs32-ep100+Dropout0.3	97.71	98.00	97.43	98.00	98.00	98.00	97.71	98.14	97.57	94.86	97.54	0.92
FrozenCERoBERTa-bs32-ep100	1.71	0.14	37.29	14.43	3.43	19.43	4.86	0.86	8.43	1.57	9.21	11.13
FrozenCERoBERTa-bs32-ep100+Dropout0.1	97.00	98.14	98.00	97.29	97.57	97.71	98.00	98.00	97.29	97.71	97.67	0.36
FrozenCERoBERTa-bs32-ep100+Dropout0.3	98.00	98.43	98.00	97.86	98.29	97.43	97.71	96.86	98.00	96.14	97.67	0.66
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Branches	97.71	97.71	97.86	98.29	97.86	96.57	95.71	97.29	98.57	98.14	97.57	0.81
EmbraceRoBERTa-bs32+Branches+Dropout0.1	98.43	98.57	98.29	97.43	97.86	97.00	97.43	98.43	98.43	97.57	97.94	0.53
EmbraceRoBERTa-bs32+Branches+Dropout0.3	97.86	98.14	98.29	97.71	98.14	98.00	98.14	98.57	97.00	98.14	98.00	0.40
CondensedEmbraceRoBERTa-bs32+Branches	98.14	98.29	98.14	98.29	97.86	97.71	97.71	97.57	98.00	97.14	97.89	0.34
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.1	97.29	98.00	97.29	96.86	96.43	98.57	97.71	97.43	97.29	98.57	97.54	0.65
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.3	98.14	98.57	97.86	97.00	97.71	97.43	98.43	97.14	97.71	98.29	97.83	0.51
FrozenEmbraceRoBERTa-bs32+Branches	97.86	98.43	97.86	98.43	98.00	97.57	97.43	96.43	98.00	98.00	97.80	0.55
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.1	98.29	98.00	98.43	96.71	98.00	98.00	98.00	97.43	97.71	97.71	97.83	0.46
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.3	98.29	98.29	97.00	96.86	97.57	98.14	98.14	97.29	97.29	97.57	97.64	0.51
FrozenCERoBERTa-bs32+Branches	98.14	97.86	97.86	98.29	97.57	98.00	97.86	96.57	97.86	97.57	97.76	0.45
FrozenCERoBERTa-bs32+Branches+Dropout0.1	97.71	97.86	98.57	98.00	97.29	98.71	98.43	97.14	97.57	98.00	97.93	0.50
FrozenCERoBERTa-bs32+Branches+Dropout0.3	98.00	98.00	98.14	98.14	98.00	97.00	98.43	97.00	98.29	98.29	97.93	0.48

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	98.43	98.57	98.43	98.29	98.29	98.14	98.14	98.43	98.71	98.43	98.39	0.17
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	98.57	97.86	98.43	98.00	98.29	98.43	98.43	98.57	98.57	98.29	98.34	0.23
BERTwithAttClsProjection-bs32	98.71	98.43	98.71	98.86	98.29	98.29	98.43	98.43	98.43	98.71	98.53	0.19
BERTwithProjection-bs32	98.71	98.71	98.14	98.86	98.57	98.57	98.43	98.86	98.71	98.43	98.60	0.21
BERTwithProjectionAtt-bs32	98.71	98.43	98.57	98.71	98.57	98.43	98.57	98.71	98.43	98.71	98.59	0.12
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	98.43	98.43	98.57	98.14	98.43	98.29	98.57	98.00	98.57	98.71	98.41	0.21
EmbraceBERT-bs32-p_attclsqw	98.57	98.43	98.29	98.86	98.71	98.57	98.00	98.86	98.43	98.29	98.50	0.26
EmbraceBERTwithProj-bs32	98.57	98.57	98.43	98.57	98.57	98.43	98.86	98.57	98.00	98.43	98.50	0.20
EmbraceBERTwithProj-bs32-p_attclsqw	99.00	98.57	98.57	98.14	98.71	98.71	98.43	98.71	98.29	98.71	98.59	0.23
EBERTconcatAtt-bs32	98.43	98.57	98.43	98.29	98.57	98.57	98.43	98.43	98.29	98.57	98.46	0.11
EBERTconcatAtt-bs32-p_attclsqw	98.29	98.57	98.29	98.29	98.57	98.71	98.43	98.43	98.43	98.71	98.47	0.16
EBERTconcatAttwithProj-bs32	98.71	98.43	99.00	98.29	98.29	98.43	98.71	98.71	98.71	98.29	98.56	0.23
EBERTconcatAttwithProj-bs32-p_attclsqw	98.71	98.29	98.43	98.57	98.43	98.71	98.71	98.00	98.43	98.86	98.51	0.24
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery	98.86	98.86	98.71	98.71	98.86	98.57	98.57	98.57	98.57	98.57	98.69	0.12
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	97.86	98.57	98.71	98.29	98.57	98.43	98.86	98.57	98.71	98.86	98.54	0.28
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery	98.71	98.57	98.57	99.00	98.57	98.57	98.14	98.71	98.71	98.71	98.63	0.20
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	98.57	98.57	98.29	98.14	98.43	98.57	98.71	98.29	98.71	98.86	98.51	0.21
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery	98.14	98.71	98.29	98.43	98.57	98.14	98.57	98.29	98.71	98.71	98.46	0.22
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	98.57	98.29	98.57	98.43	98.14	98.43	98.71	98.71	98.71	98.29	98.49	0.19
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery	98.71	98.86	98.71	98.71	98.43	98.57	98.43	98.29	98.71	98.57	98.60	0.17
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	98.71	98.57	98.29	98.86	98.14	98.71	98.57	98.86	98.71	98.29	98.57	0.24

• Epoch 100 - bs48 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	98.00	98.57	98.29	98.57	98.00	98.29	98.29	98.43	98.14	98.29	98.29	0.19
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	98.00	98.29	98.57	98.86	98.57	98.57	98.14	98.29	98.71	98.86	98.49	0.28
BERTwithAttClsProjection-bs48	98.71	98.86	98.43	98.71	98.57	98.71	98.71	98.71	98.14	98.57	98.61	0.19
BERTwithProjection-bs48	98.71	98.57	98.43	98.29	98.43	98.57	98.43	98.57	98.57	98.57	98.51	0.11
BERTwithProjectionAtt-bs48	98.43	98.86	98.43	98.43	98.57	98.71	98.29	98.57	98.14	98.43	98.49	0.19
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	98.29	98.71	98.29	98.29	98.71	98.71	98.00	98.43	98.86	98.43	98.47	0.26
EmbraceBERT-bs48-p_attclsqw	98.57	98.29	98.29	98.86	98.14	98.71	98.57	98.57	98.43	98.29	98.47	0.21
EmbraceBERTwithProj-bs48	98.57	98.71	98.57	98.57	98.57	98.00	98.29	98.57	98.29	98.57	98.47	0.20
EmbraceBERTwithProj-bs48-p_attclsqw	98.71	98.71	98.57	99.14	98.57	98.57	98.57	98.86	98.57	98.71	98.70	0.17
EBERTconcatAtt-bs48	98.57	98.14	98.71	99.00	98.43	99.00	98.57	98.29	98.57	98.71	98.60	0.26
EBERTconcatAtt-bs48-p_attclsqw	98.57	98.57	98.57	98.43	98.43	98.29	98.29	98.57	98.57	98.29	98.46	0.12
EBERTconcatAttwithProj-bs48	98.57	98.57	98.57	98.43	98.57	98.57	98.43	98.86	98.43	98.71	98.57	0.13
EBERTconcatAttwithProj-bs48-p_attclsqw	98.43	98.43	98.86	98.57	98.71	98.71	98.43	98.29	98.71	98.43	98.56	0.17
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	98.43	98.14	98.71	98.71	98.71	98.29	98.14	98.57	98.14	98.86	98.47	0.26
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.29	98.29	98.86	98.71	98.43	98.57	98.29	98.57	98.29	98.86	98.51	0.22
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	98.57	98.71	98.71	98.86	98.71	99.00	98.43	98.43	98.57	98.57	98.66	0.17
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.71	98.43	98.00	98.43	98.71	98.57	98.57	98.86	98.86	98.43	98.56	0.24
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	98.57	98.57	98.86	98.86	98.00	98.71	98.57	98.57	98.57	98.00	98.53	0.29
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.86	98.57	98.57	98.14	98.29	99.00	98.71	98.29	98.43	98.29	98.51	0.26
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	98.29	98.57	98.86	98.71	98.71	99.00	98.43	98.57	98.71	98.86	98.67	0.20
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	98.71	98.43	98.14	98.14	98.57	98.57	98.57	98.86	98.57	98.57	98.51	0.21

macsay-google - good with all but RoBERTa bs16

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	73.71	74.43	75.43	75.43	72.00	75.86	75.86	73.43	72.71	75.14	74.40	1.30
BERT-bs16+Dropout0.1	74.14	77.86	76.29	72.29	74.43	77.71	75.14	70.86	74.43	74.14	74.73	2.08
BERT-bs16+Dropout0.3	76.14	72.43	73.43	78.43	73.14	77.29	74.00	72.00	74.57	77.71	74.91	2.20
FrozenBERT-bs16-ep100	71.86	75.14	76.43	76.43	73.00	75.57	73.14	75.86	71.29	72.71	74.14	1.85
FrozenBERT-bs16-ep100+Dropout0.1	71.86	75.57	71.43	70.29	78.43	75.00	77.29	74.86	72.71	72.71	74.01	2.51
FrozenBERT-bs16-ep100+Dropout0.3	72.71	77.14	74.43	71.43	74.71	77.86	71.14	75.57	73.43	77.29	74.57	2.28
EmbraceBERT-bs16	76.57	71.14	73.86	74.71	68.57	74.86	75.00	73.86	73.14	70.71	73.24	2.29
EmbraceBERT-bs16+Dropout0.1	74.43	71.14	70.14	73.14	74.14	74.43	75.00	74.43	71.00	72.86	73.07	1.64
EmbraceBERT-bs16+Dropout0.3	74.14	75.29	77.29	75.00	70.86	74.43	73.86	75.29	72.43	73.86	74.24	1.65
CondensedEmbraceBERT-bs16	13.71	10.43	11.57	15.14	11.00	11.71	8.14	18.43	13.86	8.29	12.23	2.99
CondensedEmbraceBERT-bs16+Dropout0.1	73.14	76.29	74.43	73.43	74.29	76.71	73.71	75.71	73.71	73.00	74.44	1.27
CondensedEmbraceBERT-bs16+Dropout0.3	71.00	74.00	77.14	76.14	73.00	75.00	71.29	77.57	72.43	75.71	74.33	2.23
FrozenEBERT-bs16-ep100	73.00	68.71	74.29	74.86	71.29	75.29	78.00	73.29	73.43	72.14	73.43	2.36
FrozenEBERT-bs16-ep100+Dropout0.1	72.86	75.57	72.14	70.57	70.86	78.00	73.00	74.29	71.43	75.71	73.44	2.29
FrozenEBERT-bs16-ep100+Dropout0.3	68.29	70.57	77.86	71.14	73.00	72.43	72.00	74.14	73.71	69.00	72.21	2.61
FrozenCEBERT-bs16-ep100	12.00	7.43	7.14	11.57	9.86	7.86	7.14	17.00	28.71	10.00	11.87	6.31
FrozenCEBERT-bs16-ep100+Dropout0.1	79.43	76.29	74.00	69.57	71.00	72.29	73.57	79.86	74.57	71.43	74.20	3.28
FrozenCEBERT-bs16-ep100+Dropout0.3	75.71	76.57	77.43	76.14	68.29	72.86	74.71	78.29	75.71	74.29	75.00	2.68
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	72.29	69.86	72.43	72.71	72.57	75.00	75.57	70.57	71.43	73.86	72.63	1.71
EmbraceBERT-bs16+Branches+Dropout0.1	72.43	70.00	73.86	73.29	73.29	78.29	74.71	72.00	77.14	78.14	74.31	2.62
EmbraceBERT-bs16+Branches+Dropout0.3	74.43	74.57	72.14	76.57	73.29	72.71	74.43	73.43	73.00	76.14	74.07	1.37
CondensedEmbraceBERT-bs16+Branches	76.29	72.86	75.29	71.14	73.71	72.43	75.29	76.14	72.14	75.57	74.09	1.76
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	74.57	76.57	75.86	73.29	76.43	72.57	73.71	71.43	72.86	70.71	73.80	1.93
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	72.57	78.57	77.86	76.71	71.29	75.14	71.29	74.57	74.71	74.43	74.71	2.39
FrozenEmbraceBERT-bs16+Branches	75.43	72.57	71.14	77.14	75.86	76.14	76.14	75.57	69.43	76.00	74.54	2.43
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	75.00	76.29	77.43	76.29	71.14	73.29	72.29	71.86	73.86	74.29	74.17	1.98
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	74.29	75.86	75.57	72.71	73.57	75.43	69.86	75.00	76.86	77.00	74.61	2.04
FrozenCEBERT-bs16+Branches	78.86	76.57	76.14	72.57	73.43	76.71	73.14	74.43	74.86	76.00	75.27	1.84
FrozenCEBERT-bs16+Branches+Dropout0.1	73.71	76.71	75.43	70.29	73.29	74.14	72.57	75.00	76.29	74.57	74.20	1.79
FrozenCEBERT-bs16+Branches+Dropout0.3	74.71	74.86	79.29	74.57	76.29	76.43	75.57	76.43	77.29	71.86	75.73	1.86
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	75.86	75.00	75.71	72.71	72.43	74.57	76.57	73.57	72.71	76.71	74.59	1.56
BERT-bs32+Dropout0.1	73.29	74.14	75.57	73.57	74.86	72.71	75.71	75.00	73.57	77.43	74.59	1.34
BERT-bs32+Dropout0.3	73.29	78.57	77.14	74.29	76.14	74.43	74.86	77.14	72.57	75.71	75.41	1.78
FrozenBERT-bs32-ep100	75.57	74.00	76.00	73.57	72.14	73.71	74.00	72.14	73.57	74.43	73.91	1.18
FrozenBERT-bs32-ep100+Dropout0.1	81.29	72.00	73.00	76.43	74.57	75.57	73.00	73.00	74.00	76.43	74.93	2.57
FrozenBERT-bs32-ep100+Dropout0.3	72.86	77.00	74.14	73.29	73.86	72.57	74.57	75.71	72.00	76.57	74.26	1.62
EmbraceBERT-bs32	73.57	75.00	74.29	72.43	68.57	73.14	74.14	72.71	77.29	74.86	73.60	2.13
EmbraceBERT-bs32+Dropout0.1	70.57	72.14	74.57	76.14	73.43	75.86	76.71	79.43	74.71	71.14	74.47	2.59
EmbraceBERT-bs32+Dropout0.3	75.57	72.14	79.14	72.86	73.00	70.57	71.00	72.29	73.43	73.57	73.36	2.34
CondensedEmbraceBERT-bs32	13.57	10.57	9.43	11.14	16.71	12.00	7.86	25.29	29.57	9.71	14.59	6.89
CondensedEmbraceBERT-bs32+Dropout0.1	76.57	75.43	74.71	72.86	72.71	73.14	75.14	83.86	77.43	76.71	75.86	3.10
CondensedEmbraceBERT-bs32+Dropout0.3	75.86	71.57	76.29	72.29	74.29	74.43	72.14	76.57	72.00	75.14	74.06	1.82
FrozenEBERT-bs32-ep100	74.43	73.00	77.14	70.43	69.86	73.86	72.14	73.00	71.14	71.71	72.67	2.03
FrozenEBERT-bs32-ep100+Dropout0.1	73.14	73.86	72.71	73.43	72.43	72.71	76.14	74.71	74.14	71.29	73.46	1.28
FrozenEBERT-bs32-ep100+Dropout0.3	73.86	71.86	73.71	74.43	71.00	71.86	70.57	74.86	74.57	71.86	72.86	1.51
FrozenCEBERT-bs32-ep100	14.29	11.43	9.57	9.57	17.29	12.00	9.14	25.14	28.57	10.71	14.77	6.52
FrozenCEBERT-bs32-ep100+Dropout0.1	75.29	75.43	73.00	72.00	71.57	72.43	75.00	74.00	69.71	72.29	73.07	1.75
FrozenCEBERT-bs32-ep100+Dropout0.3	75.43	72.71	74.14	74.14	73.00	72.57	73.71	77.29	73.86	75.29	74.21	1.37
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32+Branches	72.71	73.29	71.57	73.00	70.86	72.71	73.57	73.43	75.14	76.29	73.26	1.49
EmbraceBERT-bs32+Branches+Dropout0.1	76.14	73.14	72.14	74.14	74.43	72.57	74.29	76.86	73.43	72.57	73.97	1.47
EmbraceBERT-bs32+Branches+Dropout0.3	74.00	75.29	73.29	73.43	72.86	74.14	77.14	72.43	72.71	74.43	73.97	1.34
CondensedEmbraceBERT-bs32+Branches	74.71	72.71	73.43	73.29	75.71	73.86	73.43	78.14	69.71	74.29	73.93	2.05
CondensedEmbraceBERT-bs32+Branches+Dropout0.1	73.43	74.57	71.29	76.43	73.86	76.29	72.29	73.57	74.43	74.43	74.06	1.50
CondensedEmbraceBERT-bs32+Branches+Dropout0.3	76.29	74.86	72.57	75.00	73.86	71.86	70.43	71.43	73.00	77.29	73.66	2.09
FrozenEmbraceBERT-bs32+Branches	74.29	77.29	77.14	72.43	71.71	71.43	70.86	78.43	74.00	71.71	73.93	2.64
FrozenEmbraceBERT-bs32+Branches+Dropout0.1	75.29	68.57	74.00	71.29	74.29	79.29	72.71	78.86	72.57	70.43	73.73	3.25
FrozenEmbraceBERT-bs32+Branches+Dropout0.3	70.57	72.14	72.00	70.71	72.57	72.29	73.71	74.57	73.71	72.29	72.46	1.21
FrozenCEBERT-bs32+Branches	72.71	70.86	73.57	76.71	74.71	77.00	74.14	72.71	72.00	79.43	74.39	2.49
FrozenCEBERT-bs32+Branches+Dropout0.1	80.00	77.71	74.29	69.71	72.43	74.29	72.57	78.29	71.71	75.86	74.69	3.09
FrozenCEBERT-bs32+Branches+Dropout0.3	74.00	70.29	72.86	76.57	75.00	77.00	69.29	78.00	74.00	73.57	74.06	2.65
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	77.43	75.43	75.86	76.29	75.71	77.00	77.43	74.43	74.71	75.71	76.00	0.99
RoBERTa-bs16+Dropout0.1	73.29	76.86	74.14	71.57	79.29	72.71	73.00	74.57	75.43	74.57	74.54	2.12
RoBERTa-bs16+Dropout0.3	74.43	74.00	73.43	73.14	72.29	71.43	77.29	72.86	74.86	75.43	73.91	1.60
FrozenRoBERTa-bs16-ep100	73.71	73.43	73.57	70.14	79.86	73.43	75.71	74.43	68.71	73.29	73.63	2.83
FrozenRoBERTa-bs16-ep100+Dropout0.1	75.00	73.29	72.29	73.14	72.29	74.57	75.00	75.00	72.00	76.86	73.94	1.50
FrozenRoBERTa-bs16-ep100+Dropout0.3	70.43	69.14	73.43	72.14	73.57	74.57	73.43	74.86	69.86	73.57	72.50	1.91
EmbraceRoBERTa-bs16	75.14	74.43	73.00	71.86	72.29	73.86	74.14	73.71	72.00	76.43	73.69	1.38
EmbraceRoBERTa-bs16+Dropout0.1	75.43	71.29	73.71	75.29	78.14	77.43	78.29	72.00	73.14	74.57	74.93	2.34
EmbraceRoBERTa-bs16+Dropout0.3	75.71	72.86	73.57	75.57	78.43	75.00	73.57	71.14	74.00	79.00	74.89	2.30
CondensedEmbraceRoBERTa-bs16	7.29	11.57	30.57	9.86	5.29	24.14	13.29	11.43	20.14	16.00	14.96	7.49
CondensedEmbraceRoBERTa-bs16+Dropout0.1	71.29	75.00	73.86	71.14	75.71	75.71	73.57	76.29	73.14	75.71	74.14	1.77
CondensedEmbraceRoBERTa-bs16+Dropout0.3	71.43	75.71	74.00	76.14	75.00	75.71	76.57	74.00	75.43	77.14	75.11	1.56
FrozenERoBERTa-bs16-ep100	75.86	74.71	74.57	75.29	72.29	76.29	75.00	70.71	72.00	74.29	74.10	1.73
FrozenERoBERTa-bs16-ep100+Dropout0.1	73.00	72.14	72.57	74.71	72.14	73.71	74.00	72.43	70.29	76.71	73.17	1.65
FrozenERoBERTa-bs16-ep100+Dropout0.3	74.43	72.43	72.43	74.57	75.86	74.00	75.00	72.86	73.57	79.43	74.46	1.97
FrozenCERoBERTa-bs16-ep100	7.86	12.57	30.86	6.71	3.57	22.00	16.29	4.86	17.71	10.14	13.26	8.13
FrozenCERoBERTa-bs16-ep100+Dropout0.1	74.43	69.71	74.14	70.86	73.00	71.00	79.29	79.00	71.71	74.14	73.73	3.10
FrozenCERoBERTa-bs16-ep100+Dropout0.3	75.71	76.00	71.71	73.14	74.71	72.57	72.57	79.71	72.00	74.14	74.23	2.32
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	71.57	77.00	73.43	75.00	73.86	73.00	73.29	75.00	78.86	77.00	74.80	2.12
EmbraceRoBERTa-bs16+Branches+Dropout0.1	71.57	74.00	76.14	76.14	74.00	75.00	73.71	74.43	72.71	72.86	74.06	1.39
EmbraceRoBERTa-bs16+Branches+Dropout0.3	74.57	73.14	73.86	74.14	75.86	74.57	74.86	73.14	73.43	75.00	74.26	0.84
CondensedEmbraceRoBERTa-bs16+Branches	76.71	75.43	77.14	71.71	75.14	75.00	71.86	74.86	73.14	77.00	74.80	1.88
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	71.43	74.29	73.71	72.57	75.29	76.71	72.86	76.71	71.71	77.14	74.24	2.03
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	72.57	76.86	71.86	75.57	77.29	77.71	74.14	74.86	73.29	78.57	75.27	2.19
FrozenEmbraceRoBERTa-bs16+Branches	72.14	73.14	72.14	75.43	75.43	75.43	75.29	73.00	80.57	77.86	75.04	2.52
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	69.29	77.14	72.00	78.29	74.57	73.00	71.14	75.00	72.86	72.57	73.59	2.58
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	74.57	71.86	70.14	75.00	73.14	69.14	78.14	73.14	72.86	74.29	73.23	2.42
FrozenCERoBERTa-bs16+Branches	72.29	71.29	75.86	77.57	71.00	72.86	72.86	79.57	75.00	74.86	74.31	2.64
FrozenCERoBERTa-bs16+Branches+Dropout0.1	75.71	71.86	76.00	72.43	71.00	76.29	77.71	73.86	77.14	78.29	75.03	2.44
FrozenCERoBERTa-bs16+Branches+Dropout0.3	73.57	75.57	75.57	72.71	76.29	75.57	75.29	77.14	72.86	70.57	74.51	1.91
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	75.86	75.29	74.86	74.57	75.71	74.29	77.29	74.71	75.57	75.43	75.36	0.81
RoBERTa-bs32+Dropout0.1	73.57	73.00	68.86	71.86	74.29	73.00	71.71	73.29	72.43	74.00	72.60	1.48
RoBERTa-bs32+Dropout0.3	73.57	73.57	69.57	72.71	74.71	74.71	71.57	75.43	73.29	71.86	73.10	1.66
FrozenRoBERTa-bs32-ep100	74.29	71.43	71.14	72.86	79.86	77.00	72.14	74.29	73.43	76.29	74.27	2.60
FrozenRoBERTa-bs32-ep100+Dropout0.1	73.29	75.86	71.14	77.00	76.00	77.14	71.71	72.86	71.86	76.57	74.34	2.27
FrozenRoBERTa-bs32-ep100+Dropout0.3	74.71	72.86	71.71	73.00	73.71	72.43	73.71	77.29	70.57	73.86	73.39	1.72
EmbraceRoBERTa-bs32	72.71	75.29	71.71	77.43	73.71	76.00	76.14	81.29	72.57	74.29	75.11	2.68
EmbraceRoBERTa-bs32+Dropout0.1	75.71	71.57	74.57	74.71	74.14	71.00	74.00	77.14	72.57	74.29	73.97	1.75
EmbraceRoBERTa-bs32+Dropout0.3	75.86	73.86	75.71	75.71	72.86	73.00	74.00	74.00	70.43	72.29	73.77	1.64
CondensedEmbraceRoBERTa-bs32	7.00	5.71	28.71	9.71	3.86	14.71	11.29	2.29	14.00	11.00	10.83	7.13
CondensedEmbraceRoBERTa-bs32+Dropout0.1	76.71	75.00	75.86	75.57	74.29	76.57	76.29	78.14	73.71	77.14	75.93	1.27
CondensedEmbraceRoBERTa-bs32+Dropout0.3	76.00	72.71	74.71	72.57	73.14	74.00	72.43	76.86	73.29	75.43	74.11	1.48
FrozenERoBERTa-bs32-ep100	73.29	72.14	73.14	70.86	74.00	77.43	75.57	74.43	72.43	74.00	73.73	1.75
FrozenERoBERTa-bs32-ep100+Dropout0.1	72.14	72.00	71.43	73.71	72.29	73.43	77.86	73.29	73.71	76.29	73.61	1.91
FrozenERoBERTa-bs32-ep100+Dropout0.3	71.00	70.29	76.71	74.71	73.29	75.71	75.14	76.71	74.43	77.00	74.50	2.22
FrozenCERoBERTa-bs32-ep100	4.43	4.43	36.00	10.29	7.14	15.00	12.29	3.86	14.57	6.14	11.41	9.11
FrozenCERoBERTa-bs32-ep100+Dropout0.1	75.43	76.71	76.00	76.43	76.57	74.00	75.29	78.71	75.86	72.43	75.74	1.59
FrozenCERoBERTa-bs32-ep100+Dropout0.3	72.14	73.71	75.29	77.00	77.29	76.86	74.14	77.00	75.86	77.86	75.71	1.76
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Branches	76.14	73.86	74.29	75.71	76.86	76.86	78.71	72.14	71.71	73.29	74.96	2.15
EmbraceRoBERTa-bs32+Branches+Dropout0.1	76.00	73.86	77.86	77.57	73.86	74.43	72.43	74.29	71.00	75.71	74.70	2.04
EmbraceRoBERTa-bs32+Branches+Dropout0.3	76.86	76.43	72.14	74.43	73.14	75.86	77.00	72.29	73.71	77.14	74.90	1.89
CondensedEmbraceRoBERTa-bs32+Branches	70.29	71.29	75.29	78.71	76.86	75.29	77.00	77.57	74.57	77.57	75.44	2.62
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.1	74.00	74.71	73.14	74.14	72.43	73.57	77.00	79.29	74.57	78.29	75.11	2.17
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.3	79.43	76.86	76.86	70.86	71.29	78.29	75.57	75.71	73.00	76.43	75.43	2.71
FrozenEmbraceRoBERTa-bs32+Branches	73.86	74.14	73.14	74.14	73.57	75.14	78.86	72.86	73.00	79.00	74.77	2.17
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.1	77.57	77.57	74.00	75.14	72.86	71.43	71.43	75.71	76.43	75.43	74.76	2.16
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.3	74.71	75.00	76.43	78.14	76.00	73.29	72.71	75.71	80.14	72.00	75.41	2.35
FrozenCERoBERTa-bs32+Branches	74.71	75.43	74.43	74.57	75.00	75.00	78.43	76.43	73.71	74.14	75.19	1.29
FrozenCERoBERTa-bs32+Branches+Dropout0.1	72.57	76.14	74.29	70.29	71.86	73.14	74.29	76.14	72.86	74.86	73.64	1.77
FrozenCERoBERTa-bs32+Branches+Dropout0.3	72.00	76.43	78.00	71.71	73.29	74.14	73.71	78.14	71.71	75.29	74.44	2.32

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	75.29	75.43	74.43	73.43	75.00	74.14	76.00	77.14	71.57	75.71	74.81	1.46
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	73.00	74.71	76.71	75.57	74.29	75.57	77.86	74.14	76.14	76.14	75.41	1.34
BERTwithAttClsProjection-bs32	78.86	73.86	76.71	74.43	75.86	75.71	78.29	76.86	77.29	73.57	76.14	1.71
BERTwithProjection-bs32	75.71	73.43	75.86	76.00	75.29	76.00	74.00	75.14	75.86	76.14	75.34	0.88
BERTwithProjectionAtt-bs32	76.29	75.43	75.00	72.43	76.00	75.57	77.00	73.71	75.29	75.14	75.19	1.23
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	75.29	73.14	76.00	76.43	73.00	73.86	76.57	79.00	73.00	75.71	75.20	1.86
EmbraceBERT-bs32-p_attclsqw	76.00	78.00	74.71	75.29	74.71	79.43	72.29	75.29	76.43	73.00	75.51	2.02
EmbraceBERTwithProj-bs32	76.14	75.57	76.57	76.57	76.86	76.14	75.71	75.86	77.43	78.86	76.57	0.93
EmbraceBERTwithProj-bs32-p_attclsqw	77.14	76.14	76.71	73.29	76.29	77.14	74.57	76.14	75.43	77.43	76.03	1.23
EBERTconcatAtt-bs32	74.86	72.14	74.57	75.00	73.00	76.29	72.71	74.71	77.71	73.71	74.47	1.60
EBERTconcatAtt-bs32-p_attclsqw	75.57	72.86	76.43	77.29	73.86	75.86	74.14	73.86	75.71	76.71	75.23	1.39
EBERTconcatAttwithProj-bs32	78.57	73.00	76.14	73.43	75.00	75.29	74.00	78.29	74.00	75.00	75.27	1.81
EBERTconcatAttwithProj-bs32-p_attclsqw	73.43	75.71	74.29	74.86	73.86	74.71	73.57	77.71	79.43	76.14	75.37	1.84
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery	73.71	75.86	73.86	77.14	75.71	79.14	76.14	73.29	73.29	73.57	75.17	1.87
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	76.29	75.29	76.29	76.71	77.29	74.14	73.43	74.71	73.14	76.00	75.33	1.35
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery	78.00	78.00	75.43	75.00	75.86	75.14	76.86	78.14	80.29	76.86	76.96	1.59
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	75.14	77.14	72.57	74.86	75.57	76.86	73.43	76.14	76.00	74.29	75.20	1.39
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery	74.43	75.00	76.86	75.00	75.00	75.57	76.00	74.57	75.86	74.00	75.23	0.81
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	75.43	76.00	74.14	76.57	74.00	75.57	74.86	74.71	76.43	74.43	75.21	0.88
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery	76.29	74.43	72.43	75.43	74.00	76.00	75.14	78.14	74.57	74.57	75.10	1.45
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	73.86	75.71	75.43	74.71	74.57	77.57	76.86	75.71	79.29	73.57	75.73	1.67

• Epoch 100 - bs48 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	75.14	73.43	74.14	75.71	72.71	74.14	75.00	75.86	71.71	76.14	74.40	1.38
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	73.71	72.86	75.00	75.71	72.14	75.71	76.86	73.43	72.86	74.71	74.30	1.45
BERTwithAttClsProjection-bs48	77.14	74.29	72.43	72.71	79.43	75.71	78.00	75.57	74.14	77.71	75.71	2.22
BERTwithProjection-bs48	73.00	73.57	75.29	76.00	75.71	78.86	76.43	76.14	76.71	76.14	75.79	1.55
BERTwithProjectionAtt-bs48	77.29	73.29	72.86	75.29	75.57	77.43	77.29	74.86	73.29	76.57	75.37	1.68
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	74.86	71.71	75.71	75.57	72.00	75.00	76.71	73.29	74.57	75.57	74.50	1.57
EmbraceBERT-bs48-p_attclsqw	78.29	72.86	75.57	74.71	73.43	73.14	72.71	75.14	76.29	76.00	74.81	1.71
EmbraceBERTwithProj-bs48	77.43	76.00	76.43	75.14	76.43	77.57	75.14	73.00	77.43	77.14	76.17	1.36
EmbraceBERTwithProj-bs48-p_attclsqw	75.14	71.86	73.14	74.43	76.29	75.57	75.14	80.00	75.14	77.29	75.40	2.11
EBERTconcatAtt-bs48	76.57	76.71	76.29	76.29	75.71	72.86	73.57	73.00	75.00	76.14	75.21	1.44
EBERTconcatAtt-bs48-p_attclsqw	77.86	73.86	76.43	76.00	73.14	77.86	73.57	74.00	77.29	75.14	75.51	1.73
EBERTconcatAttwithProj-bs48	78.71	77.14	78.29	73.86	75.43	73.71	74.14	78.86	77.00	74.14	76.13	2.00
EBERTconcatAttwithProj-bs48-p_attclsqw	75.86	75.43	74.14	76.43	76.14	74.57	75.57	75.29	78.00	73.29	75.47	1.24
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	74.14	77.00	75.43	74.57	75.14	75.57	77.00	77.14	77.00	74.29	75.73	1.15
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	74.57	75.43	76.57	74.71	72.00	74.00	72.71	74.86	76.29	75.57	74.67	1.38
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	76.43	76.43	77.86	75.86	77.71	75.00	75.43	77.71	78.57	75.00	76.60	1.23
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	72.86	76.14	73.14	72.29	72.29	75.43	76.00	77.71	76.71	75.71	74.83	1.89
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	75.14	76.86	73.14	76.29	72.57	73.57	73.86	75.43	74.86	73.86	74.56	1.32
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	72.71	76.71	74.29	73.71	74.29	77.57	74.14	74.00	76.71	74.86	74.90	1.48
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	76.86	75.29	75.71	76.14	77.00	77.57	74.71	74.86	73.29	78.14	75.96	1.40
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	74.29	76.57	75.00	74.86	75.86	77.57	74.57	76.00	77.14	72.43	75.43	1.45

macsay-sphinx - only good with RoBERTa bs32

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	47.71	46.57	47.57	45.86	44.86	49.00	48.00	47.57	46.43	47.86	47.14	1.14
BERT-bs16+Dropout0.1	48.29	49.86	46.29	44.71	46.29	46.00	50.57	46.14	45.71	47.71	47.16	1.80
BERT-bs16+Dropout0.3	47.86	45.71	46.29	50.71	44.86	46.00	48.43	43.86	47.57	49.14	47.04	1.98
FrozenBERT-bs16-ep100	50.00	46.14	46.00	49.14	45.71	46.71	45.43	47.86	42.14	46.86	46.60	2.06
FrozenBERT-bs16-ep100+Dropout0.1	47.86	49.57	43.43	44.71	47.71	44.29	46.71	45.57	44.86	46.86	46.16	1.81
FrozenBERT-bs16-ep100+Dropout0.3	47.57	52.43	42.14	43.71	48.29	48.57	41.00	47.43	46.43	50.00	46.76	3.36
EmbraceBERT-bs16	47.14	45.57	50.14	45.14	38.14	48.29	48.29	46.86	49.57	45.14	46.43	3.22
EmbraceBERT-bs16+Dropout0.1	45.43	41.14	39.71	45.57	45.43	48.43	50.29	50.29	46.86	47.00	46.01	3.29
EmbraceBERT-bs16+Dropout0.3	47.29	48.86	48.00	43.00	39.14	46.29	44.57	47.14	49.86	49.14	46.33	3.11
CondensedEmbraceBERT-bs16	12.57	13.57	12.29	12.29	14.29	14.14	9.00	18.71	15.57	9.86	13.23	2.63
CondensedEmbraceBERT-bs16+Dropout0.1	49.00	44.00	45.14	46.71	47.86	46.14	44.57	46.43	44.86	42.57	45.73	1.80
CondensedEmbraceBERT-bs16+Dropout0.3	45.71	47.29	45.29	49.14	48.71	46.43	43.00	47.00	45.71	44.29	46.26	1.79
FrozenEBERT-bs16-ep100	49.14	39.14	45.29	49.43	42.71	45.29	48.29	46.14	44.14	42.71	45.23	3.07
FrozenEBERT-bs16-ep100+Dropout0.1	45.14	46.00	43.86	48.71	49.00	48.00	44.14	49.86	46.71	48.57	47.00	2.03
FrozenEBERT-bs16-ep100+Dropout0.3	44.43	41.00	45.57	47.43	47.86	47.71	43.57	47.43	49.29	44.29	45.86	2.40
FrozenCEBERT-bs16-ep100	11.57	9.43	9.86	12.14	14.71	11.86	9.00	14.71	20.43	10.43	12.41	3.27
FrozenCEBERT-bs16-ep100+Dropout0.1	49.43	46.86	40.00	45.43	44.29	40.43	46.29	44.43	42.71	43.43	44.33	2.74
FrozenCEBERT-bs16-ep100+Dropout0.3	48.71	48.86	43.86	45.29	46.14	45.57	44.71	49.43	44.57	45.57	46.27	1.89
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	43.43	40.71	40.29	41.29	43.29	41.71	49.57	44.57	46.00	43.43	43.43	2.65
EmbraceBERT-bs16+Branches+Dropout0.1	49.43	44.29	43.29	44.00	42.00	50.00	49.00	44.29	48.71	47.14	46.21	2.80
EmbraceBERT-bs16+Branches+Dropout0.3	48.71	43.43	40.57	47.29	44.71	44.86	47.86	45.14	47.57	47.00	45.71	2.34
CondensedEmbraceBERT-bs16+Branches	49.00	48.14	46.00	44.43	46.43	44.71	46.71	48.43	41.71	46.43	46.20	2.06
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	50.43	45.43	40.57	42.29	49.29	45.29	46.71	42.43	47.57	43.14	45.31	3.06
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	51.43	45.29	46.29	47.43	44.43	47.43	46.71	46.43	48.57	46.71	47.07	1.82
FrozenEmbraceBERT-bs16+Branches	48.71	46.57	40.29	49.86	43.43	47.86	45.57	48.00	40.29	47.43	45.80	3.22
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	45.00	49.43	46.14	49.71	41.00	47.29	42.71	46.29	49.57	49.43	46.66	2.90
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	44.71	48.57	45.43	42.29	45.29	52.29	45.57	48.43	49.00	44.29	46.59	2.77
FrozenCEBERT-bs16+Branches	48.29	41.71	47.14	39.00	43.29	48.14	45.00	44.86	41.57	45.00	44.40	2.88
FrozenCEBERT-bs16+Branches+Dropout0.1	45.43	44.71	42.43	44.00	44.57	48.14	46.86	43.29	46.00	44.14	44.96	1.61
FrozenCEBERT-bs16+Branches+Dropout0.3	48.86	46.71	47.29	46.14	47.71	47.29	50.00	46.86	53.14	41.14	47.51	2.89
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	46.71	46.14	47.43	43.86	43.86	47.14	47.71	48.71	45.57	48.57	46.57	1.64
BERT-bs32+Dropout0.1	48.57	44.86	44.71	47.00	45.57	49.00	50.57	43.43	43.29	51.00	46.80	2.71
BERT-bs32+Dropout0.3	46.86	46.57	46.14	49.00	48.14	46.14	48.71	49.43	44.71	50.57	47.63	1.73
FrozenBERT-bs32-ep100	47.29	45.29	50.00	46.43	45.29	45.00	46.29	47.57	48.14	47.57	46.89	1.47
FrozenBERT-bs32-ep100+Dropout0.1	48.57	47.86	42.14	46.86	47.14	46.71	49.14	43.71	48.43	48.57	46.91	2.16
FrozenBERT-bs32-ep100+Dropout0.3	45.00	50.14	45.14	43.57	43.86	45.71	44.29	45.57	48.57	44.00	45.59	2.04
EmbraceBERT-bs32	46.00	46.86	40.00	44.14	42.57	46.57	46.14	46.14	44.86	45.00	44.83	2.02
EmbraceBERT-bs32+Dropout0.1	46.29	46.71	47.71	48.29	44.71	44.71	47.86	48.71	45.86	44.57	46.54	1.48
EmbraceBERT-bs32+Dropout0.3	46.43	45.57	44.71	48.71	39.00	45.43	43.14	47.00	45.71	46.00	45.17	2.47
CondensedEmbraceBERT-bs32	16.00	13.71	11.57	10.71	17.00	12.71	8.57	25.00	23.29	10.57	14.91	5.21
CondensedEmbraceBERT-bs32+Dropout0.1	48.14	45.86	44.14	43.29	46.29	45.29	47.14	50.86	44.29	47.86	46.31	2.15
CondensedEmbraceBERT-bs32+Dropout0.3	45.86	41.14	47.00	46.14	48.00	45.86	46.43	46.71	43.00	46.00	45.61	1.92
FrozenEBERT-bs32-ep100	48.29	45.00	47.00	40.29	44.00	44.86	46.29	44.71	42.86	43.29	44.66	2.15
FrozenEBERT-bs32-ep100+Dropout0.1	45.57	44.57	45.14	43.57	41.71	45.86	48.14	47.71	42.14	45.43	44.99	1.99
FrozenEBERT-bs32-ep100+Dropout0.3	45.71	42.57	47.43	44.14	41.86	45.00	41.86	48.14	47.43	46.14	45.03	2.23
FrozenCEBERT-bs32-ep100	16.86	14.29	11.00	10.00	17.14	12.29	9.00	25.43	22.86	10.43	14.93	5.33
FrozenCEBERT-bs32-ep100+Dropout0.1	46.00	46.71	42.29	41.86	44.57	45.57	47.00	45.14	43.43	44.29	44.69	1.66
FrozenCEBERT-bs32-ep100+Dropout0.3	46.00	44.86	43.00	45.86	48.14	45.29	42.14	40.43	46.71	46.14	44.86	2.21
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32+Branches	44.14	48.00	38.71	44.57	45.57	43.43	47.00	47.43	45.14	43.43	44.74	2.53
EmbraceBERT-bs32+Branches+Dropout0.1	52.29	49.71	41.86	46.57	44.14	47.00	46.86	47.00	47.00	45.86	46.83	2.67
EmbraceBERT-bs32+Branches+Dropout0.3	46.57	44.71	42.14	44.86	46.43	44.29	48.71	48.00	47.29	46.29	45.93	1.85
CondensedEmbraceBERT-bs32+Branches	48.29	48.86	44.43	44.29	48.29	43.43	44.71	44.29	42.43	45.86	45.49	2.13
CondensedEmbraceBERT-bs32+Branches+Dropout0.1	46.71	45.71	41.71	47.71	45.43	47.14	42.71	44.71	48.14	44.14	45.41	2.01
CondensedEmbraceBERT-bs32+Branches+Dropout0.3	48.86	44.86	44.29	47.14	42.86	45.86	42.00	45.29	45.29	47.57	45.40	1.99
FrozenEmbraceBERT-bs32+Branches	45.71	46.86	46.14	44.86	41.43	42.57	47.00	46.86	43.57	47.86	45.29	2.02
FrozenEmbraceBERT-bs32+Branches+Dropout0.1	48.57	40.14	41.86	46.86	42.86	45.29	45.57	50.43	45.86	44.43	45.19	2.91
FrozenEmbraceBERT-bs32+Branches+Dropout0.3	41.71	47.43	37.57	48.14	44.29	43.43	47.57	50.14	46.57	47.86	45.47	3.55
FrozenCEBERT-bs32+Branches	45.43	40.71	41.86	47.43	48.57	46.29	44.57	43.00	45.29	50.00	45.31	2.78
FrozenCEBERT-bs32+Branches+Dropout0.1	49.00	44.71	47.29	42.14	42.00	48.43	46.00	45.00	44.57	48.00	45.71	2.35
FrozenCEBERT-bs32+Branches+Dropout0.3	48.29	38.29	45.57	46.00	44.29	45.71	41.29	51.00	46.71	43.86	45.10	3.35
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	51.86	48.86	50.14	49.57	48.57	49.86	49.71	48.86	50.86	49.14	49.74	0.96
RoBERTa-bs16+Dropout0.1	47.57	51.14	50.57	44.71	49.86	49.00	50.43	50.00	50.71	45.43	48.94	2.17
RoBERTa-bs16+Dropout0.3	47.86	52.86	50.00	45.86	49.14	47.86	47.57	48.29	50.29	49.71	48.94	1.82
FrozenRoBERTa-bs16-ep100	46.86	49.29	49.43	45.86	47.71	49.29	51.14	48.71	45.71	50.71	48.47	1.79
FrozenRoBERTa-bs16-ep100+Dropout0.1	46.57	47.57	45.00	48.43	49.00	48.00	48.43	46.71	44.86	45.71	47.03	1.41
FrozenRoBERTa-bs16-ep100+Dropout0.3	47.71	46.71	48.43	47.43	47.71	48.43	47.71	46.71	48.29	47.00	47.61	0.62
EmbraceRoBERTa-bs16	46.14	44.71	48.86	47.86	48.71	48.57	51.00	49.57	49.57	51.57	48.66	1.95
EmbraceRoBERTa-bs16+Dropout0.1	48.14	49.29	46.86	41.43	52.14	47.57	47.29	47.29	48.29	49.29	47.76	2.56
EmbraceRoBERTa-bs16+Dropout0.3	48.14	47.29	48.86	48.86	44.71	48.71	47.71	45.71	45.57	45.29	47.09	1.53
CondensedEmbraceRoBERTa-bs16	7.43	11.71	25.86	10.29	11.29	19.29	15.14	13.29	12.71	16.00	14.30	4.95
CondensedEmbraceRoBERTa-bs16+Dropout0.1	49.43	48.71	47.43	50.00	49.00	49.00	49.57	44.43	50.29	46.71	48.46	1.70
CondensedEmbraceRoBERTa-bs16+Dropout0.3	49.43	44.57	49.14	48.14	46.71	51.57	47.57	48.71	46.00	47.71	47.96	1.85
FrozenERoBERTa-bs16-ep100	50.57	47.00	46.57	46.71	45.43	48.86	49.86	45.71	45.57	48.86	47.51	1.78
FrozenERoBERTa-bs16-ep100+Dropout0.1	46.71	45.29	46.57	47.86	45.14	45.57	49.71	48.29	48.14	44.00	46.73	1.67
FrozenERoBERTa-bs16-ep100+Dropout0.3	49.29	45.86	47.86	47.86	48.14	49.00	46.71	47.29	45.43	47.86	47.53	1.18
FrozenCERoBERTa-bs16-ep100	9.43	15.43	25.71	10.29	8.86	18.14	14.00	14.43	8.86	11.14	13.63	5.01
FrozenCERoBERTa-bs16-ep100+Dropout0.1	45.00	43.86	48.29	47.71	52.57	45.71	48.14	46.29	50.57	50.29	47.84	2.59
FrozenCERoBERTa-bs16-ep100+Dropout0.3	45.57	45.86	47.29	51.00	48.00	46.71	48.29	49.29	47.14	50.71	47.99	1.77
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	48.86	48.00	48.00	47.57	48.29	48.14	50.57	51.00	49.43	49.71	48.96	1.12
EmbraceRoBERTa-bs16+Branches+Dropout0.1	50.86	46.14	48.57	50.00	49.29	47.43	48.57	50.86	48.14	47.57	48.74	1.45
EmbraceRoBERTa-bs16+Branches+Dropout0.3	49.43	45.29	50.00	49.00	48.57	49.43	47.43	51.29	45.86	47.57	48.39	1.77
CondensedEmbraceRoBERTa-bs16+Branches	46.29	48.86	47.29	49.14	50.57	50.00	46.57	51.71	44.29	47.00	48.17	2.16
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	48.14	50.43	51.57	47.29	49.86	50.57	51.14	46.00	50.71	48.43	49.41	1.75
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	46.57	45.57	48.86	46.57	47.57	45.86	48.86	50.00	48.29	48.29	47.64	1.38
FrozenEmbraceRoBERTa-bs16+Branches	46.29	49.00	46.57	48.14	47.14	51.71	51.43	48.71	47.86	48.57	48.54	1.74
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	48.86	47.86	46.57	48.00	45.00	48.43	46.71	50.57	50.86	48.14	48.10	1.68
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	51.71	47.43	45.43	48.71	47.29	44.71	49.43	51.29	47.00	48.14	48.11	2.15
FrozenCERoBERTa-bs16+Branches	46.57	46.86	48.57	47.43	50.43	50.29	47.14	48.71	48.57	47.86	48.24	1.27
FrozenCERoBERTa-bs16+Branches+Dropout0.1	47.29	50.14	49.14	48.00	48.57	47.71	46.71	51.57	49.86	45.29	48.43	1.74
FrozenCERoBERTa-bs16+Branches+Dropout0.3	48.86	45.71	48.71	48.86	47.43	48.86	46.00	48.86	48.71	44.29	47.63	1.61
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	50.29	45.86	51.29	50.29	50.00	47.71	50.86	47.57	47.43	48.71	49.00	1.71
RoBERTa-bs32+Dropout0.1	47.00	48.71	46.71	47.86	46.71	52.43	49.57	47.86	52.14	50.14	48.91	2.01
RoBERTa-bs32+Dropout0.3	45.14	49.86	45.29	47.00	50.43	50.14	48.43	50.29	53.14	45.86	48.56	2.53
FrozenRoBERTa-bs32-ep100	47.29	48.43	46.29	47.43	48.86	43.86	49.43	50.71	49.14	49.71	48.11	1.88
FrozenRoBERTa-bs32-ep100+Dropout0.1	46.29	47.57	47.57	45.29	47.43	48.29	49.86	45.43	48.43	48.43	47.46	1.36
FrozenRoBERTa-bs32-ep100+Dropout0.3	49.14	48.71	45.57	46.57	46.29	47.14	45.00	50.71	47.43	44.14	47.07	1.91
EmbraceRoBERTa-bs32	50.29	49.57	46.29	49.86	49.14	48.29	50.29	46.71	48.43	47.43	48.63	1.37
EmbraceRoBERTa-bs32+Dropout0.1	48.29	50.71	48.14	50.29	49.43	48.29	48.29	47.86	49.71	49.86	49.09	0.98
EmbraceRoBERTa-bs32+Dropout0.3	48.86	49.43	47.00	49.14	48.29	48.00	49.86	48.57	48.57	47.00	48.47	0.90
CondensedEmbraceRoBERTa-bs32	7.71	11.43	26.14	12.29	11.14	17.29	10.71	11.71	11.14	12.57	13.21	4.85
CondensedEmbraceRoBERTa-bs32+Dropout0.1	48.71	47.00	48.57	48.14	52.14	51.43	48.71	47.71	50.14	49.00	49.16	1.54
CondensedEmbraceRoBERTa-bs32+Dropout0.3	47.29	44.71	47.00	50.57	51.57	51.14	46.00	49.29	50.00	49.00	48.66	2.19
FrozenERoBERTa-bs32-ep100	47.57	48.57	49.29	49.43	47.00	47.14	49.57	49.00	47.43	51.71	48.67	1.38
FrozenERoBERTa-bs32-ep100+Dropout0.1	47.57	46.43	49.00	44.57	52.00	48.57	47.29	49.14	49.57	47.86	48.20	1.89
FrozenERoBERTa-bs32-ep100+Dropout0.3	45.14	47.29	49.57	45.57	48.14	47.43	49.43	49.57	47.57	47.29	47.70	1.47
FrozenCERoBERTa-bs32-ep100	7.29	11.57	23.00	9.29	10.43	17.00	12.00	9.43	7.00	11.00	11.80	4.59
FrozenCERoBERTa-bs32-ep100+Dropout0.1	45.29	47.43	46.29	51.43	47.43	47.43	48.29	48.14	44.57	50.14	47.64	1.95
FrozenCERoBERTa-bs32-ep100+Dropout0.3	46.71	48.71	47.71	50.00	49.71	47.57	47.71	45.00	47.71	49.29	48.01	1.42
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Branches	46.14	48.43	49.71	47.71	50.14	45.71	49.57	47.86	47.14	48.14	48.06	1.40
EmbraceRoBERTa-bs32+Branches+Dropout0.1	49.57	49.43	48.14	49.86	46.86	43.43	44.71	49.57	49.57	50.57	48.17	2.29
EmbraceRoBERTa-bs32+Branches+Dropout0.3	46.43	47.43	48.43	48.86	46.71	51.43	48.29	50.00	47.71	51.43	48.67	1.69
CondensedEmbraceRoBERTa-bs32+Branches	46.86	45.57	49.00	51.43	50.00	53.14	46.14	46.57	48.57	47.00	48.43	2.35
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.1	51.29	48.00	47.57	49.71	48.29	52.57	47.57	47.86	49.00	50.29	49.21	1.63
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.3	50.71	48.57	50.00	46.86	50.14	48.86	49.86	50.14	47.14	52.57	49.49	1.61
FrozenEmbraceRoBERTa-bs32+Branches	45.57	47.00	50.14	51.00	50.71	45.29	47.57	47.86	45.29	49.29	47.97	2.11
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.1	47.71	49.43	50.00	44.57	47.43	50.29	46.57	47.29	49.43	48.57	48.13	1.68
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.3	47.57	49.57	43.29	49.29	43.43	46.71	51.43	45.00	49.14	48.00	47.34	2.59
FrozenCERoBERTa-bs32+Branches	50.14	47.29	49.00	49.43	52.00	49.29	50.29	44.57	47.43	48.86	48.83	1.92
FrozenCERoBERTa-bs32+Branches+Dropout0.1	47.29	46.00	49.29	49.00	46.71	50.43	50.00	52.57	48.57	47.00	48.69	1.90
FrozenCERoBERTa-bs32+Branches+Dropout0.3	45.14	49.57	45.57	50.71	50.43	45.57	49.29	49.57	44.86	50.14	48.09	2.33

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	49.43	48.86	46.57	49.86	49.57	48.43	52.43	50.00	46.14	48.00	48.93	1.71
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	50.00	50.57	47.71	50.14	46.71	49.00	52.43	46.71	49.14	49.14	49.16	1.69
BERTwithAttClsProjection-bs32	50.71	46.14	49.86	47.57	47.29	49.14	51.00	52.57	52.00	50.14	49.64	1.99
BERTwithProjection-bs32	50.29	50.14	47.71	48.43	49.43	47.43	49.43	51.86	48.71	51.14	49.46	1.36
BERTwithProjectionAtt-bs32	48.43	53.71	48.43	49.43	47.71	48.71	50.43	47.86	48.14	50.86	49.37	1.76
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	47.00	54.29	45.43	48.71	49.86	46.29	49.29	51.43	45.71	49.43	48.74	2.63
EmbraceBERT-bs32-p_attclsqw	44.29	48.29	48.14	50.00	50.71	48.14	47.57	48.00	49.29	46.00	48.04	1.76
EmbraceBERTwithProj-bs32	51.57	50.00	52.57	48.14	52.29	48.29	52.86	49.57	52.71	49.29	50.73	1.78
EmbraceBERTwithProj-bs32-p_attclsqw	53.00	48.86	52.57	49.43	52.57	49.43	49.43	48.29	50.00	49.86	50.34	1.62
EBERTconcatAtt-bs32	46.29	44.43	47.00	47.29	49.29	48.71	49.00	47.43	50.00	48.57	47.80	1.57
EBERTconcatAtt-bs32-p_attclsqw	49.71	49.14	47.29	47.43	47.71	47.14	49.14	46.57	47.14	49.57	48.09	1.11
EBERTconcatAttwithProj-bs32	51.57	50.00	54.43	49.14	49.86	48.86	52.57	49.43	48.43	51.86	50.61	1.82
EBERTconcatAttwithProj-bs32-p_attclsqw	49.43	50.71	50.71	50.57	48.57	50.86	50.29	47.86	52.00	50.86	50.19	1.16
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery	51.71	50.29	43.71	48.43	49.00	51.14	50.29	50.14	48.57	47.57	49.09	2.17
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	49.57	46.71	49.14	51.86	48.29	49.29	52.57	47.00	49.86	49.14	49.34	1.75
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery	51.71	52.29	49.86	48.00	50.86	50.00	52.57	52.86	52.71	49.71	51.06	1.55
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	48.43	50.43	48.43	51.00	51.43	48.57	49.29	48.14	48.86	53.00	49.76	1.55
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery	50.29	50.71	46.71	48.14	48.43	50.00	51.71	48.29	51.57	49.57	49.54	1.54
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	48.00	46.14	48.14	50.71	46.57	46.71	47.29	51.14	51.86	47.86	48.44	1.95
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery	53.29	52.14	48.57	50.86	48.43	51.00	51.29	52.29	50.43	50.00	50.83	1.48
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	48.71	50.86	49.43	49.57	49.00	50.29	50.86	51.00	51.29	51.14	50.21	0.91

• Epoch 100 - bs48 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	45.57	49.43	49.00	48.00	44.43	48.29	52.00	49.29	43.71	46.86	47.66	2.41
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	50.43	49.00	49.86	47.29	47.29	49.00	45.43	48.43	47.86	47.14	48.17	1.40
BERTwithAttClsProjection-bs48	47.57	48.00	50.29	49.00	52.71	51.29	48.71	51.00	48.00	49.00	49.56	1.60
BERTwithProjection-bs48	50.86	47.86	46.43	48.86	48.86	53.00	52.14	49.71	49.14	52.71	49.96	2.06
BERTwithProjectionAtt-bs48	47.14	46.86	49.14	51.14	44.57	46.57	51.14	51.43	47.14	47.71	48.29	2.21
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	47.29	49.57	47.71	46.00	48.14	46.29	47.86	45.86	46.43	46.86	47.20	1.10
EmbraceBERT-bs48-p_attclsqw	48.14	42.71	48.86	50.57	43.57	49.57	48.14	51.43	50.29	46.57	47.99	2.76
EmbraceBERTwithProj-bs48	52.43	49.86	53.14	46.14	53.43	48.14	49.14	49.14	51.29	50.29	50.30	2.20
EmbraceBERTwithProj-bs48-p_attclsqw	49.00	50.43	48.86	48.00	49.29	48.86	48.00	51.00	51.43	50.71	49.56	1.18
EBERTconcatAtt-bs48	46.00	47.57	48.43	44.71	46.29	43.86	45.86	44.86	47.14	44.00	45.87	1.45
EBERTconcatAtt-bs48-p_attclsqw	45.00	47.71	48.00	50.43	48.86	49.29	49.14	47.57	51.00	50.00	48.70	1.64
EBERTconcatAttwithProj-bs48	53.00	50.14	50.00	49.00	47.29	51.00	51.29	50.14	50.14	47.29	49.93	1.66
EBERTconcatAttwithProj-bs48-p_attclsqw	48.86	47.57	50.71	51.29	50.00	50.00	51.86	50.14	49.00	48.57	49.80	1.24
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	50.14	46.57	49.14	47.86	47.57	46.00	45.00	48.71	51.00	49.57	48.16	1.81
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	49.57	46.57	51.14	50.14	47.14	47.00	45.71	50.00	46.14	47.00	48.04	1.85
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	49.71	49.71	51.29	48.86	47.86	51.71	51.71	52.00	49.57	50.14	50.26	1.31
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	52.86	50.57	50.00	51.29	49.29	49.00	52.43	49.57	49.14	47.86	50.20	1.51
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	44.14	50.14	48.00	47.00	48.00	48.29	46.14	45.71	45.00	46.43	46.89	1.68
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	46.71	49.29	46.29	45.86	45.71	49.71	49.86	42.14	50.71	47.71	47.40	2.46
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	47.86	49.86	48.00	48.43	47.14	50.86	47.14	49.43	47.57	53.71	49.00	1.95
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	51.86	52.14	49.14	49.29	48.29	45.71	48.29	48.14	52.57	45.86	49.13	2.31

macsay-witai - not good

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	94.71	94.71	94.43	93.86	94.29	94.71	94.57	94.43	94.14	94.57	94.44	0.27
BERT-bs16+Dropout0.1	94.43	94.86	95.00	93.43	94.71	94.43	95.14	94.29	94.43	94.86	94.56	0.46
BERT-bs16+Dropout0.3	94.00	94.29	94.57	94.43	94.29	94.86	94.71	93.86	94.71	94.71	94.44	0.32
FrozenBERT-bs16-ep100	93.71	93.71	94.00	94.14	93.71	93.43	92.71	94.29	92.71	94.57	93.70	0.58
FrozenBERT-bs16-ep100+Dropout0.1	94.14	94.00	93.86	94.00	94.43	94.57	94.14	94.29	94.00	93.71	94.11	0.25
FrozenBERT-bs16-ep100+Dropout0.3	93.43	94.43	94.14	94.14	94.14	94.14	94.14	94.57	93.86	94.29	94.13	0.30
EmbraceBERT-bs16	94.86	93.29	94.29	94.43	92.86	94.43	93.86	94.00	93.86	94.43	94.03	0.57
EmbraceBERT-bs16+Dropout0.1	94.29	92.57	93.29	94.57	94.14	94.29	94.43	94.14	94.00	94.43	94.01	0.59
EmbraceBERT-bs16+Dropout0.3	93.14	95.14	94.14	94.00	93.00	93.57	95.00	94.29	93.86	94.71	94.09	0.69
CondensedEmbraceBERT-bs16	16.00	4.43	10.00	15.71	12.14	9.29	1.71	19.29	8.43	2.57	9.96	5.63
CondensedEmbraceBERT-bs16+Dropout0.1	93.71	93.71	92.86	94.71	93.57	93.71	94.57	94.14	94.86	94.14	94.00	0.58
CondensedEmbraceBERT-bs16+Dropout0.3	93.43	94.57	93.57	94.57	92.71	94.00	94.29	93.71	94.29	94.71	93.99	0.60
FrozenEBERT-bs16-ep100	93.43	90.86	93.71	94.00	94.00	94.29	94.86	93.57	93.29	92.71	93.47	1.03
FrozenEBERT-bs16-ep100+Dropout0.1	93.71	94.43	93.71	93.71	92.86	94.71	93.86	93.14	93.29	94.71	93.81	0.61
FrozenEBERT-bs16-ep100+Dropout0.3	92.57	94.00	94.71	93.57	93.43	94.29	94.57	94.29	93.86	93.00	93.83	0.65
FrozenCEBERT-bs16-ep100	13.14	1.86	6.00	12.00	11.86	6.43	3.00	20.14	21.14	2.29	9.79	6.69
FrozenCEBERT-bs16-ep100+Dropout0.1	94.29	93.71	93.57	94.14	93.14	92.43	94.14	94.29	93.86	94.29	93.79	0.58
FrozenCEBERT-bs16-ep100+Dropout0.3	94.14	95.00	94.86	94.29	92.00	94.00	94.00	94.00	94.00	93.29	93.96	0.80
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	94.43	92.71	93.29	94.57	93.57	94.14	94.71	93.86	94.43	94.71	94.04	0.64
EmbraceBERT-bs16+Branches+Dropout0.1	93.00	93.14	93.29	93.29	94.29	95.00	94.43	94.29	94.14	95.14	94.00	0.74
EmbraceBERT-bs16+Branches+Dropout0.3	94.14	94.29	93.71	94.43	94.29	92.86	94.57	92.71	94.43	94.71	94.01	0.67
CondensedEmbraceBERT-bs16+Branches	94.57	94.57	94.86	94.29	94.00	94.86	94.00	95.00	93.57	95.14	94.49	0.48
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	94.57	94.57	93.71	94.29	94.29	93.71	94.71	93.14	94.14	92.71	93.99	0.62
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	94.29	94.14	93.86	95.00	92.86	94.14	92.71	93.86	94.43	93.86	93.91	0.65
FrozenEmbraceBERT-bs16+Branches	93.71	93.43	92.57	94.86	93.14	93.86	94.29	94.14	93.29	94.29	93.76	0.64
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	94.00	93.29	93.57	94.00	93.00	94.43	93.14	92.57	93.86	95.71	93.76	0.84
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	93.57	94.14	94.29	93.29	94.00	94.29	93.14	94.29	95.71	94.14	94.09	0.68
FrozenCEBERT-bs16+Branches	95.00	93.43	94.00	93.43	93.29	94.14	93.86	93.29	94.14	93.00	93.76	0.56
FrozenCEBERT-bs16+Branches+Dropout0.1	94.14	95.00	94.57	93.14	93.71	94.57	93.57	93.71	94.43	93.57	94.04	0.56
FrozenCEBERT-bs16+Branches+Dropout0.3	93.86	94.71	93.86	93.86	93.00	94.00	94.43	94.29	94.14	92.57	93.87	0.61
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	94.71	94.86	94.57	94.00	94.43	94.43	94.71	94.71	94.14	94.71	94.53	0.26
BERT-bs32+Dropout0.1	93.43	94.43	94.29	94.00	94.29	94.14	94.86	94.14	94.71	93.86	94.21	0.39
BERT-bs32+Dropout0.3	94.43	95.00	94.43	94.29	94.14	94.43	94.71	94.43	94.29	94.71	94.49	0.24
FrozenBERT-bs32-ep100	94.71	94.29	94.71	94.14	94.43	94.29	94.57	94.29	94.57	94.86	94.49	0.22
FrozenBERT-bs32-ep100+Dropout0.1	94.86	94.43	94.29	94.29	94.43	94.43	93.43	94.29	94.71	94.57	94.37	0.36
FrozenBERT-bs32-ep100+Dropout0.3	94.14	94.57	94.14	93.71	94.29	93.71	94.14	94.86	94.00	94.43	94.20	0.34
EmbraceBERT-bs32	94.71	94.57	94.00	94.14	93.29	94.00	94.14	95.14	93.71	94.57	94.23	0.51
EmbraceBERT-bs32+Dropout0.1	94.14	93.57	94.57	94.86	94.43	94.29	94.57	95.29	94.14	93.57	94.34	0.50
EmbraceBERT-bs32+Dropout0.3	94.43	94.14	94.57	94.43	93.14	94.00	93.14	94.14	94.57	94.29	94.09	0.50
CondensedEmbraceBERT-bs32	11.86	9.43	7.43	3.00	16.14	8.57	3.57	26.57	27.43	3.00	11.70	8.60
CondensedEmbraceBERT-bs32+Dropout0.1	95.00	94.86	94.29	94.29	94.57	94.29	94.71	95.57	93.86	93.29	94.47	0.60
CondensedEmbraceBERT-bs32+Dropout0.3	94.57	93.86	94.14	94.57	93.86	95.14	94.29	94.57	94.00	94.14	94.31	0.38
FrozenEBERT-bs32-ep100	94.00	94.86	93.57	93.57	92.43	94.29	94.00	93.86	93.14	94.00	93.77	0.62
FrozenEBERT-bs32-ep100+Dropout0.1	94.57	94.57	92.29	93.86	93.86	94.29	93.14	94.43	93.43	93.86	93.83	0.68
FrozenEBERT-bs32-ep100+Dropout0.3	93.71	95.00	93.86	94.86	93.86	94.57	92.86	94.57	94.71	94.14	94.21	0.62
FrozenCEBERT-bs32-ep100	12.29	8.71	7.57	3.71	18.00	9.57	3.86	25.71	27.71	3.14	12.03	8.49
FrozenCEBERT-bs32-ep100+Dropout0.1	93.86	94.43	93.29	93.57	94.57	94.00	93.29	94.00	93.29	94.00	93.83	0.44
FrozenCEBERT-bs32-ep100+Dropout0.3	94.29	94.00	94.14	94.71	93.57	94.71	93.29	94.00	94.43	94.29	94.14	0.43
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32+Branches	93.57	95.00	91.71	94.14	94.00	94.29	94.29	94.43	94.57	93.57	93.96	0.85
EmbraceBERT-bs32+Branches+Dropout0.1	94.00	94.00	93.71	94.71	94.29	94.86	93.71	94.71	93.43	94.14	94.16	0.46
EmbraceBERT-bs32+Branches+Dropout0.3	94.29	93.14	93.29	94.71	94.29	94.86	94.71	95.57	94.71	94.43	94.40	0.69
CondensedEmbraceBERT-bs32+Branches	94.57	95.14	93.57	94.29	95.00	94.29	94.43	94.86	94.00	93.86	94.40	0.48
CondensedEmbraceBERT-bs32+Branches+Dropout0.1	92.86	94.29	92.86	94.14	94.00	94.29	93.71	93.71	93.86	94.14	93.79	0.50
CondensedEmbraceBERT-bs32+Branches+Dropout0.3	94.86	94.29	93.86	94.29	93.43	93.43	93.71	94.43	94.57	94.00	94.09	0.46
FrozenEmbraceBERT-bs32+Branches	93.43	95.00	93.86	94.00	93.14	93.57	92.86	94.71	94.00	93.43	93.80	0.63
FrozenEmbraceBERT-bs32+Branches+Dropout0.1	93.71	93.43	93.43	94.00	95.00	94.00	92.86	94.57	93.71	93.29	93.80	0.60
FrozenEmbraceBERT-bs32+Branches+Dropout0.3	92.57	94.14	91.86	93.71	93.86	94.14	94.57	95.14	93.43	93.29	93.67	0.90
FrozenCEBERT-bs32+Branches	93.86	93.43	93.00	93.86	94.29	94.86	92.71	94.43	92.86	94.14	93.74	0.68
FrozenCEBERT-bs32+Branches+Dropout0.1	94.86	94.43	94.29	93.00	93.43	94.57	94.57	94.29	92.43	94.71	94.06	0.78
FrozenCEBERT-bs32+Branches+Dropout0.3	94.57	93.57	92.43	93.86	94.00	93.86	92.14	94.43	93.71	94.00	93.66	0.75
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	93.57	93.57	94.00	94.00	94.86	94.29	93.14	94.00	93.86	93.86	93.91	0.43
RoBERTa-bs16+Dropout0.1	92.86	93.71	93.71	91.57	93.00	93.29	92.86	92.14	94.43	92.71	93.03	0.78
RoBERTa-bs16+Dropout0.3	93.57	94.43	93.71	92.43	93.29	92.29	93.71	92.86	93.86	93.00	93.31	0.64
FrozenRoBERTa-bs16-ep100	92.43	93.57	92.86	92.86	92.86	91.29	92.57	94.43	91.29	92.86	92.70	0.89
FrozenRoBERTa-bs16-ep100+Dropout0.1	92.00	92.86	92.71	92.86	92.57	92.57	93.86	93.71	93.29	93.71	93.01	0.58
FrozenRoBERTa-bs16-ep100+Dropout0.3	92.71	90.29	94.14	91.57	93.86	93.29	94.14	93.29	93.14	92.43	92.89	1.15
EmbraceRoBERTa-bs16	92.71	94.86	92.86	94.00	92.14	93.57	93.57	92.86	91.86	93.71	93.21	0.86
EmbraceRoBERTa-bs16+Dropout0.1	92.43	92.86	92.71	91.29	94.00	93.00	94.29	92.57	93.86	92.71	92.97	0.84
EmbraceRoBERTa-bs16+Dropout0.3	92.29	91.71	93.86	94.29	92.29	93.00	94.14	92.14	94.14	94.00	93.19	0.95
CondensedEmbraceRoBERTa-bs16	3.29	5.00	27.57	15.57	3.86	25.57	6.29	8.29	11.29	15.00	12.17	8.28
CondensedEmbraceRoBERTa-bs16+Dropout0.1	93.00	93.71	92.00	93.57	92.43	91.86	93.00	93.14	93.43	94.29	93.04	0.73
CondensedEmbraceRoBERTa-bs16+Dropout0.3	91.71	92.71	93.86	93.71	93.57	93.14	94.14	92.71	91.43	92.14	92.91	0.89
FrozenERoBERTa-bs16-ep100	94.43	94.14	93.14	92.86	94.14	94.43	93.57	93.14	93.00	93.57	93.64	0.57
FrozenERoBERTa-bs16-ep100+Dropout0.1	92.57	93.86	92.71	93.14	93.57	94.00	92.43	93.29	91.86	93.29	93.07	0.64
FrozenERoBERTa-bs16-ep100+Dropout0.3	92.86	93.29	94.00	92.57	92.57	93.29	94.00	93.00	91.00	93.86	93.04	0.85
FrozenCERoBERTa-bs16-ep100	1.29	17.71	28.14	14.86	2.00	23.00	7.86	6.14	12.00	5.43	11.84	8.55
FrozenCERoBERTa-bs16-ep100+Dropout0.1	93.14	93.57	94.29	91.14	94.29	93.00	93.43	92.43	93.00	93.14	93.14	0.86
FrozenCERoBERTa-bs16-ep100+Dropout0.3	92.14	93.43	93.14	93.43	92.57	93.14	93.86	93.00	93.71	94.71	93.31	0.67
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	93.29	93.71	92.86	93.43	92.86	93.43	93.29	93.71	92.43	93.57	93.26	0.40
EmbraceRoBERTa-bs16+Branches+Dropout0.1	93.57	93.14	92.71	93.71	92.86	93.86	94.29	93.00	92.86	93.14	93.31	0.49
EmbraceRoBERTa-bs16+Branches+Dropout0.3	92.71	91.86	93.00	92.71	92.86	93.43	93.57	93.57	94.14	94.57	93.24	0.74
CondensedEmbraceRoBERTa-bs16+Branches	92.43	93.71	93.29	93.14	93.00	93.57	93.43	93.86	92.43	93.57	93.24	0.47
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	93.14	92.86	94.14	94.14	94.29	93.57	92.57	93.43	94.14	91.86	93.41	0.77
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	93.14	93.71	93.14	93.14	93.71	93.86	93.43	93.43	91.71	93.71	93.30	0.59
FrozenEmbraceRoBERTa-bs16+Branches	92.57	93.29	92.57	93.00	93.00	93.71	93.71	92.86	94.43	92.29	93.14	0.62
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	90.71	92.43	93.43	93.71	93.00	93.86	93.57	93.43	92.43	93.71	93.03	0.91
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	93.14	92.86	92.86	92.29	91.71	92.00	93.57	92.57	93.43	93.14	92.76	0.58
FrozenCERoBERTa-bs16+Branches	93.00	93.57	92.43	91.57	92.57	92.71	93.14	93.00	93.43	94.29	92.97	0.69
FrozenCERoBERTa-bs16+Branches+Dropout0.1	92.29	94.14	93.71	94.00	93.29	93.43	92.57	92.57	93.00	93.14	93.21	0.59
FrozenCERoBERTa-bs16+Branches+Dropout0.3	93.14	92.57	92.57	92.86	91.86	92.86	92.71	92.43	93.14	94.00	92.81	0.53
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	93.86	94.29	93.29	94.14	94.29	93.71	93.57	93.57	93.43	94.00	93.81	0.34
RoBERTa-bs32+Dropout0.1	93.29	94.29	92.00	92.43	93.71	92.86	93.57	94.29	93.29	93.43	93.31	0.70
RoBERTa-bs32+Dropout0.3	91.86	93.29	93.57	92.57	93.86	92.57	93.43	94.29	92.57	91.29	92.93	0.88
FrozenRoBERTa-bs32-ep100	92.86	93.14	92.43	92.71	93.29	92.57	94.00	92.14	92.57	94.14	92.99	0.63
FrozenRoBERTa-bs32-ep100+Dropout0.1	92.57	93.29	93.57	93.57	93.14	92.29	93.14	91.86	93.29	94.57	93.13	0.72
FrozenRoBERTa-bs32-ep100+Dropout0.3	93.14	92.71	93.14	93.00	92.86	92.43	93.14	92.86	92.43	94.00	92.97	0.43
EmbraceRoBERTa-bs32	93.29	93.14	93.57	93.29	94.43	93.00	93.29	93.86	93.57	93.86	93.53	0.40
EmbraceRoBERTa-bs32+Dropout0.1	94.14	93.43	93.86	94.00	93.71	92.57	93.86	94.14	92.57	94.29	93.66	0.59
EmbraceRoBERTa-bs32+Dropout0.3	93.57	92.43	93.43	93.14	93.00	94.00	93.57	93.00	91.71	92.71	93.06	0.62
CondensedEmbraceRoBERTa-bs32	9.86	3.29	27.29	15.14	2.71	15.00	4.71	1.86	8.00	9.43	9.73	7.39
CondensedEmbraceRoBERTa-bs32+Dropout0.1	92.29	93.86	93.29	92.57	93.29	94.29	92.57	94.00	93.43	93.14	93.27	0.62
CondensedEmbraceRoBERTa-bs32+Dropout0.3	93.43	94.00	93.43	92.43	92.57	93.86	93.86	92.00	92.86	93.71	93.21	0.66
FrozenERoBERTa-bs32-ep100	93.29	93.29	94.29	92.29	93.29	93.43	92.86	93.29	92.71	92.86	93.16	0.50
FrozenERoBERTa-bs32-ep100+Dropout0.1	93.71	94.00	93.14	94.14	93.71	93.71	94.29	91.71	90.29	93.86	93.26	1.21
FrozenERoBERTa-bs32-ep100+Dropout0.3	93.00	92.57	93.43	93.43	93.29	93.71	92.86	92.86	93.00	90.71	92.89	0.79
FrozenCERoBERTa-bs32-ep100	1.71	2.00	36.71	15.14	3.86	19.57	6.14	1.43	7.86	2.71	9.71	10.71
FrozenCERoBERTa-bs32-ep100+Dropout0.1	93.00	94.43	94.00	92.71	93.71	94.00	94.00	93.00	93.14	92.71	93.47	0.59
FrozenCERoBERTa-bs32-ep100+Dropout0.3	93.00	93.86	93.57	92.86	93.43	93.57	94.00	92.57	94.14	91.57	93.26	0.74
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs32+Branches	93.86	93.71	92.86	94.00	93.14	93.29	91.71	93.29	93.57	93.43	93.29	0.62
EmbraceRoBERTa-bs32+Branches+Dropout0.1	94.57	93.57	94.71	92.00	93.71	92.57	91.86	93.86	93.57	93.14	93.36	0.92
EmbraceRoBERTa-bs32+Branches+Dropout0.3	93.43	93.57	93.71	92.71	93.43	93.00	92.86	92.57	92.57	93.00	93.09	0.40
CondensedEmbraceRoBERTa-bs32+Branches	93.71	94.00	93.71	92.71	92.86	93.14	93.29	93.14	93.57	93.14	93.33	0.39
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.1	92.86	93.14	93.43	92.29	91.86	94.14	94.00	92.29	92.43	94.29	93.07	0.82
CondensedEmbraceRoBERTa-bs32+Branches+Dropout0.3	94.00	93.43	93.71	93.29	92.29	93.29	93.43	92.14	93.14	93.29	93.20	0.55
FrozenEmbraceRoBERTa-bs32+Branches	92.86	93.71	92.86	93.57	93.71	93.00	93.86	92.14	92.86	94.00	93.26	0.57
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.1	93.43	93.43	94.14	92.29	92.86	92.29	93.43	92.86	92.57	93.43	93.07	0.57
FrozenEmbraceRoBERTa-bs32+Branches+Dropout0.3	93.57	92.71	93.00	92.14	93.86	93.14	92.71	93.29	93.00	93.57	93.10	0.48
FrozenCERoBERTa-bs32+Branches	94.43	92.29	93.29	93.57	92.14	93.29	92.86	91.57	93.00	93.14	92.96	0.76
FrozenCERoBERTa-bs32+Branches+Dropout0.1	92.71	92.57	94.14	92.57	92.43	93.71	94.14	92.86	93.29	93.57	93.20	0.63
FrozenCERoBERTa-bs32+Branches+Dropout0.3	94.14	93.29	93.86	93.29	93.86	92.71	93.00	93.14	94.14	93.29	93.47	0.47

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	94.71	94.14	94.43	92.57	94.29	93.29	93.86	94.43	93.00	93.86	93.86	0.66
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	92.86	94.00	94.00	93.00	94.00	94.29	94.71	94.43	94.86	94.29	94.04	0.62
BERTwithAttClsProjection-bs32	95.57	94.29	93.57	94.43	94.57	94.14	94.71	93.57	93.71	93.57	94.21	0.61
BERTwithProjection-bs32	94.43	93.43	93.86	93.86	93.86	93.29	94.14	94.00	94.57	94.43	93.99	0.40
BERTwithProjectionAtt-bs32	94.57	94.43	93.43	94.14	94.86	94.43	93.86	93.71	94.43	94.57	94.24	0.42
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	93.71	92.71	93.57	93.57	93.00	94.29	94.14	94.43	93.14	94.29	93.69	0.57
EmbraceBERT-bs32-p_attclsqw	94.29	94.00	93.86	94.00	94.14	94.00	93.29	94.29	94.43	93.43	93.97	0.35
EmbraceBERTwithProj-bs32	94.29	94.00	94.43	93.57	94.00	93.71	94.14	93.00	93.71	94.43	93.93	0.42
EmbraceBERTwithProj-bs32-p_attclsqw	94.00	94.29	94.00	94.14	94.29	93.00	93.57	94.71	94.71	94.57	94.13	0.50
EBERTconcatAtt-bs32	94.57	93.14	94.57	94.00	94.29	94.71	93.57	93.86	93.86	94.43	94.10	0.48
EBERTconcatAtt-bs32-p_attclsqw	94.14	93.71	93.29	94.00	94.00	94.29	93.00	93.43	94.71	94.29	93.89	0.50
EBERTconcatAttwithProj-bs32	93.71	94.00	94.57	94.14	94.29	93.71	94.00	94.57	94.14	93.86	94.10	0.29
EBERTconcatAttwithProj-bs32-p_attclsqw	94.14	93.57	92.57	94.86	93.57	93.86	93.86	94.29	94.29	94.29	93.93	0.58
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery	94.00	94.43	94.00	94.00	94.71	94.14	94.57	93.14	94.29	93.86	94.11	0.42
EmbraceBERT-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	94.14	94.57	94.57	94.00	94.14	94.57	94.29	93.43	93.57	94.29	94.16	0.38
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery	94.57	94.14	93.86	93.71	92.71	94.57	93.71	94.43	93.86	92.86	93.84	0.61
EmbraceBERTwithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	93.29	93.71	94.14	93.14	93.57	94.43	93.71	94.29	94.43	94.14	93.89	0.44
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery	93.00	93.86	94.29	93.29	94.29	93.29	94.57	93.43	94.71	93.43	93.81	0.58
EmbraceBERTconcatatt-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	93.86	94.29	93.43	94.43	94.57	93.86	94.29	94.14	94.14	93.71	94.07	0.33
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery	94.57	93.29	93.00	94.71	93.71	94.57	94.14	94.00	94.00	94.14	94.01	0.53
EmbraceBERTconcatattWithProj-bs32-p_multiheadatt_bertKeyValQuery_attclsqw	94.71	95.14	94.00	94.00	94.29	94.86	94.43	93.86	94.57	93.43	94.33	0.49

• Epoch 100 - bs48 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	94.00	94.57	94.57	94.29	94.00	94.29	94.71	94.71	94.00	94.57	94.37	0.28
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	93.57	94.57	93.86	94.57	93.86	94.29	94.29	93.71	94.00	94.14	94.09	0.33
BERTwithAttClsProjection-bs48	95.00	94.00	94.14	93.86	93.71	94.71	94.57	93.71	94.00	94.57	94.23	0.43
BERTwithProjection-bs48	94.43	93.57	94.29	94.14	94.14	93.86	94.57	93.71	94.71	94.71	94.21	0.38
BERTwithProjectionAtt-bs48	94.71	94.00	93.71	94.14	94.43	94.57	94.29	93.57	93.86	94.14	94.14	0.35
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	94.00	93.43	93.14	94.14	94.00	93.43	93.71	93.00	94.14	94.29	93.73	0.43
EmbraceBERT-bs48-p_attclsqw	94.14	93.86	94.57	93.86	93.86	94.57	94.29	94.43	94.57	94.43	94.26	0.29
EmbraceBERTwithProj-bs48	94.86	94.43	94.57	94.14	93.71	94.43	93.57	94.29	94.00	94.43	94.24	0.37
EmbraceBERTwithProj-bs48-p_attclsqw	94.29	94.29	93.71	95.00	94.71	94.00	94.71	94.43	94.71	94.43	94.43	0.36
EBERTconcatAtt-bs48	94.00	94.00	93.86	94.86	94.14	93.86	94.57	94.00	95.14	94.57	94.30	0.43
EBERTconcatAtt-bs48-p_attclsqw	94.00	94.29	94.57	94.00	93.86	94.43	93.71	94.57	94.29	94.29	94.20	0.28
EBERTconcatAttwithProj-bs48	94.71	94.71	94.57	94.14	94.00	93.29	93.71	95.29	94.57	94.57	94.36	0.55
EBERTconcatAttwithProj-bs48-p_attclsqw	93.71	94.29	93.00	93.57	94.29	94.71	94.29	93.86	93.86	94.29	93.99	0.46
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	94.71	93.71	94.00	94.14	94.86	94.29	94.14	93.86	94.43	93.86	94.20	0.36
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	94.00	94.14	94.57	93.43	94.00	94.14	94.43	93.43	94.29	94.29	94.07	0.36
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	94.71	94.71	94.14	94.29	94.29	95.00	94.14	94.14	95.14	94.00	94.46	0.38
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	93.71	93.86	93.43	95.00	93.14	95.00	93.00	94.86	94.43	94.71	94.11	0.74
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	94.14	94.00	94.14	94.14	93.00	94.43	94.71	94.14	94.00	94.14	94.09	0.42
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	94.29	94.57	94.14	94.00	94.00	94.43	93.86	94.14	93.86	95.00	94.23	0.34
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	94.00	93.00	93.86	94.14	94.57	94.86	94.29	94.14	94.00	94.43	94.13	0.47
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	94.86	94.43	94.00	93.43	94.14	94.86	93.57	93.86	93.43	93.43	94.00	0.53

STT Error: trained and tested with INCOMPLETE data

gtts-google

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	75.14	74.71	75.71	75.43	75.00	75.57	76.00	75.86	75.43	75.57	75.44	0.37
BERT-bs16+Dropout0.1	75.14	74.86	75.43	75.43	75.29	74.29	75.00	75.57	75.00	75.14	75.11	0.35
EmbraceBERT-bs16-WRONG	75.43	75.57	75.00	76.14	75.14	74.86	75.29	75.43	74.71	76.00	75.36	0.44
EmbraceBERT-bs16+Dropout0.1-WRONG
CondensedEmbraceBERT-bs16	76.29	74.57	75.14	75.29	75.43	74.86	75.29	76.00	74.86	75.71	75.34	0.51
CondensedEmbraceBERT-bs16+Dropout0.1	75.43	75.00	75.71	75.29	74.43	75.57	75.29	75.57	75.29	75.57	75.31	0.35
CondensedEmbraceBERT-bs16+Dropout0.3	75.71	75.14	75.57	75.86	75.14	75.14	74.29	75.00	74.00	75.29	75.11	0.56
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	75.86	74.29	74.71	74.71	71.14	75.71	74.57	74.57	74.71	75.43	74.57	1.25
FrozenEBERT-bs16-ep100-WRONG	75.14	74.00	76.14	75.29	74.71	75.00	75.00	75.00	74.57	74.57	74.94	0.53
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	75.00	75.00	75.14	75.43	74.43	74.29	75.43	75.57	75.43	75.14	75.09	0.41
BERT-bs32+Dropout0.1	75.00	74.86	75.43	76.00	75.57	75.00	75.57	75.57	74.57	75.29	75.29	0.40
EmbraceBERT-bs32-WRONG	74.71	74.86	74.43	75.43	74.43	75.14	74.86	75.71	74.57	75.86	75.00	0.49
EmbraceBERT-bs32+Dropout0.1-WRONG	74.57	75.00	74.71	75.14	75.14	75.14	75.43	74.86	75.14	75.00	75.01	0.23
CondensedEmbraceBERT-bs32	75.14	74.57	75.00	77.00	74.86	75.00	75.14	75.00	74.71	75.00	75.14	0.64
CondensedEmbraceBERT-bs32+Dropout0.1	75.14	75.14	75.00	77.00	74.14	75.57	75.14	75.57	74.71	75.29	75.27	0.70
CondensedEmbraceBERT-bs32+Dropout0.3	75.14	75.00	75.43	75.14	74.71	75.00	75.43	75.29	75.00	75.00	75.11	0.21
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	74.57	75.14	75.00	75.29	72.43	74.86	74.71	75.14	75.00	72.14	74.43	1.09
FrozenEBERT-bs32-ep100-WRONG	75.29	74.71	75.71	75.71	74.14	75.29	74.00	71.71	73.86	74.57	74.50	1.13
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenCEBERT-bs32-ep100
FrozenCEBERT-bs32-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	75.57	76.57	76.57	76.00	74.86	75.14	75.29	75.71	75.57	75.43	75.67	0.54
RoBERTa-bs16+Dropout0.1	75.57	74.86	75.57	75.86	75.43	75.43	75.71	75.14	76.14	75.00	75.47	0.37
EmbraceRoBERTa-bs16-WRONG	75.71	75.00	75.29	74.86	75.00	75.29	76.00	75.71	75.29	76.00	75.41	0.40
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	76.14	76.00	76.00	76.00	75.14	75.43	75.00	74.86	75.14	75.71	75.54	0.46
EmbraceRoBERTa-bs16+Dropout0.3
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	75.29	75.29	75.86	74.71	74.71	75.00	75.43	75.29	74.86	75.86	75.23	0.39
CondensedEmbraceRoBERTa-bs16+Dropout0.3	75.00	75.43	75.71	74.43	75.00	75.14	74.86	74.86	75.57	75.14	75.11	0.36
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	75.00	75.71	74.71	71.29	74.29	75.86	74.71	75.29	74.57	74.86	74.63	1.21
FrozenERoBERTa-bs16-ep100-WRONG	71.43	71.86	75.71	74.86	74.71	75.71	74.71	75.29	75.29	75.00	74.46	1.45
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	74.86	75.29	75.86	75.43	74.57	76.14	75.57	74.86	75.86	74.29	75.27	0.58
RoBERTa-bs32+Dropout0.1	72.00	75.71	75.57	76.00	75.57	75.57	75.71	74.57	75.29	74.86	75.09	1.11
EmbraceRoBERTa-bs32-WRONG	75.57	75.43	74.29	75.14	74.57	72.29	74.86	75.14	75.14	75.86	74.83	0.95
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	75.71	74.86	75.57	75.43	74.57	75.14	75.86	75.14	74.86	76.00	75.31	0.45
EmbraceRoBERTa-bs32+Dropout0.3
CondensedEmbraceRoBERTa-bs32
CondensedEmbraceRoBERTa-bs32+Dropout0.1	75.57	74.43	76.29	74.86	74.43	75.71	74.43	75.43	75.00	76.00	75.21	0.65
CondensedEmbraceRoBERTa-bs32+Dropout0.3	75.29	75.71	75.43	74.71	74.43	75.14	75.43	74.71	74.57	75.71	75.11	0.45
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	74.71	76.14	74.00	75.14	71.71	75.71	74.86	71.14	75.29	74.71	74.34	1.56
FrozenERoBERTa-bs32-ep100-WRONG	74.00	75.00	74.71	75.14	74.29	74.57	72.43	74.29	73.57	74.71	74.27	0.76
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100
FrozenCERoBERTa-bs32-ep100+Dropout0.1

• Epoch 30 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ----------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | RoBERTa-bs16 | | EmbraceRoBERTa-bs16-WRONG |74.86|75.14|74.86|75.71|75.57|76.14|75.86|74.43|74.71| - |75.25|0.55|

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std

gtts-sphinx

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	96.29	95.86	96.29	96.57	96.71	96.29	95.86	96.00	96.29	96.00	96.21	0.27
BERT-bs16+Dropout0.1	96.43	95.43	95.57	96.29	96.29	96.43	96.14	96.57	96.00	96.43	96.16	0.36
EmbraceBERT-bs16-WRONG	95.86	96.00	96.29	95.86	96.29	96.43	95.71	95.86	96.29	95.71	96.03	0.25
EmbraceBERT-bs16+Dropout0.1-WRONG
CondensedEmbraceBERT-bs16	95.71	96.00	95.71	96.43	95.71	95.71	96.43	95.43	95.71	95.43	95.83	0.34
CondensedEmbraceBERT-bs16+Dropout0.1	95.86	95.71	96.00	95.86	95.57	96.43	95.86	95.43	96.00	95.71	95.84	0.26
CondensedEmbraceBERT-bs16+Dropout0.3	96.00	95.57	95.86	95.86	95.71	95.57	96.14	95.86	96.43	95.57	95.86	0.26
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	96.43	95.57	95.86	95.29	96.71	95.86	95.43	95.86	96.00	95.29	95.83	0.45
FrozenEBERT-bs16-ep100-WRONG	96.00	95.00	95.29	96.00	95.14	94.43	96.00	95.86	96.29	94.86	95.49	0.59
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	96.29	95.86	95.71	95.71	96.14	95.57	95.43	96.43	95.86	95.86	95.89	0.30
BERT-bs32+Dropout0.1	96.00	95.86	96.14	96.00	96.14	96.00	96.29	96.14	96.14	96.29	96.10	0.13
EmbraceBERT-bs32-WRONG	95.14	95.00	95.14	95.14	95.86	96.00	95.71	96.00	95.57	95.71	95.53	0.37
EmbraceBERT-bs32+Dropout0.1-WRONG	95.00	95.29	94.71	95.86	95.14	95.29	95.86	95.57	96.00	95.86	95.46	0.41
CondensedEmbraceBERT-bs32	95.29	95.29	95.43	95.00	95.57	95.57	95.86	95.71	96.29	95.14	95.51	0.36
CondensedEmbraceBERT-bs32+Dropout0.1	95.86	95.29	95.00	95.71	96.43	95.29	96.00	95.86	95.29	95.57	95.63	0.41
CondensedEmbraceBERT-bs32+Dropout0.3	95.29	95.43	95.43	95.86	96.71	95.00	96.29	95.86	95.86	96.14	95.79	0.49
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	96.29	95.00	95.86	95.29	96.57	95.00	95.57	95.86	95.71	96.14	95.73	0.50
FrozenEBERT-bs32-ep100-WRONG	95.00	96.14	95.57	95.29	94.14	94.29	96.14	95.14	96.43	96.00	95.41	0.75
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenCEBERT-bs32-ep100
FrozenCEBERT-bs32-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	96.29	96.57	96.43	96.29	96.00	96.43	96.00	96.29	96.43	96.29	96.30	0.17
RoBERTa-bs16+Dropout0.1	96.43	96.14	96.57	95.71	96.86	96.57	96.14	95.86	97.14	96.29	96.37	0.42
EmbraceRoBERTa-bs16-WRONG	96.86	95.57	95.86	95.71	96.43	96.00	96.29	96.00	96.43	96.29	96.14	0.37
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	95.86	96.14	96.43	96.43	95.43	96.43	95.57	95.43	96.57	96.57	96.09	0.45
EmbraceRoBERTa-bs16+Dropout0.3
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	96.57	95.43	96.86	96.71	96.29	96.57	96.86	95.00	97.14	96.14	96.36	0.64
CondensedEmbraceRoBERTa-bs16+Dropout0.3	96.14	96.29	96.43	95.86	96.57	96.43	96.29	95.71	96.71	96.00	96.24	0.30
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	94.71	95.43	95.86	96.00	94.43	96.14	95.71	95.71	95.57	95.29	95.49	0.52
FrozenERoBERTa-bs16-ep100-WRONG	94.71	96.29	95.57	94.29	94.86	94.86	95.86	95.57	96.14	96.29	95.44	0.68
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	96.43	95.86	96.29	96.29	96.57	96.14	96.43	95.57	97.00	96.29	96.29	0.37
RoBERTa-bs32+Dropout0.1	96.57	95.71	96.57	95.43	96.43	95.71	96.57	95.29	96.86	96.29	96.14	0.53
EmbraceRoBERTa-bs32-WRONG	96.14	96.14	96.14	96.29	96.29	96.43	96.57	95.86	96.43	95.71	96.20	0.25
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	96.14	96.29	96.43	96.43	96.43	95.86	96.14	96.00	96.14	95.43	96.13	0.30
EmbraceRoBERTa-bs32+Dropout0.3
CondensedEmbraceRoBERTa-bs32
CondensedEmbraceRoBERTa-bs32+Dropout0.1	96.43	95.86	96.29	95.71	96.29	96.29	96.29	95.86	96.14	95.86	96.10	0.24
CondensedEmbraceRoBERTa-bs32+Dropout0.3	96.71	95.57	96.43	95.86	95.57	96.00	96.29	95.29	96.43	95.57	95.97	0.45
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	94.57	95.57	96.14	95.57	95.71	95.14	95.57	95.29	94.86	95.29	95.37	0.42
FrozenERoBERTa-bs32-ep100-WRONG	96.57	96.29	94.71	96.71	96.14	96.00	95.29	94.00	96.71	95.00	95.74	0.89
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100
FrozenCERoBERTa-bs32-ep100+Dropout0.1

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std

gtts-witai

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	98.71	98.71	98.57	98.57	98.43	98.57	98.57	99.00	98.57	98.71	98.64	0.15
BERT-bs16+Dropout0.1	98.57	98.43	98.29	98.43	98.57	98.57	98.71	98.43	98.43	98.71	98.51	0.13
EmbraceBERT-bs16-WRONG	98.29	98.71	98.43	98.57	98.86	98.86	98.57	98.14	98.29	98.43	98.51	0.23
EmbraceBERT-bs16+Dropout0.1-WRONG
CondensedEmbraceBERT-bs16	98.43	98.43	98.43	98.57	98.71	98.43	98.43	98.43	98.86	98.57	98.53	0.14
CondensedEmbraceBERT-bs16+Dropout0.1	98.43	98.43	98.43	98.43	98.43	98.71	98.57	98.29	98.57	98.43	98.47	0.11
CondensedEmbraceBERT-bs16+Dropout0.3	98.57	98.29	98.29	98.29	98.57	98.29	98.43	98.57	98.71	98.71	98.47	0.17
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	98.29	98.71	98.14	98.57	97.71	98.14	98.29	98.00	98.00	98.29	98.21	0.27
FrozenEBERT-bs16-ep100-WRONG	97.71	98.29	98.71	98.43	98.29	98.43	99.14	98.29	99.29	99.00	98.56	0.45
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	98.86	98.71	98.43	98.43	98.57	98.71	98.71	98.57	98.29	98.29	98.56	0.19
BERT-bs32+Dropout0.1	98.57	98.43	98.29	98.43	98.29	98.43	98.57	98.43	98.57	98.57	98.46	0.11
EmbraceBERT-bs32-WRONG	98.57	98.43	98.57	98.43	98.57	98.71	98.43	98.86	98.43	98.71	98.57	0.14
EmbraceBERT-bs32+Dropout0.1-WRONG	98.43	98.86	98.86	98.71	98.43	98.57	98.14	98.43	98.43	98.57	98.54	0.21
CondensedEmbraceBERT-bs32	98.57	98.71	98.57	98.29	98.43	98.71	98.71	98.43	98.57	98.71	98.57	0.14
CondensedEmbraceBERT-bs32+Dropout0.1	98.57	98.86	98.57	98.43	98.71	98.43	98.71	98.29	98.71	98.57	98.59	0.16
CondensedEmbraceBERT-bs32+Dropout0.3	98.29	98.43	98.43	98.86	98.71	98.43	98.71	98.14	98.71	98.71	98.54	0.22
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	99.14	98.29	98.71	98.14	98.43	98.29	98.71	98.57	99.00	98.71	98.60	0.31
FrozenEBERT-bs32-ep100-WRONG	98.86	98.00	98.57	98.43	98.57	98.43	98.43	98.86	98.86	98.86	98.59	0.27
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenCEBERT-bs32-ep100
FrozenCEBERT-bs32-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	98.43	98.71	98.71	98.57	98.71	98.71	98.71	98.43	98.71	98.43	98.61	0.13
RoBERTa-bs16+Dropout0.1	98.43	99.00	98.71	98.57	98.71	98.86	98.71	98.71	98.57	98.43	98.67	0.17
EmbraceRoBERTa-bs16-WRONG	98.71	98.71	98.86	98.71	98.29	98.86	98.57	98.71	98.71	98.57	98.67	0.16
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	98.57	98.71	98.86	98.71	98.43	99.00	98.71	98.71	98.57	98.71	98.70	0.15
EmbraceRoBERTa-bs16+Dropout0.3
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	98.86	98.71	98.86	98.86	98.57	98.57	98.57	98.71	98.86	98.57	98.71	0.13
CondensedEmbraceRoBERTa-bs16+Dropout0.3	98.43	98.71	98.86	98.57	98.43	98.57	98.43	98.71	98.86	98.86	98.64	0.17
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	98.71	98.29	98.71	98.29	98.00	98.71	98.43	98.43	98.43	98.71	98.47	0.23
FrozenERoBERTa-bs16-ep100-WRONG	97.14	98.43	98.14	98.71	98.29	98.43	98.29	98.57	98.57	98.43	98.30	0.42
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	98.86	98.71	98.86	98.57	98.71	98.71	98.86	98.86	98.86	98.71	98.77	0.09
RoBERTa-bs32+Dropout0.1	99.00	98.86	98.71	98.57	98.71	98.57	98.57	98.86	98.71	98.86	98.74	0.14
EmbraceRoBERTa-bs32-WRONG	98.57	99.00	98.57	98.71	98.71	98.71	98.71	98.86	98.71	98.57	98.71	0.13
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	98.57	98.71	98.71	98.86	98.86	98.57	98.29	99.14	98.86	98.43	98.70	0.23
EmbraceRoBERTa-bs32+Dropout0.3
CondensedEmbraceRoBERTa-bs32
CondensedEmbraceRoBERTa-bs32+Dropout0.1	98.57	98.86	98.71	98.71	98.57	98.71	98.57	98.57	98.71	98.86	98.69	0.11
CondensedEmbraceRoBERTa-bs32+Dropout0.3	98.71	98.86	98.86	98.43	98.71	98.71	98.71	98.71	98.57	98.71	98.70	0.12
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	98.86	98.71	98.43	98.71	98.43	98.71	98.57	98.86	98.57	98.43	98.63	0.16
FrozenERoBERTa-bs32-ep100-WRONG	98.86	98.71	98.14	98.57	98.43	98.71	98.57	98.43	99.14	98.71	98.63	0.26
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100
FrozenCERoBERTa-bs32-ep100+Dropout0.1

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std

macsay-google

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	91.57	92.43	91.57	92.29	92.71	91.86	91.57	92.29	92.14	92.57	92.10	0.41
BERT-bs16+Dropout0.1	92.14	92.29	91.57	93.00	92.57	92.57	92.43	92.29	91.57	92.29	92.27	0.42
EmbraceBERT-bs16-WRONG	92.71	91.29	91.29	92.14	92.00	92.00	92.00	91.43	92.43	91.57	91.89	0.46
EmbraceBERT-bs16+Dropout0.1-WRONG
CondensedEmbraceBERT-bs16	92.00	92.29	92.71	91.71	92.00	91.57	91.86	91.29	92.14	91.71	91.93	0.38
CondensedEmbraceBERT-bs16+Dropout0.1	91.43	91.86	92.43	91.71	91.86	92.29	91.71	91.43	92.14	91.71	91.86	0.32
CondensedEmbraceBERT-bs16+Dropout0.3	91.71	92.00	92.57	91.86	92.00	91.71	92.43	90.71	92.14	92.43	91.96	0.50
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	92.29	93.71	92.14	91.57	91.14	92.71	92.57	91.57	91.00	92.57	92.13	0.78
FrozenEBERT-bs16-ep100-WRONG	93.00	91.71	93.43	93.14	91.00	90.71	92.00	91.57	91.57	91.29	91.94	0.89
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	91.00	91.86	91.57	91.71	91.29	91.71	92.29	91.71	91.86	92.43	91.74	0.40
BERT-bs32+Dropout0.1	91.00	92.29	92.00	90.86	90.71	92.14	91.57	92.14	92.00	92.14	91.69	0.57
EmbraceBERT-bs32-WRONG	91.29	92.43	92.43	91.14	90.86	91.14	91.00	91.57	91.57	91.29	91.47	0.52
EmbraceBERT-bs32+Dropout0.1-WRONG	90.86	92.00	91.43	91.86	92.71	91.00	91.29	90.86	92.29	91.29	91.56	0.60
CondensedEmbraceBERT-bs32	91.43	92.71	92.00	90.86	92.29	91.86	91.00	91.14	92.14	91.14	91.66	0.60
CondensedEmbraceBERT-bs32+Dropout0.1	91.14	92.14	90.29	90.86	91.71	91.43	91.14	90.86	92.29	91.00	91.29	0.59
CondensedEmbraceBERT-bs32+Dropout0.3	91.14	92.00	90.71	90.86	91.86	91.86	91.00	91.71	92.29	91.14	91.46	0.52
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	90.86	92.57	91.57	91.00	91.57	91.14	92.14	92.00	92.14	91.86	91.69	0.53
FrozenEBERT-bs32-ep100-WRONG	91.57	91.00	91.57	91.57	90.43	92.29	91.43	91.43	92.14	92.14	91.56	0.53
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenCEBERT-bs32-ep100
FrozenCEBERT-bs32-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	92.86	92.29	92.86	93.14	92.57	91.57	92.71	92.00	92.71	93.14	92.59	0.48
RoBERTa-bs16+Dropout0.1	92.14	91.71	92.00	93.57	92.86	92.29	92.29	91.86	92.86	93.29	92.49	0.59
EmbraceRoBERTa-bs16-WRONG	92.57	91.86	90.86	91.71	93.00	92.71	92.86	93.00	91.86	92.14	92.26	0.66
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	92.43	91.86	91.43	92.29	91.86	93.43	92.86	91.71	92.57	91.86	92.23	0.58
EmbraceRoBERTa-bs16+Dropout0.3
CondensedEmbraceRoBERTa-bs16	92.57	91.71	91.86	91.57	91.86	92.14	91.71	91.86	92.29	92.29	91.99	0.30
CondensedEmbraceRoBERTa-bs16+Dropout0.1	92.43	90.86	91.57	92.43	92.14	92.71	92.43	91.29	92.14	92.57	92.06	0.58
CondensedEmbraceRoBERTa-bs16+Dropout0.3	92.71	92.57	92.29	92.29	92.57	92.43	91.86	92.43	93.29	92.57	92.50	0.35
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	92.00	92.00	92.14	92.86	91.57	92.14	92.43	92.71	90.14	92.57	92.06	0.73
FrozenERoBERTa-bs16-ep100-WRONG	92.86	91.57	92.57	91.57	92.57	92.43	92.29	91.57	91.57	92.43	92.14	0.49
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	92.71	92.00	92.71	93.00	92.29	91.86	92.43	91.86	92.00	92.14	92.30	0.38
RoBERTa-bs32+Dropout0.1	91.57	92.43	92.86	92.43	92.00	91.86	91.71	91.14	92.14	91.00	91.91	0.55
EmbraceRoBERTa-bs32-WRONG	91.43	92.14	91.29	91.00	92.86	92.00	90.86	92.57	92.43	91.43	91.80	0.66
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	92.00	92.57	91.43	90.00	90.71	92.00	91.57	91.14	93.14	91.14	91.57	0.86
EmbraceRoBERTa-bs32+Dropout0.3
CondensedEmbraceRoBERTa-bs32	92.43	92.57	91.14	91.71	91.00	92.14	91.29	92.71	92.29	91.71	91.90	0.59
CondensedEmbraceRoBERTa-bs32-Dropout0.1	92.43	92.14	91.43	91.71	92.14	92.29	91.57	91.86	92.00	91.14	91.87	0.39
CondensedEmbraceRoBERTa-bs32+Dropout0.3	92.29	92.29	91.57	92.14	91.86	91.71	91.71	92.14	90.86	91.29	91.79	0.44
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	92.86	93.43	91.57	92.86	91.57	92.57	92.14	91.43	92.00	92.14	92.26	0.63
FrozenERoBERTa-bs32-ep100-WRONG	91.29	91.43	90.43	92.71	91.00	91.43	91.86	93.86	92.29	92.14	91.84	0.92
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100
FrozenCERoBERTa-bs32-ep100+Dropout0.1

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std

macsay-sphinx

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	90.29	90.00	90.00	89.14	90.00	89.86	88.43	91.00	90.29	89.57	89.86	0.66
BERT-bs16+Dropout0.1	89.43	89.00	89.43	90.29	89.14	90.00	89.57	90.57	89.57	91.00	89.80	0.61
EmbraceBERT-bs16-WRONG	89.00	87.71	87.57	89.29	88.00	87.71	88.14	89.29	88.57	88.43	88.37	0.62
EmbraceBERT-bs16+Dropout0.1-WRONG
CondensedEmbraceBERT-bs16	88.00	88.71	88.71	87.29	88.14	87.57	88.86	87.71	88.57	88.00	88.16	0.51
CondensedEmbraceBERT-bs16+Dropout0.1	88.14	87.86	89.14	87.00	88.00	88.14	89.29	89.00	88.29	87.14	88.20	0.74
CondensedEmbraceBERT-bs16+Dropout0.3	87.57	88.29	89.57	89.43	86.86	87.86	88.86	88.71	87.43	89.14	88.37	0.87
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	89.86	90.57	90.29	90.29	90.29	87.71	89.86	90.43	89.86	88.14	89.73	0.94
FrozenEBERT-bs16-ep100-WRONG	88.86	88.86	88.57	89.14	88.86	86.86	87.71	88.43	88.14	89.86	88.53	0.78
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	88.14	89.71	88.43	88.14	87.86	88.29	87.71	89.14	88.57	89.43	88.54	0.64
BERT-bs32+Dropout0.1	87.71	88.57	89.29	88.57	87.00	87.71	88.14	89.29	88.00	89.71	88.40	0.81
EmbraceBERT-bs32-WRONG	85.00	85.71	85.14	88.43	87.00	86.86	87.14	86.29	86.86	86.57	86.50	0.97
EmbraceBERT-bs32+Dropout0.1-WRONG	86.14	87.29	85.86	87.43	86.86	87.00	86.86	85.43	86.57	87.57	86.70	0.67
CondensedEmbraceBERT-bs32	86.14	86.00	86.71	86.57	86.71	86.71	87.57	86.14	85.29	87.14	86.50	0.61
CondensedEmbraceBERT-bs32+Dropout0.1	86.71	86.71	86.57	87.57	87.71	87.57	87.00	87.14	87.29	85.29	86.96	0.67
CondensedEmbraceBERT-bs32+Dropout0.3	86.00	86.86	86.14	86.86	86.86	87.14	87.43	87.00	86.57	85.71	86.66	0.52
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	89.71	90.00	88.29	89.43	90.14	89.43	89.57	88.43	90.00	89.43	89.44	0.60
FrozenEBERT-bs32-ep100-WRONG	88.29	88.29	86.29	88.86	89.86	88.14	89.29	89.00	88.14	88.43	88.46	0.90
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenCEBERT-bs32-ep100
FrozenCEBERT-bs32-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	90.00	89.71	91.29	89.43	90.57	90.86	88.86	88.57	90.29	90.57	90.01	0.83
RoBERTa-bs16+Dropout0.1	89.14	89.14	89.00	89.57	90.43	89.71	90.43	88.71	89.86	89.57	89.56	0.55
EmbraceRoBERTa-bs16-WRONG	90.14	90.86	90.57	89.14	89.29	90.57	89.43	89.00	90.43	90.00	89.94	0.64
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	88.71	88.57	88.86	88.71	89.71	90.14	90.29	89.00	88.86	89.86	89.27	0.62
EmbraceRoBERTa-bs16+Dropout0.3
CondensedEmbraceRoBERTa-bs16	89.86	89.43	90.57	90.57	88.86	89.57	89.71	89.14	89.29	89.43	89.64	0.54
CondensedEmbraceRoBERTa-bs16+Dropout0.1	89.00	89.43	88.71	89.14	89.14	89.29	90.29	89.29	89.14	89.43	89.29	0.39
CondensedEmbraceRoBERTa-bs16+Dropout0.3	89.29	89.29	89.00	89.43	89.29	89.14	88.71	89.14	89.29	89.57	89.21	0.22
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	90.57	90.29	89.86	89.57	90.43	89.29	89.14	90.57	89.29	88.29	89.73	0.71
FrozenERoBERTa-bs16-ep100-WRONG	89.57	89.00	91.00	90.29	89.57	89.00	88.57	90.57	91.00	90.29	89.89	0.82
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	89.00	89.00	89.00	89.43	88.00	88.00	88.29	89.00	89.00	88.71	88.74	0.46
RoBERTa-bs32+Dropout0.1	90.14	89.00	89.29	90.00	89.00	88.86	89.57	88.71	88.29	88.29	89.11	0.61
EmbraceRoBERTa-bs32-WRONG	88.71	88.57	90.00	89.00	89.29	89.00	88.14	89.00	88.57	87.86	88.81	0.57
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	87.86	88.57	89.71	87.71	89.00	88.57	89.43	89.29	89.43	89.14	88.87	0.64
EmbraceRoBERTa-bs32+Dropout0.3
CondensedEmbraceRoBERTa-bs32	89.14	88.57	88.57	88.57	88.71	88.29	89.14	88.43	89.14	88.71	88.73	0.30
CondensedEmbraceRoBERTa-bs32+Dropout0.1	87.57	88.14	89.14	88.29	88.71	88.14	89.29	88.00	90.29	88.71	88.63	0.75
CondensedEmbraceRoBERTa-bs32+Dropout0.3	87.57	88.86	88.71	88.71	89.14	89.14	89.00	88.43	89.71	88.71	88.80	0.53
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	88.71	91.14	87.57	88.71	89.29	91.00	89.14	88.57	89.57	89.43	89.31	1.03
FrozenERoBERTa-bs32-ep100-WRONG	90.43	87.14	89.00	90.43	88.86	90.14	91.43	90.29	89.86	87.86	89.54	1.25
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100
FrozenCERoBERTa-bs32-ep100+Dropout0.1

• Epoch 100 - bs32 (COLING paper)

BERTwithProjection deleted by mistake - Re-running now (LDS3-GPU0)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
LDS1 NAS - BERT-bs32	90.57	92.00	91.29	92.43	91.14	92.14	90.57	92.14	92.14	91.86	91.63	0.65
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS1 - BERTwithAtt-bs32	91.43	92.29	91.43	91.71	91.71	90.71	92.43	91.29	90.57	92.86	91.64	0.69
LDS1 - BERTwithAttClsProjection-bs32	90.57	91.14	91.71	92.14	91.14	91.71	90.71	91.43	90.57	90.86	91.20	0.51
LDS1 - BERTwithProjection-bs32	90.14	92.14	91.86	91.29	92.14	92.29	91.00	91.14	91.43	90.57	91.40	0.68
LDS1 - BERTwithProjectionAtt-bs32	91.14	91.14	90.14	91.14	92.14	91.14	91.00	91.00	92.29	91.43	91.26	0.57
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS1 - EmbraceBERT-bs32	91.00	90.43	89.86	91.29	91.71	92.00	91.00	91.00	89.57	90.43	90.83	0.73
LDS1 - EmbraceBERT-bs32-p_attclsqw	89.71	90.29	92.86	92.86	91.43	91.14	90.57	91.00	92.00	91.71	91.36	0.98
LDS1 - EmbraceBERTwithProj-bs32	92.00	91.43	90.71	91.00	92.00	91.29	91.29	90.29	91.43	91.71	91.31	0.51
LDS1 - EmbraceBERTwithProj-bs32-p_attclsqw	91.29	92.00	91.57	91.57	91.43	91.57	91.29	91.57	91.00	90.43	91.37	0.40
LOCAL - EmbraceBERTconcatAtt-bs32	92.14	91.29	91.14	91.86	91.14	93.14	90.86	91.43	91.43	91.71	91.61	0.62
LOCAL - EmbraceBERTconcatAtt-bs32-p_attclsqw	91.14	92.57	93.57	91.57	90.86	92.00	90.86	90.43	91.00	90.57	91.46	0.94
LDS3 - EmbraceBERTconcatAttwithProj-bs32	91.43	91.57	90.71	91.71	91.14	90.86	91.57	90.86	90.71	91.14	91.17	0.36
LDS3 - EmbraceBERTconcatAttwithProj-bs32-p_attclsqw	91.43	91.86	91.86	91.43	92.71	91.14	90.86	91.00	91.57	91.14	91.50	0.52
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LOCAL - EmbraceBERT-KVQ-bs32	92.29	91.57	90.43	91.86	91.71	91.43	91.71	90.86	92.00	91.14	91.50	0.53
LOCAL - EmbraceBERT-KVQ-bs32-p_attclsqw	92.00	90.29	90.57	90.71	91.29	90.71	90.86	92.14	90.86	90.57	91.00	0.59
LDS1 - EmbraceBERT-KVQ-withProj-bs32	91.43	92.29	92.00	91.57	92.00	90.86	91.57	91.86	91.71	91.71	91.70	0.37
LDS1 - EmbraceBERT-KVQ-withProj-bs32-p_attclsqw	90.71	91.86	91.43	91.71	91.00	90.86	92.14	90.57	90.86	90.86	91.20	0.52
LDS3 - EmbraceBERTconcatAtt-KVQ-bs32	92.14	91.43	90.57	91.00	92.14	91.57	91.71	91.00	90.43	90.57	91.26	0.61
LDS3 - EmbraceBERTconcatAtt-KVQ-bs32-p_attclsqw	91.86	91.71	91.43	90.57	91.29	92.29	91.14	92.14	91.14	91.00	91.46	0.51
LDS3 - EmbraceBERTconcatAtt-KVQ-withProj-bs32	92.00	91.71	90.57	92.00	91.43	92.00	91.29	92.00	92.00	91.57	91.66	0.44
LDS3 - EmbraceBERTconcatAtt-KVQ-withProj-bs32-p_attclsqw	91.71	92.14	90.86	90.86	91.57	91.00	90.71	91.57	91.43	91.71	91.36	0.45

• Epoch 100 - BERT bs48

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
LDS3 - BERT-bs48	90.71	90.86	92.14	91.43	90.71	91.57	91.86	92.14	91.71	91.71	91.49	0.52
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS1 - BERTwithAtt-bs48	91.57	91.86	90.86	91.71	91.00	91.86	91.71	91.29	90.14	91.29	91.33	0.52
LDS1 - BERTwithAttClsProjection-bs48	91.00	90.43	91.00	91.71	91.43	91.43	91.57	91.71	89.86	90.57	91.07	0.59
LDS1 - BERTwithProjection-bs48	91.14	91.14	91.57	89.86	91.43	91.00	91.43	90.57	90.71	91.29	91.01	0.49
LDS1 - BERTwithProjectionAtt-bs48	91.29	90.43	89.43	91.71	91.00	90.14	90.86	90.71	91.00	91.14	90.77	0.61
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS3 - EmbraceBERT-bs48	92.57	91.86	90.71	90.43	90.14	90.43	90.71	91.86	91.14	91.29	91.11	0.74
LDS3 - EmbraceBERT-bs48-p_attclsqw	90.43	91.43	90.86	91.43	91.00	90.43	90.14	92.29	89.57	90.43	90.80	0.73
LDS3 - EmbraceBERTwithProj-bs48	91.71	90.86	92.00	89.71	90.71	91.86	91.71	90.71	92.00	92.00	91.33	0.74
LDS3 - EmbraceBERTwithProj-bs48-p_attclsqw	91.57	91.00	90.86	90.86	90.00	91.43	91.14	91.00	91.14	91.00	91.00	0.40
LDS3 - EmbraceBERTconcatAtt-bs48	90.57	91.00	92.14	91.29	90.57	92.14	90.86	89.86	91.29	91.71	91.14	0.69
LDS3 - EmbraceBERTconcatAtt-bs48-p_attclsqw	91.14	91.71	91.57	91.29	91.86	91.29	91.86	91.71	90.29	91.57	91.43	0.45
LDS3 - EmbraceBERTconcatAttwithProj-bs48	91.14	91.57	91.43	91.14	91.86	93.00	90.71	91.43	91.71	91.43	91.54	0.57
LDS3 - EmbraceBERTconcatAttwithProj-bs48-p_attclsqw	92.29	91.00	92.14	91.14	91.00	92.00	90.29	90.29	91.57	90.57	91.23	0.71
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
LDS3 - EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	91.71	90.71	89.57	91.86	92.29	90.71	91.43	91.14	91.14	89.00	90.96	0.96
LDS3 - EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	92.14	91.43	91.14	90.86	91.14	92.00	90.86	90.71	91.14	91.29	91.27	0.45
LDS3 - EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	90.86	91.86	91.29	91.29	91.00	90.86	89.14	91.29	91.29	92.43	91.13	0.80
LDS3 - EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	92.00	92.14	91.14	90.00	91.43	91.43	91.14	91.14	91.57	91.43	91.34	0.55
LDS1 - EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	91.43	91.71	91.86	92.00	91.43	90.71	90.71	92.00	92.14	91.14	91.51	0.50
LDS1 - EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	91.00	90.14	91.43	91.71	91.43	89.71	90.86	91.71	91.29	90.29	90.96	0.66
LDS1 - EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	90.86	91.29	91.71	91.00	90.29	91.29	90.71	90.57	91.00	90.57	90.93	0.40
LDS1 - EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	90.71	91.43	91.86	91.86	90.57	90.43	90.43	91.14	91.43	91.71	91.16	0.55

macsay-witai

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs16	96.14	96.14	96.71	96.00	96.43	96.14	96.43	96.57	96.00	96.00	96.26	0.25
BERT-bs16+Dropout0.1	96.43	96.00	96.43	96.29	96.14	96.14	96.57	96.57	96.43	96.00	96.30	0.21
EmbraceBERT-bs16-WRONG	95.86	96.00	96.43	95.86	96.71	96.71	96.00	96.14	96.00	96.14	96.19	0.31
EmbraceBERT-bs16+Dropout0.1-WRONG
CondensedEmbraceBERT-bs16	96.14	96.14	96.00	95.86	96.14	96.43	95.86	96.14	96.29	96.57	96.16	0.22
CondensedEmbraceBERT-bs16+Dropout0.1	96.14	95.86	95.86	96.43	96.57	96.43	95.86	96.00	96.00	96.14	96.13	0.25
CondensedEmbraceBERT-bs16+Dropout0.3	95.86	96.00	96.14	96.57	96.29	96.43	96.29	95.86	96.00	96.29	96.17	0.23
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	96.00	96.71	96.29	95.86	95.71	95.86	96.29	95.43	95.43	96.43	96.00	0.40
FrozenEBERT-bs16-ep100-WRONG	95.57	95.43	95.71	96.43	96.00	96.43	95.86	96.29	96.14	95.71	95.96	0.34
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs32	96.00	96.14	96.43	96.57	96.14	96.14	96.43	96.14	96.43	96.57	96.30	0.20
BERT-bs32+Dropout0.1	96.29	96.29	96.57	96.57	96.43	96.43	96.14	96.14	95.29	96.71	96.29	0.38
EmbraceBERT-bs32-WRONG	96.29	96.14	96.14	96.43	96.57	97.00	95.86	96.29	96.29	96.43	96.34	0.29
EmbraceBERT-bs32+Dropout0.1-WRONG	96.14	95.86	96.14	96.43	96.43	96.29	96.00	96.00	96.00	95.86	96.11	0.20
CondensedEmbraceBERT-bs32	96.71	95.86	96.14	96.43	96.57	97.14	96.00	96.00	96.14	97.00	96.40	0.42
CondensedEmbraceBERT-bs32+Dropout0.1	95.71	96.00	96.43	96.71	96.43	96.14	95.71	95.57	96.57	96.00	96.13	0.37
CondensedEmbraceBERT-bs32+Dropout0.3	95.86	95.86	96.29	96.57	96.43	96.29	95.71	95.57	96.57	96.14	96.13	0.34
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs32-ep100	95.71	95.86	96.29	96.43	95.86	96.00	96.43	96.00	96.00	96.86	96.14	0.33
FrozenEBERT-bs32-ep100-WRONG	96.14	96.43	96.29	96.43	96.14	97.00	95.43	96.14	96.71	96.57	96.33	0.40
FrozenEBERT-bs32-ep100+Dropout0.1
FrozenCEBERT-bs32-ep100
FrozenCEBERT-bs32-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	96.29	96.29	96.57	96.43	96.43	96.43	96.29	96.43	96.00	96.57	96.37	0.16
RoBERTa-bs16+Dropout0.1	96.43	96.43	96.14	96.86	96.43	96.71	96.57	95.71	96.57	96.57	96.44	0.30
EmbraceRoBERTa-bs16-WRONG	95.86	96.29	95.71	96.29	96.14	96.14	95.86	96.14	96.14	96.00	96.06	0.18
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	96.29	96.00	96.43	96.57	95.57	96.57	96.29	96.43	96.71	96.43	96.33	0.31
EmbraceRoBERTa-bs16+Dropout0.3
CondensedEmbraceRoBERTa-bs16	96.00	96.43	96.00	96.29	96.29	96.43	96.14	96.14	96.57	96.00	96.23	0.19
CondensedEmbraceRoBERTa-bs16+Dropout0.1	96.71	96.43	96.43	95.86	96.43	96.43	96.00	96.14	96.86	96.43	96.37	0.29
CondensedEmbraceRoBERTa-bs16+Dropout0.3	96.43	96.43	96.43	96.00	96.14	95.71	96.14	96.57	96.14	96.29	96.23	0.24
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	96.57	96.00	96.57	96.29	95.57	95.14	96.00	95.71	95.71	95.29	95.89	0.47
FrozenERoBERTa-bs16-ep100-WRONG	96.14	95.43	96.00	96.00	95.86	96.29	95.43	95.86	96.43	95.71	95.91	0.31
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs32	96.43	96.29	95.86	96.43	96.29	96.43	96.00	96.43	96.00	96.57	96.27	0.23
RoBERTa-bs32+Dropout0.1	95.71	96.29	96.71	96.43	96.29	95.86	96.00	96.86	96.43	96.57	96.31	0.35
EmbraceRoBERTa-bs32-WRONG	96.14	95.86	96.43	96.57	96.57	96.29	95.71	96.43	96.86	95.86	96.27	0.35
EmbraceRoBERTa-bs32+Dropout0.1-WRONG	95.57	96.71	96.29	95.86	96.00	95.86	96.14	96.29	96.43	96.57	96.17	0.34
EmbraceRoBERTa-bs32+Dropout0.3
CondensedEmbraceRoBERTa-bs32	95.57	95.86	95.86	96.14	96.43	96.00	95.86	96.57	96.43	96.00	96.07	0.30
CondensedEmbraceRoBERTa-bs32+Dropout0.1	95.71	96.57	95.71	96.29	95.86	96.14	95.71	96.00	96.14	96.00	96.01	0.27
CondensedEmbraceRoBERTa-bs32+Dropout0.3	95.86	96.71	95.86	96.14	96.00	96.14	96.14	96.00	96.29	96.43	96.16	0.25
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs32-ep100	96.71	96.43	96.00	96.57	96.29	96.57	96.43	96.14	96.00	95.71	96.29	0.30
FrozenERoBERTa-bs32-ep100-WRONG	96.43	96.00	96.14	95.29	95.57	96.29	95.43	96.71	96.71	96.43	96.10	0.49
FrozenERoBERTa-bs32-ep100+Dropout0.1
FrozenCERoBERTa-bs32-ep100
FrozenCERoBERTa-bs32-ep100+Dropout0.1

• Epoch 100 - bs32 (COLING paper)

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs32	96.71	96.57	96.14	96.43	96.71	96.43	96.71	96.86	96.43	96.43	96.54	0.20
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs32	96.14	96.00	96.14	96.29	97.00	96.29	96.71	96.29	96.71	96.57	96.41	0.30
BERTwithAttClsProjection-bs32	96.14	96.71	96.71	96.00	96.57	96.57	96.57	96.43	96.43	96.71	96.49	0.23
BERTwithProjection-bs32	96.71	97.29	95.86	96.71	96.00	96.57	96.14	96.86	96.43	96.00	96.46	0.43
BERTwithProjectionAtt-bs32	96.86	96.57	96.86	96.43	96.43	96.86	96.43	96.71	96.43	96.86	96.64	0.19
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs32	96.57	96.14	97.14	96.71	96.29	96.71	96.57	96.43	96.00	96.43	96.50	0.31
EmbraceBERT-bs32-p_attclsqw	96.14	96.43	96.14	97.00	96.43	96.71	96.43	96.71	96.71	96.57	96.53	0.26
EmbraceBERTwithProj-bs32	96.14	96.29	96.57	96.43	96.43	96.43	96.71	96.14	96.71	96.86	96.47	0.23
EmbraceBERTwithProj-bs32-p_attclsqw	96.00	96.29	96.29	96.57	95.86	96.43	96.57	96.29	97.00	96.29	96.36	0.30
EmbraceBERTconcatAtt-bs32	96.43	96.29	95.86	96.71	96.57	96.29	96.57	96.29	96.57	96.29	96.39	0.23
EmbraceBERTconcatAtt-bs32-p_attclsqw	96.43	96.00	97.14	96.71	95.86	96.14	96.57	96.43	96.29	96.43	96.40	0.35
EmbraceBERTconcatAttwithProj-bs32	96.43	96.71	96.29	96.71	96.43	96.43	96.29	96.71	96.57	96.29	96.49	0.17
EmbraceBERTconcatAttwithProj-bs32-p_attclsqw	96.43	96.86	97.00	96.29	96.71	96.86	96.86	96.71	97.00	96.86	96.76	0.22
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-KVQ-bs32	96.00	96.43	96.86	96.71	95.86	96.43	96.14	96.71	96.29	96.43	96.39	0.31
EmbraceBERT-KVQ-bs32-p_attclsqw	96.29	96.43	95.86	96.29	96.43	96.29	96.86	96.29	96.29	96.43	96.34	0.23
EmbraceBERT-KVQ-withProj-bs32	96.43	96.29	96.57	96.43	96.57	96.57	96.43	96.14	96.29	96.86	96.46	0.19
EmbraceBERT-KVQ-withProj-bs32-p_attclsqw	96.57	96.86	96.00	97.00	96.71	96.86	96.43	96.71	96.57	96.57	96.63	0.26
EmbraceBERTconcatAtt-KVQ-bs32	96.00	96.57	96.71	97.00	96.29	96.43	96.14	96.29	96.57	96.43	96.44	0.27
EmbraceBERTconcatAtt-KVQ-bs32-p_attclsqw	96.29	96.57	95.71	96.57	96.29	96.29	96.29	96.00	96.14	96.14	96.23	0.24
EmbraceBERTconcatAtt-KVQ-withProj-bs32	96.14	96.43	96.86	97.00	96.14	97.00	96.71	96.29	96.43	96.86	96.59	0.32
EmbraceBERTconcatAtt-KVQ-withProj-bs32-p_attclsqw	96.29	96.71	96.71	96.71	96.43	96.71	96.57	96.57	96.29	96.86	96.59	0.19

• Epoch 100 - BERT bs48

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs48	96.29	96.43	96.57	96.86	96.29	96.86	96.29	96.57	95.86	96.57	96.46	0.28
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs48	96.29	96.57	96.14	96.14	96.29	96.71	96.43	96.57	96.43	96.14	96.37	0.19
BERTwithAttClsProjection-bs48	95.86	96.71	96.86	96.29	96.00	96.86	96.43	96.43	96.71	96.71	96.49	0.33
BERTwithProjection-bs48	96.29	96.14	96.14	96.14	96.71	96.14	96.43	96.43	96.43	96.43	96.33	0.18
BERTwithProjectionAtt-bs48	96.57	95.71	96.14	95.86	96.86	96.71	96.86	96.29	96.57	96.71	96.43	0.39
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48	96.14	96.71	96.43	96.86	97.00	96.57	96.71	96.43	96.00	97.14	96.60	0.34
EmbraceBERT-bs48-p_attclsqw	96.71	96.00	96.14	96.29	96.14	96.43	96.43	96.14	96.57	96.57	96.34	0.22
EmbraceBERTwithProj-bs48	96.00	96.29	95.71	96.71	96.43	96.57	96.29	96.57	96.00	96.86	96.34	0.34
EmbraceBERTwithProj-bs48-p_attclsqw	96.71	96.57	96.29	96.14	96.86	97.00	96.57	96.57	96.71	96.43	96.59	0.24
EmbraceBERTconcatAtt-bs48	96.29	96.29	96.14	96.43	96.00	96.57	96.57	96.43	96.57	96.14	96.34	0.19
EmbraceBERTconcatAtt-bs48-p_attclsqw	96.71	96.14	96.57	96.29	96.14	96.29	96.43	96.29	96.57	96.29	96.37	0.18
EmbraceBERTconcatAttwithProj-bs48	96.14	95.86	96.71	97.14	96.43	96.00	96.00	96.43	96.43	96.29	96.34	0.36
EmbraceBERTconcatAttwithProj-bs48-p_attclsqw	96.43	96.43	96.43	96.71	96.71	96.14	96.43	96.43	96.29	96.57	96.46	0.17
--------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery	96.29	96.57	96.14	96.57	96.71	96.14	96.57	96.14	96.57	96.29	96.40	0.21
EmbraceBERT-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	96.14	96.57	96.14	96.29	96.86	96.29	96.57	96.43	96.14	96.86	96.43	0.26
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery	97.14	96.14	96.57	96.00	96.29	96.43	97.14	96.14	96.00	96.57	96.44	0.40
EmbraceBERTwithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	96.57	96.29	96.71	96.43	96.29	96.29	96.00	96.43	96.71	96.71	96.44	0.23
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery	96.29	96.14	96.43	96.71	96.43	96.29	96.71	97.00	96.57	96.29	96.49	0.25
EmbraceBERTconcatatt-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	96.29	96.57	96.29	96.29	97.00	96.14	96.71	96.71	96.71	96.71	96.54	0.26
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery	96.29	96.43	97.14	96.57	96.57	96.57	95.71	95.71	96.57	96.86	96.44	0.43
EmbraceBERTconcatattWithProj-bs48-p_multiheadatt_bertKeyValQuery_attclsqw	96.43	96.71	96.43	96.71	97.14	96.43	96.57	96.57	96.43	96.86	96.63	0.22