webapplications.md

Results

Train/Test with Complete • Test with Incomplete • Train/Test with Incomplete

Complete

• Epoch 3

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	38.98	33.90	40.68	54.24	25.42	10.17	44.07	33.90	47.46	33.90	36.27	11.62
BERT-bs4+Dropout0.1
EmbraceBERT-bs4-WRONG	38.98	27.12	38.98	33.90	35.59	42.37	52.54	45.76	52.54	32.20	40.00	7.99
EmbraceBERT-bs4+Dropout0.1-WRONG	40.68	25.42	45.76	30.51	35.59	40.68	32.20	45.76	42.37	27.12	36.61	7.12
EmbraceBERT-bs4+Dropout0.3-WRONG	23.73	28.81	42.37	27.12	35.59	40.68	33.90	42.37	27.12	25.42	32.71	6.87
CondensedEmbraceBERT-bs4	33.90	32.20	44.07	37.29	40.68	47.46	45.76	42.37	40.68	27.12	39.15	6.13
CondensedEmbraceBERT-bs4+Dropout0.1	30.51	25.42	47.46	35.59	38.98	50.85	28.81	40.68	38.98	25.42	36.27	8.34
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	59.32	50.85	55.93	59.32	54.24	54.24	59.32	52.54	61.02	54.24	56.10	3.26
FrozenEBERT-bs4-ep100-WRONG	57.63	55.93	54.24	64.41	57.63	62.71	64.41	57.63	55.93	57.63	58.81	3.48
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	44.07	23.73	27.12	28.81	22.03	10.17	16.95	23.73	37.29	35.59	26.95	9.51
BERT-bs16+Dropout0.1	40.68	13.56	30.51	16.95	18.64	10.17	6.78	13.56	35.59	22.03	20.85	10.69
EmbraceBERT-bs16-WRONG	10.17	18.64	32.20	28.81	22.03	30.51	30.51	23.73	27.12	23.73	24.75	6.35
EmbraceBERT-bs16+Dropout0.1-WRONG	10.17	22.03	28.81	30.51	22.03	32.20	28.81	18.64	28.81	23.73	24.58	6.35
EmbraceBERT-bs16+Dropout0.3-WRONG	10.17	20.34	28.81	28.81	22.03	32.20	25.42	22.03	30.51	22.03	24.24	6.11
CondensedEmbraceBERT-bs16	10.17	35.59	28.81	25.42	27.12	32.20	25.42	25.42	27.12	22.03	25.93	6.39
CondensedEmbraceBERT-bs16+Dropout0.1	10.17	32.20	28.81	25.42	27.12	33.90	27.12	22.03	18.64	22.03	24.75	6.57
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	44.07	22.03	37.29	52.54	33.90	15.25	38.98	52.54	54.24	32.20	38.31	12.44
FrozenEBERT-bs16-ep100-WRONG	42.37	27.12	47.46	30.51	52.54	47.46	45.76	52.54	33.90	42.37	42.20	8.46
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	27.12	33.90	23.73	44.07	27.12	16.95	27.12	50.85	22.03	22.03	29.49	10.03
RoBERTa-bs4+Dropout0.1
EmbraceRoBERTa-bs4-WRONG	27.12	16.95	16.95	33.90	27.12	20.34	25.42	27.12	27.12	16.95	23.90	5.49
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	27.12	16.95	16.95	27.12	25.42	32.20	23.73	27.12	18.64	22.03	23.73	4.79
EmbraceRoBERTa-bs4+Dropout0.3-WRONG	27.12	16.95	16.95	27.12	16.95	20.34	23.73	27.12	16.95	23.73	21.69	4.34
CondensedEmbraceRoBERTa-bs4	27.12	22.03	18.64	27.12	22.03	33.90	32.20	27.12	30.51	16.95	25.76	5.40
CondensedEmbraceRoBERTa-bs4+Dropout0.1	27.12	16.95	18.64	27.12	22.03	37.29	27.12	27.12	22.03	22.03	24.75	5.48
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	66.10	61.02	64.41	64.41	61.02	59.32	59.32	67.80	59.32	33.90	59.66	9.06
FrozenERoBERTa-bs4-ep100-WRONG	61.02	45.76	33.90	47.46	61.02	55.93	57.63	42.37	42.37	61.02	50.85	9.22
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	27.12	33.90	5.08	32.20	27.12	25.42	27.12	27.12	16.95	23.73	24.58	7.81
RoBERTa-bs16+Dropout0.1	15.25	25.42	5.08	30.51	27.12	37.29	27.12	27.12	18.64	23.73	23.73	8.44
EmbraceRoBERTa-bs16-WRONG	27.12	16.95	16.95	27.12	16.95	28.81	23.73	27.12	10.17	28.81	22.37	6.24
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	27.12	16.95	16.95	27.12	16.95	28.81	30.51	27.12	10.17	25.42	22.71	6.49
EmbraceRoBERTa-bs16+Dropout0.3-WRONG	27.12	16.95	16.95	27.12	8.47	28.81	25.42	27.12	10.17	27.12	21.53	7.31
CondensedEmbraceRoBERTa-bs16	27.12	16.95	20.34	27.12	10.17	33.90	25.42	27.12	13.56	20.34	22.20	6.88
CondensedEmbraceRoBERTa-bs16+Dropout0.1	27.12	16.95	15.25	27.12	8.47	30.51	28.81	27.12	18.64	22.03	22.20	6.80
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	55.93	27.12	30.51	54.24	27.12	33.90	33.90	30.51	49.15	40.68	38.31	10.48
FrozenERoBERTa-bs16-ep100-WRONG	30.51	16.95	22.03	27.12	16.95	33.90	32.20	27.12	18.64	49.15	27.46	9.34
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	76.27	69.49	66.10	71.19	67.80	69.49	67.80	64.41	69.49	66.10	68.81	3.14
BERT-bs4+Dropout0.1	76.27	71.19	69.49	74.58	69.49	69.49	74.58	67.80	74.58	69.49	71.69	2.84
BERT-bs4+Dropout0.3	76.27	67.80	69.49	72.88	67.80	69.49	72.88	67.80	69.49	69.49	70.34	2.65
EmbraceBERT-bs4	71.19	72.88	61.02	71.19	64.41	67.80	74.58	74.58	69.49	72.88	70.00	4.22
EmbraceBERT-bs4+Dropout0.1	71.19	72.88	69.49	66.10	74.58	66.10	71.19	72.88	74.58	77.97	71.69	3.56
EmbraceBERT-bs4+Dropout0.3	72.88	67.80	67.80	67.80	74.58	66.10	74.58	71.19	71.19	67.80	70.17	2.96
EmbraceBERT-bs4+Branches
CondensedEmbraceBERT-bs4	64.41	67.80	62.71	64.41	67.80	69.49	71.19	69.49	67.80	72.88	67.80	3.03
CondensedEmbraceBERT-bs4+Dropout0.1	74.58	64.41	64.41	71.19	72.88	67.80	76.27	72.88	76.27	84.75	72.54	5.81
CondensedEmbraceBERT-bs4+Dropout0.3	69.49	66.10	66.10	69.49	72.88	62.71	72.88	72.88	74.58	74.58	70.17	3.88
CondensedEmbraceBERT-bs4+branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	74.58	69.49	72.88	74.58	67.80	72.88	71.19	71.19	74.58	71.19	72.03	2.18
FrozenEBERT-bs4-ep100	72.88	74.58	62.71	71.19	62.71	72.88	71.19	72.88	71.19	69.49	70.17	3.95
FrozenEBERT-bs4-ep100+Dropout0.1	74.58	72.88	69.49	69.49	72.88	72.88	71.19	71.19	76.27	77.97	72.88	2.63
FrozenEBERT-bs4-ep100+Dropout0.3	76.27	69.49	67.80	69.49	74.58	66.10	69.49	76.27	74.58	74.58	71.86	3.57
FrozenEBERT-bs4-ep100+Branches
FrozenCEBERT-bs4-ep100	69.49	72.88	67.80	71.19	69.49	71.19	74.58	69.49	71.19	76.27	71.36	2.45
FrozenCEBERT-bs4-ep100+Dropout0.1	74.58	64.41	64.41	71.19	71.19	74.58	71.19	72.88	76.27	77.97	71.86	4.30
FrozenCEBERT-bs4-ep100+Dropout0.3	71.19	64.41	67.80	71.19	76.27	71.19	76.27	74.58	69.49	74.58	71.69	3.64
FrozenCEBERT-bs4-ep100+Branches	67.80	74.58	64.41	66.10	66.10	72.88	69.49	57.63	66.10	74.58	67.97	4.94
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	62.71	62.71	64.41	66.10	57.63	59.32	57.63	59.32	59.32	57.63	60.68	2.92
BERT-bs16+Dropout0.1	71.19	66.10	64.41	67.80	59.32	67.80	59.32	61.02	67.80	62.71	64.75	3.85
BERT-bs16+Dropout0.3	67.80	62.71	66.10	67.80	59.32	66.10	57.63	59.32	64.41	62.71	63.39	3.49
EmbraceBERT-bs16	71.19	59.32	57.63	64.41	66.10	67.80	66.10	66.10	66.10	74.58	65.93	4.70
EmbraceBERT-bs16+Dropout0.1	71.19	64.41	49.15	67.80	69.49	66.10	71.19	66.10	64.41	76.27	66.61	6.78
EmbraceBERT-bs16+Dropout0.3	67.80	62.71	57.63	69.49	66.10	67.80	69.49	67.80	64.41	76.27	66.95	4.63
EmbraceBERT-bs16+Branches
CondensedEmbraceBERT-bs16	67.80	59.32	54.24	64.41	61.02	62.71	64.41	69.49	62.71	72.88	63.90	5.03
CondensedEmbraceBERT-bs16+Dropout0.1	71.19	57.63	59.32	67.80	69.49	66.10	64.41	67.80	66.10	79.66	66.95	5.83
CondensedEmbraceBERT-bs16+Dropout0.3	71.19	59.32	55.93	66.10	64.41	67.80	62.71	71.19	66.10	71.19	65.59	4.92
CondensedEmbraceBERT-bs16+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	77.97	71.19	67.80	71.19	69.49	69.49	69.49	64.41	71.19	67.80	70.00	3.31
FrozenEBERT-bs16-ep100	71.19	57.63	62.71	71.19	67.80	69.49	66.10	69.49	69.49	72.88	67.80	4.35
FrozenEBERT-bs16-ep100+Dropout0.1	71.19	66.10	61.02	69.49	69.49	69.49	72.88	66.10	71.19	83.05	70.00	5.42
FrozenEBERT-bs16-ep100+Dropout0.3	71.19	62.71	61.02	67.80	67.80	69.49	69.49	67.80	71.19	77.97	68.64	4.44
FrozenEBERT-bs16-ep100+Branches
FrozenCEBERT-bs16-ep100	74.58	66.10	64.41	69.49	71.19	67.80	69.49	67.80	69.49	74.58	69.49	3.13
FrozenCEBERT-bs16-ep100+Dropout0.1	74.58	61.02	67.80	64.41	69.49	67.80	69.49	69.49	67.80	77.97	68.98	4.49
FrozenCEBERT-bs16-ep100+Dropout0.3	74.58	64.41	66.10	71.19	71.19	69.49	67.80	71.19	67.80	79.66	70.34	4.17
FrozenCEBERT-bs16-ep100+Branches	61.02	76.27	64.41	69.49	69.49	62.71	69.49	57.63	66.10	62.71	65.93	5.11
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	88.14	91.53	86.44	84.75	86.44	88.14	86.44	83.05	86.44	81.36	86.27	2.67
RoBERTa-bs4+Dropout0.1	89.83	86.44	91.53	86.44	89.83	88.14	86.44	79.66	86.44	86.44	87.12	3.05
RoBERTa-bs4+Dropout0.3	91.53	88.14	89.83	81.36	91.53	88.14	88.14	88.14	84.75	88.14	87.97	2.88
EmbraceRoBERTa-bs4	83.05	91.53	88.14	86.44	86.44	88.14	84.75	84.75	84.75	83.05	86.10	2.49
EmbraceRoBERTa-bs4+Dropout0.1	83.05	88.14	91.53	89.83	88.14	83.05	91.53	79.66	83.05	91.53	86.95	4.16
EmbraceRoBERTa-bs4+Dropout0.3	83.05	88.14	89.83	86.44	84.75	81.36	86.44	84.75	86.44	86.44	85.76	2.30
EmbraceRoBERTa-bs4+Branches
CondensedEmbraceRoBERTa-bs4	86.44	88.14	77.97	83.05	86.44	89.83	89.83	77.97	88.14	86.44	85.42	4.17
CondensedEmbraceRoBERTa-bs4+Dropout0.1	83.05	89.83	91.53	86.44	89.83	83.05	88.14	83.05	84.75	91.53	87.12	3.32
CondensedEmbraceRoBERTa-bs4+Dropout0.3	86.44	89.83	91.53	86.44	88.14	84.75	91.53	81.36	86.44	86.44	87.29	2.96
CondensedEmbraceRoBERTa-bs4+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	88.14	89.83	89.83	84.75	86.44	91.53	88.14	84.75	89.83	86.44	87.97	2.20
FrozenERoBERTa-bs4-ep100	84.75	91.53	89.83	86.44	83.05	89.83	84.75	84.75	84.75	84.75	86.44	2.73
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenERoBERTa-bs4-ep100+Branches
FrozenCERoBERTa-bs4-ep100	88.14	88.14	89.83	84.75	84.75	89.83	88.14	83.05	86.44	84.75	86.78	2.25
FrozenCERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100+Dropout0.3
FrozenCERoBERTa-bs4-ep100+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	83.05	67.80	84.75	77.97	71.19	81.36	69.49	79.66	76.27	76.27	76.78	5.47
RoBERTa-bs16+Dropout0.1	88.14	77.97	89.83	84.75	86.44	88.14	79.66	83.05	86.44	83.05	84.75	3.64
RoBERTa-bs16+Dropout0.3	86.44	79.66	88.14	83.05	86.44	83.05	77.97	81.36	86.44	79.66	83.22	3.34
EmbraceRoBERTa-bs16	83.05	83.05	66.10	83.05	77.97	83.05	83.05	79.66	71.19	86.44	79.66	6.02
EmbraceRoBERTa-bs16+Dropout0.1	77.97	76.27	59.32	83.05	79.66	84.75	89.83	71.19	77.97	88.14	78.81	8.42
EmbraceRoBERTa-bs16+Dropout0.3	69.49	69.49	59.32	77.97	71.19	79.66	71.19	71.19	71.19	86.44	72.71	6.88
EmbraceRoBERTa-bs16+Branches
CondensedEmbraceRoBERTa-bs16	61.02	59.32	61.02	66.10	57.63	59.32	62.71	61.02	61.02	67.80	61.69	2.96
CondensedEmbraceRoBERTa-bs16+Dropout0.1	77.97	81.36	62.71	79.66	81.36	86.44	86.44	67.80	74.58	88.14	78.64	7.85
CondensedEmbraceRoBERTa-bs16+Dropout0.3	74.58	72.88	62.71	74.58	64.41	71.19	74.58	71.19	79.66	86.44	73.22	6.47
CondensedEmbraceRoBERTa-bs16+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	88.14	83.05	88.14	83.05	84.75	86.44	88.14	83.05	88.14	81.36	85.42	2.54
FrozenERoBERTa-bs16-ep100	88.14	86.44	81.36	88.14	88.14	84.75	91.53	81.36	89.83	89.83	86.95	3.31
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenERoBERTa-bs16-ep100+Branches
FrozenCERoBERTa-bs16-ep100	88.14	86.44	88.14	88.14	88.14	84.75	93.22	79.66	88.14	89.83	87.46	3.32
FrozenCERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100+Dropout0.3
FrozenCERoBERTa-bs16-ep100+Branches

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	79.66	69.49	72.88	74.58	67.80	72.88	69.49	69.49	72.88	72.88	72.20	3.23
BERT-bs4+Dropout0.1	76.27	69.49	71.19	74.58	69.49	72.88	74.58	71.19	76.27	67.80	72.37	2.84
BERT-bs4+Dropout0.3	77.97	71.19	72.88	72.88	69.49	69.49	77.97	71.19	74.58	71.19	72.88	2.94
EmbraceBERT-bs4	74.58	71.19	66.10	71.19	64.41	66.10	72.88	71.19	71.19	72.88	70.17	3.23
EmbraceBERT-bs4+Dropout0.1	74.58	72.88	71.19	69.49	77.97	71.19	71.19	71.19	79.66	76.27	73.56	3.23
EmbraceBERT-bs4+Dropout0.3	74.58	69.49	67.80	74.58	71.19	66.10	72.88	76.27	77.97	72.88	72.37	3.56
EmbraceBERT-bs4+Branches
CondensedEmbraceBERT-bs4	72.88	72.88	67.80	72.88	67.80	69.49	72.88	69.49	72.88	72.88	71.19	2.14
CondensedEmbraceBERT-bs4+Dropout0.1	72.88	66.10	62.71	72.88	72.88	71.19	71.19	74.58	76.27	81.36	72.20	4.87
CondensedEmbraceBERT-bs4+Dropout0.3	69.49	72.88	61.02	71.19	74.58	69.49	69.49	77.97	74.58	74.58	71.53	4.41
CondensedEmbraceBERT-bs4+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAttClsProjection-bs4	64.41	71.19	69.49	66.10	66.10	71.19	72.88	67.80	71.19	71.19	69.15	2.71
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	79.66	71.19	74.58	74.58	67.80	76.27	71.19	72.88	74.58	74.58	73.73	3.06
FrozenEBERT-bs4-ep100	76.27	71.19	71.19	72.88	71.19	67.80	74.58	77.97	77.97	72.88	73.39	3.13
FrozenEBERT-bs4-ep100+Dropout0.1	79.66	76.27	72.88	69.49	74.58	74.58	74.58	74.58	77.97	81.36	75.59	3.23
FrozenEBERT-bs4-ep100+Dropout0.3	76.27	74.58	69.49	72.88	72.88	69.49	76.27	77.97	74.58	76.27	74.07	2.74
FrozenEBERT-bs4-ep100+Branches
FrozenCEBERT-bs4-ep100	74.58	79.66	67.80	72.88	67.80	69.49	72.88	69.49	72.88	72.88	72.03	3.41
FrozenCEBERT-bs4-ep100+Dropout0.1	72.88	67.80	69.49	69.49	72.88	74.58	72.88	76.27	77.97	77.97	73.22	3.37
FrozenCEBERT-bs4-ep100+Dropout0.3	69.49	71.19	61.02	74.58	76.27	74.58	71.19	76.27	76.27	74.58	72.54	4.47
FrozenCEBERT-bs4-ep100+Branches	67.80	74.58	71.19	71.19	72.88	76.27	77.97	64.41	66.10	71.19	71.36	4.11
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	74.58	71.19	69.49	74.58	69.49	74.58	71.19	66.10	72.88	71.19	71.53	2.60
BERT-bs16+Dropout0.1	72.88	71.19	67.80	72.88	66.10	72.88	71.19	67.80	71.19	71.19	70.51	2.30
BERT-bs16+Dropout0.3	67.80	71.19	67.80	69.49	67.80	71.19	71.19	67.80	71.19	69.49	69.49	1.52
EmbraceBERT-bs16	74.58	62.71	61.02	69.49	67.80	71.19	69.49	69.49	69.49	72.88	68.81	3.95
EmbraceBERT-bs16+Dropout0.1	71.19	64.41	64.41	69.49	66.10	67.80	69.49	71.19	72.88	77.97	69.49	3.94
EmbraceBERT-bs16+Dropout0.3	72.88	59.32	66.10	69.49	66.10	67.80	69.49	71.19	69.49	74.58	68.64	4.03
EmbraceBERT-bs16+Branches
CondensedEmbraceBERT-bs16	72.88	64.41	64.41	69.49	72.88	67.80	69.49	69.49	71.19	74.58	69.66	3.26
CondensedEmbraceBERT-bs16+Dropout0.1	74.58	61.02	67.80	66.10	72.88	72.88	71.19	66.10	69.49	77.97	70.00	4.68
CondensedEmbraceBERT-bs16+Dropout0.3	71.19	64.41	66.10	66.10	74.58	72.88	64.41	74.58	71.19	77.97	70.34	4.56
CondensedEmbraceBERT-bs16+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	72.88	72.88	71.19	74.58	71.19	76.27	72.88	66.10	74.58	71.19	72.37	2.63
FrozenEBERT-bs16-ep100	72.88	64.41	66.10	74.58	69.49	69.49	74.58	71.19	71.19	74.58	70.85	3.37
FrozenEBERT-bs16-ep100+Dropout0.1	72.88	69.49	64.41	72.88	67.80	69.49	72.88	72.88	72.88	76.27	71.19	3.22
FrozenEBERT-bs16-ep100+Dropout0.3	71.19	64.41	67.80	72.88	67.80	67.80	71.19	71.19	72.88	74.58	70.17	2.96
FrozenEBERT-bs16-ep100+Branches
FrozenCEBERT-bs16-ep100	71.19	67.80	64.41	71.19	74.58	69.49	72.88	71.19	72.88	72.88	70.85	2.82
FrozenCEBERT-bs16-ep100+Dropout0.1	74.58	64.41	71.19	69.49	71.19	71.19	71.19	69.49	69.49	76.27	70.85	3.01
FrozenCEBERT-bs16-ep100+Dropout0.3	72.88	66.10	62.71	69.49	77.97	71.19	69.49	74.58	72.88	76.27	71.36	4.38
FrozenCEBERT-bs16-ep100+Branches	69.49	72.88	66.10	74.58	71.19	62.71	74.58	61.02	69.49	62.71	68.47	4.81
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	88.14	91.53	89.83	84.75	86.44	89.83	84.75	86.44	88.14	84.75	87.46	2.30
RoBERTa-bs4+Dropout0.1	89.83	86.44	91.53	86.44	91.53	88.14	86.44	81.36	86.44	83.05	87.12	3.14
RoBERTa-bs4+Dropout0.3	91.53	86.44	91.53	83.05	91.53	88.14	89.83	86.44	84.75	88.14	88.14	2.84
EmbraceRoBERTa-bs4	81.36	86.44	88.14	86.44	84.75	89.83	84.75	86.44	86.44	83.05	85.76	2.30
EmbraceRoBERTa-bs4+Dropout0.1	83.05	89.83	86.44	86.44	86.44	84.75	86.44	81.36	84.75	91.53	86.10	2.82
EmbraceRoBERTa-bs4+Dropout0.3	83.05	86.44	91.53	86.44	84.75	83.05	86.44	86.44	86.44	86.44	86.10	2.25
EmbraceRoBERTa-bs4+Branches
CondensedEmbraceRoBERTa-bs4	88.14	89.83	86.44	84.75	84.75	86.44	89.83	81.36	86.44	86.44	86.44	2.40
CondensedEmbraceRoBERTa-bs4+Dropout0.1	84.75	86.44	89.83	88.14	88.14	86.44	88.14	84.75	83.05	91.53	87.12	2.42
CondensedEmbraceRoBERTa-bs4+Dropout0.3	88.14	86.44	93.22	86.44	83.05	81.36	91.53	81.36	89.83	84.75	86.61	3.90
CondensedEmbraceRoBERTa-bs4+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	89.83	89.83	91.53	84.75	84.75	91.53	83.05	88.14	89.83	84.75	87.80	3.01
FrozenERoBERTa-bs4-ep100	88.14	88.14	84.75	88.14	86.44	89.83	84.75	86.44	88.14	84.75	86.95	1.70
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenERoBERTa-bs4-ep100+Branches
FrozenCERoBERTa-bs4-ep100	84.75	89.83	88.14	86.44	84.75	86.44	89.83	84.75	83.05	86.44	86.44	2.14
FrozenCERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100+Dropout0.3
FrozenCERoBERTa-bs4-ep100+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	89.83	81.36	89.83	83.05	86.44	86.44	88.14	86.44	88.14	81.36	86.10	3.01
RoBERTa-bs16+Dropout0.1	88.14	83.05	89.83	84.75	89.83	91.53	84.75	84.75	89.83	84.75	87.12	2.86
RoBERTa-bs16+Dropout0.3	88.14	84.75	89.83	84.75	88.14	84.75	83.05	86.44	86.44	77.97	85.42	3.14
EmbraceRoBERTa-bs16	86.44	88.14	83.05	89.83	86.44	86.44	93.22	77.97	89.83	89.83	87.12	4.03
EmbraceRoBERTa-bs16+Dropout0.1	88.14	86.44	89.83	89.83	84.75	88.14	93.22	83.05	88.14	86.44	87.80	2.71
EmbraceRoBERTa-bs16+Dropout0.3	84.75	84.75	86.44	86.44	86.44	88.14	94.92	88.14	86.44	86.44	87.29	2.76
EmbraceRoBERTa-bs16+Branches
CondensedEmbraceRoBERTa-bs16	88.14	86.44	89.83	89.83	84.75	86.44	93.22	84.75	88.14	88.14	87.97	2.45
CondensedEmbraceRoBERTa-bs16+Dropout0.1	86.44	88.14	84.75	84.75	89.83	88.14	93.22	83.05	89.83	84.75	87.29	2.96
CondensedEmbraceRoBERTa-bs16+Dropout0.3	86.44	89.83	88.14	88.14	83.05	88.14	88.14	79.66	89.83	86.44	86.78	3.01
CondensedEmbraceRoBERTa-bs16+Branches
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	88.14	84.75	89.83	83.05	88.14	86.44	88.14	86.44	88.14	84.75	86.78	1.98
FrozenERoBERTa-bs16-ep100	83.05	86.44	83.05	91.53	86.44	86.44	89.83	83.05	89.83	88.14	86.78	2.92
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenERoBERTa-bs16-ep100+Branches
FrozenCERoBERTa-bs16-ep100	86.44	84.75	89.83	89.83	84.75	88.14	88.14	84.75	89.83	88.14	87.46	2.03
FrozenCERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100+Dropout0.3
FrozenCERoBERTa-bs16-ep100+Branches

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |76.27|72.88|69.49|72.88|69.49|71.19|74.58|69.49|74.58|71.19|72.20|2.30| | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERTwithAtt-bs8 |71.19|69.49|67.80|67.80|69.49|67.80|62.71|74.58|62.71|79.66|69.32|4.82| | BERTwithProjection-bs8 |62.71|69.49|72.88|69.49|66.10|69.49|55.93|71.19|62.71|72.88|67.29|5.14| | BERTwithProjectionAtt-bs8 |71.19|74.58|72.88|74.58|67.80|67.80|76.27|72.88|71.19|67.80|71.69|2.94| | BERTwithAttClsProjection-bs8 |62.71|67.80|66.10|64.41|59.32|71.19|69.49|66.10|69.49|69.49|66.61|3.48| | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |74.58|66.10|62.71|71.19|66.10|69.49|66.10|72.88|67.80|74.58|69.15|3.85| | EmbraceBERT-bs8-p_att_clsquery_weights |69.49|66.10|74.58|66.10|71.19|72.88|66.10|76.27|67.80|72.88|70.34|3.58| | EmbraceBERTwithProj-bs8 |69.49|71.19|71.19|69.49|71.19|69.49|71.19|69.49|71.19|71.19|70.51|0.83| | EmbraceBERTwithProj-bs8-p_attclsqw |66.10|69.49|71.19|66.10|67.80|69.49|71.19|66.10|67.80|64.41|67.97|2.20| | EBERTconcatAtt-bs8 |64.41|69.49|69.49|66.10|72.88|67.80|72.88|71.19|69.49|74.58|69.83|3.01| | EBERTconcatAtt-bs8_attclsqw |71.19|67.80|67.80|62.71|66.10|69.49|67.80|69.49|69.49|69.49|68.14|2.25| | EBERTconcatAttwithProj-bs8 |74.58|62.71|64.41|62.71|66.10|69.49|66.10|69.49|66.10|67.80|66.95|3.41| | EBERTconcatAttwithProj-bs8_attclsqw |71.19|64.41|71.19|69.49|62.71|66.10|71.19|69.49|74.58|67.80|68.81|3.41| | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |72.88|67.80|64.41|67.80|71.19|72.88|71.19|74.58|69.49|66.10|69.83|3.11| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|72.88|67.80|67.80|71.19|76.27|72.88|72.88|71.19|71.19|71.19|2.63| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |69.49|62.71|69.49|71.19|71.19|72.88|67.80|66.10|71.19|67.80|68.98|2.84| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |66.10|66.10|59.32|69.49|71.19|66.10|67.80|71.19|69.49|64.41|67.12|3.41| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |69.49|72.88|66.10|74.58|71.19|66.10|64.41|67.80|72.88|71.19|69.66|3.26| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|71.19|67.80|69.49|74.58|72.88|64.41|72.88|66.10|72.88|70.00|3.22| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |66.10|66.10|72.88|66.10|69.49|71.19|72.88|64.41|69.49|71.19|68.98|2.94| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |69.49|67.80|71.19|61.02|71.19|71.19|66.10|62.71|69.49|69.49|67.97|3.43|

STT Error: trained with COMPLETE data and tested with INCOMPLETE data

gtts-google - only good with RoBERTa bs16

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	62.71	61.02	64.41	62.71	59.32	59.32	64.41	55.93	61.02	57.63	60.85	2.67
BERT-bs4+Dropout0.1	66.10	62.71	57.63	61.02	57.63	62.71	69.49	57.63	59.32	61.02	61.53	3.72
BERT-bs4+Dropout0.3	64.41	59.32	57.63	61.02	57.63	66.10	66.10	59.32	62.71	57.63	61.19	3.26
FrozenBERT-bs4-ep100	62.71	61.02	64.41	61.02	59.32	57.63	67.80	57.63	61.02	59.32	61.19	2.98
FrozenBERT-bs4-ep100+Dropout0.1	59.32	61.02	59.32	61.02	57.63	62.71	69.49	54.24	62.71	57.63	60.51	3.87
FrozenBERT-bs4-ep100+Dropout0.3	61.02	61.02	61.02	59.32	55.93	64.41	66.10	59.32	61.02	61.02	61.02	2.63
EmbraceBERT-bs4	54.24	59.32	62.71	62.71	57.63	57.63	62.71	61.02	62.71	61.02	60.17	2.76
EmbraceBERT-bs4+Dropout0.1	66.10	57.63	64.41	61.02	66.10	59.32	55.93	57.63	57.63	64.41	61.02	3.71
EmbraceBERT-bs4+Dropout0.3	62.71	59.32	55.93	61.02	66.10	57.63	57.63	66.10	57.63	64.41	60.85	3.59
CondensedEmbraceBERT-bs4	6.78	16.95	18.64	3.39	28.81	13.56	8.47	11.86	18.64	3.39	13.05	7.58
CondensedEmbraceBERT-bs4+Dropout0.1	57.63	50.85	61.02	62.71	57.63	59.32	61.02	61.02	61.02	61.02	59.32	3.22
CondensedEmbraceBERT-bs4+Dropout0.3	55.93	59.32	62.71	64.41	64.41	59.32	55.93	62.71	57.63	64.41	60.68	3.29
FrozenEBERT-bs4-ep100	55.93	59.32	66.10	62.71	61.02	59.32	61.02	55.93	66.10	59.32	60.68	3.37
FrozenEBERT-bs4-ep100+Dropout0.1	61.02	59.32	62.71	59.32	62.71	62.71	61.02	57.63	57.63	64.41	60.85	2.20
FrozenEBERT-bs4-ep100+Dropout0.3	62.71	62.71	55.93	62.71	61.02	55.93	57.63	67.80	57.63	64.41	60.85	3.75
FrozenCEBERT-bs4-ep100	10.17	16.95	18.64	1.69	27.12	11.86	11.86	11.86	13.56	3.39	12.71	6.92
FrozenCEBERT-bs4-ep100+Dropout0.1	59.32	52.54	59.32	67.80	57.63	57.63	57.63	62.71	55.93	61.02	59.15	3.90
FrozenCEBERT-bs4-ep100+Dropout0.3	50.85	61.02	61.02	64.41	67.80	61.02	54.24	59.32	57.63	62.71	60.00	4.62
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs4	55.93	57.63	50.85	61.02	59.32	57.63	54.24	62.71	54.24	61.02	57.46	3.51
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4+Branches	61.02	55.93	57.63	59.32	59.32	57.63	62.71	57.63	57.63	61.02	58.98	1.98
EmbraceBERT-bs4+Branches+Dropout0.1	64.41	62.71	54.24	61.02	57.63	66.10	57.63	59.32	61.02	57.63	60.17	3.41
EmbraceBERT-bs4+Branches+Dropout0.3	66.10	61.02	61.02	61.02	66.10	59.32	57.63	61.02	59.32	54.24	60.68	3.37
CondensedEmbraceBERT-bs4+Branches	61.02	52.54	59.32	54.24	59.32	61.02	59.32	55.93	57.63	61.02	58.14	2.84
CondensedEmbraceBERT-bs4+Branches+Dropout0.1	57.63	59.32	55.93	55.93	57.63	59.32	64.41	62.71	61.02	59.32	59.32	2.63
CondensedEmbraceBERT-bs4+Branches+Dropout0.3	61.02	64.41	59.32	57.63	57.63	61.02	61.02	61.02	64.41	54.24	60.17	2.96
FrozenEmbraceBERT-bs4+Branches	59.32	57.63	59.32	61.02	61.02	57.63	62.71	59.32	59.32	57.63	59.49	1.60
FrozenEmbraceBERT-bs4+Branches+Dropout0.1	62.71	59.32	55.93	57.63	59.32	66.10	57.63	57.63	62.71	55.93	59.49	3.17
FrozenEmbraceBERT-bs4+Branches+Dropout0.3	62.71	61.02	61.02	62.71	64.41	59.32	62.71	62.71	61.02	57.63	61.53	1.86
FrozenCEBERT-bs4+Branches	61.02	59.32	59.32	55.93	61.02	62.71	59.32	52.54	64.41	54.24	58.98	3.54
FrozenCEBERT-bs4+Branches+Dropout0.1	61.02	59.32	54.24	57.63	59.32	59.32	64.41	59.32	61.02	61.02	59.66	2.49
FrozenCEBERT-bs4+Branches+Dropout0.3	57.63	61.02	57.63	55.93	62.71	61.02	62.71	57.63	64.41	54.24	59.49	3.17
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	62.71	61.02	55.93	55.93	57.63	62.71	55.93	54.24	54.24	59.32	57.97	3.11
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw	59.32	64.41	55.93	55.93	59.32	62.71	64.41	59.32	59.32	59.32	60.00	2.86
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	64.41	61.02	59.32	61.02	55.93	61.02	69.49	55.93	59.32	61.02	60.85	3.75
BERT-bs16+Dropout0.1	64.41	61.02	59.32	55.93	57.63	62.71	66.10	59.32	59.32	57.63	60.34	3.05
BERT-bs16+Dropout0.3	62.71	59.32	59.32	55.93	59.32	64.41	64.41	57.63	57.63	54.24	59.49	3.26
FrozenBERT-bs16-ep100	62.71	59.32	59.32	62.71	59.32	62.71	71.19	55.93	61.02	61.02	61.53	3.79
FrozenBERT-bs16-ep100+Dropout0.1	64.41	62.71	61.02	61.02	59.32	64.41	66.10	57.63	62.71	57.63	61.69	2.75
FrozenBERT-bs16-ep100+Dropout0.3	59.32	66.10	57.63	59.32	61.02	64.41	66.10	55.93	61.02	57.63	60.85	3.43
EmbraceBERT-bs16	57.63	50.85	59.32	55.93	66.10	59.32	64.41	54.24	57.63	61.02	58.64	4.30
EmbraceBERT-bs16+Dropout0.1	61.02	50.85	55.93	62.71	62.71	64.41	66.10	54.24	54.24	61.02	59.32	4.85
EmbraceBERT-bs16+Dropout0.3	61.02	49.15	62.71	64.41	61.02	62.71	62.71	61.02	61.02	59.32	60.51	4.01
CondensedEmbraceBERT-bs16	6.78	25.42	15.25	0.00	8.47	5.08	15.25	10.17	13.56	3.39	10.34	6.97
CondensedEmbraceBERT-bs16+Dropout0.1	62.71	47.46	59.32	59.32	61.02	64.41	62.71	61.02	57.63	59.32	59.49	4.45
CondensedEmbraceBERT-bs16+Dropout0.3	62.71	49.15	59.32	57.63	64.41	61.02	57.63	66.10	57.63	57.63	59.32	4.48
FrozenEBERT-bs16-ep100	59.32	50.85	59.32	55.93	66.10	59.32	59.32	52.54	59.32	57.63	57.97	4.00
FrozenEBERT-bs16-ep100+Dropout0.1	57.63	50.85	57.63	62.71	62.71	57.63	62.71	54.24	57.63	57.63	58.14	3.64
FrozenEBERT-bs16-ep100+Dropout0.3	57.63	54.24	61.02	62.71	62.71	61.02	61.02	54.24	61.02	61.02	59.66	3.01
FrozenCEBERT-bs16-ep100	6.78	27.12	15.25	0.00	8.47	6.78	13.56	13.56	11.86	3.39	10.68	7.15
FrozenCEBERT-bs16-ep100+Dropout0.1	61.02	49.15	61.02	62.71	52.54	57.63	62.71	55.93	55.93	55.93	57.46	4.25
FrozenCEBERT-bs16-ep100+Dropout0.3	62.71	54.24	61.02	57.63	64.41	59.32	61.02	61.02	54.24	62.71	59.83	3.31
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	57.63	50.85	57.63	59.32	64.41	59.32	62.71	57.63	59.32	55.93	58.47	3.49
EmbraceBERT-bs16+Branches+Dropout0.1	61.02	62.71	62.71	55.93	61.02	55.93	64.41	54.24	66.10	59.32	60.34	3.73
EmbraceBERT-bs16+Branches+Dropout0.3	59.32	62.71	61.02	55.93	57.63	55.93	64.41	59.32	67.80	59.32	60.34	3.57
CondensedEmbraceBERT-bs16+Branches	57.63	50.85	59.32	59.32	59.32	57.63	59.32	57.63	54.24	55.93	57.12	2.63
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	55.93	57.63	59.32	59.32	59.32	57.63	62.71	55.93	61.02	57.63	58.64	2.03
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	47.46	64.41	57.63	61.02	59.32	54.24	66.10	55.93	66.10	54.24	58.64	5.68
FrozenEmbraceBERT-bs16+Branches	59.32	52.54	61.02	62.71	62.71	61.02	61.02	55.93	64.41	62.71	60.34	3.41
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	59.32	61.02	55.93	57.63	57.63	59.32	59.32	57.63	66.10	57.63	59.15	2.67
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	59.32	64.41	59.32	55.93	59.32	57.63	59.32	55.93	67.80	59.32	59.83	3.48
FrozenCEBERT-bs16+Branches	62.71	59.32	55.93	57.63	55.93	55.93	59.32	59.32	64.41	50.85	58.14	3.64
FrozenCEBERT-bs16+Branches+Dropout0.1	59.32	59.32	59.32	55.93	62.71	59.32	61.02	57.63	61.02	57.63	59.32	1.86
FrozenCEBERT-bs16+Branches+Dropout0.3	57.63	62.71	57.63	54.24	62.71	57.63	62.71	59.32	66.10	59.32	60.00	3.32
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	62.71	59.32	55.93	55.93	61.02	59.32	55.93	55.93	52.54	59.32	57.80	2.88
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	76.27	76.27	72.88	62.71	76.27	76.27	76.27	67.80	72.88	74.58	73.22	4.34
RoBERTa-bs4+Dropout0.1	77.97	74.58	74.58	62.71	74.58	71.19	69.49	71.19	69.49	74.58	72.03	4.03
RoBERTa-bs4+Dropout0.3	76.27	74.58	72.88	71.19	76.27	69.49	71.19	71.19	76.27	71.19	73.05	2.45
FrozenRoBERTa-bs4-ep100	77.97	72.88	74.58	64.41	71.19	74.58	72.88	67.80	69.49	72.88	71.86	3.65
FrozenRoBERTa-bs4-ep100+Dropout0.1	74.58	71.19	72.88	61.02	74.58	74.58	69.49	69.49	69.49	74.58	71.19	4.01
FrozenRoBERTa-bs4-ep100+Dropout0.3	71.19	69.49	71.19	67.80	76.27	74.58	66.10	67.80	71.19	67.80	70.34	3.06
EmbraceRoBERTa-bs4	71.19	74.58	72.88	74.58	71.19	61.02	72.88	66.10	72.88	72.88	71.02	4.04
EmbraceRoBERTa-bs4+Dropout0.1	72.88	76.27	69.49	71.19	72.88	72.88	69.49	62.71	67.80	72.88	70.85	3.54
EmbraceRoBERTa-bs4+Dropout0.3	71.19	72.88	71.19	74.58	74.58	71.19	67.80	72.88	72.88	76.27	72.54	2.25
CondensedEmbraceRoBERTa-bs4	27.12	33.90	33.90	10.17	22.03	13.56	25.42	15.25	1.69	3.39	18.64	11.06
CondensedEmbraceRoBERTa-bs4+Dropout0.1	69.49	74.58	77.97	74.58	72.88	72.88	72.88	71.19	74.58	67.80	72.88	2.73
CondensedEmbraceRoBERTa-bs4+Dropout0.3	61.02	76.27	72.88	74.58	74.58	72.88	69.49	74.58	71.19	74.58	72.20	4.17
FrozenERoBERTa-bs4-ep100	72.88	74.58	67.80	74.58	69.49	69.49	69.49	69.49	72.88	72.88	71.36	2.33
FrozenERoBERTa-bs4-ep100+Dropout0.1	71.19	72.88	67.80	74.58	67.80	72.88	67.80	69.49	74.58	72.88	71.19	2.63
FrozenERoBERTa-bs4-ep100+Dropout0.3	66.10	72.88	71.19	71.19	69.49	72.88	66.10	67.80	74.58	71.19	70.34	2.76
FrozenCERoBERTa-bs4-ep100	20.34	33.90	37.29	11.86	23.73	13.56	25.42	15.25	3.39	5.08	18.98	10.74
FrozenCERoBERTa-bs4-ep100+Dropout0.1	74.58	74.58	69.49	69.49	69.49	76.27	71.19	67.80	72.88	72.88	71.86	2.65
FrozenCERoBERTa-bs4-ep100+Dropout0.3	64.41	74.58	72.88	71.19	64.41	74.58	69.49	72.88	71.19	74.58	71.02	3.67
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs4+Branches	69.49	72.88	76.27	74.58	72.88	74.58	69.49	66.10	71.19	74.58	72.20	2.96
EmbraceRoBERTa-bs4+Branches+Dropout0.1	71.19	72.88	74.58	74.58	74.58	71.19	74.58	72.88	74.58	72.88	73.39	1.32
EmbraceRoBERTa-bs4+Branches+Dropout0.3	71.19	71.19	74.58	72.88	72.88	76.27	72.88	74.58	74.58	77.97	73.90	2.03
CondensedEmbraceRoBERTa-bs4+Branches	69.49	76.27	74.58	76.27	72.88	74.58	71.19	72.88	72.88	74.58	73.56	2.03
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1	67.80	74.58	71.19	74.58	71.19	72.88	72.88	74.58	77.97	76.27	73.39	2.74
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3	69.49	72.88	76.27	74.58	72.88	74.58	69.49	71.19	76.27	74.58	73.22	2.37
FrozenEmbraceRoBERTa-bs4+Branches	66.10	71.19	72.88	72.88	71.19	74.58	69.49	67.80	71.19	72.88	71.02	2.45
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1	71.19	72.88	76.27	74.58	72.88	71.19	74.58	67.80	74.58	71.19	72.71	2.33
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3	71.19	66.10	71.19	72.88	69.49	74.58	74.58	69.49	74.58	74.58	71.86	2.75
FrozenCERoBERTa-bs4+Branches	69.49	74.58	74.58	76.27	74.58	69.49	67.80	69.49	72.88	74.58	72.37	2.84
FrozenCERoBERTa-bs4+Branches+Dropout0.1	66.10	74.58	76.27	74.58	74.58	69.49	69.49	74.58	71.19	74.58	72.54	3.11
FrozenCERoBERTa-bs4+Branches+Dropout0.3	67.80	71.19	77.97	72.88	74.58	67.80	67.80	67.80	76.27	74.58	71.86	3.73
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	74.58	69.49	72.88	69.49	72.88	72.88	74.58	69.49	72.88	69.49	71.86	2.03
RoBERTa-bs16+Dropout0.1	74.58	71.19	72.88	72.88	66.10	76.27	69.49	72.88	71.19	69.49	71.69	2.74
RoBERTa-bs16+Dropout0.3	76.27	72.88	72.88	74.58	76.27	69.49	69.49	71.19	69.49	67.80	72.03	2.86
FrozenRoBERTa-bs16-ep100	74.58	67.80	72.88	67.80	74.58	71.19	71.19	66.10	72.88	71.19	71.02	2.78
FrozenRoBERTa-bs16-ep100+Dropout0.1	71.19	71.19	76.27	64.41	74.58	71.19	71.19	67.80	71.19	71.19	71.02	3.07
FrozenRoBERTa-bs16-ep100+Dropout0.3	72.88	69.49	72.88	61.02	74.58	72.88	74.58	67.80	67.80	71.19	70.51	3.95
EmbraceRoBERTa-bs16	72.88	74.58	69.49	76.27	72.88	74.58	74.58	69.49	76.27	76.27	73.73	2.43
EmbraceRoBERTa-bs16+Dropout0.1	74.58	72.88	76.27	74.58	72.88	72.88	72.88	72.88	74.58	74.58	73.90	1.12
EmbraceRoBERTa-bs16+Dropout0.3	74.58	71.19	69.49	76.27	74.58	71.19	74.58	71.19	76.27	76.27	73.56	2.42
CondensedEmbraceRoBERTa-bs16	23.73	35.59	30.51	16.95	15.25	13.56	32.20	16.95	5.08	22.03	21.19	9.01
CondensedEmbraceRoBERTa-bs16+Dropout0.1	76.27	72.88	71.19	72.88	72.88	74.58	74.58	66.10	76.27	74.58	73.22	2.82
CondensedEmbraceRoBERTa-bs16+Dropout0.3	76.27	69.49	74.58	76.27	72.88	74.58	72.88	64.41	76.27	76.27	73.39	3.64
FrozenERoBERTa-bs16-ep100	72.88	72.88	67.80	76.27	71.19	71.19	71.19	66.10	76.27	76.27	72.20	3.32
FrozenERoBERTa-bs16-ep100+Dropout0.1	72.88	74.58	72.88	74.58	72.88	74.58	71.19	69.49	74.58	74.58	73.22	1.66
FrozenERoBERTa-bs16-ep100+Dropout0.3	72.88	76.27	77.97	76.27	72.88	72.88	71.19	76.27	71.19	74.58	74.24	2.25
FrozenCERoBERTa-bs16-ep100	23.73	40.68	32.20	13.56	11.86	11.86	32.20	22.03	3.39	23.73	21.53	10.88
FrozenCERoBERTa-bs16-ep100+Dropout0.1	74.58	74.58	69.49	76.27	72.88	74.58	76.27	69.49	72.88	74.58	73.56	2.30
FrozenCERoBERTa-bs16-ep100+Dropout0.3	71.19	76.27	71.19	76.27	71.19	72.88	71.19	72.88	74.58	76.27	73.39	2.15
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	74.58	67.80	72.88	72.88	66.10	69.49	72.88	66.10	71.19	74.58	70.85	3.11
EmbraceRoBERTa-bs16+Branches+Dropout0.1	76.27	77.97	74.58	74.58	72.88	76.27	76.27	69.49	74.58	76.27	74.92	2.25
EmbraceRoBERTa-bs16+Branches+Dropout0.3	74.58	77.97	72.88	74.58	72.88	74.58	76.27	69.49	74.58	74.58	74.24	2.12
CondensedEmbraceRoBERTa-bs16+Branches	71.19	67.80	64.41	72.88	67.80	69.49	69.49	66.10	71.19	76.27	69.66	3.26
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	76.27	71.19	72.88	77.97	72.88	74.58	71.19	72.88	74.58	76.27	74.07	2.15
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	77.97	72.88	71.19	71.19	71.19	72.88	72.88	72.88	74.58	74.58	73.22	1.98
FrozenEmbraceRoBERTa-bs16+Branches	72.88	69.49	69.49	72.88	74.58	74.58	72.88	69.49	74.58	74.58	72.54	2.12
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	72.88	76.27	72.88	69.49	71.19	76.27	72.88	64.41	72.88	74.58	72.37	3.31
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	72.88	77.97	71.19	69.49	71.19	76.27	72.88	66.10	74.58	72.88	72.54	3.20
FrozenCERoBERTa-bs16+Branches	74.58	79.66	74.58	72.88	71.19	74.58	72.88	72.88	74.58	74.58	74.24	2.12
FrozenCERoBERTa-bs16+Branches+Dropout0.1	71.19	74.58	72.88	74.58	71.19	74.58	69.49	67.80	74.58	74.58	72.54	2.37
FrozenCERoBERTa-bs16+Branches+Dropout0.3	72.88	76.27	69.49	69.49	71.19	74.58	71.19	69.49	74.58	74.58	72.37	2.40

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |64.41|61.02|61.02|61.02|54.24|59.32|67.80|57.63|61.02|61.02|60.85|3.43| | BERTwithAtt-bs8 |61.02|55.93|57.63|57.63|61.02|57.63|55.93|57.63|50.85|62.71|57.80|3.17| | BERTwithAttClsProjection-bs8 |59.32|55.93|62.71|61.02|61.02|66.10|61.02|55.93|61.02|66.10|61.02|3.30| | BERTwithProjection-bs8 |55.93|59.32|66.10|61.02|57.63|59.32|45.76|61.02|50.85|64.41|58.14|5.78| | BERTwithProjectionAtt-bs8 |61.02|62.71|59.32|61.02|61.02|62.71|62.71|61.02|64.41|61.02|61.69|1.36| | EmbraceBERT-bs8 |61.02|54.24|61.02|59.32|61.02|57.63|55.93|61.02|55.93|57.63|58.47|2.43| | EmbraceBERT-bs8-p_att_clsquery_weights |64.41|57.63|52.54|59.32|61.02|61.02|54.24|55.93|62.71|61.02|58.98|3.62| | EmbraceBERTwithProj-bs8 |61.02|62.71|64.41|71.19|64.41|62.71|64.41|62.71|59.32|59.32|63.22|3.22| | EmbraceBERTwithProj-bs8-p_attclsqw |54.24|59.32|62.71|64.41|64.41|66.10|61.02|61.02|59.32|61.02|61.36|3.20| | EBERTconcatAtt-bs8 |57.63|59.32|52.54|57.63|55.93|57.63|57.63|54.24|59.32|62.71|57.46|2.67| | EBERTconcatAtt-bs8_attclsqw |59.32|54.24|54.24|50.85|62.71|55.93|57.63|57.63|57.63|59.32|56.95|3.14| | EBERTconcatAttwithProj-bs8 |61.02|54.24|62.71|55.93|57.63|61.02|55.93|64.41|61.02|59.32|59.32|3.13| | EBERTconcatAttwithProj-bs8_attclsqw |52.54|59.32|64.41|59.32|55.93|57.63|62.71|57.63|62.71|57.63|58.98|3.37| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |66.10|57.63|50.85|52.54|57.63|57.63|61.02|61.02|55.93|59.32|57.97|4.14| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |57.63|64.41|59.32|57.63|57.63|61.02|62.71|61.02|62.71|62.71|60.68|2.37| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |59.32|57.63|62.71|59.32|61.02|62.71|62.71|57.63|61.02|62.71|60.68|1.98| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |61.02|64.41|55.93|55.93|62.71|57.63|59.32|66.10|66.10|61.02|61.02|3.64| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |57.63|64.41|59.32|61.02|62.71|59.32|52.54|64.41|61.02|61.02|60.34|3.32| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |57.63|54.24|55.93|59.32|62.71|55.93|59.32|59.32|52.54|66.10|58.31|3.81| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |61.02|54.24|64.41|55.93|61.02|62.71|62.71|57.63|59.32|61.02|60.00|3.05| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |61.02|59.32|66.10|55.93|62.71|61.02|54.24|57.63|62.71|61.02|60.17|3.33|

gtts-sphinx - good RoBERTa all

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	57.63	45.76	55.93	52.54	47.46	59.32	50.85	52.54	57.63	50.85	53.05	4.29
BERT-bs4+Dropout0.1	62.71	50.85	59.32	50.85	49.15	55.93	52.54	49.15	55.93	47.46	53.39	4.69
BERT-bs4+Dropout0.3	59.32	49.15	55.93	49.15	47.46	54.24	55.93	50.85	50.85	54.24	52.71	3.59
FrozenBERT-bs4-ep100	59.32	47.46	52.54	52.54	45.76	57.63	50.85	49.15	59.32	52.54	52.71	4.51
FrozenBERT-bs4-ep100+Dropout0.1	59.32	47.46	54.24	50.85	54.24	57.63	54.24	55.93	59.32	47.46	54.07	4.11
FrozenBERT-bs4-ep100+Dropout0.3	59.32	47.46	57.63	47.46	52.54	57.63	52.54	55.93	59.32	52.54	54.24	4.22
EmbraceBERT-bs4	55.93	47.46	54.24	49.15	49.15	49.15	52.54	50.85	57.63	49.15	51.53	3.23
EmbraceBERT-bs4+Dropout0.1	44.07	55.93	47.46	47.46	59.32	55.93	52.54	49.15	54.24	44.07	51.02	5.05
EmbraceBERT-bs4+Dropout0.3	54.24	50.85	50.85	52.54	61.02	50.85	55.93	52.54	52.54	50.85	53.22	3.05
CondensedEmbraceBERT-bs4	8.47	15.25	18.64	11.86	25.42	10.17	8.47	1.69	20.34	3.39	12.37	7.15
CondensedEmbraceBERT-bs4+Dropout0.1	42.37	47.46	49.15	52.54	55.93	49.15	52.54	47.46	57.63	47.46	50.17	4.30
CondensedEmbraceBERT-bs4+Dropout0.3	42.37	50.85	47.46	52.54	54.24	52.54	45.76	50.85	52.54	52.54	50.17	3.57
FrozenEBERT-bs4-ep100	54.24	52.54	55.93	47.46	49.15	50.85	61.02	49.15	52.54	50.85	52.37	3.75
FrozenEBERT-bs4-ep100+Dropout0.1	44.07	47.46	49.15	47.46	50.85	59.32	49.15	50.85	55.93	44.07	49.83	4.56
FrozenEBERT-bs4-ep100+Dropout0.3	50.85	50.85	50.85	50.85	52.54	50.85	55.93	59.32	54.24	47.46	52.37	3.17
FrozenCEBERT-bs4-ep100	8.47	18.64	16.95	15.25	18.64	13.56	15.25	0.00	20.34	5.08	13.22	6.29
FrozenCEBERT-bs4-ep100+Dropout0.1	47.46	44.07	45.76	52.54	54.24	47.46	55.93	49.15	54.24	45.76	49.66	4.01
FrozenCEBERT-bs4-ep100+Dropout0.3	42.37	55.93	47.46	49.15	52.54	54.24	52.54	52.54	54.24	50.85	51.19	3.77
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs4	55.93	54.24	45.76	50.85	49.15	50.85	47.46	44.07	42.37	59.32	50.00	5.10
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4+Branches	44.07	49.15	49.15	49.15	45.76	50.85	50.85	49.15	40.68	45.76	47.46	3.13
EmbraceBERT-bs4+Branches+Dropout0.1	57.63	54.24	49.15	38.98	54.24	50.85	49.15	42.37	55.93	49.15	50.17	5.58
EmbraceBERT-bs4+Branches+Dropout0.3	54.24	54.24	45.76	47.46	52.54	55.93	49.15	52.54	57.63	49.15	51.86	3.65
CondensedEmbraceBERT-bs4+Branches	55.93	54.24	45.76	47.46	54.24	52.54	52.54	47.46	45.76	45.76	50.17	3.88
CondensedEmbraceBERT-bs4+Branches+Dropout0.1	57.63	52.54	47.46	44.07	54.24	45.76	49.15	55.93	54.24	49.15	51.02	4.32
CondensedEmbraceBERT-bs4+Branches+Dropout0.3	62.71	47.46	40.68	44.07	50.85	44.07	44.07	52.54	62.71	44.07	49.32	7.48
FrozenEmbraceBERT-bs4+Branches	47.46	50.85	49.15	45.76	52.54	49.15	47.46	52.54	49.15	44.07	48.81	2.60
FrozenEmbraceBERT-bs4+Branches+Dropout0.1	57.63	55.93	50.85	42.37	54.24	49.15	52.54	47.46	55.93	45.76	51.19	4.72
FrozenEmbraceBERT-bs4+Branches+Dropout0.3	54.24	49.15	49.15	49.15	52.54	38.98	52.54	54.24	54.24	45.76	50.00	4.56
FrozenCEBERT-bs4+Branches	54.24	49.15	55.93	44.07	55.93	52.54	55.93	42.37	61.02	49.15	52.03	5.52
FrozenCEBERT-bs4+Branches+Dropout0.1	54.24	50.85	47.46	45.76	47.46	45.76	49.15	45.76	54.24	52.54	49.32	3.26
FrozenCEBERT-bs4+Branches+Dropout0.3	59.32	50.85	38.98	45.76	45.76	42.37	47.46	47.46	62.71	47.46	48.81	6.86
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	45.76	50.85	54.24	42.37	54.24	42.37	38.98	54.24	42.37	50.85	47.63	5.59
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw	52.54	52.54	44.07	49.15	45.76	50.85	55.93	45.76	52.54	55.93	50.51	4.00
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	55.93	50.85	57.63	49.15	45.76	50.85	55.93	50.85	50.85	52.54	52.03	3.39
BERT-bs16+Dropout0.1	55.93	47.46	52.54	49.15	45.76	59.32	50.85	52.54	49.15	50.85	51.36	3.79
BERT-bs16+Dropout0.3	61.02	49.15	54.24	49.15	45.76	54.24	54.24	54.24	49.15	54.24	52.54	4.08
FrozenBERT-bs16-ep100	57.63	50.85	59.32	52.54	45.76	57.63	57.63	49.15	52.54	52.54	53.56	4.17
FrozenBERT-bs16-ep100+Dropout0.1	57.63	50.85	50.85	52.54	47.46	59.32	55.93	55.93	55.93	49.15	53.56	3.73
FrozenBERT-bs16-ep100+Dropout0.3	57.63	49.15	54.24	49.15	49.15	59.32	52.54	52.54	50.85	49.15	52.37	3.51
EmbraceBERT-bs16	50.85	38.98	42.37	54.24	54.24	50.85	52.54	47.46	54.24	52.54	49.83	5.04
EmbraceBERT-bs16+Dropout0.1	52.54	45.76	47.46	45.76	52.54	52.54	57.63	55.93	47.46	49.15	50.68	3.97
EmbraceBERT-bs16+Dropout0.3	50.85	47.46	55.93	50.85	59.32	50.85	47.46	47.46	49.15	47.46	50.68	3.82
CondensedEmbraceBERT-bs16	3.39	20.34	16.95	3.39	16.95	11.86	18.64	13.56	13.56	5.08	12.37	6.02
CondensedEmbraceBERT-bs16+Dropout0.1	50.85	40.68	47.46	47.46	50.85	52.54	52.54	50.85	50.85	49.15	49.32	3.34
CondensedEmbraceBERT-bs16+Dropout0.3	50.85	47.46	57.63	49.15	55.93	52.54	49.15	50.85	47.46	49.15	51.02	3.26
FrozenEBERT-bs16-ep100	47.46	44.07	44.07	45.76	52.54	52.54	45.76	47.46	52.54	49.15	48.14	3.23
FrozenEBERT-bs16-ep100+Dropout0.1	40.68	45.76	47.46	47.46	50.85	52.54	49.15	54.24	49.15	50.85	48.81	3.62
FrozenEBERT-bs16-ep100+Dropout0.3	50.85	47.46	50.85	47.46	55.93	54.24	49.15	49.15	50.85	50.85	50.68	2.56
FrozenCEBERT-bs16-ep100	3.39	20.34	16.95	3.39	15.25	13.56	16.95	16.95	15.25	5.08	12.71	5.98
FrozenCEBERT-bs16-ep100+Dropout0.1	54.24	44.07	47.46	47.46	50.85	54.24	52.54	45.76	45.76	47.46	48.98	3.51
FrozenCEBERT-bs16-ep100+Dropout0.3	54.24	50.85	50.85	42.37	61.02	59.32	49.15	54.24	47.46	45.76	51.53	5.53
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	47.46	44.07	52.54	49.15	55.93	49.15	47.46	47.46	44.07	49.15	48.64	3.39
EmbraceBERT-bs16+Branches+Dropout0.1	47.46	49.15	47.46	49.15	57.63	49.15	54.24	50.85	57.63	50.85	51.36	3.64
EmbraceBERT-bs16+Branches+Dropout0.3	50.85	54.24	45.76	45.76	50.85	54.24	55.93	50.85	52.54	54.24	51.53	3.32
CondensedEmbraceBERT-bs16+Branches	44.07	44.07	54.24	47.46	52.54	50.85	47.46	52.54	45.76	50.85	48.98	3.51
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	54.24	50.85	49.15	52.54	59.32	50.85	45.76	49.15	54.24	49.15	51.53	3.57
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	49.15	54.24	45.76	52.54	49.15	50.85	37.29	49.15	52.54	54.24	49.49	4.78
FrozenEmbraceBERT-bs16+Branches	44.07	47.46	50.85	47.46	50.85	54.24	52.54	49.15	55.93	47.46	50.00	3.41
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	52.54	52.54	45.76	49.15	54.24	44.07	57.63	44.07	59.32	49.15	50.85	5.08
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	57.63	54.24	44.07	45.76	52.54	54.24	59.32	44.07	50.85	55.93	51.86	5.26
FrozenCEBERT-bs16+Branches	50.85	52.54	49.15	49.15	52.54	42.37	49.15	55.93	54.24	52.54	50.85	3.56
FrozenCEBERT-bs16+Branches+Dropout0.1	54.24	47.46	50.85	47.46	55.93	49.15	42.37	47.46	57.63	52.54	50.51	4.41
FrozenCEBERT-bs16+Branches+Dropout0.3	52.54	52.54	47.46	44.07	50.85	49.15	49.15	45.76	52.54	44.07	48.81	3.20
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	54.24	44.07	52.54	45.76	55.93	50.85	45.76	52.54	44.07	47.46	49.32	4.18
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	67.80	67.80	67.80	59.32	76.27	67.80	64.41	71.19	66.10	59.32	66.78	4.81
RoBERTa-bs4+Dropout0.1	66.10	66.10	72.88	64.41	77.97	61.02	67.80	67.80	67.80	66.10	67.80	4.42
RoBERTa-bs4+Dropout0.3	71.19	71.19	72.88	66.10	71.19	67.80	69.49	71.19	64.41	67.80	69.32	2.56
FrozenRoBERTa-bs4-ep100	64.41	72.88	67.80	59.32	77.97	67.80	62.71	69.49	66.10	61.02	66.95	5.32
FrozenRoBERTa-bs4-ep100+Dropout0.1	71.19	67.80	74.58	55.93	76.27	57.63	64.41	67.80	64.41	64.41	66.44	6.24
FrozenRoBERTa-bs4-ep100+Dropout0.3	69.49	64.41	71.19	57.63	74.58	62.71	61.02	74.58	62.71	64.41	66.27	5.54
EmbraceRoBERTa-bs4	64.41	67.80	64.41	69.49	74.58	72.88	69.49	62.71	71.19	71.19	68.81	3.73
EmbraceRoBERTa-bs4+Dropout0.1	67.80	67.80	66.10	69.49	67.80	74.58	72.88	66.10	71.19	74.58	69.83	3.11
EmbraceRoBERTa-bs4+Dropout0.3	64.41	66.10	71.19	74.58	67.80	76.27	69.49	67.80	69.49	64.41	69.15	3.77
CondensedEmbraceRoBERTa-bs4	15.25	30.51	28.81	8.47	23.73	16.95	23.73	18.64	3.39	6.78	17.63	8.81
CondensedEmbraceRoBERTa-bs4+Dropout0.1	67.80	71.19	69.49	66.10	69.49	76.27	74.58	76.27	71.19	71.19	71.36	3.26
CondensedEmbraceRoBERTa-bs4+Dropout0.3	71.19	66.10	66.10	71.19	71.19	74.58	76.27	67.80	66.10	69.49	70.00	3.39
FrozenERoBERTa-bs4-ep100	64.41	71.19	61.02	69.49	69.49	69.49	67.80	61.02	72.88	69.49	67.63	3.90
FrozenERoBERTa-bs4-ep100+Dropout0.1	69.49	67.80	69.49	66.10	76.27	72.88	66.10	66.10	72.88	67.80	69.49	3.30
FrozenERoBERTa-bs4-ep100+Dropout0.3	69.49	66.10	67.80	67.80	74.58	74.58	67.80	69.49	72.88	66.10	69.66	3.07
FrozenCERoBERTa-bs4-ep100	8.47	32.20	33.90	10.17	20.34	13.56	20.34	23.73	6.78	5.08	17.46	9.80
FrozenCERoBERTa-bs4-ep100+Dropout0.1	67.80	72.88	66.10	67.80	76.27	74.58	74.58	64.41	72.88	71.19	70.85	3.85
FrozenCERoBERTa-bs4-ep100+Dropout0.3	71.19	66.10	62.71	66.10	71.19	76.27	69.49	71.19	71.19	66.10	69.15	3.70
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs4+Branches	67.80	71.19	74.58	74.58	72.88	74.58	72.88	64.41	69.49	72.88	71.53	3.20
EmbraceRoBERTa-bs4+Branches+Dropout0.1	71.19	69.49	72.88	69.49	72.88	74.58	69.49	74.58	71.19	72.88	71.86	1.89
EmbraceRoBERTa-bs4+Branches+Dropout0.3	69.49	71.19	72.88	71.19	72.88	69.49	77.97	74.58	77.97	72.88	73.05	2.88
CondensedEmbraceRoBERTa-bs4+Branches	72.88	71.19	77.97	67.80	71.19	74.58	76.27	66.10	69.49	66.10	71.36	3.90
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1	69.49	71.19	62.71	67.80	76.27	67.80	69.49	69.49	72.88	69.49	69.66	3.34
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3	71.19	71.19	71.19	71.19	74.58	76.27	66.10	67.80	72.88	72.88	71.53	2.82
FrozenEmbraceRoBERTa-bs4+Branches	66.10	69.49	74.58	76.27	76.27	71.19	72.88	71.19	67.80	71.19	71.69	3.22
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1	72.88	67.80	72.88	67.80	74.58	76.27	71.19	74.58	71.19	69.49	71.86	2.75
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3	69.49	66.10	72.88	72.88	77.97	72.88	77.97	76.27	72.88	69.49	72.88	3.64
FrozenCERoBERTa-bs4+Branches	69.49	71.19	74.58	71.19	72.88	71.19	71.19	71.19	71.19	66.10	71.02	2.07
FrozenCERoBERTa-bs4+Branches+Dropout0.1	69.49	64.41	59.32	69.49	79.66	69.49	69.49	72.88	71.19	74.58	70.00	5.20
FrozenCERoBERTa-bs4+Branches+Dropout0.3	67.80	67.80	72.88	64.41	74.58	67.80	66.10	69.49	72.88	74.58	69.83	3.46
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	64.41	55.93	71.19	57.63	76.27	64.41	69.49	72.88	69.49	69.49	67.12	6.17
RoBERTa-bs16+Dropout0.1	67.80	62.71	66.10	62.71	76.27	69.49	62.71	69.49	64.41	62.71	66.44	4.21
RoBERTa-bs16+Dropout0.3	64.41	62.71	69.49	66.10	71.19	66.10	62.71	72.88	66.10	62.71	66.44	3.46
FrozenRoBERTa-bs16-ep100	66.10	66.10	69.49	57.63	79.66	64.41	71.19	71.19	69.49	71.19	68.64	5.43
FrozenRoBERTa-bs16-ep100+Dropout0.1	67.80	66.10	64.41	57.63	67.80	66.10	69.49	69.49	62.71	64.41	65.59	3.39
FrozenRoBERTa-bs16-ep100+Dropout0.3	69.49	59.32	64.41	57.63	69.49	62.71	71.19	66.10	57.63	61.02	63.90	4.80
EmbraceRoBERTa-bs16	69.49	71.19	66.10	71.19	64.41	71.19	67.80	64.41	69.49	69.49	68.47	2.54
EmbraceRoBERTa-bs16+Dropout0.1	69.49	66.10	67.80	71.19	71.19	76.27	76.27	62.71	74.58	71.19	70.68	4.16
EmbraceRoBERTa-bs16+Dropout0.3	66.10	66.10	66.10	69.49	72.88	69.49	77.97	57.63	74.58	69.49	68.98	5.31
CondensedEmbraceRoBERTa-bs16	11.86	30.51	25.42	6.78	16.95	15.25	20.34	18.64	8.47	15.25	16.95	6.91
CondensedEmbraceRoBERTa-bs16+Dropout0.1	67.80	71.19	64.41	64.41	74.58	72.88	72.88	62.71	71.19	69.49	69.15	3.92
CondensedEmbraceRoBERTa-bs16+Dropout0.3	69.49	67.80	72.88	69.49	69.49	69.49	72.88	62.71	72.88	66.10	69.32	3.07
FrozenERoBERTa-bs16-ep100	67.80	71.19	64.41	72.88	69.49	74.58	71.19	67.80	72.88	71.19	70.34	2.86
FrozenERoBERTa-bs16-ep100+Dropout0.1	69.49	72.88	62.71	61.02	69.49	71.19	69.49	71.19	72.88	64.41	68.47	4.03
FrozenERoBERTa-bs16-ep100+Dropout0.3	66.10	74.58	74.58	64.41	67.80	72.88	71.19	69.49	74.58	67.80	70.34	3.58
FrozenCERoBERTa-bs16-ep100	10.17	30.51	22.03	5.08	15.25	18.64	27.12	16.95	8.47	15.25	16.95	7.62
FrozenCERoBERTa-bs16-ep100+Dropout0.1	71.19	69.49	62.71	67.80	74.58	74.58	69.49	69.49	74.58	69.49	70.34	3.49
FrozenCERoBERTa-bs16-ep100+Dropout0.3	74.58	69.49	62.71	67.80	57.63	72.88	71.19	69.49	71.19	67.80	68.47	4.75
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	69.49	67.80	64.41	57.63	61.02	69.49	67.80	64.41	62.71	67.80	65.25	3.73
EmbraceRoBERTa-bs16+Branches+Dropout0.1	69.49	72.88	64.41	64.41	71.19	72.88	74.58	59.32	69.49	71.19	68.98	4.55
EmbraceRoBERTa-bs16+Branches+Dropout0.3	66.10	74.58	67.80	59.32	71.19	71.19	76.27	61.02	71.19	72.88	69.15	5.30
CondensedEmbraceRoBERTa-bs16+Branches	66.10	67.80	57.63	59.32	64.41	66.10	62.71	66.10	61.02	64.41	63.56	3.15
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	74.58	71.19	67.80	66.10	71.19	71.19	71.19	62.71	74.58	71.19	70.17	3.49
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	67.80	71.19	66.10	61.02	74.58	72.88	67.80	69.49	62.71	71.19	68.47	4.10
FrozenEmbraceRoBERTa-bs16+Branches	71.19	69.49	67.80	69.49	72.88	72.88	72.88	72.88	71.19	74.58	71.53	1.98
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	66.10	71.19	66.10	69.49	74.58	72.88	74.58	67.80	69.49	71.19	70.34	2.96
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	66.10	72.88	64.41	67.80	69.49	71.19	71.19	67.80	71.19	72.88	69.49	2.73
FrozenCERoBERTa-bs16+Branches	72.88	72.88	66.10	72.88	72.88	74.58	67.80	74.58	74.58	71.19	72.03	2.76
FrozenCERoBERTa-bs16+Branches+Dropout0.1	67.80	74.58	62.71	71.19	71.19	76.27	69.49	66.10	72.88	72.88	70.51	3.88
FrozenCERoBERTa-bs16+Branches+Dropout0.3	64.41	74.58	67.80	66.10	76.27	72.88	67.80	71.19	67.80	72.88	70.17	3.73

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |55.93|47.46|57.63|49.15|47.46|54.24|61.02|55.93|54.24|47.46|53.05|4.61| | BERTwithAtt-bs8 |52.54|49.15|52.54|52.54|54.24|45.76|54.24|50.85|49.15|54.24|51.53|2.65| | BERTwithAttClsProjection-bs8 |55.93|47.46|52.54|50.85|44.07|49.15|54.24|57.63|49.15|52.54|51.36|3.87| | BERTwithProjection-bs8 |55.93|55.93|55.93|49.15|50.85|57.63|38.98|49.15|47.46|54.24|51.53|5.37| | BERTwithProjectionAtt-bs8 |62.71|57.63|50.85|50.85|57.63|54.24|55.93|52.54|52.54|50.85|54.58|3.70| | EmbraceBERT-bs8 |45.76|45.76|45.76|40.68|50.85|55.93|49.15|49.15|49.15|38.98|47.12|4.66| | EmbraceBERT-bs8-p_att_clsquery_weights |49.15|44.07|54.24|47.46|54.24|47.46|45.76|50.85|55.93|50.85|50.00|3.73| | EmbraceBERTwithProj-bs8 |52.54|61.02|55.93|54.24|52.54|49.15|47.46|57.63|54.24|57.63|54.24|3.87| | EmbraceBERTwithProj-bs8-p_attclsqw |47.46|45.76|50.85|54.24|50.85|45.76|50.85|55.93|47.46|52.54|50.17|3.32| | EBERTconcatAtt-bs8 |52.54|52.54|47.46|45.76|54.24|45.76|49.15|49.15|47.46|57.63|50.17|3.73| | EBERTconcatAtt-bs8_attclsqw |47.46|54.24|42.37|47.46|57.63|45.76|44.07|50.85|47.46|49.15|48.64|4.36| | EBERTconcatAttwithProj-bs8 |49.15|49.15|52.54|44.07|45.76|52.54|45.76|57.63|49.15|47.46|49.32|3.82| | EBERTconcatAttwithProj-bs8_attclsqw |55.93|49.15|55.93|55.93|47.46|47.46|49.15|55.93|49.15|50.85|51.69|3.58| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |54.24|50.85|49.15|45.76|47.46|45.76|49.15|54.24|44.07|52.54|49.32|3.43| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |47.46|57.63|50.85|54.24|47.46|64.41|47.46|50.85|55.93|50.85|52.71|5.17| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |47.46|52.54|50.85|49.15|57.63|42.37|49.15|49.15|45.76|47.46|49.15|3.87| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |55.93|52.54|49.15|54.24|49.15|49.15|49.15|52.54|52.54|52.54|51.69|2.31| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |45.76|55.93|42.37|55.93|50.85|52.54|55.93|50.85|52.54|54.24|51.69|4.30| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |50.85|52.54|54.24|49.15|49.15|55.93|52.54|47.46|47.46|55.93|51.53|3.05| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |55.93|52.54|47.46|50.85|55.93|47.46|52.54|54.24|49.15|50.85|51.69|2.96| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |50.85|55.93|54.24|47.46|52.54|49.15|49.15|47.46|54.24|54.24|51.53|2.96|

gtts-witai - good BERT bs4, RoBERTa all

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	76.27	69.49	69.49	71.19	66.10	71.19	69.49	66.10	71.19	69.49	70.00	2.74
BERT-bs4+Dropout0.1	72.88	67.80	67.80	71.19	67.80	71.19	72.88	69.49	69.49	67.80	69.83	1.98
BERT-bs4+Dropout0.3	74.58	69.49	67.80	71.19	69.49	71.19	74.58	69.49	72.88	69.49	71.02	2.20
FrozenBERT-bs4-ep100	76.27	71.19	71.19	69.49	67.80	72.88	71.19	69.49	72.88	71.19	71.36	2.20
FrozenBERT-bs4-ep100+Dropout0.1	72.88	69.49	69.49	72.88	71.19	72.88	72.88	66.10	74.58	71.19	71.36	2.33
FrozenBERT-bs4-ep100+Dropout0.3	76.27	71.19	71.19	71.19	69.49	72.88	72.88	69.49	72.88	71.19	71.86	1.89
EmbraceBERT-bs4	71.19	64.41	69.49	71.19	64.41	66.10	69.49	72.88	72.88	74.58	69.66	3.43
EmbraceBERT-bs4+Dropout0.1	74.58	69.49	69.49	71.19	72.88	69.49	67.80	69.49	76.27	74.58	71.53	2.71
EmbraceBERT-bs4+Dropout0.3	74.58	69.49	66.10	72.88	72.88	69.49	74.58	72.88	74.58	71.19	71.86	2.65
CondensedEmbraceBERT-bs4	6.78	20.34	23.73	10.17	16.95	6.78	6.78	6.78	15.25	6.78	12.03	6.18
CondensedEmbraceBERT-bs4+Dropout0.1	71.19	66.10	64.41	67.80	71.19	69.49	69.49	71.19	72.88	76.27	70.00	3.22
CondensedEmbraceBERT-bs4+Dropout0.3	67.80	72.88	64.41	72.88	71.19	69.49	72.88	71.19	74.58	71.19	70.85	2.82
FrozenEBERT-bs4-ep100	72.88	66.10	74.58	71.19	71.19	69.49	74.58	69.49	77.97	76.27	72.37	3.39
FrozenEBERT-bs4-ep100+Dropout0.1	74.58	72.88	69.49	71.19	76.27	74.58	76.27	69.49	76.27	74.58	73.56	2.54
FrozenEBERT-bs4-ep100+Dropout0.3	77.97	69.49	66.10	72.88	72.88	71.19	71.19	72.88	72.88	74.58	72.20	2.96
FrozenCEBERT-bs4-ep100	10.17	18.64	22.03	6.78	18.64	6.78	8.47	5.08	22.03	6.78	12.54	6.57
FrozenCEBERT-bs4-ep100+Dropout0.1	69.49	72.88	66.10	71.19	71.19	72.88	72.88	71.19	74.58	74.58	71.69	2.40
FrozenCEBERT-bs4-ep100+Dropout0.3	67.80	74.58	67.80	72.88	72.88	71.19	67.80	72.88	74.58	69.49	71.19	2.63
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs4	67.80	71.19	64.41	69.49	67.80	69.49	61.02	69.49	67.80	71.19	67.97	2.98
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4+Branches	69.49	71.19	67.80	66.10	67.80	67.80	72.88	69.49	69.49	72.88	69.49	2.14
EmbraceBERT-bs4+Branches+Dropout0.1	71.19	72.88	64.41	71.19	72.88	72.88	69.49	71.19	71.19	67.80	70.51	2.54
EmbraceBERT-bs4+Branches+Dropout0.3	71.19	77.97	69.49	74.58	69.49	69.49	72.88	69.49	71.19	62.71	70.85	3.77
CondensedEmbraceBERT-bs4+Branches	71.19	72.88	67.80	66.10	72.88	67.80	72.88	69.49	67.80	71.19	70.00	2.40
CondensedEmbraceBERT-bs4+Branches+Dropout0.1	66.10	74.58	67.80	64.41	71.19	69.49	76.27	67.80	66.10	69.49	69.32	3.59
CondensedEmbraceBERT-bs4+Branches+Dropout0.3	69.49	74.58	69.49	62.71	69.49	72.88	72.88	66.10	67.80	66.10	69.15	3.46
FrozenEmbraceBERT-bs4+Branches	71.19	67.80	67.80	66.10	72.88	64.41	77.97	69.49	77.97	71.19	70.68	4.36
FrozenEmbraceBERT-bs4+Branches+Dropout0.1	72.88	72.88	67.80	74.58	76.27	81.36	74.58	71.19	72.88	66.10	73.05	4.04
FrozenEmbraceBERT-bs4+Branches+Dropout0.3	71.19	74.58	69.49	76.27	71.19	77.97	76.27	71.19	74.58	64.41	72.71	3.82
FrozenCEBERT-bs4+Branches	69.49	72.88	69.49	69.49	69.49	72.88	76.27	66.10	66.10	69.49	70.17	2.96
FrozenCEBERT-bs4+Branches+Dropout0.1	67.80	71.19	69.49	66.10	72.88	72.88	74.58	67.80	69.49	71.19	70.34	2.54
FrozenCEBERT-bs4+Branches+Dropout0.3	69.49	76.27	66.10	71.19	71.19	74.58	74.58	67.80	72.88	69.49	71.36	3.07
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	72.88	69.49	67.80	66.10	76.27	77.97	71.19	69.49	71.19	71.19	71.36	3.43
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw	69.49	69.49	71.19	62.71	62.71	71.19	76.27	71.19	72.88	69.49	69.66	3.97
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	72.88	69.49	66.10	67.80	66.10	76.27	71.19	64.41	69.49	67.80	69.15	3.37
BERT-bs16+Dropout0.1	71.19	69.49	64.41	69.49	67.80	72.88	66.10	66.10	67.80	67.80	68.31	2.40
BERT-bs16+Dropout0.3	72.88	71.19	64.41	67.80	69.49	71.19	66.10	67.80	69.49	67.80	68.81	2.42
FrozenBERT-bs16-ep100	72.88	69.49	66.10	71.19	67.80	76.27	72.88	66.10	71.19	69.49	70.34	3.06
FrozenBERT-bs16-ep100+Dropout0.1	72.88	72.88	64.41	67.80	67.80	72.88	72.88	64.41	69.49	71.19	69.66	3.26
FrozenBERT-bs16-ep100+Dropout0.3	72.88	67.80	64.41	67.80	66.10	71.19	74.58	66.10	69.49	69.49	68.98	3.04
EmbraceBERT-bs16	69.49	61.02	66.10	66.10	74.58	64.41	67.80	67.80	66.10	74.58	67.80	4.01
EmbraceBERT-bs16+Dropout0.1	71.19	62.71	66.10	67.80	67.80	67.80	69.49	71.19	66.10	71.19	68.14	2.60
EmbraceBERT-bs16+Dropout0.3	69.49	59.32	67.80	69.49	71.19	64.41	67.80	69.49	67.80	72.88	67.97	3.59
CondensedEmbraceBERT-bs16	8.47	28.81	11.86	5.08	8.47	8.47	22.03	16.95	13.56	6.78	13.05	7.15
CondensedEmbraceBERT-bs16+Dropout0.1	69.49	62.71	67.80	62.71	69.49	69.49	69.49	71.19	64.41	76.27	68.31	3.94
CondensedEmbraceBERT-bs16+Dropout0.3	69.49	62.71	67.80	66.10	74.58	69.49	64.41	72.88	66.10	79.66	69.32	4.88
FrozenEBERT-bs16-ep100	69.49	64.41	66.10	71.19	72.88	71.19	69.49	72.88	67.80	74.58	70.00	3.04
FrozenEBERT-bs16-ep100+Dropout0.1	69.49	64.41	62.71	69.49	67.80	64.41	69.49	71.19	67.80	77.97	68.47	4.10
FrozenEBERT-bs16-ep100+Dropout0.3	69.49	61.02	66.10	67.80	69.49	62.71	74.58	66.10	69.49	79.66	68.64	5.15
FrozenCEBERT-bs16-ep100	8.47	25.42	11.86	5.08	13.56	6.78	13.56	13.56	10.17	5.08	11.36	5.67
FrozenCEBERT-bs16-ep100+Dropout0.1	72.88	64.41	66.10	64.41	67.80	71.19	71.19	69.49	67.80	79.66	69.49	4.35
FrozenCEBERT-bs16-ep100+Dropout0.3	72.88	66.10	62.71	67.80	76.27	69.49	67.80	76.27	66.10	79.66	70.51	5.21
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	67.80	62.71	67.80	67.80	72.88	64.41	67.80	67.80	66.10	74.58	67.97	3.34
EmbraceBERT-bs16+Branches+Dropout0.1	71.19	71.19	72.88	71.19	64.41	69.49	66.10	59.32	69.49	66.10	68.14	3.92
EmbraceBERT-bs16+Branches+Dropout0.3	67.80	72.88	67.80	66.10	66.10	74.58	71.19	62.71	69.49	67.80	68.64	3.33
CondensedEmbraceBERT-bs16+Branches	67.80	64.41	67.80	62.71	71.19	66.10	67.80	66.10	62.71	74.58	67.12	3.49
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	67.80	72.88	64.41	69.49	71.19	67.80	71.19	59.32	69.49	62.71	67.63	4.04
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	66.10	67.80	69.49	71.19	62.71	67.80	72.88	61.02	72.88	66.10	67.80	3.79
FrozenEmbraceBERT-bs16+Branches	71.19	62.71	71.19	69.49	71.19	64.41	72.88	69.49	67.80	74.58	69.49	3.47
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	72.88	72.88	76.27	69.49	66.10	71.19	74.58	61.02	71.19	69.49	70.51	4.17
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	71.19	72.88	71.19	69.49	66.10	72.88	67.80	61.02	69.49	69.49	69.15	3.37
FrozenCEBERT-bs16+Branches	69.49	71.19	64.41	69.49	69.49	64.41	76.27	62.71	67.80	61.02	67.63	4.32
FrozenCEBERT-bs16+Branches+Dropout0.1	67.80	72.88	64.41	67.80	71.19	67.80	74.58	61.02	69.49	62.71	67.97	4.11
FrozenCEBERT-bs16+Branches+Dropout0.3	67.80	69.49	66.10	69.49	71.19	71.19	74.58	62.71	69.49	69.49	69.15	3.01
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	67.80	64.41	69.49	62.71	66.10	69.49	69.49	67.80	67.80	62.71	66.78	2.54
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	84.75	83.05	84.75	84.75	88.14	83.05	83.05	86.44	84.75	81.36	84.41	1.83
RoBERTa-bs4+Dropout0.1	86.44	81.36	89.83	84.75	84.75	84.75	86.44	84.75	81.36	84.75	84.92	2.33
RoBERTa-bs4+Dropout0.3	88.14	81.36	84.75	79.66	86.44	88.14	83.05	86.44	83.05	83.05	84.41	2.71
FrozenRoBERTa-bs4-ep100	84.75	84.75	84.75	84.75	84.75	83.05	83.05	84.75	81.36	81.36	83.73	1.36
FrozenRoBERTa-bs4-ep100+Dropout0.1	86.44	81.36	86.44	83.05	86.44	84.75	81.36	84.75	83.05	81.36	83.90	2.04
FrozenRoBERTa-bs4-ep100+Dropout0.3	83.05	81.36	88.14	86.44	86.44	83.05	83.05	86.44	84.75	77.97	84.07	2.86
EmbraceRoBERTa-bs4	79.66	81.36	86.44	86.44	81.36	84.75	86.44	79.66	83.05	79.66	82.88	2.78
EmbraceRoBERTa-bs4+Dropout0.1	83.05	86.44	83.05	81.36	83.05	83.05	86.44	81.36	81.36	84.75	83.39	1.83
EmbraceRoBERTa-bs4+Dropout0.3	79.66	83.05	88.14	84.75	86.44	84.75	83.05	83.05	79.66	79.66	83.22	2.78
CondensedEmbraceRoBERTa-bs4	28.81	40.68	44.07	10.17	25.42	6.78	28.81	20.34	6.78	5.08	21.69	13.53
CondensedEmbraceRoBERTa-bs4+Dropout0.1	84.75	81.36	88.14	83.05	86.44	84.75	86.44	86.44	83.05	88.14	85.25	2.15
CondensedEmbraceRoBERTa-bs4+Dropout0.3	81.36	83.05	86.44	84.75	81.36	76.27	88.14	81.36	81.36	84.75	82.88	3.17
FrozenERoBERTa-bs4-ep100	83.05	81.36	84.75	84.75	81.36	83.05	86.44	84.75	81.36	83.05	83.39	1.66
FrozenERoBERTa-bs4-ep100+Dropout0.1	84.75	81.36	83.05	81.36	81.36	83.05	83.05	83.05	81.36	83.05	82.54	1.09
FrozenERoBERTa-bs4-ep100+Dropout0.3	81.36	77.97	86.44	81.36	81.36	86.44	84.75	86.44	86.44	81.36	83.39	2.92
FrozenCERoBERTa-bs4-ep100	10.17	42.37	45.76	15.25	25.42	6.78	30.51	23.73	3.39	3.39	20.68	14.69
FrozenCERoBERTa-bs4-ep100+Dropout0.1	83.05	83.05	83.05	81.36	79.66	83.05	84.75	77.97	84.75	81.36	82.20	2.04
FrozenCERoBERTa-bs4-ep100+Dropout0.3	81.36	86.44	86.44	83.05	83.05	83.05	83.05	88.14	84.75	86.44	84.58	2.07
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs4+Branches	84.75	83.05	86.44	84.75	83.05	81.36	84.75	84.75	81.36	84.75	83.90	1.56
EmbraceRoBERTa-bs4+Branches+Dropout0.1	79.66	83.05	88.14	88.14	86.44	88.14	84.75	89.83	81.36	84.75	85.42	3.14
EmbraceRoBERTa-bs4+Branches+Dropout0.3	81.36	83.05	84.75	81.36	84.75	84.75	89.83	83.05	88.14	86.44	84.75	2.63
CondensedEmbraceRoBERTa-bs4+Branches	83.05	86.44	89.83	83.05	81.36	88.14	88.14	86.44	86.44	83.05	85.59	2.65
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1	83.05	83.05	86.44	88.14	86.44	86.44	84.75	81.36	83.05	81.36	84.41	2.25
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3	86.44	81.36	86.44	81.36	86.44	81.36	84.75	86.44	86.44	84.75	84.58	2.20
FrozenEmbraceRoBERTa-bs4+Branches	81.36	83.05	86.44	83.05	84.75	86.44	83.05	84.75	86.44	81.36	84.07	1.89
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1	81.36	83.05	89.83	89.83	88.14	89.83	86.44	88.14	81.36	83.05	86.10	3.37
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3	84.75	83.05	88.14	81.36	89.83	89.83	89.83	83.05	86.44	84.75	86.10	3.01
FrozenCERoBERTa-bs4+Branches	83.05	88.14	89.83	84.75	84.75	83.05	86.44	86.44	88.14	84.75	85.93	2.15
FrozenCERoBERTa-bs4+Branches+Dropout0.1	83.05	81.36	86.44	88.14	86.44	84.75	83.05	81.36	83.05	81.36	83.90	2.31
FrozenCERoBERTa-bs4+Branches+Dropout0.3	86.44	81.36	86.44	83.05	86.44	79.66	83.05	86.44	88.14	84.75	84.58	2.56
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	81.36	76.27	84.75	83.05	91.53	84.75	83.05	84.75	83.05	84.75	83.73	3.57
RoBERTa-bs16+Dropout0.1	84.75	79.66	83.05	81.36	86.44	84.75	79.66	81.36	77.97	79.66	81.86	2.63
RoBERTa-bs16+Dropout0.3	84.75	81.36	83.05	86.44	88.14	81.36	77.97	81.36	79.66	77.97	82.20	3.24
FrozenRoBERTa-bs16-ep100	84.75	74.58	84.75	84.75	91.53	83.05	83.05	83.05	84.75	84.75	83.90	3.88
FrozenRoBERTa-bs16-ep100+Dropout0.1	83.05	77.97	81.36	83.05	84.75	81.36	83.05	81.36	79.66	79.66	81.53	1.93
FrozenRoBERTa-bs16-ep100+Dropout0.3	86.44	74.58	79.66	84.75	83.05	81.36	84.75	81.36	76.27	79.66	81.19	3.59
EmbraceRoBERTa-bs16	86.44	84.75	79.66	86.44	79.66	86.44	84.75	77.97	83.05	86.44	83.56	3.13
EmbraceRoBERTa-bs16+Dropout0.1	88.14	86.44	84.75	88.14	83.05	83.05	89.83	79.66	81.36	84.75	84.92	3.07
EmbraceRoBERTa-bs16+Dropout0.3	88.14	84.75	79.66	86.44	83.05	84.75	89.83	79.66	83.05	84.75	84.41	3.11
CondensedEmbraceRoBERTa-bs16	28.81	33.90	37.29	10.17	23.73	5.08	30.51	18.64	3.39	20.34	21.19	11.30
CondensedEmbraceRoBERTa-bs16+Dropout0.1	89.83	81.36	86.44	84.75	83.05	81.36	89.83	76.27	86.44	83.05	84.24	3.94
CondensedEmbraceRoBERTa-bs16+Dropout0.3	86.44	83.05	88.14	86.44	84.75	83.05	86.44	76.27	88.14	84.75	84.75	3.30
FrozenERoBERTa-bs16-ep100	86.44	84.75	83.05	84.75	79.66	89.83	83.05	84.75	88.14	83.05	84.75	2.73
FrozenERoBERTa-bs16-ep100+Dropout0.1	84.75	84.75	84.75	79.66	79.66	84.75	83.05	81.36	84.75	83.05	83.05	2.01
FrozenERoBERTa-bs16-ep100+Dropout0.3	84.75	84.75	91.53	81.36	79.66	83.05	86.44	89.83	79.66	84.75	84.58	3.75
FrozenCERoBERTa-bs16-ep100	20.34	42.37	30.51	11.86	23.73	6.78	38.98	18.64	5.08	22.03	22.03	11.89
FrozenCERoBERTa-bs16-ep100+Dropout0.1	86.44	84.75	79.66	84.75	79.66	81.36	88.14	81.36	81.36	84.75	83.22	2.78
FrozenCERoBERTa-bs16-ep100+Dropout0.3	81.36	84.75	81.36	84.75	74.58	79.66	84.75	84.75	81.36	84.75	82.20	3.15
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	81.36	79.66	79.66	81.36	74.58	79.66	83.05	72.88	77.97	84.75	79.49	3.43
EmbraceRoBERTa-bs16+Branches+Dropout0.1	84.75	86.44	83.05	84.75	79.66	86.44	88.14	81.36	83.05	84.75	84.24	2.40
EmbraceRoBERTa-bs16+Branches+Dropout0.3	83.05	83.05	84.75	84.75	79.66	88.14	84.75	83.05	81.36	84.75	83.73	2.17
CondensedEmbraceRoBERTa-bs16+Branches	84.75	83.05	74.58	81.36	77.97	84.75	76.27	74.58	79.66	84.75	80.17	3.94
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	84.75	83.05	86.44	83.05	84.75	84.75	89.83	83.05	84.75	88.14	85.25	2.15
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	84.75	83.05	79.66	83.05	79.66	84.75	84.75	86.44	81.36	83.05	83.05	2.14
FrozenEmbraceRoBERTa-bs16+Branches	84.75	81.36	83.05	83.05	86.44	81.36	86.44	88.14	81.36	84.75	84.07	2.30
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	83.05	84.75	88.14	83.05	81.36	84.75	83.05	84.75	83.05	84.75	84.07	1.73
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	83.05	86.44	86.44	83.05	84.75	86.44	84.75	83.05	83.05	84.75	84.58	1.41
FrozenCERoBERTa-bs16+Branches	81.36	86.44	88.14	83.05	81.36	84.75	89.83	86.44	88.14	84.75	85.42	2.75
FrozenCERoBERTa-bs16+Branches+Dropout0.1	81.36	84.75	84.75	84.75	84.75	86.44	86.44	81.36	81.36	84.75	84.07	1.89
FrozenCERoBERTa-bs16+Branches+Dropout0.3	83.05	83.05	86.44	81.36	81.36	86.44	84.75	88.14	81.36	84.75	84.07	2.30

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |72.88|69.49|64.41|66.10|64.41|71.19|74.58|66.10|69.49|69.49|68.81|3.32| | BERTwithAtt-bs8 |74.58|69.49|67.80|66.10|66.10|69.49|64.41|74.58|62.71|76.27|69.15|4.41| | BERTwithAttClsProjection-bs8 |64.41|66.10|66.10|66.10|62.71|69.49|71.19|64.41|67.80|66.10|66.44|2.37| | BERTwithProjection-bs8 |61.02|64.41|76.27|67.80|66.10|71.19|57.63|71.19|61.02|71.19|66.78|5.53| | BERTwithProjectionAtt-bs8 |74.58|67.80|67.80|69.49|69.49|67.80|71.19|66.10|69.49|66.10|68.98|2.40| | EmbraceBERT-bs8 |69.49|62.71|64.41|69.49|67.80|67.80|64.41|69.49|64.41|71.19|67.12|2.75| | EmbraceBERT-bs8-p_att_clsquery_weights |67.80|67.80|66.10|64.41|71.19|69.49|66.10|72.88|67.80|72.88|68.64|2.76| | EmbraceBERTwithProj-bs8 |66.10|72.88|71.19|67.80|69.49|66.10|67.80|69.49|69.49|71.19|69.15|2.12| | EmbraceBERTwithProj-bs8-p_attclsqw |66.10|67.80|67.80|69.49|69.49|69.49|71.19|67.80|67.80|67.80|68.47|1.36| | EBERTconcatAtt-bs8 |66.10|71.19|69.49|66.10|66.10|66.10|69.49|67.80|71.19|74.58|68.81|2.75| | EBERTconcatAtt-bs8_attclsqw |67.80|69.49|66.10|59.32|66.10|64.41|67.80|67.80|66.10|67.80|66.27|2.67| | EBERTconcatAttwithProj-bs8 |69.49|62.71|66.10|62.71|62.71|66.10|61.02|69.49|69.49|67.80|65.76|3.11| | EBERTconcatAttwithProj-bs8_attclsqw |69.49|64.41|71.19|69.49|62.71|66.10|67.80|71.19|71.19|67.80|68.14|2.82| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |71.19|67.80|62.71|64.41|64.41|69.49|69.49|69.49|69.49|67.80|67.63|2.67| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|74.58|69.49|72.88|69.49|71.19|72.88|71.19|67.80|67.80|70.51|2.30| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |67.80|64.41|64.41|72.88|69.49|69.49|67.80|66.10|67.80|69.49|67.97|2.45| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |64.41|64.41|61.02|62.71|67.80|64.41|66.10|67.80|67.80|67.80|65.42|2.30| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |69.49|74.58|67.80|67.80|66.10|67.80|62.71|67.80|72.88|67.80|68.47|3.14| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |69.49|72.88|69.49|69.49|72.88|71.19|66.10|69.49|66.10|71.19|69.83|2.25| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |64.41|66.10|71.19|67.80|69.49|71.19|67.80|64.41|66.10|67.80|67.63|2.33| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|67.80|69.49|61.02|67.80|69.49|64.41|62.71|66.10|67.80|66.44|2.71|

macsay-google - good BERT bs4, RoBERTa all

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	67.80	66.10	64.41	62.71	62.71	62.71	59.32	55.93	64.41	61.02	62.71	3.22
BERT-bs4+Dropout0.1	66.10	64.41	61.02	64.41	61.02	62.71	64.41	64.41	61.02	64.41	63.39	1.73
BERT-bs4+Dropout0.3	66.10	64.41	61.02	64.41	61.02	62.71	64.41	59.32	62.71	62.71	62.88	1.93
FrozenBERT-bs4-ep100	66.10	64.41	66.10	62.71	59.32	62.71	62.71	57.63	67.80	64.41	63.39	2.96
FrozenBERT-bs4-ep100+Dropout0.1	61.02	64.41	61.02	62.71	59.32	61.02	61.02	57.63	69.49	57.63	61.53	3.31
FrozenBERT-bs4-ep100+Dropout0.3	66.10	66.10	62.71	62.71	61.02	64.41	62.71	61.02	69.49	59.32	63.56	2.86
EmbraceBERT-bs4	62.71	62.71	55.93	62.71	61.02	61.02	62.71	62.71	64.41	64.41	62.03	2.30
EmbraceBERT-bs4+Dropout0.1	62.71	64.41	62.71	61.02	67.80	67.80	62.71	62.71	62.71	66.10	64.07	2.25
EmbraceBERT-bs4+Dropout0.3	66.10	59.32	59.32	64.41	62.71	61.02	61.02	64.41	62.71	69.49	63.05	3.01
CondensedEmbraceBERT-bs4	8.47	13.56	20.34	6.78	23.73	8.47	11.86	6.78	18.64	5.08	12.37	6.16
CondensedEmbraceBERT-bs4+Dropout0.1	67.80	54.24	64.41	66.10	62.71	66.10	62.71	61.02	66.10	66.10	63.73	3.73
CondensedEmbraceBERT-bs4+Dropout0.3	55.93	61.02	59.32	62.71	66.10	57.63	66.10	69.49	62.71	62.71	62.37	3.92
FrozenEBERT-bs4-ep100	62.71	59.32	61.02	66.10	57.63	62.71	62.71	67.80	64.41	64.41	62.88	2.88
FrozenEBERT-bs4-ep100+Dropout0.1	66.10	64.41	59.32	61.02	66.10	66.10	67.80	62.71	64.41	67.80	64.58	2.67
FrozenEBERT-bs4-ep100+Dropout0.3	62.71	61.02	57.63	66.10	62.71	64.41	64.41	66.10	62.71	67.80	63.56	2.76
FrozenCEBERT-bs4-ep100	8.47	13.56	22.03	13.56	22.03	13.56	11.86	8.47	15.25	6.78	13.56	4.97
FrozenCEBERT-bs4-ep100+Dropout0.1	64.41	59.32	59.32	64.41	64.41	69.49	62.71	62.71	62.71	64.41	63.39	2.75
FrozenCEBERT-bs4-ep100+Dropout0.3	55.93	64.41	57.63	62.71	66.10	69.49	64.41	69.49	61.02	62.71	63.39	4.23
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs4	57.63	61.02	54.24	64.41	54.24	64.41	52.54	62.71	61.02	61.02	59.32	4.15
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4+Branches	59.32	61.02	59.32	66.10	61.02	62.71	62.71	59.32	61.02	64.41	61.69	2.17
EmbraceBERT-bs4+Branches+Dropout0.1	55.93	64.41	64.41	59.32	66.10	64.41	67.80	57.63	59.32	52.54	61.19	4.70
EmbraceBERT-bs4+Branches+Dropout0.3	61.02	64.41	55.93	61.02	67.80	59.32	66.10	61.02	59.32	59.32	61.53	3.39
CondensedEmbraceBERT-bs4+Branches	64.41	64.41	62.71	64.41	62.71	61.02	64.41	57.63	66.10	57.63	62.54	2.78
CondensedEmbraceBERT-bs4+Branches+Dropout0.1	59.32	61.02	66.10	55.93	61.02	67.80	61.02	66.10	59.32	62.71	62.03	3.49
CondensedEmbraceBERT-bs4+Branches+Dropout0.3	62.71	62.71	55.93	59.32	64.41	64.41	61.02	62.71	64.41	55.93	61.36	3.11
FrozenEmbraceBERT-bs4+Branches	67.80	59.32	59.32	57.63	67.80	59.32	67.80	62.71	67.80	61.02	63.05	4.07
FrozenEmbraceBERT-bs4+Branches+Dropout0.1	54.24	61.02	61.02	59.32	62.71	67.80	66.10	57.63	64.41	54.24	60.85	4.38
FrozenEmbraceBERT-bs4+Branches+Dropout0.3	55.93	61.02	55.93	61.02	62.71	66.10	71.19	62.71	61.02	59.32	61.69	4.30
FrozenCEBERT-bs4+Branches	59.32	64.41	57.63	61.02	62.71	62.71	67.80	57.63	66.10	61.02	62.03	3.23
FrozenCEBERT-bs4+Branches+Dropout0.1	61.02	59.32	64.41	57.63	69.49	59.32	66.10	62.71	59.32	64.41	62.37	3.54
FrozenCEBERT-bs4+Branches+Dropout0.3	57.63	61.02	59.32	62.71	61.02	61.02	62.71	59.32	64.41	55.93	60.51	2.40
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	57.63	62.71	57.63	57.63	66.10	66.10	66.10	59.32	59.32	61.02	61.36	3.46
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw	62.71	59.32	59.32	57.63	55.93	66.10	71.19	59.32	62.71	61.02	61.53	4.22
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	62.71	64.41	59.32	62.71	62.71	59.32	61.02	54.24	64.41	61.02	61.19	2.88
BERT-bs16+Dropout0.1	62.71	62.71	64.41	62.71	59.32	64.41	55.93	57.63	64.41	61.02	61.53	2.84
BERT-bs16+Dropout0.3	64.41	66.10	66.10	59.32	61.02	62.71	59.32	62.71	62.71	62.71	62.71	2.27
FrozenBERT-bs16-ep100	64.41	62.71	61.02	62.71	61.02	62.71	62.71	54.24	64.41	59.32	61.53	2.84
FrozenBERT-bs16-ep100+Dropout0.1	61.02	66.10	61.02	64.41	61.02	64.41	62.71	57.63	61.02	59.32	61.86	2.43
FrozenBERT-bs16-ep100+Dropout0.3	64.41	67.80	62.71	64.41	61.02	64.41	61.02	61.02	61.02	61.02	62.88	2.20
EmbraceBERT-bs16	61.02	54.24	59.32	64.41	59.32	64.41	55.93	67.80	69.49	59.32	61.53	4.68
EmbraceBERT-bs16+Dropout0.1	62.71	54.24	61.02	62.71	59.32	61.02	59.32	66.10	64.41	64.41	61.53	3.22
EmbraceBERT-bs16+Dropout0.3	61.02	52.54	61.02	61.02	59.32	61.02	59.32	64.41	61.02	62.71	60.34	2.96
CondensedEmbraceBERT-bs16	3.39	23.73	15.25	5.08	16.95	8.47	11.86	15.25	6.78	10.17	11.69	5.89
CondensedEmbraceBERT-bs16+Dropout0.1	59.32	55.93	62.71	59.32	59.32	64.41	61.02	66.10	61.02	62.71	61.19	2.78
CondensedEmbraceBERT-bs16+Dropout0.3	62.71	57.63	59.32	64.41	72.88	64.41	52.54	66.10	59.32	64.41	62.37	5.24
FrozenEBERT-bs16-ep100	61.02	55.93	57.63	64.41	59.32	61.02	57.63	67.80	69.49	62.71	61.69	4.23
FrozenEBERT-bs16-ep100+Dropout0.1	57.63	59.32	57.63	64.41	66.10	55.93	59.32	64.41	61.02	61.02	60.68	3.20
FrozenEBERT-bs16-ep100+Dropout0.3	59.32	52.54	62.71	66.10	61.02	57.63	62.71	61.02	69.49	62.71	61.53	4.36
FrozenCEBERT-bs16-ep100	3.39	27.12	15.25	5.08	15.25	8.47	10.17	20.34	5.08	10.17	12.03	7.13
FrozenCEBERT-bs16-ep100+Dropout0.1	61.02	59.32	62.71	66.10	57.63	62.71	59.32	64.41	62.71	66.10	62.20	2.74
FrozenCEBERT-bs16-ep100+Dropout0.3	62.71	57.63	59.32	66.10	67.80	61.02	55.93	62.71	57.63	64.41	61.53	3.72
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	61.02	59.32	62.71	62.71	57.63	57.63	59.32	64.41	59.32	59.32	60.34	2.17
EmbraceBERT-bs16+Branches+Dropout0.1	61.02	64.41	62.71	62.71	62.71	62.71	59.32	54.24	66.10	66.10	62.20	3.31
EmbraceBERT-bs16+Branches+Dropout0.3	59.32	64.41	61.02	64.41	62.71	62.71	64.41	52.54	62.71	59.32	61.36	3.46
CondensedEmbraceBERT-bs16+Branches	59.32	59.32	61.02	57.63	66.10	61.02	61.02	66.10	57.63	64.41	61.36	3.01
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	59.32	62.71	61.02	61.02	62.71	62.71	61.02	57.63	61.02	59.32	60.85	1.60
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	59.32	66.10	57.63	62.71	61.02	62.71	64.41	55.93	64.41	61.02	61.53	3.04
FrozenEmbraceBERT-bs16+Branches	59.32	57.63	61.02	64.41	64.41	61.02	66.10	66.10	66.10	62.71	62.88	2.88
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	62.71	62.71	61.02	57.63	61.02	57.63	64.41	59.32	69.49	62.71	61.86	3.33
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	61.02	61.02	61.02	64.41	57.63	59.32	62.71	55.93	59.32	62.71	60.51	2.40
FrozenCEBERT-bs16+Branches	59.32	59.32	62.71	64.41	64.41	57.63	66.10	57.63	66.10	52.54	61.02	4.22
FrozenCEBERT-bs16+Branches+Dropout0.1	59.32	62.71	64.41	62.71	64.41	66.10	69.49	57.63	62.71	62.71	63.22	3.13
FrozenCEBERT-bs16+Branches+Dropout0.3	59.32	64.41	61.02	62.71	64.41	62.71	69.49	57.63	62.71	62.71	62.71	3.03
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	61.02	57.63	57.63	59.32	61.02	57.63	66.10	55.93	59.32	52.54	58.81	3.39
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	76.27	79.66	81.36	72.88	74.58	77.97	74.58	79.66	72.88	77.97	76.78	2.84
RoBERTa-bs4+Dropout0.1	76.27	74.58	83.05	77.97	74.58	76.27	76.27	72.88	69.49	77.97	75.93	3.37
RoBERTa-bs4+Dropout0.3	79.66	76.27	74.58	72.88	76.27	72.88	77.97	76.27	76.27	74.58	75.76	2.01
FrozenRoBERTa-bs4-ep100	74.58	77.97	81.36	71.19	71.19	76.27	69.49	76.27	71.19	76.27	74.58	3.56
FrozenRoBERTa-bs4-ep100+Dropout0.1	77.97	71.19	77.97	67.80	71.19	79.66	71.19	76.27	71.19	76.27	74.07	3.79
FrozenRoBERTa-bs4-ep100+Dropout0.3	71.19	74.58	76.27	71.19	72.88	81.36	72.88	74.58	69.49	72.88	73.73	3.15
EmbraceRoBERTa-bs4	71.19	77.97	72.88	76.27	74.58	77.97	79.66	76.27	74.58	74.58	75.59	2.42
EmbraceRoBERTa-bs4+Dropout0.1	76.27	77.97	77.97	71.19	72.88	79.66	76.27	76.27	76.27	76.27	76.10	2.33
EmbraceRoBERTa-bs4+Dropout0.3	71.19	72.88	74.58	72.88	71.19	72.88	76.27	72.88	71.19	74.58	73.05	1.60
CondensedEmbraceRoBERTa-bs4	32.20	28.81	35.59	16.95	18.64	10.17	20.34	23.73	5.08	6.78	19.83	9.94
CondensedEmbraceRoBERTa-bs4+Dropout0.1	72.88	77.97	77.97	77.97	72.88	77.97	77.97	77.97	76.27	72.88	76.27	2.27
CondensedEmbraceRoBERTa-bs4+Dropout0.3	71.19	76.27	79.66	76.27	72.88	74.58	77.97	77.97	72.88	74.58	75.42	2.54
FrozenERoBERTa-bs4-ep100	69.49	77.97	67.80	74.58	74.58	74.58	74.58	77.97	69.49	74.58	73.56	3.32
FrozenERoBERTa-bs4-ep100+Dropout0.1	71.19	72.88	74.58	71.19	69.49	72.88	71.19	72.88	72.88	74.58	72.37	1.53
FrozenERoBERTa-bs4-ep100+Dropout0.3	69.49	72.88	77.97	76.27	71.19	69.49	74.58	69.49	72.88	74.58	72.88	2.84
FrozenCERoBERTa-bs4-ep100	25.42	30.51	37.29	18.64	15.25	15.25	18.64	20.34	3.39	10.17	19.49	9.26
FrozenCERoBERTa-bs4-ep100+Dropout0.1	72.88	71.19	72.88	72.88	72.88	76.27	77.97	74.58	69.49	74.58	73.56	2.30
FrozenCERoBERTa-bs4-ep100+Dropout0.3	69.49	72.88	72.88	72.88	71.19	77.97	72.88	72.88	71.19	77.97	73.22	2.60
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs4+Branches	81.36	76.27	76.27	79.66	79.66	77.97	74.58	77.97	74.58	81.36	77.97	2.40
EmbraceRoBERTa-bs4+Branches+Dropout0.1	74.58	77.97	79.66	77.97	77.97	76.27	81.36	81.36	77.97	74.58	77.97	2.27
EmbraceRoBERTa-bs4+Branches+Dropout0.3	72.88	72.88	74.58	76.27	72.88	77.97	76.27	76.27	72.88	76.27	74.92	1.83
CondensedEmbraceRoBERTa-bs4+Branches	76.27	74.58	81.36	77.97	76.27	79.66	76.27	81.36	74.58	77.97	77.63	2.37
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1	71.19	76.27	74.58	79.66	74.58	76.27	76.27	79.66	69.49	81.36	75.93	3.54
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3	74.58	76.27	69.49	74.58	76.27	71.19	67.80	81.36	77.97	77.97	74.75	3.97
FrozenEmbraceRoBERTa-bs4+Branches	76.27	76.27	74.58	76.27	72.88	77.97	71.19	77.97	76.27	76.27	75.59	2.03
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1	71.19	79.66	77.97	74.58	77.97	76.27	79.66	77.97	74.58	69.49	75.93	3.29
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3	72.88	72.88	77.97	71.19	76.27	74.58	77.97	74.58	76.27	72.88	74.75	2.20
FrozenCERoBERTa-bs4+Branches	76.27	74.58	77.97	76.27	77.97	77.97	76.27	76.27	71.19	72.88	75.76	2.15
FrozenCERoBERTa-bs4+Branches+Dropout0.1	71.19	77.97	72.88	77.97	74.58	71.19	76.27	74.58	71.19	77.97	74.58	2.73
FrozenCERoBERTa-bs4+Branches+Dropout0.3	71.19	76.27	71.19	71.19	76.27	71.19	69.49	76.27	72.88	77.97	73.39	2.84
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	74.58	74.58	76.27	74.58	74.58	77.97	74.58	76.27	74.58	74.58	75.25	1.12
RoBERTa-bs16+Dropout0.1	74.58	76.27	77.97	72.88	81.36	83.05	76.27	71.19	72.88	72.88	75.93	3.70
RoBERTa-bs16+Dropout0.3	77.97	74.58	76.27	71.19	72.88	76.27	77.97	71.19	71.19	74.58	74.41	2.56
FrozenRoBERTa-bs16-ep100	76.27	74.58	74.58	71.19	72.88	76.27	76.27	77.97	74.58	79.66	75.42	2.31
FrozenRoBERTa-bs16-ep100+Dropout0.1	74.58	71.19	79.66	67.80	72.88	77.97	77.97	67.80	72.88	72.88	73.56	3.88
FrozenRoBERTa-bs16-ep100+Dropout0.3	76.27	69.49	77.97	69.49	74.58	81.36	76.27	67.80	71.19	72.88	73.73	4.10
EmbraceRoBERTa-bs16	77.97	76.27	76.27	79.66	77.97	83.05	77.97	77.97	83.05	79.66	78.98	2.30
EmbraceRoBERTa-bs16+Dropout0.1	81.36	76.27	76.27	81.36	79.66	79.66	81.36	76.27	77.97	81.36	79.15	2.15
EmbraceRoBERTa-bs16+Dropout0.3	72.88	76.27	76.27	83.05	77.97	83.05	83.05	79.66	77.97	79.66	78.98	3.23
CondensedEmbraceRoBERTa-bs16	25.42	40.68	32.20	18.64	15.25	13.56	25.42	15.25	3.39	22.03	21.19	9.95
CondensedEmbraceRoBERTa-bs16+Dropout0.1	74.58	77.97	77.97	79.66	76.27	79.66	81.36	74.58	77.97	77.97	77.80	2.07
CondensedEmbraceRoBERTa-bs16+Dropout0.3	74.58	74.58	76.27	81.36	77.97	77.97	77.97	76.27	77.97	81.36	77.63	2.25
FrozenERoBERTa-bs16-ep100	71.19	72.88	72.88	76.27	76.27	77.97	74.58	76.27	77.97	76.27	75.25	2.17
FrozenERoBERTa-bs16-ep100+Dropout0.1	71.19	77.97	74.58	76.27	77.97	76.27	74.58	76.27	79.66	79.66	76.44	2.45
FrozenERoBERTa-bs16-ep100+Dropout0.3	69.49	76.27	79.66	76.27	77.97	76.27	79.66	74.58	76.27	79.66	76.61	2.92
FrozenCERoBERTa-bs16-ep100	22.03	40.68	32.20	16.95	11.86	16.95	28.81	15.25	6.78	23.73	21.53	9.65
FrozenCERoBERTa-bs16-ep100+Dropout0.1	72.88	77.97	71.19	79.66	77.97	77.97	77.97	72.88	76.27	76.27	76.10	2.67
FrozenCERoBERTa-bs16-ep100+Dropout0.3	74.58	76.27	72.88	79.66	72.88	77.97	76.27	74.58	76.27	77.97	75.93	2.12
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	79.66	74.58	76.27	79.66	67.80	77.97	79.66	77.97	77.97	76.27	76.78	3.39
EmbraceRoBERTa-bs16+Branches+Dropout0.1	79.66	79.66	76.27	79.66	77.97	77.97	81.36	81.36	76.27	77.97	78.81	1.74
EmbraceRoBERTa-bs16+Branches+Dropout0.3	79.66	74.58	72.88	77.97	79.66	81.36	79.66	83.05	76.27	77.97	78.31	2.92
CondensedEmbraceRoBERTa-bs16+Branches	81.36	72.88	71.19	76.27	71.19	77.97	72.88	76.27	79.66	77.97	75.76	3.39
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	76.27	81.36	74.58	79.66	79.66	77.97	77.97	86.44	76.27	79.66	78.98	3.14
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	77.97	79.66	77.97	77.97	79.66	76.27	74.58	81.36	77.97	79.66	78.31	1.83
FrozenEmbraceRoBERTa-bs16+Branches	72.88	72.88	76.27	76.27	71.19	79.66	79.66	81.36	74.58	77.97	76.27	3.22
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	76.27	74.58	76.27	72.88	74.58	79.66	76.27	76.27	74.58	76.27	75.76	1.70
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	74.58	76.27	77.97	72.88	76.27	81.36	72.88	79.66	74.58	74.58	76.10	2.67
FrozenCERoBERTa-bs16+Branches	77.97	76.27	79.66	81.36	79.66	77.97	77.97	77.97	74.58	76.27	77.97	1.86
FrozenCERoBERTa-bs16+Branches+Dropout0.1	74.58	72.88	76.27	76.27	76.27	76.27	76.27	84.75	74.58	77.97	76.61	3.01
FrozenCERoBERTa-bs16+Branches+Dropout0.3	74.58	76.27	76.27	74.58	77.97	77.97	74.58	77.97	72.88	77.97	76.10	1.77

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |66.10|61.02|59.32|62.71|59.32|67.80|61.02|59.32|67.80|61.02|62.54|3.26| | BERTwithAtt-bs8 |57.63|59.32|57.63|59.32|59.32|59.32|55.93|62.71|54.24|67.80|59.32|3.56| | BERTwithAttClsProjection-bs8 |55.93|59.32|62.71|54.24|55.93|62.71|64.41|57.63|61.02|57.63|59.15|3.26| | BERTwithProjection-bs8 |57.63|57.63|62.71|61.02|62.71|62.71|54.24|57.63|54.24|61.02|59.15|3.17| | BERTwithProjectionAtt-bs8 |62.71|64.41|62.71|61.02|61.02|59.32|67.80|62.71|57.63|59.32|61.86|2.76| | EmbraceBERT-bs8 |64.41|55.93|61.02|61.02|57.63|61.02|57.63|64.41|64.41|62.71|61.02|2.94| | EmbraceBERT-bs8-p_att_clsquery_weights |61.02|62.71|57.63|57.63|59.32|61.02|62.71|66.10|55.93|59.32|60.34|2.86| | EmbraceBERTwithProj-bs8 |64.41|66.10|62.71|69.49|69.49|61.02|61.02|64.41|61.02|64.41|64.41|3.03| | EmbraceBERTwithProj-bs8-p_attclsqw |55.93|62.71|59.32|67.80|57.63|61.02|64.41|64.41|61.02|61.02|61.53|3.31| | EBERTconcatAtt-bs8 |59.32|62.71|55.93|55.93|57.63|55.93|62.71|57.63|61.02|66.10|59.49|3.34| | EBERTconcatAtt-bs8_attclsqw |61.02|62.71|61.02|55.93|55.93|57.63|62.71|59.32|57.63|59.32|59.32|2.40| | EBERTconcatAttwithProj-bs8 |62.71|59.32|59.32|57.63|55.93|59.32|54.24|61.02|62.71|59.32|59.15|2.56| | EBERTconcatAttwithProj-bs8_attclsqw |61.02|57.63|64.41|64.41|54.24|55.93|57.63|57.63|57.63|59.32|58.98|3.20| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |64.41|54.24|55.93|61.02|59.32|59.32|64.41|61.02|59.32|55.93|59.49|3.26| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |59.32|62.71|66.10|61.02|62.71|67.80|61.02|61.02|64.41|59.32|62.54|2.67| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |62.71|62.71|59.32|62.71|66.10|61.02|61.02|57.63|62.71|61.02|61.69|2.17| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |62.71|57.63|54.24|62.71|61.02|62.71|59.32|67.80|64.41|62.71|61.53|3.56| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |61.02|61.02|55.93|59.32|61.02|59.32|50.85|59.32|61.02|57.63|58.64|3.05| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |57.63|59.32|61.02|59.32|62.71|62.71|55.93|62.71|57.63|66.10|60.51|2.94| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |59.32|59.32|62.71|55.93|61.02|61.02|59.32|59.32|59.32|57.63|59.49|1.77| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |61.02|57.63|64.41|55.93|66.10|62.71|55.93|57.63|59.32|61.02|60.17|3.33|

macsay-sphinx - good RoBERTa bs16

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	38.98	32.20	38.98	33.90	32.20	28.81	35.59	32.20	40.68	30.51	34.41	3.79
BERT-bs4+Dropout0.1	38.98	28.81	35.59	38.98	33.90	27.12	28.81	28.81	40.68	30.51	33.22	4.81
BERT-bs4+Dropout0.3	35.59	27.12	33.90	38.98	32.20	35.59	28.81	32.20	38.98	30.51	33.39	3.79
FrozenBERT-bs4-ep100	42.37	28.81	37.29	37.29	32.20	27.12	30.51	32.20	35.59	32.20	33.56	4.34
FrozenBERT-bs4-ep100+Dropout0.1	42.37	38.98	28.81	30.51	37.29	25.42	32.20	35.59	42.37	27.12	34.07	5.84
FrozenBERT-bs4-ep100+Dropout0.3	33.90	32.20	35.59	28.81	32.20	27.12	32.20	33.90	40.68	30.51	32.71	3.56
EmbraceBERT-bs4	35.59	32.20	28.81	30.51	28.81	35.59	37.29	30.51	40.68	33.90	33.39	3.72
EmbraceBERT-bs4+Dropout0.1	23.73	37.29	32.20	32.20	40.68	38.98	33.90	30.51	32.20	25.42	32.71	5.14
EmbraceBERT-bs4+Dropout0.3	35.59	32.20	33.90	30.51	30.51	32.20	30.51	30.51	37.29	35.59	32.88	2.42
CondensedEmbraceBERT-bs4	11.86	6.78	18.64	3.39	11.86	11.86	18.64	10.17	22.03	3.39	11.86	6.06
CondensedEmbraceBERT-bs4+Dropout0.1	28.81	33.90	33.90	32.20	35.59	38.98	32.20	28.81	42.37	27.12	33.39	4.49
CondensedEmbraceBERT-bs4+Dropout0.3	23.73	32.20	28.81	40.68	35.59	40.68	27.12	30.51	44.07	40.68	34.41	6.57
FrozenEBERT-bs4-ep100	33.90	35.59	40.68	32.20	33.90	32.20	33.90	28.81	33.90	30.51	33.56	3.01
FrozenEBERT-bs4-ep100+Dropout0.1	22.03	37.29	33.90	28.81	37.29	40.68	30.51	27.12	32.20	27.12	31.69	5.42
FrozenEBERT-bs4-ep100+Dropout0.3	30.51	35.59	33.90	33.90	30.51	38.98	32.20	27.12	33.90	28.81	32.54	3.29
FrozenCEBERT-bs4-ep100	11.86	5.08	16.95	8.47	11.86	11.86	16.95	1.69	20.34	5.08	11.02	5.68
FrozenCEBERT-bs4-ep100+Dropout0.1	27.12	35.59	35.59	30.51	25.42	38.98	30.51	30.51	42.37	30.51	32.71	5.03
FrozenCEBERT-bs4-ep100+Dropout0.3	23.73	33.90	27.12	35.59	35.59	40.68	37.29	30.51	33.90	38.98	33.73	5.00
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs4	37.29	33.90	28.81	38.98	30.51	33.90	33.90	32.20	32.20	38.98	34.07	3.26
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4+Branches	28.81	44.07	35.59	35.59	30.51	33.90	37.29	23.73	23.73	38.98	33.22	6.21
EmbraceBERT-bs4+Branches+Dropout0.1	40.68	38.98	30.51	27.12	25.42	42.37	30.51	32.20	30.51	33.90	33.22	5.42
EmbraceBERT-bs4+Branches+Dropout0.3	35.59	35.59	32.20	32.20	37.29	42.37	35.59	30.51	35.59	35.59	35.25	3.11
CondensedEmbraceBERT-bs4+Branches	28.81	40.68	35.59	32.20	32.20	33.90	35.59	25.42	28.81	32.20	32.54	4.07
CondensedEmbraceBERT-bs4+Branches+Dropout0.1	30.51	30.51	32.20	35.59	23.73	30.51	38.98	38.98	32.20	33.90	32.71	4.29
CondensedEmbraceBERT-bs4+Branches+Dropout0.3	37.29	35.59	30.51	30.51	32.20	42.37	25.42	37.29	38.98	28.81	33.90	4.97
FrozenEmbraceBERT-bs4+Branches	25.42	42.37	37.29	30.51	30.51	27.12	38.98	28.81	30.51	35.59	32.71	5.25
FrozenEmbraceBERT-bs4+Branches+Dropout0.1	40.68	38.98	28.81	27.12	37.29	38.98	32.20	32.20	33.90	28.81	33.90	4.61
FrozenEmbraceBERT-bs4+Branches+Dropout0.3	37.29	35.59	35.59	38.98	37.29	28.81	37.29	33.90	32.20	32.20	34.92	2.96
FrozenCEBERT-bs4+Branches	44.07	28.81	33.90	25.42	30.51	38.98	42.37	28.81	30.51	35.59	33.90	5.92
FrozenCEBERT-bs4+Branches+Dropout0.1	33.90	32.20	32.20	40.68	27.12	30.51	37.29	38.98	33.90	32.20	33.90	3.87
FrozenCEBERT-bs4+Branches+Dropout0.3	35.59	33.90	27.12	33.90	33.90	33.90	30.51	37.29	33.90	32.20	33.22	2.65
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	27.12	38.98	35.59	30.51	38.98	20.34	23.73	33.90	35.59	35.59	32.03	6.09
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw	45.76	40.68	33.90	37.29	32.20	30.51	40.68	25.42	32.20	27.12	34.58	6.12
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	40.68	28.81	38.98	33.90	30.51	30.51	28.81	32.20	40.68	30.51	33.56	4.54
BERT-bs16+Dropout0.1	32.20	30.51	37.29	35.59	32.20	27.12	33.90	37.29	37.29	30.51	33.39	3.31
BERT-bs16+Dropout0.3	37.29	27.12	35.59	33.90	30.51	32.20	37.29	40.68	37.29	30.51	34.24	3.92
FrozenBERT-bs16-ep100	42.37	32.20	37.29	40.68	30.51	30.51	33.90	33.90	40.68	25.42	34.75	5.15
FrozenBERT-bs16-ep100+Dropout0.1	37.29	28.81	38.98	35.59	32.20	37.29	30.51	38.98	45.76	27.12	35.25	5.35
FrozenBERT-bs16-ep100+Dropout0.3	40.68	25.42	40.68	35.59	33.90	38.98	32.20	42.37	40.68	30.51	36.10	5.25
EmbraceBERT-bs16	27.12	30.51	32.20	37.29	37.29	35.59	37.29	28.81	40.68	30.51	33.73	4.25
EmbraceBERT-bs16+Dropout0.1	25.42	33.90	37.29	37.29	30.51	37.29	37.29	35.59	28.81	28.81	33.22	4.23
EmbraceBERT-bs16+Dropout0.3	30.51	32.20	37.29	37.29	33.90	35.59	32.20	33.90	28.81	25.42	32.71	3.56
CondensedEmbraceBERT-bs16	10.17	13.56	16.95	8.47	10.17	6.78	15.25	13.56	11.86	11.86	11.86	2.94
CondensedEmbraceBERT-bs16+Dropout0.1	37.29	32.20	35.59	40.68	37.29	30.51	38.98	35.59	33.90	35.59	35.76	2.88
CondensedEmbraceBERT-bs16+Dropout0.3	30.51	37.29	35.59	38.98	35.59	30.51	27.12	33.90	32.20	35.59	33.73	3.43
FrozenEBERT-bs16-ep100	28.81	32.20	38.98	32.20	33.90	33.90	35.59	23.73	40.68	25.42	32.54	5.13
FrozenEBERT-bs16-ep100+Dropout0.1	23.73	35.59	35.59	37.29	28.81	33.90	30.51	27.12	35.59	32.20	32.03	4.18
FrozenEBERT-bs16-ep100+Dropout0.3	28.81	32.20	38.98	32.20	32.20	35.59	28.81	23.73	37.29	25.42	31.53	4.68
FrozenCEBERT-bs16-ep100	10.17	16.95	20.34	5.08	5.08	10.17	16.95	15.25	11.86	8.47	12.03	4.94
FrozenCEBERT-bs16-ep100+Dropout0.1	27.12	35.59	35.59	38.98	30.51	33.90	35.59	25.42	27.12	30.51	32.03	4.32
FrozenCEBERT-bs16-ep100+Dropout0.3	32.20	33.90	33.90	37.29	32.20	33.90	32.20	27.12	27.12	32.20	32.20	2.94
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	25.42	38.98	32.20	42.37	25.42	32.20	33.90	33.90	23.73	30.51	31.86	5.66
EmbraceBERT-bs16+Branches+Dropout0.1	35.59	37.29	28.81	33.90	30.51	38.98	38.98	30.51	27.12	32.20	33.39	4.01
EmbraceBERT-bs16+Branches+Dropout0.3	35.59	35.59	30.51	27.12	32.20	33.90	33.90	35.59	32.20	33.90	33.05	2.54
CondensedEmbraceBERT-bs16+Branches	27.12	37.29	35.59	33.90	37.29	37.29	27.12	35.59	22.03	32.20	32.54	5.07
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	33.90	38.98	42.37	32.20	32.20	32.20	33.90	38.98	32.20	32.20	34.92	3.57
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	32.20	37.29	32.20	30.51	28.81	33.90	37.29	28.81	37.29	35.59	33.39	3.22
FrozenEmbraceBERT-bs16+Branches	22.03	42.37	35.59	38.98	18.64	33.90	30.51	27.12	35.59	33.90	31.86	7.02
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	40.68	33.90	35.59	37.29	28.81	37.29	30.51	37.29	27.12	33.90	34.24	4.07
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	44.07	35.59	27.12	35.59	32.20	30.51	44.07	40.68	30.51	35.59	35.59	5.52
FrozenCEBERT-bs16+Branches	42.37	42.37	37.29	37.29	30.51	30.51	28.81	37.29	35.59	32.20	35.42	4.58
FrozenCEBERT-bs16+Branches+Dropout0.1	35.59	38.98	35.59	33.90	37.29	37.29	35.59	37.29	27.12	30.51	34.92	3.41
FrozenCEBERT-bs16+Branches+Dropout0.3	32.20	33.90	27.12	38.98	32.20	35.59	28.81	35.59	37.29	35.59	33.73	3.51
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	30.51	33.90	33.90	37.29	40.68	37.29	32.20	27.12	33.90	32.20	33.90	3.64
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	50.85	47.46	47.46	32.20	50.85	38.98	45.76	40.68	38.98	49.15	44.24	5.89
RoBERTa-bs4+Dropout0.1	54.24	49.15	47.46	47.46	49.15	40.68	50.85	47.46	45.76	44.07	47.63	3.51
RoBERTa-bs4+Dropout0.3	52.54	49.15	42.37	44.07	54.24	50.85	45.76	49.15	54.24	45.76	48.81	4.00
FrozenRoBERTa-bs4-ep100	42.37	49.15	49.15	33.90	44.07	40.68	45.76	44.07	38.98	45.76	43.39	4.43
FrozenRoBERTa-bs4-ep100+Dropout0.1	49.15	37.29	49.15	33.90	49.15	50.85	50.85	42.37	38.98	49.15	45.08	6.03
FrozenRoBERTa-bs4-ep100+Dropout0.3	44.07	38.98	57.63	40.68	52.54	44.07	49.15	40.68	47.46	50.85	46.61	5.68
EmbraceRoBERTa-bs4	35.59	47.46	47.46	50.85	40.68	52.54	50.85	52.54	47.46	44.07	46.95	5.20
EmbraceRoBERTa-bs4+Dropout0.1	42.37	52.54	42.37	45.76	47.46	50.85	45.76	45.76	45.76	47.46	46.61	3.06
EmbraceRoBERTa-bs4+Dropout0.3	35.59	54.24	47.46	47.46	44.07	49.15	47.46	45.76	45.76	44.07	46.10	4.47
CondensedEmbraceRoBERTa-bs4	18.64	22.03	22.03	18.64	25.42	20.34	18.64	15.25	6.78	1.69	16.95	6.95
CondensedEmbraceRoBERTa-bs4+Dropout0.1	38.98	55.93	45.76	52.54	42.37	50.85	52.54	49.15	45.76	45.76	47.97	4.92
CondensedEmbraceRoBERTa-bs4+Dropout0.3	38.98	44.07	47.46	45.76	45.76	55.93	52.54	49.15	38.98	45.76	46.44	5.04
FrozenERoBERTa-bs4-ep100	32.20	42.37	42.37	49.15	45.76	49.15	44.07	47.46	47.46	44.07	44.41	4.72
FrozenERoBERTa-bs4-ep100+Dropout0.1	42.37	52.54	45.76	44.07	47.46	44.07	44.07	49.15	42.37	45.76	45.76	3.03
FrozenERoBERTa-bs4-ep100+Dropout0.3	45.76	50.85	40.68	40.68	47.46	44.07	40.68	49.15	45.76	44.07	44.92	3.41
FrozenCERoBERTa-bs4-ep100	11.86	22.03	23.73	16.95	16.95	23.73	13.56	18.64	6.78	5.08	15.93	6.26
FrozenCERoBERTa-bs4-ep100+Dropout0.1	44.07	47.46	49.15	40.68	42.37	47.46	40.68	50.85	44.07	45.76	45.25	3.31
FrozenCERoBERTa-bs4-ep100+Dropout0.3	44.07	45.76	45.76	45.76	44.07	45.76	44.07	49.15	42.37	47.46	45.42	1.83
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs4+Branches	44.07	50.85	55.93	42.37	44.07	50.85	40.68	50.85	40.68	47.46	46.78	4.92
EmbraceRoBERTa-bs4+Branches+Dropout0.1	44.07	52.54	47.46	40.68	49.15	49.15	40.68	50.85	44.07	49.15	46.78	3.95
EmbraceRoBERTa-bs4+Branches+Dropout0.3	50.85	49.15	38.98	40.68	54.24	47.46	44.07	47.46	47.46	50.85	47.12	4.47
CondensedEmbraceRoBERTa-bs4+Branches	44.07	44.07	54.24	38.98	47.46	54.24	40.68	49.15	38.98	45.76	45.76	5.31
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1	47.46	50.85	44.07	44.07	50.85	47.46	42.37	50.85	45.76	45.76	46.95	2.94
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3	45.76	50.85	45.76	45.76	42.37	44.07	37.29	52.54	47.46	44.07	45.59	4.04
FrozenEmbraceRoBERTa-bs4+Branches	42.37	45.76	50.85	40.68	47.46	50.85	42.37	49.15	40.68	45.76	45.59	3.75
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1	47.46	50.85	50.85	38.98	52.54	49.15	42.37	50.85	45.76	49.15	47.80	4.07
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3	50.85	47.46	45.76	47.46	57.63	49.15	44.07	47.46	49.15	50.85	48.98	3.51
FrozenCERoBERTa-bs4+Branches	44.07	44.07	49.15	40.68	49.15	47.46	40.68	50.85	44.07	45.76	45.59	3.34
FrozenCERoBERTa-bs4+Branches+Dropout0.1	44.07	52.54	49.15	44.07	52.54	49.15	38.98	50.85	45.76	47.46	47.46	4.08
FrozenCERoBERTa-bs4+Branches+Dropout0.3	42.37	45.76	50.85	44.07	44.07	47.46	42.37	52.54	47.46	45.76	46.27	3.22
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	50.85	44.07	44.07	47.46	49.15	40.68	45.76	42.37	47.46	49.15	46.10	3.11
RoBERTa-bs16+Dropout0.1	45.76	47.46	47.46	40.68	47.46	49.15	45.76	40.68	44.07	47.46	45.59	2.78
RoBERTa-bs16+Dropout0.3	45.76	47.46	47.46	42.37	49.15	50.85	42.37	45.76	45.76	47.46	46.44	2.54
FrozenRoBERTa-bs16-ep100	49.15	44.07	45.76	33.90	49.15	44.07	47.46	45.76	40.68	50.85	45.08	4.68
FrozenRoBERTa-bs16-ep100+Dropout0.1	50.85	44.07	42.37	38.98	50.85	52.54	49.15	42.37	47.46	44.07	46.27	4.29
FrozenRoBERTa-bs16-ep100+Dropout0.3	49.15	44.07	44.07	35.59	50.85	45.76	49.15	45.76	38.98	47.46	45.08	4.50
EmbraceRoBERTa-bs16	44.07	49.15	40.68	49.15	47.46	45.76	42.37	40.68	40.68	47.46	44.75	3.32
EmbraceRoBERTa-bs16+Dropout0.1	44.07	47.46	47.46	45.76	42.37	45.76	44.07	45.76	47.46	45.76	45.59	1.60
EmbraceRoBERTa-bs16+Dropout0.3	44.07	47.46	45.76	44.07	47.46	47.46	42.37	40.68	45.76	45.76	45.08	2.17
CondensedEmbraceRoBERTa-bs16	18.64	22.03	18.64	11.86	18.64	20.34	15.25	18.64	6.78	6.78	15.76	5.20
CondensedEmbraceRoBERTa-bs16+Dropout0.1	42.37	50.85	45.76	44.07	44.07	52.54	52.54	38.98	44.07	45.76	46.10	4.27
CondensedEmbraceRoBERTa-bs16+Dropout0.3	42.37	49.15	45.76	45.76	49.15	54.24	49.15	42.37	44.07	42.37	46.44	3.73
FrozenERoBERTa-bs16-ep100	37.29	50.85	42.37	50.85	50.85	59.32	42.37	45.76	40.68	45.76	46.61	6.12
FrozenERoBERTa-bs16-ep100+Dropout0.1	45.76	49.15	47.46	40.68	45.76	42.37	44.07	52.54	40.68	49.15	45.76	3.71
FrozenERoBERTa-bs16-ep100+Dropout0.3	45.76	49.15	52.54	40.68	49.15	50.85	44.07	47.46	40.68	45.76	46.61	3.81
FrozenCERoBERTa-bs16-ep100	13.56	27.12	18.64	11.86	15.25	23.73	11.86	15.25	5.08	8.47	15.08	6.32
FrozenCERoBERTa-bs16-ep100+Dropout0.1	44.07	47.46	45.76	42.37	42.37	44.07	42.37	49.15	40.68	45.76	44.41	2.49
FrozenCERoBERTa-bs16-ep100+Dropout0.3	45.76	49.15	49.15	42.37	42.37	45.76	45.76	50.85	42.37	45.76	45.93	2.88
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	42.37	42.37	40.68	37.29	37.29	42.37	40.68	42.37	38.98	40.68	40.51	1.93
EmbraceRoBERTa-bs16+Branches+Dropout0.1	45.76	49.15	49.15	40.68	45.76	49.15	44.07	49.15	42.37	45.76	46.10	2.92
EmbraceRoBERTa-bs16+Branches+Dropout0.3	44.07	52.54	42.37	40.68	47.46	49.15	47.46	49.15	45.76	47.46	46.61	3.33
CondensedEmbraceRoBERTa-bs16+Branches	40.68	42.37	40.68	35.59	42.37	44.07	38.98	45.76	35.59	40.68	40.68	3.13
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	44.07	45.76	44.07	38.98	44.07	45.76	42.37	45.76	44.07	42.37	43.73	1.98
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	40.68	49.15	44.07	40.68	40.68	45.76	37.29	47.46	37.29	44.07	42.71	3.85
FrozenEmbraceRoBERTa-bs16+Branches	44.07	49.15	47.46	45.76	44.07	49.15	42.37	42.37	45.76	47.46	45.76	2.40
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	47.46	49.15	47.46	45.76	45.76	54.24	44.07	47.46	44.07	45.76	47.12	2.82
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	40.68	50.85	54.24	40.68	42.37	54.24	44.07	47.46	50.85	45.76	47.12	4.96
FrozenCERoBERTa-bs16+Branches	44.07	52.54	44.07	42.37	45.76	47.46	44.07	44.07	44.07	45.76	45.42	2.71
FrozenCERoBERTa-bs16+Branches+Dropout0.1	44.07	50.85	44.07	40.68	38.98	52.54	42.37	45.76	42.37	44.07	44.58	4.01
FrozenCERoBERTa-bs16+Branches+Dropout0.3	40.68	50.85	49.15	42.37	44.07	52.54	42.37	50.85	42.37	44.07	45.93	4.18

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |42.37|28.81|38.98|37.29|33.90|28.81|32.20|35.59|47.46|28.81|35.42|5.94| | BERTwithAtt-bs8 |27.12|35.59|32.20|28.81|37.29|32.20|32.20|32.20|32.20|44.07|33.39|4.49| | BERTwithAttClsProjection-bs8 |35.59|30.51|40.68|37.29|35.59|33.90|30.51|32.20|30.51|35.59|34.24|3.20| | BERTwithProjection-bs8 |37.29|32.20|37.29|37.29|22.03|37.29|20.34|32.20|33.90|40.68|33.05|6.44| | BERTwithProjectionAtt-bs8 |32.20|35.59|30.51|30.51|30.51|30.51|28.81|33.90|33.90|32.20|31.86|1.98| | EmbraceBERT-bs8 |27.12|45.76|33.90|30.51|33.90|35.59|23.73|25.42|37.29|25.42|31.86|6.47| | EmbraceBERT-bs8-p_att_clsquery_weights |42.37|30.51|38.98|28.81|35.59|35.59|32.20|28.81|37.29|32.20|34.24|4.27| | EmbraceBERTwithProj-bs8 |40.68|35.59|37.29|32.20|35.59|40.68|37.29|37.29|35.59|38.98|37.12|2.45| | EmbraceBERTwithProj-bs8-p_attclsqw |27.12|37.29|33.90|35.59|33.90|35.59|33.90|28.81|28.81|28.81|32.37|3.43| | EBERTconcatAtt-bs8 |38.98|32.20|37.29|33.90|35.59|32.20|33.90|30.51|38.98|35.59|34.92|2.75| | EBERTconcatAtt-bs8_attclsqw |30.51|37.29|25.42|30.51|38.98|27.12|33.90|33.90|27.12|30.51|31.53|4.23| | EBERTconcatAttwithProj-bs8 |37.29|32.20|32.20|25.42|35.59|42.37|33.90|40.68|27.12|32.20|33.90|5.08| | EBERTconcatAttwithProj-bs8_attclsqw |30.51|30.51|33.90|32.20|30.51|30.51|27.12|30.51|38.98|32.20|31.69|2.94| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |37.29|33.90|32.20|40.68|33.90|30.51|35.59|30.51|38.98|33.90|34.75|3.24| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |38.98|42.37|30.51|47.46|32.20|33.90|30.51|35.59|32.20|28.81|35.25|5.66| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |30.51|37.29|35.59|23.73|38.98|35.59|37.29|30.51|30.51|38.98|33.90|4.67| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |40.68|32.20|28.81|40.68|32.20|30.51|33.90|33.90|35.59|42.37|35.08|4.42| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |37.29|44.07|25.42|33.90|38.98|35.59|38.98|42.37|40.68|40.68|37.80|5.03| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |33.90|30.51|28.81|38.98|25.42|32.20|38.98|33.90|27.12|32.20|32.20|4.29| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |42.37|30.51|38.98|28.81|38.98|33.90|33.90|42.37|33.90|32.20|35.59|4.55| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |40.68|32.20|32.20|32.20|30.51|30.51|37.29|37.29|27.12|35.59|33.56|3.85|

macsay-witai - good RoBERTa bs16

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	77.97	69.49	76.27	74.58	67.80	74.58	67.80	71.19	72.88	72.88	72.54	3.29
BERT-bs4+Dropout0.1	74.58	71.19	76.27	71.19	64.41	71.19	71.19	67.80	72.88	72.88	71.36	3.17
BERT-bs4+Dropout0.3	72.88	69.49	74.58	72.88	66.10	71.19	76.27	71.19	69.49	74.58	71.86	2.86
FrozenBERT-bs4-ep100	76.27	74.58	74.58	74.58	69.49	77.97	72.88	74.58	72.88	72.88	74.07	2.15
FrozenBERT-bs4-ep100+Dropout0.1	74.58	71.19	72.88	71.19	74.58	72.88	69.49	67.80	76.27	72.88	72.37	2.40
FrozenBERT-bs4-ep100+Dropout0.3	77.97	71.19	74.58	72.88	72.88	74.58	72.88	71.19	74.58	74.58	73.73	1.89
EmbraceBERT-bs4	71.19	67.80	69.49	67.80	67.80	62.71	74.58	72.88	69.49	74.58	69.83	3.46
EmbraceBERT-bs4+Dropout0.1	69.49	71.19	67.80	67.80	77.97	67.80	71.19	66.10	71.19	67.80	69.83	3.20
EmbraceBERT-bs4+Dropout0.3	74.58	74.58	66.10	71.19	72.88	67.80	77.97	72.88	76.27	67.80	72.20	3.73
CondensedEmbraceBERT-bs4	8.47	18.64	22.03	5.08	20.34	10.17	11.86	11.86	20.34	3.39	13.22	6.38
CondensedEmbraceBERT-bs4+Dropout0.1	72.88	69.49	66.10	66.10	71.19	72.88	72.88	66.10	76.27	77.97	71.19	4.01
CondensedEmbraceBERT-bs4+Dropout0.3	72.88	69.49	62.71	71.19	74.58	64.41	74.58	72.88	71.19	69.49	70.34	3.81
FrozenEBERT-bs4-ep100	72.88	67.80	71.19	69.49	69.49	64.41	79.66	72.88	71.19	74.58	71.36	3.90
FrozenEBERT-bs4-ep100+Dropout0.1	71.19	71.19	71.19	67.80	76.27	72.88	76.27	67.80	74.58	71.19	72.03	2.86
FrozenEBERT-bs4-ep100+Dropout0.3	77.97	76.27	66.10	71.19	69.49	69.49	77.97	74.58	72.88	69.49	72.54	3.85
FrozenCEBERT-bs4-ep100	8.47	16.95	22.03	3.39	23.73	8.47	5.08	10.17	22.03	5.08	12.54	7.47
FrozenCEBERT-bs4-ep100+Dropout0.1	72.88	74.58	67.80	69.49	71.19	66.10	72.88	66.10	81.36	74.58	71.69	4.42
FrozenCEBERT-bs4-ep100+Dropout0.3	72.88	72.88	66.10	74.58	77.97	72.88	71.19	74.58	77.97	72.88	73.39	3.22
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERTwithAtt-bs4	76.27	71.19	64.41	64.41	67.80	64.41	62.71	71.19	64.41	71.19	67.80	4.22
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4+Branches	69.49	69.49	62.71	64.41	66.10	66.10	72.88	64.41	71.19	71.19	67.80	3.30
EmbraceBERT-bs4+Branches+Dropout0.1	66.10	71.19	64.41	71.19	67.80	74.58	69.49	67.80	69.49	62.71	68.47	3.32
EmbraceBERT-bs4+Branches+Dropout0.3	67.80	71.19	67.80	72.88	67.80	72.88	76.27	62.71	69.49	66.10	69.49	3.71
CondensedEmbraceBERT-bs4+Branches	66.10	67.80	66.10	64.41	71.19	69.49	74.58	64.41	69.49	69.49	68.31	3.04
CondensedEmbraceBERT-bs4+Branches+Dropout0.1	67.80	71.19	74.58	67.80	67.80	71.19	72.88	66.10	64.41	71.19	69.49	3.03
CondensedEmbraceBERT-bs4+Branches+Dropout0.3	71.19	71.19	66.10	61.02	72.88	69.49	71.19	69.49	64.41	62.71	67.97	3.90
FrozenEmbraceBERT-bs4+Branches	67.80	62.71	64.41	66.10	74.58	62.71	77.97	64.41	76.27	71.19	68.81	5.48
FrozenEmbraceBERT-bs4+Branches+Dropout0.1	69.49	71.19	64.41	72.88	72.88	77.97	72.88	71.19	69.49	62.71	70.51	4.17
FrozenEmbraceBERT-bs4+Branches+Dropout0.3	66.10	72.88	71.19	79.66	72.88	72.88	76.27	62.71	69.49	67.80	71.19	4.67
FrozenCEBERT-bs4+Branches	77.97	71.19	69.49	67.80	69.49	69.49	76.27	66.10	61.02	72.88	70.17	4.62
FrozenCEBERT-bs4+Branches+Dropout0.1	69.49	69.49	69.49	67.80	69.49	74.58	72.88	71.19	67.80	72.88	70.51	2.17
FrozenCEBERT-bs4+Branches+Dropout0.3	69.49	71.19	67.80	69.49	74.58	72.88	74.58	71.19	72.88	66.10	71.02	2.67
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	69.49	67.80	66.10	67.80	76.27	71.19	72.88	74.58	72.88	74.58	71.36	3.26
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery_attclsqw	71.19	71.19	74.58	64.41	66.10	71.19	74.58	74.58	72.88	69.49	71.02	3.34
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	71.19	71.19	69.49	71.19	67.80	71.19	71.19	59.32	67.80	71.19	69.15	3.54
BERT-bs16+Dropout0.1	71.19	72.88	71.19	71.19	71.19	71.19	71.19	62.71	67.80	71.19	70.17	2.75
BERT-bs16+Dropout0.3	69.49	74.58	72.88	69.49	69.49	71.19	67.80	62.71	71.19	69.49	69.83	3.01
FrozenBERT-bs16-ep100	72.88	72.88	72.88	72.88	69.49	72.88	74.58	64.41	67.80	69.49	71.02	2.98
FrozenBERT-bs16-ep100+Dropout0.1	74.58	71.19	69.49	69.49	72.88	72.88	67.80	62.71	71.19	71.19	70.34	3.15
FrozenBERT-bs16-ep100+Dropout0.3	72.88	69.49	69.49	71.19	72.88	71.19	72.88	62.71	69.49	69.49	70.17	2.86
EmbraceBERT-bs16	64.41	54.24	62.71	62.71	72.88	57.63	67.80	62.71	67.80	72.88	64.58	5.70
EmbraceBERT-bs16+Dropout0.1	62.71	59.32	61.02	66.10	64.41	66.10	69.49	67.80	66.10	69.49	65.25	3.24
EmbraceBERT-bs16+Dropout0.3	66.10	57.63	62.71	67.80	66.10	61.02	67.80	66.10	66.10	74.58	65.59	4.29
CondensedEmbraceBERT-bs16	5.08	23.73	10.17	3.39	13.56	13.56	23.73	16.95	11.86	6.78	12.88	6.70
CondensedEmbraceBERT-bs16+Dropout0.1	67.80	59.32	66.10	64.41	67.80	64.41	69.49	62.71	64.41	76.27	66.27	4.32
CondensedEmbraceBERT-bs16+Dropout0.3	64.41	61.02	62.71	67.80	71.19	64.41	66.10	71.19	62.71	77.97	66.95	4.93
FrozenEBERT-bs16-ep100	69.49	61.02	66.10	67.80	71.19	66.10	69.49	69.49	69.49	72.88	68.31	3.13
FrozenEBERT-bs16-ep100+Dropout0.1	66.10	59.32	61.02	67.80	66.10	61.02	71.19	74.58	66.10	74.58	66.78	5.15
FrozenEBERT-bs16-ep100+Dropout0.3	69.49	61.02	64.41	71.19	67.80	61.02	71.19	66.10	71.19	76.27	67.97	4.64
FrozenCEBERT-bs16-ep100	5.08	25.42	11.86	3.39	18.64	6.78	11.86	13.56	8.47	6.78	11.19	6.40
FrozenCEBERT-bs16-ep100+Dropout0.1	71.19	64.41	67.80	69.49	62.71	67.80	71.19	62.71	66.10	74.58	67.80	3.71
FrozenCEBERT-bs16-ep100+Dropout0.3	69.49	62.71	64.41	69.49	71.19	64.41	69.49	74.58	64.41	79.66	68.98	5.03
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16+Branches	62.71	62.71	61.02	62.71	64.41	62.71	67.80	59.32	66.10	72.88	64.24	3.67
EmbraceBERT-bs16+Branches+Dropout0.1	74.58	67.80	71.19	67.80	61.02	64.41	61.02	59.32	66.10	66.10	65.93	4.51
EmbraceBERT-bs16+Branches+Dropout0.3	71.19	66.10	62.71	66.10	61.02	71.19	66.10	55.93	66.10	64.41	65.08	4.30
CondensedEmbraceBERT-bs16+Branches	64.41	61.02	64.41	61.02	64.41	62.71	64.41	67.80	62.71	71.19	64.41	2.94
CondensedEmbraceBERT-bs16+Branches+Dropout0.1	69.49	66.10	64.41	71.19	64.41	62.71	67.80	57.63	62.71	55.93	64.24	4.58
CondensedEmbraceBERT-bs16+Branches+Dropout0.3	69.49	66.10	64.41	71.19	59.32	62.71	66.10	59.32	66.10	62.71	64.75	3.70
FrozenEmbraceBERT-bs16+Branches	66.10	61.02	62.71	67.80	66.10	62.71	76.27	64.41	67.80	81.36	67.63	6.09
FrozenEmbraceBERT-bs16+Branches+Dropout0.1	74.58	71.19	74.58	74.58	64.41	72.88	69.49	66.10	69.49	71.19	70.85	3.37
FrozenEmbraceBERT-bs16+Branches+Dropout0.3	74.58	69.49	67.80	69.49	64.41	69.49	67.80	62.71	64.41	67.80	67.80	3.22
FrozenCEBERT-bs16+Branches	69.49	72.88	67.80	72.88	66.10	62.71	72.88	64.41	64.41	59.32	67.29	4.49
FrozenCEBERT-bs16+Branches+Dropout0.1	71.19	69.49	67.80	74.58	69.49	67.80	69.49	61.02	62.71	64.41	67.80	3.87
FrozenCEBERT-bs16+Branches+Dropout0.3	71.19	69.49	67.80	76.27	69.49	69.49	69.49	64.41	66.10	69.49	69.32	2.98
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	59.32	64.41	66.10	61.02	61.02	62.71	67.80	61.02	69.49	62.71	63.56	3.15
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	81.36	81.36	81.36	79.66	83.05	79.66	83.05	83.05	83.05	77.97	81.36	1.69
RoBERTa-bs4+Dropout0.1	81.36	84.75	83.05	84.75	84.75	79.66	86.44	81.36	79.66	79.66	82.54	2.40
RoBERTa-bs4+Dropout0.3	84.75	83.05	81.36	76.27	84.75	84.75	81.36	86.44	83.05	81.36	82.71	2.71
FrozenRoBERTa-bs4-ep100	83.05	83.05	79.66	81.36	83.05	77.97	79.66	81.36	77.97	77.97	80.51	2.04
FrozenRoBERTa-bs4-ep100+Dropout0.1	84.75	83.05	86.44	77.97	84.75	83.05	79.66	81.36	81.36	76.27	81.86	3.04
FrozenRoBERTa-bs4-ep100+Dropout0.3	81.36	81.36	84.75	83.05	84.75	81.36	77.97	84.75	81.36	76.27	81.69	2.71
EmbraceRoBERTa-bs4	76.27	81.36	84.75	83.05	83.05	83.05	86.44	74.58	83.05	77.97	81.36	3.64
EmbraceRoBERTa-bs4+Dropout0.1	77.97	84.75	79.66	81.36	81.36	84.75	88.14	84.75	76.27	84.75	82.37	3.49
EmbraceRoBERTa-bs4+Dropout0.3	81.36	81.36	84.75	84.75	84.75	84.75	83.05	83.05	77.97	77.97	82.37	2.54
CondensedEmbraceRoBERTa-bs4	25.42	38.98	35.59	1.69	20.34	10.17	27.12	23.73	1.69	6.78	19.15	12.77
CondensedEmbraceRoBERTa-bs4+Dropout0.1	81.36	81.36	86.44	79.66	86.44	84.75	86.44	86.44	79.66	84.75	83.73	2.75
CondensedEmbraceRoBERTa-bs4+Dropout0.3	81.36	81.36	84.75	83.05	81.36	79.66	86.44	79.66	81.36	83.05	82.20	2.04
FrozenERoBERTa-bs4-ep100	76.27	79.66	81.36	81.36	81.36	83.05	86.44	79.66	79.66	79.66	80.85	2.52
FrozenERoBERTa-bs4-ep100+Dropout0.1	83.05	81.36	81.36	79.66	83.05	83.05	84.75	81.36	79.66	79.66	81.69	1.66
FrozenERoBERTa-bs4-ep100+Dropout0.3	83.05	79.66	83.05	83.05	81.36	84.75	81.36	83.05	84.75	77.97	82.20	2.04
FrozenCERoBERTa-bs4-ep100	11.86	40.68	40.68	5.08	18.64	8.47	28.81	23.73	3.39	6.78	18.81	13.44
FrozenCERoBERTa-bs4-ep100+Dropout0.1	83.05	83.05	81.36	79.66	81.36	83.05	86.44	79.66	84.75	79.66	82.20	2.18
FrozenCERoBERTa-bs4-ep100+Dropout0.3	81.36	83.05	81.36	84.75	81.36	84.75	83.05	83.05	84.75	79.66	82.71	1.66
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs4+Branches	84.75	81.36	84.75	81.36	79.66	84.75	84.75	83.05	77.97	81.36	82.37	2.30
EmbraceRoBERTa-bs4+Branches+Dropout0.1	79.66	83.05	83.05	84.75	86.44	86.44	86.44	83.05	81.36	86.44	84.07	2.30
EmbraceRoBERTa-bs4+Branches+Dropout0.3	81.36	81.36	83.05	77.97	83.05	83.05	89.83	81.36	86.44	83.05	83.05	3.03
CondensedEmbraceRoBERTa-bs4+Branches	84.75	84.75	83.05	79.66	81.36	86.44	86.44	83.05	84.75	81.36	83.56	2.15
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.1	83.05	81.36	79.66	79.66	83.05	89.83	86.44	79.66	83.05	79.66	82.54	3.22
CondensedEmbraceRoBERTa-bs4+Branches+Dropout0.3	86.44	83.05	83.05	83.05	84.75	84.75	88.14	77.97	83.05	79.66	83.39	2.82
FrozenEmbraceRoBERTa-bs4+Branches	79.66	84.75	83.05	81.36	84.75	86.44	81.36	84.75	83.05	81.36	83.05	2.01
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.1	81.36	81.36	84.75	86.44	89.83	88.14	88.14	86.44	81.36	81.36	84.92	3.17
FrozenEmbraceRoBERTa-bs4+Branches+Dropout0.3	79.66	81.36	84.75	81.36	86.44	86.44	89.83	81.36	84.75	81.36	83.73	3.05
FrozenCERoBERTa-bs4+Branches	83.05	84.75	84.75	81.36	84.75	83.05	86.44	84.75	84.75	81.36	83.90	1.56
FrozenCERoBERTa-bs4+Branches+Dropout0.1	84.75	81.36	81.36	83.05	84.75	86.44	84.75	79.66	83.05	81.36	83.05	2.01
FrozenCERoBERTa-bs4+Branches+Dropout0.3	84.75	83.05	83.05	81.36	84.75	83.05	86.44	81.36	84.75	84.75	83.73	1.55
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	81.36	74.58	81.36	79.66	84.75	77.97	83.05	81.36	79.66	76.27	80.00	2.92
RoBERTa-bs16+Dropout0.1	81.36	77.97	86.44	79.66	81.36	88.14	76.27	79.66	77.97	76.27	80.51	3.81
RoBERTa-bs16+Dropout0.3	83.05	81.36	83.05	79.66	84.75	79.66	77.97	81.36	77.97	77.97	80.68	2.30
FrozenRoBERTa-bs16-ep100	83.05	76.27	83.05	79.66	83.05	77.97	81.36	83.05	79.66	79.66	80.68	2.30
FrozenRoBERTa-bs16-ep100+Dropout0.1	81.36	77.97	81.36	74.58	83.05	77.97	81.36	81.36	77.97	77.97	79.49	2.45
FrozenRoBERTa-bs16-ep100+Dropout0.3	84.75	76.27	79.66	79.66	81.36	79.66	83.05	79.66	74.58	76.27	79.49	2.98
EmbraceRoBERTa-bs16	83.05	81.36	76.27	81.36	77.97	88.14	84.75	77.97	81.36	84.75	81.69	3.46
EmbraceRoBERTa-bs16+Dropout0.1	84.75	79.66	81.36	81.36	81.36	84.75	84.75	77.97	84.75	83.05	82.37	2.30
EmbraceRoBERTa-bs16+Dropout0.3	83.05	79.66	77.97	81.36	79.66	84.75	86.44	74.58	81.36	81.36	81.02	3.20
CondensedEmbraceRoBERTa-bs16	28.81	38.98	27.12	5.08	20.34	6.78	30.51	20.34	8.47	16.95	20.34	10.67
CondensedEmbraceRoBERTa-bs16+Dropout0.1	86.44	81.36	79.66	79.66	83.05	83.05	86.44	74.58	83.05	77.97	81.53	3.51
CondensedEmbraceRoBERTa-bs16+Dropout0.3	83.05	81.36	84.75	79.66	81.36	84.75	81.36	76.27	83.05	81.36	81.69	2.37
FrozenERoBERTa-bs16-ep100	81.36	81.36	79.66	83.05	79.66	89.83	83.05	79.66	84.75	83.05	82.54	2.94
FrozenERoBERTa-bs16-ep100+Dropout0.1	83.05	83.05	81.36	77.97	79.66	88.14	81.36	83.05	83.05	77.97	81.86	2.84
FrozenERoBERTa-bs16-ep100+Dropout0.3	83.05	83.05	79.66	79.66	79.66	84.75	83.05	84.75	77.97	81.36	81.69	2.25
FrozenCERoBERTa-bs16-ep100	15.25	42.37	30.51	5.08	15.25	8.47	38.98	20.34	10.17	15.25	20.17	12.21
FrozenCERoBERTa-bs16-ep100+Dropout0.1	86.44	81.36	74.58	81.36	79.66	84.75	88.14	79.66	83.05	83.05	82.20	3.65
FrozenCERoBERTa-bs16-ep100+Dropout0.3	83.05	79.66	81.36	81.36	72.88	81.36	81.36	81.36	81.36	81.36	80.51	2.65
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceRoBERTa-bs16+Branches	84.75	74.58	72.88	77.97	71.19	83.05	79.66	69.49	77.97	81.36	77.29	4.87
EmbraceRoBERTa-bs16+Branches+Dropout0.1	84.75	79.66	81.36	83.05	81.36	84.75	88.14	81.36	83.05	83.05	83.05	2.27
EmbraceRoBERTa-bs16+Branches+Dropout0.3	83.05	81.36	79.66	79.66	83.05	86.44	84.75	79.66	83.05	81.36	82.20	2.18
CondensedEmbraceRoBERTa-bs16+Branches	83.05	77.97	69.49	77.97	74.58	84.75	76.27	71.19	79.66	81.36	77.63	4.66
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.1	84.75	79.66	79.66	81.36	83.05	86.44	84.75	83.05	86.44	81.36	83.05	2.40
CondensedEmbraceRoBERTa-bs16+Branches+Dropout0.3	83.05	79.66	72.88	81.36	81.36	86.44	83.05	84.75	77.97	79.66	81.02	3.62
FrozenEmbraceRoBERTa-bs16+Branches	84.75	79.66	79.66	81.36	83.05	83.05	86.44	88.14	83.05	81.36	83.05	2.63
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.1	83.05	83.05	84.75	84.75	81.36	84.75	84.75	84.75	81.36	79.66	83.22	1.77
FrozenEmbraceRoBERTa-bs16+Branches+Dropout0.3	79.66	84.75	83.05	77.97	83.05	84.75	86.44	83.05	83.05	83.05	82.88	2.33
FrozenCERoBERTa-bs16+Branches	81.36	81.36	81.36	79.66	79.66	86.44	88.14	84.75	86.44	83.05	83.22	2.88
FrozenCERoBERTa-bs16+Branches+Dropout0.1	81.36	83.05	81.36	81.36	83.05	86.44	84.75	81.36	84.75	79.66	82.71	1.98
FrozenCERoBERTa-bs16+Branches+Dropout0.3	83.05	81.36	79.66	79.66	83.05	86.44	81.36	86.44	79.66	83.05	82.37	2.42

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |74.58|71.19|67.80|69.49|67.80|72.88|72.88|66.10|69.49|72.88|70.51|2.65| | BERTwithAtt-bs8 |69.49|67.80|64.41|62.71|66.10|64.41|72.88|71.19|54.24|72.88|66.61|5.36| | BERTwithAttClsProjection-bs8 |61.02|62.71|61.02|62.71|55.93|69.49|69.49|59.32|64.41|62.71|62.88|3.97| | BERTwithProjection-bs8 |61.02|66.10|74.58|67.80|62.71|71.19|50.85|72.88|55.93|72.88|65.59|7.51| | BERTwithProjectionAtt-bs8 |72.88|69.49|66.10|67.80|67.80|64.41|72.88|72.88|64.41|62.71|68.14|3.62| | EmbraceBERT-bs8 |69.49|61.02|64.41|67.80|66.10|64.41|69.49|69.49|59.32|71.19|66.27|3.75| | EmbraceBERT-bs8-p_att_clsquery_weights |72.88|66.10|66.10|62.71|67.80|71.19|67.80|72.88|66.10|77.97|69.15|4.27| | EmbraceBERTwithProj-bs8 |69.49|67.80|69.49|72.88|64.41|74.58|67.80|71.19|64.41|71.19|69.32|3.17| | EmbraceBERTwithProj-bs8-p_attclsqw |62.71|71.19|74.58|66.10|69.49|69.49|71.19|64.41|62.71|66.10|67.80|3.79| | EBERTconcatAtt-bs8 |67.80|69.49|66.10|66.10|67.80|59.32|62.71|67.80|67.80|77.97|67.29|4.55| | EBERTconcatAtt-bs8_attclsqw |64.41|64.41|67.80|55.93|64.41|61.02|66.10|64.41|64.41|67.80|64.07|3.29| | EBERTconcatAttwithProj-bs8 |66.10|62.71|61.02|61.02|57.63|62.71|61.02|66.10|66.10|62.71|62.71|2.63| | EBERTconcatAttwithProj-bs8_attclsqw |64.41|62.71|69.49|71.19|61.02|64.41|66.10|64.41|67.80|67.80|65.93|2.98| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |67.80|64.41|64.41|66.10|64.41|61.02|72.88|72.88|67.80|67.80|66.95|3.58| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |64.41|76.27|67.80|71.19|72.88|66.10|67.80|71.19|67.80|67.80|69.32|3.34| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |64.41|62.71|67.80|67.80|76.27|66.10|69.49|62.71|69.49|66.10|67.29|3.79| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |62.71|66.10|57.63|62.71|66.10|62.71|62.71|66.10|69.49|67.80|64.41|3.22| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |66.10|69.49|64.41|64.41|64.41|62.71|59.32|66.10|72.88|62.71|65.25|3.58| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |64.41|66.10|66.10|67.80|76.27|69.49|62.71|67.80|64.41|69.49|67.46|3.62| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |61.02|66.10|69.49|69.49|66.10|72.88|66.10|64.41|61.02|64.41|66.10|3.56| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|69.49|69.49|59.32|66.10|66.10|67.80|57.63|61.02|64.41|64.92|4.01|

STT Error: trained and tested with INCOMPLETE data

TODO: redo webapplications ep100, bs16 (bertquery, bertKeyVal -> deleted by mistake)

gtts-google

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	59.32	62.71	64.41	59.32	59.32	57.63	66.10	61.02	61.02	66.10	61.69	2.86
BERT-bs4+Dropout0.1	61.02	59.32	62.71	59.32	57.63	61.02	62.71	59.32	66.10	61.02	61.02	2.27
EmbraceBERT-bs4-WRONG	59.32	59.32	54.24	57.63	61.02	61.02	55.93	59.32	59.32	59.32	58.64	2.03
EmbraceBERT-bs4+Dropout0.1-WRONG	59.32	52.54	55.93	55.93	59.32	62.71	55.93	59.32	57.63	59.32	57.80	2.67
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4	59.32	61.02	57.63	59.32	64.41	57.63	59.32	57.63	67.80	61.02	60.51	3.13
CondensedEmbraceBERT-bs4+Dropout0.1	61.02	57.63	55.93	59.32	57.63	61.02	62.71	55.93	59.32	57.63	58.81	2.15
CondensedEmbraceBERT-bs4+Dropout0.3	54.24	61.02	57.63	61.02	64.41	57.63	61.02	57.63	55.93	54.24	58.47	3.15
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	66.10	64.41	66.10	66.10	62.71	64.41	66.10	64.41	64.41	67.80	65.25	1.37
FrozenEBERT-bs4-ep100-WRONG	62.71	64.41	55.93	61.02	64.41	62.71	61.02	61.02	62.71	62.71	61.86	2.31
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	59.32	54.24	54.24	55.93	47.46	54.24	50.85	54.24	47.46	54.24	53.22	3.49
BERT-bs16+Dropout0.1	54.24	49.15	59.32	44.07	49.15	54.24	49.15	55.93	49.15	54.24	51.86	4.23
EmbraceBERT-bs16-WRONG	55.93	50.85	49.15	54.24	55.93	50.85	54.24	61.02	47.46	47.46	52.71	4.11
EmbraceBERT-bs16+Dropout0.1-WRONG	55.93	50.85	50.85	47.46	59.32	52.54	59.32	50.85	44.07	55.93	52.71	4.70
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16	47.46	54.24	54.24	59.32	61.02	59.32	50.85	61.02	42.37	44.07	53.39	6.62
CondensedEmbraceBERT-bs16+Dropout0.1	55.93	54.24	49.15	55.93	57.63	62.71	57.63	57.63	47.46	52.54	55.08	4.24
CondensedEmbraceBERT-bs16+Dropout0.3	54.24	54.24	47.46	49.15	59.32	61.02	50.85	61.02	42.37	54.24	53.39	5.79
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	62.71	61.02	61.02	59.32	61.02	55.93	61.02	59.32	67.80	62.71	61.19	2.88
FrozenEBERT-bs16-ep100-WRONG	54.24	61.02	55.93	62.71	57.63	61.02	55.93	59.32	55.93	54.24	57.80	2.88
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	64.41	62.71	62.71	54.24	64.41	61.02	62.71	67.80	57.63	62.71	62.03	3.57
RoBERTa-bs4+Dropout0.1	57.63	66.10	64.41	59.32	61.02	54.24	61.02	62.71	71.19	57.63	61.53	4.61
EmbraceRoBERTa-bs4-WRONG	66.10	59.32	61.02	64.41	55.93	57.63	64.41	57.63	61.02	69.49	61.69	4.10
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	54.24	59.32	59.32	62.71	61.02	59.32	54.24	52.54	67.80	69.49	60.00	5.32
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4	55.93	57.63	32.20	66.10	55.93	66.10	64.41	55.93	62.71	69.49	58.64	10.00
CondensedEmbraceRoBERTa-bs4+Dropout0.1	59.32	61.02	59.32	64.41	54.24	59.32	59.32	50.85	72.88	64.41	60.51	5.67
CondensedEmbraceRoBERTa-bs4+Dropout0.3	59.32	62.71	62.71	61.02	54.24	61.02	52.54	55.93	57.63	66.10	59.32	4.01
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	66.10	62.71	64.41	62.71	67.80	67.80	64.41	69.49	64.41	62.71	65.25	2.31
FrozenERoBERTa-bs4-ep100-WRONG	62.71	59.32	61.02	69.49	66.10	64.41	67.80	62.71	66.10	64.41	64.41	2.94
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	62.71	59.32	54.24	61.02	55.93	55.93	59.32	57.63	54.24	61.02	58.14	2.84
RoBERTa-bs16+Dropout0.1	54.24	55.93	57.63	55.93	52.54	61.02	57.63	57.63	52.54	59.32	56.44	2.63
EmbraceRoBERTa-bs16-WRONG	44.07	45.76	37.29	47.46	37.29	59.32	45.76	44.07	40.68	57.63	45.93	7.09
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	52.54	45.76	42.37	42.37	37.29	50.85	49.15	49.15	45.76	54.24	46.95	4.97
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	47.46	49.15	40.68	44.07	37.29	52.54	44.07	42.37	49.15	50.85	45.76	4.61
CondensedEmbraceRoBERTa-bs16+Dropout0.3	38.98	42.37	27.12	35.59	38.98	44.07	42.37	45.76	40.68	49.15	40.51	5.75
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	59.32	64.41	67.80	59.32	62.71	61.02	61.02	64.41	66.10	61.02	62.71	2.73
FrozenERoBERTa-bs16-ep100-WRONG	61.02	62.71	50.85	67.80	54.24	59.32	57.63	55.93	59.32	69.49	59.83	5.47
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	64.41	62.71	69.49	62.71	62.71	62.71	64.41	62.71	62.71	67.80	64.24	2.33
BERT-bs4+Dropout0.1	64.41	62.71	62.71	64.41	69.49	64.41	64.41	67.80	74.58	59.32	65.42	4.03
EmbraceBERT-bs4	64.41	66.10	57.63	62.71	64.41	55.93	55.93	57.63	66.10	66.10	61.69	4.17
EmbraceBERT-bs4+Dropout0.1	61.02	67.80	62.71	64.41	62.71	66.10	59.32	62.71	67.80	57.63	63.22	3.22
EmbraceBERT-bs4-p_selfatt	59.32	62.71	66.10	64.41	55.93	62.71	57.63	61.02	54.24	66.10	61.02	3.94
EmbraceBERT-bs4-p_att_clsquery_weights	61.02	61.02	66.10	61.02	64.41	64.41	59.32	62.71	59.32	62.71	62.20	2.15
CondensedEmbraceBERT-bs4	62.71	66.10	59.32	62.71	67.80	61.02	59.32	59.32	64.41	64.41	62.71	2.84
CondensedEmbraceBERT-bs4+Dropout0.1	57.63	64.41	62.71	66.10	61.02	62.71	61.02	59.32	62.71	61.02	61.86	2.31
CondensedEmbraceBERT-bs4+Dropout0.3	59.32	64.41	62.71	61.02	64.41	67.80	64.41	64.41	62.71	54.24	62.54	3.51
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	69.49	66.10	64.41	64.41	62.71	66.10	66.10	62.71	64.41	67.80	65.42	2.03
FrozenEBERT-bs4-ep100-WRONG	61.02	62.71	59.32	59.32	66.10	59.32	57.63	62.71	62.71	64.41	61.53	2.52
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	55.93	64.41	61.02	61.02	57.63	66.10	55.93	67.80	61.02	52.54	60.34	4.62
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	61.02	64.41	64.41	64.41	52.54	62.71	57.63	61.02	61.02	54.24	60.34	4.03
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	61.02	66.10	67.80	62.71	61.02	62.71	64.41	57.63	64.41	62.71	63.05	2.71
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	61.02	64.41	62.71	55.93	57.63	66.10	61.02	61.02	61.02	55.93	60.68	3.20
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	59.32	64.41	64.41	61.02	64.41	62.71	64.41	59.32	66.10	62.71	62.88	2.20
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	61.02	64.41	55.93	57.63	64.41	66.10	64.41	61.02	55.93	64.41	61.53	3.64
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	64.41	62.71	61.02	59.32	61.02	57.63	61.02	64.41	69.49	64.41	62.54	3.17
BERT-bs16+Dropout0.1	62.71	66.10	59.32	59.32	62.71	59.32	59.32	59.32	57.63	66.10	61.19	2.88
EmbraceBERT-bs16	59.32	61.02	59.32	61.02	59.32	64.41	57.63	64.41	59.32	57.63	60.34	2.30
EmbraceBERT-bs16+Dropout0.1	57.63	59.32	54.24	59.32	64.41	59.32	59.32	61.02	61.02	62.71	59.83	2.63
CondensedEmbraceBERT-bs16	57.63	61.02	61.02	59.32	61.02	61.02	55.93	62.71	62.71	55.93	59.83	2.40
CondensedEmbraceBERT-bs16+Dropout0.1	57.63	55.93	61.02	61.02	61.02	59.32	62.71	57.63	57.63	57.63	59.15	2.07
CondensedEmbraceBERT-bs16+Dropout0.3	55.93	57.63	55.93	57.63	59.32	62.71	59.32	55.93	57.63	59.32	58.14	2.01
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	64.41	61.02	64.41	67.80	66.10	57.63	61.02	62.71	69.49	69.49	64.41	3.71
FrozenEBERT-bs16-ep100-WRONG	61.02	61.02	59.32	62.71	59.32	67.80	62.71	67.80	55.93	59.32	61.69	3.57
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	57.63	62.71	62.71	59.32	59.32	61.02	59.32	52.54	57.63	61.02	59.32	2.84
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	55.93	59.32	57.63	55.93	59.32	50.85	55.93	64.41	57.63	52.54	56.95	3.57
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	64.41	64.41	64.41	59.32	67.80	61.02	66.10	71.19	62.71	62.71	64.41	3.22
RoBERTa-bs4+Dropout0.1	62.71	74.58	72.88	62.71	71.19	62.71	62.71	67.80	67.80	64.41	66.95	4.37
EmbraceRoBERTa-bs4-WRONG	66.10	66.10	62.71	72.88	66.10	66.10	71.19	67.80	77.97	66.10	68.31	4.22
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	67.80	61.02	71.19	69.49	69.49	59.32	72.88	59.32	74.58	66.10	67.12	5.26
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1	64.41	62.71	67.80	67.80	64.41	66.10	67.80	57.63	71.19	74.58	66.44	4.41
CondensedEmbraceRoBERTa-bs4+Dropout0.3	69.49	64.41	67.80	66.10	62.71	66.10	62.71	71.19	74.58	66.10	67.12	3.57
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	61.02	66.10	61.02	61.02	72.88	62.71	64.41	76.27	69.49	64.41	65.93	5.05
FrozenERoBERTa-bs4-ep100-WRONG	67.80	57.63	66.10	72.88	69.49	67.80	72.88	64.41	79.66	71.19	68.98	5.57
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	61.02	62.71	67.80	62.71	62.71	57.63	61.02	64.41	64.41	64.41	62.88	2.56
RoBERTa-bs16+Dropout0.1	66.10	62.71	64.41	62.71	62.71	62.71	62.71	64.41	67.80	61.02	63.73	1.89
EmbraceRoBERTa-bs16-WRONG	62.71	61.02	54.24	66.10	59.32	61.02	59.32	59.32	61.02	64.41	60.85	3.07
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	69.49	61.02	61.02	66.10	57.63	59.32	59.32	59.32	62.71	61.02	61.69	3.41
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	64.41	64.41	61.02	71.19	64.41	62.71	61.02	59.32	66.10	71.19	64.58	3.82
CondensedEmbraceRoBERTa-bs16+Dropout0.3	62.71	62.71	62.71	69.49	62.71	55.93	62.71	61.02	61.02	64.41	62.54	3.17
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	61.02	64.41	66.10	64.41	62.71	59.32	67.80	66.10	64.41	64.41	64.07	2.37
FrozenERoBERTa-bs16-ep100-WRONG	72.88	61.02	66.10	69.49	61.02	62.71	64.41	67.80	67.80	69.49	66.27	3.75
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |64.41|67.80|61.02|59.32|62.71|57.63|64.41|59.32|64.41|67.80|62.88|3.34| | BERTwithAtt-bs8 |61.02|62.71|57.63|71.19|61.02|59.32|59.32|61.02|67.80|61.02|62.20|3.94| | BERTwithAttClsProjection-bs8 |54.24|61.02|59.32|59.32|52.54|64.41|64.41|64.41|61.02|64.41|60.51|4.09| | BERTwithProjection-bs8 |55.93|61.02|67.80|59.32|66.10|59.32|57.63|62.71|54.24|54.24|59.83|4.42| | BERTwithProjectionAtt-bs8 |57.63|66.10|62.71|62.71|61.02|64.41|62.71|57.63|61.02|55.93|61.19|3.07| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |61.02|62.71|59.32|66.10|57.63|62.71|61.02|59.32|61.02|59.32|61.02|2.27| | EmbraceBERT-bs8-p_att_clsquery_weights |62.71|59.32|57.63|64.41|62.71|61.02|57.63|59.32|64.41|55.93|60.51|2.84| | EmbraceBERTwithProj-bs8 |57.63|59.32|64.41|62.71|59.32|59.32|64.41|64.41|61.02|61.02|61.36|2.37| | EmbraceBERTwithProj-bs8-p_attclsqw |52.54|61.02|57.63|62.71|66.10|61.02|66.10|61.02|62.71|66.10|61.69|4.03| | EBERTconcatAtt-bs8 |64.41|62.71|61.02|61.02|57.63|52.54|61.02|64.41|62.71|66.10|61.36|3.70| | EBERTconcatAtt-bs8_attclsqw |64.41|66.10|59.32|57.63|66.10|54.24|64.41|57.63|64.41|64.41|61.86|4.03| | EBERTconcatAttwithProj-bs8 |64.41|57.63|62.71|64.41|61.02|57.63|62.71|62.71|61.02|55.93|61.02|2.84| | EBERTconcatAttwithProj-bs8_attclsqw |59.32|59.32|57.63|54.24|69.49|59.32|61.02|64.41|59.32|64.41|60.85|4.04| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |59.32|64.41|57.63|59.32|59.32|52.54|61.02|67.80|61.02|64.41|60.68|4.00| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |62.71|57.63|62.71|57.63|61.02|62.71|62.71|57.63|61.02|62.71|60.85|2.20| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |66.10|62.71|67.80|57.63|61.02|62.71|61.02|57.63|66.10|67.80|63.05|3.62| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |66.10|62.71|57.63|62.71|64.41|62.71|62.71|62.71|61.02|61.02|62.37|2.12| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |61.02|59.32|50.85|64.41|61.02|54.24|64.41|57.63|62.71|64.41|60.00|4.37| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |66.10|59.32|61.02|59.32|66.10|61.02|61.02|66.10|57.63|66.10|62.37|3.20| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |57.63|59.32|61.02|64.41|55.93|59.32|66.10|59.32|57.63|61.02|60.17|2.96| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |64.41|57.63|62.71|61.02|61.02|57.63|61.02|64.41|61.02|62.71|61.36|2.25|

gtts-sphinx

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	62.71	55.93	59.32	54.24	52.54	59.32	52.54	50.85	59.32	54.24	56.10	3.67
BERT-bs4+Dropout0.1	59.32	54.24	57.63	57.63	52.54	54.24	50.85	55.93	61.02	52.54	55.59	3.11
EmbraceBERT-bs4-WRONG	52.54	55.93	55.93	54.24	54.24	57.63	55.93	54.24	54.24	54.24	54.92	1.36
EmbraceBERT-bs4+Dropout0.1-WRONG	52.54	55.93	54.24	61.02	57.63	57.63	54.24	57.63	50.85	54.24	55.59	2.82
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4	52.54	55.93	54.24	57.63	50.85	52.54	52.54	57.63	50.85	57.63	54.24	2.63
CondensedEmbraceBERT-bs4+Dropout0.1	52.54	55.93	50.85	59.32	55.93	57.63	55.93	55.93	50.85	55.93	55.08	2.65
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	66.10	57.63	61.02	55.93	52.54	59.32	54.24	52.54	59.32	54.24	57.29	4.07
FrozenEBERT-bs4-ep100-WRONG	55.93	55.93	54.24	54.24	52.54	50.85	55.93	59.32	45.76	49.15	53.39	3.73
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	54.24	52.54	49.15	55.93	44.07	47.46	40.68	52.54	50.85	49.15	49.66	4.42
BERT-bs16+Dropout0.1	47.46	50.85	45.76	45.76	49.15	44.07	40.68	44.07	47.46	47.46	46.27	2.74
EmbraceBERT-bs16-WRONG	49.15	50.85	45.76	49.15	44.07	54.24	54.24	52.54	47.46	50.85	49.83	3.23
EmbraceBERT-bs16+Dropout0.1-WRONG	44.07	50.85	50.85	50.85	45.76	54.24	54.24	50.85	45.76	50.85	49.83	3.32
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16	47.46	52.54	52.54	52.54	40.68	50.85	55.93	49.15	47.46	52.54	50.17	4.03
CondensedEmbraceBERT-bs16+Dropout0.1	47.46	49.15	45.76	52.54	47.46	52.54	52.54	49.15	44.07	54.24	49.49	3.20
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	61.02	55.93	57.63	59.32	52.54	57.63	54.24	57.63	55.93	61.02	57.29	2.60
FrozenEBERT-bs16-ep100-WRONG	52.54	54.24	52.54	54.24	54.24	57.63	57.63	57.63	55.93	57.63	55.42	2.01
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	72.88	71.19	71.19	69.49	64.41	69.49	71.19	74.58	66.10	69.49	70.00	2.84
RoBERTa-bs4+Dropout0.1	62.71	74.58	64.41	66.10	69.49	62.71	72.88	76.27	69.49	67.80	68.64	4.56
EmbraceRoBERTa-bs4-WRONG	69.49	71.19	67.80	72.88	71.19	66.10	69.49	67.80	66.10	69.49	69.15	2.12
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	67.80	64.41	72.88	67.80	69.49	67.80	66.10	61.02	71.19	69.49	67.80	3.22
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4	69.49	72.88	67.80	69.49	74.58	69.49	69.49	72.88	67.80	71.19	70.51	2.17
CondensedEmbraceRoBERTa-bs4+Dropout0.1	69.49	66.10	66.10	67.80	62.71	64.41	67.80	72.88	71.19	66.10	67.46	2.92
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	74.58	74.58	66.10	69.49	64.41	69.49	71.19	71.19	69.49	72.88	70.34	3.15
FrozenERoBERTa-bs4-ep100-WRONG	69.49	69.49	67.80	72.88	74.58	74.58	69.49	67.80	76.27	71.19	71.36	2.88
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	59.32	55.93	61.02	55.93	59.32	54.24	59.32	59.32	54.24	61.02	57.97	2.49
RoBERTa-bs16+Dropout0.1	57.63	55.93	55.93	54.24	52.54	55.93	57.63	59.32	54.24	61.02	56.44	2.40
EmbraceRoBERTa-bs16-WRONG	49.15	45.76	44.07	50.85	45.76	50.85	49.15	40.68	40.68	49.15	46.61	3.65
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	47.46	44.07	35.59	44.07	40.68	52.54	54.24	40.68	50.85	50.85	46.10	5.76
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	52.54	50.85	40.68	52.54	44.07	50.85	50.85	37.29	52.54	49.15	48.14	5.21
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	71.19	64.41	69.49	69.49	66.10	66.10	72.88	72.88	67.80	66.10	68.64	2.86
FrozenERoBERTa-bs16-ep100-WRONG	69.49	71.19	71.19	67.80	72.88	66.10	67.80	62.71	71.19	71.19	69.15	2.92
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	66.10	55.93	62.71	59.32	54.24	61.02	54.24	55.93	59.32	55.93	58.47	3.73
BERT-bs4+Dropout0.1	62.71	57.63	57.63	61.02	55.93	57.63	55.93	55.93	61.02	55.93	58.14	2.40
EmbraceBERT-bs4	55.93	57.63	50.85	54.24	52.54	62.71	57.63	54.24	54.24	54.24	55.42	3.13
EmbraceBERT-bs4+Dropout0.1	54.24	54.24	47.46	55.93	57.63	59.32	59.32	59.32	52.54	55.93	55.59	3.54
EmbraceBERT-bs4-p_selfatt	54.24	54.24	59.32	57.63	54.24	49.15	59.32	55.93	55.93	57.63	55.76	2.88
EmbraceBERT-bs4-p_att_clsquery_weights	59.32	54.24	52.54	54.24	54.24	52.54	55.93	57.63	59.32	54.24	55.42	2.40
CondensedEmbraceBERT-bs4	57.63	57.63	52.54	55.93	49.15	59.32	54.24	55.93	54.24	55.93	55.25	2.75
CondensedEmbraceBERT-bs4+Dropout0.1	52.54	57.63	47.46	57.63	57.63	54.24	57.63	55.93	52.54	59.32	55.25	3.41
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	64.41	57.63	62.71	61.02	52.54	59.32	54.24	57.63	57.63	55.93	58.31	3.49
FrozenEBERT-bs4-ep100-WRONG	55.93	55.93	50.85	59.32	55.93	59.32	54.24	59.32	52.54	54.24	55.76	2.78
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	55.93	57.63	54.24	55.93	52.54	55.93	54.24	61.02	55.93	50.85	55.42	2.63
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	57.63	54.24	59.32	59.32	55.93	59.32	57.63	59.32	50.85	50.85	56.44	3.22
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	54.24	54.24	59.32	52.54	55.93	55.93	57.63	59.32	55.93	57.63	56.27	2.12
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	52.54	57.63	54.24	55.93	55.93	57.63	57.63	57.63	55.93	45.76	55.08	3.49
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	61.02	50.85	61.02	52.54	54.24	55.93	61.02	61.02	59.32	57.63	57.46	3.67
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	55.93	52.54	54.24	57.63	61.02	57.63	52.54	55.93	55.93	49.15	55.25	3.14
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	61.02	54.24	59.32	57.63	54.24	57.63	52.54	57.63	54.24	57.63	56.61	2.54
BERT-bs16+Dropout0.1	59.32	54.24	55.93	57.63	57.63	55.93	55.93	57.63	54.24	54.24	56.27	1.66
EmbraceBERT-bs16	54.24	57.63	55.93	52.54	52.54	61.02	57.63	57.63	52.54	61.02	56.27	3.11
EmbraceBERT-bs16+Dropout0.1	49.15	57.63	55.93	55.93	47.46	59.32	57.63	54.24	50.85	55.93	54.41	3.75
CondensedEmbraceBERT-bs16	52.54	57.63	55.93	52.54	55.93	59.32	57.63	57.63	54.24	59.32	56.27	2.37
CondensedEmbraceBERT-bs16+Dropout0.1	54.24	57.63	55.93	55.93	50.85	59.32	57.63	54.24	52.54	57.63	55.59	2.49
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	62.71	57.63	57.63	59.32	55.93	57.63	59.32	57.63	59.32	55.93	58.31	1.89
FrozenEBERT-bs16-ep100-WRONG	54.24	55.93	55.93	55.93	54.24	57.63	57.63	57.63	54.24	55.93	55.93	1.31
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	57.63	55.93	57.63	55.93	59.32	50.85	52.54	57.63	55.93	52.54	55.59	2.60
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	59.32	62.71	57.63	55.93	54.24	54.24	55.93	59.32	52.54	50.85	56.27	3.37
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	74.58	77.97	69.49	69.49	66.10	71.19	72.88	72.88	71.19	71.19	71.69	3.04
RoBERTa-bs4+Dropout0.1	66.10	72.88	64.41	64.41	72.88	64.41	74.58	74.58	74.58	66.10	69.49	4.48
EmbraceRoBERTa-bs4-WRONG	74.58	71.19	71.19	71.19	74.58	74.58	71.19	69.49	71.19	71.19	72.03	1.74
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	69.49	71.19	74.58	71.19	74.58	67.80	69.49	69.49	74.58	72.88	71.53	2.37
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1	69.49	71.19	71.19	66.10	74.58	71.19	69.49	71.19	71.19	69.49	70.51	2.03
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	74.58	76.27	69.49	72.88	69.49	71.19	72.88	72.88	74.58	71.19	72.54	2.12
FrozenERoBERTa-bs4-ep100-WRONG	71.19	74.58	72.88	72.88	72.88	74.58	69.49	71.19	69.49	72.88	72.20	1.73
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	71.19	72.88	71.19	66.10	69.49	66.10	71.19	72.88	71.19	72.88	70.51	2.42
RoBERTa-bs16+Dropout0.1	69.49	67.80	69.49	67.80	66.10	67.80	67.80	76.27	67.80	72.88	69.32	2.88
EmbraceRoBERTa-bs16-WRONG	67.80	69.49	69.49	66.10	72.88	66.10	67.80	61.02	69.49	74.58	68.47	3.57
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	72.88	69.49	76.27	67.80	67.80	66.10	67.80	66.10	71.19	74.58	70.00	3.39
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	71.19	71.19	67.80	66.10	69.49	67.80	71.19	69.49	71.19	72.88	69.83	1.98
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	74.58	71.19	71.19	66.10	69.49	64.41	71.19	76.27	72.88	74.58	71.19	3.56
FrozenERoBERTa-bs16-ep100-WRONG	67.80	71.19	71.19	69.49	72.88	67.80	69.49	69.49	72.88	76.27	70.85	2.49
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |59.32|50.85|59.32|61.02|50.85|57.63|55.93|59.32|57.63|59.32|57.12|3.39| | BERTwithAtt-bs8 |52.54|55.93|52.54|57.63|54.24|52.54|62.71|57.63|59.32|59.32|56.44|3.31| | BERTwithAttClsProjection-bs8 |57.63|50.85|55.93|55.93|54.24|57.63|54.24|52.54|54.24|59.32|55.25|2.42| | BERTwithProjection-bs8 |55.93|55.93|59.32|61.02|57.63|59.32|52.54|55.93|50.85|52.54|56.10|3.17| | BERTwithProjectionAtt-bs8 |55.93|54.24|61.02|59.32|55.93|59.32|57.63|55.93|57.63|55.93|57.29|1.98| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |45.76|55.93|59.32|59.32|55.93|57.63|57.63|59.32|50.85|55.93|55.76|4.11| | EmbraceBERT-bs8-p_att_clsquery_weights |55.93|54.24|50.85|55.93|57.63|52.54|54.24|57.63|52.54|50.85|54.24|2.40| | EmbraceBERTwithProj-bs8 |61.02|54.24|57.63|55.93|50.85|54.24|52.54|55.93|52.54|59.32|55.42|3.04| | EmbraceBERTwithProj-bs8-p_attclsqw |57.63|57.63|50.85|49.15|55.93|52.54|54.24|54.24|55.93|50.85|53.90|2.82| | EBERTconcatAtt-bs8 |55.93|54.24|52.54|55.93|61.02|52.54|55.93|52.54|50.85|54.24|54.58|2.71| | EBERTconcatAtt-bs8_attclsqw |50.85|59.32|52.54|61.02|54.24|57.63|52.54|55.93|59.32|59.32|56.27|3.37| | EBERTconcatAttwithProj-bs8 |57.63|52.54|59.32|50.85|52.54|49.15|52.54|57.63|55.93|55.93|54.41|3.17| | EBERTconcatAttwithProj-bs8_attclsqw |55.93|52.54|55.93|55.93|54.24|55.93|54.24|54.24|49.15|59.32|54.75|2.52| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |62.71|52.54|49.15|55.93|55.93|50.85|57.63|55.93|59.32|52.54|55.25|3.88| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |61.02|55.93|54.24|61.02|62.71|55.93|64.41|50.85|55.93|55.93|57.80|4.04| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |54.24|52.54|54.24|62.71|61.02|54.24|55.93|49.15|52.54|50.85|54.75|4.01| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |55.93|55.93|49.15|54.24|55.93|59.32|54.24|54.24|49.15|50.85|53.90|3.11| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |50.85|59.32|50.85|55.93|52.54|57.63|50.85|50.85|55.93|55.93|54.07|3.07| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |50.85|52.54|62.71|50.85|52.54|57.63|55.93|54.24|54.24|57.63|54.92|3.49| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |55.93|61.02|57.63|54.24|50.85|54.24|57.63|57.63|54.24|50.85|55.42|3.04| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |59.32|54.24|50.85|54.24|54.24|59.32|54.24|61.02|57.63|59.32|56.44|3.13|

gtts-witai

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	69.49	69.49	67.80	67.80	64.41	71.19	69.49	67.80	66.10	67.80	68.14	1.83
BERT-bs4+Dropout0.1	69.49	67.80	69.49	67.80	67.80	67.80	69.49	66.10	67.80	66.10	67.97	1.19
EmbraceBERT-bs4-WRONG	71.19	69.49	69.49	71.19	71.19	64.41	67.80	76.27	69.49	76.27	70.68	3.39
EmbraceBERT-bs4+Dropout0.1-WRONG	66.10	69.49	72.88	71.19	72.88	69.49	71.19	72.88	69.49	71.19	70.68	2.01
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4	69.49	69.49	71.19	69.49	71.19	61.02	69.49	71.19	69.49	72.88	69.49	3.03
CondensedEmbraceBERT-bs4+Dropout0.1	69.49	71.19	74.58	69.49	71.19	66.10	69.49	74.58	69.49	72.88	70.85	2.49
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	72.88	71.19	74.58	69.49	67.80	69.49	71.19	69.49	69.49	67.80	70.34	2.04
FrozenEBERT-bs4-ep100-WRONG	71.19	71.19	74.58	71.19	71.19	72.88	67.80	74.58	71.19	77.97	72.37	2.63
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	67.80	61.02	62.71	62.71	49.15	59.32	55.93	61.02	61.02	55.93	59.66	4.78
BERT-bs16+Dropout0.1	67.80	59.32	64.41	59.32	47.46	62.71	57.63	61.02	61.02	57.63	59.83	5.09
EmbraceBERT-bs16-WRONG	67.80	66.10	55.93	62.71	62.71	64.41	66.10	67.80	66.10	69.49	64.92	3.64
EmbraceBERT-bs16+Dropout0.1-WRONG	69.49	61.02	57.63	61.02	59.32	62.71	64.41	66.10	64.41	71.19	63.73	4.10
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16	67.80	69.49	55.93	61.02	64.41	62.71	69.49	62.71	66.10	69.49	64.92	4.22
CondensedEmbraceBERT-bs16+Dropout0.1	69.49	61.02	55.93	67.80	67.80	67.80	71.19	66.10	62.71	69.49	65.93	4.45
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	72.88	69.49	66.10	67.80	64.41	71.19	69.49	66.10	69.49	71.19	68.81	2.54
FrozenEBERT-bs16-ep100-WRONG	71.19	71.19	67.80	69.49	67.80	69.49	69.49	72.88	67.80	74.58	70.17	2.17
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	84.75	86.44	84.75	88.14	84.75	83.05	84.75	83.05	88.14	83.05	85.08	1.83
RoBERTa-bs4+Dropout0.1	83.05	83.05	88.14	83.05	89.83	84.75	83.05	84.75	81.36	83.05	84.41	2.49
EmbraceRoBERTa-bs4-WRONG	84.75	81.36	81.36	86.44	84.75	79.66	84.75	83.05	84.75	86.44	83.73	2.17
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	84.75	79.66	81.36	86.44	79.66	79.66	79.66	77.97	89.83	77.97	81.69	3.77
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4	84.75	84.75	72.88	81.36	77.97	86.44	84.75	83.05	84.75	86.44	82.71	4.07
CondensedEmbraceRoBERTa-bs4+Dropout0.1	88.14	81.36	83.05	81.36	83.05	88.14	79.66	84.75	88.14	79.66	83.73	3.23
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	86.44	88.14	89.83	88.14	86.44	84.75	86.44	84.75	89.83	86.44	87.12	1.73
FrozenERoBERTa-bs4-ep100-WRONG	84.75	88.14	84.75	83.05	88.14	89.83	88.14	89.83	83.05	88.14	86.78	2.49
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	67.80	71.19	77.97	67.80	64.41	67.80	67.80	71.19	71.19	66.10	69.32	3.59
RoBERTa-bs16+Dropout0.1	66.10	67.80	79.66	67.80	62.71	67.80	67.80	71.19	66.10	64.41	68.14	4.41
EmbraceRoBERTa-bs16-WRONG	61.02	59.32	61.02	61.02	55.93	66.10	59.32	59.32	59.32	67.80	61.02	3.30
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	61.02	59.32	61.02	59.32	55.93	66.10	61.02	61.02	62.71	66.10	61.36	2.92
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	62.71	59.32	61.02	62.71	57.63	64.41	61.02	59.32	61.02	66.10	61.53	2.40
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	83.05	84.75	83.05	89.83	88.14	84.75	84.75	81.36	83.05	79.66	84.24	2.84
FrozenERoBERTa-bs16-ep100-WRONG	91.53	86.44	83.05	88.14	84.75	79.66	83.05	77.97	88.14	84.75	84.75	3.87
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	74.58	71.19	76.27	69.49	67.80	69.49	71.19	72.88	72.88	71.19	71.69	2.40
BERT-bs4+Dropout0.1	72.88	67.80	72.88	67.80	67.80	72.88	74.58	69.49	72.88	66.10	70.51	2.86
EmbraceBERT-bs4	69.49	72.88	76.27	76.27	76.27	74.58	71.19	72.88	72.88	77.97	74.07	2.52
EmbraceBERT-bs4+Dropout0.1	67.80	71.19	71.19	74.58	77.97	71.19	67.80	67.80	76.27	71.19	71.69	3.39
EmbraceBERT-bs4-p_selfatt	71.19	71.19	71.19	74.58	67.80	69.49	74.58	72.88	67.80	76.27	71.69	2.74
EmbraceBERT-bs4-p_att_clsquery_weights	76.27	69.49	69.49	67.80	72.88	71.19	71.19	74.58	72.88	74.58	72.03	2.54
CondensedEmbraceBERT-bs4	72.88	71.19	71.19	72.88	71.19	71.19	69.49	71.19	71.19	76.27	71.86	1.73
CondensedEmbraceBERT-bs4+Dropout0.1	69.49	74.58	66.10	71.19	74.58	76.27	69.49	69.49	74.58	79.66	72.54	3.85
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	76.27	72.88	77.97	69.49	67.80	69.49	72.88	72.88	74.58	69.49	72.37	3.13
FrozenEBERT-bs4-ep100-WRONG	72.88	74.58	71.19	74.58	67.80	71.19	71.19	76.27	71.19	79.66	73.05	3.17
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	69.49	72.88	72.88	69.49	74.58	69.49	71.19	72.88	71.19	66.10	71.02	2.33
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	71.19	72.88	69.49	67.80	72.88	71.19	77.97	72.88	69.49	66.10	71.19	3.13
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	72.88	67.80	72.88	72.88	67.80	69.49	76.27	64.41	72.88	74.58	71.19	3.47
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	72.88	77.97	72.88	67.80	72.88	74.58	69.49	72.88	71.19	66.10	71.86	3.23
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	67.80	67.80	72.88	69.49	66.10	67.80	77.97	66.10	72.88	76.27	70.51	4.03
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	64.41	71.19	72.88	67.80	72.88	69.49	72.88	66.10	74.58	72.88	70.51	3.23
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	72.88	69.49	66.10	67.80	64.41	71.19	71.19	66.10	71.19	71.19	69.15	2.71
BERT-bs16+Dropout0.1	71.19	66.10	66.10	67.80	67.80	69.49	71.19	66.10	72.88	69.49	68.81	2.30
EmbraceBERT-bs16	69.49	69.49	67.80	69.49	72.88	67.80	71.19	74.58	71.19	79.66	71.36	3.43
EmbraceBERT-bs16+Dropout0.1	71.19	71.19	66.10	67.80	72.88	67.80	66.10	67.80	69.49	77.97	69.83	3.46
CondensedEmbraceBERT-bs16	71.19	71.19	66.10	67.80	72.88	67.80	71.19	74.58	71.19	79.66	71.36	3.67
CondensedEmbraceBERT-bs16+Dropout0.1	71.19	67.80	66.10	69.49	72.88	67.80	69.49	69.49	67.80	79.66	70.17	3.65
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	74.58	69.49	67.80	67.80	66.10	76.27	71.19	69.49	71.19	71.19	70.51	2.96
FrozenEBERT-bs16-ep100-WRONG	67.80	72.88	69.49	72.88	72.88	66.10	69.49	67.80	69.49	74.58	70.34	2.65
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	69.49	72.88	71.19	71.19	69.49	67.80	64.41	66.10	67.80	67.80	68.81	2.42
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	66.10	66.10	69.49	67.80	74.58	74.58	69.49	66.10	71.19	66.10	69.15	3.20
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	86.44	88.14	91.53	88.14	84.75	83.05	86.44	86.44	88.14	86.44	86.95	2.15
RoBERTa-bs4+Dropout0.1	86.44	84.75	88.14	84.75	89.83	84.75	88.14	86.44	84.75	83.05	86.10	1.98
EmbraceRoBERTa-bs4-WRONG	88.14	83.05	86.44	84.75	86.44	88.14	88.14	88.14	86.44	84.75	86.44	1.69
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	86.44	84.75	88.14	86.44	84.75	86.44	84.75	86.44	88.14	89.83	86.61	1.60
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1	88.14	88.14	86.44	84.75	84.75	86.44	84.75	88.14	86.44	81.36	85.93	2.01
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	86.44	86.44	84.75	86.44	88.14	86.44	89.83	81.36	83.05	84.75	85.76	2.30
FrozenERoBERTa-bs4-ep100-WRONG	86.44	86.44	84.75	86.44	84.75	88.14	84.75	88.14	84.75	86.44	86.10	1.27
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	83.05	84.75	83.05	88.14	88.14	84.75	84.75	81.36	83.05	83.05	84.41	2.12
RoBERTa-bs16+Dropout0.1	84.75	84.75	86.44	88.14	86.44	83.05	83.05	83.05	83.05	84.75	84.75	1.69
EmbraceRoBERTa-bs16-WRONG	91.53	86.44	86.44	89.83	86.44	86.44	86.44	83.05	89.83	88.14	87.46	2.30
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	89.83	86.44	86.44	86.44	89.83	86.44	88.14	86.44	88.14	86.44	87.46	1.36
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1	91.53	88.14	88.14	88.14	88.14	81.36	84.75	81.36	89.83	86.44	86.78	3.20
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	86.44	88.14	91.53	88.14	86.44	83.05	86.44	84.75	88.14	86.44	86.95	2.15
FrozenERoBERTa-bs16-ep100-WRONG	89.83	84.75	89.83	89.83	83.05	86.44	84.75	84.75	88.14	86.44	86.78	2.37
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |72.88|69.49|67.80|67.80|69.49|71.19|72.88|69.49|71.19|69.49|70.17|1.73| | BERTwithAtt-bs8 |66.10|76.27|67.80|67.80|67.80|76.27|67.80|72.88|69.49|72.88|70.51|3.57| | BERTwithAttClsProjection-bs8 |64.41|72.88|67.80|66.10|61.02|67.80|69.49|67.80|66.10|67.80|67.12|2.96| | BERTwithProjection-bs8 |59.32|72.88|76.27|69.49|71.19|72.88|62.71|71.19|61.02|71.19|68.81|5.42| | BERTwithProjectionAtt-bs8 |67.80|67.80|69.49|67.80|69.49|69.49|71.19|69.49|69.49|71.19|69.32|1.19| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |69.49|69.49|69.49|69.49|74.58|67.80|67.80|69.49|67.80|72.88|69.83|2.12| | EmbraceBERT-bs8-p_att_clsquery_weights |71.19|67.80|74.58|67.80|69.49|72.88|69.49|74.58|71.19|76.27|71.53|2.82| | EmbraceBERTwithProj-bs8 |64.41|71.19|67.80|72.88|69.49|69.49|67.80|67.80|66.10|71.19|68.81|2.42| | EmbraceBERTwithProj-bs8-p_attclsqw |67.80|66.10|69.49|67.80|67.80|67.80|69.49|64.41|67.80|66.10|67.46|1.48| | EBERTconcatAtt-bs8 |69.49|72.88|69.49|71.19|67.80|66.10|66.10|69.49|71.19|72.88|69.66|2.33| | EBERTconcatAtt-bs8_attclsqw |71.19|72.88|66.10|61.02|66.10|71.19|64.41|77.97|71.19|72.88|69.49|4.73| | EBERTconcatAttwithProj-bs8 |67.80|66.10|61.02|67.80|67.80|64.41|62.71|69.49|69.49|67.80|66.44|2.71| | EBERTconcatAttwithProj-bs8_attclsqw |66.10|69.49|67.80|66.10|71.19|62.71|67.80|71.19|67.80|72.88|68.31|2.84| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |71.19|57.63|69.49|71.19|72.88|76.27|71.19|64.41|76.27|67.80|69.83|5.30| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|72.88|69.49|76.27|69.49|76.27|71.19|67.80|69.49|67.80|70.85|3.11| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |71.19|67.80|66.10|71.19|71.19|67.80|69.49|67.80|71.19|67.80|69.15|1.83| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |69.49|67.80|66.10|71.19|67.80|69.49|69.49|67.80|66.10|64.41|67.97|1.93| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |69.49|69.49|72.88|69.49|64.41|71.19|67.80|67.80|74.58|66.10|69.32|2.88| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |71.19|74.58|66.10|71.19|71.19|74.58|69.49|74.58|71.19|71.19|71.53|2.49| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |69.49|61.02|71.19|72.88|67.80|69.49|69.49|67.80|64.41|62.71|67.63|3.59| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |71.19|66.10|67.80|71.19|67.80|71.19|69.49|62.71|69.49|71.19|68.81|2.65|

macsay-google

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	57.63	59.32	57.63	55.93	55.93	61.02	55.93	57.63	62.71	49.15	57.29	3.46
BERT-bs4+Dropout0.1	55.93	54.24	54.24	57.63	55.93	62.71	52.54	57.63	57.63	49.15	55.76	3.43
EmbraceBERT-bs4-WRONG	57.63	61.02	54.24	61.02	54.24	55.93	59.32	57.63	54.24	54.24	56.95	2.65
EmbraceBERT-bs4+Dropout0.1-WRONG	54.24	59.32	54.24	61.02	55.93	59.32	52.54	61.02	57.63	55.93	57.12	2.84
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4	59.32	57.63	55.93	62.71	59.32	61.02	55.93	61.02	54.24	54.24	58.14	2.84
CondensedEmbraceBERT-bs4+Dropout0.1
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	59.32	62.71	52.54	52.54	62.71	62.71	61.02	59.32	61.02	52.54	58.64	4.17
FrozenEBERT-bs4-ep100-WRONG	54.24	64.41	52.54	59.32	52.54	61.02	52.54	62.71	52.54	61.02	57.29	4.60
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	57.63	55.93	50.85	49.15	45.76	50.85	50.85	57.63	45.76	45.76	51.02	4.45
BERT-bs16+Dropout0.1	49.15	50.85	47.46	49.15	45.76	50.85	49.15	54.24	49.15	50.85	49.66	2.15
EmbraceBERT-bs16-WRONG	52.54	55.93	50.85	50.85	52.54	49.15	50.85	61.02	54.24	52.54	53.05	3.22
EmbraceBERT-bs16+Dropout0.1-WRONG	54.24	52.54	50.85	52.54	54.24	52.54	49.15	62.71	47.46	54.24	53.05	3.87
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16	50.85	57.63	47.46	54.24	49.15	47.46	52.54	61.02	52.54	55.93	52.88	4.21
CondensedEmbraceBERT-bs16+Dropout0.1
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	55.93	57.63	54.24	52.54	54.24	62.71	57.63	57.63	57.63	55.93	56.61	2.65
FrozenEBERT-bs16-ep100-WRONG	52.54	64.41	61.02	59.32	54.24	62.71	54.24	57.63	59.32	50.85	57.63	4.29
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	77.97	79.66	72.88	71.19	76.27	71.19	76.27	76.27	72.88	74.58	74.92	2.71
RoBERTa-bs4+Dropout0.1	71.19	74.58	72.88	77.97	71.19	69.49	74.58	76.27	76.27	71.19	73.56	2.65
EmbraceRoBERTa-bs4-WRONG	72.88	71.19	59.32	71.19	64.41	67.80	79.66	67.80	77.97	72.88	70.51	5.73
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	71.19	69.49	72.88	74.58	71.19	67.80	67.80	64.41	74.58	71.19	70.51	3.05
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4	67.80	69.49	59.32	72.88	64.41	72.88	71.19	71.19	74.58	77.97	70.17	5.04
CondensedEmbraceRoBERTa-bs4+Dropout0.1
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	76.27	77.97	74.58	74.58	74.58	69.49	79.66	77.97	74.58	74.58	75.42	2.65
FrozenERoBERTa-bs4-ep100-WRONG	74.58	71.19	74.58	74.58	74.58	71.19	72.88	74.58	76.27	71.19	73.56	1.73
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	61.02	64.41	74.58	61.02	57.63	61.02	62.71	66.10	61.02	59.32	62.88	4.51
RoBERTa-bs16+Dropout0.1	61.02	61.02	74.58	57.63	55.93	57.63	55.93	66.10	57.63	61.02	60.85	5.44
EmbraceRoBERTa-bs16-WRONG	54.24	49.15	42.37	54.24	40.68	54.24	54.24	47.46	42.37	54.24	49.32	5.44
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	50.85	50.85	47.46	49.15	38.98	54.24	55.93	42.37	54.24	55.93	50.00	5.43
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	76.27	71.19	74.58	69.49	72.88	74.58	72.88	72.88	71.19	71.19	72.71	1.93
FrozenERoBERTa-bs16-ep100-WRONG	72.88	74.58	69.49	72.88	69.49	71.19	79.66	71.19	76.27	69.49	72.71	3.17
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	59.32	62.71	55.93	52.54	62.71	61.02	57.63	61.02	61.02	54.24	58.81	3.39
BERT-bs4+Dropout0.1	61.02	66.10	55.93	57.63	59.32	59.32	54.24	61.02	59.32	55.93	58.98	3.20
EmbraceBERT-bs4	59.32	57.63	55.93	59.32	59.32	61.02	55.93	59.32	61.02	52.54	58.14	2.52
EmbraceBERT-bs4+Dropout0.1	61.02	62.71	54.24	59.32	62.71	64.41	59.32	55.93	61.02	57.63	59.83	3.04
EmbraceBERT-bs4-p_selfatt	57.63	59.32	61.02	52.54	55.93	50.85	55.93	54.24	57.63	64.41	56.95	3.81
EmbraceBERT-bs4-p_att_clsquery_weights	55.93	54.24	59.32	54.24	54.24	55.93	55.93	59.32	57.63	59.32	56.61	2.03
CondensedEmbraceBERT-bs4	54.24	57.63	59.32	61.02	62.71	62.71	55.93	57.63	59.32	55.93	58.64	2.75
CondensedEmbraceBERT-bs4+Dropout0.1
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	61.02	64.41	57.63	54.24	64.41	62.71	61.02	61.02	59.32	54.24	60.00	3.49
FrozenEBERT-bs4-ep100-WRONG	54.24	55.93	62.71	57.63	54.24	57.63	59.32	61.02	54.24	57.63	57.46	2.78
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	55.93	62.71	50.85	57.63	61.02	64.41	49.15	52.54	54.24	50.85	55.93	5.08
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	55.93	64.41	54.24	55.93	55.93	64.41	50.85	50.85	54.24	47.46	55.42	5.20
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	59.32	64.41	57.63	64.41	57.63	59.32	57.63	55.93	61.02	62.71	60.00	2.86
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	57.63	64.41	50.85	55.93	61.02	64.41	52.54	54.24	55.93	47.46	56.44	5.31
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	59.32	66.10	61.02	62.71	59.32	57.63	61.02	54.24	61.02	61.02	60.34	2.96
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	61.02	55.93	52.54	59.32	55.93	66.10	59.32	59.32	54.24	55.93	57.97	3.70
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	61.02	55.93	55.93	55.93	57.63	61.02	57.63	59.32	62.71	54.24	58.14	2.63
BERT-bs16+Dropout0.1	59.32	57.63	55.93	54.24	59.32	64.41	61.02	59.32	62.71	55.93	58.98	3.01
EmbraceBERT-bs16	61.02	59.32	55.93	59.32	54.24	61.02	50.85	55.93	61.02	52.54	57.12	3.56
EmbraceBERT-bs16+Dropout0.1	59.32	57.63	52.54	57.63	54.24	54.24	54.24	57.63	52.54	55.93	55.59	2.25
CondensedEmbraceBERT-bs16	61.02	61.02	55.93	59.32	59.32	61.02	50.85	55.93	55.93	54.24	57.46	3.26
CondensedEmbraceBERT-bs16+Dropout0.1
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	59.32	61.02	55.93	52.54	57.63	66.10	59.32	59.32	67.80	55.93	59.49	4.38
FrozenEBERT-bs16-ep100-WRONG	54.24	62.71	55.93	57.63	52.54	54.24	55.93	55.93	55.93	54.24	55.93	2.63
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	55.93	59.32	59.32	50.85	55.93	54.24	55.93	54.24	57.63	54.24	55.76	2.45
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	62.71	57.63	52.54	52.54	57.63	52.54	57.63	54.24	52.54	49.15	54.92	3.73
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	76.27	81.36	77.97	71.19	76.27	76.27	79.66	77.97	74.58	72.88	76.44	2.88
RoBERTa-bs4+Dropout0.1	72.88	77.97	71.19	72.88	72.88	74.58	81.36	77.97	76.27	79.66	75.76	3.22
EmbraceRoBERTa-bs4-WRONG	76.27	74.58	69.49	72.88	76.27	76.27	74.58	74.58	76.27	71.19	74.24	2.25
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	72.88	72.88	77.97	79.66	72.88	67.80	72.88	76.27	77.97	74.58	74.58	3.30
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	79.66	74.58	77.97	74.58	76.27	81.36	81.36	77.97	72.88	77.97	77.46	2.74
FrozenERoBERTa-bs4-ep100-WRONG	76.27	72.88	72.88	72.88	77.97	72.88	71.19	77.97	72.88	74.58	74.24	2.25
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	74.58	69.49	77.97	67.80	74.58	76.27	72.88	71.19	71.19	72.88	72.88	2.94
RoBERTa-bs16+Dropout0.1	71.19	79.66	71.19	66.10	76.27	72.88	72.88	74.58	69.49	74.58	72.88	3.56
EmbraceRoBERTa-bs16-WRONG	72.88	77.97	74.58	81.36	71.19	72.88	76.27	79.66	74.58	74.58	75.59	3.05
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	72.88	76.27	71.19	79.66	67.80	71.19	74.58	74.58	76.27	74.58	73.90	3.14
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	72.88	74.58	76.27	69.49	76.27	77.97	74.58	72.88	71.19	71.19	73.73	2.54
FrozenERoBERTa-bs16-ep100-WRONG	72.88	74.58	77.97	77.97	74.58	72.88	72.88	79.66	79.66	74.58	75.76	2.63
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |57.63|61.02|57.63|52.54|59.32|61.02|59.32|59.32|66.10|61.02|59.49|3.26| | BERTwithAtt-bs8 |55.93|62.71|55.93|62.71|52.54|62.71|55.93|64.41|52.54|64.41|58.98|4.60| | BERTwithAttClsProjection-bs8 |52.54|57.63|59.32|57.63|52.54|67.80|66.10|54.24|52.54|55.93|57.63|5.20| | BERTwithProjection-bs8 |57.63|55.93|64.41|50.85|54.24|66.10|54.24|59.32|50.85|61.02|57.46|5.00| | BERTwithProjectionAtt-bs8 |57.63|67.80|57.63|57.63|55.93|57.63|55.93|61.02|52.54|54.24|57.80|3.97| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |57.63|57.63|55.93|62.71|57.63|59.32|54.24|52.54|49.15|54.24|56.10|3.59| | EmbraceBERT-bs8-p_att_clsquery_weights |54.24|50.85|54.24|59.32|54.24|59.32|57.63|62.71|54.24|57.63|56.44|3.31| | EmbraceBERTwithProj-bs8 |55.93|57.63|54.24|66.10|59.32|59.32|61.02|62.71|61.02|62.71|60.00|3.32| | EmbraceBERTwithProj-bs8-p_attclsqw |52.54|57.63|55.93|55.93|55.93|61.02|64.41|59.32|52.54|52.54|56.78|3.73| | EBERTconcatAtt-bs8 |67.80|62.71|50.85|59.32|55.93|54.24|54.24|55.93|54.24|54.24|56.95|4.75| | EBERTconcatAtt-bs8_attclsqw |49.15|55.93|55.93|61.02|54.24|52.54|54.24|57.63|57.63|52.54|55.08|3.15| | EBERTconcatAttwithProj-bs8 |50.85|47.46|55.93|61.02|55.93|54.24|62.71|54.24|61.02|52.54|55.59|4.60| | EBERTconcatAttwithProj-bs8_attclsqw |50.85|55.93|55.93|61.02|54.24|59.32|55.93|64.41|59.32|61.02|57.80|3.75| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |49.15|57.63|59.32|50.85|55.93|54.24|57.63|57.63|61.02|54.24|55.76|3.51| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |54.24|59.32|54.24|52.54|52.54|66.10|64.41|52.54|52.54|49.15|55.76|5.33| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |61.02|61.02|62.71|54.24|57.63|55.93|55.93|55.93|57.63|64.41|58.64|3.23| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |62.71|52.54|50.85|64.41|61.02|55.93|59.32|57.63|62.71|61.02|58.81|4.29| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |55.93|52.54|52.54|59.32|62.71|61.02|54.24|52.54|54.24|57.63|56.27|3.54| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |55.93|57.63|57.63|57.63|57.63|55.93|57.63|52.54|59.32|55.93|56.78|1.74| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |57.63|55.93|55.93|59.32|57.63|61.02|55.93|57.63|55.93|59.32|57.63|1.69| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |57.63|54.24|57.63|54.24|59.32|57.63|61.02|57.63|59.32|54.24|57.29|2.25|

macsay-sphinx

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	37.29	40.68	49.15	49.15	38.98	40.68	40.68	44.07	47.46	40.68	42.88	4.09
BERT-bs4+Dropout0.1	33.90	42.37	42.37	49.15	37.29	44.07	38.98	42.37	50.85	42.37	42.37	4.79
EmbraceBERT-bs4-WRONG	38.98	35.59	33.90	42.37	28.81	45.76	42.37	40.68	37.29	42.37	38.81	4.76
EmbraceBERT-bs4+Dropout0.1-WRONG	37.29	38.98	33.90	42.37	35.59	44.07	44.07	40.68	35.59	54.24	40.68	5.67
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4	42.37	44.07	28.81	35.59	32.20	44.07	44.07	38.98	40.68	44.07	39.49	5.25
CondensedEmbraceBERT-bs4+Dropout0.1
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	40.68	45.76	57.63	52.54	47.46	47.46	47.46	49.15	55.93	47.46	49.15	4.73
FrozenEBERT-bs4-ep100-WRONG	45.76	38.98	38.98	42.37	37.29	40.68	42.37	50.85	40.68	50.85	42.88	4.55
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	35.59	35.59	38.98	32.20	35.59	28.81	30.51	45.76	40.68	33.90	35.76	4.76
BERT-bs16+Dropout0.1	33.90	38.98	35.59	38.98	25.42	27.12	32.20	38.98	38.98	32.20	34.24	4.78
EmbraceBERT-bs16-WRONG	35.59	38.98	30.51	35.59	32.20	37.29	42.37	38.98	32.20	38.98	36.27	3.57
EmbraceBERT-bs16+Dropout0.1-WRONG	37.29	40.68	30.51	35.59	28.81	40.68	42.37	38.98	25.42	37.29	35.76	5.38
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16	35.59	45.76	37.29	38.98	30.51	37.29	40.68	38.98	27.12	37.29	36.95	4.90
CondensedEmbraceBERT-bs16+Dropout0.1
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	35.59	44.07	47.46	55.93	37.29	42.37	33.90	44.07	52.54	44.07	43.73	6.69
FrozenEBERT-bs16-ep100-WRONG	38.98	40.68	30.51	42.37	38.98	47.46	38.98	45.76	33.90	44.07	40.17	4.92
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	50.85	47.46	54.24	47.46	49.15	44.07	54.24	52.54	42.37	49.15	49.15	3.79
RoBERTa-bs4+Dropout0.1	49.15	49.15	49.15	45.76	44.07	44.07	45.76	49.15	45.76	50.85	47.29	2.33
EmbraceRoBERTa-bs4-WRONG	47.46	45.76	54.24	50.85	49.15	47.46	44.07	47.46	42.37	47.46	47.63	3.17
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	50.85	45.76	55.93	49.15	45.76	47.46	45.76	42.37	42.37	40.68	46.61	4.30
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4	50.85	52.54	49.15	55.93	44.07	47.46	45.76	50.85	47.46	50.85	49.49	3.29
CondensedEmbraceRoBERTa-bs4+Dropout0.1
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	50.85	54.24	61.02	47.46	50.85	47.46	55.93	57.63	49.15	50.85	52.54	4.29
FrozenERoBERTa-bs4-ep100-WRONG	52.54	50.85	59.32	55.93	52.54	52.54	47.46	52.54	50.85	52.54	52.71	2.98
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	40.68	42.37	52.54	35.59	45.76	37.29	40.68	44.07	42.37	42.37	42.37	4.42
RoBERTa-bs16+Dropout0.1	42.37	44.07	38.98	35.59	40.68	38.98	38.98	38.98	37.29	37.29	39.32	2.37
EmbraceRoBERTa-bs16-WRONG	27.12	35.59	37.29	37.29	30.51	35.59	37.29	23.73	33.90	35.59	33.39	4.49
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	32.20	35.59	40.68	33.90	32.20	33.90	32.20	30.51	35.59	32.20	33.90	2.73
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	52.54	49.15	54.24	44.07	49.15	47.46	42.37	55.93	47.46	57.63	50.00	4.75
FrozenERoBERTa-bs16-ep100-WRONG	52.54	49.15	49.15	47.46	40.68	57.63	49.15	50.85	50.85	44.07	49.15	4.35
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	42.37	49.15	57.63	54.24	49.15	44.07	45.76	45.76	55.93	47.46	49.15	4.91
BERT-bs4+Dropout0.1	42.37	49.15	49.15	45.76	40.68	45.76	47.46	44.07	59.32	59.32	48.31	6.08
EmbraceBERT-bs4	44.07	38.98	28.81	44.07	35.59	49.15	42.37	49.15	40.68	55.93	42.88	7.23
EmbraceBERT-bs4+Dropout0.1	44.07	42.37	35.59	45.76	40.68	40.68	47.46	40.68	37.29	57.63	43.22	5.88
EmbraceBERT-bs4-p_selfatt	45.76	42.37	47.46	47.46	42.37	44.07	44.07	44.07	42.37	45.76	44.58	1.86
EmbraceBERT-bs4-p_att_clsquery_weights	45.76	45.76	38.98	49.15	47.46	49.15	47.46	47.46	52.54	50.85	47.46	3.47
CondensedEmbraceBERT-bs4	50.85	45.76	28.81	44.07	44.07	47.46	44.07	47.46	44.07	49.15	44.58	5.73
CondensedEmbraceBERT-bs4+Dropout0.1
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	44.07	45.76	57.63	54.24	54.24	49.15	52.54	45.76	57.63	49.15	51.02	4.70
FrozenEBERT-bs4-ep100-WRONG	45.76	40.68	42.37	47.46	40.68	49.15	42.37	49.15	40.68	54.24	45.25	4.42
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	50.85	44.07	42.37	52.54	44.07	45.76	40.68	42.37	44.07	49.15	45.59	3.75
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	42.37	42.37	45.76	38.98	47.46	37.29	44.07	50.85	40.68	50.85	44.07	4.42
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	55.93	37.29	54.24	50.85	40.68	45.76	45.76	38.98	44.07	37.29	45.08	6.44
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	49.15	40.68	42.37	49.15	45.76	44.07	42.37	47.46	44.07	49.15	45.42	3.01
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	52.54	40.68	61.02	47.46	38.98	44.07	45.76	40.68	44.07	40.68	45.59	6.41
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	42.37	57.63	50.85	47.46	42.37	35.59	38.98	59.32	42.37	42.37	45.93	7.41
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	38.98	42.37	47.46	54.24	40.68	45.76	32.20	45.76	57.63	47.46	45.25	6.95
BERT-bs16+Dropout0.1	37.29	37.29	40.68	54.24	33.90	49.15	35.59	47.46	55.93	54.24	44.58	8.13
EmbraceBERT-bs16	33.90	47.46	33.90	37.29	38.98	44.07	45.76	37.29	35.59	49.15	40.34	5.46
EmbraceBERT-bs16+Dropout0.1	35.59	42.37	33.90	33.90	33.90	44.07	44.07	45.76	42.37	50.85	40.68	5.67
CondensedEmbraceBERT-bs16	32.20	45.76	32.20	40.68	40.68	45.76	42.37	40.68	37.29	47.46	40.51	5.05
CondensedEmbraceBERT-bs16+Dropout0.1
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	42.37	45.76	54.24	54.24	44.07	49.15	44.07	45.76	55.93	49.15	48.47	4.62
FrozenEBERT-bs16-ep100-WRONG	35.59	42.37	23.73	49.15	40.68	40.68	52.54	44.07	45.76	54.24	42.88	8.37
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	40.68	45.76	44.07	47.46	47.46	45.76	38.98	49.15	42.37	47.46	44.92	3.15
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	40.68	47.46	38.98	42.37	49.15	38.98	38.98	47.46	50.85	40.68	43.56	4.42
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	54.24	52.54	54.24	47.46	50.85	49.15	49.15	61.02	50.85	52.54	52.20	3.62
RoBERTa-bs4+Dropout0.1	49.15	54.24	52.54	45.76	47.46	49.15	50.85	54.24	50.85	57.63	51.19	3.37
EmbraceRoBERTa-bs4-WRONG	52.54	52.54	59.32	55.93	52.54	55.93	47.46	54.24	49.15	55.93	53.56	3.32
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	55.93	59.32	55.93	50.85	54.24	52.54	57.63	55.93	49.15	52.54	54.41	2.98
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	57.63	57.63	54.24	45.76	54.24	49.15	54.24	57.63	59.32	57.63	54.75	4.09
FrozenERoBERTa-bs4-ep100-WRONG	55.93	54.24	59.32	59.32	50.85	54.24	49.15	54.24	52.54	59.32	54.92	3.41
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	55.93	50.85	55.93	45.76	50.85	52.54	47.46	54.24	55.93	55.93	52.54	3.56
RoBERTa-bs16+Dropout0.1	57.63	50.85	61.02	47.46	52.54	49.15	61.02	49.15	47.46	54.24	53.05	4.97
EmbraceRoBERTa-bs16-WRONG	55.93	50.85	54.24	52.54	47.46	59.32	55.93	50.85	45.76	47.46	52.03	4.16
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	55.93	55.93	59.32	55.93	47.46	55.93	50.85	50.85	54.24	52.54	53.90	3.29
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	54.24	49.15	61.02	47.46	50.85	52.54	49.15	57.63	54.24	55.93	53.22	4.03
FrozenERoBERTa-bs16-ep100-WRONG	61.02	57.63	59.32	52.54	50.85	59.32	55.93	54.24	50.85	47.46	54.92	4.23
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |38.98|50.85|55.93|54.24|40.68|44.07|45.76|44.07|52.54|52.54|47.97|5.67| | BERTwithAtt-bs8 |44.07|47.46|45.76|52.54|45.76|44.07|49.15|47.46|45.76|52.54|47.46|2.94| | BERTwithAttClsProjection-bs8 |42.37|42.37|44.07|38.98|42.37|47.46|45.76|40.68|40.68|44.07|42.88|2.40| | BERTwithProjection-bs8 |44.07|45.76|44.07|40.68|50.85|47.46|49.15|45.76|35.59|40.68|44.41|4.27| | BERTwithProjectionAtt-bs8 |49.15|50.85|44.07|45.76|38.98|45.76|47.46|40.68|45.76|38.98|44.75|3.88| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |40.68|42.37|40.68|42.37|44.07|40.68|47.46|42.37|44.07|49.15|43.39|2.75| | EmbraceBERT-bs8-p_att_clsquery_weights |47.46|44.07|45.76|44.07|50.85|40.68|42.37|49.15|44.07|38.98|44.75|3.49| | EmbraceBERTwithProj-bs8 |42.37|44.07|30.51|47.46|38.98|45.76|44.07|45.76|40.68|40.68|42.03|4.60| | EmbraceBERTwithProj-bs8-p_attclsqw |52.54|45.76|49.15|42.37|47.46|47.46|38.98|44.07|44.07|47.46|45.93|3.59| | EBERTconcatAtt-bs8 |44.07|38.98|37.29|54.24|40.68|44.07|42.37|40.68|42.37|47.46|43.22|4.56| | EBERTconcatAtt-bs8_attclsqw |52.54|37.29|38.98|45.76|52.54|49.15|38.98|37.29|38.98|44.07|43.56|5.82| | EBERTconcatAttwithProj-bs8 |54.24|37.29|49.15|44.07|38.98|42.37|44.07|44.07|42.37|49.15|44.58|4.80| | EBERTconcatAttwithProj-bs8_attclsqw |44.07|42.37|38.98|40.68|42.37|47.46|38.98|42.37|52.54|45.76|43.56|3.94| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |38.98|54.24|47.46|52.54|52.54|37.29|42.37|55.93|44.07|47.46|47.29|6.18| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |42.37|45.76|44.07|45.76|42.37|38.98|52.54|37.29|38.98|40.68|42.88|4.22| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |37.29|40.68|52.54|40.68|49.15|44.07|45.76|37.29|45.76|40.68|43.39|4.75| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |44.07|44.07|38.98|40.68|44.07|42.37|45.76|40.68|49.15|45.76|43.56|2.84| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |38.98|38.98|44.07|49.15|40.68|52.54|52.54|44.07|35.59|45.76|44.24|5.54| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |42.37|45.76|50.85|45.76|45.76|47.46|38.98|44.07|35.59|42.37|43.90|4.11| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |40.68|40.68|42.37|37.29|37.29|37.29|42.37|37.29|44.07|40.68|40.00|2.42| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |49.15|37.29|50.85|45.76|35.59|52.54|49.15|54.24|45.76|45.76|46.61|5.79|

macsay-witai

• Epoch 30

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	67.80	69.49	66.10	64.41	66.10	64.41	64.41	64.41	74.58	69.49	67.12	3.14
BERT-bs4+Dropout0.1	64.41	64.41	64.41	66.10	67.80	66.10	59.32	62.71	67.80	66.10	64.92	2.40
EmbraceBERT-bs4-WRONG	72.88	62.71	71.19	66.10	64.41	66.10	67.80	64.41	67.80	61.02	66.44	3.46
EmbraceBERT-bs4+Dropout0.1-WRONG	72.88	61.02	59.32	66.10	55.93	62.71	71.19	61.02	66.10	79.66	65.59	6.82
EmbraceBERT-bs4+Dropout0.3-WRONG
CondensedEmbraceBERT-bs4	71.19	64.41	67.80	64.41	61.02	64.41	69.49	66.10	66.10	69.49	66.44	2.92
CondensedEmbraceBERT-bs4+Dropout0.1
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	74.58	69.49	74.58	67.80	71.19	67.80	71.19	71.19	76.27	71.19	71.53	2.71
FrozenEBERT-bs4-ep100-WRONG	76.27	69.49	71.19	64.41	71.19	67.80	71.19	71.19	67.80	71.19	70.17	2.96
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	55.93	61.02	61.02	52.54	49.15	52.54	44.07	59.32	52.54	52.54	54.07	5.11
BERT-bs16+Dropout0.1	54.24	62.71	57.63	54.24	45.76	50.85	45.76	55.93	54.24	50.85	53.22	4.92
EmbraceBERT-bs16-WRONG	50.85	54.24	50.85	55.93	55.93	55.93	64.41	62.71	47.46	55.93	55.42	4.92
EmbraceBERT-bs16+Dropout0.1-WRONG	55.93	47.46	52.54	55.93	54.24	59.32	62.71	59.32	49.15	54.24	55.08	4.44
EmbraceBERT-bs16+Dropout0.3-WRONG
CondensedEmbraceBERT-bs16	62.71	57.63	50.85	54.24	54.24	55.93	64.41	61.02	52.54	55.93	56.95	4.23
CondensedEmbraceBERT-bs16+Dropout0.1
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	64.41	71.19	66.10	67.80	61.02	62.71	62.71	66.10	67.80	67.80	65.76	2.92
FrozenEBERT-bs16-ep100-WRONG	72.88	69.49	61.02	61.02	67.80	62.71	71.19	67.80	59.32	64.41	65.76	4.47
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	84.75	83.05	81.36	83.05	83.05	81.36	81.36	81.36	83.05	76.27	81.86	2.15
RoBERTa-bs4+Dropout0.1	79.66	81.36	81.36	81.36	81.36	77.97	79.66	81.36	84.75	83.05	81.19	1.77
EmbraceRoBERTa-bs4-WRONG	84.75	72.88	74.58	81.36	76.27	84.75	79.66	79.66	79.66	79.66	79.32	3.70
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	74.58	74.58	76.27	76.27	72.88	76.27	72.88	89.83	81.36	81.36	77.63	4.96
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4	83.05	77.97	72.88	79.66	76.27	81.36	77.97	81.36	81.36	81.36	79.32	2.92
CondensedEmbraceRoBERTa-bs4+Dropout0.1
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	83.05	86.44	81.36	81.36	83.05	84.75	79.66	81.36	83.05	81.36	82.54	1.86
FrozenERoBERTa-bs4-ep100-WRONG	81.36	79.66	88.14	77.97	81.36	83.05	88.14	84.75	84.75	83.05	83.22	3.17
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	59.32	67.80	67.80	64.41	57.63	62.71	64.41	67.80	64.41	64.41	64.07	3.29
RoBERTa-bs16+Dropout0.1	57.63	62.71	66.10	61.02	57.63	64.41	61.02	64.41	62.71	62.71	62.03	2.65
EmbraceRoBERTa-bs16-WRONG	61.02	57.63	54.24	59.32	55.93	61.02	57.63	55.93	50.85	61.02	57.46	3.17
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	61.02	55.93	59.32	44.07	54.24	61.02	59.32	57.63	54.24	59.32	56.61	4.81
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	84.75	83.05	83.05	77.97	76.27	77.97	76.27	77.97	81.36	79.66	79.83	2.88
FrozenERoBERTa-bs16-ep100-WRONG	86.44	79.66	74.58	81.36	79.66	89.83	79.66	84.75	89.83	81.36	82.71	4.66
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	71.19	71.19	76.27	69.49	72.88	66.10	69.49	71.19	76.27	71.19	71.53	2.92
BERT-bs4+Dropout0.1	67.80	69.49	72.88	72.88	69.49	67.80	74.58	69.49	74.58	74.58	71.36	2.67
EmbraceBERT-bs4	74.58	67.80	67.80	67.80	69.49	72.88	72.88	66.10	69.49	76.27	70.51	3.23
EmbraceBERT-bs4+Dropout0.1	79.66	71.19	67.80	66.10	74.58	76.27	71.19	66.10	74.58	62.71	71.02	5.05
EmbraceBERT-bs4-p_selfatt	71.19	67.80	74.58	71.19	62.71	67.80	72.88	54.24	67.80	69.49	67.97	5.54
EmbraceBERT-bs4-p_att_clsquery_weights	66.10	67.80	64.41	66.10	66.10	72.88	66.10	76.27	64.41	71.19	68.14	3.77
CondensedEmbraceBERT-bs4	77.97	72.88	71.19	71.19	69.49	72.88	74.58	66.10	72.88	81.36	73.05	4.04
CondensedEmbraceBERT-bs4+Dropout0.1
CondensedEmbraceBERT-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs4-ep100	71.19	69.49	76.27	69.49	74.58	66.10	76.27	81.36	79.66	71.19	73.56	4.62
FrozenEBERT-bs4-ep100-WRONG	76.27	69.49	76.27	69.49	76.27	69.49	74.58	62.71	71.19	77.97	72.37	4.49
FrozenEBERT-bs4-ep100+Dropout0.1
FrozenCEBERT-bs4-ep100
FrozenCEBERT-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	67.80	77.97	67.80	64.41	74.58	72.88	64.41	72.88	64.41	69.49	69.66	4.51
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	72.88	71.19	66.10	67.80	64.41	81.36	62.71	66.10	64.41	66.10	68.31	5.25
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	61.02	72.88	59.32	74.58	71.19	71.19	67.80	64.41	69.49	67.80	67.97	4.76
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	76.27	76.27	74.58	66.10	71.19	76.27	69.49	74.58	64.41	64.41	71.36	4.70
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	69.49	71.19	64.41	76.27	69.49	69.49	69.49	66.10	66.10	69.49	69.15	3.11
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	62.71	67.80	64.41	67.80	72.88	69.49	76.27	66.10	72.88	69.49	68.98	3.94
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	67.80	72.88	69.49	72.88	66.10	62.71	64.41	67.80	67.80	69.49	68.14	3.11
BERT-bs16+Dropout0.1	69.49	69.49	71.19	67.80	69.49	64.41	66.10	66.10	69.49	71.19	68.47	2.17
EmbraceBERT-bs16	66.10	66.10	62.71	62.71	66.10	72.88	67.80	69.49	72.88	72.88	67.97	3.75
EmbraceBERT-bs16+Dropout0.1	71.19	62.71	59.32	62.71	67.80	67.80	69.49	66.10	62.71	72.88	66.27	4.11
CondensedEmbraceBERT-bs16	72.88	61.02	62.71	61.02	64.41	67.80	72.88	66.10	66.10	71.19	66.61	4.29
CondensedEmbraceBERT-bs16+Dropout0.1
CondensedEmbraceBERT-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenBERT-bs16-ep100	69.49	74.58	69.49	72.88	69.49	64.41	72.88	71.19	71.19	71.19	70.68	2.63
FrozenEBERT-bs16-ep100-WRONG	71.19	67.80	66.10	69.49	69.49	67.80	71.19	71.19	62.71	72.88	68.98	2.84
FrozenEBERT-bs16-ep100+Dropout0.1
FrozenCEBERT-bs16-ep100
FrozenCEBERT-bs16-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	66.10	67.80	72.88	64.41	62.71	66.10	61.02	59.32	67.80	64.41	65.25	3.65
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10
EmbraceBERT-bs16-p_multiheadatt_bertKeyValQuery	57.63	64.41	66.10	64.41	59.32	67.80	67.80	67.80	67.80	59.32	64.24	3.82
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs4	84.75	83.05	84.75	81.36	81.36	81.36	83.05	83.05	86.44	81.36	83.05	1.69
RoBERTa-bs4+Dropout0.1	86.44	88.14	79.66	81.36	83.05	83.05	77.97	84.75	88.14	83.05	83.56	3.22
EmbraceRoBERTa-bs4-WRONG	86.44	79.66	84.75	77.97	84.75	88.14	86.44	77.97	86.44	83.05	83.56	3.56
EmbraceRoBERTa-bs4+Dropout0.1-WRONG	83.05	79.66	81.36	81.36	84.75	84.75	84.75	89.83	84.75	88.14	84.24	2.94
EmbraceRoBERTa-bs4+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs4
CondensedEmbraceRoBERTa-bs4+Dropout0.1
CondensedEmbraceRoBERTa-bs4+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs4-ep100	84.75	86.44	88.14	79.66	83.05	81.36	81.36	83.05	86.44	84.75	83.90	2.54
FrozenERoBERTa-bs4-ep100-WRONG	86.44	79.66	86.44	79.66	86.44	84.75	89.83	84.75	86.44	84.75	84.92	2.98
FrozenERoBERTa-bs4-ep100+Dropout0.1
FrozenCERoBERTa-bs4-ep100
FrozenCERoBERTa-bs4-ep100+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
RoBERTa-bs16	84.75	79.66	83.05	77.97	79.66	77.97	79.66	77.97	79.66	79.66	80.00	2.12
RoBERTa-bs16+Dropout0.1	79.66	81.36	79.66	74.58	84.75	79.66	74.58	76.27	81.36	83.05	79.49	3.26
EmbraceRoBERTa-bs16-WRONG	86.44	77.97	81.36	79.66	81.36	89.83	79.66	84.75	88.14	83.05	83.22	3.75
EmbraceRoBERTa-bs16+Dropout0.1-WRONG	77.97	81.36	81.36	77.97	79.66	88.14	83.05	83.05	88.14	83.05	82.37	3.41
EmbraceRoBERTa-bs16+Dropout0.3-WRONG
CondensedEmbraceRoBERTa-bs16
CondensedEmbraceRoBERTa-bs16+Dropout0.1
CondensedEmbraceRoBERTa-bs16+Dropout0.3
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
FrozenRoBERTa-bs16-ep100	84.75	84.75	83.05	79.66	81.36	79.66	83.05	79.66	81.36	77.97	81.53	2.20
FrozenERoBERTa-bs16-ep100-WRONG	84.75	79.66	83.05	77.97	81.36	91.53	81.36	86.44	89.83	83.05	83.90	4.10
FrozenERoBERTa-bs16-ep100+Dropout0.1
FrozenCERoBERTa-bs16-ep100
FrozenCERoBERTa-bs16-ep100+Dropout0.1

• Epoch 100 - bs8 | Model |Run1 |Run2 |Run3 |Run4 |Run5 |Run6 |Run7 |Run8 |Run9 |Run10|Mean |Std | | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-bs8 |71.19|74.58|72.88|74.58|66.10|61.02|74.58|69.49|72.88|71.19|70.85|4.14| | BERTwithAtt-bs8 |72.88|64.41|69.49|72.88|66.10|72.88|66.10|64.41|69.49|71.19|68.98|3.31| | BERTwithAttClsProjection-bs8 |54.24|59.32|62.71|55.93|66.10|72.88|64.41|57.63|62.71|61.02|61.69|5.15| | BERTwithProjection-bs8 |55.93|62.71|72.88|64.41|71.19|71.19|55.93|66.10|64.41|64.41|64.92|5.57| | BERTwithProjectionAtt-bs8 |64.41|77.97|64.41|72.88|69.49|74.58|64.41|67.80|69.49|61.02|68.64|5.04| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8 |69.49|66.10|69.49|66.10|66.10|69.49|74.58|69.49|72.88|71.19|69.49|2.73| | EmbraceBERT-bs8-p_att_clsquery_weights |67.80|57.63|72.88|64.41|67.80|67.80|66.10|74.58|61.02|77.97|67.80|5.82| | EmbraceBERTwithProj-bs8 |62.71|66.10|67.80|67.80|59.32|71.19|66.10|67.80|67.80|69.49|66.61|3.22| | EmbraceBERTwithProj-bs8-p_attclsqw |55.93|62.71|67.80|67.80|64.41|67.80|62.71|59.32|62.71|64.41|63.56|3.65| | EBERTconcatAtt-bs8 |69.49|64.41|69.49|71.19|61.02|61.02|72.88|59.32|61.02|47.46|63.73|7.16| | EBERTconcatAtt-bs8_attclsqw |72.88|66.10|62.71|57.63|57.63|64.41|69.49|69.49|66.10|66.10|65.25|4.69| | EBERTconcatAttwithProj-bs8 |66.10|61.02|62.71|62.71|61.02|66.10|62.71|61.02|64.41|66.10|63.39|2.03| | EBERTconcatAttwithProj-bs8_attclsqw |61.02|67.80|67.80|55.93|55.93|66.10|66.10|64.41|67.80|67.80|64.07|4.54| | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery |59.32|64.41|62.71|62.71|71.19|66.10|69.49|59.32|64.41|64.41|64.41|3.64| | EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |66.10|69.49|67.80|61.02|67.80|76.27|71.19|69.49|69.49|72.88|69.15|3.85| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery |66.10|57.63|69.49|64.41|71.19|61.02|71.19|61.02|69.49|67.80|65.93|4.51| | EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |67.80|66.10|57.63|67.80|64.41|64.41|62.71|62.71|62.71|62.71|63.90|2.84| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery |67.80|59.32|66.10|62.71|61.02|69.49|61.02|71.19|64.41|72.88|65.59|4.42| | EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |61.02|72.88|64.41|69.49|67.80|64.41|64.41|66.10|69.49|66.10|66.61|3.22| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery |59.32|47.46|67.80|64.41|62.71|67.80|66.10|71.19|67.80|62.71|63.73|6.31| | EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw |66.10|66.10|66.10|59.32|61.02|72.88|64.41|59.32|66.10|69.49|65.08|4.10|

STT Error: trained and tested with INCOMPLETE and COMPLETE data

gtts-google

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	64.41	64.41	62.71	62.71	66.10	61.02	69.49	61.02	64.41	64.41	64.07	2.37
BERT-bs4+Dropout0.1	66.10	62.71	61.02	66.10	66.10	64.41	69.49	67.80	62.71	66.10	65.25	2.43
EmbraceBERT-bs4	62.71	66.10	62.71	62.71	59.32	62.71	57.63	66.10	66.10	59.32	62.54	2.88
EmbraceBERT-bs4+Dropout0.1	62.71	61.02	62.71	62.71	67.80	69.49	61.02	62.71	64.41	64.41	63.90	2.63
EmbraceBERT-bs4-p_selfatt	62.71	66.10	67.80	67.80	66.10	64.41	67.80	61.02	64.41	61.02	64.92	2.52
EmbraceBERT-bs4-p_att_clsquery_weights	66.10	66.10	61.02	69.49	61.02	71.19	66.10	62.71	62.71	64.41	65.08	3.23
CondensedEmbraceBERT-bs4
CondensedEmbraceBERT-bs4+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0	61.02	62.71	54.24	64.41	57.63	61.02	62.71	64.41	62.71	59.32	61.02	3.03
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	57.63	59.32	67.80	57.63	55.93	59.32	62.71	62.71	57.63	52.54	59.32	4.01
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	62.71	62.71	62.71	67.80	57.63	62.71	64.41	59.32	59.32	47.46	60.68	5.19
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	64.41	55.93	64.41	64.41	54.24	62.71	62.71	61.02	66.10	64.41	62.03	3.73
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	62.71	57.63	62.71	55.93	55.93	61.02	59.32	64.41	54.24	54.24	58.81	3.56
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	66.10	61.02	64.41	64.41	54.24	62.71	59.32	57.63	64.41	61.02	61.53	3.48
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	62.71	67.80	59.32	69.49	66.10	61.02	55.93	67.80	64.41	61.02	63.56	4.10
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	67.80	61.02	67.80	64.41	62.71	64.41	66.10	62.71	66.10	66.10	64.92	2.15
BERT-bs16+Dropout0.1	64.41	62.71	64.41	64.41	66.10	64.41	64.41	64.41	66.10	67.80	64.92	1.32
EmbraceBERT-bs16	66.10	64.41	57.63	61.02	59.32	66.10	64.41	62.71	62.71	61.02	62.54	2.67
EmbraceBERT-bs16+Dropout0.1	62.71	62.71	57.63	67.80	62.71	62.71	64.41	62.71	66.10	66.10	63.56	2.65
CondensedEmbraceBERT-bs16
CondensedEmbraceBERT-bs16+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3	59.32	64.41	59.32	62.71	57.63	55.93	57.63	59.32	61.02	57.63	59.49	2.45
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	59.32	64.41	59.32	59.32	55.93	66.10	59.32	52.54	55.93	62.71	59.49	3.90
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10	55.93	66.10	59.32	61.02	55.93	61.02	62.71	55.93	62.71	61.02	60.17	3.24
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3	61.02	64.41	57.63	62.71	59.32	55.93	59.32	59.32	55.93	52.54	58.81	3.31
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10	55.93	64.41	59.32	64.41	55.93	62.71	57.63	59.32	62.71	64.41	60.68	3.29

• Epoch 100 - bs8

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	62.71	61.02	66.10	66.10	67.80	61.02	69.49	61.02	64.41	62.71	64.24	2.88
BERTwithAtt-bs8	66.10	64.41	61.02	67.80	64.41	62.71	61.02	64.41	64.41	62.71	63.90	2.01
BERTwithAttClsProjection-bs8	62.71	66.10	61.02	66.10	67.80	69.49	66.10	57.63	67.80	67.80	65.25	3.49
BERTwithProjection-bs8	59.32	59.32	69.49	62.71	64.41	69.49	54.24	66.10	61.02	66.10	63.22	4.61
BERTwithProjectionAtt-bs8	62.71	61.02	71.19	64.41	66.10	66.10	66.10	64.41	61.02	66.10	64.92	2.84
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	62.71	66.10	61.02	62.71	62.71	67.80	67.80	67.80	64.41	66.10	64.92	2.40
EmbraceBERT-bs8-p_att_clsquery_weights	64.41	62.71	61.02	62.71	61.02	62.71	62.71	62.71	69.49	62.71	63.22	2.28
EmbraceBERTwithProj-bs8	71.19	66.10	66.10	66.10	66.10	66.10	71.19	66.10	62.71	57.63	65.93	3.67
EmbraceBERTwithProj-bs8-p_attclsqw	61.02	64.41	64.41	64.41	67.80	67.80	66.10	69.49	66.10	61.02	65.25	2.65
EBERTconcatAtt-bs8	64.41	64.41	50.85	64.41	64.41	54.24	61.02	64.41	69.49	61.02	61.86	5.21
EBERTconcatAtt-bs8_attclsqw	67.80	67.80	62.71	61.02	62.71	54.24	67.80	61.02	62.71	61.02	62.88	3.97
EBERTconcatAttwithProj-bs8	67.80	64.41	66.10	69.49	64.41	62.71	67.80	62.71	66.10	62.71	65.42	2.30
EBERTconcatAttwithProj-bs8_attclsqw	66.10	67.80	62.71	62.71	67.80	59.32	69.49	61.02	62.71	67.80	64.75	3.29
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	66.10	66.10	67.80	16.95	10.17	64.41	67.80	27.12	64.41	64.41	51.53	22.26
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	64.41	66.10	59.32	67.80	64.41	66.10	62.71	66.10	62.71	66.10	64.58	2.33
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	67.80	67.80	8.47	66.10	11.86	64.41	66.10	66.10	67.80	1.69	48.81	27.26
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	69.49	66.10	62.71	62.71	64.41	67.80	62.71	69.49	67.80	61.02	65.42	2.96
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	28.81	67.80	15.25	15.25	64.41	61.02	15.25	15.25	18.64	67.80	36.95	23.50
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	62.71	62.71	69.49	62.71	67.80	61.02	62.71	64.41	64.41	62.71	64.07	2.49
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	72.88	72.88	67.80	6.78	32.20	66.10	69.49	11.86	61.02	20.34	48.14	25.71
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	62.71	62.71	64.41	66.10	66.10	59.32	67.80	69.49	64.41	62.71	64.58	2.78

gtts-sphinx

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	52.54	62.71	66.10	45.76	54.24	54.24	57.63	59.32	55.93	66.10	57.46	6.04
BERT-bs4+Dropout0.1	61.02	61.02	52.54	52.54	61.02	57.63	55.93	61.02	55.93	61.02	57.97	3.37
EmbraceBERT-bs4	59.32	49.15	57.63	57.63	59.32	59.32	57.63	54.24	59.32	55.93	56.95	3.05
EmbraceBERT-bs4+Dropout0.1	55.93	55.93	55.93	52.54	55.93	61.02	61.02	57.63	59.32	57.63	57.29	2.49
EmbraceBERT-bs4-p_selfatt	57.63	61.02	54.24	61.02	57.63	52.54	54.24	55.93	59.32	52.54	56.61	3.05
EmbraceBERT-bs4-p_att_clsquery_weights	59.32	49.15	52.54	57.63	55.93	57.63	55.93	57.63	55.93	61.02	56.27	3.20
CondensedEmbraceBERT-bs4
CondensedEmbraceBERT-bs4+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0	55.93	59.32	52.54	55.93	57.63	55.93	57.63	49.15	57.63	57.63	55.93	2.84
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	59.32	52.54	57.63	55.93	52.54	55.93	59.32	57.63	52.54	54.24	55.76	2.56
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	57.63	54.24	59.32	59.32	55.93	59.32	57.63	59.32	50.85	50.85	56.44	3.22
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	54.24	52.54	61.02	57.63	45.76	57.63	54.24	55.93	62.71	54.24	55.59	4.47
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	55.93	54.24	54.24	57.63	54.24	57.63	55.93	61.02	49.15	64.41	56.44	3.94
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	57.63	52.54	61.02	59.32	44.07	52.54	54.24	62.71	55.93	49.15	54.92	5.37
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	54.24	52.54	57.63	55.93	61.02	54.24	50.85	54.24	54.24	59.32	55.42	2.94
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	54.24	55.93	61.02	55.93	57.63	55.93	54.24	64.41	62.71	57.63	57.97	3.37
BERT-bs16+Dropout0.1	55.93	57.63	57.63	50.85	52.54	54.24	59.32	61.02	57.63	55.93	56.27	2.92
EmbraceBERT-bs16	50.85	61.02	57.63	52.54	55.93	50.85	55.93	52.54	50.85	49.15	53.73	3.56
EmbraceBERT-bs16+Dropout0.1	55.93	54.24	59.32	57.63	50.85	55.93	54.24	57.63	50.85	59.32	55.59	2.92
CondensedEmbraceBERT-bs16
CondensedEmbraceBERT-bs16+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3	59.32	52.54	55.93	59.32	55.93	54.24	54.24	49.15	55.93	50.85	54.75	3.13
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	57.63	55.93	57.63	55.93	59.32	50.85	52.54	57.63	55.93	52.54	55.59	2.60
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10	55.93	59.32	59.32	57.63	50.85	55.93	50.85	57.63	61.02	55.93	56.44	3.22
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3	59.32	54.24	55.93	59.32	57.63	54.24	54.24	49.15	55.93	50.85	55.08	3.15
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10	52.54	59.32	64.41	57.63	50.85	57.63	50.85	55.93	59.32	54.24	56.27	4.07

• Epoch 100 - bs8

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	57.63	55.93	57.63	52.54	54.24	57.63	54.24	62.71	61.02	57.63	57.12	2.94
BERTwithAtt-bs8	54.24	66.10	54.24	55.93	55.93	57.63	59.32	61.02	57.63	62.71	58.47	3.65
BERTwithAttClsProjection-bs8	55.93	57.63	61.02	57.63	57.63	57.63	62.71	59.32	59.32	62.71	59.15	2.20
BERTwithProjection-bs8	57.63	55.93	66.10	55.93	64.41	62.71	49.15	59.32	57.63	64.41	59.32	4.91
BERTwithProjectionAtt-bs8	59.32	54.24	57.63	52.54	59.32	62.71	55.93	55.93	61.02	55.93	57.46	2.98
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	55.93	52.54	52.54	54.24	54.24	55.93	59.32	55.93	55.93	50.85	54.75	2.28
EmbraceBERT-bs8-p_att_clsquery_weights	57.63	52.54	52.54	59.32	55.93	62.71	47.46	54.24	59.32	55.93	55.76	4.11
EmbraceBERTwithProj-bs8	61.02	66.10	61.02	61.02	57.63	59.32	59.32	54.24	57.63	57.63	59.49	2.98
EmbraceBERTwithProj-bs8-p_attclsqw	55.93	57.63	55.93	57.63	62.71	62.71	57.63	55.93	55.93	59.32	58.14	2.52
EBERTconcatAtt-bs8	59.32	55.93	61.02	61.02	59.32	54.24	57.63	54.24	49.15	45.76	55.76	4.82
EBERTconcatAtt-bs8_attclsqw	52.54	54.24	54.24	61.02	55.93	52.54	47.46	52.54	57.63	59.32	54.75	3.72
EBERTconcatAttwithProj-bs8	59.32	52.54	57.63	57.63	57.63	55.93	59.32	57.63	55.93	57.63	57.12	1.86
EBERTconcatAttwithProj-bs8_attclsqw	50.85	57.63	57.63	50.85	55.93	61.02	55.93	62.71	57.63	59.32	56.95	3.65
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	20.34	50.85	52.54	57.63	54.24	57.63	15.25	20.34	52.54	18.64	40.00	17.60
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	59.32	64.41	54.24	61.02	47.46	57.63	55.93	49.15	54.24	50.85	55.42	5.09
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	57.63	57.63	8.47	6.78	62.71	62.71	28.81	59.32	55.93	8.47	40.85	23.43
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	59.32	59.32	55.93	61.02	59.32	55.93	61.02	55.93	59.32	59.32	58.64	1.89
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	13.56	10.17	52.54	18.64	54.24	20.34	8.47	49.15	15.25	61.02	30.34	19.97
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	61.02	52.54	62.71	52.54	59.32	61.02	59.32	61.02	52.54	54.24	57.63	3.94
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	13.56	59.32	57.63	61.02	57.63	59.32	27.12	10.17	20.34	57.63	42.37	20.51
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	59.32	59.32	55.93	54.24	62.71	64.41	61.02	55.93	61.02	61.02	59.49	3.07

gtts-witai

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	69.49	74.58	74.58	67.80	71.19	72.88	74.58	74.58	76.27	74.58	73.05	2.56
BERT-bs4+Dropout0.1	74.58	71.19	66.10	71.19	69.49	71.19	76.27	71.19	76.27	79.66	72.71	3.75
EmbraceBERT-bs4	67.80	67.80	67.80	72.88	71.19	74.58	71.19	76.27	74.58	76.27	72.03	3.24
EmbraceBERT-bs4+Dropout0.1	72.88	72.88	64.41	77.97	74.58	74.58	76.27	72.88	71.19	74.58	73.22	3.46
EmbraceBERT-bs4-p_selfatt	74.58	79.66	71.19	77.97	67.80	69.49	67.80	72.88	69.49	71.19	72.20	3.88
EmbraceBERT-bs4-p_att_clsquery_weights	71.19	64.41	81.36	72.88	69.49	74.58	71.19	72.88	76.27	72.88	72.71	4.18
CondensedEmbraceBERT-bs4
CondensedEmbraceBERT-bs4+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0	71.19	72.88	74.58	67.80	71.19	69.49	69.49	72.88	74.58	72.88	71.69	2.15
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	72.88	67.80	71.19	67.80	72.88	74.58	64.41	74.58	71.19	71.19	70.85	3.11
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	71.19	72.88	69.49	67.80	72.88	71.19	77.97	72.88	69.49	66.10	71.19	3.13
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	77.97	67.80	72.88	67.80	67.80	71.19	79.66	69.49	74.58	76.27	72.54	4.21
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	71.19	72.88	71.19	67.80	77.97	74.58	67.80	74.58	69.49	66.10	71.36	3.51
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	76.27	64.41	69.49	69.49	69.49	76.27	77.97	72.88	72.88	74.58	72.37	3.94
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	74.58	72.88	71.19	76.27	74.58	72.88	67.80	74.58	69.49	71.19	72.54	2.49
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	77.97	72.88	67.80	69.49	69.49	76.27	67.80	72.88	74.58	71.19	72.03	3.33
BERT-bs16+Dropout0.1	74.58	67.80	69.49	69.49	66.10	76.27	72.88	71.19	72.88	71.19	71.19	2.94
EmbraceBERT-bs16	71.19	69.49	69.49	66.10	76.27	67.80	69.49	71.19	72.88	76.27	71.02	3.17
EmbraceBERT-bs16+Dropout0.1	71.19	67.80	69.49	71.19	76.27	71.19	72.88	69.49	72.88	76.27	71.86	2.65
CondensedEmbraceBERT-bs16
CondensedEmbraceBERT-bs16+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3	71.19	71.19	72.88	72.88	64.41	64.41	69.49	67.80	69.49	71.19	69.49	2.94
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	69.49	72.88	71.19	71.19	69.49	67.80	64.41	66.10	67.80	67.80	68.81	2.42
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10	69.49	67.80	67.80	76.27	67.80	69.49	77.97	62.71	69.49	69.49	69.83	4.14
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3	67.80	71.19	71.19	72.88	67.80	71.19	67.80	67.80	69.49	69.49	69.66	1.77
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10	71.19	67.80	69.49	76.27	66.10	69.49	74.58	66.10	72.88	69.49	70.34	3.24

• Epoch 100 - bs8

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	74.58	71.19	67.80	71.19	67.80	71.19	69.49	69.49	74.58	76.27	71.36	2.78
BERTwithAtt-bs8	74.58	77.97	64.41	66.10	72.88	71.19	67.80	76.27	72.88	76.27	72.03	4.37
BERTwithAttClsProjection-bs8	69.49	72.88	69.49	72.88	71.19	69.49	74.58	69.49	72.88	72.88	71.53	1.83
BERTwithProjection-bs8	61.02	69.49	81.36	71.19	71.19	69.49	62.71	69.49	66.10	66.10	68.81	5.32
BERTwithProjectionAtt-bs8	77.97	72.88	72.88	71.19	71.19	74.58	72.88	76.27	71.19	66.10	72.71	3.07
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	76.27	71.19	67.80	67.80	71.19	74.58	71.19	72.88	72.88	76.27	72.20	2.86
EmbraceBERT-bs8-p_att_clsquery_weights	67.80	67.80	67.80	67.80	72.88	66.10	69.49	74.58	74.58	74.58	70.34	3.24
EmbraceBERTwithProj-bs8	69.49	69.49	74.58	74.58	71.19	72.88	71.19	71.19	67.80	67.80	71.02	2.33
EmbraceBERTwithProj-bs8-p_attclsqw	69.49	71.19	69.49	69.49	76.27	69.49	72.88	69.49	72.88	71.19	71.19	2.14
EBERTconcatAtt-bs8	67.80	69.49	71.19	66.10	71.19	62.71	69.49	69.49	72.88	69.49	68.98	2.74
EBERTconcatAtt-bs8_attclsqw	76.27	71.19	69.49	69.49	74.58	72.88	67.80	72.88	76.27	71.19	72.20	2.75
EBERTconcatAttwithProj-bs8	69.49	69.49	67.80	66.10	71.19	67.80	66.10	69.49	64.41	67.80	67.97	1.93
EBERTconcatAttwithProj-bs8_attclsqw	67.80	67.80	71.19	67.80	72.88	69.49	67.80	77.97	69.49	69.49	70.17	3.05
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	69.49	13.56	72.88	66.10	10.17	18.64	71.19	67.80	71.19	67.80	52.88	25.51
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	71.19	71.19	69.49	71.19	71.19	72.88	74.58	71.19	71.19	72.88	71.69	1.32
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	13.56	71.19	67.80	69.49	74.58	67.80	28.81	5.08	71.19	1.69	47.12	29.26
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	66.10	69.49	67.80	69.49	74.58	74.58	69.49	69.49	69.49	69.49	70.00	2.52
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	69.49	10.17	76.27	16.95	8.47	67.80	67.80	71.19	16.95	18.64	42.37	28.37
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	74.58	67.80	74.58	69.49	71.19	71.19	67.80	77.97	71.19	69.49	71.53	3.11
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	71.19	67.80	22.03	72.88	71.19	3.39	71.19	67.80	27.12	11.86	48.64	27.24
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	71.19	74.58	69.49	72.88	69.49	69.49	71.19	71.19	66.10	72.88	70.85	2.25

macsay-google

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	55.93	64.41	62.71	57.63	62.71	61.02	62.71	54.24	72.88	67.80	62.20	5.25
BERT-bs4+Dropout0.1	64.41	64.41	62.71	55.93	64.41	62.71	57.63	57.63	64.41	71.19	62.54	4.25
EmbraceBERT-bs4	62.71	62.71	54.24	57.63	61.02	62.71	66.10	62.71	62.71	67.80	62.03	3.65
EmbraceBERT-bs4+Dropout0.1	57.63	64.41	54.24	57.63	66.10	61.02	61.02	59.32	64.41	66.10	61.19	3.82
EmbraceBERT-bs4-p_selfatt	66.10	64.41	66.10	71.19	55.93	57.63	57.63	61.02	59.32	66.10	62.54	4.70
EmbraceBERT-bs4-p_att_clsquery_weights	67.80	47.46	62.71	55.93	59.32	64.41	62.71	66.10	69.49	59.32	61.53	6.11
CondensedEmbraceBERT-bs4
CondensedEmbraceBERT-bs4+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0	57.63	55.93	59.32	54.24	61.02	52.54	55.93	49.15	55.93	59.32	56.10	3.34
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	54.24	61.02	62.71	52.54	57.63	61.02	50.85	61.02	57.63	52.54	57.12	4.09
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	57.63	64.41	54.24	55.93	55.93	64.41	50.85	50.85	54.24	47.46	55.59	5.24
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	52.54	57.63	55.93	57.63	54.24	59.32	55.93	61.02	61.02	59.32	57.46	2.67
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	50.85	59.32	61.02	47.46	57.63	64.41	52.54	59.32	57.63	52.54	56.27	4.96
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	54.24	57.63	64.41	62.71	52.54	64.41	55.93	64.41	61.02	64.41	60.17	4.44
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	62.71	61.02	59.32	62.71	62.71	59.32	64.41	66.10	62.71	61.02	62.20	2.01
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	61.02	66.10	57.63	62.71	62.71	66.10	57.63	54.24	64.41	59.32	61.19	3.75
BERT-bs16+Dropout0.1	61.02	62.71	62.71	59.32	61.02	61.02	57.63	54.24	64.41	64.41	60.85	2.98
EmbraceBERT-bs16	64.41	59.32	57.63	64.41	55.93	61.02	64.41	61.02	61.02	57.63	60.68	2.92
EmbraceBERT-bs16+Dropout0.1	62.71	61.02	57.63	59.32	64.41	59.32	62.71	66.10	67.80	67.80	62.88	3.43
CondensedEmbraceBERT-bs16
CondensedEmbraceBERT-bs16+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3	54.24	54.24	64.41	55.93	49.15	55.93	52.54	52.54	55.93	52.54	54.75	3.79
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	55.93	59.32	59.32	50.85	55.93	54.24	55.93	54.24	57.63	54.24	55.76	2.45
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10	50.85	59.32	61.02	55.93	54.24	61.02	67.80	61.02	61.02	59.32	59.15	4.38
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3	50.85	52.54	64.41	55.93	52.54	54.24	49.15	52.54	59.32	54.24	54.58	4.21
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10	50.85	55.93	59.32	54.24	50.85	62.71	64.41	59.32	62.71	61.02	58.14	4.68

• Epoch 100 - bs8

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	57.63	64.41	62.71	64.41	59.32	57.63	61.02	59.32	59.32	61.02	60.68	2.37
BERTwithAtt-bs8	66.10	71.19	57.63	64.41	57.63	66.10	61.02	62.71	67.80	66.10	64.07	4.14
BERTwithAttClsProjection-bs8	62.71	67.80	57.63	59.32	57.63	67.80	62.71	57.63	61.02	66.10	62.03	3.88
BERTwithProjection-bs8	59.32	62.71	66.10	67.80	62.71	69.49	54.24	61.02	55.93	59.32	61.86	4.69
BERTwithProjectionAtt-bs8	57.63	67.80	64.41	59.32	57.63	59.32	64.41	67.80	66.10	59.32	62.37	3.92
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	67.80	55.93	55.93	59.32	57.63	67.80	61.02	61.02	61.02	64.41	61.19	4.11
EmbraceBERT-bs8-p_att_clsquery_weights	55.93	55.93	61.02	62.71	57.63	64.41	57.63	64.41	57.63	62.71	60.00	3.23
EmbraceBERTwithProj-bs8	57.63	66.10	64.41	64.41	64.41	62.71	57.63	62.71	61.02	62.71	62.37	2.71
EmbraceBERTwithProj-bs8-p_attclsqw	64.41	61.02	64.41	62.71	62.71	66.10	59.32	59.32	62.71	62.71	62.54	2.07
EBERTconcatAtt-bs8	59.32	61.02	62.71	62.71	62.71	59.32	64.41	59.32	64.41	62.71	61.86	1.89
EBERTconcatAtt-bs8_attclsqw	64.41	59.32	55.93	64.41	55.93	57.63	62.71	59.32	62.71	67.80	61.02	3.79
EBERTconcatAttwithProj-bs8	64.41	59.32	59.32	61.02	62.71	57.63	64.41	61.02	62.71	57.63	61.02	2.40
EBERTconcatAttwithProj-bs8_attclsqw	55.93	66.10	59.32	66.10	61.02	59.32	69.49	66.10	59.32	62.71	62.54	4.04
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	57.63	59.32	61.02	16.95	64.41	20.34	15.25	23.73	59.32	55.93	43.39	20.07
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	61.02	59.32	61.02	59.32	59.32	67.80	62.71	62.71	66.10	61.02	62.03	2.75
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	62.71	62.71	11.86	66.10	67.80	15.25	62.71	59.32	62.71	64.41	53.56	20.13
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	64.41	62.71	61.02	66.10	64.41	64.41	61.02	64.41	64.41	62.71	63.56	1.56
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	11.86	62.71	61.02	64.41	66.10	54.24	15.25	18.64	64.41	13.56	43.22	23.43
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	64.41	64.41	64.41	64.41	62.71	64.41	57.63	64.41	67.80	66.10	64.07	2.49
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	59.32	59.32	64.41	57.63	59.32	61.02	28.81	59.32	28.81	64.41	54.24	12.89
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	62.71	59.32	67.80	59.32	54.24	64.41	67.80	66.10	64.41	64.41	63.05	4.07

macsay-sphinx

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	52.54	50.85	55.93	52.54	55.93	50.85	49.15	57.63	57.63	52.54	53.56	2.86
BERT-bs4+Dropout0.1	54.24	55.93	50.85	45.76	57.63	50.85	57.63	54.24	55.93	49.15	53.22	3.73
EmbraceBERT-bs4	52.54	50.85	45.76	42.37	52.54	50.85	50.85	47.46	42.37	49.15	48.47	3.65
EmbraceBERT-bs4+Dropout0.1	47.46	40.68	42.37	47.46	37.29	49.15	40.68	61.02	38.98	50.85	45.59	6.76
EmbraceBERT-bs4-p_selfatt	45.76	55.93	52.54	50.85	47.46	49.15	40.68	52.54	54.24	49.15	49.83	4.23
EmbraceBERT-bs4-p_att_clsquery_weights	52.54	47.46	47.46	55.93	54.24	47.46	52.54	57.63	47.46	47.46	51.02	3.82
CondensedEmbraceBERT-bs4
CondensedEmbraceBERT-bs4+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0	47.46	47.46	42.37	44.07	40.68	49.15	44.07	37.29	50.85	47.46	45.08	3.95
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	47.46	44.07	50.85	38.98	47.46	54.24	40.68	37.29	40.68	52.54	45.42	5.66
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	42.37	42.37	45.76	38.98	47.46	37.29	44.07	50.85	40.68	50.85	44.07	4.42
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	44.07	52.54	49.15	45.76	42.37	49.15	44.07	38.98	33.90	38.98	43.90	5.33
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	47.46	49.15	49.15	45.76	44.07	45.76	50.85	38.98	42.37	47.46	46.10	3.37
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	52.54	50.85	47.46	54.24	44.07	47.46	44.07	40.68	44.07	50.85	47.63	4.18
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	45.76	54.24	54.24	47.46	44.07	38.98	52.54	57.63	55.93	47.46	49.83	5.68
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	49.15	52.54	52.54	50.85	54.24	49.15	47.46	52.54	64.41	52.54	52.54	4.42
BERT-bs16+Dropout0.1	50.85	55.93	50.85	50.85	55.93	49.15	44.07	52.54	55.93	47.46	51.36	3.72
EmbraceBERT-bs16	49.15	47.46	40.68	44.07	42.37	45.76	40.68	55.93	33.90	52.54	45.25	6.07
EmbraceBERT-bs16+Dropout0.1	49.15	40.68	42.37	50.85	44.07	50.85	47.46	55.93	32.20	49.15	46.27	6.34
CondensedEmbraceBERT-bs16
CondensedEmbraceBERT-bs16+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3	44.07	45.76	38.98	57.63	45.76	44.07	33.90	44.07	38.98	49.15	44.24	6.09
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	40.68	45.76	44.07	47.46	47.46	45.76	38.98	49.15	42.37	47.46	44.92	3.15
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10	44.07	32.20	45.76	42.37	35.59	47.46	33.90	45.76	45.76	40.68	41.36	5.26
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3	45.76	47.46	42.37	50.85	45.76	38.98	40.68	42.37	40.68	45.76	44.07	3.47
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10	44.07	32.20	44.07	44.07	35.59	50.85	33.90	35.59	42.37	42.37	40.51	5.59

• Epoch 100 - bs8

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	44.07	54.24	55.93	52.54	49.15	44.07	42.37	54.24	54.24	54.24	50.51	4.90
BERTwithAtt-bs8	44.07	59.32	50.85	50.85	54.24	47.46	52.54	62.71	52.54	50.85	52.54	5.08
BERTwithAttClsProjection-bs8	49.15	55.93	47.46	45.76	47.46	50.85	45.76	45.76	49.15	54.24	49.15	3.39
BERTwithProjection-bs8	42.37	50.85	50.85	49.15	50.85	59.32	47.46	49.15	42.37	52.54	49.49	4.66
BERTwithProjectionAtt-bs8	54.24	50.85	40.68	54.24	44.07	54.24	59.32	47.46	54.24	42.37	50.17	5.88
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	44.07	47.46	42.37	52.54	40.68	49.15	47.46	54.24	42.37	49.15	46.95	4.29
EmbraceBERT-bs8-p_att_clsquery_weights	47.46	38.98	45.76	42.37	50.85	50.85	45.76	47.46	40.68	44.07	45.42	3.77
EmbraceBERTwithProj-bs8	50.85	45.76	47.46	49.15	45.76	55.93	52.54	47.46	44.07	52.54	49.15	3.56
EmbraceBERTwithProj-bs8-p_attclsqw	52.54	52.54	49.15	45.76	47.46	47.46	47.46	47.46	50.85	47.46	48.81	2.25
EBERTconcatAtt-bs8	44.07	47.46	47.46	50.85	49.15	44.07	49.15	54.24	45.76	49.15	48.14	2.96
EBERTconcatAtt-bs8_attclsqw	49.15	50.85	50.85	50.85	50.85	44.07	47.46	55.93	57.63	47.46	50.51	3.77
EBERTconcatAttwithProj-bs8	50.85	38.98	54.24	45.76	45.76	52.54	49.15	49.15	49.15	44.07	47.97	4.22
EBERTconcatAttwithProj-bs8_attclsqw	44.07	52.54	52.54	40.68	50.85	50.85	44.07	52.54	44.07	49.15	48.14	4.23
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	20.34	50.85	18.64	49.15	18.64	45.76	52.54	45.76	45.76	44.07	39.15	13.29
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	40.68	45.76	49.15	47.46	50.85	44.07	49.15	49.15	59.32	47.46	48.31	4.63
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	18.64	11.86	10.17	49.15	44.07	10.17	54.24	44.07	45.76	50.85	33.90	17.68
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	49.15	44.07	52.54	44.07	45.76	52.54	42.37	47.46	50.85	49.15	47.80	3.46
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	52.54	44.07	13.56	18.64	1.69	55.93	45.76	16.95	25.42	10.17	28.47	18.41
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	52.54	45.76	40.68	49.15	55.93	57.63	54.24	50.85	45.76	57.63	51.02	5.38
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	47.46	50.85	42.37	42.37	52.54	47.46	45.76	45.76	47.46	50.85	47.29	3.26
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	50.85	49.15	52.54	49.15	45.76	52.54	44.07	52.54	52.54	54.24	50.34	3.13

macsay-witai

• Epoch 100

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs4	72.88	71.19	74.58	74.58	72.88	76.27	71.19	72.88	76.27	71.19	73.39	1.86
BERT-bs4+Dropout0.1	72.88	74.58	71.19	74.58	71.19	72.88	74.58	77.97	79.66	79.66	74.92	3.01
EmbraceBERT-bs4	76.27	66.10	66.10	74.58	77.97	69.49	72.88	72.88	69.49	71.19	71.69	3.79
EmbraceBERT-bs4+Dropout0.1	69.49	72.88	66.10	72.88	74.58	72.88	76.27	72.88	74.58	71.19	72.37	2.74
EmbraceBERT-bs4-p_selfatt	77.97	69.49	71.19	77.97	64.41	71.19	72.88	72.88	76.27	74.58	72.88	3.94
EmbraceBERT-bs4-p_att_clsquery_weights	72.88	67.80	77.97	69.49	76.27	71.19	79.66	76.27	74.58	72.88	73.90	3.57
CondensedEmbraceBERT-bs4
CondensedEmbraceBERT-bs4+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ0	67.80	72.88	77.97	76.27	69.49	71.19	74.58	62.71	71.19	71.19	71.53	4.14
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3	69.49	72.88	71.19	62.71	66.10	74.58	66.10	57.63	74.58	66.10	68.14	5.19
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ3+Dropout0.1	72.88	71.19	66.10	67.80	64.41	81.36	62.71	66.10	64.41	66.10	68.31	5.25
EmbraceBERT-bs4-p_multiheadatt_bertquery_epQ10	67.80	67.80	72.88	74.58	66.10	71.19	67.80	67.80	69.49	76.27	70.17	3.23
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ3	71.19	71.19	71.19	61.02	66.10	79.66	64.41	62.71	72.88	64.41	68.47	5.42
EmbraceBERT-bs4-p_multiheadatt_bertKeyVal_epQ10	66.10	71.19	71.19	72.88	64.41	69.49	64.41	64.41	71.19	76.27	69.15	3.92
EmbraceBERT-bs4-p_multiheadatt_bertKeyValQuery	79.66	72.88	69.49	76.27	76.27	67.80	77.97	77.97	71.19	76.27	74.58	3.79
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
BERT-bs16	76.27	74.58	72.88	72.88	72.88	72.88	69.49	69.49	69.49	74.58	72.54	2.25
BERT-bs16+Dropout0.1	77.97	74.58	74.58	72.88	71.19	71.19	72.88	71.19	72.88	71.19	73.05	2.07
EmbraceBERT-bs16	71.19	69.49	67.80	67.80	74.58	69.49	71.19	77.97	69.49	74.58	71.36	3.17
EmbraceBERT-bs16+Dropout0.1	71.19	74.58	69.49	69.49	72.88	69.49	74.58	71.19	71.19	77.97	72.20	2.65
CondensedEmbraceBERT-bs16
CondensedEmbraceBERT-bs16+Dropout0.1
-----------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3	64.41	55.93	72.88	62.71	61.02	67.80	59.32	64.41	66.10	66.10	64.07	4.47
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ3+Dropout0.1	66.10	67.80	72.88	64.41	62.71	66.10	61.02	59.32	67.80	64.41	65.25	3.65
EmbraceBERT-bs16-p_multiheadatt_bertquery_epQ10	64.41	72.88	61.02	71.19	67.80	71.19	66.10	61.02	66.10	72.88	67.46	4.27
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ3	62.71	55.93	72.88	61.02	64.41	64.41	55.93	61.02	66.10	64.41	62.88	4.70
EmbraceBERT-bs16-p_multiheadatt_bertKeyVal_epQ10	61.02	71.19	61.02	62.71	66.10	69.49	64.41	59.32	64.41	72.88	65.25	4.37

• Epoch 100 - bs8

Model	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9	Run10	Mean	Std
BERT-bs8	77.97	72.88	76.27	72.88	69.49	72.88	74.58	74.58	71.19	71.19	73.39	2.40
BERTwithAtt-bs8	69.49	72.88	67.80	71.19	66.10	69.49	69.49	69.49	69.49	74.58	70.00	2.28
BERTwithAttClsProjection-bs8	62.71	69.49	71.19	69.49	62.71	69.49	69.49	59.32	67.80	67.80	66.95	3.73
BERTwithProjection-bs8	59.32	62.71	76.27	67.80	67.80	74.58	61.02	74.58	59.32	71.19	67.46	6.24
BERTwithProjectionAtt-bs8	72.88	71.19	62.71	67.80	71.19	74.58	74.58	72.88	69.49	67.80	70.51	3.49
---------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8	76.27	67.80	64.41	67.80	72.88	72.88	74.58	72.88	69.49	72.88	71.19	3.47
EmbraceBERT-bs8-p_att_clsquery_weights	69.49	71.19	69.49	69.49	72.88	71.19	71.19	76.27	72.88	74.58	71.86	2.17
EmbraceBERTwithProj-bs8	69.49	72.88	74.58	76.27	69.49	71.19	71.19	69.49	69.49	72.88	71.69	2.28
EmbraceBERTwithProj-bs8-p_attclsqw	74.58	74.58	71.19	69.49	69.49	72.88	71.19	66.10	69.49	67.80	70.68	2.63
EBERTconcatAtt-bs8	69.49	71.19	72.88	74.58	69.49	59.32	69.49	69.49	77.97	67.80	70.17	4.62
EBERTconcatAtt-bs8_attclsqw	76.27	66.10	67.80	67.80	71.19	69.49	64.41	69.49	74.58	66.10	69.32	3.59
EBERTconcatAttwithProj-bs8	69.49	66.10	64.41	67.80	71.19	72.88	62.71	66.10	67.80	67.80	67.63	2.88
EBERTconcatAttwithProj-bs8_attclsqw	69.49	72.88	72.88	71.19	71.19	69.49	72.88	74.58	69.49	71.19	71.53	1.66
-------------------------------------	---	---	---	---	---	---	---	---	---	---	---	---
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery	25.42	11.86	16.95	66.10	11.86	74.58	72.88	74.58	69.49	62.71	48.64	26.68
EmbraceBERT-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	72.88	71.19	74.58	71.19	72.88	77.97	76.27	72.88	76.27	69.49	73.56	2.54
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery	72.88	64.41	71.19	71.19	71.19	67.80	74.58	66.10	72.88	67.80	70.00	3.13
EmbraceBERTwithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	69.49	71.19	66.10	71.19	69.49	74.58	71.19	72.88	72.88	74.58	71.36	2.45
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery	71.19	72.88	71.19	71.19	10.17	28.81	64.41	22.03	20.34	71.19	50.34	24.95
EmbraceBERTconcatatt-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	72.88	76.27	72.88	71.19	79.66	76.27	62.71	67.80	69.49	74.58	72.37	4.61
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery	72.88	71.19	74.58	71.19	67.80	71.19	74.58	69.49	67.80	72.88	71.36	2.33
EmbraceBERTconcatattWithProj-bs8-p_multiheadatt_bertKeyValQuery_attclsqw	69.49	71.19	72.88	66.10	67.80	71.19	72.88	69.49	72.88	72.88	70.68	2.28