Fail to evaluate baichuan-7b and bloom-7b1 on CEVAL

[yangjianxin1]

my dependency is:

transformers==4.32.0
torch==1.13

my config are as follows:

from mmengine.config import read_base
from opencompass.models import HuggingFaceCausalLM

batch_size = 20
# 指定评测模型
model_name_or_paths = [
    'baichuan-inc/Baichuan-7B',
    'bigscience/bloom-7b1',
    'baichuan-inc/Baichuan-13B-Base',
    'baichuan-inc/Baichuan-13B-Chat',
    
]

models = []
for model_name_or_path in model_name_or_paths:
    abbr = model_name_or_path.split('/')[-1]
    model = dict(
            type=HuggingFaceCausalLM,
            abbr=abbr,
            path=model_name_or_path,
            tokenizer_path=model_name_or_path,
            tokenizer_kwargs=dict(padding_side='left',
                                  truncation_side='left',
                                  use_fast=False,
                                  trust_remote_code=True
                                  ),
            max_out_len=100,
            max_seq_len=2048,
            batch_size=batch_size,
            model_kwargs=dict(device_map='auto', trust_remote_code=True),
            batch_padding=False, # if false, inference with for-loop without batch padding
            run_cfg=dict(num_gpus=2, num_procs=2),
        )
    models.append(model)


# 指定评测集
with read_base():
    from .datasets.ceval.ceval_ppl import ceval_datasets
    from .datasets.cmmlu.cmmlu_gen import cmmlu_datasets

    from .summarizers.example import summarizer

datasets = [*ceval_datasets]

I run the above config, the result have some problem with Baichaun-7B and bloom-7b1：

dataset	version	metric	mode	Baichuan-7B	bloom-7b1	Baichuan-13B-Base	Baichuan-13B-Chat
ceval-computer_network	9b9417	accuracy	ppl	-	-	36.84	42.11
ceval-operating_system	b2b8cf	accuracy	ppl	-	-	36.84	42.11
ceval-computer_architecture	1bd275	accuracy	ppl	-	-	61.9	57.14
ceval-college_programming	2d0833	accuracy	ppl	-	-	56.76	56.76
ceval-college_physics	fb7e04	accuracy	ppl	-	-	31.58	26.32
ceval-college_chemistry	916b7d	accuracy	ppl	-	-	58.33	37.5
ceval-advanced_mathematics	5cad2a	accuracy	ppl	-	-	26.32	36.84
ceval-probability_and_statistics	a6b30e	accuracy	ppl	-	-	22.22	16.67
ceval-discrete_mathematics	68be68	accuracy	ppl	-	-	37.5	31.25
ceval-electrical_engineer	056c2e	accuracy	ppl	-	-	43.24	29.73
ceval-metrology_engineer	4a757a	accuracy	ppl	-	-	45.83	33.33
ceval-high_school_mathematics	a8ed21	accuracy	ppl	-	-	22.22	33.33
ceval-high_school_physics	e1fc86	accuracy	ppl	-	-	42.11	47.37
ceval-high_school_chemistry	9021c6	accuracy	ppl	-	-	68.42	26.32
ceval-high_school_biology	c7f5a1	accuracy	ppl	-	-	47.37	52.63
ceval-middle_school_mathematics	213989	accuracy	ppl	-	-	47.37	36.84
ceval-middle_school_biology	ce0420	accuracy	ppl	-	-	85.71	80.95
ceval-middle_school_physics	78f3af	accuracy	ppl	-	-	57.89	52.63
ceval-middle_school_chemistry	d071d2	accuracy	ppl	-	-	70	70
ceval-veterinary_medicine	cd3a07	accuracy	ppl	-	-	30.43	34.78
ceval-college_economics	a35346	accuracy	ppl	-	-	47.27	49.09
ceval-business_administration	69dd6a	accuracy	ppl	-	-	51.52	48.48
ceval-marxism	283ce0	accuracy	ppl	-	-	73.68	57.89
ceval-mao_zedong_thought	f38cd1	accuracy	ppl	-	-	79.17	75
ceval-education_science	fbd65c	accuracy	ppl	-	-	72.41	58.62
ceval-teacher_qualification	c77f1f	accuracy	ppl	-	-	81.82	81.82
ceval-high_school_politics	bbac37	accuracy	ppl	-	-	57.89	73.68
ceval-high_school_geography	730a30	accuracy	ppl	-	-	68.42	47.37
ceval-middle_school_politics	15b2d7	accuracy	ppl	-	-	80.95	85.71
ceval-middle_school_geography	b00167	accuracy	ppl	-	-	83.33	75
ceval-modern_chinese_history	5a04cd	accuracy	ppl	-	-	56.52	56.52
ceval-ideological_and_moral_cultivation	0829ff	accuracy	ppl	-	-	89.47	89.47
ceval-logic	c9c394	accuracy	ppl	-	-	36.36	45.45
ceval-law	cbd3c5	accuracy	ppl	-	-	37.5	29.17
ceval-chinese_language_and_literature	716ab3	accuracy	ppl	-	-	56.52	60.87
ceval-art_studies	476114	accuracy	ppl	-	-	69.7	66.67
ceval-professional_tour_guide	70f30f	accuracy	ppl	-	-	68.97	68.97
ceval-legal_professional	f19cf5	accuracy	ppl	-	-	30.43	43.48
ceval-high_school_chinese	931614	accuracy	ppl	-	-	26.32	36.84
ceval-high_school_history	4d6364	accuracy	ppl	-	-	60	70
ceval-middle_school_history	7f6356	accuracy	ppl	-	-	81.82	77.27
ceval-civil_servant	a5dcb8	accuracy	ppl	-	-	40.43	46.81
ceval-sports_science	192553	accuracy	ppl	-	-	73.68	68.42
ceval-plant_protection	f7ff86	accuracy	ppl	-	-	40.91	54.55
ceval-basic_medicine	a95a09	accuracy	ppl	-	-	78.95	52.63
ceval-clinical_medicine	664b54	accuracy	ppl	-	-	45.45	40.91
ceval-urban_and_rural_planner	fdae6f	accuracy	ppl	-	-	60.87	58.7
ceval-accountant	d810a1	accuracy	ppl	-	-	44.9	48.98
ceval-fire_engineer	bb924d	accuracy	ppl	-	-	58.06	54.84
ceval-environmental_impact_assessment_engineer	d59200	accuracy	ppl	-	-	51.61	38.71
ceval-tax_accountant	9e16f2	accuracy	ppl	-	-	53.06	48.98
ceval-physician	0e90d5	accuracy	ppl	-	-	53.06	51.02
ceval-stem	-	naive_average	ppl	-	-	46.45	42.23
ceval-social-science	-	naive_average	ppl	-	-	69.65	65.27
ceval-humanities	-	naive_average	ppl	-	-	55.78	58.61
ceval-other	-	naive_average	ppl	-	-	54.64	51.32
ceval-hard	-	naive_average	ppl	-	-	38.59	31.95
ceval	-	naive_average	ppl	-	-	54.61	52.05

[Leymore]

Is any more detailed log can be provided?

You may find log in folders like {work_dir}/logs/infer/Baichuan-7B