[Question] Fintune llava-v1.5-7b just use all textvqa data, but get Accuracy: 0.00%

[ethanyys]

Question

Fintune llava-v1.5-7b just use all textvqa data, but get Accuracy: 0.00%

Trained on 8*A800(80G) with BF16

Evaluated follow docs: CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh

By reviewing eval jsonl file, we can see finetuned llava-v1.5-7b refuse to follow the prompt "Answer the question using a single word or phrase."

{"question_id": "fb39f5b120097669", "prompt": "What year was this taken?\nReference OCR token: yip, AE, Mht, juerYU, MhnN, 02/14/2012\nAnswer the question using a single word or phrase.", "text": "A bookshelf is full of movies such as The Powerpuff Girls and AE.", "answer_id": "ejxFDG2X62UneJRnGALrMt", "model_id": "llava-v1.5-7b", "metadata": {}}
{"question_id": "a3f1fe4ebd510d4e", "prompt": "What kind of comupter is this?\nReference OCR token: Select, 0o-Back, Contique, MacBook\nAnswer the question using a single word or phrase.", "text": "A MacBook computer shows a screen with a Continue button.", "answer_id": "FcrDZNThYoejXViQUc45H8", "model_id": "llava-v1.5-7b", "metadata": {}}
{"question_id": "a3f1fe4ebd510d4e", "prompt": "What does the screen say to do?\nReference OCR token: Select, 0o-Back, Contique, MacBook\nAnswer the question using a single word or phrase.", "text": "A MacBook computer monitor with a open window on the screen.", "answer_id": "5o4MXbQ5sKzpLky97JhKSY", "model_id": "llava-v1.5-7b", "metadata": {}}

How can I fix this problem?

[UCASlemontea]

do you fix this problem？thanks