apply_chat_template: consistent behaviour for return_assistant_tokens_mask=True return_tensors=True#35582

Open

mrsndmn wants to merge 4 commits intohuggingface:mainfrom mrsndmn:apply_chat_template_return_assistant_masks_return_tensors

+61-1

Commits on Jan 9, 2025

apply_chat_template: consistent return_tensors behaviour with return_assistant_tokens_mask flag

Eduard Allakhverdov
authored and
d.tarasov
committed
test_chat_template_return_assistant_tokens_mask: support tokenizers with no attention mask

Eduard Allakhverdov
authored and
d.tarasov
committed
test_chat_template_return_assistant_tokens_mask: skip tokenizers with no padding token
d.tarasov
committed
test_chat_template_return_assistant_tokens_mask: force tokenizer padding_side=right
d.tarasov
committed