jinja2 is a necessary dependency, but it is not currently specified

[FlyingCarlos]

System Info

• transformers version: 4.45.2
• Platform: Linux-3.10.0-1160.114.2.el7.x86_64-x86_64-with-glibc2.17
• Python version: 3.10.15
• Huggingface_hub version: 0.26.1
• Safetensors version: 0.4.5
• Accelerate version: 1.0.1
• Accelerate config: not found
• PyTorch version (GPU?): 1.12.1+cu102 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I have addressed the issue: if Jinja2 is not installed, an ‘Extension’ undefined error will occur.

Expected behavior

NA

[FlyingCarlos]

transformers/src/transformers/utils/chat_template_utils.py

Lines 362 to 366 in 30c76d5

	@lru_cache
	def _compile_jinja_template(chat_template):
	class AssistantTracker(Extension):
	# This extension is used to track the indices of assistant-generated tokens in the rendered chat
	tags = {"generation"}

[FlyingCarlos]

And "Extension" is from jinja2

transformers/src/transformers/utils/chat_template_utils.py

Lines 28 to 33 in 30c76d5

	if is_jinja_available():
	import jinja2
	from jinja2.ext import Extension
	from jinja2.sandbox import ImmutableSandboxedEnvironment
	else:
	jinja2 = None

[tibor-reiss]

This is a design decision to keep the number of installed packages small. It's listed here: https://github.com/huggingface/transformers/blob/main/setup.py#L127. As you see, not even torch is installed by default: https://github.com/huggingface/transformers/blob/main/setup.py#L184, https://github.com/huggingface/transformers/blob/main/setup.py#L423

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[TweedBeetle]

Thanks for flagging this. I was getting the 'Extension' undefined error and was quire confused until I found this.

[egoldschmidt]

Likewise, I ran into the same problem - really would be wonderful if there was at least a specific error message that directed the user to add jinja2 as a dep!