Datasets tutorial improvements

[RdoubleA]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (please add here)

Please link to any issues this PR addresses.

Changelog

• Add needed details around using local datasets, custom instruct/chat datasets, how to use built-in datasets, sample packing, and more all with python/config/CLI examples
• Clean up some config utilities and let users specify any path for custom instruct templates or chat formats. This also needed to update some test utils
• Make column optional in TextCompletionDataset for local datasets
• Remove redundant truncate in tokenizer utils

Sample packing will have a dedicated page, so details on that are limited for now.

Test plan

doc build, pytest tests

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/994

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a573b14 with merge base bbf0010 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[joecummings]

Long CLI commands are getting hard to understand. Might be good to break up over multiple lines with backslash operator.

[joecummings]

PreferenceDataset is not added to our documentation, therefore the class isn't rendering here.

[pbontrager]

Really great tutorial updates. I've left some comments and questions, some of them just because this was my first time going through the tutorial. One concern I have with the way we're doing deepdives/tutorials is that we're putting a lot of important information here while our API references are pretty sparse. I'm afraid that people won't take the time to go through the tutorial and jump to what they're looking for in the api reference section.

[pbontrager]

Thanks for doing this and updating from all the comments.