[FEA]: Use Rapids Dependency Manger For Handling all Dependencies in Morpheus

[mdemoret-nv]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

Currently, to run many of our examples, its required to install the examples dependencies from docker/conda/environments/cuda11.8_examples.yml. Since this file only includes the additional dependencies needed for the examples, its required to first install the dev dependencies. A naive user might do the following:

mamba env update -n morpheus -f docker/conda/environments/cuda11.8_dev.yml
mamba env update -n morpheus -f docker/conda/environments/cuda11.8_examples.yml

Unfortunately, any package restrictions in cuda11.8_dev.yml will not be enforced when updating from cuda11.8_examples.yml. The solution is to first merge the environments and then install the merged environment. This is what happens in CI and looks like:

# Assuming conda-merge has been installed into the base environment
conda-merge docker/conda/environments/cuda11.8_dev.yml docker/conda/environments/cuda11.8_examples.yml > /tmp/merged_env.yml

# Install the merged environment with --prune
mamba env update -n morpheus --prune -f /tmp/merged_env.yml

This is cumbersome, prone to errors and hard to remember.

Describe your ideal solution

Instead, we should use rapids dependency file generator (see here for how to use it) to manage all dependencies in the Morpheus repo combined with some simple scripts.

Using the dependency file generator has several advantages:

1. Allows using a single file as the source of truth for all dependencies
2. Allows for chaining dependencies together
   1. i.e. llm_examples -> examples_common -> dev -> runtime
3. Can generate both conda and pip environment files
4. More feature rich than using conda-merge

Completion Criteria

• Update dependencies.yaml in the project root (file exists already but is not used)
  • Add dependency sections for runtime, dev, examples, docs, testing and any other useful dependency groupings
  • Ensure the values contained in the current conda environment files are mirrored in dependencies.yaml
• Delete the environment files in docker/conda/environments
• Update CI to use rapids-dependency-file-generator
• Update documentation to show how to install dependencies with the new tool
  • This will include reverting some changes made in PR Docs update to indicate use of conda-merge to generate install files #1387
• Reassess the current dependencies
  • Any dependency which is only used by a single stage or single example should be converted to an optional dependency
  • Dependencies which are no longer required or not directly used in the repo should be removed
• [Optional] Create a utility script to run rapids-dependency-file-generator and update the environment in a single command

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[cwharris]

Fixed by #1468