Skip to content

Commit

Permalink
Populate all the LFS data needed for running examples within the rele…
Browse files Browse the repository at this point in the history
…ase container (#1976)

Move the data files needed for log-parsing from models to examples/data to avoid copying models to the release container

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: #1976
  • Loading branch information
AnuradhaKaruppiah authored Oct 25, 2024
1 parent cf1ec22 commit be7e90f
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 6 deletions.
3 changes: 1 addition & 2 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ RUN --mount=type=bind,source=.,target=/opt/host_repo \
git clone file:///opt/host_repo/${MORPHEUS_ROOT_HOST} /tmp/morpheus_repo &&\
cd /tmp/morpheus_repo &&\
git lfs install &&\
./scripts/fetch_data.py fetch datasets examples
./scripts/fetch_data.py fetch examples

# ============ Stage: runtime ============
# Setup container for runtime environment
Expand All @@ -360,7 +360,6 @@ COPY --from=git_clone "/tmp/morpheus_repo/conda/environments/*.yaml" "./conda/en
COPY --from=git_clone "/tmp/morpheus_repo/docker" "./docker"
COPY --from=build_docs "/workspace/${MORPHEUS_ROOT_HOST}/build/docs/html" "./docs"
COPY --from=git_clone "/tmp/morpheus_repo/examples" "./examples"
COPY --from=git_clone "/tmp/morpheus_repo/models/datasets" "./models/datasets"
COPY --from=git_clone "/tmp/morpheus_repo/scripts" "./scripts"
COPY --from=git_clone "/tmp/morpheus_repo/*.md" "./"
COPY --from=git_clone "/tmp/morpheus_repo/LICENSE" "./"
Expand Down
3 changes: 3 additions & 0 deletions examples/data/log-parsing-config-20220418.json
Git LFS file not shown
3 changes: 3 additions & 0 deletions examples/data/log-parsing-validation-data-input.csv
Git LFS file not shown
6 changes: 3 additions & 3 deletions examples/log_parsing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,11 @@ Run the following from the root of the Morpheus repo to start the log parsing pi

```bash
python examples/log_parsing/run.py \
--input_file=./models/datasets/validation-data/log-parsing-validation-data-input.csv \
--input_file=./examples/data/log-parsing-validation-data-input.csv \
--model_vocab_hash_file=data/bert-base-cased-hash.txt \
--model_vocab_file=./models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt \
--model_vocab_file=data/bert-base-cased-vocab.txt \
--model_name log-parsing-onnx \
--model_config_file=./models/log-parsing-models/log-parsing-config-20220418.json
--model_config_file=./examples/data/log-parsing-config-20220418.json
```

Use `--help` to display information about the command line options:
Expand Down
2 changes: 1 addition & 1 deletion examples/log_parsing/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
help="Model vocab hash file to use for pre-processing.")
@click.option('--model_vocab_file',
required=True,
type=click.Path(exists=True, dir_okay=False),
type=MorpheusRelativePath(exists=True, dir_okay=False),
help="Model vocab file to use for post-processing.")
@click.option("--model_seq_length",
default=256,
Expand Down

0 comments on commit be7e90f

Please sign in to comment.