to-file stage failure

[pdmack]

Traceback (most recent call last):
  File "/opt/conda/envs/morpheus/bin/morpheus", line 11, in <module>
    sys.exit(run_cli())
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/cli.py", line 1381, in run_cli
    cli(obj={}, auto_envvar_prefix='CLX', show_default=True, prog_name="morpheus")
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1689, in invoke
    return _process_result(rv)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 1626, in _process_result
    value = ctx.invoke(self._result_callback, value, **ctx.params)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/cli.py", line 543, in post_pipeline
    pipeline.run()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 1239, in run
    asyncio.run(self._do_run())
  File "/opt/conda/envs/morpheus/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/envs/morpheus/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 1214, in _do_run
    self.build_and_start()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 1038, in build_and_start
    self.start()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 998, in start
    self._neo_executor.start()
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 960, in inner_build
    s.build(seg)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 526, in build
    dep.build(seg, do_propagate=do_propagate)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 526, in build
    dep.build(seg, do_propagate=do_propagate)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 526, in build
    dep.build(seg, do_propagate=do_propagate)
  [Previous line repeated 10 more times]
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 504, in build
    out_ports_pair = self._build(seg, in_ports_pairs)
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/pipeline.py", line 783, in _build
    return [self._build_single(seg, in_ports_streams[0])]
  File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/pipeline/output/to_file.py", line 113, in _build_single
    to_file = neos.WriteToFileStage(seg,
ValueError: Read/Write mode ('+') is not supported by WriteToFileStage. Mode: w+

[mdemoret-nv]

Do you have a repro script?

[cwharris]

WriteToFileStage supports w (write, truncate) and a (write, append) modes, but not w+ (read/write truncate). Are you providing the mode explicitly? If so, just change the mode to w, since WriteToFileStage doesn't need to read. If not, how are you producing this error, and what is the expected result?

[pdmack]

This is simply using the to-file stage like this:

morpheus run --use_cpp=True --num_threads=8 --pipeline_batch_size=8192 --model_max_batch_size=32 --edge_buffer_size=32  pipeline-nlp --model_seq_length=256 from-file --filename=./data/pcap_dump.jsonlines dropna monitor --description='Drop Null Attributes rate' deserialize monitor --description='Deserialization rate' preprocess --vocab_hash_file=./data/bert-base-uncased-hash.txt --truncation=True --do_lower_case=True --add_special_tokens=False monitor --description='Preprocessing rate' inf-triton --force_convert_inputs=True --model_name=sid-minibert-onnx --server_url=ai-engine:8001 --use_shared_memory=True monitor --description='Inference rate' --smoothing=0.001 --unit inf add-class monitor --description='Classification rate' serialize --exclude '^ts_' monitor --description='Serialization rate' to-file --filename=/common/data/output/sid-minibert-onnx-output.jsonlines --overwrite

there are no arguments that relate to modes save --overwrite I suppose.

[pdmack]

https://github.com/NVIDIA/Morpheus/blob/branch-22.04/morpheus/pipeline/output/to_file.py#L115

[pdmack]

Works if I drop the --overwrite arg

[cwharris]

What should happen to the output file if the overwrite flag is not provided? Should it be appended to, or should the write fail? Something else?