From b83114b21c8fbab368e4efcdf667dfff54f2016a Mon Sep 17 00:00:00 2001 From: Bhargav Suryadevara Date: Wed, 12 Jul 2023 10:01:19 -0500 Subject: [PATCH 1/3] updated loader docs --- docs/source/loaders/core/sql_loader.md | 9 ++++----- docs/source/modules/core/payload_batcher.md | 12 ++++++------ 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/source/loaders/core/sql_loader.md b/docs/source/loaders/core/sql_loader.md index c3af1ed606..bd7978394c 100644 --- a/docs/source/loaders/core/sql_loader.md +++ b/docs/source/loaders/core/sql_loader.md @@ -17,9 +17,8 @@ limitations under the License. ## SQL Loader -[DataLoader](../../modules/core/data_loader.md) module is configured to use this loader function. It is responsible for -loading data from external sources using the fsspec library, and returns the updated ControlMessage object with payload -as MessageMeta, which contains dataframe (with filenames). +[DataLoader](../../modules/core/data_loader.md) module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload +as MessageMeta. ### Example Loader Configuration @@ -44,7 +43,7 @@ The parameters that can be configured for this specific loader at load task leve |--------------|--------|------------------------------------------|--------------------|---------------| | `strategy` | string | Strategy for combining queries | "aggregate" | `aggregate` | | `loader_id` | string | Unique identifier for the loader | "file_to_df" | `[Required]` | -| `sql_config` | dict | Dictionary containing SQL queries to run | "file_to_df" | `See below` | +| `sql_config` | dictionary | Dictionary containing SQL queries to run | "file_to_df" | `See below` | `sql_config` @@ -58,7 +57,7 @@ The parameters that can be configured for this specific loader at load task leve |---------------------|--------|--------------------------------------|-----------------------------------------------------------------|---------------| | `connection_string` | string | Strategy for combining queries | "postgresql://postgres:postgres@localhost:5432/postgres" | `[required]` | | `query` | string | SQL Query to execute | "SELECT * FROM test_table WHERE id IN (?, ?, ?)" | `[Required]` | -| `params` | dict | Named or positional paramters values | "[foo, bar, baz]" | `-` | +| `params` | dictionary | Named or positional paramters values | "[foo, bar, baz]" | `-` | ### Example Load Task Configuration diff --git a/docs/source/modules/core/payload_batcher.md b/docs/source/modules/core/payload_batcher.md index 34bb2ee363..8df1ea2766 100644 --- a/docs/source/modules/core/payload_batcher.md +++ b/docs/source/modules/core/payload_batcher.md @@ -23,13 +23,13 @@ This module batches incoming control message data payload into smaller batches b | Parameter | Type | Description | Example Value | Default Value | |-----------------------------|------------|-----------------------------------|---------------------------------|---------------| -| `max_batch_size` | int | The maximum size of each batch | 256 | `256` | -| `raise_on_failure` | bool | Whether to raise an exception if a failure occurs during processing | false | `false` | +| `max_batch_size` | integer | The maximum size of each batch | 256 | `256` | +| `raise_on_failure` | boolean | Whether to raise an exception if a failure occurs during processing | false | `false` | | `group_by_columns` | list | The column names to group by when batching | ["col1", "col2"] | `[]` | -| `disable_max_batch_size` | bool | Whether to disable the `max_batch_size` and only batch by group | false | `false` | -| `timestamp_column_name` | str | The name of the timestamp column | None | `None` | -| `timestamp_pattern` | str | The pattern to parse the timestamp column | None | `None` | -| `period` | str | The period for grouping by timestamp | H | `D` | +| `disable_max_batch_size` | boolean | Whether to disable the `max_batch_size` and only batch by group | false | `false` | +| `timestamp_column_name` | string | The name of the timestamp column | None | `None` | +| `timestamp_pattern` | string | The pattern to parse the timestamp column | None | `None` | +| `period` | string | The period for grouping by timestamp | H | `D` | ### Example JSON Configuration From cde3d8fb17601d49ba755d72c05a12c3bf75fc6b Mon Sep 17 00:00:00 2001 From: Bhargav Suryadevara Date: Wed, 12 Jul 2023 10:18:32 -0500 Subject: [PATCH 2/3] updated module docs --- docs/source/loaders/morpheus_loaders.md | 3 +- .../spear_phishing/sp_email_enrichment.md | 6 ++-- .../spear_phishing/sp_inference_intent.md | 22 +++++++------- .../sp_inference_sp_classifier.md | 6 ++-- .../spear_phishing/sp_label_and_score.md | 4 +-- .../spear_phishing/sp_preprocessing.md | 6 ++-- .../sp_sender_sketch_aggregator.md | 4 +-- .../sp_sender_sketch_query_constructor.md | 4 +-- .../spear_phishing/sp_sender_sketch_update.md | 4 +-- .../sp_spear_phishing_post_inference.md | 28 ++++++++--------- .../sp_spear_phishing_pre_inference.md | 30 +++++++++---------- docs/source/modules/morpheus_modules.md | 1 + 12 files changed, 59 insertions(+), 59 deletions(-) diff --git a/docs/source/loaders/morpheus_loaders.md b/docs/source/loaders/morpheus_loaders.md index 3558ef8dbb..5a74426efb 100644 --- a/docs/source/loaders/morpheus_loaders.md +++ b/docs/source/loaders/morpheus_loaders.md @@ -21,8 +21,7 @@ Custom functions called "Loaders" can be utilized by the DataLoader Module to lo choose to register their own customized loader function and add it to a dataloader registry, which will then become accessible to the DataLoader module during module loading. -**Note** : Loaders receive configuration from the `load` task -via [control message](../../source/control_message_guide.md) during runtime. +**Note** : Loaders receive configuration from the `load` task via control message during runtime. ## Core Loaders diff --git a/docs/source/modules/examples/spear_phishing/sp_email_enrichment.md b/docs/source/modules/examples/spear_phishing/sp_email_enrichment.md index 6cc25cc4a3..89106aaf13 100644 --- a/docs/source/modules/examples/spear_phishing/sp_email_enrichment.md +++ b/docs/source/modules/examples/spear_phishing/sp_email_enrichment.md @@ -28,8 +28,8 @@ This module performs spear phishing email enrichment. |--------------------------|------|---------------------------------------------------------------------|------------------------|---------------| | `sender_sketches` | list | List of sender strings naming sender sketch inputs. | ["sender1", "sender2"] | `[]` | | `intents` | list | List of intent strings naming computed intent inputs. | ["intent1", "intent2"] | `[]` | -| `raise_on_failure` | bool | Indicate if we should treat processing errors as pipeline failures. | false | `false` | -| `token_length_threshold` | int | Minimum token length to use when computing syntax similarity | 5 | None | +| `raise_on_failure` | boolean | Indicate if we should treat processing errors as pipeline failures. | false | `false` | +| `token_length_threshold` | integer | Minimum token length to use when computing syntax similarity | 5 | None | ### Example JSON Configuration @@ -39,4 +39,4 @@ This module performs spear phishing email enrichment. "intents": ["intent1", "intent2"], "raise_on_failure": false, "token_length_threshold": 5 -} \ No newline at end of file +} diff --git a/docs/source/modules/examples/spear_phishing/sp_inference_intent.md b/docs/source/modules/examples/spear_phishing/sp_inference_intent.md index 7267ad8bd2..1e47267a3e 100644 --- a/docs/source/modules/examples/spear_phishing/sp_inference_intent.md +++ b/docs/source/modules/examples/spear_phishing/sp_inference_intent.md @@ -26,16 +26,16 @@ Infers an 'intent' for a given email body. | Parameter | Type | Description | Example Value | Default Value | |--------------------|------|-----------------------------------------|-----------------------|-------------------------| -| `intent` | str | The intent for the model | "classify" | `None` | -| `task` | str | The task for the model | "text-classification" | `"text-classification"` | -| `model_path` | str | The path to the model | "/path/to/model" | `None` | -| `truncation` | bool | If true, truncates inputs to max_length | true | `true` | -| `max_length` | int | Maximum length for model input | 512 | `512` | -| `batch_size` | int | The size of batches for processing | 256 | `256` | -| `feature_col` | str | The feature column to use | "body" | `"body"` | -| `label_col` | str | The label column to use | "label" | `"label"` | -| `device` | int | The device to run on | 0 | `0` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `intent` | string | The intent for the model | "classify" | `None` | +| `task` | string | The task for the model | "text-classification" | `"text-classification"` | +| `model_path` | string | The path to the model | "/path/to/model" | `None` | +| `truncation` | boolean | If true, truncates inputs to max_length | true | `true` | +| `max_length` | integer | Maximum length for model input | 512 | `512` | +| `batch_size` | integer | The size of batches for processing | 256 | `256` | +| `feature_col` | string | The feature column to use | "body" | `"body"` | +| `label_col` | string | The label column to use | "label" | `"label"` | +| `device` | integer | The device to run on | 0 | `0` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration @@ -51,4 +51,4 @@ Infers an 'intent' for a given email body. "label_col": "label", "device": 0, "raise_on_failure": false -} \ No newline at end of file +} diff --git a/docs/source/modules/examples/spear_phishing/sp_inference_sp_classifier.md b/docs/source/modules/examples/spear_phishing/sp_inference_sp_classifier.md index e1844a20de..11d8c811a4 100644 --- a/docs/source/modules/examples/spear_phishing/sp_inference_sp_classifier.md +++ b/docs/source/modules/examples/spear_phishing/sp_inference_sp_classifier.md @@ -26,10 +26,10 @@ This module defines a setup for spear-phishing inference. | Parameter | Type | Description | Example Value | Default Value | |------------------------|------|---------------------------------------|--------------------|---------------| -| `tracking_uri` | str | The tracking URI for the model | "/path/to/uri" | `None` | -| `registered_model` | str | The registered model for inference | "model_1" | `None` | +| `tracking_uri` | string | The tracking URI for the model | "/path/to/uri" | `None` | +| `registered_model` | string | The registered model for inference | "model_1" | `None` | | `input_model_features` | list | The input features for the model | ["feat1", "feat2"] | `[]` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration diff --git a/docs/source/modules/examples/spear_phishing/sp_label_and_score.md b/docs/source/modules/examples/spear_phishing/sp_label_and_score.md index d66003ecc3..048f3556d9 100644 --- a/docs/source/modules/examples/spear_phishing/sp_label_and_score.md +++ b/docs/source/modules/examples/spear_phishing/sp_label_and_score.md @@ -26,8 +26,8 @@ This module defines a setup for spear-phishing email scoring. | Parameter | Type | Description | Example Value | Default Value | |--------------------|------|---------------------------------------|---------------------------|---------------| -| `scoring_config` | dict | The scoring configuration | {"method": "probability"} | `None` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `scoring_config` | dictionary | The scoring configuration | {"method": "probability"} | `None` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration diff --git a/docs/source/modules/examples/spear_phishing/sp_preprocessing.md b/docs/source/modules/examples/spear_phishing/sp_preprocessing.md index 0d8fa1a6bf..7444a94f6b 100644 --- a/docs/source/modules/examples/spear_phishing/sp_preprocessing.md +++ b/docs/source/modules/examples/spear_phishing/sp_preprocessing.md @@ -26,8 +26,8 @@ This module defines a pre-processing setup for the spear phishing inference pipe | Parameter | Type | Description | Example Value | Default Value | |--------------------|------|---------------------------------------------------|---------------|---------------| -| `attach_uuid` | bool | If true, attach a unique identifier to each input | true | `false` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `attach_uuid` | boolean | If true, attach a unique identifier to each input | true | `false` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration @@ -35,4 +35,4 @@ This module defines a pre-processing setup for the spear phishing inference pipe { "attach_uuid": false, "raise_on_failure": false -} \ No newline at end of file +} diff --git a/docs/source/modules/examples/spear_phishing/sp_sender_sketch_aggregator.md b/docs/source/modules/examples/spear_phishing/sp_sender_sketch_aggregator.md index 6505b3397f..9ad5690560 100644 --- a/docs/source/modules/examples/spear_phishing/sp_sender_sketch_aggregator.md +++ b/docs/source/modules/examples/spear_phishing/sp_sender_sketch_aggregator.md @@ -33,7 +33,7 @@ This module aggregates sender sketches in the spear phishing detection pipeline. | Key | Type | Description | Example Value | Default Value | |--------------------|-------|------------------------------------------|------------------------|---------------| | `sender_sketches` | array | The list of sender sketches to aggregate | ["sketch1", "sketch2"] | `[]` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration @@ -43,4 +43,4 @@ This module aggregates sender sketches in the spear phishing detection pipeline. "sender_sketches": [], "raise_on_failure": false } -} \ No newline at end of file +} diff --git a/docs/source/modules/examples/spear_phishing/sp_sender_sketch_query_constructor.md b/docs/source/modules/examples/spear_phishing/sp_sender_sketch_query_constructor.md index 487efe7577..275039db67 100644 --- a/docs/source/modules/examples/spear_phishing/sp_sender_sketch_query_constructor.md +++ b/docs/source/modules/examples/spear_phishing/sp_sender_sketch_query_constructor.md @@ -27,11 +27,11 @@ This module constructs SQL sender sketch queries in the spear phishing detection | Parameter | Type | Description | Example Value | Default Value | |--------------------|------|---------------------------------------|---------------|---------------| -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration ```json { "raise_on_failure": false -} \ No newline at end of file +} diff --git a/docs/source/modules/examples/spear_phishing/sp_sender_sketch_update.md b/docs/source/modules/examples/spear_phishing/sp_sender_sketch_update.md index 19d8005c5c..3581558b94 100644 --- a/docs/source/modules/examples/spear_phishing/sp_sender_sketch_update.md +++ b/docs/source/modules/examples/spear_phishing/sp_sender_sketch_update.md @@ -36,7 +36,7 @@ This module updates the sender sketch for spear phishing detection. | `endpoint` | string | The endpoint configuration | "http://my-endpoint.com" | `None` | | `required_intents` | list | List of required intents | ["intent1", "intent2"] | `[]` | | `sender_sketch_tables_config` | dictionary | Configuration for sender sketch tables | {"table1": "config1"} | `{}` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` | ### Example JSON Configuration @@ -48,4 +48,4 @@ This module updates the sender sketch for spear phishing detection. "sender_sketch_tables_config": {"table1": "config1"}, "raise_on_failure": false } -} \ No newline at end of file +} diff --git a/docs/source/modules/examples/spear_phishing/sp_spear_phishing_post_inference.md b/docs/source/modules/examples/spear_phishing/sp_spear_phishing_post_inference.md index 55d54f0206..e4980d54a0 100644 --- a/docs/source/modules/examples/spear_phishing/sp_spear_phishing_post_inference.md +++ b/docs/source/modules/examples/spear_phishing/sp_spear_phishing_post_inference.md @@ -25,37 +25,37 @@ label and score module, updates the sender sketch, and prepares the final output ## Configurable Parameters -| Parameter | Type | Description | +| Parameter | Type | Description | |------------------------|------|--------------------------------------------------------------------------------------------------------------| -| `scoring_config` | dict | Configuration for scoring, can include custom parameters for the scoring module. See below for more details. | -| `sender_sketch_config` | dict | Configuration for sender sketch module, including parameters such as endpoint details and sketch settings. | +| `scoring_config` | dictionary | Configuration for scoring, can include custom parameters for the scoring module. See below for more details. | +| `sender_sketch_config` | dictionary | Configuration for sender sketch module, including parameters such as endpoint details and sketch settings. | #### `scoring_config` | Key | Type | Description | |--------------------|-------|--------------------------------------------------------------------| | `threshold` | float | Detection threshold for scoring. | -| `scoring_type` | str | Type of scoring to use. Currently only "probability" is supported. | -| `raise_on_failure` | bool | If true, raise exceptions on failures. Default is False. | +| `scoring_type` | string | Type of scoring to use. Currently only "probability" is supported. | +| `raise_on_failure` | boolean | If true, raise exceptions on failures. Default is False. | #### `sender_sketch_config` | Key | Type | Description | Default Value | |-------------------------------|------|--------------------------------------------------------------|---------------| -| `endpoint` | dict | See `endpoint` subparameters | `None` | +| `endpoint` | dictionary | See `endpoint` subparameters | `None` | | `sender_sketches` | list | List of sender sketches | `[]` | | `required_intents` | list | List of required intents | `[]` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | `False` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | `False` | | `token_length_threshold` | int | Minimum token length to use when computing syntax similarity | `3` | -| `sender_sketch_tables_config` | dict | Configuration for sender sketch tables | `None` | +| `sender_sketch_tables_config` | dictionary | Configuration for sender sketch tables | `None` | ##### `endpoint` | Key | Type | Description | |--------------|------|--------------------------------------------| -| `database` | str | Sender sketch database name | -| `drivername` | str | Driver name for the sender sketch database | -| `host` | str | Host of the sender sketch database | -| `port` | str | Port of the sender sketch database | -| `username` | str | Username for the sender sketch database | -| `password` | str | Password for the sender sketch database | +| `database` | string | Sender sketch database name | +| `drivername` | string | Driver name for the sender sketch database | +| `host` | string | Host of the sender sketch database | +| `port` | string | Port of the sender sketch database | +| `username` | string | Username for the sender sketch database | +| `password` | string | Password for the sender sketch database | diff --git a/docs/source/modules/examples/spear_phishing/sp_spear_phishing_pre_inference.md b/docs/source/modules/examples/spear_phishing/sp_spear_phishing_pre_inference.md index aa9eb8f475..35d91f521f 100644 --- a/docs/source/modules/examples/spear_phishing/sp_spear_phishing_pre_inference.md +++ b/docs/source/modules/examples/spear_phishing/sp_spear_phishing_pre_inference.md @@ -27,36 +27,36 @@ required connections between modules. | Parameter | Type | Description | Default Value | |------------------------|------|------------------------------------------|---------------| -| `raise_on_failure` | bool | If true, raise exceptions on failures | `False` | -| `max_batch_size` | int | Maximum size of each batch | `500` | -| `intent_config` | dict | See `intent_config` subparameters | `{}` | -| `sender_sketch_config` | dict | See `sender_sketch_config` subparameters | `None` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | `False` | +| `max_batch_size` | integer | Maximum size of each batch | `500` | +| `intent_config` | dictionary | See `intent_config` subparameters | `{}` | +| `sender_sketch_config` | dictionary | See `sender_sketch_config` subparameters | `None` | #### `intent_config` | Key | Type | Description | Default Value | |---------------------|------|---------------------------------|---------------| | `required_intents` | list | List of required intents | `[]` | -| `available_intents` | dict | Dictionary of available intents | `{}` | +| `available_intents` | dictionary | Dictionary of available intents | `{}` | #### `sender_sketch_config` | Key | Type | Description | Default Value | |-------------------------------|------|--------------------------------------------------------------|---------------| -| `endpoint` | dict | See `endpoint` subparameters | `None` | +| `endpoint` | dictionary | See `endpoint` subparameters | `None` | | `sender_sketches` | list | List of sender sketches | `[]` | | `required_intents` | list | List of required intents | `[]` | -| `raise_on_failure` | bool | If true, raise exceptions on failures | `False` | -| `token_length_threshold` | int | Minimum token length to use when computing syntax similarity | `3` | -| `sender_sketch_tables_config` | dict | Configuration for sender sketch tables | `None` | +| `raise_on_failure` | boolean | If true, raise exceptions on failures | `False` | +| `token_length_threshold` | integer | Minimum token length to use when computing syntax similarity | `3` | +| `sender_sketch_tables_config` | dictionary | Configuration for sender sketch tables | `None` | ##### `endpoint` | Key | Type | Description | |--------------|------|--------------------------------------------| -| `database` | str | Sender sketch database name | -| `drivername` | str | Driver name for the sender sketch database | -| `host` | str | Host of the sender sketch database | -| `port` | str | Port of the sender sketch database | -| `username` | str | Username for the sender sketch database | -| `password` | str | Password for the sender sketch database | +| `database` | string | Sender sketch database name | +| `drivername` | string | Driver name for the sender sketch database | +| `host` | string | Host of the sender sketch database | +| `port` | string | Port of the sender sketch database | +| `username` | string | Username for the sender sketch database | +| `password` | string | Password for the sender sketch database | diff --git a/docs/source/modules/morpheus_modules.md b/docs/source/modules/morpheus_modules.md index b59fa9d694..d2ef4bc6f4 100644 --- a/docs/source/modules/morpheus_modules.md +++ b/docs/source/modules/morpheus_modules.md @@ -26,6 +26,7 @@ limitations under the License. - [Filter Detections](./core/filter_detections.md) - [MLflow Model Writer](./core/mlflow_model_writer.md) - [Multiplexer](./core/multiplexer.md) +- [Payload Batcher](./core/payload_batcher.md) - [Serializer](./core/serializer.md) - [To Control Message](./core/to_control_message.md) - [Write to File](./core/write_to_file.md) From cdd3e666f47aefa46b9efe2c3bdce33d1c761f2d Mon Sep 17 00:00:00 2001 From: Bhargav Suryadevara Date: Wed, 12 Jul 2023 10:33:21 -0500 Subject: [PATCH 3/3] updated module docs --- morpheus/utils/execution_chain.py | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/morpheus/utils/execution_chain.py b/morpheus/utils/execution_chain.py index f2aea2ad64..0a88bd471e 100644 --- a/morpheus/utils/execution_chain.py +++ b/morpheus/utils/execution_chain.py @@ -15,27 +15,6 @@ import inspect import typing -# def execution_chain_annotations(*annotations): -# def inner_annotate(func): -# def wrapper(*args, **kwargs): -# _annocations = list(*annotations) -# -# result = func(*args, **kwargs) -# signature = inspect.signature(func) -# -# return {key: value for key, value in result.items() if key in signature.return_annotation} -# -# return wrapper -# -# return inner_annotate -# -# -# @execution_chain_annotations("x", "y", "z") -# def myfunc(a, b, c) -> (int, int, int): -# x, y, z = 1, 2, 3 -# -# return x, y, z - class ExecutionChain: """