Skip to content

Commit

Permalink
Merge pull request #15 from bsuryadevara/doc-updates
Browse files Browse the repository at this point in the history 
Doc updates
  • Loading branch information
@drobison00
drobison00 authored Jul 12, 2023
2 parents e7b4c6b + cdd3e66 commit 8e5c1bf
Show file tree
Hide file tree
Showing 15 changed files with 69 additions and 91 deletions.
9 changes: 4 additions & 5 deletions docs/source/loaders/core/sql_loader.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,8 @@ limitations under the License.

## SQL Loader

[DataLoader](../../modules/core/data_loader.md) module is configured to use this loader function. It is responsible for
loading data from external sources using the fsspec library, and returns the updated ControlMessage object with payload
as MessageMeta, which contains dataframe (with filenames).
[DataLoader](../../modules/core/data_loader.md) module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload
as MessageMeta.

### Example Loader Configuration

Expand All @@ -44,7 +43,7 @@ The parameters that can be configured for this specific loader at load task leve
|--------------|--------|------------------------------------------|--------------------|---------------|
| `strategy` | string | Strategy for combining queries | "aggregate" | `aggregate` |
| `loader_id` | string | Unique identifier for the loader | "file_to_df" | `[Required]` |
| `sql_config` | dict | Dictionary containing SQL queries to run | "file_to_df" | `See below` |
| `sql_config` | dictionary | Dictionary containing SQL queries to run | "file_to_df" | `See below` |

`sql_config`

Expand All @@ -58,7 +57,7 @@ The parameters that can be configured for this specific loader at load task leve
|---------------------|--------|--------------------------------------|-----------------------------------------------------------------|---------------|
| `connection_string` | string | Strategy for combining queries | "postgresql://postgres:postgres@localhost:5432/postgres" | `[required]` |
| `query` | string | SQL Query to execute | "SELECT * FROM test_table WHERE id IN (?, ?, ?)" | `[Required]` |
| `params` | dict | Named or positional paramters values | "[foo, bar, baz]" | `-` |
| `params` | dictionary | Named or positional paramters values | "[foo, bar, baz]" | `-` |

### Example Load Task Configuration

Expand Down
3 changes: 1 addition & 2 deletions docs/source/loaders/morpheus_loaders.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,7 @@ Custom functions called "Loaders" can be utilized by the DataLoader Module to lo
choose to register their own customized loader function and add it to a dataloader registry, which will then become
accessible to the DataLoader module during module loading.

**Note** : Loaders receive configuration from the `load` task
via [control message](../../source/control_message_guide.md) during runtime.
**Note** : Loaders receive configuration from the `load` task via control message during runtime.

## Core Loaders

Expand Down
12 changes: 6 additions & 6 deletions docs/source/modules/core/payload_batcher.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ This module batches incoming control message data payload into smaller batches b

| Parameter | Type | Description | Example Value | Default Value |
|-----------------------------|------------|-----------------------------------|---------------------------------|---------------|
| `max_batch_size` | int | The maximum size of each batch | 256 | `256` |
| `raise_on_failure` | bool | Whether to raise an exception if a failure occurs during processing | false | `false` |
| `max_batch_size` | integer | The maximum size of each batch | 256 | `256` |
| `raise_on_failure` | boolean | Whether to raise an exception if a failure occurs during processing | false | `false` |
| `group_by_columns` | list | The column names to group by when batching | ["col1", "col2"] | `[]` |
| `disable_max_batch_size` | bool | Whether to disable the `max_batch_size` and only batch by group | false | `false` |
| `timestamp_column_name` | str | The name of the timestamp column | None | `None` |
| `timestamp_pattern` | str | The pattern to parse the timestamp column | None | `None` |
| `period` | str | The period for grouping by timestamp | H | `D` |
| `disable_max_batch_size` | boolean | Whether to disable the `max_batch_size` and only batch by group | false | `false` |
| `timestamp_column_name` | string | The name of the timestamp column | None | `None` |
| `timestamp_pattern` | string | The pattern to parse the timestamp column | None | `None` |
| `period` | string | The period for grouping by timestamp | H | `D` |


### Example JSON Configuration
Expand Down
6 changes: 3 additions & 3 deletions docs/source/modules/examples/spear_phishing/sp_email_enrichment.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ This module performs spear phishing email enrichment.
|--------------------------|------|---------------------------------------------------------------------|------------------------|---------------|
| `sender_sketches` | list | List of sender strings naming sender sketch inputs. | ["sender1", "sender2"] | `[]` |
| `intents` | list | List of intent strings naming computed intent inputs. | ["intent1", "intent2"] | `[]` |
| `raise_on_failure` | bool | Indicate if we should treat processing errors as pipeline failures. | false | `false` |
| `token_length_threshold` | int | Minimum token length to use when computing syntax similarity | 5 | None |
| `raise_on_failure` | boolean | Indicate if we should treat processing errors as pipeline failures. | false | `false` |
| `token_length_threshold` | integer | Minimum token length to use when computing syntax similarity | 5 | None |

### Example JSON Configuration

Expand All @@ -39,4 +39,4 @@ This module performs spear phishing email enrichment.
"intents": ["intent1", "intent2"],
"raise_on_failure": false,
"token_length_threshold": 5
}
}
22 changes: 11 additions & 11 deletions docs/source/modules/examples/spear_phishing/sp_inference_intent.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,16 @@ Infers an 'intent' for a given email body.

| Parameter | Type | Description | Example Value | Default Value |
|--------------------|------|-----------------------------------------|-----------------------|-------------------------|
| `intent` | str | The intent for the model | "classify" | `None` |
| `task` | str | The task for the model | "text-classification" | `"text-classification"` |
| `model_path` | str | The path to the model | "/path/to/model" | `None` |
| `truncation` | bool | If true, truncates inputs to max_length | true | `true` |
| `max_length` | int | Maximum length for model input | 512 | `512` |
| `batch_size` | int | The size of batches for processing | 256 | `256` |
| `feature_col` | str | The feature column to use | "body" | `"body"` |
| `label_col` | str | The label column to use | "label" | `"label"` |
| `device` | int | The device to run on | 0 | `0` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `intent` | string | The intent for the model | "classify" | `None` |
| `task` | string | The task for the model | "text-classification" | `"text-classification"` |
| `model_path` | string | The path to the model | "/path/to/model" | `None` |
| `truncation` | boolean | If true, truncates inputs to max_length | true | `true` |
| `max_length` | integer | Maximum length for model input | 512 | `512` |
| `batch_size` | integer | The size of batches for processing | 256 | `256` |
| `feature_col` | string | The feature column to use | "body" | `"body"` |
| `label_col` | string | The label column to use | "label" | `"label"` |
| `device` | integer | The device to run on | 0 | `0` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

Expand All @@ -51,4 +51,4 @@ Infers an 'intent' for a given email body.
"label_col": "label",
"device": 0,
"raise_on_failure": false
}
}
6 changes: 3 additions & 3 deletions docs/source/modules/examples/spear_phishing/sp_inference_sp_classifier.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,10 @@ This module defines a setup for spear-phishing inference.

| Parameter | Type | Description | Example Value | Default Value |
|------------------------|------|---------------------------------------|--------------------|---------------|
| `tracking_uri` | str | The tracking URI for the model | "/path/to/uri" | `None` |
| `registered_model` | str | The registered model for inference | "model_1" | `None` |
| `tracking_uri` | string | The tracking URI for the model | "/path/to/uri" | `None` |
| `registered_model` | string | The registered model for inference | "model_1" | `None` |
| `input_model_features` | list | The input features for the model | ["feat1", "feat2"] | `[]` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

Expand Down
4 changes: 2 additions & 2 deletions docs/source/modules/examples/spear_phishing/sp_label_and_score.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ This module defines a setup for spear-phishing email scoring.

| Parameter | Type | Description | Example Value | Default Value |
|--------------------|------|---------------------------------------|---------------------------|---------------|
| `scoring_config` | dict | The scoring configuration | {"method": "probability"} | `None` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `scoring_config` | dictionary | The scoring configuration | {"method": "probability"} | `None` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

Expand Down
6 changes: 3 additions & 3 deletions docs/source/modules/examples/spear_phishing/sp_preprocessing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@ This module defines a pre-processing setup for the spear phishing inference pipe

| Parameter | Type | Description | Example Value | Default Value |
|--------------------|------|---------------------------------------------------|---------------|---------------|
| `attach_uuid` | bool | If true, attach a unique identifier to each input | true | `false` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `attach_uuid` | boolean | If true, attach a unique identifier to each input | true | `false` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

```json
{
"attach_uuid": false,
"raise_on_failure": false
}
}
4 changes: 2 additions & 2 deletions docs/source/modules/examples/spear_phishing/sp_sender_sketch_aggregator.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ This module aggregates sender sketches in the spear phishing detection pipeline.
| Key | Type | Description | Example Value | Default Value |
|--------------------|-------|------------------------------------------|------------------------|---------------|
| `sender_sketches` | array | The list of sender sketches to aggregate | ["sketch1", "sketch2"] | `[]` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

Expand All @@ -43,4 +43,4 @@ This module aggregates sender sketches in the spear phishing detection pipeline.
"sender_sketches": [],
"raise_on_failure": false
}
}
}
4 changes: 2 additions & 2 deletions docs/source/modules/examples/spear_phishing/sp_sender_sketch_query_constructor.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ This module constructs SQL sender sketch queries in the spear phishing detection

| Parameter | Type | Description | Example Value | Default Value |
|--------------------|------|---------------------------------------|---------------|---------------|
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

```json
{
"raise_on_failure": false
}
}
4 changes: 2 additions & 2 deletions docs/source/modules/examples/spear_phishing/sp_sender_sketch_update.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ This module updates the sender sketch for spear phishing detection.
| `endpoint` | string | The endpoint configuration | "http://my-endpoint.com" | `None` |
| `required_intents` | list | List of required intents | ["intent1", "intent2"] | `[]` |
| `sender_sketch_tables_config` | dictionary | Configuration for sender sketch tables | {"table1": "config1"} | `{}` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | false | `false` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | false | `false` |

### Example JSON Configuration

Expand All @@ -48,4 +48,4 @@ This module updates the sender sketch for spear phishing detection.
"sender_sketch_tables_config": {"table1": "config1"},
"raise_on_failure": false
}
}
}
28 changes: 14 additions & 14 deletions docs/source/modules/examples/spear_phishing/sp_spear_phishing_post_inference.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,37 +25,37 @@ label and score module, updates the sender sketch, and prepares the final output

## Configurable Parameters

| Parameter | Type | Description |
| Parameter | Type | Description |
|------------------------|------|--------------------------------------------------------------------------------------------------------------|
| `scoring_config` | dict | Configuration for scoring, can include custom parameters for the scoring module. See below for more details. |
| `sender_sketch_config` | dict | Configuration for sender sketch module, including parameters such as endpoint details and sketch settings. |
| `scoring_config` | dictionary | Configuration for scoring, can include custom parameters for the scoring module. See below for more details. |
| `sender_sketch_config` | dictionary | Configuration for sender sketch module, including parameters such as endpoint details and sketch settings. |

#### `scoring_config`

| Key | Type | Description |
|--------------------|-------|--------------------------------------------------------------------|
| `threshold` | float | Detection threshold for scoring. |
| `scoring_type` | str | Type of scoring to use. Currently only "probability" is supported. |
| `raise_on_failure` | bool | If true, raise exceptions on failures. Default is False. |
| `scoring_type` | string | Type of scoring to use. Currently only "probability" is supported. |
| `raise_on_failure` | boolean | If true, raise exceptions on failures. Default is False. |

#### `sender_sketch_config`

| Key | Type | Description | Default Value |
|-------------------------------|------|--------------------------------------------------------------|---------------|
| `endpoint` | dict | See `endpoint` subparameters | `None` |
| `endpoint` | dictionary | See `endpoint` subparameters | `None` |
| `sender_sketches` | list | List of sender sketches | `[]` |
| `required_intents` | list | List of required intents | `[]` |
| `raise_on_failure` | bool | If true, raise exceptions on failures | `False` |
| `raise_on_failure` | boolean | If true, raise exceptions on failures | `False` |
| `token_length_threshold` | int | Minimum token length to use when computing syntax similarity | `3` |
| `sender_sketch_tables_config` | dict | Configuration for sender sketch tables | `None` |
| `sender_sketch_tables_config` | dictionary | Configuration for sender sketch tables | `None` |

##### `endpoint`

| Key | Type | Description |
|--------------|------|--------------------------------------------|
| `database` | str | Sender sketch database name |
| `drivername` | str | Driver name for the sender sketch database |
| `host` | str | Host of the sender sketch database |
| `port` | str | Port of the sender sketch database |
| `username` | str | Username for the sender sketch database |
| `password` | str | Password for the sender sketch database |
| `database` | string | Sender sketch database name |
| `drivername` | string | Driver name for the sender sketch database |
| `host` | string | Host of the sender sketch database |
| `port` | string | Port of the sender sketch database |
| `username` | string | Username for the sender sketch database |
| `password` | string | Password for the sender sketch database |
Loading

0 comments on commit 8e5c1bf

Please sign in to comment.