Imports configuration

Synthese can manage data import from the command line and through a web interface (not yet implemented), offering the following features:
– run imports automatically from a script
– email notification of import failures or warnings
– save archives of the import logs and results
– web interface for uploading files to the Synthese server
– web interface for running imports, querying old import logs, …

Concepts

Import template: defines default parameters and other configuration related to a specific import.

Import (0:n Import in an Import Tepmlate): an instance of an import template, containing the files to import and optionnaly the parameters that differ from the tmeplate defaults

Import Run (0:n Import Run in an Import): execution of an import. Contains import logs and statistics.

Configuration

The configuration is done in a file manager/imports_config.py inside the project.

Example configuration

import_templates = [{
    'id': 'my_import',
    'label': 'My Import (Trident)',
    'defaults': [
        ('ds', '16607027920896001'),
        ('itt', '1'),
        ('mr', '1'),
        ('impstp', '1'),
        ('asa', '1'),
        ('impjun', '0'),
        ('dtd', '5'),
        {
            'id': 'dr',
            'label': 'Directory containing files to import',
            'type': 'directory',
            'default': 'files',
        },
        ('clean_old_data', '1'),
        ('from_today', '1'),
    ],
    'admins': [('root', '[email protected]')],
    'uploaders': [('uploader', ['[email protected]'])],

    'run_results_notifications': {
         'load': ('[email protected]',),
         'warn': ('[email protected]',),
     },
}]

Configuration options

• run_results_notifications:
  This contains a Python dict which specifies what users should receive an email depending on what error level appears in the import logs.
  For instance, in the example above:
  [email protected] would get an email if the import has messages with the load level.
  [email protected] and [email protected] would get an email of the import has messages with the warn level

Command line interface

List templates:
sy -crel,demo imports --cmd list_templates

Create a new import:
sy -crel,demo imports --cmd create --template-id my_import

Execute an import
sy -crel,demo imports --cmd execute --template-id my_import --import-id :latest