Loaders

Version Date Notes By
0.2 2018-06-01 Adding new loaders and parameters for existing loaders ROB
0.1 2018-01-23 Initial release ROB

Loaders are the operations that insert / update items in the database.

LaravelTableLoader

This table loader is similar to the original table loader (next loader bellow) but it uses Laravel Query Builder instead of the PDO object of the Etl package. This loader returns the items to the job (possibly with the ids it inserted with) so that further transformations and loaders may be chained to it.

Parameters

Name Type Description
connection String name of the connection to load the information to
include_columns Array - String array of columns to load (excludes all others from loading), set to null to load everything
primary_table Boolean if this is set, the loader will append to each row the newly created id for further transformation / loading.
wipe_table Boolean if this is set, the table will be cleaned before insertion of data.
batch_insert Boolean if this is set, the data will be batch inserted (false will insert the data row by row).
batch_insert_count Integer This controls how many rows are in each batch (default is 150)
store_id_diff Boolean if this is set, the difference between new and old ids will be stored in the sequence to be used by future jobs (using the TransformColumnAccordingToTableSequence)
additional_diff_overrides Array This appends to the stored diff, additional old id / new id pairs (that are entered manually) 
->load('App\Etl\Loaders\LaravelTableLoader', 't00_functions', [
    'connection'      => 'destination_db', // connection to store in
    'include_columns' => [ // you may use this parameter to select the columns to insert
        'status_id'
    ],
    'primary_table'   => true // if this is true, then the ids will be preserved in items[$x]['id'] allowing for further chaining of transformers / loaders
    'additional_diff_overrides' => [
        [ // this is the accepted format for diff overrides
            'old_id' => 1,
            'new_id' => 1
        ]
    ]
])

LaravelUpdaterLoader

This loader, instead of batch inserting new information into a destination table, it performs a single update per item.

Parameters

Name Type Description
connection String name of the connection to load the information to
include_columns Array - String or null array of columns to update, if omited, all columns in items list will be updated
query_callback Callback($query, $item, $all_item) : $query callback that allows for the update query to be altered, for example by adding a where = condition. The diference between the item and all_item, is that the all_item is not constrained by the include_columns parameter. Ensure you return the query. 
$job
->load('App\Etl\Loaders\LaravelUpdaterLoader', 't01_families', [
    'connection'      => 'destination_db', // connection
    'include_columns' => [ // only these columns will be updated with the values they contain at the moment
         'workflow_id'
    ],
    'query_callback' => function($query, $item, $all_item) { // this callback allows for the altering of the update query
         $query->where('id', '=', $all_item['v9_family_id']);

         return $query;
    }
])

CsvLoader

The csv loader allows for loading info into a .csv file.

Parameters

Name Type Description
separator String The character used as separator for the csv file (default ',')
excel_hinting Boolean If this flag is true, then a small line will be aded to the top of the file to help Excel determine the separator being used
header Array or null Creates a header (using the array passed)

JsonLoader

The json loader allows for the loading into a json source file (eg .json)

(The json loader has no parameters)

$job->extract(
    'App\Etl\Extractors\JsonExtractor', 
    '~/t00_translation_lines.json', // instead of table, enter location of file
    []
)

Be aware of the operating system WeMigrane™ is running on (Windows, Linux, ...) when entering paths for files.

Table (deprecated, do not use)

The table loader loads the info resulting from the transformers into a set destination table / connection:

$job->load('Table', 't00_users', [
    'connection' => 'destination_db'
]);