QRunIO/qqq
2
0
Fork 0
mirror of https://github.com/Kingsrook/qqq.git synced 2025-07-18 05:01:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Complete migration of run to runOneStep in streamed-ETL load & transform steps

Browse Source
This commit is contained in:
darin.kelkhoff
2024-07-11 08:36:33 -05:00
parent 6cae86b6c9
commit 881ce8514e
5 changed files with 17 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/misc/ProcessBackendSteps.adoc
View File

@ -128,15 +128,15 @@ These steps are:
** The Extract step is called before the Preview, Validate, and Result screens, though for the Preview screen, it is set to only extract a small number of records (10).
* *Transform* - a subclass of `AbstractTransformStep` - is responsible for applying the majority of the business logic of the process.
In ETL terminology, this is the "Transform" action - which means applying some type of logical transformation an input record (found by the Extract step) to generate an output record (stored by the Load step).
** A Transform step's `run` method will be called, potentially, multiple times, each time with a page of records in the `runBackendStepInput` parameter.
** A Transform step's `runOnePage` method will be called, potentially, multiple times, each time with a page of records in the `runBackendStepInput` parameter.
** This method is responsible for adding records to the `runBackendStepOutput`, which will then be passed to the *Load* step.
** This class is also responsible for implementing the method `getProcessSummary`, which provides the data to the *Validate* screen.
** The run method will generally update ProcessSummaryLine objects to facilitate this functionality.
** The `runOnePage` method will generally update ProcessSummaryLine objects to facilitate this functionality.
** The Transform step is called before the Preview, Validate, and Result screens, consuming all records selected by the Extract step.
* *Load* - a subclass of `AbstractLoadStep` - is responsible for the Load function of the ETL job.
_A quick word on terminology - this step is actually doing what we are more likely to think of as storing data - which feels like the opposite of “loading” - but we use the name Load to keep in line with the ETL naming convention…_
** The Load step is ONLY called before the Result screen is presented (possibly after Preview, if the user chose to skip validation, otherwise, after validation).
** Similar to the Transform step, the Load step's `run` method will be called potentially multiple times, with pages of records in its input.
** Similar to the Transform step, the Load step's `runOnePage` method will be called potentially multiple times, with pages of records in its input.
** As such, the Load step is generally the only step where data writes should occur.
*** e.g., a Transform step should not do any writes, as it will be called when the user is going to the Preview & Validate screens - e.g., before the user confirmed that they want to execute the action!
** A common pattern is that the Load step just needs to insert or update the list of records output by the Transform step, in which case the QQQ-provided `LoadViaInsertStep` or `LoadViaUpdateStep` can be used, but custom use-cases can be built as well.