QRunIO/qqq
2
0
Fork 0
mirror of https://github.com/Kingsrook/qqq.git synced 2025-07-17 20:50:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Complete migration of run to runOneStep in streamed-ETL load & transform steps

Browse Source
This commit is contained in:
darin.kelkhoff
2024-07-11 08:36:33 -05:00
parent 6cae86b6c9
commit 881ce8514e
5 changed files with 17 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/metaData/Processes.adoc
View File

@ -190,8 +190,8 @@ To improve this programmer-interface, an inner `Builder` class exists within `St
*StreamedETLWithFrontendProcess.Builder methods:*
* `withName(String name) - Set the name for the process.
* `withLabel(String label) - Set the label for the process.
* `withName(String name)` - Set the name for the process.
* `withLabel(String label)` - Set the label for the process.
* `withIcon(QIcon icon)` - Set an {link-icon} to be display with the process in the UI.
* `withExtractStepClass(Class<? extends AbstractExtractStep>)` - Define the Extract step for the process.
If no special extraction logic is needed, `ExtractViaQuery.class` is often a reasonable default.

6
docs/misc/ProcessBackendSteps.adoc
View File

@ -128,15 +128,15 @@ These steps are:
** The Extract step is called before the Preview, Validate, and Result screens, though for the Preview screen, it is set to only extract a small number of records (10).
* *Transform* - a subclass of `AbstractTransformStep` - is responsible for applying the majority of the business logic of the process.
In ETL terminology, this is the "Transform" action - which means applying some type of logical transformation an input record (found by the Extract step) to generate an output record (stored by the Load step).
** A Transform step's `run` method will be called, potentially, multiple times, each time with a page of records in the `runBackendStepInput` parameter.
** A Transform step's `runOnePage` method will be called, potentially, multiple times, each time with a page of records in the `runBackendStepInput` parameter.
** This method is responsible for adding records to the `runBackendStepOutput`, which will then be passed to the *Load* step.
** This class is also responsible for implementing the method `getProcessSummary`, which provides the data to the *Validate* screen.
** The run method will generally update ProcessSummaryLine objects to facilitate this functionality.
** The `runOnePage` method will generally update ProcessSummaryLine objects to facilitate this functionality.
** The Transform step is called before the Preview, Validate, and Result screens, consuming all records selected by the Extract step.
* *Load* - a subclass of `AbstractLoadStep` - is responsible for the Load function of the ETL job.
_A quick word on terminology - this step is actually doing what we are more likely to think of as storing data - which feels like the opposite of “loading” - but we use the name Load to keep in line with the ETL naming convention…_
** The Load step is ONLY called before the Result screen is presented (possibly after Preview, if the user chose to skip validation, otherwise, after validation).
** Similar to the Transform step, the Load step's `run` method will be called potentially multiple times, with pages of records in its input.
** Similar to the Transform step, the Load step's `runOnePage` method will be called potentially multiple times, with pages of records in its input.
** As such, the Load step is generally the only step where data writes should occur.
*** e.g., a Transform step should not do any writes, as it will be called when the user is going to the Preview & Validate screens - e.g., before the user confirmed that they want to execute the action!
** A common pattern is that the Load step just needs to insert or update the list of records output by the Transform step, in which case the QQQ-provided `LoadViaInsertStep` or `LoadViaUpdateStep` can be used, but custom use-cases can be built as well.

24
qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/AbstractLoadStep.java
View File

@ -24,6 +24,7 @@ package com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwit
import java.util.Optional;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
@ -51,31 +52,18 @@ public abstract class AbstractLoadStep
/*******************************************************************************
**
** Do the load logic for one page of records
*******************************************************************************/
@Deprecated
public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
runOnePage(runBackendStepInput, runBackendStepOutput);
}
public abstract void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException;
/*******************************************************************************
** todo - make abstract when run is deleted.
*******************************************************************************/
public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
}
/*******************************************************************************
** Allow subclasses to do an action before the run is complete - before any
** pages of records are passed in.
** Allow subclasses to do an action before any pages of records are processed.
*******************************************************************************/
public void preRun(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
this.session = runBackendStepInput.getSession();
this.session = QContext.getQSession();
}

21
qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/AbstractTransformStep.java
View File

@ -46,29 +46,14 @@ public abstract class AbstractTransformStep implements ProcessSummaryProviderInt
/*******************************************************************************
**
** Do the transform logic for one page of records
*******************************************************************************/
@Deprecated
public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
runOnePage(runBackendStepInput, runBackendStepOutput);
}
public abstract void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException;
/*******************************************************************************
** todo - make abstract when run is deleted.
*******************************************************************************/
public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
}
/*******************************************************************************
** Allow subclasses to do an action before the run is complete - before any
** pages of records are passed in.
** Allow subclasses to do an action before any pages of records are processed.
*******************************************************************************/
public void preRun(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{

7
qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidatorTest.java
View File

@ -37,7 +37,6 @@ import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.dashboard.PersonsByCreateDateBarChart;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ParentWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.actions.processes.CancelProcessActionTest;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
@ -2279,7 +2278,7 @@ public class QInstanceValidatorTest extends BaseTest
///////////////////////////////////////////////
// test classes for validating process steps //
///////////////////////////////////////////////
public abstract class TestAbstractClass extends AbstractTransformStep implements BackendStep
public abstract class TestAbstractClass extends AbstractTransformStep
{
public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
@ -2291,7 +2290,7 @@ public class QInstanceValidatorTest extends BaseTest
///////////////////////////////////////////////
// //
///////////////////////////////////////////////
private class TestPrivateClass extends AbstractTransformStep implements BackendStep
private class TestPrivateClass extends AbstractTransformStep
{
public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
{
@ -2311,7 +2310,7 @@ public class QInstanceValidatorTest extends BaseTest
///////////////////////////////////////////////
// //
///////////////////////////////////////////////
public class TestNoArgsConstructorClass extends AbstractTransformStep implements BackendStep
public class TestNoArgsConstructorClass extends AbstractTransformStep
{
public TestNoArgsConstructorClass(int i)
{