== Process Backend Steps
include::../variables.adoc[]

In many QQQ applications, much of the code that engineers write will take the form of Backend Steps for {link-processes}.
Such code is defined in classes which implement the interface `BackendStep`.  
This interface defines only a single method:
[source,java]
.BackendStep.java
----
public interface BackendStep
{
   /*******************************************************************************
    ** Execute the backend step - using the request as input, and the result as output.
    **
    *******************************************************************************/
   void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException;
}

----

Process backend steps have access to state information - specifically, a list of records, and a map of name=value pairs - in the input & output objects.
This state data is persisted by QQQ between steps (e.g., if a frontend step is presented to a user between backend steps).


=== RunBackendStepInput
All input data to the step is available in the `RunBackendStepInput` object.
Key methods in this class are:

* `getRecords()` - Returns the List of <<QRecords>> that are currently being acted on in the process.
* `getValues()` - Returns a Map of String -> Serializable; name=value pairs that are the state-data of the process.
** Values can be added to this state from a process's meta-data, from a screen, or from another backend step.
* `getValue(String fieldName)` - Returns a specific value, by name, from the process's state.
** This method has several variations that return the value as a specific type, such as `getValueString`, `getValueInteger`, `getValueBoolean`...
* `getAsyncJobCallback()` - Accessor for an `AsyncJobCallback` object, which provides a way for the process backend step to communicate about its status or progress with a user running the process in a frontend.
Provides methods:
** `updateStatus(String message)` - Where general status messages can be given.
For example, `"Loading census data"`
** `updateStatus(int current, int total)` - For updating a progress meter.
e.g., "47 of 1701" would be display by calling `.updateStatus(47, 1701)`
* `getFrontendStepBehavior()` - Enum, indicating what should happen when a frontend step is encountered as the process's next step to run.
Possible values are:
** `BREAK` - Indicates that the process's execution should be suspended, so that the screen represented by the frontend step can be presented to a user.
This would be the expected behavior if a process is being run by a user from a UI.
** `SKIP` - Indicates that frontend steps should be skipped.
This would be the expected behavior if a process is running from a scheduled job (without a user present to drive it), for example.
** `FAIL` - Indicates that the process should end with an exception if a frontend step is encountered.
** A backend step may want to act differently based on its frontendStepBehavior.
For example, additional data may be looked up for displaying to a user if the behavior is `BREAK`.
* `getBasepullLastRunTime()` - For <<BasepullConfiguration,Basepull>> processes, this is the `Instant` stored in the basepull table as the process's last run time.

=== RunBackendStepOutput
All output from a process step should be placed in its `RunBackendStepOutput` object (and/or stored to a backend, as appropriate).
Key methods in this class are:

* `addValue(String fieldName, Serializable value)` - Adds a single named value to the process's state, overwriting it the value if it already exists.
* `addRecord(QRecord record)` - Add a `<<QRecord>>` to the process's output.
* `addAuditSingleInput(AuditSingleInput auditSingleInput)` - Add a new entry to the process's list of audit inputs, to be stored at the completion of the process.
** An `AuditSingleInput` object can most easily be built with the constructor: `AuditSingleInput(String tableName, QRecord record, String auditMessage)`.
** Additional audit details messages (sub-bullets that accompany the primary `auditMessage`) can be added to an `AuditSingleInput` via the `addDetail(String message)` method.
** _Note that at this time, the automatic storing of these audits is only provided by the execute step of a StreamedETLWithFrontendProcesses._

=== Example

[source,java]
.Example of a BackendStep
----
/*******************************************************************************
 ** For the "person" table's "Add Age" process -
 ** For each input person record, add the specified yearsToAdd to their age.
 *******************************************************************************/
public class AddAge implements BackendStep
{
   /*******************************************************************************
    **
    *******************************************************************************/
   @Override
   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput)
   {
      /////////////////////////////////////////////////////////////////
      // get the yearsToAdd input field value from the process input //
      /////////////////////////////////////////////////////////////////
      Integer yearsToAdd = runBackendStepInput.getValueInteger("yearsToAdd");
      int totalYearsAdded = 0;

      ///////////////////////////////////////////////////
      // loop over the records passed into the process //
      ///////////////////////////////////////////////////
      for(QRecord record : runBackendStepInput.getRecords())
      {
         Integer age = record.getValueInteger("age");
         age += yearsToAdd;
         totalYearsAdded += yearsToAdd;

         ////////////////////////////////////////////////////////////////////////////////////////////
         // update the record with the new "age" value.                                            //
         // note that this update record object will implicitly be available to the process's next //
         // backend step, via the sharing of the processState object.                              //
         ////////////////////////////////////////////////////////////////////////////////////////////
         record.setValue("age", age);
      }

      /////////////////////////////////////////
      // set an output value for the process //
      /////////////////////////////////////////
      runBackendStepOutput.addValue("totalYearsAdded", totalYearsAdded);
   }
}

----


=== Backend Steps for StreamedETLWithFrontendProcesses

For <<StreamedETLWithFrontendProcess>> type processes, backend steps are defined a little bit differently than they are for other process types.
In this type of process, the process meta-data defines 3 backend steps which are built-in to QQQ, and which do not have any custom application logic.
These steps are:

* `StreamedETLPreviewStep`
* `StreamedETLValidateStep`
* `StreamedETLExecuteStep`

For custom application logic to be implemented in a StreamedETLWithFrontendProcesses, an application engineer must define (up to) 3 backend step classes which are loaded by the steps listed above.
These application-defined steps must extend specific classes (which themselves implement the `BackendStep` interface), to provide the needed logic of this style of process.
These steps are:

* *Extract* - a subclass of `AbstractExtractStep` - is responsible for Extracting records from the source table.
** For this step, we can often use the QQQ-provided class `ExtractViaQueryStep`, or sometimes a subclass of it.
** The Extract step is called before the Preview, Validate, and Result screens, though for the Preview screen, it is set to only extract a small number of records (10).
* *Transform* - a subclass of `AbstractTransformStep` - is responsible for applying the majority of the business logic of the process.
In ETL terminology, this is the "Transform" action - which means applying some type of logical transformation an input record (found by the Extract step) to generate an output record (stored by the Load step).
** A Transform step's `run` method will be called, potentially, multiple times, each time with a page of records in the `runBackendStepInput` parameter.
** This method is responsible for adding records to the `runBackendStepOutput`, which will then be passed to the *Load* step.
** This class is also responsible for implementing the method `getProcessSummary`, which provides the data to the *Validate* screen.
** The run method will generally update ProcessSummaryLine objects to facilitate this functionality.
** The Transform step is called before the Preview, Validate, and Result screens, consuming all records selected by the Extract step.
* *Load* - a subclass of `AbstractLoadStep` - is responsible for the Load function of the ETL job.
_A quick word on terminology - this step is actually doing what we are more likely to think of as storing data - which feels like the opposite of “loading” - but we use the name Load to keep in line with the ETL naming convention…_
** The Load step is ONLY called before the Result screen is presented (possibly after Preview, if the user chose to skip validation, otherwise, after validation).
** Similar to the Transform step, the Load step's `run` method will be called potentially multiple times, with pages of records in its input.
** As such, the Load step is generally the only step where data writes should occur.
*** e.g., a Transform step should not do any writes, as it will be called when the user is going to the Preview & Validate screens - e.g., before the user confirmed that they want to execute the action!
** A common pattern is that the Load step just needs to insert or update the list of records output by the Transform step, in which case the QQQ-provided `LoadViaInsertStep` or `LoadViaUpdateStep` can be used, but custom use-cases can be built as well.

Another distinction between StreamedELTWithFrontendProcess steps and general QQQ process backend steps, is that the list of records in the input & output objects is NOT shared for StreamedELTWithFrontendProcess steps.
The direct implication of this is, that a Transform step MUST explicitly call `output.addRecord()` for any records that it wants to pass along to the Load step.

==== Example

[source,java]
.Examples of a Transform and Load step for a StreamedELTWithFrontendProcess
----
   // todo!
----

#todo: more details on these 3 specialized types of process steps (e.g., method to overload, when stuff like pre-action is called; how summaries work).#
