mirror of
https://github.com/Kingsrook/qqq.git
synced 2025-07-18 05:01:07 +00:00
Add some asciidocs that haven't been previously committed
This commit is contained in:
155
docs/misc/ProcessBackendSteps.adoc
Normal file
155
docs/misc/ProcessBackendSteps.adoc
Normal file
@ -0,0 +1,155 @@
|
||||
== Process Backend Steps
|
||||
include::../variables.adoc[]
|
||||
|
||||
In many QQQ applications, much of the code that engineers write will take the form of Backend Steps for {link-processes}.
|
||||
Such code is defined in classes which implement the interface `BackendStep`.
|
||||
This interface defines only a single method:
|
||||
[source,java]
|
||||
.BackendStep.java
|
||||
----
|
||||
public interface BackendStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Execute the backend step - using the request as input, and the result as output.
|
||||
**
|
||||
*******************************************************************************/
|
||||
void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException;
|
||||
}
|
||||
|
||||
----
|
||||
|
||||
Process backend steps have access to state information - specifically, a list of records, and a map of name=value pairs - in the input & output objects.
|
||||
This state data is persisted by QQQ between steps (e.g., if a frontend step is presented to a user between backend steps).
|
||||
|
||||
|
||||
=== RunBackendStepInput
|
||||
All input data to the step is available in the `RunBackendStepInput` object.
|
||||
Key methods in this class are:
|
||||
|
||||
* `getRecords()` - Returns the List of <<QRecords>> that are currently being acted on in the process.
|
||||
* `getValues()` - Returns a Map of String -> Serializable; name=value pairs that are the state-data of the process.
|
||||
** Values can be added to this state from a process's meta-data, from a screen, or from another backend step.
|
||||
* `getValue(String fieldName)` - Returns a specific value, by name, from the process's state.
|
||||
** This method has several variations that return the value as a specific type, such as `getValueString`, `getValueInteger`, `getValueBoolean`...
|
||||
* `getAsyncJobCallback()` - Accessor for an `AsyncJobCallback` object, which provides a way for the process backend step to communicate about its status or progress with a user running the process in a frontend.
|
||||
Provides methods:
|
||||
** `updateStatus(String message)` - Where general status messages can be given.
|
||||
For example, `"Loading census data"`
|
||||
** `updateStatus(int current, int total)` - For updating a progress meter.
|
||||
e.g., "47 of 1701" would be display by calling `.updateStatus(47, 1701)`
|
||||
* `getFrontendStepBehavior()` - Enum, indicating what should happen when a frontend step is encountered as the process's next step to run.
|
||||
Possible values are:
|
||||
** `BREAK` - Indicates that the process's execution should be suspended, so that the screen represented by the frontend step can be presented to a user.
|
||||
This would be the expected behavior if a process is being run by a user from a UI.
|
||||
** `SKIP` - Indicates that frontend steps should be skipped.
|
||||
This would be the expected behavior if a process is running from a scheduled job (without a user present to drive it), for example.
|
||||
** `FAIL` - Indicates that the process should end with an exception if a frontend step is encountered.
|
||||
** A backend step may want to act differently based on its frontendStepBehavior.
|
||||
For example, additional data may be looked up for displaying to a user if the behavior is `BREAK`.
|
||||
* `getBasepullLastRunTime()` - For <<BasepullConfiguration,Basepull>> processes, this is the `Instant` stored in the basepull table as the process's last run time.
|
||||
|
||||
=== RunBackendStepOutput
|
||||
All output from a process step should be placed in its `RunBackendStepOutput` object (and/or stored to a backend, as appropriate).
|
||||
Key methods in this class are:
|
||||
|
||||
* `addValue(String fieldName, Serializable value)` - Adds a single named value to the process's state, overwriting it the value if it already exists.
|
||||
* `addRecord(QRecord record)` - Add a `<<QRecord>>` to the process's output.
|
||||
* `addAuditSingleInput(AuditSingleInput auditSingleInput)` - Add a new entry to the process's list of audit inputs, to be stored at the completion of the process.
|
||||
** An `AuditSingleInput` object can most easily be built with the constructor: `AuditSingleInput(String tableName, QRecord record, String auditMessage)`.
|
||||
** Additional audit details messages (sub-bullets that accompany the primary `auditMessage`) can be added to an `AuditSingleInput` via the `addDetail(String message)` method.
|
||||
** _Note that at this time, the automatic storing of these audits is only provided by the execute step of a StreamedETLWithFrontendProcesses._
|
||||
|
||||
=== Example
|
||||
|
||||
[source,java]
|
||||
.Example of a BackendStep
|
||||
----
|
||||
/*******************************************************************************
|
||||
** For the "person" table's "Add Age" process -
|
||||
** For each input person record, add the specified yearsToAdd to their age.
|
||||
*******************************************************************************/
|
||||
public class AddAge implements BackendStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput)
|
||||
{
|
||||
/////////////////////////////////////////////////////////////////
|
||||
// get the yearsToAdd input field value from the process input //
|
||||
/////////////////////////////////////////////////////////////////
|
||||
Integer yearsToAdd = runBackendStepInput.getValueInteger("yearsToAdd");
|
||||
int totalYearsAdded = 0;
|
||||
|
||||
///////////////////////////////////////////////////
|
||||
// loop over the records passed into the process //
|
||||
///////////////////////////////////////////////////
|
||||
for(QRecord record : runBackendStepInput.getRecords())
|
||||
{
|
||||
Integer age = record.getValueInteger("age");
|
||||
age += yearsToAdd;
|
||||
totalYearsAdded += yearsToAdd;
|
||||
|
||||
////////////////////////////////////////////////////////////////////////////////////////////
|
||||
// update the record with the new "age" value. //
|
||||
// note that this update record object will implicitly be available to the process's next //
|
||||
// backend step, via the sharing of the processState object. //
|
||||
////////////////////////////////////////////////////////////////////////////////////////////
|
||||
record.setValue("age", age);
|
||||
}
|
||||
|
||||
/////////////////////////////////////////
|
||||
// set an output value for the process //
|
||||
/////////////////////////////////////////
|
||||
runBackendStepOutput.addValue("totalYearsAdded", totalYearsAdded);
|
||||
}
|
||||
}
|
||||
|
||||
----
|
||||
|
||||
|
||||
=== Backend Steps for StreamedETLWithFrontendProcesses
|
||||
|
||||
For <<StreamedETLWithFrontendProcess>> type processes, backend steps are defined a little bit differently than they are for other process types.
|
||||
In this type of process, the process meta-data defines 3 backend steps which are built-in to QQQ, and which do not have any custom application logic.
|
||||
These steps are:
|
||||
|
||||
* `StreamedETLPreviewStep`
|
||||
* `StreamedETLValidateStep`
|
||||
* `StreamedETLExecuteStep`
|
||||
|
||||
For custom application logic to be implemented in a StreamedETLWithFrontendProcesses, an application engineer must define (up to) 3 backend step classes which are loaded by the steps listed above.
|
||||
These application-defined steps must extend specific classes (which themselves implement the `BackendStep` interface), to provide the needed logic of this style of process.
|
||||
These steps are:
|
||||
|
||||
* *Extract* - a subclass of `AbstractExtractStep` - is responsible for Extracting records from the source table.
|
||||
** For this step, we can often use the QQQ-provided class `ExtractViaQueryStep`, or sometimes a subclass of it.
|
||||
** The Extract step is called before the Preview, Validate, and Result screens, though for the Preview screen, it is set to only extract a small number of records (10).
|
||||
* *Transform* - a subclass of `AbstractTransformStep` - is responsible for applying the majority of the business logic of the process.
|
||||
In ETL terminology, this is the "Transform" action - which means applying some type of logical transformation an input record (found by the Extract step) to generate an output record (stored by the Load step).
|
||||
** A Transform step's `run` method will be called, potentially, multiple times, each time with a page of records in the `runBackendStepInput` parameter.
|
||||
** This method is responsible for adding records to the `runBackendStepOutput`, which will then be passed to the *Load* step.
|
||||
** This class is also responsible for implementing the method `getProcessSummary`, which provides the data to the *Validate* screen.
|
||||
** The run method will generally update ProcessSummaryLine objects to facilitate this functionality.
|
||||
** The Transform step is called before the Preview, Validate, and Result screens, consuming all records selected by the Extract step.
|
||||
* *Load* - a subclass of `AbstractLoadStep` - is responsible for the Load function of the ETL job.
|
||||
_A quick word on terminology - this step is actually doing what we are more likely to think of as storing data - which feels like the opposite of “loading” - but we use the name Load to keep in line with the ETL naming convention…_
|
||||
** The Load step is ONLY called before the Result screen is presented (possibly after Preview, if the user chose to skip validation, otherwise, after validation).
|
||||
** Similar to the Transform step, the Load step's `run` method will be called potentially multiple times, with pages of records in its input.
|
||||
** As such, the Load step is generally the only step where data writes should occur.
|
||||
*** e.g., a Transform step should not do any writes, as it will be called when the user is going to the Preview & Validate screens - e.g., before the user confirmed that they want to execute the action!
|
||||
** A common pattern is that the Load step just needs to insert or update the list of records output by the Transform step, in which case the QQQ-provided `LoadViaInsertStep` or `LoadViaUpdateStep` can be used, but custom use-cases can be built as well.
|
||||
|
||||
Another distinction between StreamedELTWithFrontendProcess steps and general QQQ process backend steps, is that the list of records in the input & output objects is NOT shared for StreamedELTWithFrontendProcess steps.
|
||||
The direct implication of this is, that a Transform step MUST explicitly call `output.addRecord()` for any records that it wants to pass along to the Load step.
|
||||
|
||||
==== Example
|
||||
|
||||
[source,java]
|
||||
.Examples of a Transform and Load step for a StreamedELTWithFrontendProcess
|
||||
----
|
||||
// todo!
|
||||
----
|
||||
|
||||
#todo: more details on these 3 specialized types of process steps (e.g., method to overload, when stuff like pre-action is called; how summaries work).#
|
||||
30
docs/misc/QContext.adoc
Normal file
30
docs/misc/QContext.adoc
Normal file
@ -0,0 +1,30 @@
|
||||
== QContext
|
||||
include::../variables.adoc[]
|
||||
|
||||
The class `QContext` contains a collection of thread-local variables, to define the current context of the QQQ code that is currently running.
|
||||
For example, what `QInstance` (meta-data container) is being used, what `QSession` (user attributes) is active, etc.
|
||||
|
||||
Most of the time, main-line application code does not need to worry about setting up the `QContext` - although unit-test code can is a common exception to that rule.
|
||||
This is because all of QQQ's entry-points into execution (e.g., web context handlers, CLI startup methods, schedulers, and multi-threaded executors) take care of initializing the context.
|
||||
|
||||
It is more common though, for application code to need to get data from the context, such as the current session or any piece of meta-data from the QInstance.
|
||||
The methods to access data from the `QContext` are fairly straightforward:
|
||||
|
||||
=== Examples
|
||||
==== Get a QTableMetaData from the active QInstance
|
||||
[source,java]
|
||||
----
|
||||
QTableMeataData myTable = QContext.getQInstance().getTable("myTable");
|
||||
for(QFieldMeataData field : myTable.getFields().values())
|
||||
{
|
||||
// ...
|
||||
}
|
||||
----
|
||||
|
||||
==== Get a security key value for the current user session
|
||||
[source,java]
|
||||
----
|
||||
QSession session = QContext.getQSession();
|
||||
List<Serializable> clientIds = session.getSecurityKeyValues("clientId");
|
||||
----
|
||||
|
||||
116
docs/misc/QRecordEntities.adoc
Normal file
116
docs/misc/QRecordEntities.adoc
Normal file
@ -0,0 +1,116 @@
|
||||
== QRecordEntities
|
||||
include::../variables.adoc[]
|
||||
|
||||
While `<<QRecords>>` provide a flexible mechanism for passing around record data in a QQQ Application, they have one big disadvantage from the point-of-view of a Java Application:
|
||||
They do not provide a mechanism to ensure compile-time checks of field names or field types.
|
||||
|
||||
As such, an alternative mechanism exists, which allows records in a QQQ application to be worked with following a more familiar Java Bean (Entity Bean) like pattern.
|
||||
This mechanism is known as a `QRecordEntity`.
|
||||
|
||||
Specifically speaking, `QRecordEntity` is an abstract base class, whose purpose is to be the base class for entity-bean classes.
|
||||
Using reflection, `QRecordEntity` is able to convert objects back and forth from `QRecord` to specific entity-bean subtypes.
|
||||
|
||||
For example, the method `QRecordEntity::toQRecord()` converts a record entity object into a `QRecord`.
|
||||
|
||||
Inversely, `QRecordEntity::populateFromQRecord(QRecord record)` sets fields in a record entity object, based on the values in the supplied `QRecord`.
|
||||
It is conventional for a subclass of `QRecordEntity` to have both a no-arg (default) constructor, and a constructor that takes a `QRecord` as a parameter, and calls `populateFromQRecord`.
|
||||
|
||||
In addition to these constructors, a `QRecordEntity` subclass will generally contain:
|
||||
|
||||
* A `public static final String TABLE_NAME`, used throughout the application as a constant reference to the name for the {link-table}.
|
||||
* A series of `private` fields, corresponding to the fields in the table that the entity represents.
|
||||
** If these fields are annotated as `@QField()`, then the {link-table} meta-data for the table that the entity represents can in part be inferred by QQQ, by using the method `QTableMetaData::withFieldsFromEntity`.
|
||||
* `getX()`, `setX()`, and `withX()` methods for all of the entity's fields.
|
||||
|
||||
=== Examples
|
||||
[source,java]
|
||||
.Example Definition of a QRecordEntity subclass: Person.java
|
||||
----
|
||||
/*******************************************************************************
|
||||
** QRecordEntity for the person table.
|
||||
*******************************************************************************/
|
||||
public class Person extends QRecordEntity
|
||||
{
|
||||
public static final String TABLE_NAME = "person";
|
||||
|
||||
@QField(isEditable = false)
|
||||
private Integer id;
|
||||
|
||||
@QField()
|
||||
private String firstName;
|
||||
|
||||
@QField()
|
||||
private String lastName;
|
||||
|
||||
@QField()
|
||||
private Integer age;
|
||||
|
||||
// all other fields
|
||||
|
||||
/*******************************************************************************
|
||||
** Default constructor
|
||||
*******************************************************************************/
|
||||
public Person()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor that takes a QRecord
|
||||
*******************************************************************************/
|
||||
public Person(QRecord record)
|
||||
{
|
||||
populateFromQRecord(record);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Custom method to concatenate firstName and lastName
|
||||
*******************************************************************************/
|
||||
public String getFullName()
|
||||
{
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
// if there were more than an example, we could do some null-checks //
|
||||
// here to avoid silly output like "Bobby null" :) //
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
return (firstName + " " + lastName);
|
||||
}
|
||||
|
||||
// all getter, setter, and fluent setter (wither) methods
|
||||
}
|
||||
----
|
||||
|
||||
The core ORM actions and process-execution actions of QQQ work with `QRecords`.
|
||||
However, application engineers may want to apply patterns like the following example, to gain the compile-time safety of `QRecordEntities`:
|
||||
|
||||
[source,java]
|
||||
.Example Usage of a QRecordEntity
|
||||
----
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
// assume we're working with the "person" table & entity from above //
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
List<QRecord> recordsToUpdate = new ArrayList<>();
|
||||
for(QRecord record : inputRecordList)
|
||||
{
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
// call the constructor that copies values from the record into the entity //
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
Person person = new Person(record);
|
||||
|
||||
////////////////////////////////////////////////
|
||||
// call a custom method defined in the entity //
|
||||
////////////////////////////////////////////////
|
||||
LOG.info("Processing: " + person.getFullName());
|
||||
|
||||
/////////////////////////////////////////////////////////////
|
||||
// age is an Integer, so no type-conversion is needed here //
|
||||
/////////////////////////////////////////////////////////////
|
||||
person.setAge(person.getAge() + 1);
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
// to pass the updated records to UpdateAction, convert them to QRecords //
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
recordsToUpdate.add(person.toQRecord());
|
||||
}
|
||||
----
|
||||
|
||||
68
docs/misc/QRecords.adoc
Normal file
68
docs/misc/QRecords.adoc
Normal file
@ -0,0 +1,68 @@
|
||||
== QRecords
|
||||
include::../variables.adoc[]
|
||||
|
||||
Almost all code inside a QQQ application will be dealing with *Records* (aka Tuples or Rows).
|
||||
That is: a collection of named values, representing a single Entity, Fact, or, Row from a {link-table}.
|
||||
|
||||
The class that QQQ uses to work with records is called: `QRecord`.
|
||||
|
||||
=== Values
|
||||
At its core, a `QRecord` is a wrapper around a `Map<String, Serializable> values`.
|
||||
These are the *actual* values for the fields in the table for the record.
|
||||
That is, direct representations of the values as they are stored in the {link-backend}.
|
||||
|
||||
The keys in the `values` map are names from the {link-fields} in the {link-table}.
|
||||
|
||||
The values in `values` map are declared as `Serializable` (to help ensure the serializability of the `QRecord` as a whole).
|
||||
In practice, their types will be based on the `QFieldType` of the {link-field} that they correspond to.
|
||||
That will typically be one of: `String`, `Integer`, `Boolean`, `BigDecimal`, `Instant`, `LocalDate`, `LocalTime`, or `byte[]`.
|
||||
Be aware that `null` values may be in the `values` map, especially/per if the backend/table support `null`.
|
||||
|
||||
To work with the `values` map, the following methods are provided:
|
||||
|
||||
* `setValue(String fieldName, Serializable value)` - Sets a value for the specified field in the record.
|
||||
** Overloaded as `setValue(String fieldName, Object value)` - For some cases where the value may not be known to be `Serializable`.
|
||||
In this overload, if the value is `null` or `Serializable`, the primary version of `setValue` is called.
|
||||
Otherwise, the `value` is passed through `String::valueOf`, and the result is stored.
|
||||
** Overloaded as `setValue(QFieldMetaData field, Serializable value)` - which simply defers to the primary version of `setValue`, passing `field.getName()` as the first parameter.
|
||||
* `removeValue(String fieldName)` - Remove the given field from the `values` map.
|
||||
** Note that in some situations this is an important distinction from having a `null` value in the map (See <<UpdateAction)>>).
|
||||
* `setValues(Map<String, Serializable> values)` - Sets the full map of `values`.
|
||||
* `getValues()` - Returns the full map of `values`.
|
||||
* `getValue(String fieldName)` - Returns the value for the named field - possibly `null` - as a `Serializable`.
|
||||
* Several type-specific variations of `getValueX(String fieldName)`, where internally, values will be not exactly type-cast, but effectively converted (if possible) to the requested type.
|
||||
These conversions are done using the `ValueUtils.getValueAsX(Object)` methods.
|
||||
These methods are generally the preferred/cleanest way to get record values in application code, when it is needed in a type-specific way .
|
||||
** `getValueString(String fieldName)`
|
||||
** `getValueInteger(String fieldName)`
|
||||
** `getValueBoolean(String fieldName)`
|
||||
** `getValueBigDecimal(String fieldName)`
|
||||
** `getValueInstant(String fieldName)`
|
||||
** `getValueLocalDate(String fieldName)`
|
||||
** `getValueLocalTime(String fieldName)`
|
||||
** `getValueByteArray(String fieldName)`
|
||||
|
||||
=== Display Values
|
||||
In addition to the `values` map, a `QRecord` contains another map called `displayValues`, which only stores `String` values.
|
||||
That is to say, values for other types are stringified, based on their {link-field}'s type and `displayFormat` property.
|
||||
In addition, fields which have a `possibleValueSource` property will have their translated values set in the `displayValues` map.
|
||||
|
||||
By default, a `QRecord` will not have its `displayValues` populated.
|
||||
To populate `displayValues`, the <<QueryAction>> and <<GetAction>> classes take a property in their inputs called `shouldGenerateDisplayValues`, which must be set to `true` to generate `displayValues`.
|
||||
In addition, these two actions also have a property `shouldTranslatePossibleValues` in their inputs, which needs to be set to `true` if possible-value lookups are to be performed.
|
||||
|
||||
As an alternative to the `shouldGenerateDisplayValues` and `shouldTranslatePossibleValues` inputs to <<QueryAction>> and <<GetAction>>, one can directly call the `QValueFormatter.setDisplayValuesInRecords` and/or `qPossibleValueTranslator.translatePossibleValuesInRecords` methods.
|
||||
Or, for special cases, `setDisplayValue(String fieldName, String displayValue)` or `setDisplayValues(Map<String, String> displayValues)` can be called directly.
|
||||
|
||||
=== Backend Details
|
||||
Sometimes a backend may want to place additional data in a `QRecord` that doesn't exactly correspond to a field.
|
||||
To do this, the `Map<String, Serializable> backendDetails` member is used.
|
||||
|
||||
For example, an API backend may store the full JSON `String` that came from the API as a backend detail in a `QRecord`.
|
||||
Or fields that are marked as `isHeavy`, if the full (heavy) value of the field hasn't been fetched, then the lengths of any such heavy fields may be stored in `backendDetails`.
|
||||
|
||||
=== Errors and Warnings
|
||||
#todo#
|
||||
|
||||
=== Associated Records
|
||||
#todo#
|
||||
Reference in New Issue
Block a user