checkpoint - working version of c3p0 connection pooling, and read-only database meta-data connections (per query hint)

Feature/ce 938 order release automation

docs/metaData/Joins.adoc

@@ -2,16 +2,57 @@
 == Joins
 include::../variables.adoc[]
 
-#TODO#
+A `QJoinMetaData` is a meta-data object that tells QQQ, essentially “it is possible for these 2 tables to join, here’s how to do it”.
+
+Joins can be used then, in an application, in a number of possible ways:
+
+* In a {link-table}, we can specify joins to be “exposed”, e.g., made available to users on a query screen
+* Also in a Table, as part of an “Association”, which sets up one table as a “parent” of another,
+such that you can store (and fetch) the child-records at the same time as the parent
+** A common use-case here may be an order & lineItem table -
+such that QQQ can generate an API uses to allow you to post an order and its lines in a single request, and they get stored all together
+* In defining the security field (record lock) on a table,
+sometimes, it isn’t a field directly on the table, but instead comes from a joined table (possibly even more than just 1 table away).
+** For example, maybe a lineItem table, doesn't have a clientId, but needs secured by that field
+- so its recordLock can specify a “joinNameChain” that describes how to get from lineItem to order.clientId
+* The `QueryAction` can take (through its QueryInput object) zero or more QueryJoin objects,
+which must make a reference (implicitly or explicitly) to a QJoinMetaData.
+See the section on <<QueryJoin,QueryJoins>> for more details.
 
 === QJoinMetaData
 Joins are defined in a QQQ Instance in a `*QJoinMetaData*` object.
 
-#TODO#
+In this object, we have the concept of a "leftTable" and a "rightTable".
+There isn't generally anything special about which table is on the "left" and which is on the "right".
+But the remaining pieces of the QJoinMetaData do all need to line-up with these sides.
+
+For example:
+
+* The Type (one-to-one, one-to-many, many-to-one) - where the leftTable comes first, and rightTable comes second
+(e.g., a one-to-many means 1-row in leftTable has many-rows in rightTable associated with it)
+* In a JoinOn object, the 1st field name given is from the leftTable;
+the second fieldName from the rightTable.
 
 *QJoinMetaData Properties:*
 
-* `name` - *String, Required* - Unique name for the join within the QQQ Instance. #todo infererences or conventions?#
-
-#TODO#
+* `name` - *String, Required* - Unique name for the join within the QQQ Instance.
+** One convention is to name joins based on (leftTable + "Join" + rightTable).
+** If you do not wish to define join names yourself, the method `withInferredName()`
+can be called (which defers to
+`public static String makeInferredJoinName(String leftTable, String rightTable)`),
+to create a name for the join following the (leftTable + "Join" + rightTable) convention.
+* `leftTable` - *String, Required* - Name of a {link-table} in the {link-instance}.
+* `rightTable` - *String, Required* - Name of a {link-table} in the {link-instance}.
+* `type` - *enum, Required* - cardinality between the two tables in the join.
+** e.g., `ONE_TO_ONE`, `ONE_TO_MANY` (indicating 1 record in the left table may join
+to many records in the right table), or `MANY_TO_ONE` (vice-versa).
+** Note that there is no MANY_TO_MANY option, as a many-to-many is built as multiple QJoinMetaData's
+going through the intermediary (intersection) table.
+* `joinOns` - *List<JoinOn>, Required* - fields used to join the tables.
+Note: In the 2-arg JoinOn constructor, the leftTable's field comes first.
+Alternatively, the no-arg constructor can be used along with `.withLeftField().withRightField()`
+* `orderBys` - *List<QFilterOrderBy>* - Optional list of order-by objects,
+used in some framework-generated queries using the join.
+The field names are assumed to come from the rightTable.
 
+#TODO# what else do we need here?

docs/metaData/Processes.adoc

@@ -16,6 +16,7 @@ Processes are defined in a QQQ Instance in a `*QProcessMetaData*` object.
 In addition to directly building a `QProcessMetaData` object setting its properties directly, there are a few common process patterns that provide *Builder* objects for ease-of-use.
 See StreamedETLWithFrontendProcess below for a common example
 
+[#_QProcessMetaData_Properties]
 *QProcessMetaData Properties:*
 
 * `name` - *String, Required* - Unique name for the process within the QQQ Instance.
@@ -30,12 +31,13 @@ See below for details.
 * `permissionRules` - *QPermissionRules object* - define the permission/access rules for the process.
 See {link-permissionRules} for details.
 * `steps` and `stepList` - *Map of String → <<QStepMetaData>>* and *List of QStepMetaData* - Defines the <<QFrontendStepMetaData,screens>> and <<QBackendStepMetaData,backend code>> that makes up the process.
-** `stepList` is the list of steps in the order that they will by default be executed.
-** `steps` is a map, including all steps from `stepList`, but which may also include steps which can used by the process if its backend steps make the decision to do so, at run-time.
+** `stepList` is the list of steps in the order that they will be executed
+(that is to say - this is the _default_ order of execution - but it can be customized - see <<_custom_process_flow>> for details).
+** `steps` is a map, including all steps from `stepList`, but which may also include steps which can used by the process if its backend steps make the decision to do so, at run-time (e.g., using <<_custom_process_flow>>).
 ** A process's steps are normally defined in one of two was:
 *** 1) by a single call to `.withStepList(List<QStepMetaData>)`, which internally adds each step into the `steps` map.
 *** 2) by multiple calls to `.addStep(QStepMetaData)`, which adds a step to both the `stepList` and `steps` map.
-** If a process also needs optional steps, they should be added by a call to `.addOptionalStep(QStepMetaData)`, which only places them in the `steps` map.
+** If a process also needs optional steps (for a <<_custom_process_flow>>), they should be added by a call to `.addOptionalStep(QStepMetaData)`, which only places them in the `steps` map.
 * `schedule` - *<<QScheduleMetaData>>* - set up the process to run automatically on the specified schedule.
 See below for details.
 * `minInputRecords` - *Integer* - #not used...#
@@ -214,3 +216,112 @@ But for some cases, doing page-level transactions can reduce long-transactions a
 * `withFields(List<QFieldMetaData> fieldList)` - Adds additional input fields to the preview step of the process.
 * `withBasepullConfiguration(BasepullConfiguration basepullConfiguration)` - Add a <<BasepullConfiguration>> to the process.
 * `withSchedule(QScheduleMetaData schedule)` - Add a <<QScheduleMetaData>> to the process.
+
+[#_custom_process_flow]
+==== Custom Process Flow
+As referenced in the definition of the <<_QProcessMetaData_Properties,QProcessMetaData Properties>>, by default, a process
+will execute each of its steps in-order, as defined in the `stepList` property.
+However, a Backend Step can customize this flow #todo - write more clearly here...
+
+There are generally 2 method to call (in a `BackendStep`) to do a dynamic flow:
+
+* `RunBackendStepOutput.setOverrideLastStepName(String stepName)`
+** QQQ's `RunProcessAction` keeps track of which step it "last" ran, e.g., to tell it which one to run next.
+However, if a step sets the `OverrideLastStepName` property in its output object,
+then the step named in that property becomes the effective "last" step,
+thus determining which step comes next.
+
+* `RunBackendStepOutput.updateStepList(List<String> stepNameList)`
+** Calling this method changes the process's runtime definition of steps to be executed.
+Thus allowing a completely custom flow.
+It should be noted, that the "last" step name (as tracked by QQQ within `RunProcessAction`)
+does need to be found in the new `stepNameList` - otherwise, the framework will not know where you were,
+for figuring out where to go next.
+
+[source,java]
+.Example of a defining process that can use a flexible flow:
+----
+// for a case like this, it would be recommended to define all step names in constants:
+public final static String STEP_START = "start";
+public final static String STEP_A     = "a";
+public final static String STEP_B     = "b";
+public final static String STEP_C     = "c";
+public final static String STEP_1     = "1";
+public final static String STEP_2     = "2";
+public final static String STEP_3     = "3";
+public final static String STEP_END   = "end";
+
+// also, to define the possible flows (lists of steps) in constants as well:
+public final static List<String> LETTERS_STEP_LIST = List.of(
+  STEP_START, STEP_A, STEP_B, STEP_C, STEP_END);
+
+public final static List<String> NUMBERS_STEP_LIST = List.of(
+  STEP_START, STEP_1, STEP_2, STEP_3, STEP_END);
+
+// when we define the process's meta-data, we only give a "skeleton" stepList -
+// we must at least have our starting step, and we may want at least one frontend step
+// for the UI to show some placeholder(s):
+QProcessMetaData process = new QProcessMetaData()
+   .withName(PROCESS_NAME)
+   .withStepList(List.of(
+      new QBackendStepMetaData().withName(STEP_START)
+         .withCode(new QCodeReference(/*...*/)),
+      new QFrontendStepMetaData()
+         .withName(STEP_END)
+   ));
+
+// the additional steps get added via `addOptionalStep`, which only puts them in
+// the process's stepMap, not its stepList!
+process.addOptionalStep(new QFrontendStepMetaData().withName(STEP_A));
+process.addOptionalStep(new QBackendStepMetaData().withName(STEP_B)
+   .withCode(new QCodeReference(/*...*/)));
+process.addOptionalStep(new QFrontendStepMetaData().withName(STEP_C));
+
+process.addOptionalStep(new QBackendStepMetaData().withName(STEP_1)
+   .withCode(new QCodeReference(/*...*/)));
+process.addOptionalStep(new QFrontendStepMetaData().withName(STEP_2));
+process.addOptionalStep(new QBackendStepMetaData().withName(STEP_3)
+   .withCode(new QCodeReference(/*...*/)));
+
+----
+
+[source,java]
+.Example of a process backend step adjusting the process's runtime flow:
+----
+/***************************************************************************
+** look at the value named "which". if it's "letters", then make the process
+** go through the stepList consisting of letters; else, update the step list
+** to be the "numbers" steps.
+**
+** Also - if the "skipSomeSteps" value is give as true, then set the
+** overrideLastStepName to skip again (in the letters case, skip past A, B
+** and C; in the numbers case, skip past 1 and 2).
+***************************************************************************/
+public static class StartStep implements BackendStep
+{
+  @Override
+  public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+  {
+     Boolean skipSomeSteps = runBackendStepInput.getValueBoolean("skipSomeSteps");
+
+     if(runBackendStepInput.getValueString("which").equals("letters"))
+     {
+        runBackendStepOutput.updateStepList(LETTERS_STEP_LIST);
+        if(BooleanUtils.isTrue(skipSomeSteps))
+        {
+           runBackendStepOutput.setOverrideLastStepName(STEP_C);
+        }
+     }
+     else
+     {
+        runBackendStepOutput.updateStepList(NUMBERS_STEP_LIST);
+        if(BooleanUtils.isTrue(skipSomeSteps))
+        {
+           runBackendStepOutput.setOverrideLastStepName(STEP_2);
+        }
+     }
+  }
+}
+----
+
+

pom.xml

@@ -150,7 +150,7 @@
                <dependency>
                   <groupId>com.puppycrawl.tools</groupId>
                   <artifactId>checkstyle</artifactId>
-                  <version>9.0</version>
+                  <version>10.16.0</version>
                </dependency>
             </dependencies>
             <executions>

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/automation/AutomationStatus.java

@@ -127,7 +127,6 @@ public enum AutomationStatus implements PossibleValueEnum<Integer>
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    public String getInsertOrUpdate()
    {
       return switch(this)

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/customizers/RecordCustomizerUtilityInterface.java

@@ -50,10 +50,7 @@ public interface RecordCustomizerUtilityInterface
    /*******************************************************************************
     ** Container for an old value and a new value.
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:MethodName")
-   record Change(Serializable oldValue, Serializable newValue)
-   {
-   }
+   record Change(Serializable oldValue, Serializable newValue) {}
 
 
    /*******************************************************************************

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/dashboard/AbstractHTMLWidgetRenderer.java

@@ -161,7 +161,7 @@ public abstract class AbstractHTMLWidgetRenderer extends AbstractWidgetRenderer
    public static String linkTableCreateWithDefaultValues(RenderWidgetInput input, String tableName, Map<String, Serializable> defaultValues) throws QException
    {
       String tablePath = QContext.getQInstance().getTablePath(tableName);
-      return (tablePath + "/create?defaultValues=" + URLEncoder.encode(JsonUtils.toJson(defaultValues), Charset.defaultCharset()));
+      return (tablePath + "/create#defaultValues=" + URLEncoder.encode(JsonUtils.toJson(defaultValues), Charset.defaultCharset()));
    }
 
 
@@ -183,7 +183,6 @@ public abstract class AbstractHTMLWidgetRenderer extends AbstractWidgetRenderer
 
 
 
-
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/dashboard/widgets/DateTimeGroupBy.java

@@ -256,7 +256,6 @@ public enum DateTimeGroupBy
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    public Instant roundDown(Instant instant, ZoneId zoneId)
    {
       ZonedDateTime zoned = instant.atZone(zoneId);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/dashboard/widgets/ProcessAlertWidget.java

@@ -0,0 +1,86 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.AlertData;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
+
+
+/*******************************************************************************
+ ** Widget that can add an Alert to a process screen.
+ **
+ ** In the process, you'll want values:
+ ** - alertType - name of entry in AlertType enum (ERROR, WARNING, SUCCESS)
+ ** - alertHtml - html to display inside the alert (other than its icon)
+ *******************************************************************************/
+public class ProcessAlertWidget extends AbstractWidgetRenderer implements MetaDataProducerInterface<QWidgetMetaData>
+{
+   public static final String NAME = "ProcessAlertWidget";
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public RenderWidgetOutput render(RenderWidgetInput input) throws QException
+   {
+      AlertData.AlertType alertType = AlertData.AlertType.WARNING;
+      if(input.getQueryParams().containsKey("alertType"))
+      {
+         alertType = AlertData.AlertType.valueOf(input.getQueryParams().get("alertType"));
+      }
+
+      String html = "Warning";
+      if(input.getQueryParams().containsKey("alertHtml"))
+      {
+         html = input.getQueryParams().get("alertHtml");
+      }
+
+      return (new RenderWidgetOutput(new AlertData(alertType, html)));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public QWidgetMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QWidgetMetaData()
+         .withType(WidgetType.ALERT.getType())
+         .withGridColumns(12)
+         .withName(NAME)
+         .withIsCard(false)
+         .withShowReloadButton(false)
+         .withCodeReference(new QCodeReference(getClass()));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/metadata/JoinGraph.java

@@ -29,6 +29,7 @@ import java.util.List;
 import java.util.Objects;
 import java.util.Set;
 import java.util.TreeSet;
+import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
@@ -41,9 +42,19 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  *******************************************************************************/
 public class JoinGraph
 {
-
    private Set<Edge> edges = new HashSet<>();
 
+   ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   // as an instance grows, with the number of joins (say, more than 50?), especially as they may have a lot of connections, //
+   // it can become very very slow to process a full join graph (e.g., 10 seconds, maybe much worse, per Big-O...)           //
+   // also, it's not frequently useful to look at a join path that's more than a handful of tables long.                     //
+   // thus - this property exists - to limit the max length of a join path.  Keeping it small keeps instance enrichment      //
+   // and validation reasonably performant, at the possible cost of, some join-path that's longer than this limit may not    //
+   // be found - but - chances are, you don't want some 12-element join path to be used anyway, thus, this makes sense.      //
+   // but - it can be adjusted, per system property or ENV var.                                                              //
+   ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   private int maxPathLength = new QMetaDataVariableInterpreter().getIntegerFromPropertyOrEnvironment("qqq.instance.joinGraph.maxPathLength", "QQQ_INSTANCE_JOIN_GRAPH_MAX_PATH_LENGTH", 3);
+
 
 
    /*******************************************************************************
@@ -303,6 +314,13 @@ public class JoinGraph
 
                if(otherTableName != null)
                {
+                  if(newPath.size() > maxPathLength)
+                  {
+                     ////////////////////////////////////////////////////////////////
+                     // performance hack.  see comment at maxPathLength definition //
+                     ////////////////////////////////////////////////////////////////
+                     continue;
+                  }
 
                   JoinConnectionList newConnectionList = connectionList.copy();
                   JoinConnection     joinConnection    = new JoinConnection(otherTableName, edge.joinName);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/permissions/PermissionsHelper.java

@@ -500,7 +500,6 @@ public class PermissionsHelper
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    static PermissionSubType getEffectivePermissionSubType(QPermissionRules rules, PermissionSubType originalPermissionSubType)
    {
       if(rules == null || rules.getLevel() == null)
@@ -515,10 +514,10 @@ public class PermissionsHelper
       if(PrivatePermissionSubType.HAS_ACCESS.equals(originalPermissionSubType))
       {
          return switch(rules.getLevel())
-            {
-               case NOT_PROTECTED -> null;
-               default -> PrivatePermissionSubType.HAS_ACCESS;
-            };
+         {
+            case NOT_PROTECTED -> null;
+            default -> PrivatePermissionSubType.HAS_ACCESS;
+         };
       }
       else
       {
@@ -527,30 +526,30 @@ public class PermissionsHelper
          // permission sub-type to what we expect to be set for the table                                      //
          ////////////////////////////////////////////////////////////////////////////////////////////////////////
          return switch(rules.getLevel())
+         {
+            case NOT_PROTECTED -> null;
+            case HAS_ACCESS_PERMISSION -> PrivatePermissionSubType.HAS_ACCESS;
+            case READ_WRITE_PERMISSIONS ->
             {
-               case NOT_PROTECTED -> null;
-               case HAS_ACCESS_PERMISSION -> PrivatePermissionSubType.HAS_ACCESS;
-               case READ_WRITE_PERMISSIONS ->
+               if(PrivatePermissionSubType.READ.equals(originalPermissionSubType) || PrivatePermissionSubType.WRITE.equals(originalPermissionSubType))
                {
-                  if(PrivatePermissionSubType.READ.equals(originalPermissionSubType) || PrivatePermissionSubType.WRITE.equals(originalPermissionSubType))
-                  {
-                     yield (originalPermissionSubType);
-                  }
-                  else if(TablePermissionSubType.INSERT.equals(originalPermissionSubType) || TablePermissionSubType.EDIT.equals(originalPermissionSubType) || TablePermissionSubType.DELETE.equals(originalPermissionSubType))
-                  {
-                     yield (PrivatePermissionSubType.WRITE);
-                  }
-                  else if(TablePermissionSubType.READ.equals(originalPermissionSubType))
-                  {
-                     yield (PrivatePermissionSubType.READ);
-                  }
-                  else
-                  {
-                     throw new IllegalStateException("Unexpected permissionSubType: " + originalPermissionSubType);
-                  }
+                  yield (originalPermissionSubType);
                }
-               case READ_INSERT_EDIT_DELETE_PERMISSIONS -> originalPermissionSubType;
-            };
+               else if(TablePermissionSubType.INSERT.equals(originalPermissionSubType) || TablePermissionSubType.EDIT.equals(originalPermissionSubType) || TablePermissionSubType.DELETE.equals(originalPermissionSubType))
+               {
+                  yield (PrivatePermissionSubType.WRITE);
+               }
+               else if(TablePermissionSubType.READ.equals(originalPermissionSubType))
+               {
+                  yield (PrivatePermissionSubType.READ);
+               }
+               else
+               {
+                  throw new IllegalStateException("Unexpected permissionSubType: " + originalPermissionSubType);
+               }
+            }
+            case READ_INSERT_EDIT_DELETE_PERMISSIONS -> originalPermissionSubType;
+         };
       }
    }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/processes/CancelProcessAction.java

@@ -0,0 +1,110 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.processes;
+
+
+import java.util.Optional;
+import java.util.UUID;
+import com.kingsrook.qqq.backend.core.actions.ActionHelper;
+import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessState;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.state.StateType;
+import com.kingsrook.qqq.backend.core.state.UUIDAndTypeStateKey;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** Action handler for running the cancel step of a qqq process
+ *
+ *******************************************************************************/
+public class CancelProcessAction extends RunProcessAction
+{
+   private static final QLogger LOG = QLogger.getLogger(CancelProcessAction.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public RunProcessOutput execute(RunProcessInput runProcessInput) throws QException
+   {
+      ActionHelper.validateSession(runProcessInput);
+
+      QProcessMetaData process = runProcessInput.getInstance().getProcess(runProcessInput.getProcessName());
+      if(process == null)
+      {
+         throw new QBadRequestException("Process [" + runProcessInput.getProcessName() + "] is not defined in this instance.");
+      }
+
+      if(runProcessInput.getProcessUUID() == null)
+      {
+         throw (new QBadRequestException("Cannot cancel process - processUUID was not given."));
+      }
+
+      UUIDAndTypeStateKey    stateKey     = new UUIDAndTypeStateKey(UUID.fromString(runProcessInput.getProcessUUID()), StateType.PROCESS_STATUS);
+      Optional<ProcessState> processState = getState(runProcessInput.getProcessUUID());
+      if(processState.isEmpty())
+      {
+         throw (new QBadRequestException("Cannot cancel process - State for process UUID [" + runProcessInput.getProcessUUID() + "] was not found."));
+      }
+
+      RunProcessOutput runProcessOutput = new RunProcessOutput();
+      try
+      {
+         if(process.getCancelStep() != null)
+         {
+            LOG.info("Running cancel step for process", logPair("processName", process.getName()));
+            runBackendStep(runProcessInput, process, runProcessOutput, stateKey, process.getCancelStep(), process, processState.get());
+         }
+         else
+         {
+            LOG.debug("Process does not have a custom cancel step to run.", logPair("processName", process.getName()));
+         }
+      }
+      catch(QException qe)
+      {
+         ////////////////////////////////////////////////////////////
+         // upon exception (e.g., one thrown by a step), throw it. //
+         ////////////////////////////////////////////////////////////
+         throw (qe);
+      }
+      catch(Exception e)
+      {
+         throw (new QException("Error cancelling process", e));
+      }
+      finally
+      {
+         //////////////////////////////////////////////////////
+         // always put the final state in the process result //
+         //////////////////////////////////////////////////////
+         runProcessOutput.setProcessState(processState.get());
+      }
+
+      return (runProcessOutput);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/processes/RunBackendStepAction.java

@@ -26,6 +26,7 @@ import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
 import java.util.stream.Collectors;
 import com.kingsrook.qqq.backend.core.actions.ActionHelper;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
@@ -34,6 +35,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
@@ -71,7 +73,17 @@ public class RunBackendStepAction
       QStepMetaData stepMetaData = process.getStep(runBackendStepInput.getStepName());
       if(stepMetaData == null)
       {
-         throw new QException("Step [" + runBackendStepInput.getStepName() + "] is not defined in the process [" + process.getName() + "]");
+         if(process.getCancelStep() != null && Objects.equals(process.getCancelStep().getName(), runBackendStepInput.getStepName()))
+         {
+            /////////////////////////////////////
+            // special case for cancel step... //
+            /////////////////////////////////////
+            stepMetaData = process.getCancelStep();
+         }
+         else
+         {
+            throw new QException("Step [" + runBackendStepInput.getStepName() + "] is not defined in the process [" + process.getName() + "]");
+         }
       }
 
       if(!(stepMetaData instanceof QBackendStepMetaData backendStepMetaData))
@@ -82,7 +94,7 @@ public class RunBackendStepAction
       //////////////////////////////////////////////////////////////////////////////////////
       // ensure input data is set as needed - use callback object to get anything missing //
       //////////////////////////////////////////////////////////////////////////////////////
-      ensureRecordsAreInRequest(runBackendStepInput, backendStepMetaData);
+      ensureRecordsAreInRequest(runBackendStepInput, backendStepMetaData, process);
       ensureInputFieldsAreInRequest(runBackendStepInput, backendStepMetaData);
 
       ////////////////////////////////////////////////////////////////////
@@ -167,7 +179,7 @@ public class RunBackendStepAction
     ** check if this step uses a record list - and if so, if we need to get one
     ** via the callback
     *******************************************************************************/
-   private void ensureRecordsAreInRequest(RunBackendStepInput runBackendStepInput, QBackendStepMetaData step) throws QException
+   private void ensureRecordsAreInRequest(RunBackendStepInput runBackendStepInput, QBackendStepMetaData step, QProcessMetaData process) throws QException
    {
       QFunctionInputMetaData inputMetaData = step.getInputMetaData();
       if(inputMetaData != null && inputMetaData.getRecordListMetaData() != null)
@@ -190,9 +202,44 @@ public class RunBackendStepAction
 
             queryInput.setFilter(callback.getQueryFilter());
 
+            //////////////////////////////////////////////////////////////////////////////////////////
+            // if process has a max-no of records, set a limit on the process of that number plus 1 //
+            // (the plus 1 being so we can see "oh, you selected more than that many; error!"       //
+            //////////////////////////////////////////////////////////////////////////////////////////
+            if(process.getMaxInputRecords() != null)
+            {
+               if(callback.getQueryFilter() == null)
+               {
+                  queryInput.setFilter(new QQueryFilter());
+               }
+
+               queryInput.getFilter().setLimit(process.getMaxInputRecords() + 1);
+            }
+
             QueryOutput queryOutput = new QueryAction().execute(queryInput);
             runBackendStepInput.setRecords(queryOutput.getRecords());
-            // todo - handle 0 results found?
+
+            ////////////////////////////////////////////////////////////////////////////////
+            // if process defines a max, and more than the max were found, throw an error //
+            ////////////////////////////////////////////////////////////////////////////////
+            if(process.getMaxInputRecords() != null)
+            {
+               if(queryOutput.getRecords().size() > process.getMaxInputRecords())
+               {
+                  throw (new QUserFacingException("Too many records were selected for this process.  At most, only " + process.getMaxInputRecords() + " can be selected."));
+               }
+            }
+
+            /////////////////////////////////////////////////////////////////////////////////
+            // if process defines a min, and fewer than the min were found, throw an error //
+            /////////////////////////////////////////////////////////////////////////////////
+            if(process.getMinInputRecords() != null)
+            {
+               if(queryOutput.getRecords().size() < process.getMinInputRecords())
+               {
+                  throw (new QUserFacingException("Too few records were selected for this process.  At least " + process.getMinInputRecords() + " must be selected."));
+               }
+            }
          }
       }
    }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/processes/RunProcessAction.java

@@ -82,7 +82,7 @@ public class RunProcessAction
    public static final String BASEPULL_THIS_RUNTIME_KEY = "basepullThisRuntimeKey";
    public static final String BASEPULL_LAST_RUNTIME_KEY = "basepullLastRuntimeKey";
    public static final String BASEPULL_TIMESTAMP_FIELD  = "basepullTimestampField";
-   public static final String BASEPULL_CONFIGURATION = "basepullConfiguration";
+   public static final String BASEPULL_CONFIGURATION    = "basepullConfiguration";
 
    ////////////////////////////////////////////////////////////////////////////////////////////////
    // indicator that the timestamp field should be updated - e.g., the execute step is finished. //
@@ -190,7 +190,25 @@ public class RunProcessAction
                // Run backend steps //
                ///////////////////////
                LOG.debug("Running backend step [" + step.getName() + "] in process [" + process.getName() + "]");
-               runBackendStep(runProcessInput, process, runProcessOutput, stateKey, backendStepMetaData, process, processState);
+               RunBackendStepOutput runBackendStepOutput = runBackendStep(runProcessInput, process, runProcessOutput, stateKey, backendStepMetaData, process, processState);
+
+               /////////////////////////////////////////////////////////////////////////////////////////
+               // if the step returned an override lastStepName, use that to determine how we proceed //
+               /////////////////////////////////////////////////////////////////////////////////////////
+               if(runBackendStepOutput.getOverrideLastStepName() != null)
+               {
+                  LOG.debug("Process step [" + lastStepName + "] returned an overrideLastStepName [" + runBackendStepOutput.getOverrideLastStepName() + "]!");
+                  lastStepName = runBackendStepOutput.getOverrideLastStepName();
+               }
+
+               /////////////////////////////////////////////////////////////////////////////////////////////
+               // similarly, if the step produced an updatedFrontendStepList, propagate that data outward //
+               /////////////////////////////////////////////////////////////////////////////////////////////
+               if(runBackendStepOutput.getUpdatedFrontendStepList() != null)
+               {
+                  LOG.debug("Process step [" + lastStepName + "] generated an updatedFrontendStepList [" + runBackendStepOutput.getUpdatedFrontendStepList().stream().map(s -> s.getName()).toList() + "]!");
+                  runProcessOutput.setUpdatedFrontendStepList(runBackendStepOutput.getUpdatedFrontendStepList());
+               }
             }
             else
             {
@@ -317,6 +335,13 @@ public class RunProcessAction
          ///////////////////////////////////////////////////
          runProcessInput.seedFromProcessState(optionalProcessState.get());
 
+         ///////////////////////////////////////////////////////////////////////////////////////////////////
+         // if we're restoring an old state, we can discard a previously stored updatedFrontendStepList - //
+         // it is only needed on the transitional edge from a backend-step to a frontend step, but not    //
+         // in the other directly                                                                         //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////
+         optionalProcessState.get().setUpdatedFrontendStepList(null);
+
          ///////////////////////////////////////////////////////////////////////////
          // if there were values from the caller, put those (back) in the request //
          ///////////////////////////////////////////////////////////////////////////
@@ -339,7 +364,7 @@ public class RunProcessAction
    /*******************************************************************************
     ** Run a single backend step.
     *******************************************************************************/
-   private void runBackendStep(RunProcessInput runProcessInput, QProcessMetaData process, RunProcessOutput runProcessOutput, UUIDAndTypeStateKey stateKey, QBackendStepMetaData backendStep, QProcessMetaData qProcessMetaData, ProcessState processState) throws Exception
+   protected RunBackendStepOutput runBackendStep(RunProcessInput runProcessInput, QProcessMetaData process, RunProcessOutput runProcessOutput, UUIDAndTypeStateKey stateKey, QBackendStepMetaData backendStep, QProcessMetaData qProcessMetaData, ProcessState processState) throws Exception
    {
       RunBackendStepInput runBackendStepInput = new RunBackendStepInput(processState);
       runBackendStepInput.setProcessName(process.getName());
@@ -368,14 +393,16 @@ public class RunProcessAction
          runBackendStepInput.setBasepullLastRunTime((Instant) runProcessInput.getValues().get(BASEPULL_LAST_RUNTIME_KEY));
       }
 
-      RunBackendStepOutput lastFunctionResult = new RunBackendStepAction().execute(runBackendStepInput);
-      storeState(stateKey, lastFunctionResult.getProcessState());
+      RunBackendStepOutput runBackendStepOutput = new RunBackendStepAction().execute(runBackendStepInput);
+      storeState(stateKey, runBackendStepOutput.getProcessState());
 
-      if(lastFunctionResult.getException() != null)
+      if(runBackendStepOutput.getException() != null)
       {
-         runProcessOutput.setException(lastFunctionResult.getException());
-         throw (lastFunctionResult.getException());
+         runProcessOutput.setException(runBackendStepOutput.getException());
+         throw (runBackendStepOutput.getException());
       }
+
+      return (runBackendStepOutput);
    }
 
 
@@ -495,15 +522,15 @@ public class RunProcessAction
       String basepullKeyValue = (basepullConfiguration.getKeyValue() != null) ? basepullConfiguration.getKeyValue() : process.getName();
 
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      // if backend specifies that it uses variants, look for that data in the session and append to our basepull key //
+      // if process specifies that it uses variants, look for that data in the session and append to our basepull key //
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      if(process.getSchedule() != null && process.getVariantBackend() != null)
+      if(process.getVariantBackend() != null)
       {
          QSession         session         = QContext.getQSession();
          QBackendMetaData backendMetaData = QContext.getQInstance().getBackend(process.getVariantBackend());
          if(session.getBackendVariants() == null || !session.getBackendVariants().containsKey(backendMetaData.getVariantOptionsTableTypeValue()))
          {
-            LOG.info("Could not find Backend Variant information for Backend '" + backendMetaData.getName() + "'");
+            LOG.warn("Could not find Backend Variant information for Backend '" + backendMetaData.getName() + "'");
          }
          else
          {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/reporting/ExportAction.java

@@ -44,6 +44,7 @@ import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportOutput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountOutput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
@@ -216,7 +217,8 @@ public class ExportAction
       }
       queryInput.getFilter().setLimit(exportInput.getLimit());
       queryInput.setShouldTranslatePossibleValues(true);
-      queryInput.withQueryHint(QueryInput.QueryHint.POTENTIALLY_LARGE_NUMBER_OF_RESULTS);
+      queryInput.withQueryHint(QueryHint.POTENTIALLY_LARGE_NUMBER_OF_RESULTS);
+      queryInput.withQueryHint(QueryHint.MAY_USE_READ_ONLY_BACKEND);
 
       /////////////////////////////////////////////////////////////////
       // tell this query that it needs to put its output into a pipe //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/reporting/GenerateReportAction.java

@@ -59,6 +59,7 @@ import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountOutput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.JoinsContext;
@@ -417,7 +418,8 @@ public class GenerateReportAction extends AbstractQActionFunction<ReportInput, R
             queryInput.setTableName(dataSource.getSourceTable());
             queryInput.setFilter(queryFilter);
             queryInput.setQueryJoins(dataSource.getQueryJoins());
-            queryInput.withQueryHint(QueryInput.QueryHint.POTENTIALLY_LARGE_NUMBER_OF_RESULTS);
+            queryInput.withQueryHint(QueryHint.POTENTIALLY_LARGE_NUMBER_OF_RESULTS);
+            queryInput.withQueryHint(QueryHint.MAY_USE_READ_ONLY_BACKEND);
 
             queryInput.setShouldTranslatePossibleValues(true);
             queryInput.setFieldsToTranslatePossibleValues(setupFieldsToTranslatePossibleValues(reportInput, dataSource, new JoinsContext(reportInput.getInstance(), dataSource.getSourceTable(), dataSource.getQueryJoins(), queryInput.getFilter())));
@@ -457,7 +459,7 @@ public class GenerateReportAction extends AbstractQActionFunction<ReportInput, R
          if(finalTransformStep != null)
          {
             finalTransformStepInput.setRecords(records);
-            finalTransformStep.run(finalTransformStepInput, finalTransformStepOutput);
+            finalTransformStep.runOnePage(finalTransformStepInput, finalTransformStepOutput);
             records = finalTransformStepOutput.getRecords();
          }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/scripts/ExecuteCodeAction.java

@@ -77,7 +77,6 @@ public class ExecuteCodeAction
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    public void run(ExecuteCodeInput input, ExecuteCodeOutput output) throws QException, QCodeException
    {
       QCodeReference codeReference = input.getCodeReference();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/GetAction.java

@@ -65,16 +65,6 @@ public class GetAction
 
 
 
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   public QRecord executeForRecord(GetInput getInput) throws QException
-   {
-      return (execute(getInput).getRecord());
-   }
-
-
-
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -108,7 +98,7 @@ public class GetAction
       }
 
       GetOutput getOutput;
-      boolean usingDefaultGetInterface = false;
+      boolean   usingDefaultGetInterface = false;
       if(getInterface == null)
       {
          getInterface = new DefaultGetInterface();
@@ -140,6 +130,43 @@ public class GetAction
 
 
 
+   /*******************************************************************************
+    ** shorthand way to call for the most common use-case, when you just want the
+    ** output record to be returned.
+    *******************************************************************************/
+   public QRecord executeForRecord(GetInput getInput) throws QException
+   {
+      return (execute(getInput).getRecord());
+   }
+
+
+
+   /*******************************************************************************
+    ** more shorthand way to call for the most common use-case, when you just want the
+    ** output record to be returned, and you just want to pass in a table name and primary key.
+    *******************************************************************************/
+   public static QRecord execute(String tableName, Serializable primaryKey) throws QException
+   {
+      GetAction getAction = new GetAction();
+      GetInput  getInput  = new GetInput(tableName).withPrimaryKey(primaryKey);
+      return getAction.executeForRecord(getInput);
+   }
+
+
+
+   /*******************************************************************************
+    ** more shorthand way to call for the most common use-case, when you just want the
+    ** output record to be returned, and you just want to pass in a table name and unique key
+    *******************************************************************************/
+   public static QRecord execute(String tableName, Map<String, Serializable> uniqueKey) throws QException
+   {
+      GetAction getAction = new GetAction();
+      GetInput  getInput  = new GetInput(tableName).withUniqueKey(uniqueKey);
+      return getAction.executeForRecord(getInput);
+   }
+
+
+
    /*******************************************************************************
     ** Run a GetAction by using the QueryAction instead (e.g., with a filter made
     ** from the pkey/ukey, and returning the single record if found).

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/QueryAction.java

@@ -151,6 +151,22 @@ public class QueryAction
 
 
 
+   /*******************************************************************************
+    ** shorthand way to call for the most common use-case, when you just want the
+    ** records to be returned, and you just want to pass in a table name and filter.
+    *******************************************************************************/
+   public static List<QRecord> execute(String tableName, QQueryFilter filter) throws QException
+   {
+      QueryAction queryAction = new QueryAction();
+      QueryInput  queryInput  = new QueryInput();
+      queryInput.setTableName(tableName);
+      queryInput.setFilter(filter);
+      QueryOutput queryOutput = queryAction.execute(queryInput);
+      return (queryOutput.getRecords());
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/ReplaceAction.java

@@ -157,6 +157,17 @@ public class ReplaceAction extends AbstractQActionFunction<ReplaceInput, Replace
             output.setDeleteOutput(deleteOutput);
          }
 
+         if(input.getSetPrimaryKeyInInsertedRecords())
+         {
+            for(int i = 0; i < insertList.size(); i++)
+            {
+               if(i < insertOutput.getRecords().size())
+               {
+                  insertList.get(i).setValue(primaryKeyField, insertOutput.getRecords().get(i).getValue(primaryKeyField));
+               }
+            }
+         }
+
          if(weOwnTheTransaction)
          {
             transaction.commit();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/UpdateAction.java

@@ -24,6 +24,7 @@ package com.kingsrook.qqq.backend.core.actions.tables;
 
 import java.io.Serializable;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -335,6 +336,9 @@ public class UpdateAction
       QTableMetaData table           = updateInput.getTable();
       QFieldMetaData primaryKeyField = table.getField(table.getPrimaryKeyField());
 
+      /////////////////////////////////////////////////////////////
+      // todo - evolve to use lock tree (e.g., from multi-locks) //
+      /////////////////////////////////////////////////////////////
       List<RecordSecurityLock> onlyWriteLocks = RecordSecurityLockFilters.filterForOnlyWriteLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks()));
 
       for(List<QRecord> page : CollectionUtils.getPages(updateInput.getRecords(), 1000))
@@ -395,7 +399,7 @@ public class UpdateAction
                   QFieldType   fieldType = table.getField(lock.getFieldName()).getType();
                   Serializable lockValue = ValueUtils.getValueAsFieldType(fieldType, oldRecord.getValue(lock.getFieldName()));
 
-                  List<QErrorMessage> errors = ValidateRecordSecurityLockHelper.validateRecordSecurityValue(table, lock, lockValue, fieldType, ValidateRecordSecurityLockHelper.Action.UPDATE);
+                  List<QErrorMessage> errors = ValidateRecordSecurityLockHelper.validateRecordSecurityValue(table, lock, lockValue, fieldType, ValidateRecordSecurityLockHelper.Action.UPDATE, Collections.emptyMap());
                   if(CollectionUtils.nullSafeHasContents(errors))
                   {
                      errors.forEach(e -> record.addError(e));

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/ValidateRecordSecurityLockHelper.java

@@ -101,7 +101,7 @@ public class ValidateRecordSecurityLockHelper
       // actually check lock values //
       ////////////////////////////////
       Map<Serializable, RecordWithErrors> errorRecords = new HashMap<>();
-      evaluateRecordLocks(table, records, action, locksToCheck, errorRecords, new ArrayList<>());
+      evaluateRecordLocks(table, records, action, locksToCheck, errorRecords, new ArrayList<>(), madeUpPrimaryKeys);
 
       /////////////////////////////////
       // propagate errors to records //
@@ -141,7 +141,7 @@ public class ValidateRecordSecurityLockHelper
     ** BUT - WRITE locks - in their case, we read the record no matter what, and in
     ** here we need to verify we have a key that allows us to WRITE the record.
     *******************************************************************************/
-   private static void evaluateRecordLocks(QTableMetaData table, List<QRecord> records, Action action, RecordSecurityLock recordSecurityLock, Map<Serializable, RecordWithErrors> errorRecords, List<Integer> treePosition) throws QException
+   private static void evaluateRecordLocks(QTableMetaData table, List<QRecord> records, Action action, RecordSecurityLock recordSecurityLock, Map<Serializable, RecordWithErrors> errorRecords, List<Integer> treePosition, Map<Serializable, QRecord> madeUpPrimaryKeys) throws QException
    {
       if(recordSecurityLock instanceof MultiRecordSecurityLock multiRecordSecurityLock)
       {
@@ -152,7 +152,7 @@ public class ValidateRecordSecurityLockHelper
          for(RecordSecurityLock childLock : CollectionUtils.nonNullList(multiRecordSecurityLock.getLocks()))
          {
             treePosition.add(i);
-            evaluateRecordLocks(table, records, action, childLock, errorRecords, treePosition);
+            evaluateRecordLocks(table, records, action, childLock, errorRecords, treePosition, madeUpPrimaryKeys);
             treePosition.remove(treePosition.size() - 1);
             i++;
          }
@@ -192,7 +192,7 @@ public class ValidateRecordSecurityLockHelper
             }
 
             Serializable        recordSecurityValue = record.getValue(field.getName());
-            List<QErrorMessage> recordErrors        = validateRecordSecurityValue(table, recordSecurityLock, recordSecurityValue, field.getType(), action);
+            List<QErrorMessage> recordErrors        = validateRecordSecurityValue(table, recordSecurityLock, recordSecurityValue, field.getType(), action, madeUpPrimaryKeys);
             if(CollectionUtils.nullSafeHasContents(recordErrors))
             {
                errorRecords.computeIfAbsent(record.getValue(primaryKeyField), (k) -> new RecordWithErrors(record)).addAll(recordErrors, treePosition);
@@ -337,7 +337,7 @@ public class ValidateRecordSecurityLockHelper
 
                   for(QRecord inputRecord : inputRecords)
                   {
-                     List<QErrorMessage> recordErrors = validateRecordSecurityValue(table, recordSecurityLock, recordSecurityValue, field.getType(), action);
+                     List<QErrorMessage> recordErrors = validateRecordSecurityValue(table, recordSecurityLock, recordSecurityValue, field.getType(), action, madeUpPrimaryKeys);
                      if(CollectionUtils.nullSafeHasContents(recordErrors))
                      {
                         errorRecords.computeIfAbsent(inputRecord.getValue(primaryKeyField), (k) -> new RecordWithErrors(inputRecord)).addAll(recordErrors, treePosition);
@@ -370,14 +370,14 @@ public class ValidateRecordSecurityLockHelper
    {
       String                     primaryKeyField   = table.getPrimaryKeyField();
       Map<Serializable, QRecord> madeUpPrimaryKeys = new HashMap<>();
-      Integer                    madeUpPrimaryKey  = -1;
+      Integer                    madeUpPrimaryKey  = Integer.MIN_VALUE / 2;
       for(QRecord record : records)
       {
          if(record.getValue(primaryKeyField) == null)
          {
             madeUpPrimaryKeys.put(madeUpPrimaryKey, record);
             record.setValue(primaryKeyField, madeUpPrimaryKey);
-            madeUpPrimaryKey--;
+            madeUpPrimaryKey++;
          }
       }
       return madeUpPrimaryKeys;
@@ -390,7 +390,6 @@ public class ValidateRecordSecurityLockHelper
     ** MultiRecordSecurityLock, with only the appropriate lock-scopes being included
     ** (e.g., read-locks for selects, write-locks for insert/update/delete).
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:Indentation")
    static MultiRecordSecurityLock getRecordSecurityLocks(QTableMetaData table, Action action)
    {
       List<RecordSecurityLock> allLocksOnTable = CollectionUtils.nonNullList(table.getRecordSecurityLocks());
@@ -445,9 +444,9 @@ public class ValidateRecordSecurityLockHelper
    /*******************************************************************************
     **
     *******************************************************************************/
-   public static List<QErrorMessage> validateRecordSecurityValue(QTableMetaData table, RecordSecurityLock recordSecurityLock, Serializable recordSecurityValue, QFieldType fieldType, Action action)
+   public static List<QErrorMessage> validateRecordSecurityValue(QTableMetaData table, RecordSecurityLock recordSecurityLock, Serializable recordSecurityValue, QFieldType fieldType, Action action, Map<Serializable, QRecord> madeUpPrimaryKeys)
    {
-      if(recordSecurityValue == null)
+      if(recordSecurityValue == null || (madeUpPrimaryKeys != null && madeUpPrimaryKeys.containsKey(recordSecurityValue)))
       {
          /////////////////////////////////////////////////////////////////
          // handle null values - error if the NullValueBehavior is DENY //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/QPossibleValueTranslator.java

@@ -32,13 +32,12 @@ import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
-import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QValueException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
-import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
@@ -78,44 +77,6 @@ public class QPossibleValueTranslator
 
    private int maxSizePerPvsCache = 50_000;
 
-   private Map<String, QBackendTransaction> transactionsPerTable = new HashMap<>();
-
-   // todo not commit - remove instance & session - use Context
-
-
-   boolean useTransactionsAsConnectionPool = false;
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   private QBackendTransaction getTransaction(String tableName)
-   {
-      /////////////////////////////////////////////////////////////
-      // mmm, this does cut down on connections used -           //
-      // especially seems helpful in big exports.                //
-      // but, let's just start using connection pools instead... //
-      /////////////////////////////////////////////////////////////
-      if(useTransactionsAsConnectionPool)
-      {
-         try
-         {
-            if(!transactionsPerTable.containsKey(tableName))
-            {
-               transactionsPerTable.put(tableName, QBackendTransaction.openFor(new InsertInput(tableName)));
-            }
-
-            return (transactionsPerTable.get(tableName));
-         }
-         catch(Exception e)
-         {
-            LOG.warn("Error opening transaction for table", logPair("tableName", tableName));
-         }
-      }
-
-      return null;
-   }
 
 
    /*******************************************************************************
@@ -421,7 +382,6 @@ public class QPossibleValueTranslator
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:Indentation")
    private String doFormatPossibleValue(String formatString, List<String> valueFields, Object id, String label)
    {
       List<Object> values = new ArrayList<>();
@@ -601,7 +561,7 @@ public class QPossibleValueTranslator
             QueryInput queryInput = new QueryInput();
             queryInput.setTableName(tableName);
             queryInput.setFilter(new QQueryFilter().withCriteria(new QFilterCriteria(idField, QCriteriaOperator.IN, page)));
-            queryInput.setTransaction(getTransaction(tableName));
+            queryInput.hasQueryHint(QueryHint.MAY_USE_READ_ONLY_BACKEND);
 
             /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
             // when querying for possible values, we do want to generate their display values, which makes record labels, which are usually used as PVS labels //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceEnricher.java

@@ -384,9 +384,9 @@ public class QInstanceEnricher
          process.setLabel(nameToLabel(process.getName()));
       }
 
-      if(process.getStepList() != null)
+      for(QStepMetaData step : CollectionUtils.nonNullMap(process.getAllSteps()).values())
       {
-         process.getStepList().forEach(this::enrichStep);
+         enrichStep(step);
       }
 
       for(QSupplementalProcessMetaData supplementalProcessMetaData : CollectionUtils.nonNullMap(process.getSupplementalMetaData()).values())

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceHelpContentManager.java

@@ -43,6 +43,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.help.HelpFormat;
 import com.kingsrook.qqq.backend.core.model.metadata.help.HelpRole;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpRole;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@@ -111,6 +112,7 @@ public class QInstanceHelpContentManager
          String processName = nameValuePairs.get("process");
          String fieldName   = nameValuePairs.get("field");
          String sectionName = nameValuePairs.get("section");
+         String stepName = nameValuePairs.get("step");
          String widgetName  = nameValuePairs.get("widget");
          String slotName    = nameValuePairs.get("slot");
 
@@ -145,7 +147,7 @@ public class QInstanceHelpContentManager
          }
          else if(StringUtils.hasContent(processName))
          {
-            processHelpContentForProcess(key, processName, fieldName, roles, helpContent);
+            processHelpContentForProcess(key, processName, fieldName, stepName, roles, helpContent);
          }
          else if(StringUtils.hasContent(widgetName))
          {
@@ -208,6 +210,10 @@ public class QInstanceHelpContentManager
             optionalSection.get().removeHelpContent(roles);
          }
       }
+      else
+      {
+         LOG.info("Unrecognized key format for table help content", logPair("key", key));
+      }
    }
 
 
@@ -215,7 +221,7 @@ public class QInstanceHelpContentManager
    /*******************************************************************************
     **
     *******************************************************************************/
-   private static void processHelpContentForProcess(String key, String processName, String fieldName, Set<HelpRole> roles, QHelpContent helpContent)
+   private static void processHelpContentForProcess(String key, String processName, String fieldName, String stepName, Set<HelpRole> roles, QHelpContent helpContent)
    {
       QProcessMetaData process = QContext.getQInstance().getProcess(processName);
       if(process == null)
@@ -244,6 +250,30 @@ public class QInstanceHelpContentManager
             optionalField.get().removeHelpContent(roles);
          }
       }
+      else if(StringUtils.hasContent(stepName))
+      {
+         /////////////////////////////
+         // handle a process screen //
+         /////////////////////////////
+         QFrontendStepMetaData frontendStep = process.getFrontendStep(stepName);
+
+         if(frontendStep == null)
+         {
+            LOG.info("Unrecognized process step in help content", logPair("key", key));
+         }
+         else if(helpContent != null)
+         {
+            frontendStep.withHelpContent(helpContent);
+         }
+         else
+         {
+            frontendStep.removeHelpContent(roles);
+         }
+      }
+      else
+      {
+         LOG.info("Unrecognized key format for process help content", logPair("key", key));
+      }
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidator.java

@@ -109,6 +109,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 import com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeLambda;
 import org.quartz.CronExpression;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
 /*******************************************************************************
@@ -151,6 +152,7 @@ public class QInstanceValidator
       // once, during the enrichment/validation work, so, capture it, and store it back in the instance. //
       /////////////////////////////////////////////////////////////////////////////////////////////////////
       JoinGraph joinGraph = null;
+      long      start     = System.currentTimeMillis();
       try
       {
          /////////////////////////////////////////////////////////////////////////////////////////////////
@@ -191,6 +193,9 @@ public class QInstanceValidator
          validateUniqueTopLevelNames(qInstance);
 
          runPlugins(QInstance.class, qInstance, qInstance);
+
+         long end = System.currentTimeMillis();
+         LOG.info("Validation (and enrichment) performance", logPair("millis", (end - start)));
       }
       catch(Exception e)
       {
@@ -209,6 +214,17 @@ public class QInstanceValidator
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void revalidate(QInstance qInstance) throws QInstanceValidationException
+   {
+      qInstance.setHasBeenValidated(null);
+      validate(qInstance);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -657,17 +673,20 @@ public class QInstanceValidator
             {
                if(assertCondition(CollectionUtils.nullSafeHasContents(exposedJoin.getJoinPath()), joinPrefix + "is missing a joinPath."))
                {
-                  joinConnectionsForTable = Objects.requireNonNullElseGet(joinConnectionsForTable, () -> joinGraph.getJoinConnections(table.getName()));
-
-                  boolean foundJoinConnection = false;
-                  for(JoinGraph.JoinConnectionList joinConnectionList : joinConnectionsForTable)
+                  if(joinGraph != null)
                   {
-                     if(joinConnectionList.matchesJoinPath(exposedJoin.getJoinPath()))
+                     joinConnectionsForTable = Objects.requireNonNullElseGet(joinConnectionsForTable, () -> joinGraph.getJoinConnections(table.getName()));
+
+                     boolean foundJoinConnection = false;
+                     for(JoinGraph.JoinConnectionList joinConnectionList : joinConnectionsForTable)
                      {
-                        foundJoinConnection = true;
+                        if(joinConnectionList.matchesJoinPath(exposedJoin.getJoinPath()))
+                        {
+                           foundJoinConnection = true;
+                        }
                      }
+                     assertCondition(foundJoinConnection, joinPrefix + "specified a joinPath [" + exposedJoin.getJoinPath() + "] which does not match a valid join connection in the instance.");
                   }
-                  assertCondition(foundJoinConnection, joinPrefix + "specified a joinPath [" + exposedJoin.getJoinPath() + "] which does not match a valid join connection in the instance.");
 
                   assertCondition(!usedJoinPaths.contains(exposedJoin.getJoinPath()), tablePrefix + "has more than one join with the joinPath: " + exposedJoin.getJoinPath());
                   usedJoinPaths.add(exposedJoin.getJoinPath());
@@ -1468,7 +1487,7 @@ public class QInstanceValidator
                                  warn("Error loading expectedType for field [" + fieldMetaData.getName() + "] in process [" + processName + "]: " + e.getMessage());
                               }
 
-                              validateSimpleCodeReference("Process " + processName + " code reference: ", codeReference, expectedClass);
+                              validateSimpleCodeReference("Process " + processName + " code reference:", codeReference, expectedClass);
                            }
                         }
                      }
@@ -1476,6 +1495,14 @@ public class QInstanceValidator
                }
             }
 
+            if(process.getCancelStep() != null)
+            {
+               if(assertCondition(process.getCancelStep().getCode() != null, "Cancel step is missing a code reference, in process " + processName))
+               {
+                  validateSimpleCodeReference("Process " + processName + " cancel step code reference: ", process.getCancelStep().getCode(), BackendStep.class);
+               }
+            }
+
             ///////////////////////////////////////////////////////////////////////////////
             // if the process has a schedule, make sure required schedule data populated //
             ///////////////////////////////////////////////////////////////////////////////
@@ -1487,7 +1514,11 @@ public class QInstanceValidator
 
             if(process.getVariantBackend() != null)
             {
-               assertCondition(qInstance.getBackend(process.getVariantBackend()) != null, "Process " + processName + ", a variant backend was not found named " + process.getVariantBackend());
+               if(qInstance.getBackends() != null)
+               {
+                  assertCondition(qInstance.getBackend(process.getVariantBackend()) != null, "Process " + processName + ", a variant backend was not found named " + process.getVariantBackend());
+               }
+
                assertCondition(process.getVariantRunStrategy() != null, "A variant run strategy was not set for process " + processName + " (which does specify a variant backend)");
             }
             else

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/logging/LogPair.java

@@ -79,7 +79,7 @@ public class LogPair
       }
       else if(value instanceof LogPair[] subLogPairs)
       {
-         String subLogPairsString = Arrays.stream(subLogPairs).map(LogPair::toString).collect(Collectors.joining(","));
+         String subLogPairsString = Arrays.stream(subLogPairs).filter(Objects::nonNull).map(LogPair::toString).collect(Collectors.joining(","));
          valueString = '{' + subLogPairsString + '}';
       }
       else if(value instanceof UnsafeSupplier<?, ?> us)

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/logging/QLogger.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.logging;
 
 
+import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
@@ -147,6 +148,28 @@ public class QLogger
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public <T extends Throwable> T warnAndThrow(T t, LogPair... logPairs) throws T
+   {
+      warn(t.getMessage(), t, logPairs);
+      throw (t);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public <T extends Throwable> T errorAndThrow(T t, LogPair... logPairs) throws T
+   {
+      error(t.getMessage(), t, logPairs);
+      throw (t);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -595,7 +618,10 @@ public class QLogger
             {
                user = session.getUser().getIdReference();
             }
-            sessionLogPair = logPair("session", logPair("id", session.getUuid()), logPair("user", user));
+
+            LogPair variantsLogPair = getVariantsLogPair(session);
+
+            sessionLogPair = logPair("session", logPair("id", session.getUuid()), logPair("user", user), variantsLogPair);
          }
 
          try
@@ -615,6 +641,38 @@ public class QLogger
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static LogPair getVariantsLogPair(QSession session)
+   {
+      LogPair variantsLogPair = null;
+      try
+      {
+         if(session.getBackendVariants() != null)
+         {
+            LogPair[] variants = new LogPair[session.getBackendVariants().size()];
+
+            int i = 0;
+            for(Map.Entry<String, Serializable> entry : session.getBackendVariants().entrySet())
+            {
+               variants[i] = new LogPair(entry.getKey(), entry.getValue());
+            }
+
+            variantsLogPair = new LogPair("variants", variants);
+         }
+      }
+      catch(Exception e)
+      {
+         ////////////////
+         // leave null //
+         ////////////////
+      }
+      return variantsLogPair;
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/MetaDataProducerInterface.java

@@ -23,8 +23,8 @@ package com.kingsrook.qqq.backend.core.model;
 
 
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerOutput;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
 
 
 /*******************************************************************************
@@ -42,7 +42,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
  ** implement this interface.  or, same idea for a QRecordEntity that provides
  ** its own TableMetaData.
  *******************************************************************************/
-public interface MetaDataProducerInterface<T extends TopLevelMetaDataInterface>
+public interface MetaDataProducerInterface<T extends MetaDataProducerOutput>
 {
    int DEFAULT_SORT_ORDER = 500;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/ProcessState.java

@@ -29,6 +29,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Optional;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
 
 
 /*******************************************************************************
@@ -41,6 +42,11 @@ public class ProcessState implements Serializable
    private List<String>              stepList     = new ArrayList<>();
    private Optional<String>          nextStepName = Optional.empty();
 
+   /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   // maybe, remove this altogether - just let the frontend compute & send if needed... but how does it know last version...? //
+   /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   private List<QFrontendStepMetaData> updatedFrontendStepList = null;
+
 
 
    /*******************************************************************************
@@ -139,4 +145,36 @@ public class ProcessState implements Serializable
    {
       this.stepList = stepList;
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for updatedFrontendStepList
+    *******************************************************************************/
+   public List<QFrontendStepMetaData> getUpdatedFrontendStepList()
+   {
+      return (this.updatedFrontendStepList);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for updatedFrontendStepList
+    *******************************************************************************/
+   public void setUpdatedFrontendStepList(List<QFrontendStepMetaData> updatedFrontendStepList)
+   {
+      this.updatedFrontendStepList = updatedFrontendStepList;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for updatedFrontendStepList
+    *******************************************************************************/
+   public ProcessState withUpdatedFrontendStepList(List<QFrontendStepMetaData> updatedFrontendStepList)
+   {
+      this.updatedFrontendStepList = updatedFrontendStepList;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/RunBackendStepOutput.java

@@ -27,10 +27,14 @@ import java.math.BigDecimal;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractActionOutput;
 import com.kingsrook.qqq.backend.core.model.actions.audits.AuditInput;
 import com.kingsrook.qqq.backend.core.model.actions.audits.AuditSingleInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 
 
@@ -40,9 +44,13 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  *******************************************************************************/
 public class RunBackendStepOutput extends AbstractActionOutput implements Serializable
 {
+   private String processName;
+
    private ProcessState processState;
    private Exception    exception; // todo - make optional
 
+   private String overrideLastStepName; // todo - does this need to go into state too??
+
    private List<AuditInput> auditInputList = new ArrayList<>();
 
 
@@ -78,6 +86,7 @@ public class RunBackendStepOutput extends AbstractActionOutput implements Serial
    public void seedFromRequest(RunBackendStepInput runBackendStepInput)
    {
       this.processState = runBackendStepInput.getProcessState();
+      this.processName = runBackendStepInput.getProcessName();
    }
 
 
@@ -312,4 +321,111 @@ public class RunBackendStepOutput extends AbstractActionOutput implements Serial
       auditInput.addAuditSingleInput(auditSingleInput);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for overrideLastStepName
+    *******************************************************************************/
+   public String getOverrideLastStepName()
+   {
+      return (this.overrideLastStepName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for overrideLastStepName
+    *******************************************************************************/
+   public void setOverrideLastStepName(String overrideLastStepName)
+   {
+      this.overrideLastStepName = overrideLastStepName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for overrideLastStepName
+    *******************************************************************************/
+   public RunBackendStepOutput withOverrideLastStepName(String overrideLastStepName)
+   {
+      this.overrideLastStepName = overrideLastStepName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void updateStepList(List<String> stepList)
+   {
+      getProcessState().setStepList(stepList);
+
+      if(processName == null)
+      {
+         throw (new QRuntimeException("ProcessName was not set in this object, therefore updateStepList cannot complete successfully.  Try to manually call setProcessName as a work around."));
+      }
+
+      QProcessMetaData processMetaData = QContext.getQInstance().getProcess(processName);
+
+      ArrayList<QFrontendStepMetaData> updatedFrontendStepList = new ArrayList<>(stepList.stream()
+         .map(name -> processMetaData.getStep(name))
+         .filter(step -> step instanceof QFrontendStepMetaData)
+         .map(step -> (QFrontendStepMetaData) step)
+         .toList());
+
+      setUpdatedFrontendStepList(updatedFrontendStepList);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for processName
+    *******************************************************************************/
+   public String getProcessName()
+   {
+      return (this.processName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for processName
+    *******************************************************************************/
+   public void setProcessName(String processName)
+   {
+      this.processName = processName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for processName
+    *******************************************************************************/
+   public RunBackendStepOutput withProcessName(String processName)
+   {
+      this.processName = processName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for updatedFrontendStepList
+    *******************************************************************************/
+   public List<QFrontendStepMetaData> getUpdatedFrontendStepList()
+   {
+      return (this.processState.getUpdatedFrontendStepList());
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for updatedFrontendStepList
+    *******************************************************************************/
+   public void setUpdatedFrontendStepList(List<QFrontendStepMetaData> updatedFrontendStepList)
+   {
+      this.processState.setUpdatedFrontendStepList(updatedFrontendStepList);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/RunProcessOutput.java

@@ -32,6 +32,7 @@ import java.util.Map;
 import java.util.Optional;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractActionOutput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 
 
@@ -327,4 +328,25 @@ public class RunProcessOutput extends AbstractActionOutput implements Serializab
    {
       return exception;
    }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void setUpdatedFrontendStepList(List<QFrontendStepMetaData> updatedFrontendStepList)
+   {
+      this.processState.setUpdatedFrontendStepList(updatedFrontendStepList);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public List<QFrontendStepMetaData> getUpdatedFrontendStepList()
+   {
+      return this.processState.getUpdatedFrontendStepList();
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/QueryHint.java

@@ -0,0 +1,38 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.actions.tables;
+
+
+/*******************************************************************************
+ ** Information about the query that an application (or qqq service) may know and
+ ** want to tell the backend, that can help influence how the backend processes
+ ** query.
+ **
+ ** For example, a query with potentially a large result set, for MySQL backend,
+ ** we may want to configure the result set to stream results rather than do its
+ ** default in-memory thing.  See RDBMSQueryAction for usage.
+ *******************************************************************************/
+public enum QueryHint
+{
+   POTENTIALLY_LARGE_NUMBER_OF_RESULTS,
+   MAY_USE_READ_ONLY_BACKEND
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/aggregate/AggregateInput.java

@@ -23,8 +23,10 @@ package com.kingsrook.qqq.backend.core.model.actions.tables.aggregate;
 
 
 import java.util.ArrayList;
+import java.util.EnumSet;
 import java.util.List;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
 
@@ -44,6 +46,8 @@ public class AggregateInput extends AbstractTableActionInput
 
    private List<QueryJoin> queryJoins = null;
 
+   private EnumSet<QueryHint> queryHints = EnumSet.noneOf(QueryHint.class);
+
 
 
    /*******************************************************************************
@@ -302,4 +306,78 @@ public class AggregateInput extends AbstractTableActionInput
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for queryHints
+    *******************************************************************************/
+   public EnumSet<QueryHint> getQueryHints()
+   {
+      return (this.queryHints);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for queryHints
+    *******************************************************************************/
+   public void setQueryHints(EnumSet<QueryHint> queryHints)
+   {
+      this.queryHints = queryHints;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for queryHints
+    *******************************************************************************/
+   public AggregateInput withQueryHints(EnumSet<QueryHint> queryHints)
+   {
+      this.queryHints = queryHints;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for queryHints
+    *******************************************************************************/
+   public AggregateInput withQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints == null)
+      {
+         this.queryHints = EnumSet.noneOf(QueryHint.class);
+      }
+      this.queryHints.add(queryHint);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for queryHints
+    *******************************************************************************/
+   public AggregateInput withoutQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints != null)
+      {
+         this.queryHints.remove(queryHint);
+      }
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** null-safely check if query hints map contains the specified hint
+    *******************************************************************************/
+   public boolean hasQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints == null)
+      {
+         return (false);
+      }
+
+      return (queryHints.contains(queryHint));
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/count/CountInput.java

@@ -23,8 +23,10 @@ package com.kingsrook.qqq.backend.core.model.actions.tables.count;
 
 
 import java.util.ArrayList;
+import java.util.EnumSet;
 import java.util.List;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
 
@@ -42,6 +44,8 @@ public class CountInput extends AbstractTableActionInput
    private List<QueryJoin> queryJoins           = null;
    private Boolean         includeDistinctCount = false;
 
+   private EnumSet<QueryHint> queryHints = EnumSet.noneOf(QueryHint.class);
+
 
 
    /*******************************************************************************
@@ -207,4 +211,78 @@ public class CountInput extends AbstractTableActionInput
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for queryHints
+    *******************************************************************************/
+   public EnumSet<QueryHint> getQueryHints()
+   {
+      return (this.queryHints);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for queryHints
+    *******************************************************************************/
+   public void setQueryHints(EnumSet<QueryHint> queryHints)
+   {
+      this.queryHints = queryHints;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for queryHints
+    *******************************************************************************/
+   public CountInput withQueryHints(EnumSet<QueryHint> queryHints)
+   {
+      this.queryHints = queryHints;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for queryHints
+    *******************************************************************************/
+   public CountInput withQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints == null)
+      {
+         this.queryHints = EnumSet.noneOf(QueryHint.class);
+      }
+      this.queryHints.add(queryHint);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for queryHints
+    *******************************************************************************/
+   public CountInput withoutQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints != null)
+      {
+         this.queryHints.remove(queryHint);
+      }
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** null-safely check if query hints map contains the specified hint
+    *******************************************************************************/
+   public boolean hasQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints == null)
+      {
+         return (false);
+      }
+
+      return (queryHints.contains(queryHint));
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/insert/InsertInput.java

@@ -112,7 +112,7 @@ public class InsertInput extends AbstractTableActionInput
    /*******************************************************************************
     **
     *******************************************************************************/
-   public InsertInput withRecordEntities(List<QRecordEntity> recordEntityList)
+   public InsertInput withRecordEntities(List<? extends QRecordEntity> recordEntityList)
    {
       for(QRecordEntity recordEntity : CollectionUtils.nonNullList(recordEntityList))
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QueryInput.java

@@ -25,19 +25,24 @@ package com.kingsrook.qqq.backend.core.model.actions.tables.query;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.EnumSet;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Set;
 import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
 
 
 /*******************************************************************************
  ** Input data for the Query action
  **
+ ** Todo - maybe make a class between AbstractTableActionInput and {QueryInput,
+ ** CountInput, and AggregateInput}, with common attributes for all of these
+ ** "read" operations (like, queryHints,
  *******************************************************************************/
-public class QueryInput extends AbstractTableActionInput implements QueryOrGetInputInterface
+public class QueryInput extends AbstractTableActionInput implements QueryOrGetInputInterface, Cloneable
 {
    private QBackendTransaction transaction;
    private QQueryFilter        filter;
@@ -73,22 +78,6 @@ public class QueryInput extends AbstractTableActionInput implements QueryOrGetIn
 
 
 
-   /*******************************************************************************
-    ** Information about the query that an application (or qqq service) may know and
-    ** want to tell the backend, that can help influence how the backend processes
-    ** query.
-    **
-    ** For example, a query with potentially a large result set, for MySQL backend,
-    ** we may want to configure the result set to stream results rather than do its
-    ** default in-memory thing.  See RDBMSQueryAction for usage.
-    *******************************************************************************/
-   public enum QueryHint
-   {
-      POTENTIALLY_LARGE_NUMBER_OF_RESULTS
-   }
-
-
-
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -109,6 +98,40 @@ public class QueryInput extends AbstractTableActionInput implements QueryOrGetIn
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public QueryInput clone() throws CloneNotSupportedException
+   {
+      QueryInput clone = (QueryInput) super.clone();
+
+      if(fieldsToTranslatePossibleValues != null)
+      {
+         clone.fieldsToTranslatePossibleValues = new HashSet<>(fieldsToTranslatePossibleValues);
+      }
+
+      if(queryJoins != null)
+      {
+         clone.queryJoins = new ArrayList<>(queryJoins);
+      }
+
+      if(clone.associationNamesToInclude != null)
+      {
+         clone.associationNamesToInclude = new HashSet<>(associationNamesToInclude);
+      }
+
+      if(queryHints != null)
+      {
+         clone.queryHints = EnumSet.noneOf(QueryHint.class);
+         clone.queryHints.addAll(queryHints);
+      }
+
+      return (clone);
+   }
+
+
+
    /*******************************************************************************
     ** Getter for filter
     **
@@ -648,4 +671,19 @@ public class QueryInput extends AbstractTableActionInput implements QueryOrGetIn
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** null-safely check if query hints map contains the specified hint
+    *******************************************************************************/
+   public boolean hasQueryHint(QueryHint queryHint)
+   {
+      if(this.queryHints == null)
+      {
+         return (false);
+      }
+
+      return (queryHints.contains(queryHint));
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/replace/ReplaceInput.java

@@ -39,8 +39,9 @@ public class ReplaceInput extends AbstractTableActionInput
    private UniqueKey           key;
    private List<QRecord>       records;
    private QQueryFilter        filter;
-   private boolean             performDeletes            = true;
-   private boolean             allowNullKeyValuesToEqual = false;
+   private boolean performDeletes                 = true;
+   private boolean allowNullKeyValuesToEqual      = false;
+   private boolean setPrimaryKeyInInsertedRecords = false;
 
    private boolean omitDmlAudit = false;
 
@@ -271,4 +272,35 @@ public class ReplaceInput extends AbstractTableActionInput
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for setPrimaryKeyInInsertedRecords
+    *******************************************************************************/
+   public boolean getSetPrimaryKeyInInsertedRecords()
+   {
+      return (this.setPrimaryKeyInInsertedRecords);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for setPrimaryKeyInInsertedRecords
+    *******************************************************************************/
+   public void setSetPrimaryKeyInInsertedRecords(boolean setPrimaryKeyInInsertedRecords)
+   {
+      this.setPrimaryKeyInInsertedRecords = setPrimaryKeyInInsertedRecords;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for setPrimaryKeyInInsertedRecords
+    *******************************************************************************/
+   public ReplaceInput withSetPrimaryKeyInInsertedRecords(boolean setPrimaryKeyInInsertedRecords)
+   {
+      this.setPrimaryKeyInInsertedRecords = setPrimaryKeyInInsertedRecords;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/update/UpdateInput.java

@@ -120,7 +120,7 @@ public class UpdateInput extends AbstractTableActionInput
    /*******************************************************************************
     **
     *******************************************************************************/
-   public UpdateInput withRecordEntities(List<QRecordEntity> recordEntityList)
+   public UpdateInput withRecordEntities(List<? extends QRecordEntity> recordEntityList)
    {
       for(QRecordEntity recordEntity : CollectionUtils.nonNullList(recordEntityList))
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/dashboard/widgets/CompositeWidgetData.java

@@ -54,6 +54,7 @@ public class CompositeWidgetData extends AbstractBlockWidgetData<CompositeWidget
       /////////////////////////////////////////////////////////////
       // note, these are used in QQQ FMD CompositeWidgetData.tsx //
       /////////////////////////////////////////////////////////////
+      FLEX_COLUMN,
       FLEX_ROW_WRAPPED,
       FLEX_ROW_SPACE_BETWEEN,
       TABLE_SUB_ROW_DETAILS,

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/dashboard/widgets/FilterAndColumnsSetupData.java

@@ -0,0 +1,196 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.dashboard.widgets;
+
+
+import java.util.List;
+
+
+/*******************************************************************************
+ ** Model containing datastructure expected by frontend filter and columns setup widget
+ **
+ *******************************************************************************/
+public class FilterAndColumnsSetupData extends QWidgetData
+{
+   private String       tableName;
+   private Boolean      allowVariables = false;
+   private Boolean      hideColumns    = false;
+   private List<String> filterDefaultFieldNames;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilterAndColumnsSetupData()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilterAndColumnsSetupData(String tableName, Boolean allowVariables, Boolean hideColumns, List<String> filterDefaultFieldNames)
+   {
+      this.tableName = tableName;
+      this.allowVariables = allowVariables;
+      this.hideColumns = hideColumns;
+      this.filterDefaultFieldNames = filterDefaultFieldNames;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for type
+    **
+    *******************************************************************************/
+   public String getType()
+   {
+      return WidgetType.FILTER_AND_COLUMNS_SETUP.getType();
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    *******************************************************************************/
+   public String getTableName()
+   {
+      return (this.tableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    *******************************************************************************/
+   public void setTableName(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableName
+    *******************************************************************************/
+   public FilterAndColumnsSetupData withTableName(String tableName)
+   {
+      this.tableName = tableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for hideColumns
+    *******************************************************************************/
+   public Boolean getHideColumns()
+   {
+      return (this.hideColumns);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for hideColumns
+    *******************************************************************************/
+   public void setHideColumns(Boolean hideColumns)
+   {
+      this.hideColumns = hideColumns;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for hideColumns
+    *******************************************************************************/
+   public FilterAndColumnsSetupData withHideColumns(Boolean hideColumns)
+   {
+      this.hideColumns = hideColumns;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for filterDefaultFieldNames
+    *******************************************************************************/
+   public List<String> getFilterDefaultFieldNames()
+   {
+      return (this.filterDefaultFieldNames);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for filterDefaultFieldNames
+    *******************************************************************************/
+   public void setFilterDefaultFieldNames(List<String> filterDefaultFieldNames)
+   {
+      this.filterDefaultFieldNames = filterDefaultFieldNames;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for filterDefaultFieldNames
+    *******************************************************************************/
+   public FilterAndColumnsSetupData withFilterDefaultFieldNames(List<String> filterDefaultFieldNames)
+   {
+      this.filterDefaultFieldNames = filterDefaultFieldNames;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for allowVariables
+    *******************************************************************************/
+   public Boolean getAllowVariables()
+   {
+      return (this.allowVariables);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for allowVariables
+    *******************************************************************************/
+   public void setAllowVariables(Boolean allowVariables)
+   {
+      this.allowVariables = allowVariables;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for allowVariables
+    *******************************************************************************/
+   public FilterAndColumnsSetupData withAllowVariables(Boolean allowVariables)
+   {
+      this.allowVariables = allowVariables;
+      return (this);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/dashboard/widgets/MultiTableData.java

@@ -0,0 +1,97 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.dashboard.widgets;
+
+
+import java.util.List;
+
+
+/*******************************************************************************
+ ** Model containing datastructure expected by frontend bar chart widget
+ **
+ *******************************************************************************/
+public class MultiTableData extends QWidgetData
+{
+   List<TableData> tableDataList;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public MultiTableData()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public MultiTableData(List<TableData> tableDataList)
+   {
+      setTableDataList(tableDataList);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for type
+    **
+    *******************************************************************************/
+   public String getType()
+   {
+      return WidgetType.MULTI_TABLE.getType();
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableDataList
+    *******************************************************************************/
+   public List<TableData> getTableDataList()
+   {
+      return (this.tableDataList);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableDataList
+    *******************************************************************************/
+   public void setTableDataList(List<TableData> tableDataList)
+   {
+      this.tableDataList = tableDataList;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableDataList
+    *******************************************************************************/
+   public MultiTableData withTableDataList(List<TableData> tableDataList)
+   {
+      this.tableDataList = tableDataList;
+      return (this);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/dashboard/widgets/WidgetType.java

@@ -42,6 +42,7 @@ public enum WidgetType
    SMALL_LINE_CHART("smallLineChart"),
    LOCATION("location"),
    MULTI_STATISTICS("multiStatistics"),
+   MULTI_TABLE("multiTable"),
    PIE_CHART("pieChart"),
    QUICK_SIGHT_CHART("quickSightChart"),
    STATISTICS("statistics"),
@@ -68,7 +69,7 @@ public enum WidgetType
    DYNAMIC_FORM("dynamicForm"),
    DATA_BAG_VIEWER("dataBagViewer"),
    PIVOT_TABLE_SETUP("pivotTableSetup"),
-   REPORT_SETUP("reportSetup"),
+   FILTER_AND_COLUMNS_SETUP("filterAndColumnsSetup"),
    SCRIPT_VIEWER("scriptViewer");
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QField.java

@@ -49,6 +49,11 @@ public @interface QField
     *******************************************************************************/
    String backendName() default "";
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   boolean isPrimaryKey() default false;
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QIgnore.java

@@ -0,0 +1,39 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.data;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+
+/*******************************************************************************
+ ** Marker - that a piece of code should be ignored (e.g., a field not treated as
+ ** a @QField)
+ *******************************************************************************/
+@Target({ ElementType.FIELD, ElementType.METHOD })
+@Retention(RetentionPolicy.RUNTIME)
+public @interface QIgnore
+{
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecord.java

@@ -35,12 +35,15 @@ import java.util.LinkedHashMap;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.statusmessages.QErrorMessage;
 import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 import org.apache.commons.lang3.SerializationUtils;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
@@ -462,6 +465,7 @@ public class QRecord implements Serializable
    }
 
 
+
    /*******************************************************************************
     ** Getter for a single field's value
     **
@@ -616,6 +620,22 @@ public class QRecord implements Serializable
 
 
 
+   /*******************************************************************************
+    ** Getter for errors
+    **
+    *******************************************************************************/
+   @JsonIgnore
+   public String getErrorsAsString()
+   {
+      if(CollectionUtils.nullSafeHasContents(errors))
+      {
+         return StringUtils.join("; ", errors.stream().map(e -> e.getMessage()).toList());
+      }
+      return ("");
+   }
+
+
+
    /*******************************************************************************
     ** Setter for errors
     **
@@ -732,6 +752,22 @@ public class QRecord implements Serializable
 
 
 
+   /*******************************************************************************
+    ** Getter for warnings
+    **
+    *******************************************************************************/
+   @JsonIgnore
+   public String getWarningsAsString()
+   {
+      if(CollectionUtils.nullSafeHasContents(warnings))
+      {
+         return StringUtils.join("; ", warnings.stream().map(e -> e.getMessage()).toList());
+      }
+      return ("");
+   }
+
+
+
    /*******************************************************************************
     ** Setter for warnings
     *******************************************************************************/
@@ -742,6 +778,18 @@ public class QRecord implements Serializable
 
 
 
+   /*******************************************************************************
+    ** Fluently Add one warning to this record
+    **
+    *******************************************************************************/
+   public QRecord withWarning(QWarningMessage warning)
+   {
+      addWarning(warning);
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     ** Fluent setter for warnings
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecordEntity.java

@@ -218,6 +218,7 @@ public abstract class QRecordEntity
    }
 
 
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -296,7 +297,19 @@ public abstract class QRecordEntity
                   }
                   else
                   {
-                     LOG.debug("Skipping field without @QField annotation", logPair("class", c.getSimpleName()), logPair("fieldName", fieldName));
+                     Optional<QIgnore>      ignoreAnnotation      = getQIgnoreAnnotation(c, fieldName);
+                     Optional<QAssociation> associationAnnotation = getQAssociationAnnotation(c, fieldName);
+
+                     if(ignoreAnnotation.isPresent() || associationAnnotation.isPresent())
+                     {
+                        ////////////////////////////////////////////////////////////
+                        // silently skip if marked as an association or an ignore //
+                        ////////////////////////////////////////////////////////////
+                     }
+                     else
+                     {
+                        LOG.debug("Skipping field without @QField annotation", logPair("class", c.getSimpleName()), logPair("fieldName", fieldName));
+                     }
                   }
                }
                else
@@ -360,6 +373,16 @@ public abstract class QRecordEntity
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static Optional<QIgnore> getQIgnoreAnnotation(Class<? extends QRecordEntity> c, String ignoreName)
+   {
+      return (getAnnotationOnField(c, QIgnore.class, ignoreName));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -419,9 +442,9 @@ public abstract class QRecordEntity
          }
          else
          {
-            if(!method.getName().equals("getClass"))
+            if(!method.getName().equals("getClass") && method.getAnnotation(QIgnore.class) == null)
             {
-               LOG.debug("Method [" + method.getName() + "] looks like a getter, but its return type, [" + method.getReturnType() + "], isn't supported.");
+               LOG.debug("Method [" + method.getName() + "] in [" + method.getDeclaringClass().getSimpleName() + "] looks like a getter, but its return type, [" + method.getReturnType().getSimpleName() + "], isn't supported.");
             }
          }
       }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecordEnum.java

@@ -145,7 +145,7 @@ public interface QRecordEnum
          {
             if(!method.getName().equals("getClass") && !method.getName().equals("getDeclaringClass") && !method.getName().equals("getPossibleValueId"))
             {
-               LOG.debug("Method [" + method.getName() + "] looks like a getter, but its return type, [" + method.getReturnType() + "], isn't supported.");
+               LOG.debug("Method [" + method.getName() + "] in [" + method.getDeclaringClass().getSimpleName() + "] looks like a getter, but its return type, [" + method.getReturnType().getSimpleName() + "], isn't supported.");
             }
          }
       }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducer.java

@@ -30,7 +30,7 @@ import com.kingsrook.qqq.backend.core.model.MetaDataProducerInterface;
  ** MetaDataProducerHelper, to put point at a package full of these, and populate
  ** your whole QInstance.
  *******************************************************************************/
-public abstract class MetaDataProducer<T extends TopLevelMetaDataInterface> implements MetaDataProducerInterface<T>
+public abstract class MetaDataProducer<T extends MetaDataProducerOutput> implements MetaDataProducerInterface<T>
 {
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerHelper.java

@@ -30,6 +30,7 @@ import java.util.Comparator;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import com.google.common.collect.ImmutableSet;
 import com.google.common.reflect.ClassPath;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
@@ -50,6 +51,8 @@ public class MetaDataProducerHelper
    private static Map<Class<?>, Integer> comparatorValuesByType = new HashMap<>();
    private static Integer                defaultComparatorValue;
 
+   private static ImmutableSet<ClassPath.ClassInfo> topLevelClasses;
+
    static
    {
       ////////////////////////////////////////////////////////////////////////////////////////
@@ -70,8 +73,6 @@ public class MetaDataProducerHelper
       comparatorValuesByType.put(QAppMetaData.class, 23);
    }
 
-
-
    /*******************************************************************************
     ** Recursively find all classes in the given package, that implement MetaDataProducerInterface
     ** run them, and add their output to the given qInstance.
@@ -156,7 +157,7 @@ public class MetaDataProducerHelper
          {
             try
             {
-               TopLevelMetaDataInterface metaData = producer.produce(instance);
+               MetaDataProducerOutput metaData = producer.produce(instance);
                if(metaData != null)
                {
                   metaData.addSelfToInstance(instance);
@@ -186,7 +187,7 @@ public class MetaDataProducerHelper
       List<Class<?>> classes = new ArrayList<>();
       ClassLoader    loader  = Thread.currentThread().getContextClassLoader();
 
-      for(ClassPath.ClassInfo info : ClassPath.from(loader).getTopLevelClasses())
+      for(ClassPath.ClassInfo info : getTopLevelClasses(loader))
       {
          if(info.getName().startsWith(packageName))
          {
@@ -197,4 +198,29 @@ public class MetaDataProducerHelper
       return (classes);
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static ImmutableSet<ClassPath.ClassInfo> getTopLevelClasses(ClassLoader loader) throws IOException
+   {
+      if(topLevelClasses == null)
+      {
+         topLevelClasses = ClassPath.from(loader).getTopLevelClasses();
+      }
+
+      return (topLevelClasses);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static void clearTopLevelClassCache()
+   {
+      topLevelClasses = null;
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerMultiOutput.java

@@ -0,0 +1,101 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+
+
+/*******************************************************************************
+ ** Output object for a MetaDataProducer, which contains multiple meta-data
+ ** objects.
+ *******************************************************************************/
+public class MetaDataProducerMultiOutput implements MetaDataProducerOutput
+{
+   private List<MetaDataProducerOutput> contents;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void addSelfToInstance(QInstance instance)
+   {
+      for(MetaDataProducerOutput metaDataProducerOutput : CollectionUtils.nonNullList(contents))
+      {
+         metaDataProducerOutput.addSelfToInstance(instance);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void add(MetaDataProducerOutput metaDataProducerOutput)
+   {
+      if(contents == null)
+      {
+         contents = new ArrayList<>();
+      }
+      contents.add(metaDataProducerOutput);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public MetaDataProducerMultiOutput with(MetaDataProducerOutput metaDataProducerOutput)
+   {
+      add(metaDataProducerOutput);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public <T extends MetaDataProducerOutput> List<T> getEach(Class<T> c)
+   {
+      List<T> rs = new ArrayList<>();
+
+      for(MetaDataProducerOutput content : contents)
+      {
+         if(content instanceof MetaDataProducerMultiOutput multiOutput)
+         {
+            rs.addAll(multiOutput.getEach(c));
+         }
+         else if(c.isInstance(content))
+         {
+            rs.add(c.cast(content));
+         }
+      }
+
+      return (rs);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerOutput.java

@@ -0,0 +1,40 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata;
+
+
+/*******************************************************************************
+ ** Interface to mark objects that can be produced by a MetaDataProducer.
+ **
+ ** These would usually be TopLevelMetaData objects (a table, a process, etc)
+ ** but can also be a MetaDataProducerMultiOutput, to produce multiple objects
+ ** from one producer.
+ *******************************************************************************/
+public interface MetaDataProducerOutput
+{
+
+   /*******************************************************************************
+    ** call the appropriate methods on a QInstance to add ourselves to it.
+    *******************************************************************************/
+   void addSelfToInstance(QInstance instance);
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/TopLevelMetaDataInterface.java

@@ -26,7 +26,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
  ** Interface for meta-data classes that can be added directly (e.g, at the top
  ** level) to a QInstance (such as a QTableMetaData - not a QFieldMetaData).
  *******************************************************************************/
-public interface TopLevelMetaDataInterface
+public interface TopLevelMetaDataInterface extends MetaDataProducerOutput
 {
 
    /*******************************************************************************

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/DisplayFormat.java

@@ -49,7 +49,6 @@ public interface DisplayFormat
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:Indentation")
    static String getExcelFormat(String javaDisplayFormat)
    {
       if(javaDisplayFormat == null)
@@ -58,21 +57,21 @@ public interface DisplayFormat
       }
 
       return switch(javaDisplayFormat)
-         {
-            case DisplayFormat.DEFAULT -> null;
-            case DisplayFormat.COMMAS -> "#,##0";
-            case DisplayFormat.DECIMAL1 -> "0.0";
-            case DisplayFormat.DECIMAL2 -> "0.00";
-            case DisplayFormat.DECIMAL3 -> "0.000";
-            case DisplayFormat.DECIMAL1_COMMAS -> "#,##0.0";
-            case DisplayFormat.DECIMAL2_COMMAS -> "#,##0.00";
-            case DisplayFormat.DECIMAL3_COMMAS -> "#,##0.000";
-            case DisplayFormat.CURRENCY -> "$#,##0.00";
-            case DisplayFormat.PERCENT -> "0%";
-            case DisplayFormat.PERCENT_POINT1 -> "0.0%";
-            case DisplayFormat.PERCENT_POINT2 -> "0.00%";
-            default -> null;
-         };
+      {
+         case DisplayFormat.DEFAULT -> null;
+         case DisplayFormat.COMMAS -> "#,##0";
+         case DisplayFormat.DECIMAL1 -> "0.0";
+         case DisplayFormat.DECIMAL2 -> "0.00";
+         case DisplayFormat.DECIMAL3 -> "0.000";
+         case DisplayFormat.DECIMAL1_COMMAS -> "#,##0.0";
+         case DisplayFormat.DECIMAL2_COMMAS -> "#,##0.00";
+         case DisplayFormat.DECIMAL3_COMMAS -> "#,##0.000";
+         case DisplayFormat.CURRENCY -> "$#,##0.00";
+         case DisplayFormat.PERCENT -> "0%";
+         case DisplayFormat.PERCENT_POINT1 -> "0.0%";
+         case DisplayFormat.PERCENT_POINT2 -> "0.00%";
+         default -> null;
+      };
 
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/joins/JoinType.java

@@ -44,14 +44,13 @@ public enum JoinType
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    public JoinType flip()
    {
       return switch(this)
-         {
-            case ONE_TO_MANY -> MANY_TO_ONE;
-            case MANY_TO_ONE -> ONE_TO_MANY;
-            case MANY_TO_MANY, ONE_TO_ONE -> this;
-         };
+      {
+         case ONE_TO_MANY -> MANY_TO_ONE;
+         case MANY_TO_ONE -> ONE_TO_MANY;
+         case MANY_TO_MANY, ONE_TO_ONE -> this;
+      };
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QFrontendStepMetaData.java

@@ -26,8 +26,12 @@ import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.kingsrook.qqq.backend.core.instances.QInstanceHelpContentManager;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.help.HelpRole;
+import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 
 
 /*******************************************************************************
@@ -43,6 +47,8 @@ public class QFrontendStepMetaData extends QStepMetaData
    private List<QFieldMetaData>             recordListFields;
    private Map<String, QFieldMetaData>      formFieldMap;
 
+   private List<QHelpContent> helpContents;
+
 
 
    /*******************************************************************************
@@ -340,4 +346,61 @@ public class QFrontendStepMetaData extends QStepMetaData
       return (rs);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for helpContents
+    *******************************************************************************/
+   public List<QHelpContent> getHelpContents()
+   {
+      return (this.helpContents);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for helpContents
+    *******************************************************************************/
+   public void setHelpContents(List<QHelpContent> helpContents)
+   {
+      this.helpContents = helpContents;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for helpContents
+    *******************************************************************************/
+   public QFrontendStepMetaData withHelpContents(List<QHelpContent> helpContents)
+   {
+      this.helpContents = helpContents;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for adding 1 helpContent
+    *******************************************************************************/
+   public QFrontendStepMetaData withHelpContent(QHelpContent helpContent)
+   {
+      if(this.helpContents == null)
+      {
+         this.helpContents = new ArrayList<>();
+      }
+
+      QInstanceHelpContentManager.putHelpContentInList(helpContent, this.helpContents);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** remove a single helpContent based on its set of roles
+    *******************************************************************************/
+   public void removeHelpContent(Set<HelpRole> roles)
+   {
+      QInstanceHelpContentManager.removeHelpContentByRoleSetFromList(roles, this.helpContents);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QProcessMetaData.java

@@ -60,6 +60,8 @@ public class QProcessMetaData implements QAppChildMetaData, MetaDataWithPermissi
    private List<QStepMetaData>        stepList; // these are the steps that are ran, by-default, in the order they are ran in
    private Map<String, QStepMetaData> steps; // this is the full map of possible steps
 
+   private QBackendStepMetaData cancelStep;
+
    private QIcon icon;
 
    private QScheduleMetaData schedule;
@@ -675,6 +677,7 @@ public class QProcessMetaData implements QAppChildMetaData, MetaDataWithPermissi
    }
 
 
+
    /*******************************************************************************
     ** Getter for variantRunStrategy
     *******************************************************************************/
@@ -736,4 +739,45 @@ public class QProcessMetaData implements QAppChildMetaData, MetaDataWithPermissi
    }
 
 
+
+   /*******************************************************************************
+    ** Getter for the full map of all steps (not the step list!)
+    **
+    *******************************************************************************/
+   public Map<String, QStepMetaData> getAllSteps()
+   {
+      return steps;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for cancelStep
+    *******************************************************************************/
+   public QBackendStepMetaData getCancelStep()
+   {
+      return (this.cancelStep);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for cancelStep
+    *******************************************************************************/
+   public void setCancelStep(QBackendStepMetaData cancelStep)
+   {
+      this.cancelStep = cancelStep;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for cancelStep
+    *******************************************************************************/
+   public QProcessMetaData withCancelStep(QBackendStepMetaData cancelStep)
+   {
+      this.cancelStep = cancelStep;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/serialization/QStepMetaDataDeserializer.java

@@ -39,18 +39,21 @@ import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
  *******************************************************************************/
 public class QStepMetaDataDeserializer extends JsonDeserializer<QStepMetaData>
 {
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
    @Override
-   @SuppressWarnings("checkstyle:Indentation")
    public QStepMetaData deserialize(JsonParser jsonParser, DeserializationContext deserializationContext) throws IOException, JacksonException
    {
       TreeNode treeNode = jsonParser.readValueAsTree();
       String   stepType = DeserializerUtils.readTextValue(treeNode, "stepType");
       Class<? extends QStepMetaData> targetClass = switch(stepType)
-         {
-            case "backend" -> QBackendStepMetaData.class;
-            case "frontend" -> QFrontendStepMetaData.class;
-            default -> throw new IllegalArgumentException("Unsupported StepType " + stepType + " for deserialization");
-         };
+      {
+         case "backend" -> QBackendStepMetaData.class;
+         case "frontend" -> QFrontendStepMetaData.class;
+         default -> throw new IllegalArgumentException("Unsupported StepType " + stepType + " for deserialization");
+      };
       return (DeserializerUtils.reflectivelyDeserialize(targetClass, treeNode));
    }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/QTableMetaData.java

@@ -111,6 +111,7 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
    private ShareableTableMetaData shareableTableMetaData;
 
 
+
    /*******************************************************************************
     ** Default constructor.
     *******************************************************************************/
@@ -158,11 +159,26 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
    public QTableMetaData withFieldsFromEntity(Class<? extends QRecordEntity> entityClass) throws QException
    {
       List<QRecordEntityField> recordEntityFieldList = QRecordEntity.getFieldList(entityClass);
+
+      boolean setPrimaryKey = false;
+
       for(QRecordEntityField recordEntityField : recordEntityFieldList)
       {
          QFieldMetaData field = new QFieldMetaData(recordEntityField.getGetter());
          addField(field);
+
+         if(recordEntityField.getFieldAnnotation().isPrimaryKey())
+         {
+            if(setPrimaryKey)
+            {
+               throw (new QException("Attempt to set more than one field as primary key (" + primaryKeyField + "," + field.getName() + ")."));
+            }
+
+            setPrimaryKeyField(field.getName());
+            setPrimaryKey = true;
+         }
       }
+
       return (this);
    }
 
@@ -624,6 +640,18 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
 
 
 
+   /*******************************************************************************
+    ** fluent setter for both recordLabelFormat and recordLabelFields
+    *******************************************************************************/
+   public QTableMetaData withRecordLabelFormatAndFields(String format, String... fields)
+   {
+      setRecordLabelFormat(format);
+      setRecordLabelFields(Arrays.asList(fields));
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     ** Getter for recordLabelFields
     **
@@ -1388,6 +1416,7 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
    }
 
 
+
    /*******************************************************************************
     ** Getter for shareableTableMetaData
     *******************************************************************************/
@@ -1417,5 +1446,4 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
       return (this);
    }
 
-
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/querystats/QueryStat.java

@@ -28,6 +28,7 @@ import java.util.Set;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.data.QAssociation;
 import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QIgnore;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
@@ -77,7 +78,9 @@ public class QueryStat extends QRecordEntity
    ///////////////////////////////////////////////////////////
    // non-persistent fields - used to help build the record //
    ///////////////////////////////////////////////////////////
-   private String       tableName;
+   @QIgnore
+   private String tableName;
+
    private Set<String>  joinTableNames;
    private QQueryFilter queryFilter;
 
@@ -384,6 +387,7 @@ public class QueryStat extends QRecordEntity
    /*******************************************************************************
     ** Getter for queryFilter
     *******************************************************************************/
+   @QIgnore
    public QQueryFilter getQueryFilter()
    {
       return (this.queryFilter);
@@ -446,6 +450,7 @@ public class QueryStat extends QRecordEntity
    /*******************************************************************************
     ** Getter for joinTableNames
     *******************************************************************************/
+   @QIgnore
    public Set<String> getJoinTableNames()
    {
       return (this.joinTableNames);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/savedreports/SavedReportsFilterAndColumnsSetupRenderer.java

@@ -0,0 +1,45 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.savedreports;
+
+
+import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.FilterAndColumnsSetupData;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class SavedReportsFilterAndColumnsSetupRenderer extends AbstractWidgetRenderer
+{
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public RenderWidgetOutput render(RenderWidgetInput input) throws QException
+   {
+      return (new RenderWidgetOutput(new FilterAndColumnsSetupData(null, true, false, null)));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/savedreports/SavedReportsMetaDataProvider.java

@@ -73,6 +73,7 @@ public class SavedReportsMetaDataProvider
    public static final String RENDER_REPORT_PROCESS_VALUES_WIDGET = "renderReportProcessValuesWidget";
 
 
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -234,8 +235,8 @@ public class SavedReportsMetaDataProvider
          .withName("reportSetupWidget")
          .withLabel("Filters and Columns")
          .withIsCard(true)
-         .withType(WidgetType.REPORT_SETUP.getType())
-         .withCodeReference(new QCodeReference(DefaultWidgetRenderer.class));
+         .withType(WidgetType.FILTER_AND_COLUMNS_SETUP.getType())
+         .withCodeReference(new QCodeReference(SavedReportsFilterAndColumnsSetupRenderer.class));
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/scheduledjobs/ScheduledJob.java

@@ -31,6 +31,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.common.TimeZonePossibleValueSourceMetaDataProvider;
 import com.kingsrook.qqq.backend.core.model.data.QAssociation;
 import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QIgnore;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
@@ -434,6 +435,7 @@ public class ScheduledJob extends QRecordEntity
    /*******************************************************************************
     ** Getter for jobParameters - but a map of just the key=value pairs.
     *******************************************************************************/
+   @QIgnore
    public Map<String, String> getJobParametersMap()
    {
       if(CollectionUtils.nullSafeIsEmpty(this.jobParameters))
@@ -469,6 +471,7 @@ public class ScheduledJob extends QRecordEntity
    }
 
 
+
    /*******************************************************************************
     ** Getter for repeatSeconds
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/Auth0AuthenticationModule.java

@@ -150,10 +150,7 @@ public class Auth0AuthenticationModule implements QAuthenticationModuleInterface
    ////////////////////////////////////////////////////////////////////////////////////////////////////////////
    // this is how we allow the actions within this class to work without themselves having a logged-in user. //
    ////////////////////////////////////////////////////////////////////////////////////////////////////////////
-   private static QSession chickenAndEggSession = new QSession()
-   {
-
-   };
+   private static QSession chickenAndEggSession = null;
 
 
 
@@ -163,14 +160,29 @@ public class Auth0AuthenticationModule implements QAuthenticationModuleInterface
     *******************************************************************************/
    private QSession getChickenAndEggSession()
    {
-      for(String typeName : QContext.getQInstance().getSecurityKeyTypes().keySet())
+      if(chickenAndEggSession == null)
       {
-         QSecurityKeyType keyType = QContext.getQInstance().getSecurityKeyType(typeName);
-         if(StringUtils.hasContent(keyType.getAllAccessKeyName()))
+         ////////////////////////////////////////////////////////////////////////////////
+         // if the static field is null, then let's make a new session;                //
+         // prime it with all all-access keys; and then set it in the static field.    //
+         // and, if 2 threads get in here at the same time, no real harm will be done, //
+         // other than creating the session twice, and whoever loses the race, that'll //
+         // be the one that stays in the field                                         //
+         ////////////////////////////////////////////////////////////////////////////////
+         QSession newChickenAndEggSession = new QSession();
+
+         for(String typeName : QContext.getQInstance().getSecurityKeyTypes().keySet())
          {
-            chickenAndEggSession = chickenAndEggSession.withSecurityKeyValue(keyType.getAllAccessKeyName(), true);
+            QSecurityKeyType keyType = QContext.getQInstance().getSecurityKeyType(typeName);
+            if(StringUtils.hasContent(keyType.getAllAccessKeyName()))
+            {
+               newChickenAndEggSession.withSecurityKeyValue(keyType.getAllAccessKeyName(), true);
+            }
          }
+
+         chickenAndEggSession = newChickenAndEggSession;
       }
+
       return (chickenAndEggSession);
    }
 
@@ -196,7 +208,7 @@ public class Auth0AuthenticationModule implements QAuthenticationModuleInterface
             // process a sessionUUID - looks up userSession record - cannot create token this way. //
             /////////////////////////////////////////////////////////////////////////////////////////
             String sessionUUID = context.get(SESSION_UUID_KEY);
-            LOG.debug("Creating session from sessionUUID (userSession)", logPair("sessionUUID", maskForLog(sessionUUID)));
+            LOG.trace("Creating session from sessionUUID (userSession)", logPair("sessionUUID", maskForLog(sessionUUID)));
             if(sessionUUID != null)
             {
                accessToken = getAccessTokenFromSessionUUID(metaData, sessionUUID);
@@ -254,7 +266,7 @@ public class Auth0AuthenticationModule implements QAuthenticationModuleInterface
                // decode the credentials from the header auth //
                /////////////////////////////////////////////////
                String base64Credentials = context.get(BASIC_AUTH_KEY).trim();
-               LOG.info("Creating session from basicAuthentication", logPair("base64Credentials", maskForLog(base64Credentials)));
+               LOG.trace("Creating session from basicAuthentication", logPair("base64Credentials", maskForLog(base64Credentials)));
                accessToken = getAccessTokenFromBase64BasicAuthCredentials(metaData, auth, base64Credentials);
             }
             catch(Auth0Exception e)
@@ -273,7 +285,7 @@ public class Auth0AuthenticationModule implements QAuthenticationModuleInterface
             // process an api key - looks up client application token (creating token if needed) //
             ///////////////////////////////////////////////////////////////////////////////////////
             String apiKey = context.get(API_KEY);
-            LOG.info("Creating session from apiKey (accessTokenTable)", logPair("apiKey", maskForLog(apiKey)));
+            LOG.trace("Creating session from apiKey (accessTokenTable)", logPair("apiKey", maskForLog(apiKey)));
             if(apiKey != null)
             {
                accessToken = getAccessTokenFromApiKey(metaData, apiKey);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/memory/MemoryRecordStore.java

@@ -769,7 +769,6 @@ public class MemoryRecordStore
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    private static Serializable computeAggregate(List<QRecord> records, Aggregate aggregate, QTableMetaData table)
    {
       String            fieldName = aggregate.getFieldName();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/mock/MockCountAction.java

@@ -38,7 +38,6 @@ public class MockCountAction implements CountInterface
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:MagicNumber")
    public CountOutput execute(CountInput countInput) throws QException
    {
       try

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/mock/MockQueryAction.java

@@ -95,10 +95,8 @@ public class MockQueryAction implements QueryInterface
     ** Get a mock value to use, based on its type.
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:MagicNumber")
    public static Serializable getMockValue(QTableMetaData table, String field)
    {
-      // @formatter:off // IJ can't do new-style switch correctly yet...
       return switch(table.getField(field).getType())
       {
          case STRING -> UUID.randomUUID().toString();
@@ -112,7 +110,6 @@ public class MockQueryAction implements QueryInterface
          case PASSWORD -> "abc***234";
          default -> throw new IllegalStateException("Unexpected value: " + table.getField(field).getType());
       };
-      // @formatter:on
    }
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/utils/BackendQueryFilterUtils.java

@@ -134,7 +134,6 @@ public class BackendQueryFilterUtils
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    public static boolean doesCriteriaMatch(QFilterCriteria criterion, String fieldName, Serializable value)
    {
       ListIterator<Serializable> valueListIterator = criterion.getValues().listIterator();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/delete/BulkDeleteLoadStep.java

@@ -112,12 +112,12 @@ public class BulkDeleteLoadStep extends LoadViaDeleteStep implements ProcessSumm
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       ////////////////////////////
       // have base class delete //
       ////////////////////////////
-      super.run(runBackendStepInput, runBackendStepOutput);
+      super.runOnePage(runBackendStepInput, runBackendStepOutput);
 
       QTableMetaData             table               = runBackendStepInput.getInstance().getTable(runBackendStepInput.getTableName());
       String                     primaryKeyFieldName = table.getPrimaryKeyField();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/delete/BulkDeleteTransformStep.java

@@ -78,7 +78,7 @@ public class BulkDeleteTransformStep extends AbstractTransformStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       QTableMetaData table           = runBackendStepInput.getTable();
       String         primaryKeyField = table.getPrimaryKeyField();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/edit/BulkEditLoadStep.java

@@ -114,12 +114,12 @@ public class BulkEditLoadStep extends LoadViaUpdateStep implements ProcessSummar
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       ////////////////////////////
       // have base class update //
       ////////////////////////////
-      super.run(runBackendStepInput, runBackendStepOutput);
+      super.runOnePage(runBackendStepInput, runBackendStepOutput);
 
       ////////////////////////////////////////////////////////
       // roll up results based on output from update action //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/edit/BulkEditTransformStep.java

@@ -103,7 +103,7 @@ public class BulkEditTransformStep extends AbstractTransformStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       /////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       // on the validate step, we haven't read the full file, so we don't know how many rows there are - thus        //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertTransformStep.java

@@ -118,7 +118,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       int            rowsInThisPage = runBackendStepInput.getRecords().size();
       QTableMetaData table          = runBackendStepInput.getInstance().getTable(runBackendStepInput.getTableName());

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaDeleteStep.java

@@ -61,7 +61,7 @@ public class LoadViaDeleteStep extends AbstractLoadStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       QTableMetaData table = runBackendStepInput.getTable();
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaInsertOrUpdateStep.java

@@ -73,7 +73,7 @@ public class LoadViaInsertOrUpdateStep extends AbstractLoadStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       evaluateRecords(runBackendStepInput);
       insertAndUpdateRecords(runBackendStepInput, runBackendStepOutput);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaInsertStep.java

@@ -60,7 +60,7 @@ public class LoadViaInsertStep extends AbstractLoadStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       InsertInput insertInput = new InsertInput();
       insertInput.setInputSource(getInputSource());

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaUpdateStep.java

@@ -62,7 +62,7 @@ public class LoadViaUpdateStep extends AbstractLoadStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       UpdateInput updateInput = new UpdateInput();
       updateInput.setInputSource(getInputSource());

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/NoopLoadStep.java

@@ -40,7 +40,7 @@ public class NoopLoadStep extends AbstractLoadStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       ///////////
       // noop. //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/NoopTransformStep.java

@@ -67,7 +67,7 @@ public class NoopTransformStep extends AbstractTransformStep
     *
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       ////////////////////////////////
       // return if no input records //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/StreamedETLExecuteStep.java

@@ -58,7 +58,6 @@ public class StreamedETLExecuteStep extends BaseStreamedETLStep implements Backe
     **
     *******************************************************************************/
    @Override
-   @SuppressWarnings("checkstyle:indentation")
    public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       Optional<QBackendTransaction> transaction = Optional.empty();
@@ -174,6 +173,14 @@ public class StreamedETLExecuteStep extends BaseStreamedETLStep implements Backe
          transformStep.postRun(postRunInput, postRunOutput);
          loadStep.postRun(postRunInput, postRunOutput);
 
+         //////////////////////////////////////////////////////////////////////
+         // propagate data from inner-step state to process-level step state //
+         //////////////////////////////////////////////////////////////////////
+         if(postRunOutput.getUpdatedFrontendStepList() != null)
+         {
+            runBackendStepOutput.setUpdatedFrontendStepList(postRunOutput.getUpdatedFrontendStepList());
+         }
+
          /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
          // explicitly copy values back into the runStepOutput from the post-run output                                     //
          // this might not be needed, since they (presumably) share a processState object, but just in case that changes... //
@@ -268,18 +275,36 @@ public class StreamedETLExecuteStep extends BaseStreamedETLStep implements Backe
          /////////////////////////////////////////////////////
          // pass the records through the transform function //
          /////////////////////////////////////////////////////
-         transformStep.run(streamedBackendStepInput, streamedBackendStepOutput);
+         transformStep.runOnePage(streamedBackendStepInput, streamedBackendStepOutput);
          List<AuditInput> auditInputListFromTransform = streamedBackendStepOutput.getAuditInputList();
 
+         //////////////////////////////////////////////////////////////////////
+         // propagate data from inner-step state to process-level step state //
+         //////////////////////////////////////////////////////////////////////
+         if(streamedBackendStepOutput.getUpdatedFrontendStepList() != null)
+         {
+            runBackendStepOutput.getProcessState().setStepList(streamedBackendStepOutput.getProcessState().getStepList());
+            runBackendStepOutput.setUpdatedFrontendStepList(streamedBackendStepOutput.getUpdatedFrontendStepList());
+         }
+
          ////////////////////////////////////////////////
          // pass the records through the load function //
          ////////////////////////////////////////////////
          streamedBackendStepInput = new StreamedBackendStepInput(runBackendStepInput, streamedBackendStepOutput.getRecords());
          streamedBackendStepOutput = new StreamedBackendStepOutput(runBackendStepOutput);
 
-         loadStep.run(streamedBackendStepInput, streamedBackendStepOutput);
+         loadStep.runOnePage(streamedBackendStepInput, streamedBackendStepOutput);
          List<AuditInput> auditInputListFromLoad = streamedBackendStepOutput.getAuditInputList();
 
+         //////////////////////////////////////////////////////////////////////
+         // propagate data from inner-step state to process-level step state //
+         //////////////////////////////////////////////////////////////////////
+         if(streamedBackendStepOutput.getUpdatedFrontendStepList() != null)
+         {
+            runBackendStepOutput.getProcessState().setStepList(streamedBackendStepOutput.getProcessState().getStepList());
+            runBackendStepOutput.setUpdatedFrontendStepList(streamedBackendStepOutput.getUpdatedFrontendStepList());
+         }
+
          ///////////////////////////////////////////////////////
          // copy a small number of records to the output list //
          ///////////////////////////////////////////////////////

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/StreamedETLPreviewStep.java

@@ -144,6 +144,14 @@ public class StreamedETLPreviewStep extends BaseStreamedETLStep implements Backe
       BackendStepPostRunOutput postRunOutput = new BackendStepPostRunOutput(runBackendStepOutput);
       BackendStepPostRunInput  postRunInput  = new BackendStepPostRunInput(runBackendStepInput);
       transformStep.postRun(postRunInput, postRunOutput);
+
+      //////////////////////////////////////////////////////////////////////
+      // propagate data from inner-step state to process-level step state //
+      //////////////////////////////////////////////////////////////////////
+      if(postRunOutput.getUpdatedFrontendStepList() != null)
+      {
+         runBackendStepOutput.setUpdatedFrontendStepList(postRunOutput.getUpdatedFrontendStepList());
+      }
    }
 
 
@@ -206,7 +214,16 @@ public class StreamedETLPreviewStep extends BaseStreamedETLStep implements Backe
       /////////////////////////////////////////////////////
       // pass the records through the transform function //
       /////////////////////////////////////////////////////
-      transformStep.run(streamedBackendStepInput, streamedBackendStepOutput);
+      transformStep.runOnePage(streamedBackendStepInput, streamedBackendStepOutput);
+
+      //////////////////////////////////////////////////////////////////////
+      // propagate data from inner-step state to process-level step state //
+      //////////////////////////////////////////////////////////////////////
+      if(streamedBackendStepOutput.getUpdatedFrontendStepList() != null)
+      {
+         runBackendStepOutput.getProcessState().setStepList(streamedBackendStepOutput.getProcessState().getStepList());
+         runBackendStepOutput.setUpdatedFrontendStepList(streamedBackendStepOutput.getUpdatedFrontendStepList());
+      }
 
       ////////////////////////////////////////////////////
       // add the transformed records to the output list //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/StreamedETLValidateStep.java

@@ -141,6 +141,14 @@ public class StreamedETLValidateStep extends BaseStreamedETLStep implements Back
       BackendStepPostRunOutput postRunOutput = new BackendStepPostRunOutput(runBackendStepOutput);
       BackendStepPostRunInput  postRunInput  = new BackendStepPostRunInput(runBackendStepInput);
       transformStep.postRun(postRunInput, postRunOutput);
+
+      //////////////////////////////////////////////////////////////////////
+      // propagate data from inner-step state to process-level step state //
+      //////////////////////////////////////////////////////////////////////
+      if(postRunOutput.getUpdatedFrontendStepList() != null)
+      {
+         runBackendStepOutput.setUpdatedFrontendStepList(postRunOutput.getUpdatedFrontendStepList());
+      }
    }
 
 
@@ -170,7 +178,16 @@ public class StreamedETLValidateStep extends BaseStreamedETLStep implements Back
       /////////////////////////////////////////////////////
       // pass the records through the transform function //
       /////////////////////////////////////////////////////
-      transformStep.run(streamedBackendStepInput, streamedBackendStepOutput);
+      transformStep.runOnePage(streamedBackendStepInput, streamedBackendStepOutput);
+
+      //////////////////////////////////////////////////////////////////////
+      // propagate data from inner-step state to process-level step state //
+      //////////////////////////////////////////////////////////////////////
+      if(streamedBackendStepOutput.getUpdatedFrontendStepList() != null)
+      {
+         runBackendStepOutput.getProcessState().setStepList(streamedBackendStepOutput.getProcessState().getStepList());
+         runBackendStepOutput.setUpdatedFrontendStepList(streamedBackendStepOutput.getUpdatedFrontendStepList());
+      }
 
       ///////////////////////////////////////////////////////
       // copy a small number of records to the output list //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/garbagecollector/GarbageCollectorExtractStep.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.processes.implementations.garbagecollecto
 import java.time.Instant;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
@@ -66,7 +67,7 @@ public class GarbageCollectorExtractStep extends ExtractViaQueryStep
    @Override
    protected void customizeInputPreQuery(QueryInput queryInput)
    {
-      queryInput.withQueryHint(QueryInput.QueryHint.POTENTIALLY_LARGE_NUMBER_OF_RESULTS);
+      queryInput.withQueryHint(QueryHint.POTENTIALLY_LARGE_NUMBER_OF_RESULTS);
    }
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/garbagecollector/GarbageCollectorTransformStep.java

@@ -118,7 +118,7 @@ public class GarbageCollectorTransformStep extends AbstractTransformStep
     *
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       ////////////////////////////////
       // return if no input records //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/mergeduplicates/AbstractMergeDuplicatesTransformStep.java

@@ -203,7 +203,7 @@ public abstract class AbstractMergeDuplicatesTransformStep extends AbstractTrans
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       if(CollectionUtils.nullSafeIsEmpty(runBackendStepInput.getRecords()))
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/mergeduplicates/MergeDuplicatesLoadStep.java

@@ -61,9 +61,9 @@ public class MergeDuplicatesLoadStep extends LoadViaInsertOrUpdateStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
-      super.run(runBackendStepInput, runBackendStepOutput);
+      super.runOnePage(runBackendStepInput, runBackendStepOutput);
 
       ListingHash<String, Serializable> otherTableIdsToDelete     = (ListingHash<String, Serializable>) runBackendStepInput.getValue("otherTableIdsToDelete");
       ListingHash<String, QQueryFilter> otherTableFiltersToDelete = (ListingHash<String, QQueryFilter>) runBackendStepInput.getValue("otherTableFiltersToDelete");

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/scripts/RunRecordScriptLoadStep.java

@@ -113,7 +113,7 @@ public class RunRecordScriptLoadStep extends AbstractLoadStep implements Process
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       runBackendStepInput.getAsyncJobCallback().updateStatus("Running script");
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/scripts/RunRecordScriptTransformStep.java

@@ -88,9 +88,9 @@ public class RunRecordScriptTransformStep extends NoopTransformStep
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
-      super.run(runBackendStepInput, runBackendStepOutput);
+      super.runOnePage(runBackendStepInput, runBackendStepOutput);
 
       runBackendStepOutput.addValue(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY, doGetProcessSummary(runBackendStepOutput, false));
    }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/tablesync/AbstractTableSyncTransformStep.java

@@ -190,6 +190,8 @@ public abstract class AbstractTableSyncTransformStep extends AbstractTransformSt
          this(sourceTable, sourceTableKeyField, destinationTable, destinationTableForeignKey, true, true);
       }
 
+
+
       /*******************************************************************************
        ** artificial method, here to make jacoco see that this class is indeed
        ** included in test coverage...
@@ -207,7 +209,7 @@ public abstract class AbstractTableSyncTransformStep extends AbstractTransformSt
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       if(CollectionUtils.nullSafeIsEmpty(runBackendStepInput.getRecords()))
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/locks/ProcessLock.java

@@ -0,0 +1,398 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.locks;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
+
+
+/*******************************************************************************
+ ** QRecord Entity for ProcessLock table
+ *******************************************************************************/
+public class ProcessLock extends QRecordEntity
+{
+   public static final String TABLE_NAME = "processLock";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+   @QField(isRequired = true, maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.ERROR)
+   private String key;
+
+   @QField(possibleValueSourceName = ProcessLockType.TABLE_NAME)
+   private Integer processLockTypeId;
+
+   @QField(maxLength = 100, valueTooLongBehavior = ValueTooLongBehavior.ERROR)
+   private String userId;
+
+   @QField(label = "Session UUID", maxLength = 36, valueTooLongBehavior = ValueTooLongBehavior.ERROR)
+   private String sessionUUID;
+
+   @QField(maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.ERROR)
+   private String details;
+
+   @QField()
+   private Instant checkInTimestamp;
+
+   @QField()
+   private Instant expiresAtTimestamp;
+
+
+
+   /*******************************************************************************
+    ** Default constructor
+    *******************************************************************************/
+   public ProcessLock()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that takes a QRecord
+    *******************************************************************************/
+   public ProcessLock(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public ProcessLock withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (this.createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public ProcessLock withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return (this.modifyDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modifyDate
+    *******************************************************************************/
+   public ProcessLock withModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for key
+    *******************************************************************************/
+   public String getKey()
+   {
+      return (this.key);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for key
+    *******************************************************************************/
+   public void setKey(String key)
+   {
+      this.key = key;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for key
+    *******************************************************************************/
+   public ProcessLock withKey(String key)
+   {
+      this.key = key;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for checkInTimestamp
+    *******************************************************************************/
+   public Instant getCheckInTimestamp()
+   {
+      return (this.checkInTimestamp);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for checkInTimestamp
+    *******************************************************************************/
+   public void setCheckInTimestamp(Instant checkInTimestamp)
+   {
+      this.checkInTimestamp = checkInTimestamp;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for checkInTimestamp
+    *******************************************************************************/
+   public ProcessLock withCheckInTimestamp(Instant checkInTimestamp)
+   {
+      this.checkInTimestamp = checkInTimestamp;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for expiresAtTimestamp
+    *******************************************************************************/
+   public Instant getExpiresAtTimestamp()
+   {
+      return (this.expiresAtTimestamp);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for expiresAtTimestamp
+    *******************************************************************************/
+   public void setExpiresAtTimestamp(Instant expiresAtTimestamp)
+   {
+      this.expiresAtTimestamp = expiresAtTimestamp;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for expiresAtTimestamp
+    *******************************************************************************/
+   public ProcessLock withExpiresAtTimestamp(Instant expiresAtTimestamp)
+   {
+      this.expiresAtTimestamp = expiresAtTimestamp;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for processLockTypeId
+    *******************************************************************************/
+   public Integer getProcessLockTypeId()
+   {
+      return (this.processLockTypeId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for processLockTypeId
+    *******************************************************************************/
+   public void setProcessLockTypeId(Integer processLockTypeId)
+   {
+      this.processLockTypeId = processLockTypeId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for processLockTypeId
+    *******************************************************************************/
+   public ProcessLock withProcessLockTypeId(Integer processLockTypeId)
+   {
+      this.processLockTypeId = processLockTypeId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for userId
+    *******************************************************************************/
+   public String getUserId()
+   {
+      return (this.userId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for userId
+    *******************************************************************************/
+   public void setUserId(String userId)
+   {
+      this.userId = userId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for userId
+    *******************************************************************************/
+   public ProcessLock withUserId(String userId)
+   {
+      this.userId = userId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for sessionUUID
+    *******************************************************************************/
+   public String getSessionUUID()
+   {
+      return (this.sessionUUID);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for sessionUUID
+    *******************************************************************************/
+   public void setSessionUUID(String sessionUUID)
+   {
+      this.sessionUUID = sessionUUID;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for sessionUUID
+    *******************************************************************************/
+   public ProcessLock withSessionUUID(String sessionUUID)
+   {
+      this.sessionUUID = sessionUUID;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for details
+    *******************************************************************************/
+   public String getDetails()
+   {
+      return (this.details);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for details
+    *******************************************************************************/
+   public void setDetails(String details)
+   {
+      this.details = details;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for details
+    *******************************************************************************/
+   public ProcessLock withDetails(String details)
+   {
+      this.details = details;
+      return (this);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/locks/ProcessLockMetaDataProducer.java

@@ -0,0 +1,104 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.locks;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
+
+
+/*******************************************************************************
+ ** MetaData producer for Process Locks "system"
+ *******************************************************************************/
+public class ProcessLockMetaDataProducer implements MetaDataProducerInterface<MetaDataProducerMultiOutput>
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException
+   {
+      MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput();
+
+      ////////////////////////
+      // process lock table //
+      ////////////////////////
+      output.add(new QTableMetaData()
+         .withName(ProcessLock.TABLE_NAME)
+         .withFieldsFromEntity(ProcessLock.class)
+         .withIcon(new QIcon().withName("sync_lock"))
+         .withUniqueKey(new UniqueKey("processLockTypeId", "key"))
+         .withRecordLabelFormat("%s %s")
+         .withRecordLabelFields("processLockTypeId", "key")
+         .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1, List.of("id", "processLockTypeId", "key")))
+         .withSection(new QFieldSection("data", new QIcon().withName("text_snippet"), Tier.T2, List.of("userId", "sessionUUID", "details", "checkInTimestamp", "expiresAtTimestamp")))
+         .withSection(new QFieldSection("dates", new QIcon().withName("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")))
+      );
+
+      /////////////////////////////
+      // process lock type table //
+      /////////////////////////////
+      output.add(new QTableMetaData()
+         .withName(ProcessLockType.TABLE_NAME)
+         .withFieldsFromEntity(ProcessLockType.class)
+         .withIcon(new QIcon().withName("lock"))
+         .withUniqueKey(new UniqueKey("name"))
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("label")
+         .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1, List.of("id", "name", "label")))
+         .withSection(new QFieldSection("data", new QIcon().withName("text_snippet"), Tier.T2, List.of("defaultExpirationSeconds")))
+         .withSection(new QFieldSection("dates", new QIcon().withName("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")))
+      );
+
+      ///////////////////////////
+      // process lock type PVS //
+      ///////////////////////////
+      output.add(QPossibleValueSource.newForTable(ProcessLockType.TABLE_NAME));
+
+      /////////////////////////////////////////////////////
+      // join between process lock type and process lock //
+      /////////////////////////////////////////////////////
+      output.add(new QJoinMetaData()
+         .withLeftTable(ProcessLockType.TABLE_NAME)
+         .withRightTable(ProcessLock.TABLE_NAME)
+         .withInferredName()
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn("name", "processLockTypeId"))
+      );
+
+      return output;
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/locks/ProcessLockType.java

@@ -0,0 +1,262 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.locks;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
+
+
+/*******************************************************************************
+ ** QRecord Entity for ProcessLockType table
+ *******************************************************************************/
+public class ProcessLockType extends QRecordEntity
+{
+   public static final String TABLE_NAME = "processLockType";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+   @QField(isRequired = true, maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.ERROR)
+   private String name;
+
+   @QField(isRequired = true, maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.ERROR)
+   private String label;
+
+   @QField()
+   private Integer defaultExpirationSeconds;
+
+
+
+   /*******************************************************************************
+    ** Default constructor
+    *******************************************************************************/
+   public ProcessLockType()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that takes a QRecord
+    *******************************************************************************/
+   public ProcessLockType(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public ProcessLockType withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (this.createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public ProcessLockType withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return (this.modifyDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modifyDate
+    *******************************************************************************/
+   public ProcessLockType withModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for name
+    *******************************************************************************/
+   public String getName()
+   {
+      return (this.name);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for name
+    *******************************************************************************/
+   public void setName(String name)
+   {
+      this.name = name;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for name
+    *******************************************************************************/
+   public ProcessLockType withName(String name)
+   {
+      this.name = name;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for label
+    *******************************************************************************/
+   public String getLabel()
+   {
+      return (this.label);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for label
+    *******************************************************************************/
+   public void setLabel(String label)
+   {
+      this.label = label;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for label
+    *******************************************************************************/
+   public ProcessLockType withLabel(String label)
+   {
+      this.label = label;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for defaultExpirationSeconds
+    *******************************************************************************/
+   public Integer getDefaultExpirationSeconds()
+   {
+      return (this.defaultExpirationSeconds);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for defaultExpirationSeconds
+    *******************************************************************************/
+   public void setDefaultExpirationSeconds(Integer defaultExpirationSeconds)
+   {
+      this.defaultExpirationSeconds = defaultExpirationSeconds;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for defaultExpirationSeconds
+    *******************************************************************************/
+   public ProcessLockType withDefaultExpirationSeconds(Integer defaultExpirationSeconds)
+   {
+      this.defaultExpirationSeconds = defaultExpirationSeconds;
+      return (this);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/locks/ProcessLockUtils.java

@@ -0,0 +1,486 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.locks;
+
+
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZonedDateTime;
+import java.util.Map;
+import java.util.Optional;
+import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
+import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
+import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ObjectUtils;
+import com.kingsrook.qqq.backend.core.utils.SleepUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** Utility class for working with ProcessLock table - creating, checking-in,
+ ** and releasing process locks.
+ *******************************************************************************/
+public class ProcessLockUtils
+{
+   private static final QLogger LOG = QLogger.getLogger(ProcessLockUtils.class);
+
+   private static Memoization<String, ProcessLockType> getProcessLockTypeByNameMemoization = new Memoization<String, ProcessLockType>()
+      .withTimeout(Duration.ofHours(1))
+      .withMayStoreNullValues(false);
+
+   private static Memoization<Integer, ProcessLockType> getProcessLockTypeByIdMemoization = new Memoization<Integer, ProcessLockType>()
+      .withTimeout(Duration.ofHours(1))
+      .withMayStoreNullValues(false);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static ProcessLock create(String key, String typeName, String details) throws UnableToObtainProcessLockException, QException
+   {
+      ProcessLockType lockType = getProcessLockTypeByName(typeName);
+      if(lockType == null)
+      {
+         throw (new QException("Unrecognized process lock type: " + typeName));
+      }
+
+      QSession qSession = QContext.getQSession();
+
+      Instant now = Instant.now();
+      ProcessLock processLock = new ProcessLock()
+         .withKey(key)
+         .withProcessLockTypeId(lockType.getId())
+         .withSessionUUID(ObjectUtils.tryAndRequireNonNullElse(() -> qSession.getUuid(), null))
+         .withUserId(ObjectUtils.tryAndRequireNonNullElse(() -> qSession.getUser().getIdReference(), null))
+         .withDetails(details)
+         .withCheckInTimestamp(now);
+
+      Integer defaultExpirationSeconds = lockType.getDefaultExpirationSeconds();
+      if(defaultExpirationSeconds != null)
+      {
+         processLock.setExpiresAtTimestamp(now.plusSeconds(defaultExpirationSeconds));
+      }
+
+      QRecord insertOutputRecord = tryToInsert(processLock);
+
+      ////////////////////////////////////////////////////////////
+      // if inserting failed... see if we can get existing lock //
+      ////////////////////////////////////////////////////////////
+      StringBuilder existingLockDetails = new StringBuilder();
+      ProcessLock   existingLock        = null;
+      if(CollectionUtils.nullSafeHasContents(insertOutputRecord.getErrors()))
+      {
+         QRecord existingLockRecord = new GetAction().executeForRecord(new GetInput(ProcessLock.TABLE_NAME).withUniqueKey(Map.of("key", key, "processLockTypeId", lockType.getId())));
+         if(existingLockRecord != null)
+         {
+            existingLock = new ProcessLock(existingLockRecord);
+            if(StringUtils.hasContent(existingLock.getUserId()))
+            {
+               existingLockDetails.append("Held by: ").append(existingLock.getUserId());
+            }
+
+            if(StringUtils.hasContent(existingLock.getDetails()))
+            {
+               existingLockDetails.append("; with details: ").append(existingLock.getDetails());
+            }
+
+            Instant expiresAtTimestamp = existingLock.getExpiresAtTimestamp();
+            if(expiresAtTimestamp != null)
+            {
+               ZonedDateTime zonedExpiresAt = expiresAtTimestamp.atZone(ValueUtils.getSessionOrInstanceZoneId());
+               existingLockDetails.append("; expiring at: ").append(QValueFormatter.formatDateTimeWithZone(zonedExpiresAt));
+            }
+
+            if(expiresAtTimestamp != null && expiresAtTimestamp.isBefore(now))
+            {
+               /////////////////////////////////////////////////////////////////////////////////
+               // if existing lock has expired, then we can delete it and try to insert again //
+               /////////////////////////////////////////////////////////////////////////////////
+               LOG.info("Existing lock has expired - deleting it and trying again.", logPair("id", existingLock.getId()),
+                  logPair("key", key), logPair("type", typeName), logPair("details", details), logPair("expiresAtTimestamp", expiresAtTimestamp));
+               new DeleteAction().execute(new DeleteInput(ProcessLock.TABLE_NAME).withPrimaryKey(existingLock.getId()));
+               insertOutputRecord = tryToInsert(processLock);
+            }
+         }
+         else
+         {
+            /////////////////////////////////////////////////////////
+            // if existing lock doesn't exist, try to insert again //
+            /////////////////////////////////////////////////////////
+            insertOutputRecord = tryToInsert(processLock);
+         }
+      }
+
+      if(CollectionUtils.nullSafeHasContents(insertOutputRecord.getErrors()))
+      {
+         /////////////////////////////////////////////////////////////////////////////////
+         // if at this point, we have errors on the last attempted insert, then give up //
+         /////////////////////////////////////////////////////////////////////////////////
+         LOG.info("Errors in process lock record after attempted insert", logPair("errors", insertOutputRecord.getErrors()),
+            logPair("key", key), logPair("type", typeName), logPair("details", details));
+         throw (new UnableToObtainProcessLockException("A Process Lock already exists for key [" + key + "] of type [" + typeName + "], " + existingLockDetails)
+            .withExistingLock(existingLock));
+      }
+
+      LOG.info("Created process lock", logPair("id", processLock.getId()),
+         logPair("key", key), logPair("type", typeName), logPair("details", details), logPair("expiresAtTimestamp", processLock.getExpiresAtTimestamp()));
+      return new ProcessLock(insertOutputRecord);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static QRecord tryToInsert(ProcessLock processLock) throws QException
+   {
+      return new InsertAction().execute(new InsertInput(ProcessLock.TABLE_NAME).withRecordEntity(processLock)).getRecords().get(0);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static ProcessLock create(String key, String type, String holderId, Duration sleepBetweenTries, Duration maxWait) throws UnableToObtainProcessLockException, QException
+   {
+      Instant giveUpTime = Instant.now().plus(maxWait);
+
+      UnableToObtainProcessLockException lastCaughtUnableToObtainProcessLockException = null;
+      while(true)
+      {
+         try
+         {
+            ProcessLock processLock = create(key, type, holderId);
+            return (processLock);
+         }
+         catch(UnableToObtainProcessLockException e)
+         {
+            lastCaughtUnableToObtainProcessLockException = e;
+            if(Instant.now().plus(sleepBetweenTries).isBefore(giveUpTime))
+            {
+               SleepUtils.sleep(sleepBetweenTries);
+            }
+            else
+            {
+               break;
+            }
+         }
+      }
+
+      ///////////////////////////////////////////////////////////////////////////////////////////////////
+      // this variable can never be null with current code-path, but prefer to be defensive regardless //
+      ///////////////////////////////////////////////////////////////////////////////////////////////////
+      @SuppressWarnings("ConstantValue")
+      String suffix = lastCaughtUnableToObtainProcessLockException == null ? "" : ": " + lastCaughtUnableToObtainProcessLockException.getMessage();
+
+      //noinspection ConstantValue
+      throw (new UnableToObtainProcessLockException("Unable to obtain process lock for key [" + key + "] in type [" + type + "] after [" + maxWait + "]" + suffix)
+         .withExistingLock(lastCaughtUnableToObtainProcessLockException == null ? null : lastCaughtUnableToObtainProcessLockException.getExistingLock()));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static ProcessLock getById(Integer id) throws QException
+   {
+      if(id == null)
+      {
+         return (null);
+      }
+
+      QRecord existingLockRecord = new GetAction().executeForRecord(new GetInput(ProcessLock.TABLE_NAME).withPrimaryKey(id));
+      if(existingLockRecord != null)
+      {
+         return (new ProcessLock(existingLockRecord));
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    ** input wrapper for an overload of the checkin method, to allow more flexibility
+    ** w/ whether or not you want to update details & expiresAtTimestamp (e.g., so a
+    ** null can be passed in, to mean "set it to null" vs. "don't update it").
+    *******************************************************************************/
+   public static class CheckInInput
+   {
+      private ProcessLock processLock;
+      private Instant     expiresAtTimestamp         = null;
+      private boolean     wasGivenExpiresAtTimestamp = false;
+      private String      details                    = null;
+      private boolean     wasGivenDetails            = false;
+
+
+
+      /*******************************************************************************
+       ** Constructor
+       **
+       *******************************************************************************/
+      public CheckInInput(ProcessLock processLock)
+      {
+         this.processLock = processLock;
+      }
+
+
+
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      public CheckInInput withExpiresAtTimestamp(Instant expiresAtTimestamp)
+      {
+         this.expiresAtTimestamp = expiresAtTimestamp;
+         this.wasGivenExpiresAtTimestamp = true;
+         return (this);
+      }
+
+
+
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      public CheckInInput withDetails(String details)
+      {
+         this.details = details;
+         this.wasGivenDetails = true;
+         return (this);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Do a check-in, with a specific value for the expiresAtTimestamp - which can
+    ** be set to null to make it null in the lock.
+    **
+    ** If you don't want to specify the expiresAtTimestamp, call the overload that
+    ** doesn't take the timestamp - in which case it'll either stay the same as it
+    ** was, or will be set based on the type's default.
+    *******************************************************************************/
+   public static void checkIn(CheckInInput input)
+   {
+      ProcessLock processLock = input.processLock;
+
+      try
+      {
+         if(processLock == null)
+         {
+            LOG.debug("Null processLock passed in - will not checkin.");
+            return;
+         }
+
+         QRecord recordToUpdate = new QRecord()
+            .withValue("id", processLock.getId())
+            .withValue("checkInTimestamp", Instant.now());
+
+         ///////////////////////////////////////////////////////////////////
+         // if the input was given a details string, update the details   //
+         // use boolean instead of null to know whether or not to do this //
+         ///////////////////////////////////////////////////////////////////
+         if(input.wasGivenDetails)
+         {
+            recordToUpdate.setValue("details", input.details);
+         }
+
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // if the input object had an expires-at timestamp put in it, then use that value (null or otherwise) for the expires-at-timestamp //
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         if(input.wasGivenExpiresAtTimestamp)
+         {
+            recordToUpdate.setValue("expiresAtTimestamp", input.expiresAtTimestamp);
+         }
+         else
+         {
+            ////////////////////////////////////////////////////////////////////////////////
+            // else, do the default thing - which is, look for a default in the lock type //
+            ////////////////////////////////////////////////////////////////////////////////
+            ProcessLockType lockType = getProcessLockTypeById(processLock.getProcessLockTypeId());
+            if(lockType != null)
+            {
+               Integer defaultExpirationSeconds = lockType.getDefaultExpirationSeconds();
+               if(defaultExpirationSeconds != null)
+               {
+                  recordToUpdate.setValue("expiresAtTimestamp", Instant.now().plusSeconds(defaultExpirationSeconds));
+               }
+            }
+         }
+
+         new UpdateAction().execute(new UpdateInput(ProcessLock.TABLE_NAME).withRecord(recordToUpdate));
+         LOG.debug("Checked in on process lock", logPair("id", processLock.getId()));
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error checking-in on process lock", e, logPair("processLockId", () -> processLock.getId()));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Do a check-in, with a specific value for the expiresAtTimestamp - which can
+    ** be set to null to make it null in the lock.
+    **
+    ** If you don't want to specify the expiresAtTimestamp, call the overload that
+    ** doesn't take the timestamp - in which case it'll either stay the same as it
+    ** was, or will be set based on the type's default.
+    *******************************************************************************/
+   public static void checkIn(ProcessLock processLock, Instant expiresAtTimestamp)
+   {
+      checkIn(new CheckInInput(processLock).withExpiresAtTimestamp(expiresAtTimestamp));
+   }
+
+
+
+   /*******************************************************************************
+    ** Do a check-in, updating the expires-timestamp based on the lock type's default.
+    ** (or leaving it the same as it was (null or otherwise) if there is no default
+    ** on the type).
+    *******************************************************************************/
+   public static void checkIn(ProcessLock processLock)
+   {
+      checkIn(new CheckInInput(processLock));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static void releaseById(Integer id)
+   {
+      if(id == null)
+      {
+         LOG.debug("No id passed in to releaseById - returning with noop");
+         return;
+      }
+
+      ProcessLock processLock = null;
+      try
+      {
+         processLock = ProcessLockUtils.getById(id);
+         if(processLock == null)
+         {
+            LOG.info("Process lock not found in releaseById call", logPair("id", id));
+         }
+      }
+      catch(QException e)
+      {
+         LOG.warn("Exception releasing processLock byId", e, logPair("id", id));
+      }
+
+      if(processLock != null)
+      {
+         release(processLock);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static void release(ProcessLock processLock)
+   {
+      try
+      {
+         if(processLock == null)
+         {
+            LOG.debug("No process lock passed in to release - returning with noop");
+            return;
+         }
+
+         DeleteOutput deleteOutput = new DeleteAction().execute(new DeleteInput(ProcessLock.TABLE_NAME).withPrimaryKey(processLock.getId()));
+         if(CollectionUtils.nullSafeHasContents(deleteOutput.getRecordsWithErrors()))
+         {
+            throw (new QException("Error deleting processLock record: " + deleteOutput.getRecordsWithErrors().get(0).getErrorsAsString()));
+         }
+      }
+      catch(QException e)
+      {
+         LOG.warn("Exception releasing processLock", e, logPair("processLockId", () -> processLock.getId()));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static ProcessLockType getProcessLockTypeByName(String name)
+   {
+      Optional<ProcessLockType> result = getProcessLockTypeByNameMemoization.getResult(name, n ->
+      {
+         QRecord qRecord = new GetAction().executeForRecord(new GetInput(ProcessLockType.TABLE_NAME).withUniqueKey(Map.of("name", name)));
+
+         if(qRecord != null)
+         {
+            return (new ProcessLockType(qRecord));
+         }
+
+         return (null);
+      });
+
+      return (result.orElse(null));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static ProcessLockType getProcessLockTypeById(Integer id)
+   {
+      Optional<ProcessLockType> result = getProcessLockTypeByIdMemoization.getResult(id, i ->
+      {
+         QRecord qRecord = new GetAction().executeForRecord(new GetInput(ProcessLockType.TABLE_NAME).withPrimaryKey(id));
+
+         if(qRecord != null)
+         {
+            return (new ProcessLockType(qRecord));
+         }
+
+         return (null);
+      });
+
+      return (result.orElse(null));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/locks/UnableToObtainProcessLockException.java

@@ -0,0 +1,86 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.locks;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+
+
+/*******************************************************************************
+ ** Lock thrown by ProcessLockUtils when you can't get the lock.
+ *******************************************************************************/
+public class UnableToObtainProcessLockException extends QUserFacingException
+{
+   private ProcessLock existingLock;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public UnableToObtainProcessLockException(String message)
+   {
+      super(message);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public UnableToObtainProcessLockException(String message, Throwable cause)
+   {
+      super(message, cause);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for existingLock
+    *******************************************************************************/
+   public ProcessLock getExistingLock()
+   {
+      return (this.existingLock);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for existingLock
+    *******************************************************************************/
+   public void setExistingLock(ProcessLock existingLock)
+   {
+      this.existingLock = existingLock;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for existingLock
+    *******************************************************************************/
+   public UnableToObtainProcessLockException withExistingLock(ProcessLock existingLock)
+   {
+      this.existingLock = existingLock;
+      return (this);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/utils/GeneralProcessUtils.java

@@ -343,6 +343,36 @@ public class GeneralProcessUtils
 
 
 
+   /*******************************************************************************
+    ** Load rows from a table matching the specified filter, into a map, keyed by the keyFieldName.
+    **
+    ** Note - null values from the key field are NOT put in the map.
+    **
+    ** If multiple values are found for the key, they'll squash each other, and only
+    ** one (random) value will appear.
+    *******************************************************************************/
+   public static <T extends QRecordEntity> Map<Serializable, T> loadTableToMap(String tableName, String keyFieldName, Class<T> entityClass, QQueryFilter filter) throws QException
+   {
+      QueryInput queryInput = new QueryInput();
+      queryInput.setTableName(tableName);
+      queryInput.setFilter(filter);
+      QueryOutput   queryOutput = new QueryAction().execute(queryInput);
+      List<QRecord> records     = queryOutput.getRecords();
+
+      Map<Serializable, T> map = new HashMap<>();
+      for(QRecord record : records)
+      {
+         Serializable value = record.getValue(keyFieldName);
+         if(value != null)
+         {
+            map.put(value, QRecordEntity.fromQRecord(entityClass, record));
+         }
+      }
+      return (map);
+   }
+
+
+
    /*******************************************************************************
     ** Load rows from a table matching the specified filter, into a map, keyed by the keyFieldName.
     **
@@ -412,7 +442,7 @@ public class GeneralProcessUtils
     *******************************************************************************/
    public static <T extends QRecordEntity> Map<Serializable, T> loadTableToMap(String tableName, String keyFieldName, Class<T> entityClass) throws QException
    {
-      return (loadTableToMap(tableName, keyFieldName, entityClass, null));
+      return (loadTableToMap(tableName, keyFieldName, entityClass, (Consumer<QueryInput>) null));
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/scheduler/quartz/processes/PauseQuartzJobsProcess.java

@@ -78,7 +78,7 @@ public class PauseQuartzJobsProcess extends AbstractLoadStep implements MetaData
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       try
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/scheduler/quartz/processes/ResumeQuartzJobsProcess.java

@@ -78,7 +78,7 @@ public class ResumeQuartzJobsProcess extends AbstractLoadStep implements MetaDat
     **
     *******************************************************************************/
    @Override
-   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
       try
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/QQueryFilterDeduper.java

@@ -34,6 +34,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 import static com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.EQUALS;
 import static com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.IN;
+import static com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.IS_NOT_BLANK;
 import static com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.NOT_EQUALS;
 import static com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.NOT_IN;
 
@@ -311,6 +312,28 @@ public class QQueryFilterDeduper
                         log.add("Merge two not-equals as not-in");
                         continue;
                      }
+                     else if(IN.equals(other.getOperator()) && IS_NOT_BLANK.equals(criteria.getOperator()))
+                     {
+                        //////////////////////////////////////////////////////////////////////////
+                        // for an IN and IS_NOT_BLANK, remove the IS_NOT_BLANK - it's redundant //
+                        //////////////////////////////////////////////////////////////////////////
+                        iterator.remove();
+                        didAnyGood = true;
+                        log.add("Removing redundant is-not-blank");
+                        continue;
+                     }
+                     else if(IS_NOT_BLANK.equals(other.getOperator()) && IN.equals(criteria.getOperator()))
+                     {
+                        //////////////////////////////////////////////////////////////////////////
+                        // for an IN and IS_NOT_BLANK, remove the IS_NOT_BLANK - it's redundant //
+                        //////////////////////////////////////////////////////////////////////////
+                        other.setOperator(IN);
+                        other.setValues(new ArrayList<>(criteria.getValues()));
+                        iterator.remove();
+                        didAnyGood = true;
+                        log.add("Removing redundant is-not-blank");
+                        continue;
+                     }
                      else
                      {
                         log.add("Fail because unhandled operator pair");

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/SleepUtils.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.utils;
 
 
+import java.time.Duration;
 import java.util.concurrent.TimeUnit;
 
 
@@ -55,4 +56,14 @@ public class SleepUtils
       }
    }
 
+
+
+   /*******************************************************************************
+    ** overload for sleep that takes duration object
+    *******************************************************************************/
+   public static void sleep(Duration sleepDuration)
+   {
+      sleep(sleepDuration.toMillis(), TimeUnit.MILLISECONDS);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/ValueUtils.java

@@ -793,7 +793,6 @@ public class ValueUtils
    /*******************************************************************************
     **
     *******************************************************************************/
-   @SuppressWarnings("checkstyle:indentation")
    public static Serializable getValueAsFieldType(QFieldType type, Object value)
    {
       return switch(type)

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/memoization/Memoization.java

@@ -43,8 +43,9 @@ public class Memoization<K, V>
 
    private final Map<K, MemoizedResult<V>> map = Collections.synchronizedMap(new LinkedHashMap<>());
 
-   private Duration timeout = Duration.ofSeconds(600);
-   private Integer  maxSize = 1000;
+   private Duration timeout            = Duration.ofSeconds(600);
+   private Integer  maxSize            = 1000;
+   private boolean  mayStoreNullValues = true;
 
 
 
@@ -58,6 +59,40 @@ public class Memoization<K, V>
 
 
 
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public Memoization(Integer maxSize)
+   {
+      this.maxSize = maxSize;
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public Memoization(Duration timeout)
+   {
+      this.timeout = timeout;
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public Memoization(Duration timeout, Integer maxSize)
+   {
+      this.timeout = timeout;
+      this.maxSize = maxSize;
+   }
+
+
+
    /*******************************************************************************
     ** Get the memoized Value for a given input Key - computing it if it wasn't previously
     ** memoized (or expired).
@@ -153,6 +188,14 @@ public class Memoization<K, V>
     *******************************************************************************/
    public void storeResult(K key, V value)
    {
+      //////////////////////////////////////////////////////////////////////////////////////////
+      // if the value is null, and we're not supposed to store nulls, then return w/o storing //
+      //////////////////////////////////////////////////////////////////////////////////////////
+      if(value == null && !mayStoreNullValues)
+      {
+         return;
+      }
+
       map.put(key, new MemoizedResult<>(value));
 
       //////////////////////////////////////
@@ -277,4 +320,35 @@ public class Memoization<K, V>
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for mayStoreNullValues
+    *******************************************************************************/
+   public boolean getMayStoreNullValues()
+   {
+      return (this.mayStoreNullValues);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for mayStoreNullValues
+    *******************************************************************************/
+   public void setMayStoreNullValues(boolean mayStoreNullValues)
+   {
+      this.mayStoreNullValues = mayStoreNullValues;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for mayStoreNullValues
+    *******************************************************************************/
+   public Memoization<K, V> withMayStoreNullValues(boolean mayStoreNullValues)
+   {
+      this.mayStoreNullValues = mayStoreNullValues;
+      return (this);
+   }
+
 }