Merge branch 'rel/0.25.0'

Feature/string utils safe equals ignore case

docs/Introduction.adoc

@@ -91,7 +91,7 @@ And then having a bug in the check permission logic on the _Light Bulb Inventory
 No!
 
 All of the (really important, even though application developers hate doing it) aspects of security - you don't need to write ANY code for dealing with that.
-Just tell QQQ what Authentication provider you want to use (e.g., https://auth0.com/[Auth0]), and - to paraphrase the old https://www.youtube.com/watch?v=YHzM4avGrKI[iMac ad] - there's no step 2.
+Just tell QQQ what Authentication provider you want to use (e.g., OAuth2 or https://auth0.com/[Auth0]), and - to paraphrase the old https://www.youtube.com/watch?v=YHzM4avGrKI[iMac ad] - there's no step 2.
 QQQ just does it.
 
 ''''

docs/index.adoc

@@ -31,11 +31,9 @@ include::metaData/PermissionRules.adoc[leveloffset=+1]
 
 == Services
 
+include::misc/Javalin.adoc[leveloffset=+1]
 include::misc/ScheduledJobs.adoc[leveloffset=+1]
 
-=== Web server (Javalin)
-#todo#
-
 === API server (OpenAPI)
 #todo#
 

docs/misc/Javalin.adoc

@@ -0,0 +1,109 @@
+== QQQ Middleware:  Javalin web server
+include::../variables.adoc[]
+
+QQQ provides a standard implementation of a middleware layer - that is - code that exists between the
+QQQ backend and user interface.  This implementation is a web server built using the https://javalin.io/[Javalin framework],
+packaged and deployed in the `qqq-middleware-javalin` maven module
+
+The de facto way to create a QQQ application server is to write a class which uses an instance of one of the
+subclasses of `QApplicationJavalinServer`.
+
+For example, if your application metadata is defined in a directory of yaml files, your server class could be implemented as:
+
+[source,java]
+.ConfigFileBasedQQQApplication usage example
+----
+public static void main(String[] args)
+{
+   try
+   {
+      String path = "src/main/resources/metadata";
+      ConfigFilesBasedQQQApplication application = new ConfigFilesBasedQQQApplication(path);
+      QApplicationJavalinServer javalinServer = new QApplicationJavalinServer(application);
+      javalinServer.start();
+   }
+   catch(Exception e)
+   {
+      LOG.error("Failed to start javalin server.  See stack trace for details.", e);
+   }
+}
+----
+
+A similar class exists if your metadata is produced by a package of Java MetaDataProducer objects:  `MetaDataProducerBasedQQQApplication`.
+
+=== QApplicationJavalinServer
+This class provides the bridge between your QQQ Application (e.g., your metadata) and the QQQ Middleware layer
+served by a Javalin web server.  It has several properties to control behaviors:
+
+* `Integer port` - (default `8000`) - port to use for serving HTTP.
+* `boolean serveFrontendMaterialDashboard` - (default `true`) whether to serve the javascript frontend provided
+in the maven artifact `qqq-frontend-material-dashboard`.
+* `boolean serveLegacyUnversionedMiddlewareAPI` - (default `true`) whether to serve a version the original implementation
+of the QQQ middleware, which current version of `qqq-frontend-material-dashboard` are compatible with.
+* `List<AbstractMiddlewareVersion> middlewareVersionList` - (default contains `MiddlewareVersionV1`) - list of
+newer, formally versioned implementations of the QQQ middleware interface to be served.
+* `Consumer<Javalin> javalinConfigurationCustomizer` - (default `null`) - optional hook to customize the
+javalin service object before it is started.
+* `List<QJavalinRouteProviderInterface> additionalRouteProviders` - (default `null`) - list of fully custom
+implementations of `QJavalinRouteProviderInterface`, to add additional endpoints to the javalin server.
+** _Note, you may first want to consider using JavalinRouteProviderMetaData instead - see below._
+* `QJavalinMetaData javalinMetaData` - (default `null`) - optional alternative place to define `JavalinMetaData` (vs.
+defining it in the `QInstance`). _Note that if it is set in both places, the one in the QApplicationJavalinServer
+is used._
+
+=== JavalinMetaData
+Certain behaviors of a QQQ Javalin server are configured in a declarative manner by adding a `QJavalinMetaData`
+object to the `supplementalMetaData` in your `QInstance` (or, as mentioned above, by setting it directly on the
+`QApplicationJavalinServer`):
+
+* `List<JavalinRouteProviderMetaData> routeProviders` - (default `null`) optional list of custom route providers to
+add to the Javalin server.  See below for details.
+* `String uploadedFileArchiveTableName` - (default `null`) - reference to a QQQ Table in your application instance,
+needed to support the Bulk Load process, as well as any other processes which need to accept an uploaded file
+as input.
+* `boolean loggerDisabled` - (default `false`)
+* `Function<QJavalinAccessLogger.LogEntry, Boolean> logFilter` - (default `null`)
+* `boolean queryWithoutLimitAllowed` - (default `false`)
+* `Integer queryWithoutLimitDefault` - (default `1000`)
+* `Level queryWithoutLimitLogLevel` - (default `INFO`)
+
+==== JavalinRouteProviderMetaData
+This type of metadata allows you to add additional http route providers to your Javalin instance, e.g., for
+serving static files or for running custom code from your application (in the form of QQQ Processes) to respond
+to HTTP requests.
+
+* `String hostedPath` - (required)
+* `String fileSystemPath` - (required for a static router)
+* `String processName` - required for a dynamic, process-based router.  Must be a process name within the QQQ Instance.
+See below for additional details
+* `List<String> methods` - required list of HTTP methods (verbs) that are served by the route provider
+* `QCodeReference routeAuthenticator - Optional reference to a class that implements `RouteAuthenticatorInterface`,
+to provide security authentication to all requests handled by the route provider.
+** A default implementation is provided as `SimpleRouteAuthenticator`, which requires that a user session be present
+to access paths served by the route provider.
+
+===== Process-based route provider processes
+If you define a `JavalinRouteProviderMetaData` with a `processName` (e.g., to serve dynamic HTTP responses from your javalin
+server), the process that you implement will be called to respond to any HTTP requests received by the javalin
+server which match the `hostedPath` and `methods` that are specified in the metadata.
+
+The QQQ javalin server will marshal request data from the javalin context into the process's payload, conforming to
+the shape of the `ProcessBasedRouterPayload` class.  Similarly, the http response will be built by taking values from
+the process's output/state conforming to the fields in that class.  As such, it is recommended to use a
+`ProcessBasedRouterPayload`  instance, as show in this example:
+
+[source,java]
+.Process-based router usage example (including ProcessBasedRouterPayload)
+----
+public class MyDynamicSiteProcessStep implements BackendStep
+{
+   @Override
+   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
+   {
+      ProcessBasedRouterPayload payload = input.getProcessPayload(ProcessBasedRouterPayload.class);
+      String path = payload.getPath();
+      payload.setResponseString("You requested: " + path);
+      output.setProcessPayload(payload);
+   }
+}
+----

pom.xml

@@ -48,7 +48,7 @@
    </modules>
 
    <properties>
-      <revision>0.25.0-SNAPSHOT</revision>
+      <revision>0.25.0</revision>
 
       <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

qqq-backend-core/pom.xml

@@ -65,7 +65,11 @@
          <artifactId>aws-java-sdk-secretsmanager</artifactId>
          <version>1.12.385</version>
       </dependency>
-
+      <dependency>
+         <groupId>com.ibm.icu</groupId>
+         <artifactId>icu4j</artifactId>
+         <version>77.1</version>
+      </dependency>
       <dependency>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>jackson-databind</artifactId>
@@ -125,10 +129,16 @@
          <version>2.16.0</version>
       </dependency>
 
+      <dependency>
+         <groupId>com.nimbusds</groupId>
+         <artifactId>oauth2-oidc-sdk</artifactId>
+         <version>11.23.1</version>
+      </dependency>
+
       <dependency>
          <groupId>com.auth0</groupId>
          <artifactId>auth0</artifactId>
-         <version>2.1.0</version>
+         <version>2.18.0</version>
       </dependency>
       <dependency>
          <groupId>com.auth0</groupId>
@@ -138,12 +148,12 @@
       <dependency>
          <groupId>com.auth0</groupId>
          <artifactId>jwks-rsa</artifactId>
-         <version>0.22.0</version>
+         <version>0.22.1</version>
       </dependency>
       <dependency>
          <groupId>io.github.cdimascio</groupId>
-         <artifactId>java-dotenv</artifactId>
-         <version>5.2.2</version>
+         <artifactId>dotenv-java</artifactId>
+         <version>3.2.0</version>
       </dependency>
       <dependency>
          <groupId>org.apache.velocity</groupId>
@@ -151,16 +161,21 @@
          <version>2.3</version>
       </dependency>
 
-      <!-- the next 2 deps are for html to pdf - per https://www.baeldung.com/java-html-to-pdf -->
+      <!-- the next 3 deps are for html to pdf -->
       <dependency>
          <groupId>org.jsoup</groupId>
          <artifactId>jsoup</artifactId>
          <version>1.15.3</version>
       </dependency>
       <dependency>
-         <groupId>org.xhtmlrenderer</groupId>
-         <artifactId>flying-saucer-pdf-openpdf</artifactId>
-         <version>9.1.22</version>
+         <groupId>com.openhtmltopdf</groupId>
+         <artifactId>openhtmltopdf-core</artifactId>
+         <version>1.0.10</version>
+      </dependency>
+      <dependency>
+         <groupId>com.openhtmltopdf</groupId>
+         <artifactId>openhtmltopdf-pdfbox</artifactId>
+         <version>1.0.10</version>
       </dependency>
 
       <!-- the next 3 deps are being added for google drive support -->

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/templates/ConvertHtmlToPdfAction.java

@@ -26,22 +26,32 @@ import java.nio.file.Path;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.actions.AbstractQActionFunction;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.templates.ConvertHtmlToPdfInput;
 import com.kingsrook.qqq.backend.core.model.actions.templates.ConvertHtmlToPdfOutput;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.openhtmltopdf.css.constants.IdentValue;
+import com.openhtmltopdf.pdfboxout.PdfBoxFontResolver;
+import com.openhtmltopdf.pdfboxout.PdfBoxRenderer;
+import com.openhtmltopdf.pdfboxout.PdfRendererBuilder;
 import org.jsoup.Jsoup;
+import org.jsoup.helper.W3CDom;
 import org.jsoup.nodes.Document;
-import org.xhtmlrenderer.layout.SharedContext;
-import org.xhtmlrenderer.pdf.ITextRenderer;
 
 
 /*******************************************************************************
  ** Action to convert a string of HTML to a PDF!
  **
  ** Much credit to https://www.baeldung.com/java-html-to-pdf
- *******************************************************************************/
+ **
+ ** Updated in March 2025 to go from flying-saucer-pdf-openpdf lib to openhtmltopdf,
+ ** mostly to get support for max-height on images...
+ ********************************************************************************/
 public class ConvertHtmlToPdfAction extends AbstractQActionFunction<ConvertHtmlToPdfInput, ConvertHtmlToPdfOutput>
 {
+   private static final QLogger LOG = QLogger.getLogger(ConvertHtmlToPdfAction.class);
+
+
 
    /*******************************************************************************
     **
@@ -58,35 +68,37 @@ public class ConvertHtmlToPdfAction extends AbstractQActionFunction<ConvertHtmlT
          //////////////////////////////////////////////////////////////////
          Document document = Jsoup.parse(input.getHtml());
          document.outputSettings().syntax(Document.OutputSettings.Syntax.xml);
+         org.w3c.dom.Document w3cDoc = new W3CDom().fromJsoup(document);
 
          //////////////////////////////
          // convert the XHTML to PDF //
          //////////////////////////////
-         ITextRenderer renderer      = new ITextRenderer();
-         SharedContext sharedContext = renderer.getSharedContext();
-         sharedContext.setPrint(true);
-         sharedContext.setInteractive(false);
+         PdfRendererBuilder builder = new PdfRendererBuilder();
+         builder.toStream(input.getOutputStream());
+         builder.useFastMode();
+         builder.withW3cDocument(w3cDoc, input.getBasePath() == null ? "./" : input.getBasePath().toUri().toString());
 
-         if(input.getBasePath() != null)
+         try(PdfBoxRenderer pdfBoxRenderer = builder.buildPdfRenderer())
          {
-            String baseUrl = input.getBasePath().toUri().toURL().toString();
-            renderer.setDocumentFromString(document.html(), baseUrl);
-         }
-         else
-         {
-            renderer.setDocumentFromString(document.html());
-         }
+            pdfBoxRenderer.layout();
+            pdfBoxRenderer.getSharedContext().setPrint(true);
+            pdfBoxRenderer.getSharedContext().setInteractive(false);
 
-         //////////////////////////////////////////////////
-         // register any custom fonts the input supplied //
-         //////////////////////////////////////////////////
-         for(Map.Entry<String, Path> entry : CollectionUtils.nonNullMap(input.getCustomFonts()).entrySet())
-         {
-            renderer.getFontResolver().addFont(entry.getValue().toAbsolutePath().toString(), entry.getKey(), "UTF-8", true, null);
-         }
+            for(Map.Entry<String, Path> entry : CollectionUtils.nonNullMap(input.getCustomFonts()).entrySet())
+            {
+               LOG.warn("Note:  Custom fonts appear to not be working in this class at this time...");
+               pdfBoxRenderer.getFontResolver().addFont(
+                  entry.getValue().toAbsolutePath().toFile(), // Path to the TrueType font file
+                  entry.getKey(),                             // Font family name to use in CSS
+                  400,                                        // Font weight (e.g., 400 for normal, 700 for bold)
+                  IdentValue.NORMAL,                          // Font style (e.g., NORMAL, ITALIC)
+                  true,                                       // Whether to subset the font
+                  PdfBoxFontResolver.FontGroup.MAIN           // ??
+               );
+            }
 
-         renderer.layout();
-         renderer.createPDF(input.getOutputStream());
+            pdfBoxRenderer.createPDF();
+         }
 
          return (output);
       }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/BasicCustomPossibleValueProvider.java

@@ -0,0 +1,91 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.values;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+
+
+/*******************************************************************************
+ ** Basic implementation of a possible value provider, for where there's a limited
+ ** set of possible source objects - so you just have to define how to make one
+ ** PV from a source object, how to list all of the source objects, and how to
+ ** look up a PV from an id.
+ *******************************************************************************/
+public abstract class BasicCustomPossibleValueProvider<S, ID extends Serializable> implements QCustomPossibleValueProvider<ID>
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   protected abstract QPossibleValue<ID> makePossibleValue(S sourceObject);
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   protected abstract S getSourceObject(Serializable id);
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   protected abstract List<S> getAllSourceObjects();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QPossibleValue<ID> getPossibleValue(Serializable idValue)
+   {
+      S sourceObject = getSourceObject(idValue);
+      if(sourceObject == null)
+      {
+         return (null);
+      }
+
+      return makePossibleValue(sourceObject);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public List<QPossibleValue<ID>> search(SearchPossibleValueSourceInput input) throws QException
+   {
+      List<QPossibleValue<ID>> allPossibleValues = new ArrayList<>();
+      List<S>                  allSourceObjects  = getAllSourceObjects();
+      for(S sourceObject : allSourceObjects)
+      {
+         allPossibleValues.add(makePossibleValue(sourceObject));
+      }
+
+      return completeCustomPVSSearch(input, allPossibleValues);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/AbstractMetaDataProducerBasedQQQApplication.java

@@ -29,7 +29,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 
 /*******************************************************************************
  ** Version of AbstractQQQApplication that assumes all meta-data is produced
- ** by MetaDataProducers in a single package.
+ ** by MetaDataProducers in (or under) a single package.
  *******************************************************************************/
 public abstract class AbstractMetaDataProducerBasedQQQApplication extends AbstractQQQApplication
 {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/ConfigFilesBasedQQQApplication.java

@@ -0,0 +1,61 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.instances.loaders.MetaDataLoaderHelper;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+
+
+/*******************************************************************************
+ ** Version of AbstractQQQApplication that assumes all meta-data is defined in
+ ** config files (yaml, json, etc) under a given directory path.
+ *******************************************************************************/
+public class ConfigFilesBasedQQQApplication extends AbstractQQQApplication
+{
+   private final String path;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public ConfigFilesBasedQQQApplication(String path)
+   {
+      this.path = path;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QInstance defineQInstance() throws QException
+   {
+      QInstance qInstance = new QInstance();
+      MetaDataLoaderHelper.processAllMetaDataFilesInDirectory(qInstance, path);
+      return (qInstance);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/MetaDataProducerBasedQQQApplication.java

@@ -0,0 +1,67 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances;
+
+
+/*******************************************************************************
+ ** Version of AbstractQQQApplication that assumes all meta-data is produced
+ ** by MetaDataProducers in (or under) a single package (where you can pass that
+ ** package into the constructor, vs. the abstract base class, where you extend
+ ** it and override the getMetaDataPackageName method.
+ *******************************************************************************/
+public class MetaDataProducerBasedQQQApplication extends AbstractMetaDataProducerBasedQQQApplication
+{
+   private final String metaDataPackageName;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public MetaDataProducerBasedQQQApplication(String metaDataPackageName)
+   {
+      this.metaDataPackageName = metaDataPackageName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public MetaDataProducerBasedQQQApplication(Class<?> aClassInMetaDataPackage)
+   {
+      this(aClassInMetaDataPackage.getPackageName());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getMetaDataPackageName()
+   {
+      return (this.metaDataPackageName);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceEnricher.java

@@ -330,7 +330,21 @@ public class QInstanceEnricher
 
       if(table.getFields() != null)
       {
-         table.getFields().values().forEach(this::enrichField);
+         for(Map.Entry<String, QFieldMetaData> entry : table.getFields().entrySet())
+         {
+            String         name  = entry.getKey();
+            QFieldMetaData field = entry.getValue();
+
+            ////////////////////////////////////////////////////////////////////////////
+            // in case the field wasn't given a name, use its key from the fields map //
+            ////////////////////////////////////////////////////////////////////////////
+            if(!StringUtils.hasContent(field.getName()))
+            {
+               field.setName(name);
+            }
+
+            enrichField(field);
+         }
 
          for(QSupplementalTableMetaData supplementalTableMetaData : CollectionUtils.nonNullMap(table.getSupplementalMetaData()).values())
          {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidator.java

@@ -64,6 +64,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaDa
 import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.automation.QAutomationProviderMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceLambda;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeType;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.ParentWidgetMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
@@ -651,6 +652,8 @@ public class QInstanceValidator
             validateSimpleCodeReference("Instance Authentication meta data customizer ", authentication.getCustomizer(), QAuthenticationModuleCustomizerInterface.class);
          }
 
+         authentication.validate(qInstance, this);
+
          runPlugins(QAuthenticationMetaData.class, authentication, qInstance);
       }
    }
@@ -1406,7 +1409,7 @@ public class QInstanceValidator
             //////////////////////////////////////////////////
             // make sure the customizer can be instantiated //
             //////////////////////////////////////////////////
-            Object customizerInstance = getInstanceOfCodeReference(prefix, customizerClass);
+            Object customizerInstance = getInstanceOfCodeReference(prefix, customizerClass, codeReference);
 
             TableCustomizers tableCustomizer = TableCustomizers.forRole(roleName);
             if(tableCustomizer == null)
@@ -1467,8 +1470,13 @@ public class QInstanceValidator
    /*******************************************************************************
     **
     *******************************************************************************/
-   private Object getInstanceOfCodeReference(String prefix, Class<?> clazz)
+   private Object getInstanceOfCodeReference(String prefix, Class<?> clazz, QCodeReference codeReference)
    {
+      if(codeReference instanceof QCodeReferenceLambda<?> lambdaCodeReference)
+      {
+         return (lambdaCodeReference.getLambda());
+      }
+
       Object instance = null;
       try
       {
@@ -1647,21 +1655,26 @@ public class QInstanceValidator
             Set<String> usedStepNames = new HashSet<>();
             if(assertCondition(CollectionUtils.nullSafeHasContents(process.getStepList()), "At least 1 step must be defined in process " + processName + "."))
             {
-               int index = 0;
+               int index = -1;
                for(QStepMetaData step : process.getStepList())
                {
+                  index++;
                   if(assertCondition(StringUtils.hasContent(step.getName()), "Missing name for a step at index " + index + " in process " + processName))
                   {
                      assertCondition(!usedStepNames.contains(step.getName()), "Duplicate step name [" + step.getName() + "] in process " + processName);
                      usedStepNames.add(step.getName());
                   }
-                  index++;
 
                   ////////////////////////////////////////////
                   // validate instantiation of step classes //
                   ////////////////////////////////////////////
                   if(step instanceof QBackendStepMetaData backendStepMetaData)
                   {
+                     if(assertCondition(backendStepMetaData.getCode() != null, "Missing code for a backend step at index " + index + " in process " + processName))
+                     {
+                        validateSimpleCodeReference("Process " + processName + ", backend step at index " + index + ", code reference: ", backendStepMetaData.getCode(), BackendStep.class);
+                     }
+
                      if(backendStepMetaData.getInputMetaData() != null && CollectionUtils.nullSafeHasContents(backendStepMetaData.getInputMetaData().getFieldList()))
                      {
                         for(QFieldMetaData fieldMetaData : backendStepMetaData.getInputMetaData().getFieldList())
@@ -2247,7 +2260,7 @@ public class QInstanceValidator
             //////////////////////////////////////////////////
             // make sure the customizer can be instantiated //
             //////////////////////////////////////////////////
-            Object classInstance = getInstanceOfCodeReference(prefix, clazz);
+            Object classInstance = getInstanceOfCodeReference(prefix, clazz, codeReference);
 
             ////////////////////////////////////////////////////////////////////////
             // make sure the customizer instance can be cast to the expected type //
@@ -2270,6 +2283,11 @@ public class QInstanceValidator
       Class<?> clazz = null;
       try
       {
+         if(codeReference instanceof QCodeReferenceLambda<?> lambdaCodeReference)
+         {
+            return (lambdaCodeReference.getLambda().getClass());
+         }
+
          clazz = Class.forName(codeReference.getName());
       }
       catch(ClassNotFoundException e)

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/SecretsManagerUtils.java

@@ -72,6 +72,18 @@ public class SecretsManagerUtils
     ** and write them to a .env file (backing up any pre-existing .env files first).
     *******************************************************************************/
    public static void writeEnvFromSecretsWithNamePrefix(String prefix) throws IOException
+   {
+      writeEnvFromSecretsWithNamePrefix(prefix, true);
+   }
+
+
+
+   /*******************************************************************************
+    ** IF secret manager ENV vars are set,
+    ** THEN lookup all secrets starting with the given prefix,
+    ** and write them to a .env file (backing up any pre-existing .env files first).
+    *******************************************************************************/
+   public static void writeEnvFromSecretsWithNamePrefix(String prefix, boolean quoteValues) throws IOException
    {
       Optional<AWSSecretsManager> optionalSecretsManagerClient = getSecretsManagerClient();
       if(optionalSecretsManagerClient.isPresent())
@@ -91,7 +103,9 @@ public class SecretsManagerUtils
                Optional<String> secretValue       = getSecret(prefix, nameWithoutPrefix);
                if(secretValue.isPresent())
                {
-                  String envLine = nameWithoutPrefix + "=" + secretValue.get();
+                  String envLine = quoteValues
+                     ? nameWithoutPrefix + "=\"" + secretValue.get() + "\""
+                     : nameWithoutPrefix + "=" + secretValue.get();
                   fullEnv.append(envLine).append('\n');
                }
             }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/AbstractMetaDataLoader.java

@@ -0,0 +1,510 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Serializable;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.YamlUtils;
+import org.apache.commons.io.IOUtils;
+import static com.kingsrook.qqq.backend.core.utils.ValueUtils.getValueAsInteger;
+import static com.kingsrook.qqq.backend.core.utils.ValueUtils.getValueAsString;
+
+
+/*******************************************************************************
+ ** Abstract base class in hierarchy of classes that know how to construct &
+ ** populate QMetaDataObject instances, based on input streams (e.g., from files).
+ *******************************************************************************/
+public abstract class AbstractMetaDataLoader<T extends QMetaDataObject>
+{
+   private static final QLogger LOG = QLogger.getLogger(AbstractMetaDataLoader.class);
+
+   private String fileName;
+
+   private List<LoadingProblem> problems = new ArrayList<>();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public T fileToMetaDataObject(QInstance qInstance, InputStream inputStream, String fileName) throws QMetaDataLoaderException
+   {
+      this.fileName = fileName;
+      Map<String, Object> map = fileToMap(inputStream, fileName);
+      LoadingContext loadingContext = new LoadingContext(fileName, "/");
+      return (mapToMetaDataObject(qInstance, map, loadingContext));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public abstract T mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   protected Map<String, Object> fileToMap(InputStream inputStream, String fileName) throws QMetaDataLoaderException
+   {
+      try
+      {
+         String string = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
+         string = StringUtils.ltrim(string);
+         if(fileName.toLowerCase().endsWith(".json"))
+         {
+            return JsonUtils.toObject(string, new TypeReference<>() {});
+         }
+         else if(fileName.toLowerCase().endsWith(".yaml") || fileName.toLowerCase().endsWith(".yml"))
+         {
+            return YamlUtils.toMap(string);
+         }
+
+         throw (new QMetaDataLoaderException("Unsupported file format (based on file name: " + fileName + ")"));
+      }
+      catch(IOException e)
+      {
+         throw new QMetaDataLoaderException("Error building map from file: " + fileName, e);
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   protected void reflectivelyMap(QInstance qInstance, QMetaDataObject targetObject, Map<String, Object> map, LoadingContext context)
+   {
+      Class<? extends QMetaDataObject> targetClass    = targetObject.getClass();
+      Set<String>                      usedFieldNames = new HashSet<>();
+
+      for(Method method : targetClass.getMethods())
+      {
+         try
+         {
+            if(method.getName().startsWith("set") && method.getParameterTypes().length == 1)
+            {
+               String propertyName = StringUtils.lcFirst(method.getName().substring(3));
+
+               if(map.containsKey(propertyName))
+               {
+                  usedFieldNames.add(propertyName);
+                  Class<?> parameterType = method.getParameterTypes()[0];
+                  Object   rawValue      = map.get(propertyName);
+
+                  try
+                  {
+                     Object mappedValue = reflectivelyMapValue(qInstance, method, parameterType, rawValue, context.descendToProperty(propertyName));
+                     method.invoke(targetObject, mappedValue);
+                  }
+                  catch(NoValueException nve)
+                  {
+                     ///////////////////////
+                     // don't call setter //
+                     ///////////////////////
+                     LOG.debug("at " + context + ": No value was mapped for property [" + propertyName + "] on " + targetClass.getSimpleName() + "." + method.getName() + ", raw value: [" + rawValue + "]");
+                  }
+               }
+            }
+         }
+         catch(Exception e)
+         {
+            addProblem(new LoadingProblem(context, "Error reflectively mapping on " + targetClass.getName() + "." + method.getName(), e));
+         }
+      }
+
+      //////////////////////////
+      // mmm, slightly sus... //
+      //////////////////////////
+      map.remove("class");
+      map.remove("version");
+
+      Set<String> unrecognizedKeys = new HashSet<>(map.keySet());
+      unrecognizedKeys.removeAll(usedFieldNames);
+
+      if(!unrecognizedKeys.isEmpty())
+      {
+         addProblem(new LoadingProblem(context, unrecognizedKeys.size() + " Unrecognized " + StringUtils.plural(unrecognizedKeys, "property", "properties") + ": " + unrecognizedKeys));
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public Object reflectivelyMapValue(QInstance qInstance, Method method, Class<?> parameterType, Object rawValue, LoadingContext context) throws Exception
+   {
+      if(rawValue instanceof String s && s.matches("^\\$\\{.+\\..+}"))
+      {
+         rawValue = new QMetaDataVariableInterpreter().interpret(s);
+         LOG.debug("Interpreted raw value [" + s + "] as [" + StringUtils.maskAndTruncate(ValueUtils.getValueAsString(rawValue) + "]"));
+      }
+
+      if(parameterType.equals(String.class))
+      {
+         return (getValueAsString(rawValue));
+      }
+      else if(parameterType.equals(Integer.class))
+      {
+         try
+         {
+            return (getValueAsInteger(rawValue));
+         }
+         catch(Exception e)
+         {
+            addProblem(new LoadingProblem(context, "[" + rawValue + "] is not an Integer value."));
+         }
+      }
+      else if(parameterType.equals(Boolean.class))
+      {
+         if("true".equals(rawValue) || Boolean.TRUE.equals(rawValue))
+         {
+            return (true);
+         }
+         else if("false".equals(rawValue) || Boolean.FALSE.equals(rawValue))
+         {
+            return (false);
+         }
+         else if(rawValue == null)
+         {
+            return (null);
+         }
+         else
+         {
+            addProblem(new LoadingProblem(context, "[" + rawValue + "] is not a boolean value (must be 'true' or 'false')."));
+            return (null);
+         }
+      }
+      else if(parameterType.equals(boolean.class))
+      {
+         if("true".equals(rawValue) || Boolean.TRUE.equals(rawValue))
+         {
+            return (true);
+         }
+         else if("false".equals(rawValue) || Boolean.FALSE.equals(rawValue))
+         {
+            return (false);
+         }
+         else
+         {
+            addProblem(new LoadingProblem(context, rawValue + " is not a boolean value (must be 'true' or 'false')."));
+            throw (new NoValueException());
+         }
+      }
+      else if(parameterType.equals(List.class))
+      {
+         Type     actualTypeArgument = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[0];
+         Class<?> actualTypeClass    = Class.forName(actualTypeArgument.getTypeName());
+
+         if(rawValue instanceof @SuppressWarnings("rawtypes")List valueList)
+         {
+            List<Object> mappedValueList = new ArrayList<>();
+            for(Object o : valueList)
+            {
+               try
+               {
+                  Object mappedValue = reflectivelyMapValue(qInstance, null, actualTypeClass, o, context);
+                  mappedValueList.add(mappedValue);
+               }
+               catch(NoValueException nve)
+               {
+                  // leave off list
+               }
+            }
+            return (mappedValueList);
+         }
+      }
+      else if(parameterType.equals(Set.class))
+      {
+         Type     actualTypeArgument = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[0];
+         Class<?> actualTypeClass    = Class.forName(actualTypeArgument.getTypeName());
+
+         if(rawValue instanceof @SuppressWarnings("rawtypes")List valueList)
+         {
+            Set<Object> mappedValueSet = new LinkedHashSet<>();
+            for(Object o : valueList)
+            {
+               try
+               {
+                  Object mappedValue = reflectivelyMapValue(qInstance, null, actualTypeClass, o, context);
+                  mappedValueSet.add(mappedValue);
+               }
+               catch(NoValueException nve)
+               {
+                  // leave off list
+               }
+            }
+            return (mappedValueSet);
+         }
+      }
+      else if(parameterType.equals(Map.class))
+      {
+         Type keyType = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[0];
+         if(!keyType.equals(String.class))
+         {
+            addProblem(new LoadingProblem(context, "Unsupported key type for " + method + " got [" + keyType + "], expected [String]"));
+            throw new NoValueException();
+         }
+         // todo make sure string
+
+         Type     actualTypeArgument = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[1];
+         Class<?> actualTypeClass    = Class.forName(actualTypeArgument.getTypeName());
+
+         if(rawValue instanceof @SuppressWarnings("rawtypes")Map valueMap)
+         {
+            Map<String, Object> mappedValueMap = new LinkedHashMap<>();
+            for(Object o : valueMap.entrySet())
+            {
+               try
+               {
+                  @SuppressWarnings("unchecked")
+                  Map.Entry<String, Object> entry = (Map.Entry<String, Object>) o;
+                  Object mappedValue = reflectivelyMapValue(qInstance, null, actualTypeClass, entry.getValue(), context);
+                  mappedValueMap.put(entry.getKey(), mappedValue);
+               }
+               catch(NoValueException nve)
+               {
+                  // leave out of map
+               }
+            }
+            return (mappedValueMap);
+         }
+      }
+      else if(parameterType.isEnum())
+      {
+         String value = getValueAsString(rawValue);
+         for(Object enumConstant : parameterType.getEnumConstants())
+         {
+            if(((Enum<?>) enumConstant).name().equals(value))
+            {
+               return (enumConstant);
+            }
+         }
+
+         addProblem(new LoadingProblem(context, "Unrecognized value [" + rawValue + "].  Expected one of: " + Arrays.toString(parameterType.getEnumConstants())));
+      }
+      else if(MetaDataLoaderRegistry.hasLoaderForClass(parameterType))
+      {
+         if(rawValue instanceof @SuppressWarnings("rawtypes")Map valueMap)
+         {
+            Class<? extends AbstractMetaDataLoader<?>> loaderClass = MetaDataLoaderRegistry.getLoaderForClass(parameterType);
+            AbstractMetaDataLoader<?>                  loader      = loaderClass.getConstructor().newInstance();
+            //noinspection unchecked
+            return (loader.mapToMetaDataObject(qInstance, valueMap, context));
+         }
+      }
+      else if(QMetaDataObject.class.isAssignableFrom(parameterType))
+      {
+         if(rawValue instanceof @SuppressWarnings("rawtypes")Map valueMap)
+         {
+            QMetaDataObject childObject = (QMetaDataObject) parameterType.getConstructor().newInstance();
+            //noinspection unchecked
+            reflectivelyMap(qInstance, childObject, valueMap, context);
+            return (childObject);
+         }
+      }
+      else if(parameterType.equals(Serializable.class))
+      {
+         if(rawValue instanceof String
+            || rawValue instanceof Integer
+            || rawValue instanceof BigDecimal
+            || rawValue instanceof Boolean
+         )
+         {
+            return rawValue;
+         }
+      }
+      else
+      {
+         // todo clean up this message/level
+         addProblem(new LoadingProblem(context, "No case for " + parameterType + " (arg to: " + method + ")"));
+      }
+
+      throw new NoValueException();
+   }
+
+   ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   // unclear if the below is needed.  if so, useful to not re-write, but is hurting test coverage, so zombie until used //
+   ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   ///***************************************************************************
+   // *
+   // ***************************************************************************/
+   //protected ListOfMapOrMapOfMap getListOfMapOrMapOfMap(Map<String, Object> map, String key)
+   //{
+   //   if(map.containsKey(key))
+   //   {
+   //      if(map.get(key) instanceof List)
+   //      {
+   //         return (new ListOfMapOrMapOfMap((List<Map<String, Object>>) map.get(key)));
+   //      }
+   //      else if(map.get(key) instanceof Map)
+   //      {
+   //         return (new ListOfMapOrMapOfMap((Map<String, Map<String, Object>>) map.get(key)));
+   //      }
+   //      else
+   //      {
+   //         LOG.warn("Expected list or map under key [" + key + "] while processing [" + getClass().getSimpleName() + "] from [" + fileName + "], but found: " + (map.get(key) == null ? "null" : map.get(key).getClass().getSimpleName()));
+   //      }
+   //   }
+
+   //   return (null);
+   //}
+
+   ///***************************************************************************
+   // *
+   // ***************************************************************************/
+   //protected List<Map<String, Object>> getListOfMap(Map<String, Object> map, String key)
+   //{
+   //   if(map.containsKey(key))
+   //   {
+   //      if(map.get(key) instanceof List)
+   //      {
+   //         return (List<Map<String, Object>>) map.get(key);
+   //      }
+   //      else
+   //      {
+   //         LOG.warn("Expected list under key [" + key + "] while processing [" + getClass().getSimpleName() + "] from [" + fileName + "], but found: " + (map.get(key) == null ? "null" : map.get(key).getClass().getSimpleName()));
+   //      }
+   //   }
+
+   //   return (null);
+   //}
+
+   ///***************************************************************************
+   // *
+   // ***************************************************************************/
+   //protected Map<String, Map<String, Object>> getMapOfMap(Map<String, Object> map, String key)
+   //{
+   //   if(map.containsKey(key))
+   //   {
+   //      if(map.get(key) instanceof Map)
+   //      {
+   //         return (Map<String, Map<String, Object>>) map.get(key);
+   //      }
+   //      else
+   //      {
+   //         LOG.warn("Expected map under key [" + key + "] while processing [" + getClass().getSimpleName() + "] from [" + fileName + "], but found: " + (map.get(key) == null ? "null" : map.get(key).getClass().getSimpleName()));
+   //      }
+   //   }
+
+   //   return (null);
+   //}
+
+   ///***************************************************************************
+   // **
+   // ***************************************************************************/
+   //protected record ListOfMapOrMapOfMap(List<Map<String, Object>> listOf, Map<String, Map<String, Object>> mapOf)
+   //{
+   //   /*******************************************************************************
+   //    ** Constructor
+   //    **
+   //    *******************************************************************************/
+   //   public ListOfMapOrMapOfMap(List<Map<String, Object>> listOf)
+   //   {
+   //      this(listOf, null);
+   //   }
+
+   //   /*******************************************************************************
+   //    ** Constructor
+   //    **
+   //    *******************************************************************************/
+   //   public ListOfMapOrMapOfMap(Map<String, Map<String, Object>> mapOf)
+   //   {
+   //      this(null, mapOf);
+   //   }
+   //}
+
+
+
+   /*******************************************************************************
+    ** Getter for fileName
+    **
+    *******************************************************************************/
+   public String getFileName()
+   {
+      return fileName;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static class NoValueException extends Exception
+   {
+      /*******************************************************************************
+       ** Constructor
+       **
+       *******************************************************************************/
+      public NoValueException()
+      {
+         super("No value");
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public void addProblem(LoadingProblem problem)
+   {
+      problems.add(problem);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for problems
+    **
+    *******************************************************************************/
+   public List<LoadingProblem> getProblems()
+   {
+      return (problems);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/ClassDetectingMetaDataLoader.java

@@ -0,0 +1,120 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.InputStream;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.GenericMetaDataLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.utils.ClassPathUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.AnyKey;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+
+
+/*******************************************************************************
+ ** Generic implementation of AbstractMetaDataLoader, who "detects" the class
+ ** of meta data object to be created, then defers to an appropriate subclass
+ ** to do the work.
+ *******************************************************************************/
+public class ClassDetectingMetaDataLoader extends AbstractMetaDataLoader<QMetaDataObject>
+{
+   private static final Memoization<AnyKey, List<Class<?>>> memoizedMetaDataObjectClasses = new Memoization<>();
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public AbstractMetaDataLoader<?> getLoaderForFile(InputStream inputStream, String fileName) throws QMetaDataLoaderException
+   {
+      Map<String, Object> map = fileToMap(inputStream, fileName);
+      return (getLoaderForMap(map));
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public AbstractMetaDataLoader<?> getLoaderForMap(Map<String, Object> map) throws QMetaDataLoaderException
+   {
+      if(map.containsKey("class"))
+      {
+         String classProperty = ValueUtils.getValueAsString(map.get("class"));
+         try
+         {
+            if(MetaDataLoaderRegistry.hasLoaderForSimpleName(classProperty))
+            {
+               Class<? extends AbstractMetaDataLoader<?>> loaderClass = MetaDataLoaderRegistry.getLoaderForSimpleName(classProperty);
+               return (loaderClass.getConstructor().newInstance());
+            }
+            else
+            {
+               Optional<List<Class<?>>> metaDataClasses = memoizedMetaDataObjectClasses.getResult(AnyKey.getInstance(), k -> ClassPathUtils.getClassesContainingNameAndOfType("MetaData", QMetaDataObject.class));
+               if(metaDataClasses.isEmpty())
+               {
+                  throw (new QMetaDataLoaderException("Could not get list of metaDataObjects from class loader"));
+               }
+
+               for(Class<?> c : metaDataClasses.get())
+               {
+                  if(c.getSimpleName().equals(classProperty) && QMetaDataObject.class.isAssignableFrom(c))
+                  {
+                     @SuppressWarnings("unchecked")
+                     Class<? extends QMetaDataObject> metaDataClass = (Class<? extends QMetaDataObject>) c;
+                     return new GenericMetaDataLoader<>(metaDataClass);
+                  }
+               }
+            }
+            throw new QMetaDataLoaderException("Unexpected class [" + classProperty + "] (not a QMetaDataObject; doesn't have a registered MetaDataLoader) specified in " + getFileName());
+         }
+         catch(QMetaDataLoaderException qmdle)
+         {
+            throw (qmdle);
+         }
+         catch(Exception e)
+         {
+            throw new QMetaDataLoaderException("Error handling class [" + classProperty + "] specified in " + getFileName(), e);
+         }
+      }
+      else
+      {
+         throw new QMetaDataLoaderException("Cannot detect meta-data type, because [class] attribute was not specified in file: " + getFileName());
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QMetaDataObject mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      AbstractMetaDataLoader<?> loaderForMap = getLoaderForMap(map);
+      return loaderForMap.mapToMetaDataObject(qInstance, map, context);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/LoadingContext.java

@@ -0,0 +1,38 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+/*******************************************************************************
+ ** Record to track where loader objects are - e.g., what file they're on,
+ ** and at what property path within the file (e.g., helps report problems).
+ *******************************************************************************/
+public record LoadingContext(String fileName, String propertyPath)
+{
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public LoadingContext descendToProperty(String propertyName)
+   {
+      return new LoadingContext(fileName, propertyPath + propertyName + "/");
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/LoadingProblem.java

@@ -0,0 +1,49 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+/*******************************************************************************
+ ** record that tracks a problem that was encountered when loading files.
+ *******************************************************************************/
+public record LoadingProblem(LoadingContext context, String message, Exception exception) // todo Level if useful
+{
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public LoadingProblem(LoadingContext context, String message)
+   {
+      this(context, message, null);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String toString()
+   {
+      return "at[" + context.fileName() + "][" + context.propertyPath() + "]: " + message;
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/MetaDataLoaderHelper.java

@@ -0,0 +1,118 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.Pair;
+
+
+/*******************************************************************************
+ ** class that loads a directory full of meta data files into meta data objects,
+ ** and then sets all of them in a QInstance.
+ *******************************************************************************/
+public class MetaDataLoaderHelper
+{
+   private static final QLogger LOG = QLogger.getLogger(MetaDataLoaderHelper.class);
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public static void processAllMetaDataFilesInDirectory(QInstance qInstance, String path) throws QException
+   {
+      List<Pair<File, AbstractMetaDataLoader<?>>> loaders = new ArrayList<>();
+
+      File directory = new File(path);
+      processAllMetaDataFilesInDirectory(loaders, directory);
+
+      // todo - some version of sorting the loaders by type or possibly a sort field within the files (or file names)
+
+      for(Pair<File, AbstractMetaDataLoader<?>> pair : loaders)
+      {
+         File                      file   = pair.getA();
+         AbstractMetaDataLoader<?> loader = pair.getB();
+         try(FileInputStream fileInputStream = new FileInputStream(file))
+         {
+            QMetaDataObject qMetaDataObject = loader.fileToMetaDataObject(qInstance, fileInputStream, file.getName());
+
+            if(CollectionUtils.nullSafeHasContents(loader.getProblems()))
+            {
+               loader.getProblems().forEach(System.out::println);
+            }
+
+            if(qMetaDataObject instanceof TopLevelMetaDataInterface topLevelMetaData)
+            {
+               topLevelMetaData.addSelfToInstance(qInstance);
+            }
+            else
+            {
+               LOG.warn("Received a non-topLevelMetaDataObject from file: " + file.getAbsolutePath());
+            }
+         }
+         catch(Exception e)
+         {
+            LOG.error("Error processing file: " + file.getAbsolutePath(), e);
+         }
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   private static void processAllMetaDataFilesInDirectory(List<Pair<File, AbstractMetaDataLoader<?>>> loaders, File directory) throws QException
+   {
+      for(File file : Objects.requireNonNullElse(directory.listFiles(), new File[0]))
+      {
+         if(file.isDirectory())
+         {
+            processAllMetaDataFilesInDirectory(loaders, file);
+         }
+         else
+         {
+            try(FileInputStream fileInputStream = new FileInputStream(file))
+            {
+               AbstractMetaDataLoader<?> loader = new ClassDetectingMetaDataLoader().getLoaderForFile(fileInputStream, file.getName());
+               loaders.add(Pair.of(file, loader));
+            }
+            catch(Exception e)
+            {
+               LOG.error("Error processing file: " + file.getAbsolutePath(), e);
+            }
+         }
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/MetaDataLoaderRegistry.java

@@ -0,0 +1,120 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.QTableMetaDataLoader;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.utils.ClassPathUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MetaDataLoaderRegistry
+{
+   private static final QLogger LOG = QLogger.getLogger(AbstractMetaDataLoader.class);
+
+   private static final Map<Class<?>, Class<? extends AbstractMetaDataLoader<?>>> registeredLoaders                   = new HashMap<>();
+   private static final Map<String, Class<? extends AbstractMetaDataLoader<?>>>   registeredLoadersByTargetSimpleName = new HashMap<>();
+
+   static
+   {
+      try
+      {
+         List<Class<?>> classesInPackage = ClassPathUtils.getClassesInPackage(QTableMetaDataLoader.class.getPackageName());
+         for(Class<?> possibleLoaderClass : classesInPackage)
+         {
+            try
+            {
+               Type superClass = possibleLoaderClass.getGenericSuperclass();
+               if(superClass.getTypeName().startsWith(AbstractMetaDataLoader.class.getName() + "<"))
+               {
+                  Type actualTypeArgument = ((ParameterizedType) superClass).getActualTypeArguments()[0];
+                  if(actualTypeArgument instanceof Class)
+                  {
+                     //noinspection unchecked
+                     Class<? extends AbstractMetaDataLoader<?>> loaderClass = (Class<? extends AbstractMetaDataLoader<?>>) possibleLoaderClass;
+
+                     Class<?> metaDataObjectType = Class.forName(actualTypeArgument.getTypeName());
+                     registeredLoaders.put(metaDataObjectType, loaderClass);
+                     registeredLoadersByTargetSimpleName.put(metaDataObjectType.getSimpleName(), loaderClass);
+                  }
+               }
+            }
+            catch(Exception e)
+            {
+               LOG.info("Error on class: " + possibleLoaderClass, e);
+            }
+         }
+
+         System.out.println("Registered loaders: " + registeredLoadersByTargetSimpleName);
+      }
+      catch(Exception e)
+      {
+         LOG.error("Error in static init block for MetaDataLoaderRegistry", e);
+      }
+   }
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static boolean hasLoaderForClass(Class<?> metaDataClass)
+   {
+      return registeredLoaders.containsKey(metaDataClass);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static Class<? extends AbstractMetaDataLoader<?>> getLoaderForClass(Class<?> metaDataClass)
+   {
+      return registeredLoaders.get(metaDataClass);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static boolean hasLoaderForSimpleName(String targetSimpleName)
+   {
+      return registeredLoadersByTargetSimpleName.containsKey(targetSimpleName);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static Class<? extends AbstractMetaDataLoader<?>> getLoaderForSimpleName(String targetSimpleName)
+   {
+      return registeredLoadersByTargetSimpleName.get(targetSimpleName);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/QMetaDataLoaderException.java

@@ -0,0 +1,50 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class QMetaDataLoaderException extends Exception
+{
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public QMetaDataLoaderException(String message)
+   {
+      super(message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public QMetaDataLoaderException(String message, Throwable cause)
+   {
+      super(message, cause);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/GenericMetaDataLoader.java

@@ -0,0 +1,71 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.AbstractMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class GenericMetaDataLoader<T extends QMetaDataObject> extends AbstractMetaDataLoader<T>
+{
+   private final Class<T> metaDataClass;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public GenericMetaDataLoader(Class<T> metaDataClass)
+   {
+      this.metaDataClass = metaDataClass;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public T mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      try
+      {
+         T object = metaDataClass.getConstructor().newInstance();
+         reflectivelyMap(qInstance, object, map, context);
+         return (object);
+      }
+      catch(Exception e)
+      {
+         throw (new QMetaDataLoaderException("Error loading metaData object of type " + metaDataClass.getSimpleName(), e));
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/QStepDataLoader.java

@@ -0,0 +1,85 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.AbstractMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class QStepDataLoader extends AbstractMetaDataLoader<QStepMetaData>
+{
+   private static final QLogger LOG = QLogger.getLogger(QStepDataLoader.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QStepMetaData mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      String stepType = ValueUtils.getValueAsString(map.get("stepType"));
+
+      if(!StringUtils.hasContent(stepType))
+      {
+         throw (new QMetaDataLoaderException("stepType was not specified for process step"));
+      }
+
+      QStepMetaData step;
+      if("backend".equalsIgnoreCase(stepType))
+      {
+         step = new QBackendStepMetaData();
+         reflectivelyMap(qInstance, step, map, context);
+      }
+      else if("frontend".equalsIgnoreCase(stepType))
+      {
+         step = new QFrontendStepMetaData();
+         reflectivelyMap(qInstance, step, map, context);
+      }
+      // todo - we have custom factory methods for this, so, maybe needs all custom loader?
+      // else if("stateMachine".equalsIgnoreCase(stepType))
+      // {
+      //    step = new QStateMachineStep();
+      //    reflectivelyMap(qInstance, step, map, context);
+      // }
+      else
+      {
+         throw (new QMetaDataLoaderException("Unsupported step stepType: " + stepType));
+      }
+
+      return (step);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/QTableMetaDataLoader.java

@@ -0,0 +1,58 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.AbstractMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class QTableMetaDataLoader extends AbstractMetaDataLoader<QTableMetaData>
+{
+   private static final QLogger LOG = QLogger.getLogger(QTableMetaDataLoader.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QTableMetaData mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      QTableMetaData table = new QTableMetaData();
+
+      reflectivelyMap(qInstance, table, map, context);
+
+      // todo - handle QTableBackendDetails, based on backend's type
+
+      return (table);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/QProcessPayload.java

@@ -0,0 +1,162 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.actions.processes;
+
+
+import java.io.Serializable;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntityField;
+import com.kingsrook.qqq.backend.core.utils.ListingHash;
+import com.kingsrook.qqq.backend.core.utils.ReflectiveBeanLikeClassUtils;
+
+
+/*******************************************************************************
+ ** base-class for bean-like classes to represent the fields of a process.
+ ** similar in spirit to QRecordEntity, but for processes.
+ *******************************************************************************/
+public class QProcessPayload
+{
+   private static final QLogger LOG = QLogger.getLogger(QProcessPayload.class);
+
+   private static final ListingHash<Class<? extends QProcessPayload>, QRecordEntityField> fieldMapping = new ListingHash<>();
+
+
+
+   /*******************************************************************************
+    ** Build an entity of this QRecord type from a QRecord
+    **
+    *******************************************************************************/
+   public static <T extends QProcessPayload> T fromProcessState(Class<T> c, ProcessState processState) throws QException
+   {
+      try
+      {
+         T entity = c.getConstructor().newInstance();
+         entity.populateFromProcessState(processState);
+         return (entity);
+      }
+      catch(Exception e)
+      {
+         throw (new QException("Error building process payload from state.", e));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   protected void populateFromProcessState(ProcessState processState)
+   {
+      try
+      {
+         List<QRecordEntityField> fieldList = getFieldList(this.getClass());
+
+         for(QRecordEntityField qRecordEntityField : fieldList)
+         {
+            Serializable value      = processState.getValues().get(qRecordEntityField.getFieldName());
+            Object       typedValue = qRecordEntityField.convertValueType(value);
+            qRecordEntityField.getSetter().invoke(this, typedValue);
+         }
+      }
+      catch(Exception e)
+      {
+         throw (new QRuntimeException("Error building process payload from process state.", e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Copy the values from this payload into the given process state.
+    ** ALL fields in the entity will be set in the process state.
+    **
+    *******************************************************************************/
+   public void toProcessState(ProcessState processState) throws QRuntimeException
+   {
+      try
+      {
+         for(QRecordEntityField qRecordEntityField : getFieldList(this.getClass()))
+         {
+            processState.getValues().put(qRecordEntityField.getFieldName(), (Serializable) qRecordEntityField.getGetter().invoke(this));
+         }
+      }
+      catch(Exception e)
+      {
+         throw (new QRuntimeException("Error populating process state from process payload.", e));
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public static Set<Class<?>> allowedFieldTypes()
+   {
+      HashSet<Class<?>> classes = new HashSet<>(ReflectiveBeanLikeClassUtils.defaultAllowedTypes());
+      classes.add(Map.class);
+      classes.add(List.class);
+      return (classes);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static List<QRecordEntityField> getFieldList(Class<? extends QProcessPayload> c)
+   {
+      if(!fieldMapping.containsKey(c))
+      {
+         List<QRecordEntityField> fieldList = new ArrayList<>();
+         for(Method possibleGetter : c.getMethods())
+         {
+            if(ReflectiveBeanLikeClassUtils.isGetter(possibleGetter, false, allowedFieldTypes()))
+            {
+               Optional<Method> setter = ReflectiveBeanLikeClassUtils.getSetterForGetter(c, possibleGetter);
+
+               if(setter.isPresent())
+               {
+                  String fieldName = ReflectiveBeanLikeClassUtils.getFieldNameFromGetter(possibleGetter);
+                  fieldList.add(new QRecordEntityField(fieldName, possibleGetter, setter.get(), possibleGetter.getReturnType(), null));
+               }
+               else
+               {
+                  LOG.debug("Getter method [" + possibleGetter.getName() + "] does not have a corresponding setter.");
+               }
+            }
+         }
+         fieldMapping.put(c, fieldList);
+      }
+      return (fieldMapping.get(c));
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/RunBackendStepInput.java

@@ -628,4 +628,15 @@ public class RunBackendStepInput extends AbstractActionInput
    {
       return (QContext.getQInstance().getProcess(getProcessName()));
    }
+
+
+
+   /***************************************************************************
+    ** return a QProcessPayload subclass instance, with values populated from
+    ** the current process state.
+    ***************************************************************************/
+   public <T extends QProcessPayload> T getProcessPayload(Class<T> payloadClass) throws QException
+   {
+      return QProcessPayload.fromProcessState(payloadClass, getProcessState());
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/RunBackendStepOutput.java

@@ -445,4 +445,14 @@ public class RunBackendStepOutput extends AbstractActionOutput implements Serial
       this.processState.setProcessMetaDataAdjustment(processMetaDataAdjustment);
    }
 
+
+
+   /***************************************************************************
+    ** Update the process state with values from the input processPayload
+    ** subclass instance.
+    ***************************************************************************/
+   public void setProcessPayload(QProcessPayload processPayload)
+   {
+      processPayload.toProcessState(getProcessState());
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QFilterCriteria.java

@@ -33,6 +33,7 @@ import java.util.Set;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.serialization.QFilterCriteriaDeserializer;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
@@ -42,7 +43,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  *
  *******************************************************************************/
 @JsonDeserialize(using = QFilterCriteriaDeserializer.class)
-public class QFilterCriteria implements Serializable, Cloneable
+public class QFilterCriteria implements Serializable, Cloneable, QMetaDataObject
 {
    private static final QLogger LOG = QLogger.getLogger(QFilterCriteria.class);
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QFilterOrderBy.java

@@ -23,13 +23,14 @@ package com.kingsrook.qqq.backend.core.model.actions.tables.query;
 
 
 import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** Bean representing an element of a query order-by clause.
  **
  *******************************************************************************/
-public class QFilterOrderBy implements Serializable, Cloneable
+public class QFilterOrderBy implements Serializable, Cloneable, QMetaDataObject
 {
    private String  fieldName;
    private boolean isAscending = true;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QQueryFilter.java

@@ -36,6 +36,7 @@ import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.AbstractFilterExpression;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.FilterVariableExpression;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
@@ -45,7 +46,7 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  * Full "filter" for a query - a list of criteria and order-bys
  *
  *******************************************************************************/
-public class QQueryFilter implements Serializable, Cloneable
+public class QQueryFilter implements Serializable, Cloneable, QMetaDataObject
 {
    private static final QLogger LOG = QLogger.getLogger(QQueryFilter.class);
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/dashboard/widgets/WidgetType.java

@@ -66,6 +66,7 @@ public enum WidgetType
    // record view/edit widgets //
    //////////////////////////////
    CHILD_RECORD_LIST("childRecordList"),
+   CUSTOM_COMPONENT("customComponent"),
    DYNAMIC_FORM("dynamicForm"),
    DATA_BAG_VIEWER("dataBagViewer"),
    PIVOT_TABLE_SETUP("pivotTableSetup"),

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecord.java

@@ -468,7 +468,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public String getValueString(String fieldName)
    {
-      return (ValueUtils.getValueAsString(values.get(fieldName)));
+      return (ValueUtils.getValueAsString(getValue(fieldName)));
    }
 
 
@@ -479,7 +479,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public Integer getValueInteger(String fieldName)
    {
-      return (ValueUtils.getValueAsInteger(values.get(fieldName)));
+      return (ValueUtils.getValueAsInteger(getValue(fieldName)));
    }
 
 
@@ -490,7 +490,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public Long getValueLong(String fieldName)
    {
-      return (ValueUtils.getValueAsLong(values.get(fieldName)));
+      return (ValueUtils.getValueAsLong(getValue(fieldName)));
    }
 
 
@@ -500,7 +500,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public BigDecimal getValueBigDecimal(String fieldName)
    {
-      return (ValueUtils.getValueAsBigDecimal(values.get(fieldName)));
+      return (ValueUtils.getValueAsBigDecimal(getValue(fieldName)));
    }
 
 
@@ -510,7 +510,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public Boolean getValueBoolean(String fieldName)
    {
-      return (ValueUtils.getValueAsBoolean(values.get(fieldName)));
+      return (ValueUtils.getValueAsBoolean(getValue(fieldName)));
    }
 
 
@@ -520,7 +520,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public LocalTime getValueLocalTime(String fieldName)
    {
-      return (ValueUtils.getValueAsLocalTime(values.get(fieldName)));
+      return (ValueUtils.getValueAsLocalTime(getValue(fieldName)));
    }
 
 
@@ -530,7 +530,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public LocalDate getValueLocalDate(String fieldName)
    {
-      return (ValueUtils.getValueAsLocalDate(values.get(fieldName)));
+      return (ValueUtils.getValueAsLocalDate(getValue(fieldName)));
    }
 
 
@@ -540,7 +540,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public byte[] getValueByteArray(String fieldName)
    {
-      return (ValueUtils.getValueAsByteArray(values.get(fieldName)));
+      return (ValueUtils.getValueAsByteArray(getValue(fieldName)));
    }
 
 
@@ -550,7 +550,7 @@ public class QRecord implements Serializable
     *******************************************************************************/
    public Instant getValueInstant(String fieldName)
    {
-      return (ValueUtils.getValueAsInstant(values.get(fieldName)));
+      return (ValueUtils.getValueAsInstant(getValue(fieldName)));
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecordEntity.java

@@ -49,6 +49,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.ListingHash;
 import com.kingsrook.qqq.backend.core.utils.ObjectUtils;
+import com.kingsrook.qqq.backend.core.utils.ReflectiveBeanLikeClassUtils;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -325,13 +326,13 @@ public abstract class QRecordEntity
          List<QRecordEntityField> fieldList = new ArrayList<>();
          for(Method possibleGetter : c.getMethods())
          {
-            if(isGetter(possibleGetter))
+            if(ReflectiveBeanLikeClassUtils.isGetter(possibleGetter, true))
             {
-               Optional<Method> setter = getSetterForGetter(c, possibleGetter);
+               Optional<Method> setter = ReflectiveBeanLikeClassUtils.getSetterForGetter(c, possibleGetter);
 
                if(setter.isPresent())
                {
-                  String           fieldName       = getFieldNameFromGetter(possibleGetter);
+                  String           fieldName       = ReflectiveBeanLikeClassUtils.getFieldNameFromGetter(possibleGetter);
                   Optional<QField> fieldAnnotation = getQFieldAnnotation(c, fieldName);
 
                   if(fieldAnnotation.isPresent())
@@ -378,19 +379,19 @@ public abstract class QRecordEntity
          List<QRecordEntityAssociation> associationList = new ArrayList<>();
          for(Method possibleGetter : c.getMethods())
          {
-            if(isGetter(possibleGetter))
+            if(ReflectiveBeanLikeClassUtils.isGetter(possibleGetter, true))
             {
-               Optional<Method> setter = getSetterForGetter(c, possibleGetter);
+               Optional<Method> setter = ReflectiveBeanLikeClassUtils.getSetterForGetter(c, possibleGetter);
 
                if(setter.isPresent())
                {
-                  String                 fieldName             = getFieldNameFromGetter(possibleGetter);
+                  String                 fieldName             = ReflectiveBeanLikeClassUtils.getFieldNameFromGetter(possibleGetter);
                   Optional<QAssociation> associationAnnotation = getQAssociationAnnotation(c, fieldName);
 
                   if(associationAnnotation.isPresent())
                   {
                      @SuppressWarnings("unchecked")
-                     Class<? extends QRecordEntity> listTypeParam = (Class<? extends QRecordEntity>) getListTypeParam(possibleGetter.getReturnType(), possibleGetter.getAnnotatedReturnType());
+                     Class<? extends QRecordEntity> listTypeParam = (Class<? extends QRecordEntity>) ReflectiveBeanLikeClassUtils.getListTypeParam(possibleGetter.getReturnType(), possibleGetter.getAnnotatedReturnType());
                      associationList.add(new QRecordEntityAssociation(fieldName, possibleGetter, setter.get(), listTypeParam, associationAnnotation.orElse(null)));
                   }
                }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecordEntityField.java

@@ -28,6 +28,7 @@ import java.math.BigDecimal;
 import java.time.Instant;
 import java.time.LocalDate;
 import java.time.LocalTime;
+import java.util.Map;
 import com.kingsrook.qqq.backend.core.exceptions.QValueException;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 
@@ -170,6 +171,11 @@ public class QRecordEntityField
          {
             return (ValueUtils.getValueAsByteArray(value));
          }
+
+         if(type.equals(Map.class))
+         {
+            return (ValueUtils.getValueAsMap(value));
+         }
       }
       catch(Exception e)
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecordWithJoinedRecords.java

@@ -24,6 +24,7 @@ package com.kingsrook.qqq.backend.core.model.data;
 
 import java.io.Serializable;
 import java.util.LinkedHashMap;
+import java.util.List;
 import java.util.Map;
 import java.util.function.BiFunction;
 
@@ -198,4 +199,62 @@ public class QRecordWithJoinedRecords extends QRecord
       return (rs);
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public Map<String, List<QRecord>> getAssociatedRecords()
+   {
+      return mainRecord.getAssociatedRecords();
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QRecord withAssociatedRecord(String name, QRecord associatedRecord)
+   {
+      mainRecord.withAssociatedRecord(name, associatedRecord);
+      return (this);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QRecord withAssociatedRecords(Map<String, List<QRecord>> associatedRecords)
+   {
+      mainRecord.withAssociatedRecords(associatedRecords);
+      return (this);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void setAssociatedRecords(Map<String, List<QRecord>> associatedRecords)
+   {
+      mainRecord.setAssociatedRecords(associatedRecords);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QRecord withAssociatedRecords(String name, List<QRecord> associatedRecords)
+   {
+      mainRecord.withAssociatedRecords(name, associatedRecords);
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/helpcontent/HelpContentMetaDataProvider.java

@@ -82,6 +82,7 @@ public class HelpContentMetaDataProvider
       table.getField("key").withFieldAdornment(AdornmentType.Size.LARGE.toAdornment());
       table.getField("content").withFieldAdornment(AdornmentType.Size.LARGE.toAdornment());
       table.getField("content").withFieldAdornment(new FieldAdornment(AdornmentType.CODE_EDITOR).withValue(AdornmentType.CodeEditorValues.languageMode("html")));
+      table.getField("content").withGridColumns(12);
 
       if(backendDetailEnricher != null)
       {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QAuthenticationType.java

@@ -28,6 +28,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
  *******************************************************************************/
 public enum QAuthenticationType
 {
+   OAUTH2("OAuth2"),
    AUTH_0("auth0"),
    TABLE_BASED("tableBased"),
    FULLY_ANONYMOUS("fullyAnonymous"),

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QInstance.java

@@ -1250,7 +1250,7 @@ public class QInstance
       {
          this.supplementalMetaData = new HashMap<>();
       }
-      this.supplementalMetaData.put(supplementalMetaData.getType(), supplementalMetaData);
+      this.supplementalMetaData.put(supplementalMetaData.getName(), supplementalMetaData);
       return (this);
    }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QMetaDataObject.java

@@ -0,0 +1,34 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata;
+
+
+import java.io.Serializable;
+
+
+/*******************************************************************************
+ ** interface common among all objects that can be considered qqq meta data -
+ ** e.g., stored in a QInstance.
+ *******************************************************************************/
+public interface QMetaDataObject extends Serializable
+{
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QSupplementalInstanceMetaData.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.model.metadata;
 
 
+import java.util.function.Supplier;
 import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 
@@ -30,20 +31,13 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
  ** Base-class for instance-level meta-data defined by some supplemental module, etc,
  ** outside of qqq core
  *******************************************************************************/
-public abstract class QSupplementalInstanceMetaData implements TopLevelMetaDataInterface
+public interface QSupplementalInstanceMetaData extends TopLevelMetaDataInterface
 {
 
-   /*******************************************************************************
-    ** Getter for type
-    *******************************************************************************/
-   public abstract String getType();
-
-
-
    /*******************************************************************************
     **
     *******************************************************************************/
-   public void enrich(QTableMetaData table)
+   default void enrich(QTableMetaData table)
    {
       ////////////////////////
       // noop in base class //
@@ -55,7 +49,7 @@ public abstract class QSupplementalInstanceMetaData implements TopLevelMetaDataI
    /*******************************************************************************
     **
     *******************************************************************************/
-   public void validate(QInstance qInstance, QInstanceValidator validator)
+   default void validate(QInstance qInstance, QInstanceValidator validator)
    {
       ////////////////////////
       // noop in base class //
@@ -68,9 +62,33 @@ public abstract class QSupplementalInstanceMetaData implements TopLevelMetaDataI
     **
     *******************************************************************************/
    @Override
-   public void addSelfToInstance(QInstance qInstance)
+   default void addSelfToInstance(QInstance qInstance)
    {
       qInstance.withSupplementalMetaData(this);
    }
 
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static <S extends QSupplementalInstanceMetaData> S of(QInstance qInstance, String name)
+   {
+      return ((S) qInstance.getSupplementalMetaData(name));
+   }
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static <S extends QSupplementalInstanceMetaData> S ofOrWithNew(QInstance qInstance, String name, Supplier<S> supplier)
+   {
+      S s = (S) qInstance.getSupplementalMetaData(name);
+      if(s == null)
+      {
+         s = supplier.get();
+         s.addSelfToInstance(qInstance);
+      }
+      return (s);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/TopLevelMetaDataInterface.java

@@ -26,7 +26,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
  ** Interface for meta-data classes that can be added directly (e.g, at the top
  ** level) to a QInstance (such as a QTableMetaData - not a QFieldMetaData).
  *******************************************************************************/
-public interface TopLevelMetaDataInterface extends MetaDataProducerOutput
+public interface TopLevelMetaDataInterface extends MetaDataProducerOutput, QMetaDataObject
 {
 
    /*******************************************************************************

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/audits/QAuditRules.java

@@ -22,10 +22,13 @@
 package com.kingsrook.qqq.backend.core.model.metadata.audits;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  **
  *******************************************************************************/
-public class QAuditRules
+public class QAuditRules implements QMetaDataObject
 {
    private AuditLevel auditLevel;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/authentication/OAuth2AuthenticationMetaData.java

@@ -0,0 +1,320 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.authentication;
+
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
+import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher;
+import com.kingsrook.qqq.backend.core.modules.authentication.implementations.OAuth2AuthenticationModule;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ ** Meta-data to provide details of an OAuth2 Authentication module
+ *******************************************************************************/
+public class OAuth2AuthenticationMetaData extends QAuthenticationMetaData
+{
+   private String baseUrl;
+   private String tokenUrl;
+   private String clientId;
+   private String scopes;
+
+   private String userSessionTableName;
+   private String redirectStateTableName;
+
+   ////////////////////////////////////////////////////////////////////////////////////////
+   // keep this secret, on the server - don't let it be serialized and sent to a client! //
+   ////////////////////////////////////////////////////////////////////////////////////////
+   @JsonIgnore
+   private String clientSecret;
+
+
+
+   /*******************************************************************************
+    ** Default Constructor.
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData()
+   {
+      super();
+      setType(QAuthenticationType.OAUTH2);
+
+      //////////////////////////////////////////////////////////
+      // ensure this module is registered with the dispatcher //
+      //////////////////////////////////////////////////////////
+      QAuthenticationModuleDispatcher.registerModule(QAuthenticationType.OAUTH2.getName(), OAuth2AuthenticationModule.class.getName());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void validate(QInstance qInstance, QInstanceValidator qInstanceValidator)
+   {
+      super.validate(qInstance, qInstanceValidator);
+
+      String prefix = "OAuth2AuthenticationMetaData (named '" + getName() + "'): ";
+
+      qInstanceValidator.assertCondition(StringUtils.hasContent(baseUrl), prefix + "baseUrl must be set");
+      qInstanceValidator.assertCondition(StringUtils.hasContent(clientId), prefix + "clientId must be set");
+      qInstanceValidator.assertCondition(StringUtils.hasContent(clientSecret), prefix + "clientSecret must be set");
+      qInstanceValidator.assertCondition(StringUtils.hasContent(scopes), prefix + "scopes must be set");
+
+      if(qInstanceValidator.assertCondition(StringUtils.hasContent(userSessionTableName), prefix + "userSessionTableName must be set"))
+      {
+         qInstanceValidator.assertCondition(qInstance.getTable(userSessionTableName) != null, prefix + "userSessionTableName ('" + userSessionTableName + "') was not found in the instance");
+      }
+
+      if(qInstanceValidator.assertCondition(StringUtils.hasContent(redirectStateTableName), prefix + "redirectStateTableName must be set"))
+      {
+         qInstanceValidator.assertCondition(qInstance.getTable(redirectStateTableName) != null, prefix + "redirectStateTableName ('" + redirectStateTableName + "') was not found in the instance");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter, override to help fluent flows
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withBaseUrl(String baseUrl)
+   {
+      setBaseUrl(baseUrl);
+      return this;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for baseUrl
+    **
+    *******************************************************************************/
+   public String getBaseUrl()
+   {
+      return baseUrl;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for baseUrl
+    **
+    *******************************************************************************/
+   public void setBaseUrl(String baseUrl)
+   {
+      this.baseUrl = baseUrl;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter, override to help fluent flows
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withClientId(String clientId)
+   {
+      setClientId(clientId);
+      return this;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for clientId
+    **
+    *******************************************************************************/
+   public String getClientId()
+   {
+      return clientId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for clientId
+    **
+    *******************************************************************************/
+   public void setClientId(String clientId)
+   {
+      this.clientId = clientId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter, override to help fluent flows
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withClientSecret(String clientSecret)
+   {
+      setClientSecret(clientSecret);
+      return this;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for clientSecret
+    **
+    *******************************************************************************/
+   public String getClientSecret()
+   {
+      return clientSecret;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for clientSecret
+    **
+    *******************************************************************************/
+   public void setClientSecret(String clientSecret)
+   {
+      this.clientSecret = clientSecret;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tokenUrl
+    *******************************************************************************/
+   public String getTokenUrl()
+   {
+      return (this.tokenUrl);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tokenUrl
+    *******************************************************************************/
+   public void setTokenUrl(String tokenUrl)
+   {
+      this.tokenUrl = tokenUrl;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tokenUrl
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withTokenUrl(String tokenUrl)
+   {
+      this.tokenUrl = tokenUrl;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for userSessionTableName
+    *******************************************************************************/
+   public String getUserSessionTableName()
+   {
+      return (this.userSessionTableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for userSessionTableName
+    *******************************************************************************/
+   public void setUserSessionTableName(String userSessionTableName)
+   {
+      this.userSessionTableName = userSessionTableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for userSessionTableName
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withUserSessionTableName(String userSessionTableName)
+   {
+      this.userSessionTableName = userSessionTableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for redirectStateTableName
+    *******************************************************************************/
+   public String getRedirectStateTableName()
+   {
+      return (this.redirectStateTableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for redirectStateTableName
+    *******************************************************************************/
+   public void setRedirectStateTableName(String redirectStateTableName)
+   {
+      this.redirectStateTableName = redirectStateTableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for redirectStateTableName
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withRedirectStateTableName(String redirectStateTableName)
+   {
+      this.redirectStateTableName = redirectStateTableName;
+      return (this);
+   }
+
+
+   /*******************************************************************************
+    ** Getter for scopes
+    *******************************************************************************/
+   public String getScopes()
+   {
+      return (this.scopes);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for scopes
+    *******************************************************************************/
+   public void setScopes(String scopes)
+   {
+      this.scopes = scopes;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for scopes
+    *******************************************************************************/
+   public OAuth2AuthenticationMetaData withScopes(String scopes)
+   {
+      this.scopes = scopes;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/authentication/QAuthenticationMetaData.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.authentication;
 import java.util.HashMap;
 import java.util.Map;
 import com.fasterxml.jackson.annotation.JsonFilter;
+import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
 import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
@@ -225,4 +226,15 @@ public class QAuthenticationMetaData implements TopLevelMetaDataInterface
       return (this);
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public void validate(QInstance qInstance, QInstanceValidator qInstanceValidator)
+   {
+      //////////////////
+      // noop at base //
+      //////////////////
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/code/QCodeReference.java

@@ -23,13 +23,14 @@ package com.kingsrook.qqq.backend.core.model.metadata.code;
 
 
 import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** Pointer to code to be ran by the qqq framework, e.g., for custom behavior -
  ** maybe process steps, maybe customization to a table, etc.
  *******************************************************************************/
-public class QCodeReference implements Serializable, Cloneable
+public class QCodeReference implements Serializable, Cloneable, QMetaDataObject
 {
    private String    name;
    private QCodeType codeType;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/QFieldMetaData.java

@@ -40,11 +40,13 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.data.QField;
 import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.help.HelpRole;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ReflectiveBeanLikeClassUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
@@ -54,7 +56,7 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
  ** Meta-data to represent a single field in a table.
  **
  *******************************************************************************/
-public class QFieldMetaData implements Cloneable
+public class QFieldMetaData implements Cloneable, QMetaDataObject
 {
    private static final QLogger LOG = QLogger.getLogger(QFieldMetaData.class);
 
@@ -187,7 +189,7 @@ public class QFieldMetaData implements Cloneable
    {
       try
       {
-         this.name = QRecordEntity.getFieldNameFromGetter(getter);
+         this.name = ReflectiveBeanLikeClassUtils.getFieldNameFromGetter(getter);
          this.type = QFieldType.fromClass(getter.getReturnType());
 
          @SuppressWarnings("unchecked")

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/WhiteSpaceBehavior.java

@@ -0,0 +1,174 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.fields;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.Function;
+import com.kingsrook.qqq.backend.core.actions.values.ValueBehaviorApplier;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+
+
+/*******************************************************************************
+ ** Field behavior that changes the whitespace of string values.
+ *******************************************************************************/
+public enum WhiteSpaceBehavior implements FieldBehavior<WhiteSpaceBehavior>, FieldBehaviorForFrontend, FieldFilterBehavior<WhiteSpaceBehavior>
+{
+   NONE(null),
+   REMOVE_ALL_WHITESPACE((String s) -> s.chars().filter(c -> !Character.isWhitespace(c)).collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString()),
+   TRIM((String s) -> s.trim()),
+   TRIM_LEFT((String s) -> s.stripLeading()),
+   TRIM_RIGHT((String s) -> s.stripTrailing());
+
+
+   private final Function<String, String> function;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   WhiteSpaceBehavior(Function<String, String> function)
+   {
+      this.function = function;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public WhiteSpaceBehavior getDefault()
+   {
+      return (NONE);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void apply(ValueBehaviorApplier.Action action, List<QRecord> recordList, QInstance instance, QTableMetaData table, QFieldMetaData field)
+   {
+      if(this.equals(NONE))
+      {
+         return;
+      }
+
+      switch(this)
+      {
+         case REMOVE_ALL_WHITESPACE, TRIM, TRIM_LEFT, TRIM_RIGHT -> applyFunction(recordList, table, field);
+         default -> throw new IllegalStateException("Unexpected enum value: " + this);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private void applyFunction(List<QRecord> recordList, QTableMetaData table, QFieldMetaData field)
+   {
+      String fieldName = field.getName();
+      for(QRecord record : CollectionUtils.nonNullList(recordList))
+      {
+         String value = record.getValueString(fieldName);
+         if(value != null && function != null)
+         {
+            record.setValue(fieldName, function.apply(value));
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public Serializable applyToFilterCriteriaValue(Serializable value, QInstance instance, QTableMetaData table, QFieldMetaData field)
+   {
+      if(this.equals(NONE) || function == null)
+      {
+         return (value);
+      }
+
+      if(value instanceof String s)
+      {
+         String newValue = function.apply(s);
+         if(!Objects.equals(value, newValue))
+         {
+            return (newValue);
+         }
+      }
+
+      return (value);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public boolean allowMultipleBehaviorsOfThisType()
+   {
+      return (false);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public List<String> validateBehaviorConfiguration(QTableMetaData tableMetaData, QFieldMetaData fieldMetaData)
+   {
+      if(this == NONE)
+      {
+         return Collections.emptyList();
+      }
+
+      List<String> errors      = new ArrayList<>();
+      String       errorSuffix = " field [" + fieldMetaData.getName() + "] in table [" + tableMetaData.getName() + "]";
+
+      if(fieldMetaData.getType() != null)
+      {
+         if(!fieldMetaData.getType().isStringLike())
+         {
+            errors.add("A WhiteSpaceBehavior was a applied to a non-String-like field:" + errorSuffix);
+         }
+      }
+
+      return (errors);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/help/QHelpContent.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.help;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.Set;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
@@ -41,7 +42,7 @@ import java.util.Set;
  ** May be dynamically added to meta-data via (non-meta-) data - see
  ** HelpContentMetaDataProvider and QInstanceHelpContentManager
  *******************************************************************************/
-public class QHelpContent
+public class QHelpContent implements QMetaDataObject
 {
    private String        content;
    private HelpFormat    format;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QAppChildMetaData.java

@@ -22,11 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.layout;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** Interface shared by meta-data objects which can be placed into an App.
  ** e.g., Tables, Processes, and Apps themselves (since they can be nested)
  *******************************************************************************/
-public interface QAppChildMetaData
+public interface QAppChildMetaData extends QMetaDataObject
 {
    /*******************************************************************************
     **

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QAppSection.java

@@ -24,12 +24,13 @@ package com.kingsrook.qqq.backend.core.model.metadata.layout;
 
 import java.util.ArrayList;
 import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** A section of apps/tables/processes - a logical grouping.
  *******************************************************************************/
-public class QAppSection implements Cloneable
+public class QAppSection implements Cloneable, QMetaDataObject
 {
    private String name;
    private String label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QIcon.java

@@ -22,6 +22,9 @@
 package com.kingsrook.qqq.backend.core.model.metadata.layout;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** Icon to show associated with an App, Table, Process, etc.
  **
@@ -31,7 +34,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.layout;
  ** Future may allow something like a "namespace", and/or multiple icons for
  ** use in different frontends, etc.
  *******************************************************************************/
-public class QIcon implements Cloneable
+public class QIcon implements Cloneable, QMetaDataObject
 {
    private String name;
    private String path;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/permissions/QPermissionRules.java

@@ -22,13 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.permissions;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 
 
 /*******************************************************************************
  **
  *******************************************************************************/
-public class QPermissionRules implements Cloneable
+public class QPermissionRules implements Cloneable, QMetaDataObject
 {
    private PermissionLevel level;
    private DenyBehavior    denyBehavior;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QFrontendComponentMetaData.java

@@ -25,12 +25,13 @@ package com.kingsrook.qqq.backend.core.model.metadata.processes;
 import java.io.Serializable;
 import java.util.HashMap;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** Definition of a UI component in a frontend process steps.
  *******************************************************************************/
-public class QFrontendComponentMetaData
+public class QFrontendComponentMetaData implements QMetaDataObject
 {
    private QComponentType type;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QProcessMetaData.java

@@ -327,12 +327,23 @@ public class QProcessMetaData implements QAppChildMetaData, MetaDataWithPermissi
 
 
    /*******************************************************************************
-    ** Setter for stepList
+    ** Setter for stepList - note - calling this method ALSO overwrites the steps map!
     **
     *******************************************************************************/
    public void setStepList(List<QStepMetaData> stepList)
    {
-      this.stepList = stepList;
+      if(stepList == null)
+      {
+         this.stepList = null;
+         this.steps = null;
+      }
+      else
+      {
+         this.stepList = new ArrayList<>();
+         this.steps = new HashMap<>();
+      }
+
+      withStepList(stepList);
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QStateMachineStep.java

@@ -185,4 +185,25 @@ public class QStateMachineStep extends QStepMetaData
       return (rs);
    }
 
+
+   /*******************************************************************************
+    ** Setter for subSteps
+    *******************************************************************************/
+   public void setSubSteps(List<QStepMetaData> subSteps)
+   {
+      this.subSteps = subSteps;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for subSteps
+    *******************************************************************************/
+   public QStateMachineStep withSubSteps(List<QStepMetaData> subSteps)
+   {
+      this.subSteps = subSteps;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QStepMetaData.java

@@ -26,6 +26,7 @@ import java.util.ArrayList;
 import java.util.List;
 import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.serialization.QStepMetaDataDeserializer;
 
@@ -37,7 +38,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.serialization.QStepMetaData
  **
  *******************************************************************************/
 @JsonDeserialize(using = QStepMetaDataDeserializer.class)
-public abstract class QStepMetaData
+public abstract class QStepMetaData implements QMetaDataObject
 {
    private String name;
    private String label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/scheduleing/QScheduleMetaData.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.model.metadata.scheduleing;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
 
@@ -35,7 +36,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  ** same moment.
  **
  *******************************************************************************/
-public class QScheduleMetaData
+public class QScheduleMetaData implements QMetaDataObject
 {
    private String schedulerName;
    private String description;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/Association.java

@@ -22,11 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.tables;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** definition of a qqq table that is "associated" with another table, e.g.,
  ** managed along with it - such as child-records under a parent record.
  *******************************************************************************/
-public class Association
+public class Association implements QMetaDataObject
 {
    private String name;
    private String associatedTableName;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/QFieldSection.java

@@ -26,6 +26,7 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Set;
 import com.kingsrook.qqq.backend.core.instances.QInstanceHelpContentManager;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.help.HelpRole;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
@@ -36,7 +37,7 @@ import com.kingsrook.qqq.backend.core.utils.collections.MutableList;
  ** A section of fields - a logical grouping.
  ** TODO - this class should be named QTableSection!
  *******************************************************************************/
-public class QFieldSection
+public class QFieldSection implements QMetaDataObject
 {
    private String name;
    private String label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/TablesCustomPossibleValueProvider.java

@@ -27,11 +27,9 @@ import java.util.ArrayList;
 import java.util.List;
 import com.kingsrook.qqq.backend.core.actions.permissions.PermissionCheckResult;
 import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
-import com.kingsrook.qqq.backend.core.actions.values.QCustomPossibleValueProvider;
+import com.kingsrook.qqq.backend.core.actions.values.BasicCustomPossibleValueProvider;
 import com.kingsrook.qqq.backend.core.context.QContext;
-import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
-import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 
@@ -40,26 +38,16 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  ** possible-value source provider for the `Tables` PVS - a list of all tables
  ** in an application/qInstance.
  *******************************************************************************/
-public class TablesCustomPossibleValueProvider implements QCustomPossibleValueProvider<String>
+public class TablesCustomPossibleValueProvider extends BasicCustomPossibleValueProvider<QTableMetaData, String>
 {
 
    /***************************************************************************
     **
     ***************************************************************************/
    @Override
-   public QPossibleValue<String> getPossibleValue(Serializable idValue)
+   protected QPossibleValue<String> makePossibleValue(QTableMetaData sourceObject)
    {
-      QTableMetaData table = QContext.getQInstance().getTable(ValueUtils.getValueAsString(idValue));
-      if(table != null && !table.getIsHidden())
-      {
-         PermissionCheckResult permissionCheckResult = PermissionsHelper.getPermissionCheckResult(new QueryInput(table.getName()), table);
-         if(PermissionCheckResult.ALLOW.equals(permissionCheckResult))
-         {
-            return (new QPossibleValue<>(table.getName(), table.getLabel()));
-         }
-      }
-
-      return null;
+      return (new QPossibleValue<>(sourceObject.getName(), sourceObject.getLabel()));
    }
 
 
@@ -68,22 +56,54 @@ public class TablesCustomPossibleValueProvider implements QCustomPossibleValuePr
     **
     ***************************************************************************/
    @Override
-   public List<QPossibleValue<String>> search(SearchPossibleValueSourceInput input) throws QException
+   protected QTableMetaData getSourceObject(Serializable id)
    {
-      /////////////////////////////////////////////////////////////////////////////////////
-      // build all of the possible values (note, will be filtered by user's permissions) //
-      /////////////////////////////////////////////////////////////////////////////////////
-      List<QPossibleValue<String>> allPossibleValues = new ArrayList<>();
+      QTableMetaData table = QContext.getQInstance().getTable(ValueUtils.getValueAsString(id));
+      return isTableAllowed(table) ? table : null;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   protected List<QTableMetaData> getAllSourceObjects()
+   {
+      ArrayList<QTableMetaData> rs = new ArrayList<>();
       for(QTableMetaData table : QContext.getQInstance().getTables().values())
       {
-         QPossibleValue<String> possibleValue = getPossibleValue(table.getName());
-         if(possibleValue != null)
+         if(isTableAllowed(table))
          {
-            allPossibleValues.add(possibleValue);
+            rs.add(table);
          }
       }
+      return rs;
+   }
 
-      return completeCustomPVSSearch(input, allPossibleValues);
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private boolean isTableAllowed(QTableMetaData table)
+   {
+      if(table == null)
+      {
+         return (false);
+      }
+
+      if(table.getIsHidden())
+      {
+         return (false);
+      }
+
+      PermissionCheckResult permissionCheckResult = PermissionsHelper.getPermissionCheckResult(new QueryInput(table.getName()), table);
+      if(!PermissionCheckResult.ALLOW.equals(permissionCheckResult))
+      {
+         return (false);
+      }
+
+      return (true);
    }
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/TablesPossibleValueSourceMetaDataProvider.java

@@ -24,6 +24,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables;
 
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PVSValueFormatAndFields;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
@@ -45,6 +46,7 @@ public class TablesPossibleValueSourceMetaDataProvider
    {
       QPossibleValueSource possibleValueSource = new QPossibleValueSource()
          .withName(NAME)
+         .withIdType(QFieldType.STRING)
          .withType(QPossibleValueSourceType.CUSTOM)
          .withCustomCodeReference(new QCodeReference(TablesCustomPossibleValueProvider.class))
          .withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/UniqueKey.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
 
@@ -32,7 +33,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  ** Definition of a Unique Key (or "Constraint", if you wanna use fancy words)
  ** on a QTable.
  *******************************************************************************/
-public class UniqueKey
+public class UniqueKey implements QMetaDataObject
 {
    private List<String> fieldNames;
    private String       label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/AutomationStatusTracking.java

@@ -22,11 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.tables.automation;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** Table-automation meta-data to define how this table's per-record automation
  ** status is tracked.
  *******************************************************************************/
-public class AutomationStatusTracking
+public class AutomationStatusTracking implements QMetaDataObject
 {
    private AutomationStatusTrackingType type;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/QTableAutomationDetails.java

@@ -24,13 +24,14 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables.automation;
 
 import java.util.ArrayList;
 import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
 
 
 /*******************************************************************************
  ** Details about how this table's record automations are set up.
  *******************************************************************************/
-public class QTableAutomationDetails
+public class QTableAutomationDetails implements QMetaDataObject
 {
    private AutomationStatusTracking    statusTracking;
    private String                      providerName;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/TableAutomationAction.java

@@ -25,13 +25,14 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables.automation;
 import java.io.Serializable;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 
 
 /*******************************************************************************
  ** Definition of a specific action to run against a table
  *******************************************************************************/
-public class TableAutomationAction
+public class TableAutomationAction implements QMetaDataObject
 {
    private String       name;
    private TriggerEvent triggerEvent;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/session/QUser.java

@@ -22,10 +22,13 @@
 package com.kingsrook.qqq.backend.core.model.session;
 
 
+import java.io.Serializable;
+
+
 /*******************************************************************************
  **
  *******************************************************************************/
-public class QUser implements Cloneable
+public class QUser implements Cloneable, Serializable
 {
    private String idReference;
    private String fullName;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/QAuthenticationModuleInterface.java

@@ -25,15 +25,14 @@ package com.kingsrook.qqq.backend.core.modules.authentication;
 import java.io.Serializable;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.context.QContext;
-import com.kingsrook.qqq.backend.core.exceptions.AccessTokenException;
 import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 import org.apache.commons.lang.NotImplementedException;
 
 
+
 /*******************************************************************************
  ** Interface that a QAuthenticationModule must implement.
  **
@@ -82,12 +81,12 @@ public interface QAuthenticationModuleInterface
    }
 
 
-   /*******************************************************************************
+   /***************************************************************************
     **
-    *******************************************************************************/
-   default String createAccessToken(QAuthenticationMetaData metaData, String clientId, String clientSecret) throws AccessTokenException
+    ***************************************************************************/
+   default String getLoginRedirectUrl(String originalUrl) throws QAuthenticationException
    {
-      throw (new NotImplementedException("The method createAccessToken() is not implemented in the class: " + this.getClass().getSimpleName()));
+      throw (new NotImplementedException("The method getLoginRedirectUrl() is not implemented in the authentication module: " + this.getClass().getSimpleName()));
    }
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/Auth0AuthenticationModule.java

@@ -1020,7 +1020,7 @@ public class Auth0AuthenticationModule implements QAuthenticationModuleInterface
             // decode the accessToken and make sure it is not expired //
             ////////////////////////////////////////////////////////////
             boolean needNewToken = true;
-            if(accessToken != null)
+            if(StringUtils.hasContent(accessToken))
             {
                DecodedJWT jwt     = JWT.decode(accessToken);
                String     payload = jwt.getPayload();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/FullyAnonymousAuthenticationModule.java

@@ -24,9 +24,7 @@ package com.kingsrook.qqq.backend.core.modules.authentication.implementations;
 
 import java.util.Map;
 import java.util.UUID;
-import com.kingsrook.qqq.backend.core.exceptions.AccessTokenException;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.model.session.QUser;
 import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleInterface;
@@ -77,15 +75,4 @@ public class FullyAnonymousAuthenticationModule implements QAuthenticationModule
       return session != null;
    }
 
-
-
-   /*******************************************************************************
-    ** Load an instance of the appropriate state provider
-    **
-    *******************************************************************************/
-   public String createAccessToken(QAuthenticationMetaData metaData, String clientId, String clientSecret) throws AccessTokenException
-   {
-      return (TEST_ACCESS_TOKEN);
-   }
-
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/MockAuthenticationModule.java

@@ -24,6 +24,7 @@ package com.kingsrook.qqq.backend.core.modules.authentication.implementations;
 
 import java.util.Map;
 import java.util.UUID;
+import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
@@ -45,8 +46,13 @@ public class MockAuthenticationModule implements QAuthenticationModuleInterface
     **
     *******************************************************************************/
    @Override
-   public QSession createSession(QInstance qInstance, Map<String, String> context)
+   public QSession createSession(QInstance qInstance, Map<String, String> context) throws QAuthenticationException
    {
+      if("Deny".equalsIgnoreCase(context.get("accessToken")))
+      {
+         throw (new QAuthenticationException("Access denied (per accessToken requesting as such)"));
+      }
+
       QUser qUser = new QUser();
       qUser.setIdReference("User:" + (System.currentTimeMillis() % USER_ID_MODULO));
       qUser.setFullName("John Smith");
@@ -80,4 +86,16 @@ public class MockAuthenticationModule implements QAuthenticationModuleInterface
 
       return (true);
    }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getLoginRedirectUrl(String originalUrl)
+   {
+      return originalUrl + "?createMockSession=true";
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/OAuth2AuthenticationModule.java

@@ -0,0 +1,486 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.modules.authentication.implementations;
+
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URI;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+import com.auth0.jwt.JWT;
+import com.auth0.jwt.interfaces.DecodedJWT;
+import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.context.CapturedContext;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.authentication.OAuth2AuthenticationMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+import com.kingsrook.qqq.backend.core.model.session.QUser;
+import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleInterface;
+import com.kingsrook.qqq.backend.core.modules.authentication.implementations.model.UserSession;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+import com.nimbusds.oauth2.sdk.AuthorizationCode;
+import com.nimbusds.oauth2.sdk.AuthorizationCodeGrant;
+import com.nimbusds.oauth2.sdk.AuthorizationGrant;
+import com.nimbusds.oauth2.sdk.ErrorObject;
+import com.nimbusds.oauth2.sdk.GeneralException;
+import com.nimbusds.oauth2.sdk.ParseException;
+import com.nimbusds.oauth2.sdk.Scope;
+import com.nimbusds.oauth2.sdk.TokenRequest;
+import com.nimbusds.oauth2.sdk.TokenResponse;
+import com.nimbusds.oauth2.sdk.auth.ClientAuthentication;
+import com.nimbusds.oauth2.sdk.auth.ClientSecretBasic;
+import com.nimbusds.oauth2.sdk.auth.Secret;
+import com.nimbusds.oauth2.sdk.id.ClientID;
+import com.nimbusds.oauth2.sdk.id.Issuer;
+import com.nimbusds.oauth2.sdk.id.State;
+import com.nimbusds.oauth2.sdk.pkce.CodeVerifier;
+import com.nimbusds.oauth2.sdk.token.AccessToken;
+import com.nimbusds.openid.connect.sdk.op.OIDCProviderMetadata;
+import org.json.JSONObject;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** Implementation of OAuth2 authentication.
+ *******************************************************************************/
+public class OAuth2AuthenticationModule implements QAuthenticationModuleInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(OAuth2AuthenticationModule.class);
+
+   private static boolean mayMemoize = true;
+
+   private static final Memoization<String, String> getAccessTokenFromSessionUUIDMemoization = new Memoization<String, String>()
+      .withTimeout(Duration.of(1, ChronoUnit.MINUTES))
+      .withMaxSize(1000);
+
+   private static final Memoization<String, OIDCProviderMetadata> oidcProviderMetadataMemoization = new Memoization<String, OIDCProviderMetadata>()
+      .withMayStoreNullValues(false);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QSession createSession(QInstance qInstance, Map<String, String> context) throws QAuthenticationException
+   {
+      try
+      {
+         OAuth2AuthenticationMetaData oauth2MetaData = (OAuth2AuthenticationMetaData) qInstance.getAuthentication();
+
+         if(context.containsKey("code") && context.containsKey("state"))
+         {
+            ///////////////////////////////////////////////////////////////////////
+            // handle a callback to initially auth a user for a traditional      //
+            // (non-js) site - where the code & state params come to the backend //
+            ///////////////////////////////////////////////////////////////////////
+            AuthorizationCode code = new AuthorizationCode(context.get("code"));
+
+            /////////////////////////////////////////
+            // verify the state in our state table //
+            /////////////////////////////////////////
+            AtomicReference<String> redirectUri = new AtomicReference<>(null);
+            QContext.withTemporaryContext(new CapturedContext(qInstance, new QSystemUserSession()), () ->
+            {
+               QRecord redirectStateRecord = GetAction.execute(oauth2MetaData.getRedirectStateTableName(), Map.of("state", context.get("state")));
+               if(redirectStateRecord == null)
+               {
+                  throw (new QAuthenticationException("State not found"));
+               }
+               redirectUri.set(redirectStateRecord.getValueString("redirectUri"));
+            });
+
+            URI                    redirectURI       = new URI(redirectUri.get());
+            ClientSecretBasic      clientSecretBasic = new ClientSecretBasic(new ClientID(oauth2MetaData.getClientId()), new Secret(oauth2MetaData.getClientSecret()));
+            AuthorizationCodeGrant codeGrant         = new AuthorizationCodeGrant(code, redirectURI);
+
+            URI          tokenEndpoint = getOIDCProviderMetadata(oauth2MetaData).getTokenEndpointURI();
+            Scope        scope         = new Scope(oauth2MetaData.getScopes());
+            TokenRequest tokenRequest  = new TokenRequest(tokenEndpoint, clientSecretBasic, codeGrant, scope);
+
+            return createSessionFromTokenRequest(tokenRequest);
+         }
+         else if(context.containsKey("code") && context.containsKey("redirectUri") && context.containsKey("codeVerifier"))
+         {
+            ////////////////////////////////////////////////////////////////////////////////
+            // handle a call down to this backend code to initially auth a user for an    //
+            // SPA that received a code (where the javascript generated the codeVerifier) //
+            ////////////////////////////////////////////////////////////////////////////////
+            AuthorizationCode  code         = new AuthorizationCode(context.get("code"));
+            URI                callback     = new URI(context.get("redirectUri"));
+            CodeVerifier       codeVerifier = new CodeVerifier(context.get("codeVerifier"));
+            AuthorizationGrant codeGrant    = new AuthorizationCodeGrant(code, callback, codeVerifier);
+
+            ClientID             clientID     = new ClientID(oauth2MetaData.getClientId());
+            Secret               clientSecret = new Secret(oauth2MetaData.getClientSecret());
+            ClientAuthentication clientAuth   = new ClientSecretBasic(clientID, clientSecret);
+
+            URI          tokenEndpoint = getOIDCProviderMetadata(oauth2MetaData).getTokenEndpointURI();
+            Scope        scope         = new Scope(oauth2MetaData.getScopes());
+            TokenRequest tokenRequest  = new TokenRequest(tokenEndpoint, clientAuth, codeGrant, scope);
+
+            return createSessionFromTokenRequest(tokenRequest);
+         }
+         else if(context.containsKey("sessionUUID") || context.containsKey("sessionId") || context.containsKey("uuid"))
+         {
+            //////////////////////////////////////////////////////////////////////
+            // handle a "normal" request, where we aren't opening a new session //
+            // per-se, but instead are looking for one in our userSession table //
+            //////////////////////////////////////////////////////////////////////
+            String uuid = Objects.requireNonNullElseGet(context.get("sessionUUID"), () ->
+               Objects.requireNonNullElseGet(context.get("sessionId"), () ->
+                  context.get("uuid")));
+
+            String   accessToken = getAccessTokenFromSessionUUID(uuid);
+            QSession session     = createSessionFromToken(accessToken);
+            session.setUuid(uuid);
+
+            //////////////////////////////////////////////////////////////////
+            // todo - do we need to validate its age or ping the provider?? //
+            //////////////////////////////////////////////////////////////////
+
+            return (session);
+         }
+         else
+         {
+            String message = "Did not receive recognized values in context for creating session";
+            LOG.warn(message, logPair("contextKeys", context.keySet()));
+            throw (new QAuthenticationException(message));
+         }
+      }
+      catch(QAuthenticationException qae)
+      {
+         throw (qae);
+      }
+      catch(Exception e)
+      {
+         throw (new QAuthenticationException("Failed to create session (token)", e));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private QSession createSessionFromTokenRequest(TokenRequest tokenRequest) throws ParseException, IOException, QException
+   {
+      TokenResponse tokenResponse = TokenResponse.parse(tokenRequest.toHTTPRequest().send());
+
+      if(tokenResponse.indicatesSuccess())
+      {
+         AccessToken accessToken = tokenResponse.toSuccessResponse().getTokens().getAccessToken();
+
+         ////////////////////////////////////////////////////////////////////
+         // todo - do we want to try to do anything with a refresh token?? //
+         ////////////////////////////////////////////////////////////////////
+         // RefreshToken refreshToken = tokenResponse.toSuccessResponse().getTokens().getRefreshToken();
+
+         QSession session = createSessionFromToken(accessToken.getValue());
+         insertUserSession(accessToken.getValue(), session);
+         return (session);
+      }
+      else
+      {
+         ErrorObject errorObject = tokenResponse.toErrorResponse().getErrorObject();
+         LOG.info("Token request failed", logPair("code", errorObject.getCode()), logPair("description", errorObject.getDescription()));
+         throw (new QAuthenticationException(errorObject.getDescription()));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public boolean isSessionValid(QInstance instance, QSession session)
+   {
+      if(session instanceof QSystemUserSession)
+      {
+         return (true);
+      }
+
+      try
+      {
+         String     accessToken = getAccessTokenFromSessionUUID(session.getUuid());
+         DecodedJWT jwt         = JWT.decode(accessToken);
+         if(jwt.getExpiresAtAsInstant().isBefore(Instant.now()))
+         {
+            LOG.debug("accessToken is expired", logPair("sessionUUID", session.getUuid()));
+            return (false);
+         }
+
+         return true;
+      }
+      catch(QAuthenticationException e)
+      {
+         return (false);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getLoginRedirectUrl(String originalUrl) throws QAuthenticationException
+   {
+      try
+      {
+         QInstance                    qInstance      = QContext.getQInstance();
+         OAuth2AuthenticationMetaData oauth2MetaData = (OAuth2AuthenticationMetaData) qInstance.getAuthentication();
+         String                       authUrl        = getOIDCProviderMetadata(oauth2MetaData).getAuthorizationEndpointURI().toString();
+
+         QTableMetaData stateTable = QContext.getQInstance().getTable(oauth2MetaData.getRedirectStateTableName());
+         if(stateTable == null)
+         {
+            throw (new QAuthenticationException("The table specified as the oauthRedirectStateTableName [" + oauth2MetaData.getRedirectStateTableName() + "] is not defined in the QInstance"));
+         }
+
+         ///////////////////////////////////////////////////////////////////
+         // generate a secure state, of either default length (32 bytes), //
+         // or at a size (base64 encoded) that fits in the state table    //
+         ///////////////////////////////////////////////////////////////////
+         Integer stateStringLength = stateTable.getField("state").getMaxLength();
+         State   state             = stateStringLength == null ? new State(32) : new State((stateStringLength / 4) * 3);
+         String  stateValue        = state.getValue();
+
+         /////////////////////////////
+         // insert the state record //
+         /////////////////////////////
+         QContext.withTemporaryContext(new CapturedContext(qInstance, new QSystemUserSession()), () ->
+         {
+            QRecord insertedState = new InsertAction().execute(new InsertInput(oauth2MetaData.getRedirectStateTableName()).withRecord(new QRecord()
+               .withValue("state", stateValue)
+               .withValue("redirectUri", originalUrl))).getRecords().get(0);
+            if(CollectionUtils.nullSafeHasContents(insertedState.getErrors()))
+            {
+               throw (new QAuthenticationException("Error storing redirect state: " + insertedState.getErrorsAsString()));
+            }
+         });
+
+         return authUrl
+            + "?client_id=" + URLEncoder.encode(oauth2MetaData.getClientId(), StandardCharsets.UTF_8)
+            + "&redirect_uri=" + URLEncoder.encode(originalUrl, StandardCharsets.UTF_8)
+            + "&response_type=code"
+            + "&scope=" + URLEncoder.encode(oauth2MetaData.getScopes(), StandardCharsets.UTF_8)
+            + "&state=" + URLEncoder.encode(state.getValue(), StandardCharsets.UTF_8);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error getting login redirect url", e);
+         throw (new QAuthenticationException("Error getting login redirect url", e));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private QSession createSessionFromToken(String accessToken) throws QException
+   {
+      DecodedJWT     jwt           = JWT.decode(accessToken);
+      Base64.Decoder decoder       = Base64.getUrlDecoder();
+      String         payloadString = new String(decoder.decode(jwt.getPayload()));
+      JSONObject     payload       = new JSONObject(payloadString);
+
+      QSession session = new QSession();
+      QUser    user    = new QUser();
+      session.setUser(user);
+
+      user.setFullName("Unknown");
+      String email = Objects.requireNonNullElseGet(payload.optString("email", null), () -> payload.optString("sub", null));
+      String name  = payload.optString("name", email);
+
+      user.setIdReference(email);
+      user.setFullName(name);
+
+      ////////////////////////////////////////////////////////////
+      // todo wip - this needs to be much better standardized w/ fe //
+      ////////////////////////////////////////////////////////////
+      session.withValueForFrontend("user", new HashMap<>(Map.of("name", name, "email", email)));
+
+      return session;
+   }
+
+
+
+   /*******************************************************************************
+    ** Insert a session as a new record into userSession table
+    *******************************************************************************/
+   private void insertUserSession(String accessToken, QSession qSession) throws QException
+   {
+      CapturedContext capturedContext = QContext.capture();
+      try
+      {
+         QContext.init(capturedContext.qInstance(), new QSystemUserSession());
+
+         UserSession userSession = new UserSession()
+            .withUuid(qSession.getUuid())
+            .withUserId(qSession.getUser().getIdReference())
+            .withAccessToken(accessToken);
+
+         new InsertAction().execute(new InsertInput(UserSession.TABLE_NAME).withRecordEntity(userSession));
+      }
+      finally
+      {
+         QContext.init(capturedContext);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QSession createAutomatedSessionForUser(QInstance qInstance, Serializable userId) throws QAuthenticationException
+   {
+      return QAuthenticationModuleInterface.super.createAutomatedSessionForUser(qInstance, userId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Look up access_token from session UUID
+    **
+    *******************************************************************************/
+   private String getAccessTokenFromSessionUUID(String sessionUUID) throws QAuthenticationException
+   {
+      if(mayMemoize)
+      {
+         return getAccessTokenFromSessionUUIDMemoization.getResultThrowing(sessionUUID, (String x) ->
+            doGetAccessTokenFromSessionUUID(sessionUUID)
+         ).orElse(null);
+      }
+      else
+      {
+         return (doGetAccessTokenFromSessionUUID(sessionUUID));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private String doGetAccessTokenFromSessionUUID(String sessionUUID) throws QAuthenticationException
+   {
+      String   accessToken   = null;
+      QSession beforeSession = QContext.getQSession();
+
+      try
+      {
+         QContext.setQSession(new QSystemUserSession());
+
+         ///////////////////////////////////////
+         // query for the user session record //
+         ///////////////////////////////////////
+         QRecord userSessionRecord = new GetAction().executeForRecord(new GetInput(UserSession.TABLE_NAME)
+            .withUniqueKey(Map.of("uuid", sessionUUID))
+            .withShouldMaskPasswords(false)
+            .withShouldOmitHiddenFields(false));
+
+         if(userSessionRecord != null)
+         {
+            accessToken = userSessionRecord.getValueString("accessToken");
+
+            ////////////////////////////////////////////////////////////
+            // decode the accessToken and make sure it is not expired //
+            ////////////////////////////////////////////////////////////
+            if(accessToken != null)
+            {
+               DecodedJWT jwt = JWT.decode(accessToken);
+               if(jwt.getExpiresAtAsInstant().isBefore(Instant.now()))
+               {
+                  throw (new QAuthenticationException("accessToken is expired"));
+               }
+            }
+         }
+      }
+      catch(QAuthenticationException qae)
+      {
+         throw (qae);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error looking up userSession by sessionUUID", e);
+         throw (new QAuthenticationException("Error looking up userSession by sessionUUID", e));
+      }
+      finally
+      {
+         QContext.setQSession(beforeSession);
+      }
+
+      return (accessToken);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public boolean usesSessionIdCookie()
+   {
+      return (true);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private OIDCProviderMetadata getOIDCProviderMetadata(OAuth2AuthenticationMetaData oAuth2AuthenticationMetaData) throws GeneralException, IOException
+   {
+      return oidcProviderMetadataMemoization.getResult(oAuth2AuthenticationMetaData.getName(), (name ->
+      {
+         Issuer               issuer   = new Issuer(oAuth2AuthenticationMetaData.getBaseUrl());
+         OIDCProviderMetadata metadata = OIDCProviderMetadata.resolve(issuer);
+         return (metadata);
+      })).orElseThrow(() -> new GeneralException("Could not resolve OIDCProviderMetadata for " + oAuth2AuthenticationMetaData.getName()));
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/TableBasedAuthenticationModule.java

@@ -406,6 +406,7 @@ public class TableBasedAuthenticationModule implements QAuthenticationModuleInte
          qUser.setIdReference(userRecord.getValueString(metaData.getUserTableUsernameField()));
 
          QSession qSession = new QSession();
+         qSession.setUuid(sessionUuid);
          qSession.setIdReference(sessionUuid);
          qSession.setUser(qUser);
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/authentication/implementations/metadata/RedirectStateMetaDataProducer.java

@@ -0,0 +1,82 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2023.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.modules.authentication.implementations.metadata;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
+
+
+/*******************************************************************************
+ ** Meta Data Producer for RedirectState table
+ *******************************************************************************/
+public class RedirectStateMetaDataProducer extends MetaDataProducer<QTableMetaData>
+{
+   public static final String TABLE_NAME = "redirectState";
+
+   private final String backendName;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public RedirectStateMetaDataProducer(String backendName)
+   {
+      this.backendName = backendName;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public QTableMetaData produce(QInstance qInstance) throws QException
+   {
+      QTableMetaData tableMetaData = new QTableMetaData()
+         .withName(TABLE_NAME)
+         .withBackendName(backendName)
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("state")
+         .withPrimaryKeyField("id")
+         .withUniqueKey(new UniqueKey("state"))
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
+
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
+         .withField(new QFieldMetaData("state", QFieldType.STRING).withIsEditable(false).withMaxLength(45).withBehavior(ValueTooLongBehavior.ERROR))
+         .withField(new QFieldMetaData("redirectUri", QFieldType.STRING).withIsEditable(false).withMaxLength(4096).withBehavior(ValueTooLongBehavior.ERROR))
+         .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false));
+
+      return tableMetaData;
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertExtractStep.java

@@ -55,8 +55,6 @@ public class BulkInsertExtractStep extends AbstractExtractStep
    @Override
    public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
-      runBackendStepInput.traceMessage(BulkInsertStepUtils.getProcessTracerKeyRecordMessage(runBackendStepInput));
-
       int rowsAdded     = 0;
       int originalLimit = Objects.requireNonNullElse(getLimit(), Integer.MAX_VALUE);
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertLoadStep.java

@@ -41,6 +41,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.LoadViaInsertStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.general.ProcessSummaryWarningsAndErrorsRollup;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 
 
@@ -77,19 +78,77 @@ public class BulkInsertLoadStep extends LoadViaInsertStep implements ProcessSumm
 
       QTableMetaData table = QContext.getQInstance().getTable(runBackendStepInput.getValueString("tableName"));
 
+      /////////////////////////////////////////////////////////////////////////////////////////////
+      // the transform step builds summary lines that it predicts will insert successfully.      //
+      // but those lines don't have ids, which we'd like to have (e.g., for a process trace that //
+      // might link to the built record).  also, it's possible that there was a fail that only   //
+      // happened in the actual insert, so, basically, re-do the summary here                    //
+      /////////////////////////////////////////////////////////////////////////////////////////////
+      BulkInsertTransformStep transformStep = (BulkInsertTransformStep) getTransformStep();
+      ProcessSummaryLine      okSummary     = transformStep.okSummary;
+      okSummary.setCount(0);
+      okSummary.setPrimaryKeys(new ArrayList<>());
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // but - since errors from the transform step don't even make it through to us here in the load step,   //
+      // do re-use the ProcessSummaryWarningsAndErrorsRollup from transform step as follows:                  //
+      // clear out its warnings - we'll completely rebuild them here (with primary keys)                      //
+      // and add new error lines, e.g., in case of errors that only happened past the validation if possible. //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////
+      ProcessSummaryWarningsAndErrorsRollup processSummaryWarningsAndErrorsRollup = transformStep.processSummaryWarningsAndErrorsRollup;
+      processSummaryWarningsAndErrorsRollup.resetWarnings();
+
       List<QRecord> insertedRecords = runBackendStepOutput.getRecords();
       for(QRecord insertedRecord : insertedRecords)
       {
-         if(CollectionUtils.nullSafeIsEmpty(insertedRecord.getErrors()))
+         Serializable primaryKey = insertedRecord.getValue(table.getPrimaryKeyField());
+         if(CollectionUtils.nullSafeIsEmpty(insertedRecord.getErrors()) && primaryKey != null)
          {
+            /////////////////////////////////////////////////////////////////////////
+            // if the record had no errors, and we have a primary key for it, then //
+            // keep track of the range of primary keys (first and last)            //
+            /////////////////////////////////////////////////////////////////////////
             if(firstInsertedPrimaryKey == null)
             {
-               firstInsertedPrimaryKey = insertedRecord.getValue(table.getPrimaryKeyField());
+               firstInsertedPrimaryKey = primaryKey;
             }
 
-            lastInsertedPrimaryKey = insertedRecord.getValue(table.getPrimaryKeyField());
+            lastInsertedPrimaryKey = primaryKey;
+
+            if(!CollectionUtils.nullSafeIsEmpty(insertedRecord.getWarnings()))
+            {
+               /////////////////////////////////////////////////////////////////////////////
+               // if there were warnings on the inserted record, put it in a warning line //
+               /////////////////////////////////////////////////////////////////////////////
+               String message = insertedRecord.getWarnings().get(0).getMessage();
+               processSummaryWarningsAndErrorsRollup.addWarning(message, primaryKey);
+            }
+            else
+            {
+               ////////////////////////////////////////////////////////////////////////
+               // if no warnings for the inserted record, then put it in the OK line //
+               ////////////////////////////////////////////////////////////////////////
+               okSummary.incrementCountAndAddPrimaryKey(primaryKey);
+            }
+         }
+         else
+         {
+            //////////////////////////////////////////////////////////////////////
+            // else if there were errors or no primary key, build an error line //
+            //////////////////////////////////////////////////////////////////////
+            String message = "Failed to insert";
+            if(!CollectionUtils.nullSafeIsEmpty(insertedRecord.getErrors()))
+            {
+               //////////////////////////////////////////////////////////
+               // use the error message from the record if we have one //
+               //////////////////////////////////////////////////////////
+               message = insertedRecord.getErrors().get(0).getMessage();
+            }
+            processSummaryWarningsAndErrorsRollup.addError(message, primaryKey);
          }
       }
+
+      okSummary.pickMessage(true);
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertPrepareFileUploadStep.java

@@ -52,6 +52,11 @@ public class BulkInsertPrepareFileUploadStep implements BackendStep
    @Override
    public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
+      ////////////////////////////////////////////////////////////////////////////////////////
+      // for headless-bulk load (e.g., sftp import), set up the process tracer's key record //
+      ////////////////////////////////////////////////////////////////////////////////////////
+      runBackendStepInput.traceMessage(BulkInsertStepUtils.getProcessTracerKeyRecordMessage(runBackendStepInput));
+
       ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       // if user has come back here, clear out file (else the storageInput object that it is comes to the frontend, which isn't what we want!) //
       ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertReceiveFileMappingStep.java

@@ -46,6 +46,7 @@ import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mode
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 import org.apache.commons.lang3.BooleanUtils;
 
@@ -77,10 +78,14 @@ public class BulkInsertReceiveFileMappingStep implements BackendStep
             //////////////////////////////////////////////////////////////////////////////
             if(savedBulkLoadProfileRecord == null)
             {
-               throw (new QUserFacingException("Did not receive a saved bulk load profile record as input - unable to perform headless bulk load"));
+               throw (new QUserFacingException("Did not receive a Bulk Load Profile record as input.  Unable to perform headless bulk load"));
             }
 
             SavedBulkLoadProfile savedBulkLoadProfile = new SavedBulkLoadProfile(savedBulkLoadProfileRecord);
+            if(!StringUtils.hasContent(savedBulkLoadProfile.getMappingJson()))
+            {
+               throw (new QUserFacingException("Bulk Load Profile record's Mapping is empty.  Unable to perform headless bulk load"));
+            }
 
             try
             {
@@ -88,7 +93,7 @@ public class BulkInsertReceiveFileMappingStep implements BackendStep
             }
             catch(Exception e)
             {
-               throw (new QUserFacingException("Error processing saved bulk load profile record - unable to perform headless bulk load", e));
+               throw (new QUserFacingException("Error processing Bulk Load Profile record.  Unable to perform headless bulk load", e));
             }
          }
          else
@@ -240,6 +245,11 @@ public class BulkInsertReceiveFileMappingStep implements BackendStep
             }
          }
       }
+      catch(QUserFacingException ufe)
+      {
+         LOG.warn("User-facing error in bulk insert receive mapping", ufe);
+         throw ufe;
+      }
       catch(Exception e)
       {
          LOG.warn("Error in bulk insert receive mapping", e);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertTransformStep.java

@@ -75,9 +75,9 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  *******************************************************************************/
 public class BulkInsertTransformStep extends AbstractTransformStep
 {
-   private ProcessSummaryLine okSummary = new ProcessSummaryLine(Status.OK);
+   ProcessSummaryLine okSummary = new ProcessSummaryLine(Status.OK);
 
-   private ProcessSummaryWarningsAndErrorsRollup processSummaryWarningsAndErrorsRollup = ProcessSummaryWarningsAndErrorsRollup.build("inserted")
+   ProcessSummaryWarningsAndErrorsRollup processSummaryWarningsAndErrorsRollup = ProcessSummaryWarningsAndErrorsRollup.build("inserted")
       .withDoReplaceSingletonCountLinesWithSuffixOnly(false);
 
    private ListingHash<String, RowValue> errorToExampleRowValueMap = new ListingHash<>();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/general/ProcessSummaryWarningsAndErrorsRollup.java

@@ -195,7 +195,7 @@ public class ProcessSummaryWarningsAndErrorsRollup
             {
                if(otherWarningsSummary == null)
                {
-                  otherWarningsSummary = new ProcessSummaryLine(Status.WARNING).withMessageSuffix("records had an other warning.");
+                  otherWarningsSummary = buildOtherWarningsSummary();
                }
                processSummaryLine = otherWarningsSummary;
             }
@@ -214,6 +214,27 @@ public class ProcessSummaryWarningsAndErrorsRollup
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static ProcessSummaryLine buildOtherWarningsSummary()
+   {
+      return new ProcessSummaryLine(Status.WARNING).withMessageSuffix("records had an other warning.");
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public void resetWarnings()
+   {
+      warningSummaries.clear();
+      otherWarningsSummary = buildOtherWarningsSummary();
+   }
+
+
+
    /*******************************************************************************
     ** Wrapper around AlphaNumericComparator for ProcessSummaryLineInterface that
     ** extracts string messages out.

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/utils/RecordLookupHelper.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.processes.utils;
 
 
 import java.io.Serializable;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
@@ -51,13 +52,14 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  *******************************************************************************/
 public class RecordLookupHelper
 {
-   private Map<String, Map<Serializable, QRecord>> recordMaps = new HashMap<>();
+   private Map<String, Map<Serializable, QRecord>> recordMaps;
 
-   private Map<String, Map<Map<String, Serializable>, QRecord>> uniqueKeyMaps = new HashMap<>();
+   private Map<String, Map<Map<String, Serializable>, QRecord>> uniqueKeyMaps;
 
-   private Set<String> preloadedKeys = new HashSet<>();
+   private Set<String> preloadedKeys;
 
-   private Set<Pair<String, String>> disallowedOneOffLookups = new HashSet<>();
+   private Set<Pair<String, String>> disallowedOneOffLookups;
+   private boolean                   useSynchronizedCollections;
 
 
 
@@ -67,6 +69,33 @@ public class RecordLookupHelper
     *******************************************************************************/
    public RecordLookupHelper()
    {
+      this(false);
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public RecordLookupHelper(boolean useSynchronizedCollections)
+   {
+      this.useSynchronizedCollections = useSynchronizedCollections;
+
+      if(useSynchronizedCollections)
+      {
+         recordMaps = Collections.synchronizedMap(new HashMap<>());
+         uniqueKeyMaps = Collections.synchronizedMap(new HashMap<>());
+         preloadedKeys = Collections.synchronizedSet(new HashSet<>());
+         disallowedOneOffLookups = Collections.synchronizedSet(new HashSet<>());
+      }
+      else
+      {
+         recordMaps = new HashMap<>();
+         uniqueKeyMaps = new HashMap<>();
+         preloadedKeys = new HashSet<>();
+         disallowedOneOffLookups = new HashSet<>();
+      }
    }
 
 
@@ -77,7 +106,7 @@ public class RecordLookupHelper
    public QRecord getRecordByUniqueKey(String tableName, Map<String, Serializable> uniqueKey) throws QException
    {
       String                                  mapKey    = tableName + "." + uniqueKey.keySet().stream().sorted().collect(Collectors.joining(","));
-      Map<Map<String, Serializable>, QRecord> recordMap = uniqueKeyMaps.computeIfAbsent(mapKey, (k) -> new HashMap<>());
+      Map<Map<String, Serializable>, QRecord> recordMap = uniqueKeyMaps.computeIfAbsent(mapKey, (k) -> useSynchronizedCollections ? Collections.synchronizedMap(new HashMap<>()) : new HashMap<>());
 
       if(!recordMap.containsKey(uniqueKey))
       {
@@ -96,7 +125,7 @@ public class RecordLookupHelper
    public QRecord getRecordByKey(String tableName, String keyFieldName, Serializable key) throws QException
    {
       String                     mapKey    = tableName + "." + keyFieldName;
-      Map<Serializable, QRecord> recordMap = recordMaps.computeIfAbsent(mapKey, (k) -> new HashMap<>());
+      Map<Serializable, QRecord> recordMap = recordMaps.computeIfAbsent(mapKey, (k) -> useSynchronizedCollections ? Collections.synchronizedMap(new HashMap<>()) : new HashMap<>());
 
       ////////////////////////////////////////////////////////////
       // make sure we have they key object in the expected type //
@@ -150,7 +179,7 @@ public class RecordLookupHelper
    public void preloadRecords(String tableName, String keyFieldName, QQueryFilter filter) throws QException
    {
       String                     mapKey   = tableName + "." + keyFieldName;
-      Map<Serializable, QRecord> tableMap = recordMaps.computeIfAbsent(mapKey, s -> new HashMap<>());
+      Map<Serializable, QRecord> tableMap = recordMaps.computeIfAbsent(mapKey, s -> useSynchronizedCollections ? Collections.synchronizedMap(new HashMap<>()) : new HashMap<>());
       tableMap.putAll(GeneralProcessUtils.loadTableToMap(tableName, keyFieldName, filter));
    }
 
@@ -170,7 +199,7 @@ public class RecordLookupHelper
       }
 
       String                     mapKey   = tableName + "." + keyFieldName;
-      Map<Serializable, QRecord> tableMap = recordMaps.computeIfAbsent(mapKey, s -> new HashMap<>());
+      Map<Serializable, QRecord> tableMap = recordMaps.computeIfAbsent(mapKey, s -> useSynchronizedCollections ? Collections.synchronizedMap(new HashMap<>()) : new HashMap<>());
 
       QQueryFilter filter = new QQueryFilter(new QFilterCriteria(keyFieldName, QCriteriaOperator.IN, inList));
       tableMap.putAll(GeneralProcessUtils.loadTableToMap(tableName, keyFieldName, filter));

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/state/InMemoryStateProvider.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.state;
 import java.io.Serializable;
 import java.time.Instant;
 import java.time.temporal.ChronoUnit;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
 import java.util.Optional;
@@ -58,7 +59,7 @@ public class InMemoryStateProvider implements StateProviderInterface
     *******************************************************************************/
    private InMemoryStateProvider()
    {
-      this.map = new HashMap<>();
+      this.map = Collections.synchronizedMap(new HashMap<>());
 
       ///////////////////////////////////////////////////////////
       // Start a single thread executor to handle the cleaning //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/ClassPathUtils.java

@@ -61,6 +61,39 @@ public class ClassPathUtils
 
 
 
+   /*******************************************************************************
+    ** from https://stackoverflow.com/questions/520328/can-you-find-all-classes-in-a-package-using-reflection
+    **
+    *******************************************************************************/
+   public static List<Class<?>> getClassesContainingNameAndOfType(String nameContains, Class<?> type) throws IOException
+   {
+      List<Class<?>> classes = new ArrayList<>();
+      ClassLoader    loader  = Thread.currentThread().getContextClassLoader();
+
+      for(ClassPath.ClassInfo info : getTopLevelClasses(loader))
+      {
+         try
+         {
+            if(info.getName().contains(nameContains))
+            {
+               Class<?> testClass = info.load();
+               if(type.isAssignableFrom(testClass))
+               {
+                  classes.add(testClass);
+               }
+            }
+         }
+         catch(Throwable t)
+         {
+            // ignore - comes up for non-class entries, like module-info
+         }
+      }
+
+      return (classes);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/ReflectiveBeanLikeClassUtils.java

@@ -0,0 +1,188 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.utils;
+
+
+import java.lang.reflect.AnnotatedParameterizedType;
+import java.lang.reflect.AnnotatedType;
+import java.lang.reflect.Method;
+import java.lang.reflect.Type;
+import java.math.BigDecimal;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalTime;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Optional;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.data.QIgnore;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/*******************************************************************************
+ ** Utilities for bean-like classes (e.g., QRecordEntity, QProcessPayload) that
+ ** use reflection to understand their bean-fields
+ *******************************************************************************/
+public class ReflectiveBeanLikeClassUtils
+{
+   private static final QLogger LOG = QLogger.getLogger(ReflectiveBeanLikeClassUtils.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String getFieldNameFromGetter(Method getter)
+   {
+      String nameWithoutGet = getter.getName().replaceFirst("^get", "");
+      if(nameWithoutGet.length() == 1)
+      {
+         return (nameWithoutGet.toLowerCase(Locale.ROOT));
+      }
+      return (nameWithoutGet.substring(0, 1).toLowerCase(Locale.ROOT) + nameWithoutGet.substring(1));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static boolean isGetter(Method method, boolean allowAssociations)
+   {
+      return isGetter(method, allowAssociations, defaultAllowedTypes());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static boolean isGetter(Method method, boolean allowAssociations, Collection<Class<?>> allowedTypes)
+   {
+      if(method.getParameterTypes().length == 0 && method.getName().matches("^get[A-Z].*"))
+      {
+         if(allowedTypes.contains(method.getReturnType()) || (allowAssociations && isSupportedAssociation(method.getReturnType(), method.getAnnotatedReturnType())))
+         {
+            return (true);
+         }
+         else
+         {
+            if(!method.getName().equals("getClass") && method.getAnnotation(QIgnore.class) == null)
+            {
+               LOG.debug("Method [" + method.getName() + "] in [" + method.getDeclaringClass().getSimpleName() + "] looks like a getter, but its return type, [" + method.getReturnType().getSimpleName() + "], isn't supported.");
+            }
+         }
+      }
+      return (false);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static Optional<Method> getSetterForGetter(Class<?> c, Method getter)
+   {
+      String setterName = getter.getName().replaceFirst("^get", "set");
+      for(Method method : c.getMethods())
+      {
+         if(method.getName().equals(setterName))
+         {
+            if(method.getParameterTypes().length == 1 && method.getParameterTypes()[0].equals(getter.getReturnType()))
+            {
+               return (Optional.of(method));
+            }
+            else
+            {
+               LOG.info("Method [" + method.getName() + "] looks like a setter for [" + getter.getName() + "], but its parameters, [" + Arrays.toString(method.getParameterTypes()) + "], don't match the getter's return type [" + getter.getReturnType() + "]");
+            }
+         }
+      }
+      return (Optional.empty());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static Collection<Class<?>> defaultAllowedTypes()
+   {
+      /////////////////////////////////////////////
+      // note - this list has implications upon: //
+      // - QFieldType.fromClass                  //
+      // - QRecordEntityField.convertValueType   //
+      /////////////////////////////////////////////
+      return (Set.of(String.class,
+         Integer.class,
+         Long.class,
+         int.class,
+         Boolean.class,
+         boolean.class,
+         BigDecimal.class,
+         Instant.class,
+         LocalDate.class,
+         LocalTime.class,
+         byte[].class));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static boolean isSupportedAssociation(Class<?> returnType, AnnotatedType annotatedType)
+   {
+      Class<?> listTypeParam = getListTypeParam(returnType, annotatedType);
+      return (listTypeParam != null && QRecordEntity.class.isAssignableFrom(listTypeParam));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static Class<?> getListTypeParam(Class<?> listType, AnnotatedType annotatedType)
+   {
+      if(listType.equals(List.class))
+      {
+         if(annotatedType instanceof AnnotatedParameterizedType apt)
+         {
+            AnnotatedType[] annotatedActualTypeArguments = apt.getAnnotatedActualTypeArguments();
+            for(AnnotatedType annotatedActualTypeArgument : annotatedActualTypeArguments)
+            {
+               Type type = annotatedActualTypeArgument.getType();
+               if(type instanceof Class<?> c)
+               {
+                  return (c);
+               }
+            }
+         }
+      }
+
+      return (null);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/StringUtils.java

@@ -23,9 +23,11 @@ package com.kingsrook.qqq.backend.core.utils;
 
 
 import java.util.Collection;
+import java.util.Collections;
 import java.util.List;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
+import com.ibm.icu.text.Transliterator;
 
 
 /*******************************************************************************
@@ -462,6 +464,17 @@ public class StringUtils
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String replaceNonAsciiCharacters(String s)
+   {
+      Transliterator transliterator = Transliterator.getInstance("Any-Latin; Latin-ASCII");
+      return (transliterator.transliterate(s));
+   }
+
+
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -477,6 +490,24 @@ public class StringUtils
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static boolean safeEqualsIgnoreCase(String a, String b)
+   {
+      if(a == null && b == null)
+      {
+         return true;
+      }
+      if(a == null || b == null)
+      {
+         return false;
+      }
+      return (a.equalsIgnoreCase(b));
+   }
+
+
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -502,4 +533,59 @@ public class StringUtils
          return base + " (1)";
       }
    }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String maskAndTruncate(String value)
+   {
+      return (maskAndTruncate(value, "** MASKED **", 6, 4));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String maskAndTruncate(String value, String mask, int minLengthToMask, int charsToShowOnEnds)
+   {
+      if(!hasContent(value))
+      {
+         return ("");
+      }
+
+      if(value.length() < minLengthToMask || value.length() < 2 * charsToShowOnEnds)
+      {
+         return mask;
+      }
+
+      if(value.length() < charsToShowOnEnds * 3)
+      {
+         return (value.substring(0, charsToShowOnEnds) + mask);
+      }
+
+      return (value.substring(0, charsToShowOnEnds) + mask + value.substring(value.length() - charsToShowOnEnds));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String nCopies(int n, String s)
+   {
+      return (nCopiesWithGlue(n, s, ""));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String nCopiesWithGlue(int n, String s, String glue)
+   {
+      return (StringUtils.join(glue, Collections.nCopies(n, s)));
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/Timer.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.utils;
 
 
+import java.time.Duration;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import org.apache.logging.log4j.Level;
 
@@ -79,9 +80,36 @@ public class Timer
     **
     *******************************************************************************/
    public void mark(String message)
+   {
+      mark(message, false);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void mark(String message, boolean prettyPrint)
    {
       long now = System.currentTimeMillis();
-      LOG.log(level, String.format("%s: Last [%5d] Total [%5d] %s", name, (now - last), (now - start), message));
+
+      if(!prettyPrint)
+      {
+         LOG.log(level, String.format("%s: Last [%5d] Total [%5d] %s", name, (now - last), (now - start), message));
+      }
+      else
+      {
+
+         Duration lastDuration  = Duration.ofMillis(now - last);
+         Duration totalDuration = Duration.ofMillis(now - start);
+
+         LOG.log(level, String.format(
+            "%s: Last [%d hours, %d minutes, %d seconds, %d milliseconds] Total [%d hours, %d minutes, %d seconds, %d milliseconds] %s",
+            name, lastDuration.toHours(), lastDuration.toMinutesPart(), lastDuration.toSecondsPart(), lastDuration.toMillisPart(),
+            totalDuration.toHours(), totalDuration.toMinutesPart(), totalDuration.toSecondsPart(), totalDuration.toMillisPart(),
+            message));
+      }
+
       last = now;
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/ValueUtils.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.utils;
 
 
+import java.io.IOException;
 import java.io.Serializable;
 import java.math.BigDecimal;
 import java.math.BigInteger;
@@ -38,6 +39,7 @@ import java.time.temporal.ChronoField;
 import java.time.temporal.ChronoUnit;
 import java.util.Calendar;
 import java.util.List;
+import java.util.Map;
 import java.util.TimeZone;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QValueException;
@@ -1012,4 +1014,37 @@ public class ValueUtils
 
       return defaultIfCannotInfer;
    }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static Map getValueAsMap(Serializable value)
+   {
+      if(value == null)
+      {
+         return (null);
+      }
+      else if(value instanceof Map<?, ?> map)
+      {
+         return (map);
+      }
+      else if(value instanceof String string && string.startsWith("{") && string.endsWith("}"))
+      {
+         try
+         {
+            Map map = JsonUtils.toObject(string, Map.class);
+            return (map);
+         }
+         catch(IOException e)
+         {
+            throw new QValueException("Error parsing string to map", e);
+         }
+      }
+      else
+      {
+         throw new QValueException("Unrecognized object type in getValueAsMap: " + value.getClass().getSimpleName());
+      }
+   }
 }

qqq-backend-core/src/main/resources/log4j2.xml

@@ -24,6 +24,7 @@
       <!-- c3p0 -->
       <Logger name="com.mchange.v2" level="INFO" />
       <Logger name="org.quartz" level="INFO" />
+      <Logger name="org.apache.http" level="INFO"/>
       <Logger name="liquibase" level="INFO" />
       <Logger name="com.amazonaws" level="INFO" />
       <Root level="all">

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/templates/ConvertHtmlToPdfActionTest.java

@@ -79,6 +79,7 @@ class ConvertHtmlToPdfActionTest extends BaseTest
                      </h1>
                      <div class="myclass">
                          <p>This is a test of converting HTML to PDF!!</p>
+                         <p>This is &nbsp; a line with &bull; some entities &lt;</p>
                          <p style="font-family: SF-Pro; font-size: 24px;">(btw, is this in SF-Pro???)</p>
                      </div>
                  </div>

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/AbstractMetaDataProducerBasedQQQApplicationTest.java

@@ -43,6 +43,9 @@ class AbstractMetaDataProducerBasedQQQApplicationTest extends BaseTest
    {
       QInstance qInstance = new TestApplication().defineQInstance();
       assertEquals(1, qInstance.getTables().size());
+      assertEquals("fromProducer", qInstance.getTables().get("fromProducer").getName());
+      assertEquals(1, qInstance.getProcesses().size());
+      assertEquals("fromProducer", qInstance.getProcesses().get("fromProducer").getName());
    }
 
 

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/MetaDataProducerBasedQQQApplicationTest.java

@@ -0,0 +1,66 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.instances.producers.TestMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for MetaDataProducerBasedQQQApplication 
+ *******************************************************************************/
+class MetaDataProducerBasedQQQApplicationTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      QInstance qInstance = new MetaDataProducerBasedQQQApplication(getClass().getPackage().getName() + ".producers").defineQInstance();
+      assertEquals(1, qInstance.getTables().size());
+      assertEquals("fromProducer", qInstance.getTables().get("fromProducer").getName());
+      assertEquals(1, qInstance.getProcesses().size());
+      assertEquals("fromProducer", qInstance.getProcesses().get("fromProducer").getName());
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testConstructorThatTakeClass() throws QException
+   {
+      QInstance qInstance = new MetaDataProducerBasedQQQApplication(TestMetaDataProducer.class).defineQInstance();
+      assertEquals(1, qInstance.getTables().size());
+      assertEquals("fromProducer", qInstance.getTables().get("fromProducer").getName());
+      assertEquals(1, qInstance.getProcesses().size());
+      assertEquals("fromProducer", qInstance.getProcesses().get("fromProducer").getName());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/AbstractMetaDataLoaderTest.java

@@ -0,0 +1,136 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.GenericMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.QTableMetaDataLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for AbstractMetaDataLoader 
+ *******************************************************************************/
+class AbstractMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testVariousPropertyTypes() throws QMetaDataLoaderException
+   {
+      QProcessMetaData process = new GenericMetaDataLoader<>(QProcessMetaData.class).fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1
+         name: myProcess
+         tableName: someTable
+         maxInputRecords: 1
+         isHidden: true
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertEquals("myProcess", process.getName());
+      assertEquals("someTable", process.getTableName());
+      assertEquals(1, process.getMaxInputRecords());
+      assertTrue(process.getIsHidden());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testProblems() throws QMetaDataLoaderException
+   {
+      {
+         QTableMetaDataLoader loader = new QTableMetaDataLoader();
+         loader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+            class: QTableMetaData
+            version: 1.0
+            name: myTable
+            something: foo
+            isHidden: hi
+            icon:
+               name: account_tree
+               size: big
+               weight: bold
+            fields:
+               id:
+                  type: number
+            uniqueKeys: sure!
+            """, StandardCharsets.UTF_8), "myTable.yaml");
+
+         for(LoadingProblem problem : loader.getProblems())
+         {
+            System.out.println(problem);
+         }
+      }
+
+      {
+         GenericMetaDataLoader<QProcessMetaData> loader = new GenericMetaDataLoader<>(QProcessMetaData.class);
+         loader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+            class: QProcessMetaData
+            version: 1.0
+            name: myProcess
+            maxInputRecords: many
+            """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+         for(LoadingProblem problem : loader.getProblems())
+         {
+            System.out.println(problem);
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testEnvironmentValues() throws QMetaDataLoaderException
+   {
+      System.setProperty("myProcess.tableName", "someTable");
+      System.setProperty("myProcess.maxInputRecords", "47");
+
+      GenericMetaDataLoader<QProcessMetaData> loader = new GenericMetaDataLoader<>(QProcessMetaData.class);
+      QProcessMetaData processMetaData = loader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1.0
+         name: myProcess
+         tableName: ${prop.myProcess.tableName}
+         maxInputRecords: ${prop.myProcess.maxInputRecords}
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertEquals("someTable", processMetaData.getTableName());
+      assertEquals(47, processMetaData.getMaxInputRecords());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/ClassDetectingMetaDataLoaderTest.java

@@ -0,0 +1,115 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for ClassDetectingMetaDataLoader 
+ *******************************************************************************/
+class ClassDetectingMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testBasicSuccess() throws QMetaDataLoaderException
+   {
+      QMetaDataObject qMetaDataObject = new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QTableMetaData
+         version: 1
+         name: myTable
+         backendName: someBackend
+         """, StandardCharsets.UTF_8), "myTable.yaml");
+
+      assertThat(qMetaDataObject).isInstanceOf(QTableMetaData.class);
+      QTableMetaData qTableMetaData = (QTableMetaData) qMetaDataObject;
+      assertEquals("myTable", qTableMetaData.getName());
+      assertEquals("someBackend", qTableMetaData.getBackendName());
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testProcess() throws QMetaDataLoaderException
+   {
+      QMetaDataObject qMetaDataObject = new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1
+         name: myProcess
+         tableName: someTable
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertThat(qMetaDataObject).isInstanceOf(QProcessMetaData.class);
+      QProcessMetaData qProcessMetaData = (QProcessMetaData) qMetaDataObject;
+      assertEquals("myProcess", qProcessMetaData.getName());
+      assertEquals("someTable", qProcessMetaData.getTableName());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testUnknownClassFails()
+   {
+      assertThatThrownBy(() -> new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: ya whatever
+         version: 1
+         name: myTable
+         """, StandardCharsets.UTF_8), "whatever.yaml"))
+         .isInstanceOf(QMetaDataLoaderException.class)
+         .hasMessageContaining("Unexpected class");
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testMissingClassAttributeFails()
+   {
+      assertThatThrownBy(() -> new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         version: 1
+         name: myTable
+         """, StandardCharsets.UTF_8), "aTable.yaml"))
+         .isInstanceOf(QMetaDataLoaderException.class)
+         .hasMessageContaining("[class] attribute was not specified");
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/MetaDataLoaderHelperTest.java

@@ -0,0 +1,111 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import org.apache.commons.io.FileUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for MetaDataLoaderHelper 
+ *******************************************************************************/
+class MetaDataLoaderHelperTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws Exception
+   {
+      Path tempDirectory = Files.createTempDirectory(getClass().getSimpleName());
+
+      writeFile("myTable", ".yaml", tempDirectory, """
+         class: QTableMetaData
+         version:  1
+         name: myTable
+         label: This is My Table
+         primaryKeyField:  id
+         fields:
+            id:
+               name: id
+               type: INTEGER
+            name:
+               name: name
+               type: STRING
+            createDate:
+               name: createDate
+               type: DATE_TIME
+         """);
+
+      writeFile("yourTable", ".yaml", tempDirectory, """
+         class: QTableMetaData
+         version:  1
+         name: yourTable
+         label: Someone else's table
+         primaryKeyField:  id
+         fields:
+            id:
+               name: id
+               type: INTEGER
+            name:
+               name: name
+               type: STRING
+         """);
+
+      QInstance qInstance = new QInstance();
+      MetaDataLoaderHelper.processAllMetaDataFilesInDirectory(qInstance, tempDirectory.toFile().getAbsolutePath());
+
+      assertEquals(2, qInstance.getTables().size());
+
+      QTableMetaData myTable = qInstance.getTable("myTable");
+      assertEquals("This is My Table", myTable.getLabel());
+      assertEquals(3, myTable.getFields().size());
+      assertEquals("id", myTable.getField("id").getName());
+      assertEquals(QFieldType.INTEGER, myTable.getField("id").getType());
+
+      QTableMetaData yourTable = qInstance.getTable("yourTable");
+      assertEquals("Someone else's table", yourTable.getLabel());
+      assertEquals(2, yourTable.getFields().size());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   void writeFile(String prefix, String suffix, Path directory, String content) throws IOException
+   {
+      FileUtils.writeStringToFile(File.createTempFile(prefix, suffix, directory.toFile()), content, StandardCharsets.UTF_8);
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/QProcessMetaDataLoaderTest.java

@@ -0,0 +1,103 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+
+/*******************************************************************************
+ ** Unit test for loading a QProcessMetaData (doesn't need its own loader yet,
+ ** but is still a valuable high-level test target).
+ *******************************************************************************/
+class QProcessMetaDataLoaderTest extends BaseTest
+{
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testYaml() throws QMetaDataLoaderException
+   {
+      ClassDetectingMetaDataLoader metaDataLoader = new ClassDetectingMetaDataLoader();
+      QProcessMetaData process = (QProcessMetaData) metaDataLoader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class:  QProcessMetaData
+         version:  1.0
+         name:  myProcess
+         stepList:
+         - name: myBackendStep
+           stepType: backend
+           code:
+             name: com.kingsrook.test.processes.MyBackendStep
+         - name: myFrontendStep
+           stepType: frontend
+           components:
+             - type: HELP_TEXT
+               values:
+                  foo: bar
+             - type: VIEW_FORM
+           viewFields:
+             - name: myField
+               type: STRING
+             - name: yourField
+               type: DATE
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      CollectionUtils.nonNullList(metaDataLoader.getProblems()).forEach(System.out::println);
+
+      assertEquals("myProcess", process.getName());
+      assertEquals(2, process.getAllSteps().size());
+
+      QBackendStepMetaData myBackendStep = process.getBackendStep("myBackendStep");
+      assertNotNull(myBackendStep, "myBackendStep should not be null");
+      // todo - propagate this? assertEquals("myBackendStep", myBackendStep.getName());
+      assertEquals("com.kingsrook.test.processes.MyBackendStep", myBackendStep.getCode().getName());
+
+      QFrontendStepMetaData myFrontendStep = process.getFrontendStep("myFrontendStep");
+      assertNotNull(myFrontendStep, "myFrontendStep should not be null");
+      assertEquals(2, myFrontendStep.getComponents().size());
+      assertEquals(QComponentType.HELP_TEXT, myFrontendStep.getComponents().get(0).getType());
+      assertEquals(Map.of("foo", "bar"), myFrontendStep.getComponents().get(0).getValues());
+      assertEquals(QComponentType.VIEW_FORM, myFrontendStep.getComponents().get(1).getType());
+
+      assertEquals(2, myFrontendStep.getViewFields().size());
+      assertEquals("myField", myFrontendStep.getViewFields().get(0).getName());
+      assertEquals(QFieldType.STRING, myFrontendStep.getViewFields().get(0).getType());
+      assertEquals("yourField", myFrontendStep.getViewFields().get(1).getName());
+      assertEquals(QFieldType.DATE, myFrontendStep.getViewFields().get(1).getType());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/QTableMetaDataLoaderTest.java

@@ -0,0 +1,202 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.QTableMetaDataLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.permissions.DenyBehavior;
+import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import com.kingsrook.qqq.backend.core.utils.YamlUtils;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+
+/*******************************************************************************
+ ** Unit test for QTableMetaDataLoader 
+ *******************************************************************************/
+class QTableMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   @Disabled("Not quite yet passing - is a good goal to get to though!")
+   void testToYaml() throws QMetaDataLoaderException
+   {
+      QTableMetaData expectedTable = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON_MEMORY);
+      String expectedYaml = YamlUtils.toYaml(expectedTable);
+      QTableMetaData actualTable = new QTableMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream(expectedYaml, StandardCharsets.UTF_8), "person.yaml");
+      String actualYaml = YamlUtils.toYaml(actualTable);
+      assertEquals(expectedYaml, actualYaml);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testYaml() throws QMetaDataLoaderException
+   {
+      QTableMetaData table = new QTableMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QTableMetaData
+         version: 1.0
+         name: myTable
+         icon:
+            name: account_tree
+         fields:
+            id:
+               name: id
+               type: INTEGER
+            name:
+               name: name
+               type: STRING
+         uniqueKeys:
+         -  label: Name
+            fieldNames:
+            - name
+         associations:
+         -  name: A1
+            associatedTableName: yourTable
+            joinName: myTableJoinYourTable
+         -  name: A2
+            associatedTableName: theirTable
+            joinName: myTableJoinTheirTable
+         permissionRules:
+            level: READ_WRITE_PERMISSIONS
+            denyBehavior: HIDDEN
+            permissionBaseName: myTablePermissions
+            customPermissionChecker:
+               name: com.kingsrook.SomeChecker
+               codeType: JAVA
+         ## todo recordSecurityLocks
+         ## todo auditRules
+         ## todo backendDetails
+         ## todo automationDetails
+         sections:
+         -  name: identity
+            label: Identity
+            icon:
+               name: badge
+            tier: T1
+            fieldNames:
+            -  id
+            -  firstName
+            -  lastName
+         customizers:
+            postQueryRecord:
+               name: com.kingsrook.SomePostQuery
+               codeType: JAVA
+            preDeleteRecord:
+               name: com.kingsrook.SomePreDelete
+               codeType: JAVA
+         disabledCapabilities:
+         -   TABLE_COUNT
+         -   QUERY_STATS
+         """, StandardCharsets.UTF_8), "myTable.yaml");
+
+      assertEquals("myTable", table.getName());
+
+      assertEquals(2, table.getFields().size());
+      // assertEquals("id", table.getFields().get("id").getName());
+      assertEquals(QFieldType.INTEGER, table.getFields().get("id").getType());
+      // assertEquals("name", table.getFields().get("name").getName());
+      assertEquals(QFieldType.STRING, table.getFields().get("name").getType());
+
+      assertNotNull(table.getIcon());
+      assertEquals("account_tree", table.getIcon().getName());
+
+      assertEquals(1, table.getUniqueKeys().size());
+      assertEquals(List.of("name"), table.getUniqueKeys().get(0).getFieldNames());
+      assertEquals("Name", table.getUniqueKeys().get(0).getLabel());
+
+      assertEquals(2, table.getAssociations().size());
+      assertEquals("A1", table.getAssociations().get(0).getName());
+      assertEquals("theirTable", table.getAssociations().get(1).getAssociatedTableName());
+
+      assertNotNull(table.getPermissionRules());
+      assertEquals(PermissionLevel.READ_WRITE_PERMISSIONS, table.getPermissionRules().getLevel());
+      assertEquals(DenyBehavior.HIDDEN, table.getPermissionRules().getDenyBehavior());
+      assertEquals("myTablePermissions", table.getPermissionRules().getPermissionBaseName());
+      assertNotNull(table.getPermissionRules().getCustomPermissionChecker());
+      assertEquals("com.kingsrook.SomeChecker", table.getPermissionRules().getCustomPermissionChecker().getName());
+      assertEquals(QCodeType.JAVA, table.getPermissionRules().getCustomPermissionChecker().getCodeType());
+
+      assertEquals(1, table.getSections().size());
+      assertEquals("identity", table.getSections().get(0).getName());
+      assertEquals(Tier.T1, table.getSections().get(0).getTier());
+      assertEquals(List.of("id", "firstName", "lastName"), table.getSections().get(0).getFieldNames());
+
+      assertEquals(2, table.getCustomizers().size());
+      assertEquals("com.kingsrook.SomePostQuery", table.getCustomizers().get(TableCustomizers.POST_QUERY_RECORD.getRole()).getName());
+      assertEquals("com.kingsrook.SomePreDelete", table.getCustomizers().get(TableCustomizers.PRE_DELETE_RECORD.getRole()).getName());
+
+      assertEquals(Set.of(Capability.TABLE_COUNT, Capability.QUERY_STATS), table.getDisabledCapabilities());
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSimpleJson() throws QMetaDataLoaderException
+   {
+      QTableMetaData table = new QTableMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         {
+            "class": "QTableMetaData",
+            "version": "1.0",
+            "name": "myTable",
+            "fields":
+            {
+               "id": {"name": "id", "type": "INTEGER"},
+               "name": {"name": "name", "type": "STRING"}
+            }
+         }
+         """, StandardCharsets.UTF_8), "myTable.json");
+
+      assertEquals("myTable", table.getName());
+      assertEquals(2, table.getFields().size());
+      assertEquals("id", table.getFields().get("id").getName());
+      assertEquals(QFieldType.INTEGER, table.getFields().get("id").getType());
+      assertEquals("name", table.getFields().get("name").getName());
+      assertEquals(QFieldType.STRING, table.getFields().get("name").getType());
+   }
+
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/GenericMetaDataLoaderTest.java

@@ -0,0 +1,81 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.nio.charset.StandardCharsets;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for GenericMetaDataLoader - providing coverage for AbstractMetaDataLoader.
+ *******************************************************************************/
+class GenericMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testProcess() throws QMetaDataLoaderException
+   {
+      ////////////////////////////////////////////////////////////////////////////////
+      // trying to get some coverage of various types in here (for Abstract loader) //
+      ////////////////////////////////////////////////////////////////////////////////
+      QProcessMetaData process = new GenericMetaDataLoader<>(QProcessMetaData.class).fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1
+         name: myProcess
+         tableName: someTable
+         maxInputRecords: 1
+         isHidden: true
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertEquals("myProcess", process.getName());
+      assertEquals("someTable", process.getTableName());
+      assertEquals(1, process.getMaxInputRecords());
+      assertTrue(process.getIsHidden());
+   }
+
+
+
+   /*******************************************************************************
+    ** just here for coverage of this class, as we're failing to hit it otherwise.
+    *******************************************************************************/
+   @SuppressWarnings({ "rawtypes", "unchecked" })
+   @Test
+   void testNoValueException()
+   {
+      assertThatThrownBy(() -> new GenericMetaDataLoader(QBackendMetaData.class).reflectivelyMapValue(new QInstance(), null, GenericMetaDataLoaderTest.class, "rawValue", new LoadingContext("test.yaml", "/")));
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/producers/subpackage/TestProcessMetaDataProducer.java

@@ -0,0 +1,47 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.producers.subpackage;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class TestProcessMetaDataProducer implements MetaDataProducerInterface<QProcessMetaData>
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QProcessMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QProcessMetaData().withName("fromProducer");
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/actions/processes/ProcessSummaryLineInterfaceAssert.java

@@ -185,4 +185,13 @@ public class ProcessSummaryLineInterfaceAssert extends AbstractAssert<ProcessSum
       return (this);
    }
 
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public ProcessSummaryLineInterface getLine()
+   {
+      return actual;
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/fields/WhiteSpaceBehaviorTest.java

@@ -0,0 +1,200 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.fields;
+
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
+import com.kingsrook.qqq.backend.core.actions.values.ValueBehaviorApplier;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import com.kingsrook.qqq.backend.core.utils.collections.ListBuilder;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for WhiteSpaceBehavior
+ *******************************************************************************/
+class WhiteSpaceBehaviorTest extends BaseTest
+{
+   public static final String FIELD = "firstName";
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testNone()
+   {
+      assertNull(applyToRecord(WhiteSpaceBehavior.NONE, new QRecord(), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertNull(applyToRecord(WhiteSpaceBehavior.NONE, new QRecord().withValue(FIELD, null), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertEquals("John", applyToRecord(WhiteSpaceBehavior.NONE, new QRecord().withValue(FIELD, "John"), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+
+      assertEquals(ListBuilder.of("J.   ohn", null, "Jane\n"), applyToRecords(WhiteSpaceBehavior.NONE, List.of(
+            new QRecord().withValue(FIELD, "J.   ohn"),
+            new QRecord(),
+            new QRecord().withValue(FIELD, "Jane\n")),
+         ValueBehaviorApplier.Action.INSERT).stream().map(r -> r.getValueString(FIELD)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testRemoveWhiteSpace()
+   {
+      assertNull(applyToRecord(WhiteSpaceBehavior.REMOVE_ALL_WHITESPACE, new QRecord(), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertNull(applyToRecord(WhiteSpaceBehavior.REMOVE_ALL_WHITESPACE, new QRecord().withValue(FIELD, null), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertEquals("doobeedoobeedoo", applyToRecord(WhiteSpaceBehavior.REMOVE_ALL_WHITESPACE, new QRecord().withValue(FIELD, "doo bee doo\n bee doo"), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+
+      assertEquals(ListBuilder.of("thisistheway", null, "thatwastheway"), applyToRecords(WhiteSpaceBehavior.REMOVE_ALL_WHITESPACE, List.of(
+            new QRecord().withValue(FIELD, "this is\rthe way   \t"),
+            new QRecord(),
+            new QRecord().withValue(FIELD, "that was the way\n")),
+         ValueBehaviorApplier.Action.INSERT).stream().map(r -> r.getValueString(FIELD)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testTrimWhiteSpace()
+   {
+      assertNull(applyToRecord(WhiteSpaceBehavior.TRIM, new QRecord(), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertNull(applyToRecord(WhiteSpaceBehavior.TRIM, new QRecord().withValue(FIELD, null), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertEquals("doo bee doo\n bee doo", applyToRecord(WhiteSpaceBehavior.TRIM, new QRecord().withValue(FIELD, "    doo bee doo\n bee doo\r \n\n"), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+
+      assertEquals(ListBuilder.of("this is\rthe way", null, "that was the way"), applyToRecords(WhiteSpaceBehavior.TRIM, List.of(
+            new QRecord().withValue(FIELD, "      this is\rthe way   \t"),
+            new QRecord(),
+            new QRecord().withValue(FIELD, "that was the way\n")),
+         ValueBehaviorApplier.Action.INSERT).stream().map(r -> r.getValueString(FIELD)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testTrimLeftWhiteSpace()
+   {
+      assertNull(applyToRecord(WhiteSpaceBehavior.TRIM_LEFT, new QRecord(), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertNull(applyToRecord(WhiteSpaceBehavior.TRIM_LEFT, new QRecord().withValue(FIELD, null), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertEquals("doo bee doo\n bee doo\r \n\n", applyToRecord(WhiteSpaceBehavior.TRIM_LEFT, new QRecord().withValue(FIELD, "    doo bee doo\n bee doo\r \n\n"), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+
+      assertEquals(ListBuilder.of("this is\rthe way   \t", null, "that was the way\n"), applyToRecords(WhiteSpaceBehavior.TRIM_LEFT, List.of(
+            new QRecord().withValue(FIELD, "      this is\rthe way   \t"),
+            new QRecord(),
+            new QRecord().withValue(FIELD, "  \n that was the way\n")),
+         ValueBehaviorApplier.Action.INSERT).stream().map(r -> r.getValueString(FIELD)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testTrimRightWhiteSpace()
+   {
+      assertNull(applyToRecord(WhiteSpaceBehavior.TRIM_RIGHT, new QRecord(), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertNull(applyToRecord(WhiteSpaceBehavior.TRIM_RIGHT, new QRecord().withValue(FIELD, null), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+      assertEquals("    doo bee doo\n bee doo", applyToRecord(WhiteSpaceBehavior.TRIM_RIGHT, new QRecord().withValue(FIELD, "    doo bee doo\n bee doo\r \n\n"), ValueBehaviorApplier.Action.INSERT).getValue(FIELD));
+
+      assertEquals(ListBuilder.of("      this is\rthe way", null, "  \n that was the way"), applyToRecords(WhiteSpaceBehavior.TRIM_RIGHT, List.of(
+            new QRecord().withValue(FIELD, "      this is\rthe way   \t"),
+            new QRecord(),
+            new QRecord().withValue(FIELD, "  \n that was the way\n")),
+         ValueBehaviorApplier.Action.INSERT).stream().map(r -> r.getValueString(FIELD)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private QRecord applyToRecord(WhiteSpaceBehavior behavior, QRecord record, ValueBehaviorApplier.Action action)
+   {
+      return (applyToRecords(behavior, List.of(record), action).get(0));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private List<QRecord> applyToRecords(WhiteSpaceBehavior behavior, List<QRecord> records, ValueBehaviorApplier.Action action)
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON_MEMORY);
+      behavior.apply(action, records, QContext.getQInstance(), table, table.getField(FIELD));
+      return (records);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testValidation()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_SHAPE);
+
+      ///////////////////////////////////////////
+      // should be no errors on a string field //
+      ///////////////////////////////////////////
+      assertTrue(WhiteSpaceBehavior.TRIM.validateBehaviorConfiguration(table, table.getField("name")).isEmpty());
+
+      //////////////////////////////////////////
+      // should be an error on a number field //
+      //////////////////////////////////////////
+      assertEquals(1, WhiteSpaceBehavior.REMOVE_ALL_WHITESPACE.validateBehaviorConfiguration(table, table.getField("id")).size());
+
+      /////////////////////////////////////////
+      // NONE should be allowed on any field //
+      /////////////////////////////////////////
+      assertTrue(WhiteSpaceBehavior.NONE.validateBehaviorConfiguration(table, table.getField("id")).isEmpty());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertFullProcessTest.java

@@ -29,15 +29,26 @@ import java.util.List;
 import java.util.Map;
 import java.util.UUID;
 import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.customizers.AbstractPreInsertCustomizer;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
 import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
 import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryAssert;
+import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
+import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
+import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
 import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
@@ -176,7 +187,15 @@ class BulkInsertFullProcessTest extends BaseTest
       assertThat(runProcessOutput.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY)).isNotNull().isInstanceOf(List.class);
       assertThat(runProcessOutput.getException()).isEmpty();
 
-      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("Inserted Id values between 1 and 2");
+      ProcessSummaryLineInterface okLine = ProcessSummaryAssert.assertThat(runProcessOutput)
+         .hasLineWithMessageContaining("Person Memory records were inserted")
+         .hasStatus(Status.OK)
+         .hasCount(2)
+         .getLine();
+      assertEquals(List.of(1, 2), ((ProcessSummaryLine) okLine).getPrimaryKeys());
+
+      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("records were processed from the file").hasStatus(Status.INFO);
+      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("Inserted Id values between 1 and 2").hasStatus(Status.INFO);
 
       ////////////////////////////////////
       // query for the inserted records //
@@ -201,6 +220,86 @@ class BulkInsertFullProcessTest extends BaseTest
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSummaryLinePrimaryKeys() throws Exception
+   {
+      assertThat(TestUtils.queryTable(TestUtils.TABLE_NAME_PERSON_MEMORY)).isEmpty();
+      QContext.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON_MEMORY)
+         .withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(PersonWarnOrErrorCustomizer.class));
+
+      /////////////////////////////////////////////////////////
+      // start the process - expect to go to the upload step //
+      /////////////////////////////////////////////////////////
+      RunProcessInput  runProcessInput  = new RunProcessInput();
+      RunProcessOutput runProcessOutput = startProcess(runProcessInput);
+      String           processUUID      = runProcessOutput.getProcessUUID();
+
+      continueProcessPostUpload(runProcessInput, processUUID, simulateFileUploadForWarningCase());
+      continueProcessPostFileMapping(runProcessInput);
+      continueProcessPostValueMapping(runProcessInput);
+      runProcessOutput = continueProcessPostReviewScreen(runProcessInput);
+
+      ProcessSummaryLineInterface okLine = ProcessSummaryAssert.assertThat(runProcessOutput)
+         .hasLineWithMessageContaining("Person Memory record was inserted")
+         .hasStatus(Status.OK)
+         .hasCount(1)
+         .getLine();
+      assertEquals(List.of(1), ((ProcessSummaryLine) okLine).getPrimaryKeys());
+
+      ProcessSummaryLineInterface warnTornadoLine = ProcessSummaryAssert.assertThat(runProcessOutput)
+         .hasLineWithMessageContaining("records were inserted, but had a warning: Tornado warning")
+         .hasStatus(Status.WARNING)
+         .hasCount(2)
+         .getLine();
+      assertEquals(List.of(2, 3), ((ProcessSummaryLine) warnTornadoLine).getPrimaryKeys());
+
+      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("record was inserted, but had a warning: Hurricane warning").hasStatus(Status.WARNING).hasCount(1);
+      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("records were processed from the file").hasStatus(Status.INFO).hasCount(4);
+      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("Inserted Id values between 1 and 4").hasStatus(Status.INFO);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSummaryLineErrors() throws Exception
+   {
+      assertThat(TestUtils.queryTable(TestUtils.TABLE_NAME_PERSON_MEMORY)).isEmpty();
+      QContext.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON_MEMORY)
+         .withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(PersonWarnOrErrorCustomizer.class));
+
+      /////////////////////////////////////////////////////////
+      // start the process - expect to go to the upload step //
+      /////////////////////////////////////////////////////////
+      RunProcessInput  runProcessInput  = new RunProcessInput();
+      RunProcessOutput runProcessOutput = startProcess(runProcessInput);
+      String           processUUID      = runProcessOutput.getProcessUUID();
+
+      continueProcessPostUpload(runProcessInput, processUUID, simulateFileUploadForErrorCase());
+      continueProcessPostFileMapping(runProcessInput);
+      continueProcessPostValueMapping(runProcessInput);
+      runProcessOutput = continueProcessPostReviewScreen(runProcessInput);
+
+      ProcessSummaryAssert.assertThat(runProcessOutput).hasLineWithMessageContaining("Person Memory record was inserted.").hasStatus(Status.OK).hasCount(1);
+
+      ProcessSummaryAssert.assertThat(runProcessOutput)
+         .hasLineWithMessageContaining("plane")
+         .hasStatus(Status.ERROR)
+         .hasCount(1);
+
+      ProcessSummaryAssert.assertThat(runProcessOutput)
+         .hasLineWithMessageContaining("purifier")
+         .hasStatus(Status.ERROR)
+         .hasCount(1);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -301,6 +400,47 @@ class BulkInsertFullProcessTest extends BaseTest
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static StorageInput simulateFileUploadForWarningCase() throws Exception
+   {
+      String       storageReference = UUID.randomUUID() + ".csv";
+      StorageInput storageInput     = new StorageInput(TestUtils.TABLE_NAME_MEMORY_STORAGE).withReference(storageReference);
+      try(OutputStream outputStream = new StorageAction().createOutputStream(storageInput))
+      {
+         outputStream.write((getPersonCsvHeaderUsingLabels() + """
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","John","Doe","1980-01-01","john@doe.com","Missouri",42
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","Tornado warning","Doe","1980-01-01","john@doe.com","Missouri",42
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","Tornado warning","Doey","1980-01-01","john@doe.com","Missouri",42
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","Hurricane warning","Doe","1980-01-01","john@doe.com","Missouri",42
+            """).getBytes());
+      }
+      return storageInput;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static StorageInput simulateFileUploadForErrorCase() throws Exception
+   {
+      String       storageReference = UUID.randomUUID() + ".csv";
+      StorageInput storageInput     = new StorageInput(TestUtils.TABLE_NAME_MEMORY_STORAGE).withReference(storageReference);
+      try(OutputStream outputStream = new StorageAction().createOutputStream(storageInput))
+      {
+         outputStream.write((getPersonCsvHeaderUsingLabels() + """
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","John","Doe","1980-01-01","john@doe.com","Missouri",42
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","not-pre-Error plane","Doe","1980-01-01","john@doe.com","Missouri",42
+            "0","2021-10-26 14:39:37","2021-10-26 14:39:37","Error purifier","Doe","1980-01-01","john@doe.com","Missouri",42
+            """).getBytes());
+      }
+      return storageInput;
+   }
+
+
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -331,4 +471,47 @@ class BulkInsertFullProcessTest extends BaseTest
       )));
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static class PersonWarnOrErrorCustomizer implements TableCustomizerInterface
+   {
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public AbstractPreInsertCustomizer.WhenToRun whenToRunPreInsert(InsertInput insertInput, boolean isPreview)
+      {
+         return AbstractPreInsertCustomizer.WhenToRun.BEFORE_ALL_VALIDATIONS;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public List<QRecord> preInsert(InsertInput insertInput, List<QRecord> records, boolean isPreview) throws QException
+      {
+         for(QRecord record : records)
+         {
+            if(record.getValueString("firstName").toLowerCase().contains("warn"))
+            {
+               record.addWarning(new QWarningMessage(record.getValueString("firstName")));
+            }
+            else if(record.getValueString("firstName").toLowerCase().contains("error"))
+            {
+               if(isPreview && record.getValueString("firstName").toLowerCase().contains("not-pre-error"))
+               {
+                  continue;
+               }
+
+               record.addError(new BadInputStatusMessage(record.getValueString("firstName")));
+            }
+         }
+         return records;
+      }
+   }
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/utils/RecordLookupHelperTest.java

@@ -22,8 +22,15 @@
 package com.kingsrook.qqq.backend.core.processes.utils;
 
 
+import java.util.ArrayList;
+import java.util.ConcurrentModificationException;
 import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
 import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.context.CapturedContext;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -32,6 +39,7 @@ import com.kingsrook.qqq.backend.core.utils.TestUtils;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Fail.fail;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertNotNull;
 import static org.junit.jupiter.api.Assertions.assertNull;
@@ -195,4 +203,54 @@ class RecordLookupHelperTest extends BaseTest
       assertEquals(1, MemoryRecordStore.getStatistics().get(MemoryRecordStore.STAT_QUERIES_RAN));
    }
 
+
+
+   /*******************************************************************************
+    ** run a lot of threads (eg, 100), each trying to do lots of work in a
+    ** shared recordLookupHelper.  w/o the flag to use sync'ed collections, this
+    ** (usually?) fails with a ConcurrentModificationException - but with the sync'ed
+    ** collections, is safe.
+    *******************************************************************************/
+   @Test
+   void testConcurrentModification() throws InterruptedException, ExecutionException
+   {
+      ExecutorService    executorService    = Executors.newFixedThreadPool(100);
+      RecordLookupHelper recordLookupHelper = new RecordLookupHelper(true);
+
+      CapturedContext capture = QContext.capture();
+
+      List<Future<?>> futures = new ArrayList<>();
+      for(int i = 0; i < 100; i++)
+      {
+         int finalI = i;
+         Future<?> future = executorService.submit(() ->
+         {
+            QContext.init(capture);
+            for(int j = 0; j < 25000; j++)
+            {
+               try
+               {
+                  recordLookupHelper.getRecordByKey(String.valueOf(j), "id", j);
+               }
+               catch(ConcurrentModificationException cme)
+               {
+                  fail("CME!", cme);
+               }
+               catch(Exception e)
+               {
+                  //////////////
+                  // expected //
+                  //////////////
+               }
+            }
+         });
+         futures.add(future);
+      }
+
+      for(Future<?> future : futures)
+      {
+         future.get();
+      }
+   }
+
 }