Test coverage on new javalin routing classes

Feature/migrate sample app to new javalin server

.circleci/config.yml

@@ -127,14 +127,9 @@ commands:
           command: |
             cd docs
             asciidoctor -a docinfo=shared index.adoc
-
-  upload_docs_site:
-    steps:
-      - run:
-          name: scp html to justinsgotskinnylegs.com
-          command: |
-            cd docs
-            scp index.html dkelkhoff@45.79.44.221:/mnt/first-volume/dkelkhoff/nginx/html/justinsgotskinnylegs.com/qqq-docs.html
+      - store_artifacts:
+          path: docs/index.html
+          when: always
 
 jobs:
   mvn_test:
@@ -159,7 +154,6 @@ jobs:
     steps:
       - install_asciidoctor
       - run_asciidoctor
-      - upload_docs_site
 
 workflows:
   test_only:

docs/Introduction.adoc

@@ -136,17 +136,13 @@ This speaks to the fact that this "code" is not executable code - but rather is
 **** The Filter button in the Query Screen will present a menu listing all fields from the table for the user to build ad-hoc queries against the table.
 The data-types specified for the fields (in the meta-data) dictate what operators QQQ allows the user to use against fields (e.g., Strings offer "contains" vs Numbers offer "greater than").
 **** Values for records from the table will be formatted for presentation based on the meta-data (such as a numeric field being shown with commas if it represents a quantity, or formatted as currency).
-...
-
-[start=2]
-. *Meta Data* - declarative code - java object instances (potentially which could be read from `.yaml` files or other data sources in a future version of QQQ), which tell QQQ about the backend systems, tables, processes, reports, widgets, etc, that make up the application.
-For example:
-* Details about the database you are using, and how to connect to it.
-* A database table's name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).
-* The description of web API - its URL and authentication mechanism.
-* A table/path within a web API, and the fields returned in the JSON at that endpoint.
-* The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.
-* Details about a chart that summarizes data from a table for presentation as a dashboard widget.
+* Other kinds of information that you tell QQQ about in the form of meta-data objects includes:
+** Details about the database you are using, and how to connect to it.
+** A database table's name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).
+** The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.
+** Details about a chart that summarizes data from a table for presentation as a dashboard widget.
+** The description of web API - its URL and authentication mechanism.
+** A table/path within a web API, and the fields returned in the JSON at that endpoint.
 // the section below is kinda dumb.  like, it says you have to write application code, but
 // then it just talks about how your app code gets for-free the same shit that QQQ does.
 // it should instead say more about what your custom app code is or does.
@@ -164,7 +160,8 @@ For example:
 // * The multi-threaded, paged producer/consumer pattern used in standard framework actions is how all custom application actions are also invoked.
 // ** For example, the standard QQQ Bulk Edit action uses the same streamed-ETL process that custom application processes can use.
 // Meaning your custom processes can take full advantage of the same complex frontend, middleware, and backend structural pieces, and you can just focus on your unique busines logic needs.
-2. *Application code* - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic.
+
+. *Application code* - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic.
 QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model.
 For example:
 

docs/index.adoc

@@ -6,7 +6,10 @@
 
 include::Introduction.adoc[leveloffset=+1]
 
-== Meta Data
+== Meta Data Production
+include::metaData/MetaDataProduction.adoc[leveloffset=+1]
+
+== Meta Data Types
 // Organizational units
 include::metaData/QInstance.adoc[leveloffset=+1]
 include::metaData/Backends.adoc[leveloffset=+1]

docs/metaData/MetaDataProduction.adoc

@@ -0,0 +1,363 @@
+[#MetaDataProduction]
+include::../variables.adoc[]
+
+The first thing that an application built using QQQ needs to do is to define its meta data.
+This basically means the construction of a `QInstance` object, which is populated with the
+meta data objects defining the backend(s), tables, processes, possible-value sources, joins,
+authentication provider, etc, that make your application.
+
+There are various styles that can be used for how you define your meta data, and for the
+most part they can be mixed and matched.  They will be presented here based on the historical
+evolution of how they were added to QQQ, where we generally believe that better techniques have
+been added over time.  So, you may wish to skip the earlier techniques, and jump straight to
+the end of this section.  However, it can always be instructive to learn about the past, so,
+read at your own pace.
+
+== Omni-Provider
+At the most basic level, the way to populate a `QInstance` is the simple and direct approach of creating
+one big class, possibly with just one big method, and just doing all the work directly inline there.
+
+This may (clearly) violate several good engineering principles.  However, it does have the benefit of
+being simple - if all of your meta-data is defined in one place, it can be pretty simple to find where
+that place is.  So - especially in a small project, this technique may be worth continuing to consider.
+
+Re: "doing all the work" as mentioned above - what work are we talking about?  At a minimum, we need
+to construct the following meta-data objects, and pass them into our `QInstance`:
+
+* `QAuthenticationMetaData` - how (or if!) users will be authenticated to the application.
+* `QBackendMeataData` - a backend data store.
+* `QTableMetaData` - a table (within the backend).
+* `QAppMetaData` - an organizational unit to present the other elements in a UI.
+
+Here's what a single-method omni-provider could look like:
+
+[source,java]
+.About the simplest possible single-file meta-data provider
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+
+    qInstance.setAuthentication(new QAuthenticationMetaData()
+       .withName("anonymous")
+       .withType(QAuthenticationType.FULLY_ANONYMOUS));
+
+    qInstance.addBackend(new QBackendMetaData()
+       .withBackendType(MemoryBackendModule.class)
+       .withName("memoryBackend"));
+
+    qInstance.addTable(new QTableMetaData()
+        .withName("myTable")
+        .withPrimaryKeyField("id")
+        .withBackendName("memoryBackend")
+        .withField(new QFieldMetaData("id", QFieldType.INTEGER)));
+
+    qInstance.addApp(new QAppMetaData()
+       .withName("myApp")
+       .withSectionOfChildren(new QAppSection().withName("mySection"),
+          qInstance.getTable("myTable")))
+
+   return (qInstance);
+}
+----
+
+== Multi-method Omni-Provider
+
+The next evolution of meta-data production comes by just applying some basic better-engineering
+principles, and splitting up from a single method that constructs all the things, to at least
+using unique methods to construct each thing, then calling those methods to add their results
+to the QInstance.
+
+[source,java]
+.Multi-method omni- meta-data provider
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+    qInstance.setAuthentication(defineAuthenticationMetaData());
+    qInstance.addBackend(defineBackendMetaData());
+    qInstance.addTable(defineMyTableMetaData());
+    qInstance.addApp(defineMyAppMetaData(qInstance));
+    return qInstance;
+}
+
+public QAuthenticationMetaData defineAuthenticationMetaData()
+{
+    return new QAuthenticationMetaData()
+       .withName("anonymous")
+       .withType(QAuthenticationType.FULLY_ANONYMOUS);
+}
+
+public QBackendMetaData defineBackendMetaData()
+{
+    return new QBackendMetaData()
+       .withBackendType(MemoryBackendModule.class)
+       .withName("memoryBackend");
+}
+
+// implementations of defineMyTableMetaData() and defineMyAppMetaData(qInstance)
+// left as an exercise for the reader
+----
+
+== Multi-class Providers
+
+Then the next logical evolution would be to put each of these single meta-data producing
+objects into its own class, along with calls to those classes.  This gets us away from the
+"5000 line" single-class, and lets us stop using the word "omni":
+
+[source,java]
+.Multi-class meta-data providers
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+    qInstance.setAuthentication(new AuthMetaDataProvider().defineAuthenticationMetaData());
+    qInstance.addBackend(new BackendMetaDataProvider().defineBackendMetaData());
+    qInstance.addTable(new MyTableMetaDataProvider().defineTableMetaData());
+    qInstance.addApp(new MyAppMetaDataProvider().defineAppMetaData(qInstance));
+    return qInstance;
+}
+
+public class AuthMetaDataProvider
+{
+    public QAuthenticationMetaData defineAuthenticationMetaData()
+    {
+        return new QAuthenticationMetaData()
+           .withName("anonymous")
+           .withType(QAuthenticationType.FULLY_ANONYMOUS);
+    }
+}
+
+public class BackendMetaDataProvider
+{
+    public QBackendMetaData defineBackendMetaData()
+    {
+        return new QBackendMetaData()
+           .withBackendType(MemoryBackendModule.class)
+           .withName("memoryBackend");
+    }
+}
+
+// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
+// left as an exercise for the reader
+----
+
+== MetaDataProducerInterface
+
+As the size of your application grows, if you're doing per-object meta-data providers, you may find it
+burdensome, when adding a new object to your instance, to have to write code for it in two places -
+that is - a new class to produce that meta-data object, AND a single line of code to add that object
+to your `QInstance`.  As such, a mechanism exists to let you avoid that line-of-code for adding the object
+to the `QInstance`.
+
+This mechanism involves adding the `MetaDataProducerInterface` to all of your classes that produce a
+meta-data object.  This interface is generic, with a type parameter that will typically be the type of
+meta-data object you are producing, such as `QTableMetaData`, `QProcessMetaData`, or `QWidgetMetaData`,
+(technically, any class which implements `TopLevelMetaData`).  Implementers of the interface are then
+required to override just one method: `T produce(QInstance qInstance) throws QException;`
+
+Once you have your `MetaDataProducerInterface` classes defined, then there's a one-time call needed
+to add all of the objects produced by these classes to your `QInstance` - as shown here:
+
+[source,java]
+.Using MetaDataProducerInterface
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+    MetaDataProducerHelper.processAllMetaDataProducersInPackage(qInstance,
+      "com.mydomain.myapplication");
+    return qInstance;
+}
+
+public class AuthMetaDataProducer implements MetaDataProducerInterface<QAuthenticationMetaData>
+{
+    @Override
+    public QAuthenticationMetaData produce(QInstance qInstance)
+    {
+        return new QAuthenticationMetaData()
+           .withName("anonymous")
+           .withType(QAuthenticationType.FULLY_ANONYMOUS);
+    }
+}
+
+public class BackendMetaDataProducer implements MetaDataProducerInterface<QBackendMetaData>
+{
+    @Override
+    public QBackendMetaData defineBackendMetaData()
+    {
+        return new QBackendMetaData()
+           .withBackendType(MemoryBackendModule.class)
+           .withName("memoryBackend");
+    }
+}
+
+// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
+// left as an exercise for the reader
+----
+
+=== MetaDataProducerMultiOutput
+It is worth mentioning, that sometimes it might feel like a bridge that's a bit too far, to make
+every single one of your meta-data objects require its own class.  Some may argue that it's best
+to do it that way - single responsibility principle, etc.  But, if you're producing, say, 5 widgets
+that are all related, and it's only a handful of lines of code for each one, maybe you'd rather
+produce them all in the same class.  Or maybe when you define a table, you'd like to define its
+joins and widgets at the same time.
+
+This approach can be accomplished by making the type argument for your `MetaDataProducerInterface` be
+`MetaDataProducerMultiOutput` - a simple class that just wraps a list of other `MetaDataProducerOutput`
+objects.
+
+[source,java]
+.Returning a MetaDataProducerMultiOutput
+----
+public class MyMultiProducer implements MetaDataProducerInterface<MetaDataProducerMultiOutput>
+{
+    @Override
+    public MetaDataProducerMultiOutput produce(QInstance qInstance)
+    {
+       MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput();
+
+       output.add(new QPossibleValueSource()...);
+       output.add(new QJoinMetaData()...);
+       output.add(new QJoinMetaData()...);
+       output.add(new QWidgetMetaData()...);
+       output.add(new QTableMetaData()...);
+
+       return (output);
+    }
+}
+----
+
+
+== Aside: TableMetaData with RecordEntities
+At this point, let's take a brief aside to dig deeper into the creation of a `QTableMeta` object.
+Tables, being probably the most important meta-data type in QQQ, have a lot of information that can
+be specified in their meta-data object.
+
+At the same time, if you're writing any custom code in your QQQ application
+(e.g., any processes or table customizers), where you're working with records from tables, you may
+prefer being able to work with entity beans (e.g., java classes with typed getter & setter methods),
+rather than the default object type that QQQ's ORM actions return, the `QRecord`, which carries all
+of its values in a `Map` (where you don't get compile-time checks of field names or data types).
+QQQ has a mechanism for dealing with this - in the form of the `QRecordEntity` class.
+
+So - if you want to build your application using entity beans (which is recommended, for the compile-time
+safety that they provide in custom code), you will be writing a `QRecordEntity` class for each of your tables,
+which will look like:
+
+[source,java]
+.QRecordEntity example
+----
+public class MyTable extends QRecordEntity
+{
+   public static final String TABLE_NAME = "myTable";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField()
+   private String name;
+
+   // no-arg constructor and constructor that takes a QRecord
+   // getters & setters (and optional fluent setters)
+}
+----
+
+The point of introducing this topic here and now is, that a `QRecordEntity` can be used to shortcut to
+defining some of the attributes in a `QTableMetaData` object.  Specifically, in a `MetaDataProducer<QTableMetaData>`
+you may say:
+
+[source,java]
+.QTableMetaDataProducer using a QRecordEntity
+----
+public QTableMetaData produce(QInstance qInstance) throws QExcpetion
+{
+  return new QTableMetaData()
+     .withName(MyTable.TABLE_NAME)
+     .withFieldsFromEntity(MyTable.class)
+     .withBackendName("memoryBackend");
+}
+----
+
+That `withFieldsFromEntity` call is one of the biggest benefits of this technique.  It allows you to avoid defining
+all of the fields in you table in two places (the entity and the table meta-data).
+
+== MetaData Producing Annotations for Entities
+
+If you are using `QRecordEntity` classes that correspond to your tables, then you can take advantage of some
+additional annotations on those classes, to produce more related meta-data objects associated with those tables.
+The point of this is to eliminate boilerplate, and simplify / speed up the process of getting a new table
+built and deployed in your application, with some bells and whistles added.
+
+=== @QMetaDataProducingEntity
+This is an annotation to go on a QRecordEntity class, which you would like to be
+processed by `MetaDataProducerHelper`, to automatically produce some meta-data
+objects.  Specifically supports:
+
+* Making a possible-value-source out of the table.
+* Processing child tables to create joins and childRecordList widgets
+
+=== @ChildTable
+This is an annotation used as a value that goes inside a `@QMetadataProducingEntity` annotation, to define
+child-tables, e.g., for producing joins and childRecordList widgets related to the table defined in the entity class.
+
+=== @ChildJoin
+This is an annotation used as a value inside a `@ChildTable` inside a `@QMetadataProducingEntity` annotation,
+to control the generation of a `QJoinMetaData`, as a `ONE_TO_MANY` type join from the table represented by
+the annotated entity, to the table referenced in the `@ChildTable` annotation.
+
+=== @ChildRecordListWidget
+This is an annotation used as a value that goes inside a `@QMetadataProducingEntity` annotation, to control
+the generation of a QWidgetMetaData - for a ChildRecordList widget.
+
+[source,java]
+.QRecordEntity with meta-data producing annotations
+----
+@QMetaDataProducingEntity(
+   producePossibleValueSource = true,
+   childTables = {
+      @ChildTable(
+         childTableEntityClass = MyChildTable.class,
+         childJoin = @ChildJoin(enabled = true),
+         childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Children"))
+   }
+)
+public class MyTable extends QRecordEntity
+{
+   public static final String TABLE_NAME = "myTable";
+   // class body left as exercise for reader
+}
+----
+
+The class given in the example above, if processed by the `MetaDataProducerHelper`, would add the following
+meta-data objects to your `QInstance`:
+
+* A `QPossibleValueSource` named `myTable`, of type `TABLE`, with `myTable` as its backing table.
+* A `QJoinMetaData` named `myTableJoinMyChildTable`, as a `ONE_TO_MANY` type, between those two tables.
+* A `QWidgetMetaData` named `myTableJoinMyChildTable`, as a `CHILD_RECORD_LIST` type, that will show a list of
+records from `myChildTable` as a widget, when viewing a record from `myTable`.
+
+== Other MetaData Producing Annotations
+
+Similar to these annotations for a `RecordEntity`, a similar one exists for a `PossibleValueEnum` class,
+to automatically write the meta-data to use that enum as a possible value source in your application:
+
+=== @QMetaDataProducingPossibleValueEnum
+This is an annotation to go on a `PossibleValueEnum` class, which you would like to be
+processed by MetaDataProducerHelper, to automatically produce a PossibleValueSource meta-data
+based on the enum.
+
+[source,java]
+.PossibleValueEnum with meta-data producing annotation
+----
+@QMetaDataProducingPossibleValueEnum(producePossibleValueSource = true)
+public enum MyOptionsEnum implements PossibleValueEnum<Integer>
+{
+   // values and methods left as exercise for reader
+}
+----
+The enum given in the example above, if processed by the `MetaDataProducerHelper`, would add the following
+meta-data object to your `QInstance`:
+
+* A `QPossibleValueSource` named `MyOptionsEnum`, of type `ENUM`, with `MyOptionsEnum` as its backing enum.

docs/metaData/QInstance.adoc

@@ -29,11 +29,21 @@ service.routes(qJavalinImplementation.getRoutes());
 service.start();
 ----
 
-*QBackendMetaData Setup Methods:*
+*QInstance Setup:*
 
 These are the methods that one is most likely to use when setting up (defining) a `QInstance` object:
 
-* asdf
+* `add(TopLevelMetaDataInterface metaData)` - Generic method that takes most of the meta-data subtypes that can be added
+to an instance, such as `QBackendMetaData`, `QTableMetaData`, `QProcessMetaData`, etc.
+There are also type-specific methods (e.g., `addTable`, `addProcess`, etc), which one can call instead - this would just
+be a matter of personal preference.
 
-*QBackendMetaData Usage Methods:*
+*QInstance Usage:*
 
+Generally you will set up a `QInstance` in your application's startup flow, and then place it in the server (e.g., javalin).
+But, if, during application-code runtime, you need access to any of the meta-data in the instance, you access it
+via the `QContext` object's static `getInstance()` method.  This can be useful, for example, to get a list of the defined
+tables in the application, or fields in a table, or details about a field, etc.
+
+It is generally considered risky and/or not a good idea at all to modify the `QInstance` after it has been validated and
+a server is running.  Future versions of QQQ may in fact restrict modifications to the instance after validation.

pom.xml

@@ -47,7 +47,7 @@
    </modules>
 
    <properties>
-      <revision>0.23.0-SNAPSHOT</revision>
+      <revision>0.24.0-SNAPSHOT</revision>
 
       <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
       <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/async/AsyncRecordPipeLoop.java

@@ -186,7 +186,7 @@ public class AsyncRecordPipeLoop
 
       if(recordCount > 0)
       {
-         LOG.info("End of job summary", logPair("recordCount", recordCount), logPair("jobName", jobName), logPair("millis", endTime - jobStartTime), logPair("recordsPerSecond", 1000d * (recordCount / (.001d + (endTime - jobStartTime)))));
+         LOG.debug("End of job summary", logPair("recordCount", recordCount), logPair("jobName", jobName), logPair("millis", endTime - jobStartTime), logPair("recordsPerSecond", 1000d * (recordCount / (.001d + (endTime - jobStartTime)))));
       }
 
       return (recordCount);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/audits/AuditAction.java

@@ -44,6 +44,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetOutput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.security.MultiRecordSecurityLock;
 import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock;
 import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLockFilters;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@@ -173,26 +174,53 @@ public class AuditAction extends AbstractQActionFunction<AuditInput, AuditOutput
       Map<String, Serializable> securityKeyValues = new HashMap<>();
       for(RecordSecurityLock recordSecurityLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks())))
       {
-         Serializable keyValue = record == null ? null : record.getValue(recordSecurityLock.getFieldName());
-
-         if(keyValue == null && oldRecord.isPresent())
-         {
-            LOG.debug("Table with a securityLock, but value not found in field", logPair("table", table.getName()), logPair("field", recordSecurityLock.getFieldName()));
-            keyValue = oldRecord.get().getValue(recordSecurityLock.getFieldName());
-         }
-
-         if(keyValue == null)
-         {
-            LOG.debug("Table with a securityLock, but value not found in field", logPair("table", table.getName()), logPair("field", recordSecurityLock.getFieldName()), logPair("oldRecordIsPresent", oldRecord.isPresent()));
-         }
-
-         securityKeyValues.put(recordSecurityLock.getSecurityKeyType(), keyValue);
+         getRecordSecurityKeyValues(table, record, oldRecord, recordSecurityLock, securityKeyValues);
       }
       return securityKeyValues;
    }
 
 
 
+   /***************************************************************************
+    ** recursive implementation of getRecordSecurityKeyValues, for dealing with
+    ** multi-locks
+    ***************************************************************************/
+   private static void getRecordSecurityKeyValues(QTableMetaData table, QRecord record, Optional<QRecord> oldRecord, RecordSecurityLock recordSecurityLock, Map<String, Serializable> securityKeyValues)
+   {
+      //////////////////////////////////////////////////////
+      // special case with recursive call for multi-locks //
+      //////////////////////////////////////////////////////
+      if(recordSecurityLock instanceof MultiRecordSecurityLock multiRecordSecurityLock)
+      {
+         for(RecordSecurityLock subLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(multiRecordSecurityLock.getLocks())))
+         {
+            getRecordSecurityKeyValues(table, record, oldRecord, subLock, securityKeyValues);
+         }
+
+         return;
+      }
+
+      ///////////////////////////////////////////
+      // by default, deal with non-multi locks //
+      ///////////////////////////////////////////
+      Serializable keyValue = record == null ? null : record.getValue(recordSecurityLock.getFieldName());
+
+      if(keyValue == null && oldRecord.isPresent())
+      {
+         LOG.debug("Table with a securityLock, but value not found in field", logPair("table", table.getName()), logPair("field", recordSecurityLock.getFieldName()));
+         keyValue = oldRecord.get().getValue(recordSecurityLock.getFieldName());
+      }
+
+      if(keyValue == null)
+      {
+         LOG.debug("Table with a securityLock, but value not found in field", logPair("table", table.getName()), logPair("field", recordSecurityLock.getFieldName()), logPair("oldRecordIsPresent", oldRecord.isPresent()));
+      }
+
+      securityKeyValues.put(recordSecurityLock.getSecurityKeyType(), keyValue);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -218,21 +246,16 @@ public class AuditAction extends AbstractQActionFunction<AuditInput, AuditOutput
                   throw (new QException("Requested audit for an unrecognized table name: " + auditSingleInput.getAuditTableName()));
                }
 
-               ///////////////////////////////////////////////////
-               // validate security keys on the table are given //
-               ///////////////////////////////////////////////////
-               for(RecordSecurityLock recordSecurityLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks())))
+               ///////////////////////////////////////////////////////
+               // validate security keys on the table are given     //
+               // originally, this case threw...                    //
+               // but i think it's better to record the audit, just //
+               // missing its security key value, then to fail...   //
+               // but, maybe should be configurable, etc...         //
+               ///////////////////////////////////////////////////////
+               if(!validateSecurityKeys(auditSingleInput, table))
                {
-                  if(auditSingleInput.getSecurityKeyValues() == null || !auditSingleInput.getSecurityKeyValues().containsKey(recordSecurityLock.getSecurityKeyType()))
-                  {
-                     ///////////////////////////////////////////////////////
-                     // originally, this case threw...                    //
-                     // but i think it's better to record the audit, just //
-                     // missing its security key value, then to fail...   //
-                     ///////////////////////////////////////////////////////
-                     // throw (new QException("Missing securityKeyValue [" + recordSecurityLock.getSecurityKeyType() + "] in audit request for table " + auditSingleInput.getAuditTableName()));
-                     LOG.info("Missing securityKeyValue in audit request", logPair("table", auditSingleInput.getAuditTableName()), logPair("securityKey", recordSecurityLock.getSecurityKeyType()));
-                  }
+                  LOG.debug("Missing securityKeyValue in audit request", logPair("table", auditSingleInput.getAuditTableName()), logPair("auditMessage", auditSingleInput.getMessage()), logPair("recordId", auditSingleInput.getRecordId()));
                }
 
                ////////////////////////////////////////////////
@@ -310,6 +333,70 @@ public class AuditAction extends AbstractQActionFunction<AuditInput, AuditOutput
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static boolean validateSecurityKeys(AuditSingleInput auditSingleInput, QTableMetaData table)
+   {
+      boolean allAreValid = true;
+      for(RecordSecurityLock recordSecurityLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks())))
+      {
+         boolean lockIsValid = validateSecurityKeysForLock(auditSingleInput, recordSecurityLock);
+         if(!lockIsValid)
+         {
+            allAreValid = false;
+         }
+      }
+
+      return (allAreValid);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static boolean validateSecurityKeysForLock(AuditSingleInput auditSingleInput, RecordSecurityLock recordSecurityLock)
+   {
+      if(recordSecurityLock instanceof MultiRecordSecurityLock multiRecordSecurityLock)
+      {
+         boolean allSubLocksAreValid = true;
+         boolean anySubLocksAreValid = false;
+         for(RecordSecurityLock lock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(multiRecordSecurityLock.getLocks())))
+         {
+            boolean subLockIsValid = validateSecurityKeysForLock(auditSingleInput, lock);
+            if(subLockIsValid)
+            {
+               anySubLocksAreValid = true;
+            }
+            else
+            {
+               allSubLocksAreValid = false;
+            }
+         }
+
+         if(multiRecordSecurityLock.getOperator().equals(MultiRecordSecurityLock.BooleanOperator.OR))
+         {
+            return (anySubLocksAreValid);
+         }
+         else if(multiRecordSecurityLock.getOperator().equals(MultiRecordSecurityLock.BooleanOperator.AND))
+         {
+            return (allSubLocksAreValid);
+         }
+      }
+      else
+      {
+         if(auditSingleInput.getSecurityKeyValues() == null || !auditSingleInput.getSecurityKeyValues().containsKey(recordSecurityLock.getSecurityKeyType()))
+         {
+            return (false);
+         }
+      }
+
+      return (true);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/UniqueKeyHelper.java

@@ -50,6 +50,7 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  *******************************************************************************/
 public class UniqueKeyHelper
 {
+   private static Integer pageSize = 1000;
 
    /*******************************************************************************
     **
@@ -60,62 +61,71 @@ public class UniqueKeyHelper
       Map<List<Serializable>, Serializable> existingRecords = new HashMap<>();
       if(ukFieldNames != null)
       {
-         QueryInput queryInput = new QueryInput();
-         queryInput.setTableName(table.getName());
-         queryInput.setTransaction(transaction);
+         for(List<QRecord> page : CollectionUtils.getPages(recordList, pageSize))
+         {
+            QueryInput queryInput = new QueryInput();
+            queryInput.setTableName(table.getName());
+            queryInput.setTransaction(transaction);
 
-         QQueryFilter filter = new QQueryFilter();
-         if(ukFieldNames.size() == 1)
-         {
-            List<Serializable> values = recordList.stream()
-               .filter(r -> CollectionUtils.nullSafeIsEmpty(r.getErrors()))
-               .map(r -> r.getValue(ukFieldNames.get(0)))
-               .collect(Collectors.toList());
-            filter.addCriteria(new QFilterCriteria(ukFieldNames.get(0), QCriteriaOperator.IN, values));
-         }
-         else
-         {
-            filter.setBooleanOperator(QQueryFilter.BooleanOperator.OR);
-            for(QRecord record : recordList)
+            QQueryFilter filter = new QQueryFilter();
+            if(ukFieldNames.size() == 1)
             {
-               if(CollectionUtils.nullSafeHasContents(record.getErrors()))
+               List<Serializable> values = page.stream()
+                  .filter(r -> CollectionUtils.nullSafeIsEmpty(r.getErrors()))
+                  .map(r -> r.getValue(ukFieldNames.get(0)))
+                  .collect(Collectors.toList());
+
+               if(values.isEmpty())
                {
                   continue;
                }
 
-               QQueryFilter subFilter = new QQueryFilter();
-               filter.addSubFilter(subFilter);
-               for(String fieldName : ukFieldNames)
+               filter.addCriteria(new QFilterCriteria(ukFieldNames.get(0), QCriteriaOperator.IN, values));
+            }
+            else
+            {
+               filter.setBooleanOperator(QQueryFilter.BooleanOperator.OR);
+               for(QRecord record : page)
                {
-                  Serializable value = record.getValue(fieldName);
-                  if(value == null)
+                  if(CollectionUtils.nullSafeHasContents(record.getErrors()))
                   {
-                     subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.IS_BLANK));
+                     continue;
                   }
-                  else
+
+                  QQueryFilter subFilter = new QQueryFilter();
+                  filter.addSubFilter(subFilter);
+                  for(String fieldName : ukFieldNames)
                   {
-                     subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.EQUALS, value));
+                     Serializable value = record.getValue(fieldName);
+                     if(value == null)
+                     {
+                        subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.IS_BLANK));
+                     }
+                     else
+                     {
+                        subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.EQUALS, value));
+                     }
                   }
                }
+
+               if(CollectionUtils.nullSafeIsEmpty(filter.getSubFilters()))
+               {
+                  ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  // if we didn't build any sub-filters (because all records have errors in them), don't run a query w/ no clauses - continue to next page //
+                  ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  continue;
+               }
             }
 
-            if(CollectionUtils.nullSafeIsEmpty(filter.getSubFilters()))
+            queryInput.setFilter(filter);
+            QueryOutput queryOutput = new QueryAction().execute(queryInput);
+            for(QRecord record : queryOutput.getRecords())
             {
-               ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-               // if we didn't build any sub-filters (because all records have errors in them), don't run a query w/ no clauses - rather - return early. //
-               ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-               return (existingRecords);
-            }
-         }
-
-         queryInput.setFilter(filter);
-         QueryOutput queryOutput = new QueryAction().execute(queryInput);
-         for(QRecord record : queryOutput.getRecords())
-         {
-            Optional<List<Serializable>> keyValues = getKeyValues(table, uniqueKey, record, allowNullKeyValuesToEqual);
-            if(keyValues.isPresent())
-            {
-               existingRecords.put(keyValues.get(), record.getValue(table.getPrimaryKeyField()));
+               Optional<List<Serializable>> keyValues = getKeyValues(table, uniqueKey, record, allowNullKeyValuesToEqual);
+               if(keyValues.isPresent())
+               {
+                  existingRecords.put(keyValues.get(), record.getValue(table.getPrimaryKeyField()));
+               }
             }
          }
       }
@@ -200,4 +210,26 @@ public class UniqueKeyHelper
       }
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for pageSize
+    **
+    *******************************************************************************/
+   public static Integer getPageSize()
+   {
+      return pageSize;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for pageSize
+    **
+    *******************************************************************************/
+   public static void setPageSize(Integer pageSize)
+   {
+      UniqueKeyHelper.pageSize = pageSize;
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/QValueFormatter.java

@@ -529,10 +529,16 @@ public class QValueFormatter
                   }
                }
 
-               /////////////////////////////////////////////
-               // if field type is blob, update its value //
-               /////////////////////////////////////////////
-               if(QFieldType.BLOB.equals(field.getType()))
+               ////////////////////////////////////////////////////////////////////////////////////////////////
+               // if field type is blob OR if there's a supplemental process or code-ref that needs to run - //
+               // then update its value to be a callback-url that'll give access to the bytes to download    //
+               // implied here is that a String value (w/o supplemental code/proc) has its value stay as a   //
+               // URL, which is where the file is directly downloaded from.  And in the case of a String     //
+               // with code-to-run, then the code should run, followed by a redirect to the value URL.       //
+               ////////////////////////////////////////////////////////////////////////////////////////////////
+               if(QFieldType.BLOB.equals(field.getType())
+                  || adornmentValues.containsKey(AdornmentType.FileDownloadValues.SUPPLEMENTAL_CODE_REFERENCE)
+                  || adornmentValues.containsKey(AdornmentType.FileDownloadValues.SUPPLEMENTAL_PROCESS_NAME))
                {
                   record.setValue(field.getName(), "/data/" + table.getName() + "/" + primaryKey + "/" + field.getName() + "/" + fileName);
                }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/AbstractMetaDataProducerBasedQQQApplication.java

@@ -29,7 +29,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 
 /*******************************************************************************
  ** Version of AbstractQQQApplication that assumes all meta-data is produced
- ** by MetaDataProducers in a single package.
+ ** by MetaDataProducers in (or under) a single package.
  *******************************************************************************/
 public abstract class AbstractMetaDataProducerBasedQQQApplication extends AbstractQQQApplication
 {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/ConfigFilesBasedQQQApplication.java

@@ -0,0 +1,61 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.instances.loaders.MetaDataLoaderHelper;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+
+
+/*******************************************************************************
+ ** Version of AbstractQQQApplication that assumes all meta-data is defined in
+ ** config files (yaml, json, etc) under a given directory path.
+ *******************************************************************************/
+public class ConfigFilesBasedQQQApplication extends AbstractQQQApplication
+{
+   private final String path;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public ConfigFilesBasedQQQApplication(String path)
+   {
+      this.path = path;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QInstance defineQInstance() throws QException
+   {
+      QInstance qInstance = new QInstance();
+      MetaDataLoaderHelper.processAllMetaDataFilesInDirectory(qInstance, path);
+      return (qInstance);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/MetaDataProducerBasedQQQApplication.java

@@ -0,0 +1,67 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances;
+
+
+/*******************************************************************************
+ ** Version of AbstractQQQApplication that assumes all meta-data is produced
+ ** by MetaDataProducers in (or under) a single package (where you can pass that
+ ** package into the constructor, vs. the abstract base class, where you extend
+ ** it and override the getMetaDataPackageName method.
+ *******************************************************************************/
+public class MetaDataProducerBasedQQQApplication extends AbstractMetaDataProducerBasedQQQApplication
+{
+   private final String metaDataPackageName;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public MetaDataProducerBasedQQQApplication(String metaDataPackageName)
+   {
+      this.metaDataPackageName = metaDataPackageName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public MetaDataProducerBasedQQQApplication(Class<?> aClassInMetaDataPackage)
+   {
+      this(aClassInMetaDataPackage.getPackageName());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getMetaDataPackageName()
+   {
+      return (this.metaDataPackageName);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceEnricher.java

@@ -289,7 +289,21 @@ public class QInstanceEnricher
 
       if(table.getFields() != null)
       {
-         table.getFields().values().forEach(this::enrichField);
+         for(Map.Entry<String, QFieldMetaData> entry : table.getFields().entrySet())
+         {
+            String         name  = entry.getKey();
+            QFieldMetaData field = entry.getValue();
+
+            ////////////////////////////////////////////////////////////////////////////
+            // in case the field wasn't given a name, use its key from the fields map //
+            ////////////////////////////////////////////////////////////////////////////
+            if(!StringUtils.hasContent(field.getName()))
+            {
+               field.setName(name);
+            }
+
+            enrichField(field);
+         }
 
          for(QSupplementalTableMetaData supplementalTableMetaData : CollectionUtils.nonNullMap(table.getSupplementalMetaData()).values())
          {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidator.java

@@ -70,6 +70,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldBehavior;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppChildMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
@@ -780,14 +781,37 @@ public class QInstanceValidator
          if(assertCondition(StringUtils.hasContent(association.getName()), "missing a name for an Association on table " + table.getName()))
          {
             String messageSuffix = " for Association " + association.getName() + " on table " + table.getName();
+            boolean recognizedTable = false;
             if(assertCondition(StringUtils.hasContent(association.getAssociatedTableName()), "missing associatedTableName" + messageSuffix))
             {
-               assertCondition(qInstance.getTable(association.getAssociatedTableName()) != null, "unrecognized associatedTableName " + association.getAssociatedTableName() + messageSuffix);
+               if(assertCondition(qInstance.getTable(association.getAssociatedTableName()) != null, "unrecognized associatedTableName " + association.getAssociatedTableName() + messageSuffix))
+               {
+                  recognizedTable = true;
+               }
             }
 
             if(assertCondition(StringUtils.hasContent(association.getJoinName()), "missing joinName" + messageSuffix))
             {
-               assertCondition(qInstance.getJoin(association.getJoinName()) != null, "unrecognized joinName " + association.getJoinName() + messageSuffix);
+               QJoinMetaData join = qInstance.getJoin(association.getJoinName());
+               if(assertCondition(join != null, "unrecognized joinName " + association.getJoinName() + messageSuffix))
+               {
+                  assert join != null; // covered by the assertCondition
+
+                  if(recognizedTable)
+                  {
+                     boolean isLeftToRight = join.getLeftTable().equals(table.getName()) && join.getRightTable().equals(association.getAssociatedTableName());
+                     boolean isRightToLeft = join.getRightTable().equals(table.getName()) && join.getLeftTable().equals(association.getAssociatedTableName());
+                     assertCondition(isLeftToRight || isRightToLeft, "join [" + association.getJoinName() + "] does not connect tables [" + table.getName() + "] and [" + association.getAssociatedTableName() + "]" + messageSuffix);
+                     if(isLeftToRight)
+                     {
+                        assertCondition(join.getType().equals(JoinType.ONE_TO_MANY) || join.getType().equals(JoinType.ONE_TO_ONE), "Join type does not have 'one' on this table's side side (left)" + messageSuffix);
+                     }
+                     else if(isRightToLeft)
+                     {
+                        assertCondition(join.getType().equals(JoinType.MANY_TO_ONE) || join.getType().equals(JoinType.ONE_TO_ONE), "Join type does not have 'one' on this table's side (right)" + messageSuffix);
+                     }
+                  }
+               }
             }
          }
       }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/AbstractMetaDataLoader.java

@@ -0,0 +1,510 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Serializable;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.YamlUtils;
+import org.apache.commons.io.IOUtils;
+import static com.kingsrook.qqq.backend.core.utils.ValueUtils.getValueAsInteger;
+import static com.kingsrook.qqq.backend.core.utils.ValueUtils.getValueAsString;
+
+
+/*******************************************************************************
+ ** Abstract base class in hierarchy of classes that know how to construct &
+ ** populate QMetaDataObject instances, based on input streams (e.g., from files).
+ *******************************************************************************/
+public abstract class AbstractMetaDataLoader<T extends QMetaDataObject>
+{
+   private static final QLogger LOG = QLogger.getLogger(AbstractMetaDataLoader.class);
+
+   private String fileName;
+
+   private List<LoadingProblem> problems = new ArrayList<>();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public T fileToMetaDataObject(QInstance qInstance, InputStream inputStream, String fileName) throws QMetaDataLoaderException
+   {
+      this.fileName = fileName;
+      Map<String, Object> map = fileToMap(inputStream, fileName);
+      LoadingContext loadingContext = new LoadingContext(fileName, "/");
+      return (mapToMetaDataObject(qInstance, map, loadingContext));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public abstract T mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   protected Map<String, Object> fileToMap(InputStream inputStream, String fileName) throws QMetaDataLoaderException
+   {
+      try
+      {
+         String string = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
+         string = StringUtils.ltrim(string);
+         if(fileName.toLowerCase().endsWith(".json"))
+         {
+            return JsonUtils.toObject(string, new TypeReference<>() {});
+         }
+         else if(fileName.toLowerCase().endsWith(".yaml") || fileName.toLowerCase().endsWith(".yml"))
+         {
+            return YamlUtils.toMap(string);
+         }
+
+         throw (new QMetaDataLoaderException("Unsupported file format (based on file name: " + fileName + ")"));
+      }
+      catch(IOException e)
+      {
+         throw new QMetaDataLoaderException("Error building map from file: " + fileName, e);
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   protected void reflectivelyMap(QInstance qInstance, QMetaDataObject targetObject, Map<String, Object> map, LoadingContext context)
+   {
+      Class<? extends QMetaDataObject> targetClass    = targetObject.getClass();
+      Set<String>                      usedFieldNames = new HashSet<>();
+
+      for(Method method : targetClass.getMethods())
+      {
+         try
+         {
+            if(method.getName().startsWith("set") && method.getParameterTypes().length == 1)
+            {
+               String propertyName = StringUtils.lcFirst(method.getName().substring(3));
+
+               if(map.containsKey(propertyName))
+               {
+                  usedFieldNames.add(propertyName);
+                  Class<?> parameterType = method.getParameterTypes()[0];
+                  Object   rawValue      = map.get(propertyName);
+
+                  try
+                  {
+                     Object mappedValue = reflectivelyMapValue(qInstance, method, parameterType, rawValue, context.descendToProperty(propertyName));
+                     method.invoke(targetObject, mappedValue);
+                  }
+                  catch(NoValueException nve)
+                  {
+                     ///////////////////////
+                     // don't call setter //
+                     ///////////////////////
+                     LOG.debug("at " + context + ": No value was mapped for property [" + propertyName + "] on " + targetClass.getSimpleName() + "." + method.getName() + ", raw value: [" + rawValue + "]");
+                  }
+               }
+            }
+         }
+         catch(Exception e)
+         {
+            addProblem(new LoadingProblem(context, "Error reflectively mapping on " + targetClass.getName() + "." + method.getName(), e));
+         }
+      }
+
+      //////////////////////////
+      // mmm, slightly sus... //
+      //////////////////////////
+      map.remove("class");
+      map.remove("version");
+
+      Set<String> unrecognizedKeys = new HashSet<>(map.keySet());
+      unrecognizedKeys.removeAll(usedFieldNames);
+
+      if(!unrecognizedKeys.isEmpty())
+      {
+         addProblem(new LoadingProblem(context, unrecognizedKeys.size() + " Unrecognized " + StringUtils.plural(unrecognizedKeys, "property", "properties") + ": " + unrecognizedKeys));
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public Object reflectivelyMapValue(QInstance qInstance, Method method, Class<?> parameterType, Object rawValue, LoadingContext context) throws Exception
+   {
+      if(rawValue instanceof String s && s.matches("^\\$\\{.+\\..+}"))
+      {
+         rawValue = new QMetaDataVariableInterpreter().interpret(s);
+         LOG.debug("Interpreted raw value [" + s + "] as [" + StringUtils.maskAndTruncate(ValueUtils.getValueAsString(rawValue) + "]"));
+      }
+
+      if(parameterType.equals(String.class))
+      {
+         return (getValueAsString(rawValue));
+      }
+      else if(parameterType.equals(Integer.class))
+      {
+         try
+         {
+            return (getValueAsInteger(rawValue));
+         }
+         catch(Exception e)
+         {
+            addProblem(new LoadingProblem(context, "[" + rawValue + "] is not an Integer value."));
+         }
+      }
+      else if(parameterType.equals(Boolean.class))
+      {
+         if("true".equals(rawValue) || Boolean.TRUE.equals(rawValue))
+         {
+            return (true);
+         }
+         else if("false".equals(rawValue) || Boolean.FALSE.equals(rawValue))
+         {
+            return (false);
+         }
+         else if(rawValue == null)
+         {
+            return (null);
+         }
+         else
+         {
+            addProblem(new LoadingProblem(context, "[" + rawValue + "] is not a boolean value (must be 'true' or 'false')."));
+            return (null);
+         }
+      }
+      else if(parameterType.equals(boolean.class))
+      {
+         if("true".equals(rawValue) || Boolean.TRUE.equals(rawValue))
+         {
+            return (true);
+         }
+         else if("false".equals(rawValue) || Boolean.FALSE.equals(rawValue))
+         {
+            return (false);
+         }
+         else
+         {
+            addProblem(new LoadingProblem(context, rawValue + " is not a boolean value (must be 'true' or 'false')."));
+            throw (new NoValueException());
+         }
+      }
+      else if(parameterType.equals(List.class))
+      {
+         Type     actualTypeArgument = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[0];
+         Class<?> actualTypeClass    = Class.forName(actualTypeArgument.getTypeName());
+
+         if(rawValue instanceof @SuppressWarnings("rawtypes")List valueList)
+         {
+            List<Object> mappedValueList = new ArrayList<>();
+            for(Object o : valueList)
+            {
+               try
+               {
+                  Object mappedValue = reflectivelyMapValue(qInstance, null, actualTypeClass, o, context);
+                  mappedValueList.add(mappedValue);
+               }
+               catch(NoValueException nve)
+               {
+                  // leave off list
+               }
+            }
+            return (mappedValueList);
+         }
+      }
+      else if(parameterType.equals(Set.class))
+      {
+         Type     actualTypeArgument = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[0];
+         Class<?> actualTypeClass    = Class.forName(actualTypeArgument.getTypeName());
+
+         if(rawValue instanceof @SuppressWarnings("rawtypes")List valueList)
+         {
+            Set<Object> mappedValueSet = new LinkedHashSet<>();
+            for(Object o : valueList)
+            {
+               try
+               {
+                  Object mappedValue = reflectivelyMapValue(qInstance, null, actualTypeClass, o, context);
+                  mappedValueSet.add(mappedValue);
+               }
+               catch(NoValueException nve)
+               {
+                  // leave off list
+               }
+            }
+            return (mappedValueSet);
+         }
+      }
+      else if(parameterType.equals(Map.class))
+      {
+         Type keyType = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[0];
+         if(!keyType.equals(String.class))
+         {
+            addProblem(new LoadingProblem(context, "Unsupported key type for " + method + " got [" + keyType + "], expected [String]"));
+            throw new NoValueException();
+         }
+         // todo make sure string
+
+         Type     actualTypeArgument = ((ParameterizedType) method.getGenericParameterTypes()[0]).getActualTypeArguments()[1];
+         Class<?> actualTypeClass    = Class.forName(actualTypeArgument.getTypeName());
+
+         if(rawValue instanceof @SuppressWarnings("rawtypes")Map valueMap)
+         {
+            Map<String, Object> mappedValueMap = new LinkedHashMap<>();
+            for(Object o : valueMap.entrySet())
+            {
+               try
+               {
+                  @SuppressWarnings("unchecked")
+                  Map.Entry<String, Object> entry = (Map.Entry<String, Object>) o;
+                  Object mappedValue = reflectivelyMapValue(qInstance, null, actualTypeClass, entry.getValue(), context);
+                  mappedValueMap.put(entry.getKey(), mappedValue);
+               }
+               catch(NoValueException nve)
+               {
+                  // leave out of map
+               }
+            }
+            return (mappedValueMap);
+         }
+      }
+      else if(parameterType.isEnum())
+      {
+         String value = getValueAsString(rawValue);
+         for(Object enumConstant : parameterType.getEnumConstants())
+         {
+            if(((Enum<?>) enumConstant).name().equals(value))
+            {
+               return (enumConstant);
+            }
+         }
+
+         addProblem(new LoadingProblem(context, "Unrecognized value [" + rawValue + "].  Expected one of: " + Arrays.toString(parameterType.getEnumConstants())));
+      }
+      else if(MetaDataLoaderRegistry.hasLoaderForClass(parameterType))
+      {
+         if(rawValue instanceof @SuppressWarnings("rawtypes")Map valueMap)
+         {
+            Class<? extends AbstractMetaDataLoader<?>> loaderClass = MetaDataLoaderRegistry.getLoaderForClass(parameterType);
+            AbstractMetaDataLoader<?>                  loader      = loaderClass.getConstructor().newInstance();
+            //noinspection unchecked
+            return (loader.mapToMetaDataObject(qInstance, valueMap, context));
+         }
+      }
+      else if(QMetaDataObject.class.isAssignableFrom(parameterType))
+      {
+         if(rawValue instanceof @SuppressWarnings("rawtypes")Map valueMap)
+         {
+            QMetaDataObject childObject = (QMetaDataObject) parameterType.getConstructor().newInstance();
+            //noinspection unchecked
+            reflectivelyMap(qInstance, childObject, valueMap, context);
+            return (childObject);
+         }
+      }
+      else if(parameterType.equals(Serializable.class))
+      {
+         if(rawValue instanceof String
+            || rawValue instanceof Integer
+            || rawValue instanceof BigDecimal
+            || rawValue instanceof Boolean
+         )
+         {
+            return rawValue;
+         }
+      }
+      else
+      {
+         // todo clean up this message/level
+         addProblem(new LoadingProblem(context, "No case for " + parameterType + " (arg to: " + method + ")"));
+      }
+
+      throw new NoValueException();
+   }
+
+   ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   // unclear if the below is needed.  if so, useful to not re-write, but is hurting test coverage, so zombie until used //
+   ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   ///***************************************************************************
+   // *
+   // ***************************************************************************/
+   //protected ListOfMapOrMapOfMap getListOfMapOrMapOfMap(Map<String, Object> map, String key)
+   //{
+   //   if(map.containsKey(key))
+   //   {
+   //      if(map.get(key) instanceof List)
+   //      {
+   //         return (new ListOfMapOrMapOfMap((List<Map<String, Object>>) map.get(key)));
+   //      }
+   //      else if(map.get(key) instanceof Map)
+   //      {
+   //         return (new ListOfMapOrMapOfMap((Map<String, Map<String, Object>>) map.get(key)));
+   //      }
+   //      else
+   //      {
+   //         LOG.warn("Expected list or map under key [" + key + "] while processing [" + getClass().getSimpleName() + "] from [" + fileName + "], but found: " + (map.get(key) == null ? "null" : map.get(key).getClass().getSimpleName()));
+   //      }
+   //   }
+
+   //   return (null);
+   //}
+
+   ///***************************************************************************
+   // *
+   // ***************************************************************************/
+   //protected List<Map<String, Object>> getListOfMap(Map<String, Object> map, String key)
+   //{
+   //   if(map.containsKey(key))
+   //   {
+   //      if(map.get(key) instanceof List)
+   //      {
+   //         return (List<Map<String, Object>>) map.get(key);
+   //      }
+   //      else
+   //      {
+   //         LOG.warn("Expected list under key [" + key + "] while processing [" + getClass().getSimpleName() + "] from [" + fileName + "], but found: " + (map.get(key) == null ? "null" : map.get(key).getClass().getSimpleName()));
+   //      }
+   //   }
+
+   //   return (null);
+   //}
+
+   ///***************************************************************************
+   // *
+   // ***************************************************************************/
+   //protected Map<String, Map<String, Object>> getMapOfMap(Map<String, Object> map, String key)
+   //{
+   //   if(map.containsKey(key))
+   //   {
+   //      if(map.get(key) instanceof Map)
+   //      {
+   //         return (Map<String, Map<String, Object>>) map.get(key);
+   //      }
+   //      else
+   //      {
+   //         LOG.warn("Expected map under key [" + key + "] while processing [" + getClass().getSimpleName() + "] from [" + fileName + "], but found: " + (map.get(key) == null ? "null" : map.get(key).getClass().getSimpleName()));
+   //      }
+   //   }
+
+   //   return (null);
+   //}
+
+   ///***************************************************************************
+   // **
+   // ***************************************************************************/
+   //protected record ListOfMapOrMapOfMap(List<Map<String, Object>> listOf, Map<String, Map<String, Object>> mapOf)
+   //{
+   //   /*******************************************************************************
+   //    ** Constructor
+   //    **
+   //    *******************************************************************************/
+   //   public ListOfMapOrMapOfMap(List<Map<String, Object>> listOf)
+   //   {
+   //      this(listOf, null);
+   //   }
+
+   //   /*******************************************************************************
+   //    ** Constructor
+   //    **
+   //    *******************************************************************************/
+   //   public ListOfMapOrMapOfMap(Map<String, Map<String, Object>> mapOf)
+   //   {
+   //      this(null, mapOf);
+   //   }
+   //}
+
+
+
+   /*******************************************************************************
+    ** Getter for fileName
+    **
+    *******************************************************************************/
+   public String getFileName()
+   {
+      return fileName;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static class NoValueException extends Exception
+   {
+      /*******************************************************************************
+       ** Constructor
+       **
+       *******************************************************************************/
+      public NoValueException()
+      {
+         super("No value");
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public void addProblem(LoadingProblem problem)
+   {
+      problems.add(problem);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for problems
+    **
+    *******************************************************************************/
+   public List<LoadingProblem> getProblems()
+   {
+      return (problems);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/ClassDetectingMetaDataLoader.java

@@ -0,0 +1,120 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.InputStream;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.GenericMetaDataLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.utils.ClassPathUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.AnyKey;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+
+
+/*******************************************************************************
+ ** Generic implementation of AbstractMetaDataLoader, who "detects" the class
+ ** of meta data object to be created, then defers to an appropriate subclass
+ ** to do the work.
+ *******************************************************************************/
+public class ClassDetectingMetaDataLoader extends AbstractMetaDataLoader<QMetaDataObject>
+{
+   private static final Memoization<AnyKey, List<Class<?>>> memoizedMetaDataObjectClasses = new Memoization<>();
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public AbstractMetaDataLoader<?> getLoaderForFile(InputStream inputStream, String fileName) throws QMetaDataLoaderException
+   {
+      Map<String, Object> map = fileToMap(inputStream, fileName);
+      return (getLoaderForMap(map));
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public AbstractMetaDataLoader<?> getLoaderForMap(Map<String, Object> map) throws QMetaDataLoaderException
+   {
+      if(map.containsKey("class"))
+      {
+         String classProperty = ValueUtils.getValueAsString(map.get("class"));
+         try
+         {
+            if(MetaDataLoaderRegistry.hasLoaderForSimpleName(classProperty))
+            {
+               Class<? extends AbstractMetaDataLoader<?>> loaderClass = MetaDataLoaderRegistry.getLoaderForSimpleName(classProperty);
+               return (loaderClass.getConstructor().newInstance());
+            }
+            else
+            {
+               Optional<List<Class<?>>> metaDataClasses = memoizedMetaDataObjectClasses.getResult(AnyKey.getInstance(), k -> ClassPathUtils.getClassesContainingNameAndOfType("MetaData", QMetaDataObject.class));
+               if(metaDataClasses.isEmpty())
+               {
+                  throw (new QMetaDataLoaderException("Could not get list of metaDataObjects from class loader"));
+               }
+
+               for(Class<?> c : metaDataClasses.get())
+               {
+                  if(c.getSimpleName().equals(classProperty) && QMetaDataObject.class.isAssignableFrom(c))
+                  {
+                     @SuppressWarnings("unchecked")
+                     Class<? extends QMetaDataObject> metaDataClass = (Class<? extends QMetaDataObject>) c;
+                     return new GenericMetaDataLoader<>(metaDataClass);
+                  }
+               }
+            }
+            throw new QMetaDataLoaderException("Unexpected class [" + classProperty + "] (not a QMetaDataObject; doesn't have a registered MetaDataLoader) specified in " + getFileName());
+         }
+         catch(QMetaDataLoaderException qmdle)
+         {
+            throw (qmdle);
+         }
+         catch(Exception e)
+         {
+            throw new QMetaDataLoaderException("Error handling class [" + classProperty + "] specified in " + getFileName(), e);
+         }
+      }
+      else
+      {
+         throw new QMetaDataLoaderException("Cannot detect meta-data type, because [class] attribute was not specified in file: " + getFileName());
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QMetaDataObject mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      AbstractMetaDataLoader<?> loaderForMap = getLoaderForMap(map);
+      return loaderForMap.mapToMetaDataObject(qInstance, map, context);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/LoadingContext.java

@@ -0,0 +1,38 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+/*******************************************************************************
+ ** Record to track where loader objects are - e.g., what file they're on,
+ ** and at what property path within the file (e.g., helps report problems).
+ *******************************************************************************/
+public record LoadingContext(String fileName, String propertyPath)
+{
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public LoadingContext descendToProperty(String propertyName)
+   {
+      return new LoadingContext(fileName, propertyPath + propertyName + "/");
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/LoadingProblem.java

@@ -0,0 +1,49 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+/*******************************************************************************
+ ** record that tracks a problem that was encountered when loading files.
+ *******************************************************************************/
+public record LoadingProblem(LoadingContext context, String message, Exception exception) // todo Level if useful
+{
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public LoadingProblem(LoadingContext context, String message)
+   {
+      this(context, message, null);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String toString()
+   {
+      return "at[" + context.fileName() + "][" + context.propertyPath() + "]: " + message;
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/MetaDataLoaderHelper.java

@@ -0,0 +1,118 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.Pair;
+
+
+/*******************************************************************************
+ ** class that loads a directory full of meta data files into meta data objects,
+ ** and then sets all of them in a QInstance.
+ *******************************************************************************/
+public class MetaDataLoaderHelper
+{
+   private static final QLogger LOG = QLogger.getLogger(MetaDataLoaderHelper.class);
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   public static void processAllMetaDataFilesInDirectory(QInstance qInstance, String path) throws QException
+   {
+      List<Pair<File, AbstractMetaDataLoader<?>>> loaders = new ArrayList<>();
+
+      File directory = new File(path);
+      processAllMetaDataFilesInDirectory(loaders, directory);
+
+      // todo - some version of sorting the loaders by type or possibly a sort field within the files (or file names)
+
+      for(Pair<File, AbstractMetaDataLoader<?>> pair : loaders)
+      {
+         File                      file   = pair.getA();
+         AbstractMetaDataLoader<?> loader = pair.getB();
+         try(FileInputStream fileInputStream = new FileInputStream(file))
+         {
+            QMetaDataObject qMetaDataObject = loader.fileToMetaDataObject(qInstance, fileInputStream, file.getName());
+
+            if(CollectionUtils.nullSafeHasContents(loader.getProblems()))
+            {
+               loader.getProblems().forEach(System.out::println);
+            }
+
+            if(qMetaDataObject instanceof TopLevelMetaDataInterface topLevelMetaData)
+            {
+               topLevelMetaData.addSelfToInstance(qInstance);
+            }
+            else
+            {
+               LOG.warn("Received a non-topLevelMetaDataObject from file: " + file.getAbsolutePath());
+            }
+         }
+         catch(Exception e)
+         {
+            LOG.error("Error processing file: " + file.getAbsolutePath(), e);
+         }
+      }
+   }
+
+
+
+   /***************************************************************************
+    *
+    ***************************************************************************/
+   private static void processAllMetaDataFilesInDirectory(List<Pair<File, AbstractMetaDataLoader<?>>> loaders, File directory) throws QException
+   {
+      for(File file : Objects.requireNonNullElse(directory.listFiles(), new File[0]))
+      {
+         if(file.isDirectory())
+         {
+            processAllMetaDataFilesInDirectory(loaders, file);
+         }
+         else
+         {
+            try(FileInputStream fileInputStream = new FileInputStream(file))
+            {
+               AbstractMetaDataLoader<?> loader = new ClassDetectingMetaDataLoader().getLoaderForFile(fileInputStream, file.getName());
+               loaders.add(Pair.of(file, loader));
+            }
+            catch(Exception e)
+            {
+               LOG.error("Error processing file: " + file.getAbsolutePath(), e);
+            }
+         }
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/MetaDataLoaderRegistry.java

@@ -0,0 +1,120 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.QTableMetaDataLoader;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.utils.ClassPathUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MetaDataLoaderRegistry
+{
+   private static final QLogger LOG = QLogger.getLogger(AbstractMetaDataLoader.class);
+
+   private static final Map<Class<?>, Class<? extends AbstractMetaDataLoader<?>>> registeredLoaders                   = new HashMap<>();
+   private static final Map<String, Class<? extends AbstractMetaDataLoader<?>>>   registeredLoadersByTargetSimpleName = new HashMap<>();
+
+   static
+   {
+      try
+      {
+         List<Class<?>> classesInPackage = ClassPathUtils.getClassesInPackage(QTableMetaDataLoader.class.getPackageName());
+         for(Class<?> possibleLoaderClass : classesInPackage)
+         {
+            try
+            {
+               Type superClass = possibleLoaderClass.getGenericSuperclass();
+               if(superClass.getTypeName().startsWith(AbstractMetaDataLoader.class.getName() + "<"))
+               {
+                  Type actualTypeArgument = ((ParameterizedType) superClass).getActualTypeArguments()[0];
+                  if(actualTypeArgument instanceof Class)
+                  {
+                     //noinspection unchecked
+                     Class<? extends AbstractMetaDataLoader<?>> loaderClass = (Class<? extends AbstractMetaDataLoader<?>>) possibleLoaderClass;
+
+                     Class<?> metaDataObjectType = Class.forName(actualTypeArgument.getTypeName());
+                     registeredLoaders.put(metaDataObjectType, loaderClass);
+                     registeredLoadersByTargetSimpleName.put(metaDataObjectType.getSimpleName(), loaderClass);
+                  }
+               }
+            }
+            catch(Exception e)
+            {
+               LOG.info("Error on class: " + possibleLoaderClass, e);
+            }
+         }
+
+         System.out.println("Registered loaders: " + registeredLoadersByTargetSimpleName);
+      }
+      catch(Exception e)
+      {
+         LOG.error("Error in static init block for MetaDataLoaderRegistry", e);
+      }
+   }
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static boolean hasLoaderForClass(Class<?> metaDataClass)
+   {
+      return registeredLoaders.containsKey(metaDataClass);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static Class<? extends AbstractMetaDataLoader<?>> getLoaderForClass(Class<?> metaDataClass)
+   {
+      return registeredLoaders.get(metaDataClass);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static boolean hasLoaderForSimpleName(String targetSimpleName)
+   {
+      return registeredLoadersByTargetSimpleName.containsKey(targetSimpleName);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static Class<? extends AbstractMetaDataLoader<?>> getLoaderForSimpleName(String targetSimpleName)
+   {
+      return registeredLoadersByTargetSimpleName.get(targetSimpleName);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/QMetaDataLoaderException.java

@@ -0,0 +1,50 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class QMetaDataLoaderException extends Exception
+{
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public QMetaDataLoaderException(String message)
+   {
+      super(message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public QMetaDataLoaderException(String message, Throwable cause)
+   {
+      super(message, cause);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/GenericMetaDataLoader.java

@@ -0,0 +1,71 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.AbstractMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class GenericMetaDataLoader<T extends QMetaDataObject> extends AbstractMetaDataLoader<T>
+{
+   private final Class<T> metaDataClass;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public GenericMetaDataLoader(Class<T> metaDataClass)
+   {
+      this.metaDataClass = metaDataClass;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public T mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      try
+      {
+         T object = metaDataClass.getConstructor().newInstance();
+         reflectivelyMap(qInstance, object, map, context);
+         return (object);
+      }
+      catch(Exception e)
+      {
+         throw (new QMetaDataLoaderException("Error loading metaData object of type " + metaDataClass.getSimpleName(), e));
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/QStepDataLoader.java

@@ -0,0 +1,85 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.AbstractMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class QStepDataLoader extends AbstractMetaDataLoader<QStepMetaData>
+{
+   private static final QLogger LOG = QLogger.getLogger(QStepDataLoader.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QStepMetaData mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      String stepType = ValueUtils.getValueAsString(map.get("stepType"));
+
+      if(!StringUtils.hasContent(stepType))
+      {
+         throw (new QMetaDataLoaderException("stepType was not specified for process step"));
+      }
+
+      QStepMetaData step;
+      if("backend".equalsIgnoreCase(stepType))
+      {
+         step = new QBackendStepMetaData();
+         reflectivelyMap(qInstance, step, map, context);
+      }
+      else if("frontend".equalsIgnoreCase(stepType))
+      {
+         step = new QFrontendStepMetaData();
+         reflectivelyMap(qInstance, step, map, context);
+      }
+      // todo - we have custom factory methods for this, so, maybe needs all custom loader?
+      // else if("stateMachine".equalsIgnoreCase(stepType))
+      // {
+      //    step = new QStateMachineStep();
+      //    reflectivelyMap(qInstance, step, map, context);
+      // }
+      else
+      {
+         throw (new QMetaDataLoaderException("Unsupported step stepType: " + stepType));
+      }
+
+      return (step);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/QTableMetaDataLoader.java

@@ -0,0 +1,58 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.instances.loaders.AbstractMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class QTableMetaDataLoader extends AbstractMetaDataLoader<QTableMetaData>
+{
+   private static final QLogger LOG = QLogger.getLogger(QTableMetaDataLoader.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QTableMetaData mapToMetaDataObject(QInstance qInstance, Map<String, Object> map, LoadingContext context) throws QMetaDataLoaderException
+   {
+      QTableMetaData table = new QTableMetaData();
+
+      reflectivelyMap(qInstance, table, map, context);
+
+      // todo - handle QTableBackendDetails, based on backend's type
+
+      return (table);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QFilterCriteria.java

@@ -31,6 +31,7 @@ import java.util.Objects;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.serialization.QFilterCriteriaDeserializer;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
@@ -40,7 +41,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  *
  *******************************************************************************/
 @JsonDeserialize(using = QFilterCriteriaDeserializer.class)
-public class QFilterCriteria implements Serializable, Cloneable
+public class QFilterCriteria implements Serializable, Cloneable, QMetaDataObject
 {
    private static final QLogger LOG = QLogger.getLogger(QFilterCriteria.class);
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QFilterOrderBy.java

@@ -23,13 +23,14 @@ package com.kingsrook.qqq.backend.core.model.actions.tables.query;
 
 
 import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** Bean representing an element of a query order-by clause.
  **
  *******************************************************************************/
-public class QFilterOrderBy implements Serializable, Cloneable
+public class QFilterOrderBy implements Serializable, Cloneable, QMetaDataObject
 {
    private String  fieldName;
    private boolean isAscending = true;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QQueryFilter.java

@@ -36,6 +36,7 @@ import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.AbstractFilterExpression;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.FilterVariableExpression;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
@@ -45,7 +46,7 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  * Full "filter" for a query - a list of criteria and order-bys
  *
  *******************************************************************************/
-public class QQueryFilter implements Serializable, Cloneable
+public class QQueryFilter implements Serializable, Cloneable, QMetaDataObject
 {
    private static final QLogger LOG = QLogger.getLogger(QQueryFilter.class);
 
@@ -55,6 +56,16 @@ public class QQueryFilter implements Serializable, Cloneable
    private BooleanOperator    booleanOperator = BooleanOperator.AND;
    private List<QQueryFilter> subFilters      = new ArrayList<>();
 
+   //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   // initial intent here was - put, e.g., UNION between multiple SELECT (with the individual selects being defined in subFilters) //
+   // but, actually SQL would let us do, e.g., SELECT UNION SELECT INTERSECT SELECT                                                //
+   // so - we could see a future implementation where we:                                                                          //
+   // - used the top-level subFilterSetOperator to indicate hat we are doing a multi-query set-operation query.                    //
+   // - looked within the subFilter, to see if it specified a subFilterSetOperator - and use that operator before that query       //
+   // but - in v0, just using the one at the top-level works                                                                       //
+   //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   private SubFilterSetOperator subFilterSetOperator = null;
+
    ////////////////////////////////////////////////////////////////////////////////////////////////////////////
    // skip & limit are meant to only apply to QueryAction (at least at the initial time they are added here) //
    // e.g., they are ignored in CountAction, AggregateAction, etc, where their meanings may be less obvious  //
@@ -75,6 +86,19 @@ public class QQueryFilter implements Serializable, Cloneable
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public enum SubFilterSetOperator
+   {
+      UNION,
+      UNION_ALL,
+      INTERSECT,
+      EXCEPT
+   }
+
+
+
    /*******************************************************************************
     ** Constructor
     **
@@ -799,4 +823,35 @@ public class QQueryFilter implements Serializable, Cloneable
       }
    }
 
+
+   /*******************************************************************************
+    ** Getter for subFilterSetOperator
+    *******************************************************************************/
+   public SubFilterSetOperator getSubFilterSetOperator()
+   {
+      return (this.subFilterSetOperator);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for subFilterSetOperator
+    *******************************************************************************/
+   public void setSubFilterSetOperator(SubFilterSetOperator subFilterSetOperator)
+   {
+      this.subFilterSetOperator = subFilterSetOperator;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for subFilterSetOperator
+    *******************************************************************************/
+   public QQueryFilter withSubFilterSetOperator(SubFilterSetOperator subFilterSetOperator)
+   {
+      this.subFilterSetOperator = subFilterSetOperator;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/dashboard/widgets/ChildRecordListData.java

@@ -46,6 +46,7 @@ public class ChildRecordListData extends QWidgetData
    private Boolean disableRowClick   = false;
    private Boolean allowRecordEdit   = false;
    private Boolean allowRecordDelete = false;
+   private Boolean isInProcess       = false;
 
    private boolean                   canAddChildRecord = false;
    private Map<String, Serializable> defaultValuesForNewChildRecords;
@@ -490,6 +491,38 @@ public class ChildRecordListData extends QWidgetData
       this.allowRecordDelete = allowRecordDelete;
       return (this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for isInProcess
+    *******************************************************************************/
+   public Boolean getIsInProcess()
+   {
+      return (this.isInProcess);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for isInProcess
+    *******************************************************************************/
+   public void setIsInProcess(Boolean isInProcess)
+   {
+      this.isInProcess = isInProcess;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for isInProcess
+    *******************************************************************************/
+   public ChildRecordListData withIsInProcess(Boolean isInProcess)
+   {
+      this.isInProcess = isInProcess;
+      return (this);
+   }
+
 }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducer.java

@@ -24,7 +24,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
 
 /*******************************************************************************
  ** Abstract class that knows how to produce meta data objects.  Useful with
- ** MetaDataProducerHelper, to put point at a package full of these, and populate
+ ** MetaDataProducerHelper, to point at a package full of these, and populate
  ** your whole QInstance.
  *******************************************************************************/
 public abstract class MetaDataProducer<T extends MetaDataProducerOutput> implements MetaDataProducerInterface<T>

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerHelper.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
 
 
 import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
 import java.lang.reflect.Modifier;
 import java.util.ArrayList;
 import java.util.Comparator;
@@ -31,10 +32,23 @@ import java.util.List;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.ChildJoinFromRecordEntityGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.PossibleValueSourceOfEnumGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.PossibleValueSourceOfTableGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildRecordListWidget;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildTable;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingPossibleValueEnum;
 import com.kingsrook.qqq.backend.core.utils.ClassPathUtils;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -90,6 +104,9 @@ public class MetaDataProducerHelper
       }
       List<MetaDataProducerInterface<?>> producers = new ArrayList<>();
 
+      ////////////////////////////////////////////////////////////////////////////////////////
+      // loop over classes, processing them based on either their type or their annotations //
+      ////////////////////////////////////////////////////////////////////////////////////////
       for(Class<?> aClass : classesInPackage)
       {
          try
@@ -101,23 +118,27 @@ public class MetaDataProducerHelper
 
             if(MetaDataProducerInterface.class.isAssignableFrom(aClass))
             {
-               boolean foundValidConstructor = false;
-               for(Constructor<?> constructor : aClass.getConstructors())
-               {
-                  if(constructor.getParameterCount() == 0)
-                  {
-                     Object o = constructor.newInstance();
-                     producers.add((MetaDataProducerInterface<?>) o);
-                     foundValidConstructor = true;
-                     break;
-                  }
-               }
+               CollectionUtils.addIfNotNull(producers, processMetaDataProducer(aClass));
+            }
 
-               if(!foundValidConstructor)
+            if(aClass.isAnnotationPresent(QMetaDataProducingEntity.class))
+            {
+               QMetaDataProducingEntity qMetaDataProducingEntity = aClass.getAnnotation(QMetaDataProducingEntity.class);
+               if(qMetaDataProducingEntity.producePossibleValueSource())
                {
-                  LOG.warn("Found a class which implements MetaDataProducerInterface, but it does not have a no-arg constructor, so it cannot be used.", logPair("class", aClass.getSimpleName()));
+                  producers.addAll(processMetaDataProducingEntity(aClass));
                }
             }
+
+            if(aClass.isAnnotationPresent(QMetaDataProducingPossibleValueEnum.class))
+            {
+               QMetaDataProducingPossibleValueEnum qMetaDataProducingPossibleValueEnum = aClass.getAnnotation(QMetaDataProducingPossibleValueEnum.class);
+               if(qMetaDataProducingPossibleValueEnum.producePossibleValueSource())
+               {
+                  CollectionUtils.addIfNotNull(producers, processMetaDataProducingPossibleValueEnum(aClass));
+               }
+            }
+
          }
          catch(Exception e)
          {
@@ -168,7 +189,176 @@ public class MetaDataProducerHelper
             LOG.debug("Not using producer which is not enabled", logPair("producer", producer.getClass().getSimpleName()));
          }
       }
-
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static <T extends PossibleValueEnum<T>> MetaDataProducerInterface<?> processMetaDataProducingPossibleValueEnum(Class<?> aClass)
+   {
+      String warningPrefix = "Found a class annotated as @" + QMetaDataProducingPossibleValueEnum.class.getSimpleName();
+      if(!PossibleValueEnum.class.isAssignableFrom(aClass))
+      {
+         LOG.warn(warningPrefix + ", but which is not a " + PossibleValueEnum.class.getSimpleName() + ", so it will not be used.", logPair("class", aClass.getSimpleName()));
+         return null;
+      }
+
+      PossibleValueEnum<?>[] values = (PossibleValueEnum<?>[]) aClass.getEnumConstants();
+      return (new PossibleValueSourceOfEnumGenericMetaDataProducer<T>(aClass.getSimpleName(), (PossibleValueEnum<T>[]) values));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static List<MetaDataProducerInterface<?>> processMetaDataProducingEntity(Class<?> aClass) throws Exception
+   {
+      List<MetaDataProducerInterface<?>> rs = new ArrayList<>();
+
+      String warningPrefix = "Found a class annotated as @" + QMetaDataProducingEntity.class.getSimpleName();
+      if(!QRecordEntity.class.isAssignableFrom(aClass))
+      {
+         LOG.warn(warningPrefix + ", but which is not a " + QRecordEntity.class.getSimpleName() + ", so it will not be used.", logPair("class", aClass.getSimpleName()));
+         return (rs);
+      }
+
+      Field tableNameField = aClass.getDeclaredField("TABLE_NAME");
+      if(!tableNameField.getType().equals(String.class))
+      {
+         LOG.warn(warningPrefix + ", but whose TABLE_NAME field is not a String, so it will not be used.", logPair("class", aClass.getSimpleName()));
+         return (rs);
+      }
+
+      String tableNameValue = (String) tableNameField.get(null);
+      rs.add(new PossibleValueSourceOfTableGenericMetaDataProducer(tableNameValue));
+
+      //////////////////////////
+      // process child tables //
+      //////////////////////////
+      QMetaDataProducingEntity qMetaDataProducingEntity = aClass.getAnnotation(QMetaDataProducingEntity.class);
+      for(ChildTable childTable : qMetaDataProducingEntity.childTables())
+      {
+         Class<? extends QRecordEntity> childEntityClass = childTable.childTableEntityClass();
+         if(childTable.childJoin().enabled())
+         {
+            CollectionUtils.addIfNotNull(rs, processChildJoin(aClass, childTable));
+
+            if(childTable.childRecordListWidget().enabled())
+            {
+               CollectionUtils.addIfNotNull(rs, processChildRecordListWidget(aClass, childTable));
+            }
+         }
+         else
+         {
+            if(childTable.childRecordListWidget().enabled())
+            {
+               //////////////////////////////////////////////////////////////////////////
+               // if not doing the join, can't do the child-widget, so warn about that //
+               //////////////////////////////////////////////////////////////////////////
+               LOG.warn(warningPrefix + " requested to produce a ChildRecordListWidget, but not produce a Join - which is not allowed (must do join to do widget). ", logPair("class", aClass.getSimpleName()), logPair("childEntityClass", childEntityClass.getSimpleName()));
+            }
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static MetaDataProducerInterface<?> processChildRecordListWidget(Class<?> aClass, ChildTable childTable) throws Exception
+   {
+      Class<? extends QRecordEntity> childEntityClass = childTable.childTableEntityClass();
+      String                         parentTableName  = getTableNameStaticFieldValue(aClass);
+      String                         childTableName   = getTableNameStaticFieldValue(childEntityClass);
+
+      ChildRecordListWidget childRecordListWidget = childTable.childRecordListWidget();
+      return (new ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer(childTableName, parentTableName, childRecordListWidget));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static String findPossibleValueField(Class<? extends QRecordEntity> entityClass, String possibleValueSourceName)
+   {
+      for(Field field : entityClass.getDeclaredFields())
+      {
+         if(field.isAnnotationPresent(QField.class))
+         {
+            QField qField = field.getAnnotation(QField.class);
+            if(qField.possibleValueSourceName().equals(possibleValueSourceName))
+            {
+               return field.getName();
+            }
+         }
+      }
+
+      return (null);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static MetaDataProducerInterface<?> processChildJoin(Class<?> aClass, ChildTable childTable) throws Exception
+   {
+      Class<? extends QRecordEntity> childEntityClass = childTable.childTableEntityClass();
+
+      String parentTableName        = getTableNameStaticFieldValue(aClass);
+      String childTableName         = getTableNameStaticFieldValue(childEntityClass);
+      String possibleValueFieldName = findPossibleValueField(childEntityClass, parentTableName);
+      if(!StringUtils.hasContent(possibleValueFieldName))
+      {
+         LOG.warn("Could not find field in [" + childEntityClass.getSimpleName() + "] with possibleValueSource referencing table [" + aClass.getSimpleName() + "]");
+         return (null);
+      }
+
+      return (new ChildJoinFromRecordEntityGenericMetaDataProducer(childTableName, parentTableName, possibleValueFieldName));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static MetaDataProducerInterface<?> processMetaDataProducer(Class<?> aClass) throws Exception
+   {
+      for(Constructor<?> constructor : aClass.getConstructors())
+      {
+         if(constructor.getParameterCount() == 0)
+         {
+            Object o = constructor.newInstance();
+            return (MetaDataProducerInterface<?>) o;
+         }
+      }
+
+      LOG.warn("Found a class which implements MetaDataProducerInterface, but it does not have a no-arg constructor, so it cannot be used.", logPair("class", aClass.getSimpleName()));
+      return null;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static String getTableNameStaticFieldValue(Class<?> aClass) throws NoSuchFieldException, IllegalAccessException
+   {
+      Field tableNameField = aClass.getDeclaredField("TABLE_NAME");
+      if(!tableNameField.getType().equals(String.class))
+      {
+         return (null);
+      }
+
+      String tableNameValue = (String) tableNameField.get(null);
+      return (tableNameValue);
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerInterface.java

@@ -27,12 +27,12 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
 
 /*******************************************************************************
  ** Interface for classes that know how to produce meta data objects.  Useful with
- ** MetaDataProducerHelper, to put point at a package full of these, and populate
+ ** MetaDataProducerHelper, to point at a package full of these, and populate
  ** your whole QInstance.
  **
  ** See also MetaDataProducer - an implementer of this interface, which actually
  ** came first, and is fine to extend if producing a meta-data class is all your
- ** clas means to do (nice and "Single-responsibility principle").
+ ** class means to do (nice and "Single-responsibility principle").
  **
  ** But, in some applications you may want to, for example, have one class that
  ** defines a process step, and also produces the meta-data for that process, so

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QInstance.java

@@ -1245,7 +1245,7 @@ public class QInstance
       {
          this.supplementalMetaData = new HashMap<>();
       }
-      this.supplementalMetaData.put(supplementalMetaData.getType(), supplementalMetaData);
+      this.supplementalMetaData.put(supplementalMetaData.getName(), supplementalMetaData);
       return (this);
    }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QMetaDataObject.java

@@ -0,0 +1,34 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata;
+
+
+import java.io.Serializable;
+
+
+/*******************************************************************************
+ ** interface common among all objects that can be considered qqq meta data -
+ ** e.g., stored in a QInstance.
+ *******************************************************************************/
+public interface QMetaDataObject extends Serializable
+{
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QSupplementalInstanceMetaData.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.model.metadata;
 
 
+import java.util.function.Supplier;
 import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 
@@ -30,20 +31,13 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
  ** Base-class for instance-level meta-data defined by some supplemental module, etc,
  ** outside of qqq core
  *******************************************************************************/
-public abstract class QSupplementalInstanceMetaData implements TopLevelMetaDataInterface
+public interface QSupplementalInstanceMetaData extends TopLevelMetaDataInterface
 {
 
-   /*******************************************************************************
-    ** Getter for type
-    *******************************************************************************/
-   public abstract String getType();
-
-
-
    /*******************************************************************************
     **
     *******************************************************************************/
-   public void enrich(QTableMetaData table)
+   default void enrich(QTableMetaData table)
    {
       ////////////////////////
       // noop in base class //
@@ -55,7 +49,7 @@ public abstract class QSupplementalInstanceMetaData implements TopLevelMetaDataI
    /*******************************************************************************
     **
     *******************************************************************************/
-   public void validate(QInstance qInstance, QInstanceValidator validator)
+   default void validate(QInstance qInstance, QInstanceValidator validator)
    {
       ////////////////////////
       // noop in base class //
@@ -68,9 +62,33 @@ public abstract class QSupplementalInstanceMetaData implements TopLevelMetaDataI
     **
     *******************************************************************************/
    @Override
-   public void addSelfToInstance(QInstance qInstance)
+   default void addSelfToInstance(QInstance qInstance)
    {
       qInstance.withSupplementalMetaData(this);
    }
 
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static <S extends QSupplementalInstanceMetaData> S of(QInstance qInstance, String name)
+   {
+      return ((S) qInstance.getSupplementalMetaData(name));
+   }
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static <S extends QSupplementalInstanceMetaData> S ofOrWithNew(QInstance qInstance, String name, Supplier<S> supplier)
+   {
+      S s = (S) qInstance.getSupplementalMetaData(name);
+      if(s == null)
+      {
+         s = supplier.get();
+         s.addSelfToInstance(qInstance);
+      }
+      return (s);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/TopLevelMetaDataInterface.java

@@ -26,7 +26,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
  ** Interface for meta-data classes that can be added directly (e.g, at the top
  ** level) to a QInstance (such as a QTableMetaData - not a QFieldMetaData).
  *******************************************************************************/
-public interface TopLevelMetaDataInterface extends MetaDataProducerOutput
+public interface TopLevelMetaDataInterface extends MetaDataProducerOutput, QMetaDataObject
 {
 
    /*******************************************************************************

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/audits/QAuditRules.java

@@ -22,10 +22,13 @@
 package com.kingsrook.qqq.backend.core.model.metadata.audits;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  **
  *******************************************************************************/
-public class QAuditRules
+public class QAuditRules implements QMetaDataObject
 {
    private AuditLevel auditLevel;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/code/QCodeReference.java

@@ -23,13 +23,14 @@ package com.kingsrook.qqq.backend.core.model.metadata.code;
 
 
 import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** Pointer to code to be ran by the qqq framework, e.g., for custom behavior -
  ** maybe process steps, maybe customization to a table, etc.
  *******************************************************************************/
-public class QCodeReference implements Serializable
+public class QCodeReference implements Serializable, QMetaDataObject
 {
    private String    name;
    private QCodeType codeType;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/AdornmentType.java

@@ -68,6 +68,9 @@ public enum AdornmentType
       String DEFAULT_EXTENSION = "defaultExtension";
       String DEFAULT_MIME_TYPE = "defaultMimeType";
 
+      String SUPPLEMENTAL_PROCESS_NAME   = "supplementalProcessName";
+      String SUPPLEMENTAL_CODE_REFERENCE = "supplementalCodeReference";
+
       ////////////////////////////////////////////////////
       // use these two together, as in:                 //
       // FILE_NAME_FORMAT = "Order %s Packing Slip.pdf" //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/QFieldMetaData.java

@@ -40,6 +40,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.data.QField;
 import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.help.HelpRole;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
@@ -54,7 +55,7 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
  ** Meta-data to represent a single field in a table.
  **
  *******************************************************************************/
-public class QFieldMetaData implements Cloneable
+public class QFieldMetaData implements Cloneable, QMetaDataObject
 {
    private static final QLogger LOG = QLogger.getLogger(QFieldMetaData.class);
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/QFieldType.java

@@ -120,6 +120,16 @@ public enum QFieldType
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public boolean isTemporal()
+   {
+      return this == QFieldType.DATE || this == QFieldType.DATE_TIME || this == QFieldType.TIME;
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/help/QHelpContent.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.help;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.Set;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
@@ -41,7 +42,7 @@ import java.util.Set;
  ** May be dynamically added to meta-data via (non-meta-) data - see
  ** HelpContentMetaDataProvider and QInstanceHelpContentManager
  *******************************************************************************/
-public class QHelpContent
+public class QHelpContent implements QMetaDataObject
 {
    private String        content;
    private HelpFormat    format;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QAppChildMetaData.java

@@ -22,11 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.layout;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** Interface shared by meta-data objects which can be placed into an App.
  ** e.g., Tables, Processes, and Apps themselves (since they can be nested)
  *******************************************************************************/
-public interface QAppChildMetaData
+public interface QAppChildMetaData extends QMetaDataObject
 {
    /*******************************************************************************
     **

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QAppSection.java

@@ -24,12 +24,13 @@ package com.kingsrook.qqq.backend.core.model.metadata.layout;
 
 import java.util.ArrayList;
 import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** A section of apps/tables/processes - a logical grouping.
  *******************************************************************************/
-public class QAppSection implements Cloneable
+public class QAppSection implements Cloneable, QMetaDataObject
 {
    private String name;
    private String label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QIcon.java

@@ -22,6 +22,9 @@
 package com.kingsrook.qqq.backend.core.model.metadata.layout;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** Icon to show associated with an App, Table, Process, etc.
  **
@@ -31,7 +34,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.layout;
  ** Future may allow something like a "namespace", and/or multiple icons for
  ** use in different frontends, etc.
  *******************************************************************************/
-public class QIcon
+public class QIcon implements QMetaDataObject
 {
    private String name;
    private String path;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/permissions/QPermissionRules.java

@@ -22,13 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.permissions;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 
 
 /*******************************************************************************
  **
  *******************************************************************************/
-public class QPermissionRules implements Cloneable
+public class QPermissionRules implements Cloneable, QMetaDataObject
 {
    private PermissionLevel level;
    private DenyBehavior    denyBehavior;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/possiblevalues/QPossibleValue.java

@@ -22,6 +22,9 @@
 package com.kingsrook.qqq.backend.core.model.metadata.possiblevalues;
 
 
+import java.util.Objects;
+
+
 /*******************************************************************************
  ** An actual possible value - an id and label.
  **
@@ -76,4 +79,37 @@ public class QPossibleValue<T>
    {
       return label;
    }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public boolean equals(Object o)
+   {
+      if(this == o)
+      {
+         return true;
+      }
+
+      if(o == null || getClass() != o.getClass())
+      {
+         return false;
+      }
+
+      QPossibleValue<?> that = (QPossibleValue<?>) o;
+      return Objects.equals(id, that.id) && Objects.equals(label, that.label);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public int hashCode()
+   {
+      return Objects.hash(id, label);
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/possiblevalues/QPossibleValueSource.java

@@ -23,7 +23,10 @@ package com.kingsrook.qqq.backend.core.model.metadata.possiblevalues;
 
 
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
@@ -97,7 +100,7 @@ public class QPossibleValueSource implements TopLevelMetaDataInterface
     ** Create a new possible value source, for an enum, with default settings.
     ** e.g., type=ENUM; name from param values from the param; LABEL_ONLY format
     *******************************************************************************/
-   public static <T extends PossibleValueEnum<?>> QPossibleValueSource newForEnum(String name, T[] values)
+   public static <I, T extends PossibleValueEnum<I>> QPossibleValueSource newForEnum(String name, T[] values)
    {
       return new QPossibleValueSource()
          .withName(name)
@@ -553,11 +556,25 @@ public class QPossibleValueSource implements TopLevelMetaDataInterface
     **   myPossibleValueSource.withValuesFromEnum(MyEnum.values()));
     **
     *******************************************************************************/
-   public <T extends PossibleValueEnum<?>> QPossibleValueSource withValuesFromEnum(T[] values)
+   public <I, T extends PossibleValueEnum<I>> QPossibleValueSource withValuesFromEnum(T[] values)
    {
+      Set<I> usedIds = new HashSet<>();
+      List<I> duplicatedIds = new ArrayList<>();
+
       for(T t : values)
       {
+         if(usedIds.contains(t.getPossibleValueId()))
+         {
+            duplicatedIds.add(t.getPossibleValueId());
+         }
+
          addEnumValue(new QPossibleValue<>(t.getPossibleValueId(), t.getPossibleValueLabel()));
+         usedIds.add(t.getPossibleValueId());
+      }
+
+      if(!duplicatedIds.isEmpty())
+      {
+         throw (new QRuntimeException("Error:  Duplicated id(s) found in enum values: " + duplicatedIds));
       }
 
       return (this);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QFrontendComponentMetaData.java

@@ -25,12 +25,13 @@ package com.kingsrook.qqq.backend.core.model.metadata.processes;
 import java.io.Serializable;
 import java.util.HashMap;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 
 
 /*******************************************************************************
  ** Definition of a UI component in a frontend process steps.
  *******************************************************************************/
-public class QFrontendComponentMetaData
+public class QFrontendComponentMetaData implements QMetaDataObject
 {
    private QComponentType type;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QProcessMetaData.java

@@ -320,12 +320,23 @@ public class QProcessMetaData implements QAppChildMetaData, MetaDataWithPermissi
 
 
    /*******************************************************************************
-    ** Setter for stepList
+    ** Setter for stepList - note - calling this method ALSO overwrites the steps map!
     **
     *******************************************************************************/
    public void setStepList(List<QStepMetaData> stepList)
    {
-      this.stepList = stepList;
+      if(stepList == null)
+      {
+         this.stepList = null;
+         this.steps = null;
+      }
+      else
+      {
+         this.stepList = new ArrayList<>();
+         this.steps = new HashMap<>();
+      }
+
+      withStepList(stepList);
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QStateMachineStep.java

@@ -185,4 +185,25 @@ public class QStateMachineStep extends QStepMetaData
       return (rs);
    }
 
+
+   /*******************************************************************************
+    ** Setter for subSteps
+    *******************************************************************************/
+   public void setSubSteps(List<QStepMetaData> subSteps)
+   {
+      this.subSteps = subSteps;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for subSteps
+    *******************************************************************************/
+   public QStateMachineStep withSubSteps(List<QStepMetaData> subSteps)
+   {
+      this.subSteps = subSteps;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QStepMetaData.java

@@ -26,6 +26,7 @@ import java.util.ArrayList;
 import java.util.List;
 import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.serialization.QStepMetaDataDeserializer;
 
@@ -37,7 +38,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.serialization.QStepMetaData
  **
  *******************************************************************************/
 @JsonDeserialize(using = QStepMetaDataDeserializer.class)
-public abstract class QStepMetaData
+public abstract class QStepMetaData implements QMetaDataObject
 {
    private String name;
    private String label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/ChildJoinFromRecordEntityGenericMetaDataProducer.java

@@ -0,0 +1,105 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a QJoinMetaData, based on a
+ ** QRecordEntity and a ChildTable sub-annotation.
+ **
+ ** e.g., Orders & LineItems - on the Order entity
+ ** <code>
+ @QMetaDataProducingEntity(
+    childTables = { @ChildTable(
+       childTableEntityClass = LineItem.class,
+       childJoin = @ChildJoin(enabled = true),
+       childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Order Lines"))
+    }
+ )
+ public class Order extends QRecordEntity
+ ** </code>
+ **
+ ** A Join will be made:
+ ** - left: Order
+ ** - right: LineItem
+ ** - type:  ONE_TO_MANY (one order (parent table) has mny lines (child table))
+ ** - joinOn:  order's primary key, lineItem's orderId field
+ ** - name:  inferred, based on the table names orderJoinLineItem)
+ *******************************************************************************/
+public class ChildJoinFromRecordEntityGenericMetaDataProducer implements MetaDataProducerInterface<QJoinMetaData>
+{
+   private String childTableName; // e.g., lineItem
+   private String parentTableName; // e.g., order
+   private String foreignKeyFieldName; // e.g., orderId
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public ChildJoinFromRecordEntityGenericMetaDataProducer(String childTableName, String parentTableName, String foreignKeyFieldName)
+   {
+      Objects.requireNonNull(childTableName, "childTableName cannot be null");
+      Objects.requireNonNull(parentTableName, "parentTableName cannot be null");
+      Objects.requireNonNull(foreignKeyFieldName, "foreignKeyFieldName cannot be null");
+
+      this.childTableName = childTableName;
+      this.parentTableName = parentTableName;
+      this.foreignKeyFieldName = foreignKeyFieldName;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QJoinMetaData produce(QInstance qInstance) throws QException
+   {
+      QTableMetaData possibleValueTable = qInstance.getTable(parentTableName);
+      if(possibleValueTable == null)
+      {
+         throw (new QException("Could not find tableMetaData " + parentTableName));
+      }
+
+      QJoinMetaData join = new QJoinMetaData()
+         .withLeftTable(parentTableName)
+         .withRightTable(childTableName)
+         .withInferredName()
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn(possibleValueTable.getPrimaryKeyField(), foreignKeyFieldName));
+
+      return (join);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer.java

@@ -0,0 +1,100 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildRecordListWidget;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a ChildRecordList QWidgetMetaData, to
+ ** produce a QJoinMetaData, based on a QRecordEntity and a ChildTable sub-annotation.
+ **
+ ** e.g., Orders & LineItems - on the Order entity
+ ** <code>
+ @QMetaDataProducingEntity( childTables = { @ChildTable(
+    childTableEntityClass = LineItem.class,
+    childJoin = @ChildJoin(enabled = true),
+    childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Order Lines"))
+ })
+ public class Order extends QRecordEntity
+ ** </code>
+ **
+ *******************************************************************************/
+public class ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer implements MetaDataProducerInterface<QWidgetMetaData>
+{
+   private String childTableName; // e.g., lineItem
+   private String parentTableName; // e.g., order
+
+   private ChildRecordListWidget childRecordListWidget;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer(String childTableName, String parentTableName, ChildRecordListWidget childRecordListWidget)
+   {
+      this.childTableName = childTableName;
+      this.parentTableName = parentTableName;
+      this.childRecordListWidget = childRecordListWidget;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QWidgetMetaData produce(QInstance qInstance) throws QException
+   {
+      String        name = QJoinMetaData.makeInferredJoinName(parentTableName, childTableName);
+      QJoinMetaData join = qInstance.getJoin(name);
+
+      QWidgetMetaData widget = ChildRecordListRenderer.widgetMetaDataBuilder(join)
+         .withName(name)
+         .withLabel(childRecordListWidget.label())
+         .withCanAddChildRecord(childRecordListWidget.canAddChildRecords())
+         .getWidgetMetaData();
+
+      if(StringUtils.hasContent(childRecordListWidget.manageAssociationName()))
+      {
+         widget.withDefaultValue("manageAssociationName", childRecordListWidget.manageAssociationName());
+      }
+
+      if(childRecordListWidget.maxRows() > 0)
+      {
+         widget.withDefaultValue("maxRows", childRecordListWidget.maxRows());
+      }
+
+      return (widget);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/PossibleValueSourceOfEnumGenericMetaDataProducer.java

@@ -0,0 +1,64 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+
+
+/***************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a QPossibleValueSource meta-data
+ ** based on a PossibleValueEnum
+ **
+ ***************************************************************************/
+public class PossibleValueSourceOfEnumGenericMetaDataProducer<T extends PossibleValueEnum<T>> implements MetaDataProducerInterface<QPossibleValueSource>
+{
+   private final String                 name;
+   private final PossibleValueEnum<T>[] values;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public PossibleValueSourceOfEnumGenericMetaDataProducer(String name, PossibleValueEnum<T>[] values)
+   {
+      this.name = name;
+      this.values = values;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QPossibleValueSource produce(QInstance qInstance)
+   {
+      return (QPossibleValueSource.newForEnum(name, values));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/PossibleValueSourceOfTableGenericMetaDataProducer.java

@@ -0,0 +1,61 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+
+
+/***************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a QPossibleValueSource meta-data
+ ** based on a QRecordEntity class (which has corresponding QTableMetaData).
+ **
+ ***************************************************************************/
+public class PossibleValueSourceOfTableGenericMetaDataProducer implements MetaDataProducerInterface<QPossibleValueSource>
+{
+   private final String tableName;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public PossibleValueSourceOfTableGenericMetaDataProducer(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QPossibleValueSource produce(QInstance qInstance)
+   {
+      return (QPossibleValueSource.newForTable(tableName));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/ChildJoin.java

@@ -0,0 +1,38 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+
+
+/***************************************************************************
+ ** value that goes inside a QMetadataProducingEntity annotation, to control
+ ** the generation of a QJoinMetaData
+ ***************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface ChildJoin
+{
+   boolean enabled();
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/ChildRecordListWidget.java

@@ -0,0 +1,46 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+
+
+/***************************************************************************
+ ** value that goes inside a QMetadataProducingEntity annotation, to control
+ ** the generation of a QWidgetMetaData - for a ChildRecordList widget.
+ ***************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface ChildRecordListWidget
+{
+   boolean enabled();
+
+   String label() default "";
+
+   int maxRows() default 20;
+
+   boolean canAddChildRecords() default false;
+
+   String manageAssociationName() default "";
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/ChildTable.java

@@ -0,0 +1,45 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/***************************************************************************
+ ** value that goes inside a QMetadataProducingEntity annotation, to define
+ ** child-tables, e.g., for producing joins and childRecordList widgets
+ ***************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface ChildTable
+{
+   Class<? extends QRecordEntity> childTableEntityClass();
+
+   String joinFieldName() default "";
+
+   ChildJoin childJoin() default @ChildJoin(enabled = false);
+
+   ChildRecordListWidget childRecordListWidget() default @ChildRecordListWidget(enabled = false);
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/QMetaDataProducingEntity.java

@@ -0,0 +1,48 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+
+/*******************************************************************************
+ ** annotation to go on a QRecordEntity class, which you would like to be
+ ** processed by MetaDataProducerHelper, to automatically produce some meta-data
+ ** objects.  Specifically supports:
+ **
+ ** - Making a possible-value-source out of the table.
+ ** - Processing child tables to create joins and childRecordList widgets
+ *******************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.TYPE)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface QMetaDataProducingEntity
+{
+   boolean producePossibleValueSource() default true;
+
+   ChildTable[] childTables() default { };
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/QMetaDataProducingPossibleValueEnum.java

@@ -0,0 +1,42 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+
+/*******************************************************************************
+ ** annotation to go on a PossibleValueEnum class, which you would like to be
+ ** processed by MetaDataProducerHelper, to automatically produce possible-value-
+ ** source meta-data based on the enum.
+ *******************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.TYPE)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface QMetaDataProducingPossibleValueEnum
+{
+   boolean producePossibleValueSource() default true;
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/scheduleing/QScheduleMetaData.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.model.metadata.scheduleing;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
 
@@ -35,7 +36,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  ** same moment.
  **
  *******************************************************************************/
-public class QScheduleMetaData
+public class QScheduleMetaData implements QMetaDataObject
 {
    private String schedulerName;
    private String description;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/Association.java

@@ -22,11 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.tables;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** definition of a qqq table that is "associated" with another table, e.g.,
  ** managed along with it - such as child-records under a parent record.
  *******************************************************************************/
-public class Association
+public class Association implements QMetaDataObject
 {
    private String name;
    private String associatedTableName;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/QFieldSection.java

@@ -26,6 +26,7 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Set;
 import com.kingsrook.qqq.backend.core.instances.QInstanceHelpContentManager;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.help.HelpRole;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
@@ -36,7 +37,7 @@ import com.kingsrook.qqq.backend.core.utils.collections.MutableList;
  ** A section of fields - a logical grouping.
  ** TODO - this class should be named QTableSection!
  *******************************************************************************/
-public class QFieldSection
+public class QFieldSection implements QMetaDataObject
 {
    private String name;
    private String label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/QTableMetaData.java

@@ -1329,6 +1329,21 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
 
 
 
+   /*******************************************************************************
+    ** Getter for an association by name
+    *******************************************************************************/
+   public Optional<Association> getAssociationByName(String name)
+   {
+      if(associations == null)
+      {
+         return (Optional.empty());
+      }
+
+      return (getAssociations().stream().filter(a -> a.getName().equals(name)).findFirst());
+   }
+
+
+
    /*******************************************************************************
     ** Setter for associations
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/UniqueKey.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
 
@@ -32,7 +33,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  ** Definition of a Unique Key (or "Constraint", if you wanna use fancy words)
  ** on a QTable.
  *******************************************************************************/
-public class UniqueKey
+public class UniqueKey implements QMetaDataObject
 {
    private List<String> fieldNames;
    private String       label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/AutomationStatusTracking.java

@@ -22,11 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.metadata.tables.automation;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+
+
 /*******************************************************************************
  ** Table-automation meta-data to define how this table's per-record automation
  ** status is tracked.
  *******************************************************************************/
-public class AutomationStatusTracking
+public class AutomationStatusTracking implements QMetaDataObject
 {
    private AutomationStatusTrackingType type;
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/QTableAutomationDetails.java

@@ -24,13 +24,14 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables.automation;
 
 import java.util.ArrayList;
 import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
 
 
 /*******************************************************************************
  ** Details about how this table's record automations are set up.
  *******************************************************************************/
-public class QTableAutomationDetails
+public class QTableAutomationDetails implements QMetaDataObject
 {
    private AutomationStatusTracking    statusTracking;
    private String                      providerName;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/TableAutomationAction.java

@@ -25,13 +25,14 @@ package com.kingsrook.qqq.backend.core.model.metadata.tables.automation;
 import java.io.Serializable;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 
 
 /*******************************************************************************
  ** Definition of a specific action to run against a table
  *******************************************************************************/
-public class TableAutomationAction
+public class TableAutomationAction implements QMetaDataObject
 {
    private String       name;
    private TriggerEvent triggerEvent;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/StreamedETLExecuteStep.java

@@ -189,7 +189,7 @@ public class StreamedETLExecuteStep extends BaseStreamedETLStep implements Backe
 
          if(recordCount > 0)
          {
-            LOG.info("Processed [" + recordCount + "] records.");
+            LOG.debug("Processed [" + recordCount + "] records.");
          }
 
          //////////////////////////////////////////////////////////////////////////////

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/tablesync/AbstractTableSyncTransformStep.java

@@ -223,7 +223,7 @@ public abstract class AbstractTableSyncTransformStep extends AbstractTransformSt
    {
       if(CollectionUtils.nullSafeIsEmpty(runBackendStepInput.getRecords()))
       {
-         LOG.info("No input records were found.");
+         LOG.debug("No input records were found.");
          return;
       }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/ClassPathUtils.java

@@ -61,6 +61,39 @@ public class ClassPathUtils
 
 
 
+   /*******************************************************************************
+    ** from https://stackoverflow.com/questions/520328/can-you-find-all-classes-in-a-package-using-reflection
+    **
+    *******************************************************************************/
+   public static List<Class<?>> getClassesContainingNameAndOfType(String nameContains, Class<?> type) throws IOException
+   {
+      List<Class<?>> classes = new ArrayList<>();
+      ClassLoader    loader  = Thread.currentThread().getContextClassLoader();
+
+      for(ClassPath.ClassInfo info : getTopLevelClasses(loader))
+      {
+         try
+         {
+            if(info.getName().contains(nameContains))
+            {
+               Class<?> testClass = info.load();
+               if(type.isAssignableFrom(testClass))
+               {
+                  classes.add(testClass);
+               }
+            }
+         }
+         catch(Throwable t)
+         {
+            // ignore - comes up for non-class entries, like module-info
+         }
+      }
+
+      return (classes);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/CollectionUtils.java

@@ -700,4 +700,16 @@ public class CollectionUtils
       return (map.containsKey(key) && map.get(key) != null);
    }
 
+
+
+   /***************************************************************************
+    ** add an element to a collection, but, only if the element isn't null
+    ***************************************************************************/
+   public static <T, E extends T> void addIfNotNull(Collection<T> c, E element)
+   {
+      if(element != null)
+      {
+         c.add(element);
+      }
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/JsonUtils.java

@@ -30,11 +30,14 @@ import java.util.List;
 import java.util.Map;
 import java.util.function.Consumer;
 import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.core.JsonGenerator;
 import com.fasterxml.jackson.core.JsonProcessingException;
 import com.fasterxml.jackson.core.type.TypeReference;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.ObjectWriter;
 import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
 import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
@@ -54,6 +57,11 @@ public class JsonUtils
 {
    private static final QLogger LOG = QLogger.getLogger(JsonUtils.class);
 
+   //////////////////////////////////////////////////////////////////////
+   // see https://www.baeldung.com/jackson-map-null-values-or-null-key //
+   //////////////////////////////////////////////////////////////////////
+   public static NullKeyToEmptyStringSerializer nullKeyToEmptyStringSerializer = new NullKeyToEmptyStringSerializer();
+
 
 
    /*******************************************************************************
@@ -396,4 +404,41 @@ public class JsonUtils
       return (record);
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static class NullKeyToEmptyStringSerializer extends StdSerializer<Object>
+   {
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public NullKeyToEmptyStringSerializer()
+      {
+         this(null);
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public NullKeyToEmptyStringSerializer(Class<Object> t)
+      {
+         super(t);
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public void serialize(Object nullKey, JsonGenerator jsonGenerator, SerializerProvider unused) throws IOException
+      {
+         jsonGenerator.writeFieldName("");
+      }
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/StringUtils.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.utils;
 
 
 import java.util.Collection;
+import java.util.Collections;
 import java.util.List;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
@@ -460,4 +461,74 @@ public class StringUtils
       return (Pattern.matches("[a-f0-9]{8}(?:-[a-f0-9]{4}){4}[a-f0-9]{8}", s));
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String emptyToNull(String s)
+   {
+      if(!hasContent(s))
+      {
+         return (null);
+      }
+
+      return (s);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String maskAndTruncate(String value)
+   {
+      return (maskAndTruncate(value, "** MASKED **", 6, 4));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String maskAndTruncate(String value, String mask, int minLengthToMask, int charsToShowOnEnds)
+   {
+      if(!hasContent(value))
+      {
+         return ("");
+      }
+
+      if(value.length() < minLengthToMask || value.length() < 2 * charsToShowOnEnds)
+      {
+         return mask;
+      }
+
+      if(value.length() < charsToShowOnEnds * 3)
+      {
+         return (value.substring(0, charsToShowOnEnds) + mask);
+      }
+
+      return (value.substring(0, charsToShowOnEnds) + mask + value.substring(value.length() - charsToShowOnEnds));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String nCopies(int n, String s)
+   {
+      return (nCopiesWithGlue(n, s, ""));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String nCopiesWithGlue(int n, String s, String glue)
+   {
+      return (StringUtils.join(glue, Collections.nCopies(n, s)));
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/audits/AuditActionTest.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.actions.audits;
 
 
 import java.io.Serializable;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -37,6 +38,9 @@ import com.kingsrook.qqq.backend.core.model.actions.audits.AuditSingleInput;
 import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.security.MultiRecordSecurityLock;
+import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.model.session.QUser;
 import com.kingsrook.qqq.backend.core.processes.utils.GeneralProcessUtils;
@@ -95,14 +99,18 @@ class AuditActionTest extends BaseTest
       String userName = "John Doe";
       QContext.init(qInstance, new QSession().withUser(new QUser().withFullName(userName)));
 
-      int recordId = 1701;
+      int               recordId         = 1701;
+      QCollectingLogger collectingLogger = QLogger.activateCollectingLoggerForClass(AuditAction.class);
       AuditAction.execute(TestUtils.TABLE_NAME_ORDER, recordId, Map.of(), "Test Audit");
 
       ///////////////////////////////////////////////////////////////////
       // it should not throw, but it should also not insert the audit. //
       ///////////////////////////////////////////////////////////////////
       Optional<QRecord> auditRecord = GeneralProcessUtils.getRecordByField("audit", "recordId", recordId);
-      assertFalse(auditRecord.isPresent());
+      // assertFalse(auditRecord.isPresent());
+      assertTrue(auditRecord.isPresent());
+      assertThat(collectingLogger.getCollectedMessages()).anyMatch(clm -> clm.getMessage().contains("Missing securityKeyValue"));
+      QLogger.deactivateCollectingLoggerForClass(AuditAction.class);
 
       ////////////////////////////////////////////////////////////////////////////////////////////////
       // try again with a null value in the key - that should be ok - as at least you were thinking //
@@ -274,4 +282,118 @@ class AuditActionTest extends BaseTest
       assertEquals(MapBuilder.of(TestUtils.SECURITY_KEY_TYPE_STORE, null), auditSingleInput.getSecurityKeyValues());
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testGetRecordSecurityKeyValue()
+   {
+      QRecord record = new QRecord().withValue("red", 1).withValue("blue", 2).withValue("green", 3).withValue("white", 4);
+
+      RecordSecurityLock redLock        = new RecordSecurityLock().withSecurityKeyType("red").withFieldName("red");
+      RecordSecurityLock blueLock       = new RecordSecurityLock().withSecurityKeyType("blue").withFieldName("blue");
+      RecordSecurityLock greenLock      = new RecordSecurityLock().withSecurityKeyType("green").withFieldName("green");
+      RecordSecurityLock whiteWriteLock = new RecordSecurityLock().withSecurityKeyType("green").withFieldName("white").withLockScope(RecordSecurityLock.LockScope.WRITE);
+
+      QTableMetaData simpleLockTable = new QTableMetaData()
+         .withRecordSecurityLock(redLock);
+      assertEquals(Map.of("red", 1), AuditAction.getRecordSecurityKeyValues(simpleLockTable, record, Optional.empty()));
+
+      QTableMetaData writeOnlyLockTable = new QTableMetaData()
+         .withRecordSecurityLock(whiteWriteLock);
+      assertEquals(Collections.emptyMap(), AuditAction.getRecordSecurityKeyValues(writeOnlyLockTable, record, Optional.empty()));
+
+      QTableMetaData multiAndLockTable = new QTableMetaData()
+         .withRecordSecurityLock(new MultiRecordSecurityLock()
+            .withOperator(MultiRecordSecurityLock.BooleanOperator.AND)
+            .withLock(redLock)
+            .withLock(blueLock));
+      assertEquals(Map.of("red", 1, "blue", 2), AuditAction.getRecordSecurityKeyValues(multiAndLockTable, record, Optional.empty()));
+
+      QTableMetaData multiOrLockTable = new QTableMetaData()
+         .withRecordSecurityLock(new MultiRecordSecurityLock()
+            .withOperator(MultiRecordSecurityLock.BooleanOperator.OR)
+            .withLock(redLock)
+            .withLock(blueLock));
+      assertEquals(Map.of("red", 1, "blue", 2), AuditAction.getRecordSecurityKeyValues(multiOrLockTable, record, Optional.empty()));
+
+      QTableMetaData multiLevelLockTable = new QTableMetaData()
+         .withRecordSecurityLock(new MultiRecordSecurityLock()
+            .withOperator(MultiRecordSecurityLock.BooleanOperator.AND)
+            .withLock(redLock)
+            .withLock(whiteWriteLock)
+            .withLock(new MultiRecordSecurityLock()
+               .withOperator(MultiRecordSecurityLock.BooleanOperator.OR)
+               .withLock(blueLock)
+               .withLock(greenLock)));
+      assertEquals(Map.of("red", 1, "blue", 2, "green", 3), AuditAction.getRecordSecurityKeyValues(multiLevelLockTable, record, Optional.empty()));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testValidateSecurityKeys()
+   {
+      RecordSecurityLock redLock        = new RecordSecurityLock().withSecurityKeyType("red").withFieldName("red");
+      RecordSecurityLock blueLock       = new RecordSecurityLock().withSecurityKeyType("blue").withFieldName("blue");
+      RecordSecurityLock greenLock      = new RecordSecurityLock().withSecurityKeyType("green").withFieldName("green");
+      RecordSecurityLock whiteWriteLock = new RecordSecurityLock().withSecurityKeyType("green").withFieldName("white").withLockScope(RecordSecurityLock.LockScope.WRITE);
+
+      AuditSingleInput inputWithNoKeys         = new AuditSingleInput().withSecurityKeyValues(Collections.emptyMap());
+      AuditSingleInput inputWithRedKey         = new AuditSingleInput().withSecurityKeyValues(Map.of("red", 1));
+      AuditSingleInput inputWithBlueKey        = new AuditSingleInput().withSecurityKeyValues(Map.of("blue", 2));
+      AuditSingleInput inputWithRedAndBlueKeys = new AuditSingleInput().withSecurityKeyValues(Map.of("red", 1, "blue", 2));
+
+      QTableMetaData noLockTable = new QTableMetaData();
+      assertTrue(AuditAction.validateSecurityKeys(inputWithNoKeys, noLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedKey, noLockTable));
+
+      QTableMetaData simpleLockTable = new QTableMetaData()
+         .withRecordSecurityLock(redLock);
+      assertFalse(AuditAction.validateSecurityKeys(inputWithNoKeys, simpleLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedKey, simpleLockTable));
+      assertFalse(AuditAction.validateSecurityKeys(inputWithBlueKey, simpleLockTable));
+
+      QTableMetaData writeOnlyLockTable = new QTableMetaData()
+         .withRecordSecurityLock(whiteWriteLock);
+      assertTrue(AuditAction.validateSecurityKeys(inputWithNoKeys, writeOnlyLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedKey, writeOnlyLockTable));
+
+      QTableMetaData multiAndLockTable = new QTableMetaData()
+         .withRecordSecurityLock(new MultiRecordSecurityLock()
+            .withOperator(MultiRecordSecurityLock.BooleanOperator.AND)
+            .withLock(redLock)
+            .withLock(blueLock));
+      assertFalse(AuditAction.validateSecurityKeys(inputWithNoKeys, multiAndLockTable));
+      assertFalse(AuditAction.validateSecurityKeys(inputWithRedKey, multiAndLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedAndBlueKeys, multiAndLockTable));
+
+      QTableMetaData multiOrLockTable = new QTableMetaData()
+         .withRecordSecurityLock(new MultiRecordSecurityLock()
+            .withOperator(MultiRecordSecurityLock.BooleanOperator.OR)
+            .withLock(redLock)
+            .withLock(blueLock));
+      assertFalse(AuditAction.validateSecurityKeys(inputWithNoKeys, multiOrLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedKey, multiOrLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedAndBlueKeys, multiOrLockTable));
+
+      QTableMetaData multiLevelLockTable = new QTableMetaData()
+         .withRecordSecurityLock(new MultiRecordSecurityLock()
+            .withOperator(MultiRecordSecurityLock.BooleanOperator.AND)
+            .withLock(redLock)
+            .withLock(whiteWriteLock)
+            .withLock(new MultiRecordSecurityLock()
+               .withOperator(MultiRecordSecurityLock.BooleanOperator.OR)
+               .withLock(blueLock)
+               .withLock(greenLock)));
+      assertFalse(AuditAction.validateSecurityKeys(inputWithNoKeys, multiLevelLockTable));
+      assertFalse(AuditAction.validateSecurityKeys(inputWithRedKey, multiLevelLockTable));
+      assertTrue(AuditAction.validateSecurityKeys(inputWithRedAndBlueKeys, multiLevelLockTable));
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/UniqueKeyHelperTest.java

@@ -0,0 +1,123 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.tables.helpers;
+
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for UniqueKeyHelper 
+ *******************************************************************************/
+class UniqueKeyHelperTest extends BaseTest
+{
+   private static Integer originalPageSize;
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeAll
+   static void beforeAll()
+   {
+      originalPageSize = UniqueKeyHelper.getPageSize();
+      UniqueKeyHelper.setPageSize(5);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterAll
+   static void afterAll()
+   {
+      UniqueKeyHelper.setPageSize(originalPageSize);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeEach
+   @AfterEach
+   void beforeAndAfterEach()
+   {
+      MemoryRecordStore.fullReset();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testUniqueKey() throws QException
+   {
+      List<QRecord> recordsWithKey1Equals1AndKey2In1Through10 = List.of(
+         new QRecord().withValue("key1", 1).withValue("key2", 1),
+         new QRecord().withValue("key1", 1).withValue("key2", 2),
+         new QRecord().withValue("key1", 1).withValue("key2", 3),
+         new QRecord().withValue("key1", 1).withValue("key2", 4),
+         new QRecord().withValue("key1", 1).withValue("key2", 5),
+         new QRecord().withValue("key1", 1).withValue("key2", 6),
+         new QRecord().withValue("key1", 1).withValue("key2", 7),
+         new QRecord().withValue("key1", 1).withValue("key2", 8),
+         new QRecord().withValue("key1", 1).withValue("key2", 9),
+         new QRecord().withValue("key1", 1).withValue("key2", 10)
+      );
+
+      InsertInput insertInput = new InsertInput();
+      insertInput.setTableName(TestUtils.TABLE_NAME_TWO_KEYS);
+      insertInput.setRecords(recordsWithKey1Equals1AndKey2In1Through10);
+      InsertOutput insertOutput = new InsertAction().execute(insertInput);
+
+      MemoryRecordStore.resetStatistics();
+      MemoryRecordStore.setCollectStatistics(true);
+
+      QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_TWO_KEYS);
+      Map<List<Serializable>, Serializable> existingKeys = UniqueKeyHelper.getExistingKeys(null, table, recordsWithKey1Equals1AndKey2In1Through10, table.getUniqueKeys().get(0), false);
+      assertEquals(recordsWithKey1Equals1AndKey2In1Through10.size(), existingKeys.size());
+
+      assertEquals(2, MemoryRecordStore.getStatistics().get(MemoryRecordStore.STAT_QUERIES_RAN));
+   }
+
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/AbstractMetaDataProducerBasedQQQApplicationTest.java

@@ -43,6 +43,9 @@ class AbstractMetaDataProducerBasedQQQApplicationTest extends BaseTest
    {
       QInstance qInstance = new TestApplication().defineQInstance();
       assertEquals(1, qInstance.getTables().size());
+      assertEquals("fromProducer", qInstance.getTables().get("fromProducer").getName());
+      assertEquals(1, qInstance.getProcesses().size());
+      assertEquals("fromProducer", qInstance.getProcesses().get("fromProducer").getName());
    }
 
 

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/MetaDataProducerBasedQQQApplicationTest.java

@@ -0,0 +1,66 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.instances.producers.TestMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for MetaDataProducerBasedQQQApplication 
+ *******************************************************************************/
+class MetaDataProducerBasedQQQApplicationTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      QInstance qInstance = new MetaDataProducerBasedQQQApplication(getClass().getPackage().getName() + ".producers").defineQInstance();
+      assertEquals(1, qInstance.getTables().size());
+      assertEquals("fromProducer", qInstance.getTables().get("fromProducer").getName());
+      assertEquals(1, qInstance.getProcesses().size());
+      assertEquals("fromProducer", qInstance.getProcesses().get("fromProducer").getName());
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testConstructorThatTakeClass() throws QException
+   {
+      QInstance qInstance = new MetaDataProducerBasedQQQApplication(TestMetaDataProducer.class).defineQInstance();
+      assertEquals(1, qInstance.getTables().size());
+      assertEquals("fromProducer", qInstance.getTables().get("fromProducer").getName());
+      assertEquals(1, qInstance.getProcesses().size());
+      assertEquals("fromProducer", qInstance.getProcesses().get("fromProducer").getName());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidatorTest.java

@@ -2054,16 +2054,41 @@ public class QInstanceValidatorTest extends BaseTest
 
       assertValidationFailureReasons((qInstance -> qInstance.getTable(TestUtils.TABLE_NAME_ORDER).withAssociation(new Association().withName("myAssociation"))),
          "missing joinName for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER,
-         "missing associatedTableName for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER
-      );
+         "missing associatedTableName for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER);
 
       assertValidationFailureReasons((qInstance -> qInstance.getTable(TestUtils.TABLE_NAME_ORDER).withAssociation(new Association().withName("myAssociation").withJoinName("notAJoin").withAssociatedTableName(TestUtils.TABLE_NAME_LINE_ITEM))),
-         "unrecognized joinName notAJoin for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER
-      );
+         "unrecognized joinName notAJoin for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER);
 
       assertValidationFailureReasons((qInstance -> qInstance.getTable(TestUtils.TABLE_NAME_ORDER).withAssociation(new Association().withName("myAssociation").withJoinName("orderLineItem").withAssociatedTableName("notATable"))),
-         "unrecognized associatedTableName notATable for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER
-      );
+         "unrecognized associatedTableName notATable for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER);
+
+      //////////////////////////////////
+      // wrong join on an association //
+      //////////////////////////////////
+      assertValidationFailureReasons((qInstance ->
+      {
+         Association association = qInstance.getTable(TestUtils.TABLE_NAME_ORDER).getAssociationByName("orderLine").orElseThrow();
+         association.setJoinName("orderOrderExtrinsic");
+      }),
+         "join [orderOrderExtrinsic] does not connect tables [order] and [orderLine]");
+
+      //////////////////////////////////////////
+      // wrong table (doesn't match the join) //
+      //////////////////////////////////////////
+      assertValidationFailureReasons((qInstance ->
+      {
+         Association association = qInstance.getTable(TestUtils.TABLE_NAME_ORDER).getAssociationByName("orderLine").orElseThrow();
+         association.setAssociatedTableName(TestUtils.TABLE_NAME_ORDER_EXTRINSIC);
+      }),
+         "join [orderLineItem] does not connect tables [order] and [orderExtrinsic]");
+
+      //////////////////////////////
+      // invalid type on the join //
+      //////////////////////////////
+      assertValidationFailureReasons((qInstance -> qInstance.getJoin("orderLineItem").setType(JoinType.MANY_TO_MANY)),
+         "Join type does not have 'one' on this table's side side (left)");
+      assertValidationFailureReasons((qInstance -> qInstance.getJoin("orderLineItem").setType(JoinType.MANY_TO_ONE)),
+         "Join type does not have 'one' on this table's side side (left)");
    }
 
 
@@ -2323,7 +2348,7 @@ public class QInstanceValidatorTest extends BaseTest
       {
          int noOfReasons = actualReasons == null ? 0 : actualReasons.size();
          assertEquals(expectedReasons.length, noOfReasons, "Expected number of validation failure reasons.\nExpected reasons: " + String.join(",", expectedReasons)
-            + "\nActual reasons: " + (noOfReasons > 0 ? String.join("\n", actualReasons) : "--"));
+                                                           + "\nActual reasons: " + (noOfReasons > 0 ? String.join("\n", actualReasons) : "--"));
       }
 
       for(String reason : expectedReasons)
@@ -2451,6 +2476,7 @@ public class QInstanceValidatorTest extends BaseTest
    public static class ValidAuthCustomizer implements QAuthenticationModuleCustomizerInterface {}
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -2468,6 +2494,7 @@ public class QInstanceValidatorTest extends BaseTest
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/AbstractMetaDataLoaderTest.java

@@ -0,0 +1,136 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.GenericMetaDataLoader;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.QTableMetaDataLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for AbstractMetaDataLoader 
+ *******************************************************************************/
+class AbstractMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testVariousPropertyTypes() throws QMetaDataLoaderException
+   {
+      QProcessMetaData process = new GenericMetaDataLoader<>(QProcessMetaData.class).fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1
+         name: myProcess
+         tableName: someTable
+         maxInputRecords: 1
+         isHidden: true
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertEquals("myProcess", process.getName());
+      assertEquals("someTable", process.getTableName());
+      assertEquals(1, process.getMaxInputRecords());
+      assertTrue(process.getIsHidden());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testProblems() throws QMetaDataLoaderException
+   {
+      {
+         QTableMetaDataLoader loader = new QTableMetaDataLoader();
+         loader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+            class: QTableMetaData
+            version: 1.0
+            name: myTable
+            something: foo
+            isHidden: hi
+            icon:
+               name: account_tree
+               size: big
+               weight: bold
+            fields:
+               id:
+                  type: number
+            uniqueKeys: sure!
+            """, StandardCharsets.UTF_8), "myTable.yaml");
+
+         for(LoadingProblem problem : loader.getProblems())
+         {
+            System.out.println(problem);
+         }
+      }
+
+      {
+         GenericMetaDataLoader<QProcessMetaData> loader = new GenericMetaDataLoader<>(QProcessMetaData.class);
+         loader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+            class: QProcessMetaData
+            version: 1.0
+            name: myProcess
+            maxInputRecords: many
+            """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+         for(LoadingProblem problem : loader.getProblems())
+         {
+            System.out.println(problem);
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testEnvironmentValues() throws QMetaDataLoaderException
+   {
+      System.setProperty("myProcess.tableName", "someTable");
+      System.setProperty("myProcess.maxInputRecords", "47");
+
+      GenericMetaDataLoader<QProcessMetaData> loader = new GenericMetaDataLoader<>(QProcessMetaData.class);
+      QProcessMetaData processMetaData = loader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1.0
+         name: myProcess
+         tableName: ${prop.myProcess.tableName}
+         maxInputRecords: ${prop.myProcess.maxInputRecords}
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertEquals("someTable", processMetaData.getTableName());
+      assertEquals(47, processMetaData.getMaxInputRecords());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/ClassDetectingMetaDataLoaderTest.java

@@ -0,0 +1,115 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.QMetaDataObject;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for ClassDetectingMetaDataLoader 
+ *******************************************************************************/
+class ClassDetectingMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testBasicSuccess() throws QMetaDataLoaderException
+   {
+      QMetaDataObject qMetaDataObject = new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QTableMetaData
+         version: 1
+         name: myTable
+         backendName: someBackend
+         """, StandardCharsets.UTF_8), "myTable.yaml");
+
+      assertThat(qMetaDataObject).isInstanceOf(QTableMetaData.class);
+      QTableMetaData qTableMetaData = (QTableMetaData) qMetaDataObject;
+      assertEquals("myTable", qTableMetaData.getName());
+      assertEquals("someBackend", qTableMetaData.getBackendName());
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testProcess() throws QMetaDataLoaderException
+   {
+      QMetaDataObject qMetaDataObject = new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1
+         name: myProcess
+         tableName: someTable
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertThat(qMetaDataObject).isInstanceOf(QProcessMetaData.class);
+      QProcessMetaData qProcessMetaData = (QProcessMetaData) qMetaDataObject;
+      assertEquals("myProcess", qProcessMetaData.getName());
+      assertEquals("someTable", qProcessMetaData.getTableName());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testUnknownClassFails()
+   {
+      assertThatThrownBy(() -> new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: ya whatever
+         version: 1
+         name: myTable
+         """, StandardCharsets.UTF_8), "whatever.yaml"))
+         .isInstanceOf(QMetaDataLoaderException.class)
+         .hasMessageContaining("Unexpected class");
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testMissingClassAttributeFails()
+   {
+      assertThatThrownBy(() -> new ClassDetectingMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         version: 1
+         name: myTable
+         """, StandardCharsets.UTF_8), "aTable.yaml"))
+         .isInstanceOf(QMetaDataLoaderException.class)
+         .hasMessageContaining("[class] attribute was not specified");
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/MetaDataLoaderHelperTest.java

@@ -0,0 +1,111 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import org.apache.commons.io.FileUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for MetaDataLoaderHelper 
+ *******************************************************************************/
+class MetaDataLoaderHelperTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws Exception
+   {
+      Path tempDirectory = Files.createTempDirectory(getClass().getSimpleName());
+
+      writeFile("myTable", ".yaml", tempDirectory, """
+         class: QTableMetaData
+         version:  1
+         name: myTable
+         label: This is My Table
+         primaryKeyField:  id
+         fields:
+            id:
+               name: id
+               type: INTEGER
+            name:
+               name: name
+               type: STRING
+            createDate:
+               name: createDate
+               type: DATE_TIME
+         """);
+
+      writeFile("yourTable", ".yaml", tempDirectory, """
+         class: QTableMetaData
+         version:  1
+         name: yourTable
+         label: Someone else's table
+         primaryKeyField:  id
+         fields:
+            id:
+               name: id
+               type: INTEGER
+            name:
+               name: name
+               type: STRING
+         """);
+
+      QInstance qInstance = new QInstance();
+      MetaDataLoaderHelper.processAllMetaDataFilesInDirectory(qInstance, tempDirectory.toFile().getAbsolutePath());
+
+      assertEquals(2, qInstance.getTables().size());
+
+      QTableMetaData myTable = qInstance.getTable("myTable");
+      assertEquals("This is My Table", myTable.getLabel());
+      assertEquals(3, myTable.getFields().size());
+      assertEquals("id", myTable.getField("id").getName());
+      assertEquals(QFieldType.INTEGER, myTable.getField("id").getType());
+
+      QTableMetaData yourTable = qInstance.getTable("yourTable");
+      assertEquals("Someone else's table", yourTable.getLabel());
+      assertEquals(2, yourTable.getFields().size());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   void writeFile(String prefix, String suffix, Path directory, String content) throws IOException
+   {
+      FileUtils.writeStringToFile(File.createTempFile(prefix, suffix, directory.toFile()), content, StandardCharsets.UTF_8);
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/QProcessMetaDataLoaderTest.java

@@ -0,0 +1,103 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+
+/*******************************************************************************
+ ** Unit test for loading a QProcessMetaData (doesn't need its own loader yet,
+ ** but is still a valuable high-level test target).
+ *******************************************************************************/
+class QProcessMetaDataLoaderTest extends BaseTest
+{
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testYaml() throws QMetaDataLoaderException
+   {
+      ClassDetectingMetaDataLoader metaDataLoader = new ClassDetectingMetaDataLoader();
+      QProcessMetaData process = (QProcessMetaData) metaDataLoader.fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class:  QProcessMetaData
+         version:  1.0
+         name:  myProcess
+         stepList:
+         - name: myBackendStep
+           stepType: backend
+           code:
+             name: com.kingsrook.test.processes.MyBackendStep
+         - name: myFrontendStep
+           stepType: frontend
+           components:
+             - type: HELP_TEXT
+               values:
+                  foo: bar
+             - type: VIEW_FORM
+           viewFields:
+             - name: myField
+               type: STRING
+             - name: yourField
+               type: DATE
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      CollectionUtils.nonNullList(metaDataLoader.getProblems()).forEach(System.out::println);
+
+      assertEquals("myProcess", process.getName());
+      assertEquals(2, process.getAllSteps().size());
+
+      QBackendStepMetaData myBackendStep = process.getBackendStep("myBackendStep");
+      assertNotNull(myBackendStep, "myBackendStep should not be null");
+      // todo - propagate this? assertEquals("myBackendStep", myBackendStep.getName());
+      assertEquals("com.kingsrook.test.processes.MyBackendStep", myBackendStep.getCode().getName());
+
+      QFrontendStepMetaData myFrontendStep = process.getFrontendStep("myFrontendStep");
+      assertNotNull(myFrontendStep, "myFrontendStep should not be null");
+      assertEquals(2, myFrontendStep.getComponents().size());
+      assertEquals(QComponentType.HELP_TEXT, myFrontendStep.getComponents().get(0).getType());
+      assertEquals(Map.of("foo", "bar"), myFrontendStep.getComponents().get(0).getValues());
+      assertEquals(QComponentType.VIEW_FORM, myFrontendStep.getComponents().get(1).getType());
+
+      assertEquals(2, myFrontendStep.getViewFields().size());
+      assertEquals("myField", myFrontendStep.getViewFields().get(0).getName());
+      assertEquals(QFieldType.STRING, myFrontendStep.getViewFields().get(0).getType());
+      assertEquals("yourField", myFrontendStep.getViewFields().get(1).getName());
+      assertEquals(QFieldType.DATE, myFrontendStep.getViewFields().get(1).getType());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/QTableMetaDataLoaderTest.java

@@ -0,0 +1,202 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders;
+
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.implementations.QTableMetaDataLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.permissions.DenyBehavior;
+import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import com.kingsrook.qqq.backend.core.utils.YamlUtils;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+
+/*******************************************************************************
+ ** Unit test for QTableMetaDataLoader 
+ *******************************************************************************/
+class QTableMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   @Disabled("Not quite yet passing - is a good goal to get to though!")
+   void testToYaml() throws QMetaDataLoaderException
+   {
+      QTableMetaData expectedTable = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON_MEMORY);
+      String expectedYaml = YamlUtils.toYaml(expectedTable);
+      QTableMetaData actualTable = new QTableMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream(expectedYaml, StandardCharsets.UTF_8), "person.yaml");
+      String actualYaml = YamlUtils.toYaml(actualTable);
+      assertEquals(expectedYaml, actualYaml);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testYaml() throws QMetaDataLoaderException
+   {
+      QTableMetaData table = new QTableMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QTableMetaData
+         version: 1.0
+         name: myTable
+         icon:
+            name: account_tree
+         fields:
+            id:
+               name: id
+               type: INTEGER
+            name:
+               name: name
+               type: STRING
+         uniqueKeys:
+         -  label: Name
+            fieldNames:
+            - name
+         associations:
+         -  name: A1
+            associatedTableName: yourTable
+            joinName: myTableJoinYourTable
+         -  name: A2
+            associatedTableName: theirTable
+            joinName: myTableJoinTheirTable
+         permissionRules:
+            level: READ_WRITE_PERMISSIONS
+            denyBehavior: HIDDEN
+            permissionBaseName: myTablePermissions
+            customPermissionChecker:
+               name: com.kingsrook.SomeChecker
+               codeType: JAVA
+         ## todo recordSecurityLocks
+         ## todo auditRules
+         ## todo backendDetails
+         ## todo automationDetails
+         sections:
+         -  name: identity
+            label: Identity
+            icon:
+               name: badge
+            tier: T1
+            fieldNames:
+            -  id
+            -  firstName
+            -  lastName
+         customizers:
+            postQueryRecord:
+               name: com.kingsrook.SomePostQuery
+               codeType: JAVA
+            preDeleteRecord:
+               name: com.kingsrook.SomePreDelete
+               codeType: JAVA
+         disabledCapabilities:
+         -   TABLE_COUNT
+         -   QUERY_STATS
+         """, StandardCharsets.UTF_8), "myTable.yaml");
+
+      assertEquals("myTable", table.getName());
+
+      assertEquals(2, table.getFields().size());
+      // assertEquals("id", table.getFields().get("id").getName());
+      assertEquals(QFieldType.INTEGER, table.getFields().get("id").getType());
+      // assertEquals("name", table.getFields().get("name").getName());
+      assertEquals(QFieldType.STRING, table.getFields().get("name").getType());
+
+      assertNotNull(table.getIcon());
+      assertEquals("account_tree", table.getIcon().getName());
+
+      assertEquals(1, table.getUniqueKeys().size());
+      assertEquals(List.of("name"), table.getUniqueKeys().get(0).getFieldNames());
+      assertEquals("Name", table.getUniqueKeys().get(0).getLabel());
+
+      assertEquals(2, table.getAssociations().size());
+      assertEquals("A1", table.getAssociations().get(0).getName());
+      assertEquals("theirTable", table.getAssociations().get(1).getAssociatedTableName());
+
+      assertNotNull(table.getPermissionRules());
+      assertEquals(PermissionLevel.READ_WRITE_PERMISSIONS, table.getPermissionRules().getLevel());
+      assertEquals(DenyBehavior.HIDDEN, table.getPermissionRules().getDenyBehavior());
+      assertEquals("myTablePermissions", table.getPermissionRules().getPermissionBaseName());
+      assertNotNull(table.getPermissionRules().getCustomPermissionChecker());
+      assertEquals("com.kingsrook.SomeChecker", table.getPermissionRules().getCustomPermissionChecker().getName());
+      assertEquals(QCodeType.JAVA, table.getPermissionRules().getCustomPermissionChecker().getCodeType());
+
+      assertEquals(1, table.getSections().size());
+      assertEquals("identity", table.getSections().get(0).getName());
+      assertEquals(Tier.T1, table.getSections().get(0).getTier());
+      assertEquals(List.of("id", "firstName", "lastName"), table.getSections().get(0).getFieldNames());
+
+      assertEquals(2, table.getCustomizers().size());
+      assertEquals("com.kingsrook.SomePostQuery", table.getCustomizers().get(TableCustomizers.POST_QUERY_RECORD.getRole()).getName());
+      assertEquals("com.kingsrook.SomePreDelete", table.getCustomizers().get(TableCustomizers.PRE_DELETE_RECORD.getRole()).getName());
+
+      assertEquals(Set.of(Capability.TABLE_COUNT, Capability.QUERY_STATS), table.getDisabledCapabilities());
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSimpleJson() throws QMetaDataLoaderException
+   {
+      QTableMetaData table = new QTableMetaDataLoader().fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         {
+            "class": "QTableMetaData",
+            "version": "1.0",
+            "name": "myTable",
+            "fields":
+            {
+               "id": {"name": "id", "type": "INTEGER"},
+               "name": {"name": "name", "type": "STRING"}
+            }
+         }
+         """, StandardCharsets.UTF_8), "myTable.json");
+
+      assertEquals("myTable", table.getName());
+      assertEquals(2, table.getFields().size());
+      assertEquals("id", table.getFields().get("id").getName());
+      assertEquals(QFieldType.INTEGER, table.getFields().get("id").getType());
+      assertEquals("name", table.getFields().get("name").getName());
+      assertEquals(QFieldType.STRING, table.getFields().get("name").getType());
+   }
+
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/loaders/implementations/GenericMetaDataLoaderTest.java

@@ -0,0 +1,81 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.loaders.implementations;
+
+
+import java.nio.charset.StandardCharsets;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.instances.loaders.LoadingContext;
+import com.kingsrook.qqq.backend.core.instances.loaders.QMetaDataLoaderException;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for GenericMetaDataLoader - providing coverage for AbstractMetaDataLoader.
+ *******************************************************************************/
+class GenericMetaDataLoaderTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testProcess() throws QMetaDataLoaderException
+   {
+      ////////////////////////////////////////////////////////////////////////////////
+      // trying to get some coverage of various types in here (for Abstract loader) //
+      ////////////////////////////////////////////////////////////////////////////////
+      QProcessMetaData process = new GenericMetaDataLoader<>(QProcessMetaData.class).fileToMetaDataObject(new QInstance(), IOUtils.toInputStream("""
+         class: QProcessMetaData
+         version: 1
+         name: myProcess
+         tableName: someTable
+         maxInputRecords: 1
+         isHidden: true
+         """, StandardCharsets.UTF_8), "myProcess.yaml");
+
+      assertEquals("myProcess", process.getName());
+      assertEquals("someTable", process.getTableName());
+      assertEquals(1, process.getMaxInputRecords());
+      assertTrue(process.getIsHidden());
+   }
+
+
+
+   /*******************************************************************************
+    ** just here for coverage of this class, as we're failing to hit it otherwise.
+    *******************************************************************************/
+   @SuppressWarnings({ "rawtypes", "unchecked" })
+   @Test
+   void testNoValueException()
+   {
+      assertThatThrownBy(() -> new GenericMetaDataLoader(QBackendMetaData.class).reflectivelyMapValue(new QInstance(), null, GenericMetaDataLoaderTest.class, "rawValue", new LoadingContext("test.yaml", "/")));
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/producers/subpackage/TestProcessMetaDataProducer.java

@@ -0,0 +1,47 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.instances.producers.subpackage;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class TestProcessMetaDataProducer implements MetaDataProducerInterface<QProcessMetaData>
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QProcessMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QProcessMetaData().withName("fromProducer");
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerHelperTest.java

@@ -23,14 +23,26 @@ package com.kingsrook.qqq.backend.core.model.metadata;
 
 
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestAbstractMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestDisabledMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestImplementsMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducingChildEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducingEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducingPossibleValueEnum;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestNoInterfacesExtendsObject;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestNoValidConstructorMetaDataProducer;
 import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
 
@@ -54,6 +66,48 @@ class MetaDataProducerHelperTest
       assertFalse(qInstance.getTables().containsKey(TestNoInterfacesExtendsObject.NAME));
       assertFalse(qInstance.getTables().containsKey(TestAbstractMetaDataProducer.NAME));
       assertFalse(qInstance.getTables().containsKey(TestDisabledMetaDataProducer.NAME));
+
+      /////////////////////////////////////////////
+      // annotation on PVS enum -> PVS meta data //
+      /////////////////////////////////////////////
+      assertTrue(qInstance.getPossibleValueSources().containsKey(TestMetaDataProducingPossibleValueEnum.class.getSimpleName()));
+      QPossibleValueSource enumPVS = qInstance.getPossibleValueSource(TestMetaDataProducingPossibleValueEnum.class.getSimpleName());
+      assertEquals(QPossibleValueSourceType.ENUM, enumPVS.getType());
+      assertEquals(2, enumPVS.getEnumValues().size());
+      assertEquals(new QPossibleValue<>(1, "One"), enumPVS.getEnumValues().get(0));
+
+      //////////////////////////////////////////////
+      // annotation on PVS table -> PVS meta data //
+      //////////////////////////////////////////////
+      assertTrue(qInstance.getPossibleValueSources().containsKey(TestMetaDataProducingEntity.TABLE_NAME));
+      QPossibleValueSource tablePVS = qInstance.getPossibleValueSource(TestMetaDataProducingEntity.TABLE_NAME);
+      assertEquals(QPossibleValueSourceType.TABLE, tablePVS.getType());
+      assertEquals(TestMetaDataProducingEntity.TABLE_NAME, tablePVS.getTableName());
+
+      //////////////////////////////////////////////////////////////////
+      // annotation on parent table w/ joined child -> join meta data //
+      //////////////////////////////////////////////////////////////////
+      String joinName = QJoinMetaData.makeInferredJoinName(TestMetaDataProducingEntity.TABLE_NAME, TestMetaDataProducingChildEntity.TABLE_NAME);
+      assertTrue(qInstance.getJoins().containsKey(joinName));
+      QJoinMetaData join = qInstance.getJoin(joinName);
+      assertEquals(TestMetaDataProducingEntity.TABLE_NAME, join.getLeftTable());
+      assertEquals(TestMetaDataProducingChildEntity.TABLE_NAME, join.getRightTable());
+      assertEquals(JoinType.ONE_TO_MANY, join.getType());
+      assertEquals("id", join.getJoinOns().get(0).getLeftField());
+      assertEquals("parentId", join.getJoinOns().get(0).getRightField());
+
+      //////////////////////////////////////////////////////////////////////////////////////
+      // annotation on parent table w/ joined child -> child record list widget meta data //
+      //////////////////////////////////////////////////////////////////////////////////////
+      assertTrue(qInstance.getWidgets().containsKey(joinName));
+      QWidgetMetaDataInterface widget = qInstance.getWidget(joinName);
+      assertEquals(WidgetType.CHILD_RECORD_LIST.getType(), widget.getType());
+      assertEquals("Test Children", widget.getLabel());
+      assertEquals(joinName, widget.getDefaultValues().get("joinName"));
+      assertEquals(false, widget.getDefaultValues().get("canAddChildRecord"));
+      assertNull(widget.getDefaultValues().get("manageAssociationName"));
+      assertEquals(15, widget.getDefaultValues().get("maxRows"));
+
    }
 
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/possiblevalues/QPossibleValueSourceTest.java

@@ -0,0 +1,96 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.possiblevalues;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Unit test for QPossibleValueSource 
+ *******************************************************************************/
+class QPossibleValueSourceTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testWithValuesFromEnum()
+   {
+      assertThatThrownBy(() -> new QPossibleValueSource().withValuesFromEnum(DupeIds.values()))
+         .isInstanceOf(QRuntimeException.class)
+         .hasMessageContaining("Duplicated id(s)")
+         .hasMessageMatching(".*: \\[1]$");
+   }
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private enum DupeIds implements PossibleValueEnum<Integer>
+   {
+      ONE_A(1, "A"),
+      TWO_B(2, "B"),
+      ONE_C(1, "C");
+
+
+      private final int id;
+      private final String label;
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      DupeIds(int id, String label)
+      {
+         this.id = id;
+         this.label = label;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public Integer getPossibleValueId()
+      {
+         return id;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public String getPossibleValueLabel()
+      {
+         return label;
+      }
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/producers/TestMetaDataProducingChildEntity.java

@@ -0,0 +1,140 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ ** QRecord Entity for TestMetaDataProducingEntity table
+ *******************************************************************************/
+public class TestMetaDataProducingChildEntity extends QRecordEntity implements MetaDataProducerInterface<QTableMetaData>
+{
+   public static final String TABLE_NAME = "testMetaDataProducingChildEntity";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(possibleValueSourceName = TestMetaDataProducingEntity.TABLE_NAME)
+   private Integer parentId;
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QTableMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME)
+         .withFieldsFromEntity(TestMetaDataProducingChildEntity.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Default constructor
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that takes a QRecord
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for parentId
+    *******************************************************************************/
+   public Integer getParentId()
+   {
+      return (this.parentId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for parentId
+    *******************************************************************************/
+   public void setParentId(Integer parentId)
+   {
+      this.parentId = parentId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for parentId
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity withParentId(Integer parentId)
+   {
+      this.parentId = parentId;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/producers/TestMetaDataProducingEntity.java

@@ -0,0 +1,119 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildJoin;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildRecordListWidget;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildTable;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ ** QRecord Entity for TestMetaDataProducingEntity table
+ *******************************************************************************/
+@QMetaDataProducingEntity(producePossibleValueSource = true,
+   childTables =
+      {
+         @ChildTable(childTableEntityClass = TestMetaDataProducingChildEntity.class,
+            childJoin = @ChildJoin(enabled = true),
+            childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Test Children", maxRows = 15))
+      }
+)
+public class TestMetaDataProducingEntity extends QRecordEntity implements MetaDataProducerInterface<QTableMetaData>
+{
+   public static final String TABLE_NAME = "testMetaDataProducingEntity";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QTableMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME)
+         .withFieldsFromEntity(TestMetaDataProducingEntity.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Default constructor
+    *******************************************************************************/
+   public TestMetaDataProducingEntity()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that takes a QRecord
+    *******************************************************************************/
+   public TestMetaDataProducingEntity(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public TestMetaDataProducingEntity withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/producers/TestMetaDataProducingPossibleValueEnum.java

@@ -0,0 +1,74 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingPossibleValueEnum;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+@QMetaDataProducingPossibleValueEnum(producePossibleValueSource = true)
+public enum TestMetaDataProducingPossibleValueEnum implements PossibleValueEnum<Integer>
+{
+   ONE(1, "One"),
+   TWO(2, "Two");
+
+
+   private final int    id;
+   private final String label;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   TestMetaDataProducingPossibleValueEnum(int id, String label)
+   {
+      this.id = id;
+      this.label = label;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getPossibleValueLabel()
+   {
+      return label;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public Integer getPossibleValueId()
+   {
+      return id;
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/CollectionUtilsTest.java

@@ -26,10 +26,12 @@ import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Hashtable;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 import java.util.TreeMap;
 import java.util.function.Function;
 import com.google.gson.reflect.TypeToken;
@@ -618,4 +620,23 @@ class CollectionUtilsTest extends BaseTest
          4, Map.of("B", "B4")), output);
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testAddIfNotNull()
+   {
+      HashSet<String> s = new HashSet<>();
+      CollectionUtils.addIfNotNull(s, null);
+      assertEquals(Set.of(), s);
+
+      CollectionUtils.addIfNotNull(s, "");
+      assertEquals(Set.of(""), s);
+
+      CollectionUtils.addIfNotNull(s, "1");
+      assertEquals(Set.of("", "1"), s);
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/JsonUtilsTest.java

@@ -35,11 +35,13 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
 import org.json.JSONArray;
 import org.json.JSONException;
 import org.json.JSONObject;
 import org.junit.jupiter.api.Test;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -81,7 +83,7 @@ class JsonUtilsTest extends BaseTest
       {
          objectMapper.setSerializationInclusion(JsonInclude.Include.ALWAYS);
       });
-      
+
       assertThat(json).contains("""
          "values":{"foo":"Foo","bar":3.14159,"baz":null}""");
    }
@@ -318,4 +320,27 @@ class JsonUtilsTest extends BaseTest
       assertEquals("age", qQueryFilter.getOrderBys().get(0).getFieldName());
       assertTrue(qQueryFilter.getOrderBys().get(0).getIsAscending());
    }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testNullKeyInMap()
+   {
+      Map<Object, String> mapWithNullKey = MapBuilder.of(null, "foo");
+
+      //////////////////////////////////////////////////////
+      // assert default behavior throws with null map key //
+      //////////////////////////////////////////////////////
+      assertThatThrownBy(() -> JsonUtils.toJson(mapWithNullKey)).rootCause().hasMessageContaining("Null key for a Map not allowed in JSON");
+
+      ////////////////////////////////////////////////////////////////////////
+      // assert that the nullKeyToEmptyStringSerializer does what we expect //
+      ////////////////////////////////////////////////////////////////////////
+      assertEquals("""
+         {"":"foo"}""", JsonUtils.toJson(mapWithNullKey, mapper -> mapper.getSerializerProvider().setNullKeySerializer(JsonUtils.nullKeyToEmptyStringSerializer)));
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/StringUtilsTest.java

@@ -318,4 +318,91 @@ class StringUtilsTest extends BaseTest
       assertEquals("Apples were eaten", StringUtils.pluralFormat(2, "Apple{,s} {was,were} eaten"));
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testEmptyToNull()
+   {
+      assertNull(StringUtils.emptyToNull(null));
+      assertNull(StringUtils.emptyToNull(""));
+      assertNull(StringUtils.emptyToNull(" "));
+      assertNull(StringUtils.emptyToNull("  "));
+      assertEquals("a", StringUtils.emptyToNull("a"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testNCopies()
+   {
+      assertEquals("", StringUtils.nCopies(0, "a"));
+      assertEquals("a", StringUtils.nCopies(1, "a"));
+      assertEquals("aa", StringUtils.nCopies(2, "a"));
+      assertEquals("ab ab ab ", StringUtils.nCopies(3, "ab "));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testNCopiesWithGlue()
+   {
+      assertEquals("", StringUtils.nCopiesWithGlue(0, "a", ""));
+      assertEquals("", StringUtils.nCopiesWithGlue(0, "a", ","));
+      assertEquals("a", StringUtils.nCopiesWithGlue(1, "a", ","));
+      assertEquals("aa", StringUtils.nCopiesWithGlue(2, "a", ""));
+      assertEquals("a,a", StringUtils.nCopiesWithGlue(2, "a", ","));
+      assertEquals("ab ab ab", StringUtils.nCopiesWithGlue(3, "ab", " "));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testMaskAndTruncate()
+   {
+      assertEquals("", StringUtils.maskAndTruncate(null));
+      assertEquals("", StringUtils.maskAndTruncate(""));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("1"));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("12"));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("123"));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("1234"));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("12345"));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("123456"));
+      assertEquals("** MASKED **", StringUtils.maskAndTruncate("1234567"));
+      assertEquals("1234** MASKED **", StringUtils.maskAndTruncate("12345678"));
+      assertEquals("1234** MASKED **", StringUtils.maskAndTruncate("123456789"));
+      assertEquals("1234** MASKED **", StringUtils.maskAndTruncate("1234567890"));
+      assertEquals("1234** MASKED **", StringUtils.maskAndTruncate("12345678901"));
+      assertEquals("1234** MASKED **9012", StringUtils.maskAndTruncate("123456789012"));
+      assertEquals("1234** MASKED **6789", StringUtils.maskAndTruncate("123456789" + StringUtils.nCopies(100, "xyz") + "123456789"));
+
+      assertEquals("***", StringUtils.maskAndTruncate("12", "***", 3, 1));
+      assertEquals("1***3", StringUtils.maskAndTruncate("123", "***", 3, 1));
+      assertEquals("1***4", StringUtils.maskAndTruncate("1234", "***", 3, 1));
+      assertEquals("1***5", StringUtils.maskAndTruncate("12345", "***", 3, 1));
+      assertEquals("12***", StringUtils.maskAndTruncate("12345", "***", 3, 2));
+      assertEquals("12***56", StringUtils.maskAndTruncate("123456", "***", 3, 2));
+
+      assertEquals("***", StringUtils.maskAndTruncate("12", "***", 3, 4));
+      assertEquals("***", StringUtils.maskAndTruncate("123", "***", 3, 4));
+      assertEquals("***", StringUtils.maskAndTruncate("1234", "***", 3, 4));
+      assertEquals("***", StringUtils.maskAndTruncate("12345", "***", 3, 4));
+      assertEquals("***", StringUtils.maskAndTruncate("12345", "***", 3, 4));
+      assertEquals("***", StringUtils.maskAndTruncate("123456", "***", 3, 4));
+      assertEquals("1234***", StringUtils.maskAndTruncate("1234567890", "***", 3, 4));
+      assertEquals("1234***", StringUtils.maskAndTruncate("12345678901", "***", 3, 4));
+      assertEquals("1234***9012", StringUtils.maskAndTruncate("123456789012", "***", 3, 4));
+   }
+
 }

qqq-backend-module-rdbms/src/main/java/com/kingsrook/qqq/backend/module/rdbms/actions/AbstractRDBMSAction.java

@@ -972,10 +972,15 @@ public abstract class AbstractRDBMSAction
             {
                sql = Objects.requireNonNullElse(sql, "").toString()
                   .replaceAll("FROM ", "\nFROM\n   ")
+                  .replaceAll("UNION ", "\nUNION\n   ")
+                  .replaceAll("INTERSECT ", "\nINTERSECT\n   ")
+                  .replaceAll("EXCEPT ", "\nEXCEPT\n   ")
                   .replaceAll("INNER", "\n   INNER")
                   .replaceAll("LEFT", "\n   LEFT")
                   .replaceAll("RIGHT", "\n   RIGHT")
-                  .replaceAll("WHERE", "\nWHERE\n   ");
+                  .replaceAll("WHERE", "\nWHERE\n   ")
+                  .replaceAll("ORDER BY", "\nORDER BY\n   ")
+                  .replaceAll("GROUP BY", "\nGROUP BY\n   ");
             }
 
             if(System.getProperty("qqq.rdbms.logSQL.output", "logger").equalsIgnoreCase("system.out"))

qqq-backend-module-rdbms/src/main/java/com/kingsrook/qqq/backend/module/rdbms/actions/RDBMSAggregateAction.java

@@ -159,6 +159,10 @@ public class RDBMSAggregateAction extends AbstractRDBMSAction implements Aggrega
                         {
                            fieldType = QFieldType.DECIMAL;
                         }
+                        else if(field.getType().isTemporal() && (aggregate.getOperator().equals(AggregateOperator.COUNT)) || aggregate.getOperator().equals(AggregateOperator.COUNT_DISTINCT))
+                        {
+                           fieldType = QFieldType.INTEGER;
+                        }
                      }
 
                      if(fieldType != null)

qqq-backend-module-rdbms/src/main/java/com/kingsrook/qqq/backend/module/rdbms/actions/RDBMSQueryAction.java

@@ -36,7 +36,6 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
 import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
 import com.kingsrook.qqq.backend.core.actions.tables.helpers.ActionTimeoutHelper;
 import com.kingsrook.qqq.backend.core.context.QContext;
@@ -46,6 +45,7 @@ import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.JoinsContext;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
@@ -95,35 +95,10 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
          QTableMetaData table     = queryInput.getTable();
          String         tableName = queryInput.getTableName();
 
-         Selection     selection = makeSelection(queryInput);
-         StringBuilder sql = new StringBuilder(selection.selectClause());
+         List<Serializable> params    = new ArrayList<>();
+         Selection          selection = makeSelection(queryInput);
 
-         QQueryFilter filter       = clonedOrNewFilter(queryInput.getFilter());
-         JoinsContext joinsContext = new JoinsContext(QContext.getQInstance(), tableName, queryInput.getQueryJoins(), filter);
-
-         List<Serializable> params = new ArrayList<>();
-         sql.append(" FROM ").append(makeFromClause(QContext.getQInstance(), tableName, joinsContext, params));
-         sql.append(" WHERE ").append(makeWhereClause(joinsContext, filter, params));
-
-         if(filter != null && CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
-         {
-            sql.append(" ORDER BY ").append(makeOrderByClause(table, filter.getOrderBys(), joinsContext));
-         }
-
-         if(filter != null && filter.getLimit() != null)
-         {
-            sql.append(" LIMIT ").append(filter.getLimit());
-
-            if(filter.getSkip() != null)
-            {
-               // todo - other sql grammars?
-               sql.append(" OFFSET ").append(filter.getSkip());
-            }
-         }
-
-         // todo sql customization - can edit sql and/or param list
-
-         setSqlAndJoinsInQueryStat(sql, joinsContext);
+         StringBuilder sql = makeSQL(queryInput, selection, tableName, params, table);
 
          Connection connection;
          boolean    needToCloseConnection = false;
@@ -258,6 +233,99 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private StringBuilder makeSQL(QueryInput queryInput, Selection selection, String tableName, List<Serializable> params, QTableMetaData table) throws QException
+   {
+      QQueryFilter filter       = clonedOrNewFilter(queryInput.getFilter());
+      JoinsContext joinsContext = new JoinsContext(QContext.getQInstance(), tableName, queryInput.getQueryJoins(), filter);
+
+      StringBuilder sql = new StringBuilder();
+
+      if(filter != null && filter.getSubFilterSetOperator() != null && CollectionUtils.nullSafeHasContents(filter.getSubFilters()))
+      {
+         for(QQueryFilter subFilter : filter.getSubFilters())
+         {
+            if(!sql.isEmpty())
+            {
+               sql.append(" ").append(filter.getSubFilterSetOperator().name().replace('_', ' ')).append(" ");
+            }
+
+            sql.append(" (");
+            sql.append(selection.selectClause());
+            sql.append(" FROM ").append(makeFromClause(QContext.getQInstance(), tableName, joinsContext, params));
+            sql.append(" WHERE ").append(makeWhereClause(joinsContext, subFilter, params));
+            sql.append(") ");
+         }
+
+         if(CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
+         {
+            /////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // the base version of makeOrderByClause uses `table`.`column` style references - which don't work for //
+            // these kinds of queries... so, use this version, which does index-based ones (maybe we could/should  //
+            // switch to always use those?                                                                         //
+            // the best here might be, to alias all columns, and then use those aliases in both versions...        //
+            /////////////////////////////////////////////////////////////////////////////////////////////////////////
+            sql.append(" ORDER BY ").append(makeOrderByClauseForSubFilterSetOperationQuery(table, filter.getOrderBys(), joinsContext, selection));
+         }
+      }
+      else
+      {
+         sql.append(selection.selectClause());
+         sql.append(" FROM ").append(makeFromClause(QContext.getQInstance(), tableName, joinsContext, params));
+         sql.append(" WHERE ").append(makeWhereClause(joinsContext, filter, params));
+
+         if(filter != null && CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
+         {
+            sql.append(" ORDER BY ").append(makeOrderByClause(table, filter.getOrderBys(), joinsContext));
+         }
+      }
+
+      if(filter != null && filter.getLimit() != null)
+      {
+         sql.append(" LIMIT ").append(filter.getLimit());
+
+         if(filter.getSkip() != null)
+         {
+            // todo - other sql grammars?
+            sql.append(" OFFSET ").append(filter.getSkip());
+         }
+      }
+
+      // todo sql customization - can edit sql and/or param list
+
+      setSqlAndJoinsInQueryStat(sql, joinsContext);
+      return sql;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private String makeOrderByClauseForSubFilterSetOperationQuery(QTableMetaData table, List<QFilterOrderBy> orderBys, JoinsContext joinsContext, Selection selection)
+   {
+      List<String> clauses = new ArrayList<>();
+
+      for(QFilterOrderBy orderBy : orderBys)
+      {
+         String                                ascOrDesc                     = orderBy.getIsAscending() ? "ASC" : "DESC";
+         JoinsContext.FieldAndTableNameOrAlias otherFieldAndTableNameOrAlias = joinsContext.getFieldAndTableNameOrAlias(orderBy.getFieldName());
+
+         QFieldMetaData field  = otherFieldAndTableNameOrAlias.field();
+         String         column = getColumnName(field);
+
+         String qualifiedColumn = escapeIdentifier(otherFieldAndTableNameOrAlias.tableNameOrAlias()) + "." + escapeIdentifier(column);
+         String columnNo        = String.valueOf(selection.qualifiedColumns.indexOf(qualifiedColumn) + 1);
+         clauses.add(columnNo + " " + ascOrDesc);
+
+      }
+      return (String.join(", ", clauses));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -282,10 +350,11 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
    /***************************************************************************
     ** output wrapper for makeSelection method.
     ** - selectClause is everything from SELECT up to (but not including) FROM
+    ** - qualifiedColumns is a list of the `table`.`column` strings
     ** - fields are those being selected, in the same order, and with mutated
     ** names for join fields.
     ***************************************************************************/
-   private record Selection(String selectClause, List<QFieldMetaData> fields)
+   private record Selection(String selectClause, List<String> qualifiedColumns, List<QFieldMetaData> fields)
    {
 
    }
@@ -318,10 +387,11 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
       // map those field names to columns, joined with ", ".                                                              //
       // if a field is heavy, and heavy fields aren't being selected, then replace that field name with a LENGTH function //
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      String columns = fieldList.stream()
+      List<String> qualifiedColumns = new ArrayList<>(fieldList.stream()
          .map(field -> Pair.of(field, escapeIdentifier(tableName) + "." + escapeIdentifier(getColumnName(field))))
          .map(pair -> wrapHeavyFieldsWithLengthFunctionIfNeeded(pair, queryInput.getShouldFetchHeavyFields()))
-         .collect(Collectors.joining(", "));
+         .toList());
+      String columns = String.join(", ", qualifiedColumns);
 
       ///////////////////////////////////////////////////////////////////////////////////////////////////////////
       // figure out if distinct is being used.  then start building the select clause with the table's columns //
@@ -360,10 +430,13 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
             /////////////////////////////////////////////////////
             // map to columns, wrapping heavy fields as needed //
             /////////////////////////////////////////////////////
-            String joinColumns = joinFieldList.stream()
+            List<String> qualifiedJoinColumns = joinFieldList.stream()
                .map(field -> Pair.of(field, escapeIdentifier(tableNameOrAlias) + "." + escapeIdentifier(getColumnName(field))))
                .map(pair -> wrapHeavyFieldsWithLengthFunctionIfNeeded(pair, queryInput.getShouldFetchHeavyFields()))
-               .collect(Collectors.joining(", "));
+               .toList();
+
+            qualifiedColumns.addAll(qualifiedJoinColumns);
+            String joinColumns = String.join(", ", qualifiedJoinColumns);
 
             ////////////////////////////////////////////////////////////////////////////////////////////////
             // append to output objects.                                                                  //
@@ -380,7 +453,7 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
          }
       }
 
-      return (new Selection(selectClause.toString(), selectionFieldList));
+      return (new Selection(selectClause.toString(), qualifiedColumns, selectionFieldList));
    }
 
 

qqq-backend-module-rdbms/src/test/java/com/kingsrook/qqq/backend/module/rdbms/actions/RDBMSAggregateActionTest.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.module.rdbms.actions;
 
 
 import java.math.BigDecimal;
+import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.List;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
@@ -42,13 +43,16 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.utils.ExceptionUtils;
 import com.kingsrook.qqq.backend.module.rdbms.TestUtils;
-import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
 import static org.junit.jupiter.api.Assertions.assertNull;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 import static org.junit.jupiter.api.Assertions.fail;
@@ -83,20 +87,56 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       Aggregate      averageOfDaysWorked = new Aggregate("daysWorked", AggregateOperator.AVG);
       Aggregate      maxAnnualSalary     = new Aggregate("annualSalary", AggregateOperator.MAX);
       Aggregate      minFirstName        = new Aggregate("firstName", AggregateOperator.MIN);
+      Aggregate      countOfBirthDate    = new Aggregate("birthDate", AggregateOperator.COUNT);
       aggregateInput.withAggregate(countOfId);
       aggregateInput.withAggregate(sumOfId);
       aggregateInput.withAggregate(averageOfDaysWorked);
       aggregateInput.withAggregate(maxAnnualSalary);
       aggregateInput.withAggregate(minFirstName);
+      aggregateInput.withAggregate(countOfBirthDate);
 
       AggregateOutput aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
 
       AggregateResult aggregateResult = aggregateOutput.getResults().get(0);
-      Assertions.assertEquals(5, aggregateResult.getAggregateValue(countOfId));
-      Assertions.assertEquals(15, aggregateResult.getAggregateValue(sumOfId));
-      Assertions.assertEquals(new BigDecimal("96.4"), aggregateResult.getAggregateValue(averageOfDaysWorked));
-      Assertions.assertEquals(new BigDecimal("1000000.00"), aggregateResult.getAggregateValue(maxAnnualSalary));
-      Assertions.assertEquals("Darin", aggregateResult.getAggregateValue(minFirstName));
+      assertEquals(5, aggregateResult.getAggregateValue(countOfId));
+      assertEquals(15, aggregateResult.getAggregateValue(sumOfId));
+      assertEquals(new BigDecimal("96.4"), aggregateResult.getAggregateValue(averageOfDaysWorked));
+      assertEquals(new BigDecimal("1000000.00"), aggregateResult.getAggregateValue(maxAnnualSalary));
+      assertEquals("Darin", aggregateResult.getAggregateValue(minFirstName));
+      assertEquals(4, aggregateResult.getAggregateValue(countOfBirthDate));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   @Disabled("Interesting to see effects of all operators on all types, but failures are expected (e.g., avg(string), so not for CI.")
+   void testOperatorsCrossTypes()
+   {
+      List<String> failures = new ArrayList<>();
+      for(QFieldMetaData field : QContext.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON).getFields().values())
+      {
+         for(AggregateOperator aggregateOperator : AggregateOperator.values())
+         {
+            try
+            {
+               AggregateInput aggregateInput = initAggregateRequest();
+               Aggregate      aggregate      = new Aggregate(field.getName(), aggregateOperator);
+               aggregateInput.withAggregate(aggregate);
+               AggregateOutput aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
+               AggregateResult aggregateResult = aggregateOutput.getResults().get(0);
+               assertNotNull(aggregateResult.getAggregateValue(aggregate));
+            }
+            catch(Exception e)
+            {
+               failures.add(ExceptionUtils.getRootException(e).getMessage());
+            }
+         }
+      }
+
+      failures.forEach(System.out::println);
    }
 
 
@@ -123,11 +163,11 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       AggregateOutput aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
 
       AggregateResult aggregateResult = aggregateOutput.getResults().get(0);
-      Assertions.assertEquals(2, aggregateResult.getAggregateValue(countOfId));
-      Assertions.assertEquals(5, aggregateResult.getAggregateValue(sumOfId));
-      Assertions.assertEquals(new BigDecimal("62.0"), aggregateResult.getAggregateValue(averageOfDaysWorked));
-      Assertions.assertEquals(new BigDecimal("26000.00"), aggregateResult.getAggregateValue(maxAnnualSalary));
-      Assertions.assertEquals("James", aggregateResult.getAggregateValue(minFirstName));
+      assertEquals(2, aggregateResult.getAggregateValue(countOfId));
+      assertEquals(5, aggregateResult.getAggregateValue(sumOfId));
+      assertEquals(new BigDecimal("62.0"), aggregateResult.getAggregateValue(averageOfDaysWorked));
+      assertEquals(new BigDecimal("26000.00"), aggregateResult.getAggregateValue(maxAnnualSalary));
+      assertEquals("James", aggregateResult.getAggregateValue(minFirstName));
    }
 
 
@@ -156,15 +196,15 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       AggregateOutput aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
       {
          AggregateResult aggregateResult = aggregateOutput.getResults().get(0);
-         Assertions.assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
-         Assertions.assertEquals(2, aggregateResult.getAggregateValue(countOfId));
-         Assertions.assertEquals(17, aggregateResult.getAggregateValue(sumOfDaysWorked));
+         assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
+         assertEquals(2, aggregateResult.getAggregateValue(countOfId));
+         assertEquals(17, aggregateResult.getAggregateValue(sumOfDaysWorked));
       }
       {
          AggregateResult aggregateResult = aggregateOutput.getResults().get(1);
-         Assertions.assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
-         Assertions.assertEquals(4, aggregateResult.getAggregateValue(countOfId));
-         Assertions.assertEquals(11364, aggregateResult.getAggregateValue(sumOfDaysWorked));
+         assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
+         assertEquals(4, aggregateResult.getAggregateValue(countOfId));
+         assertEquals(11364, aggregateResult.getAggregateValue(sumOfDaysWorked));
       }
    }
 
@@ -201,29 +241,29 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       AggregateResult           aggregateResult;
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals("Donny", aggregateResult.getGroupByValue(firstNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals("Donny", aggregateResult.getGroupByValue(firstNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals("Tim", aggregateResult.getGroupByValue(firstNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals("Tim", aggregateResult.getGroupByValue(firstNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals("Aaron", aggregateResult.getGroupByValue(firstNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals("Aaron", aggregateResult.getGroupByValue(firstNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals("Darin", aggregateResult.getGroupByValue(firstNameGroupBy));
-      Assertions.assertEquals(2, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals("Darin", aggregateResult.getGroupByValue(firstNameGroupBy));
+      assertEquals(2, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals("Trevor", aggregateResult.getGroupByValue(firstNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals("Trevor", aggregateResult.getGroupByValue(firstNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
    }
 
 
@@ -255,24 +295,24 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       AggregateResult           aggregateResult;
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals(4, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Kelkhoff", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals(4, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Richardson", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Richardson", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Maes", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Maes", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Samples", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals(1, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Samples", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals(1, aggregateResult.getAggregateValue(countOfId));
 
       aggregateResult = iterator.next();
-      Assertions.assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
-      Assertions.assertEquals(2, aggregateResult.getAggregateValue(countOfId));
+      assertEquals("Chamberlain", aggregateResult.getGroupByValue(lastNameGroupBy));
+      assertEquals(2, aggregateResult.getAggregateValue(countOfId));
    }
 
 
@@ -293,7 +333,7 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       ////////////////////////////////////////////////////////////
       AggregateOutput aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
       AggregateResult aggregateResult = aggregateOutput.getResults().get(0);
-      Assertions.assertEquals(0, aggregateResult.getAggregateValue(countOfId));
+      assertEquals(0, aggregateResult.getAggregateValue(countOfId));
 
       /////////////////////////////////////////////////////////////////////////////////////////
       // but re-run w/ a group-by -- then, if no rows are found, there are 0 result objects. //
@@ -324,12 +364,12 @@ public class RDBMSAggregateActionTest extends RDBMSActionTest
       QContext.setQSession(new QSession().withSecurityKeyValue(TestUtils.SECURITY_KEY_STORE_ALL_ACCESS, true));
       aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
       aggregateResult = aggregateOutput.getResults().get(0);
-      Assertions.assertEquals(43, aggregateResult.getAggregateValue(sumOfQuantity));
+      assertEquals(43, aggregateResult.getAggregateValue(sumOfQuantity));
 
       QContext.setQSession(new QSession().withSecurityKeyValue(TestUtils.TABLE_NAME_STORE, 1));
       aggregateOutput = new RDBMSAggregateAction().execute(aggregateInput);
       aggregateResult = aggregateOutput.getResults().get(0);
-      Assertions.assertEquals(33, aggregateResult.getAggregateValue(sumOfQuantity));
+      assertEquals(33, aggregateResult.getAggregateValue(sumOfQuantity));
    }
 
 

qqq-backend-module-rdbms/src/test/java/com/kingsrook/qqq/backend/module/rdbms/actions/RDBMSQueryActionSubFilterSetOperatorTest.java

@@ -0,0 +1,171 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.rdbms.actions;
+
+
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.module.rdbms.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** test for subfilter set
+ *******************************************************************************/
+public class RDBMSQueryActionSubFilterSetOperatorTest extends RDBMSActionTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeEach
+   public void beforeEach() throws Exception
+   {
+      super.primeTestDatabase();
+
+      // AbstractRDBMSAction.setLogSQL(true, true, "system.out");
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterEach
+   void afterEach()
+   {
+      AbstractRDBMSAction.setLogSQL(false);
+      QContext.getQSession().removeValue(QSession.VALUE_KEY_USER_TIMEZONE);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private QueryInput initQueryRequest()
+   {
+      QueryInput queryInput = new QueryInput();
+      queryInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+      return queryInput;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testUnion() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.UNION)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2, 3)))
+         .withOrderBy(new QFilterOrderBy("id", false))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(3, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(3, queryOutput.getRecords().get(0).getValueInteger("id"));
+      assertEquals(2, queryOutput.getRecords().get(1).getValueInteger("id"));
+      assertEquals(1, queryOutput.getRecords().get(2).getValueInteger("id"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testUnionAll() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.UNION_ALL)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2, 3)))
+         .withOrderBy(new QFilterOrderBy("id", false))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(4, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(3, queryOutput.getRecords().get(0).getValueInteger("id"));
+      assertEquals(2, queryOutput.getRecords().get(1).getValueInteger("id"));
+      assertEquals(2, queryOutput.getRecords().get(2).getValueInteger("id"));
+      assertEquals(1, queryOutput.getRecords().get(3).getValueInteger("id"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testIntersect() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.INTERSECT)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2, 3)))
+         .withOrderBy(new QFilterOrderBy("id", false))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(1, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(2, queryOutput.getRecords().get(0).getValueInteger("id"));
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testExcept() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.EXCEPT)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2, 3)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2)))
+         .withOrderBy(new QFilterOrderBy("id", true))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(2, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(1, queryOutput.getRecords().get(0).getValueInteger("id"));
+      assertEquals(3, queryOutput.getRecords().get(1).getValueInteger("id"));
+   }
+
+}