Merged dev into feature/process-locks-bulk

docs/Introduction.adoc

@@ -136,17 +136,13 @@ This speaks to the fact that this "code" is not executable code - but rather is
 **** The Filter button in the Query Screen will present a menu listing all fields from the table for the user to build ad-hoc queries against the table.
 The data-types specified for the fields (in the meta-data) dictate what operators QQQ allows the user to use against fields (e.g., Strings offer "contains" vs Numbers offer "greater than").
 **** Values for records from the table will be formatted for presentation based on the meta-data (such as a numeric field being shown with commas if it represents a quantity, or formatted as currency).
-...
+* Other kinds of information that you tell QQQ about in the form of meta-data objects includes:
-
+** Details about the database you are using, and how to connect to it.
-[start=2]
+** A database table's name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).
-. *Meta Data* - declarative code - java object instances (potentially which could be read from `.yaml` files or other data sources in a future version of QQQ), which tell QQQ about the backend systems, tables, processes, reports, widgets, etc, that make up the application.
+** The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.
-For example:
+** Details about a chart that summarizes data from a table for presentation as a dashboard widget.
-* Details about the database you are using, and how to connect to it.
+** The description of web API - its URL and authentication mechanism.
-* A database table's name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).
+** A table/path within a web API, and the fields returned in the JSON at that endpoint.
-* The description of web API - its URL and authentication mechanism.
-* A table/path within a web API, and the fields returned in the JSON at that endpoint.
-* The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.
-* Details about a chart that summarizes data from a table for presentation as a dashboard widget.
 // the section below is kinda dumb.  like, it says you have to write application code, but
 // then it just talks about how your app code gets for-free the same shit that QQQ does.
 // it should instead say more about what your custom app code is or does.
@@ -164,7 +160,8 @@ For example:
 // * The multi-threaded, paged producer/consumer pattern used in standard framework actions is how all custom application actions are also invoked.
 // ** For example, the standard QQQ Bulk Edit action uses the same streamed-ETL process that custom application processes can use.
 // Meaning your custom processes can take full advantage of the same complex frontend, middleware, and backend structural pieces, and you can just focus on your unique busines logic needs.
-2. *Application code* - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic.
+
+. *Application code* - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic.
 QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model.
 For example:
 

docs/index.adoc

@@ -6,7 +6,10 @@
 
 include::Introduction.adoc[leveloffset=+1]
 
-== Meta Data
+== Meta Data Production
+include::metaData/MetaDataProduction.adoc[leveloffset=+1]
+
+== Meta Data Types
 // Organizational units
 include::metaData/QInstance.adoc[leveloffset=+1]
 include::metaData/Backends.adoc[leveloffset=+1]

docs/metaData/MetaDataProduction.adoc

@@ -0,0 +1,362 @@
+[#MetaDataProduction]
+include::../variables.adoc[]
+
+The first thing that an application built using QQQ needs to do is to define its meta data.
+This basically means the construction of a `QInstance` object, which is populated with the
+meta data objects defining the backend(s), tables, processes, possible-value sources, joins,
+authentication provider, etc, that make your application.
+
+There are various styles that can be used for how you define your meta data, and for the
+most part they can be mixed and matched.  They will be presented here based on the historical
+evolution of how they were added to QQQ, where we generally believe that better techniques have
+been added over time.  So, you may wish to skip the earlier techniques, and jump straight to
+the end of this section.  However, it can always be instructive to learn about the past, so,
+read at your own pace.
+
+== Omni-Provider
+At the most basic level, the way to populate a `QInstance` is the simple and direct approach of creating
+one big class, possibly with just one big method, and just doing all the work directly inline there.
+
+This may (clearly) violate several good engineering principles.  However, it does have the benefit of
+being simple - if all of your meta-data is defined in one place, it can be pretty simple to find where
+that place is.  So - especially in a small project, this technique may be worth continuing to consider.
+
+Re: "doing all the work" as mentioned above - what work are we talking about?  At a minimum, we need
+to construct the following meta-data objects, and pass them into our `QInstance`:
+
+* `QAuthenticationMetaData` - how (or if!) users will be authenticated to the application.
+* `QBackendMeataData` - a backend data store.
+* `QTableMetaData` - a table (within the backend).
+* `QAppMetaData` - an organizational unit to present the other elements in a UI.
+
+Here's what a single-method omni-provider could look like:
+
+[source,java]
+.About the simplest possible single-file meta-data provider
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+
+    qInstance.setAuthentication(new QAuthenticationMetaData()
+       .withName("anonymous")
+       .withType(QAuthenticationType.FULLY_ANONYMOUS));
+
+    qInstance.addBackend(new QBackendMetaData()
+       .withBackendType(MemoryBackendModule.class)
+       .withName("memoryBackend"));
+
+    qInstance.addTable(new QTableMetaData()
+        .withName("myTable")
+        .withPrimaryKeyField("id")
+        .withBackendName("memoryBackend")
+        .withField(new QFieldMetaData("id", QFieldType.INTEGER)));
+
+    qInstance.addApp(new QAppMetaData()
+       .withName("myApp")
+       .withSectionOfChildren(new QAppSection().withName("mySection"),
+          qInstance.getTable("myTable")))
+
+   return (qInstance);
+}
+----
+
+== Multi-method Omni-Provider
+
+The next evolution of meta-data production comes by just applying some basic better-engineering
+principles, and splitting up from a single method that constructs all the things, to at least
+using unique methods to construct each thing, then calling those methods to add their results
+to the QInstance.
+
+[source,java]
+.Multi-method omni- meta-data provider
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+    qInstance.setAuthentication(defineAuthenticationMetaData());
+    qInstance.addBackend(defineBackendMetaData());
+    qInstance.addTable(defineMyTableMetaData());
+    qInstance.addApp(defineMyAppMetaData(qInstance));
+    return qInstance;
+}
+
+public QAuthenticationMetaData defineAuthenticationMetaData()
+{
+    return new QAuthenticationMetaData()
+       .withName("anonymous")
+       .withType(QAuthenticationType.FULLY_ANONYMOUS);
+}
+
+public QBackendMetaData defineBackendMetaData()
+{
+    return new QBackendMetaData()
+       .withBackendType(MemoryBackendModule.class)
+       .withName("memoryBackend");
+}
+
+// implementations of defineMyTableMetaData() and defineMyAppMetaData(qInstance)
+// left as an exercise for the reader
+----
+
+== Multi-class Providers
+
+Then the next logical evolution would be to put each of these single meta-data producing
+objects into its own class, along with calls to those classes.  This gets us away from the
+"5000 line" single-class, and lets us stop using the word "omni":
+
+[source,java]
+.Multi-class meta-data providers
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+    qInstance.setAuthentication(new AuthMetaDataProvider().defineAuthenticationMetaData());
+    qInstance.addBackend(new BackendMetaDataProvider().defineBackendMetaData());
+    qInstance.addTable(new MyTableMetaDataProvider().defineTableMetaData());
+    qInstance.addApp(new MyAppMetaDataProvider().defineAppMetaData(qInstance));
+    return qInstance;
+}
+
+public class AuthMetaDataProvider
+{
+    public QAuthenticationMetaData defineAuthenticationMetaData()
+    {
+        return new QAuthenticationMetaData()
+           .withName("anonymous")
+           .withType(QAuthenticationType.FULLY_ANONYMOUS);
+    }
+}
+
+public class BackendMetaDataProvider
+{
+    public QBackendMetaData defineBackendMetaData()
+    {
+        return new QBackendMetaData()
+           .withBackendType(MemoryBackendModule.class)
+           .withName("memoryBackend");
+    }
+}
+
+// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
+// left as an exercise for the reader
+----
+
+== MetaDataProducerInterface
+
+As the size of your application grows, if you're doing per-object meta-data providers, you may find it
+burdensome, when adding a new object to your instance, to have to write code for it in two places -
+that is - a new class to produce that meta-data object, AND a single line of code to add that object
+to your `QInstance`.  As such, a mechanism to avoid that line-of-code to add the object to the
+`QInstance` exists.
+
+This mechanism involves adding the `MetaDataProducerInterface` to all of your classes that produce a
+meta-data object.  This interface is generic, with a type parameter that will typically be the type of
+meta-data object you are producing, such as `QTableMetaData`, `QProcessMetaData`, or `QWidgetMetaData`,
+(technically, any class which implements `TopLevelMetaData`).  Implementers of the interface are then
+required to override just one method: `T produce(QInstance qInstance) throws QException;`
+
+Once you have your `MetaDataProducerInterface` classes defined, then there's a one-time call needed
+to add all of the objects produced by these classes to your `QInstance` - as shown here:
+
+[source,java]
+.Using MetaDataProducerInterface
+----
+public QInstance defineQInstance()
+{
+    QInstance qInstance = new QInstance();
+    MetaDataProducerHelper.processAllMetaDataProducersInPackage(qInstance,
+      "com.mydomain.myapplication");
+    return qInstance;
+}
+
+public class AuthMetaDataProducer implements MetaDataProducerInterface<QAuthenticationMetaData>
+{
+    @Override
+    public QAuthenticationMetaData produce(QInstance qInstance)
+    {
+        return new QAuthenticationMetaData()
+           .withName("anonymous")
+           .withType(QAuthenticationType.FULLY_ANONYMOUS);
+    }
+}
+
+public class BackendMetaDataProducer implements MetaDataProducerInterface<QBackendMetaData>
+{
+    @Override
+    public QBackendMetaData defineBackendMetaData()
+    {
+        return new QBackendMetaData()
+           .withBackendType(MemoryBackendModule.class)
+           .withName("memoryBackend");
+    }
+}
+
+// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
+// left as an exercise for the reader
+----
+
+=== MetaDataProducerMultiOutput
+It is worth mentioning, that sometimes it might feel like a bridge that's a bit too far, to make
+every single one of your meta-data objects require its own class.  Some may argue that it's best
+to do it that way - single responsibility principle, etc.  But, if you're producing, say, 5 widgets
+that are all related, and it's only a handful of lines of code for each one, maybe you'd rather
+produce them all in the same class.  Or maybe when you define a table, you'd like to define its
+joins and widgets at the same time.
+
+This approach can be accomplished by making your `MetaDataProducerInterface`'s type argument by
+`MetaDataProducerMultiOutput` - a simple class that just wraps a list of other `MetaDataProducerOutput`
+objects.
+
+[source,java]
+.Returning a MetaDataProducerMultiOutput
+----
+public class MyMultiProducer implements MetaDataProducerInterface<MetaDataProducerMultiOutput>
+{
+    @Override
+    public MetaDataProducerMultiOutput produce(QInstance qInstance)
+    {
+       MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput();
+
+       output.add(new QPossibleValueSource()...);
+       output.add(new QJoinMetaData()...);
+       output.add(new QJoinMetaData()...);
+       output.add(new QWidgetMetaData()...);
+       output.add(new QTableMetaData()...);
+
+       return (output);
+    }
+}
+----
+
+
+== Aside: TableMetaData with RecordEntities
+At this point, let's take a brief aside to dig deeper into the creation of a `QTableMeta` object.
+Tables, being probably the most important meta-data type in QQQ, have a lot of information that can
+be specified in their meta-data object.
+
+At the same time, if you're writing any custom code in your QQQ application
+(e.g., any processes or table customizers), where you're working with records from tables, you may
+prefer being able to work with entity beans (e.g., java classes with typed getter & setter methods),
+rather than the default object type that QQQ's ORM actions return, the `QRecord`, which carriers all
+of its values in a `Map`.  QQQ has a mechanism for dealing with this - in the form of the `QRecordEntity`
+class.
+
+So - if you want to build your application using entity beans (which is recommended, for the compile-time
+safety that they provide in custom code), you will be writing a `QRecordEntity` class, which will look like:
+
+[source,java]
+.QRecordEntity example
+----
+public class MyTable extends QRecordEntity
+{
+   public static final String TABLE_NAME = "myTable";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField()
+   private String name;
+
+   // no-arg constructor and constructor that takes a QRecord
+   // getters & setters (and optional fluent setters)
+}
+----
+
+The point of introducing this topic here and now is, that a `QRecordEntity` can be used to shortcut to
+defining some of the attributes in a `QTableMetaData` object.  Specifically, in a `MetaDataProducer<QTableMetaData>`
+you may say:
+
+[source,java]
+.QTableMetaDataProducer using a QRecordEntity
+----
+public QTableMetaData produce(QInstance qInstance) throws QExcpetion
+{
+  return new QTableMetaData()
+     .withName(MyTable.TABLE_NAME)
+     .withFieldsFromEntity(MyTable.class)
+     .withBackendName("memoryBackend");
+}
+----
+
+That `withFieldsFromEntity` call is one of the biggest benefits of this technique.  It allows you to avoid defining
+all of the field in you table in two place (the entity and the table meta-data).
+
+== MetaData Producing Annotations for Entities
+
+If you are using `QRecordEntity` classes that correspond to your tables, then you can take advantage of some
+additional annotations on those classes, to produce more related meta-data objects associated with those tables.
+The point of this is to eliminate boilerplate, and simplify / speed up the process of getting a new table
+built and deployed in your application, with some bells and whistles added.
+
+=== @QMetaDataProducingEntity
+This is an annotation to go on a QRecordEntity class, which you would like to be
+processed by `MetaDataProducerHelper`, to automatically produce some meta-data
+objects.  Specifically supports:
+
+* Making a possible-value-source out of the table.
+* Processing child tables to create joins and childRecordList widgets
+
+=== @ChildTable
+This is an annotation used as a value that goes inside a `@QMetadataProducingEntity` annotation, to define
+child-tables, e.g., for producing joins and childRecordList widgets related to the table defined in the entity class.
+
+=== @ChildJoin
+This is an annotation used as a value inside a `@ChildTable` inside a `@QMetadataProducingEntity` annotation,
+to control the generation of a `QJoinMetaData`, as a `ONE_TO_MANY` type join from the table represented by
+the annotated entity, to the table referenced in the `@ChildTable` annotation.
+
+=== @ChildRecordListWidget
+This is an annotation used as a value that goes inside a `@QMetadataProducingEntity` annotation, to control
+the generation of a QWidgetMetaData - for a ChildRecordList widget.
+
+[source,java]
+.QRecordEntity with meta-data producing annotations
+----
+@QMetaDataProducingEntity(
+   producePossibleValueSource = true,
+   childTables = {
+      @ChildTable(
+         childTableEntityClass = MyChildTable.class,
+         childJoin = @ChildJoin(enabled = true),
+         childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Children"))
+   }
+)
+public class MyTable extends QRecordEntity
+{
+   public static final String TABLE_NAME = "myTable";
+   // class body left as exercise for reader
+}
+----
+
+The class given in the example above, if processed by the `MetaDataProducerHelper`, would add the following
+meta-data objects to your `QInstance`:
+
+* A `QPossibleValueSource` named `myTable`, of type `TABLE`, with `myTable` as its backing table.
+* A `QJoinMetaData` named `myTableJoinMyChildTable`, as a `ONE_TO_MANY` type, between those two tables.
+* A `QWidgetMetaData` named `myTableJoinMyChildTable`, as a `CHILD_RECORD_LIST` type, that will show a list of
+records from `myChildTable` as a widget, when viewing a record from `myTable`.
+
+== Other MetaData Producing Annotations
+
+Similar to these annotations for a `RecordEntity`, a similar one exists for a `PossibleValueEnum` class,
+to automatically write the meta-data to use that enum as a possible value source in your application:
+
+=== @QMetaDataProducingPossibleValueEnum
+This is an annotation to go on a `PossibleValueEnum` class, which you would like to be
+processed by MetaDataProducerHelper, to automatically produce a PossibleValueSource meta-data
+based on the enum.
+
+[source,java]
+.PossibleValueEnum with meta-data producing annotation
+----
+@QMetaDataProducingPossibleValueEnum(producePossibleValueSource = true)
+public enum MyOptionsEnum implements PossibleValueEnum<Integer>
+{
+   // values and methods left as exercise for reader
+}
+----
+The enum given in the example above, if processed by the `MetaDataProducerHelper`, would add the following
+meta-data object to your `QInstance`:
+
+* A `QPossibleValueSource` named `MyOptionsEnum`, of type `ENUM`, with `MyOptionsEnum` as its backing enum.

docs/metaData/QInstance.adoc

@@ -29,11 +29,21 @@ service.routes(qJavalinImplementation.getRoutes());
 service.start();
 ----
 
-*QBackendMetaData Setup Methods:*
+*QInstance Setup:*
 
 These are the methods that one is most likely to use when setting up (defining) a `QInstance` object:
 
-* asdf
+* `add(TopLevelMetaDataInterface metaData)` - Generic method that takes most of the meta-data subtypes that can be added
+to an instance, such as `QBackendMetaData`, `QTableMetaData`, `QProcessMetaData`, etc.
+There are also type-specific methods (e.g., `addTable`, `addProcess`, etc), which one can call instead - this would just
+be a matter of personal preference.
 
-*QBackendMetaData Usage Methods:*
+*QInstance Usage:*
 
+Generally you will set up a `QInstance` in your application's startup flow, and then place it in the server (e.g., javalin).
+But, if, during application-code runtime, you need access to any of the meta-data in the instance, you access it
+via the `QContext` object's static `getInstance()` method.  This can be useful, for example, to get a list of the defined
+tables in the application, or fields in a table, or details about a field, etc.
+
+It is generally considered risky and/or not a good idea at all to modify the `QInstance` after it has been validated and
+a server is running.  Future versions of QQQ may in fact restrict modifications to the instance after validation.

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/UniqueKeyHelper.java

@@ -50,6 +50,7 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
  *******************************************************************************/
 public class UniqueKeyHelper
 {
+   private static Integer pageSize = 1000;
 
    /*******************************************************************************
     **
@@ -60,62 +61,71 @@ public class UniqueKeyHelper
       Map<List<Serializable>, Serializable> existingRecords = new HashMap<>();
       if(ukFieldNames != null)
       {
-         QueryInput queryInput = new QueryInput();
+         for(List<QRecord> page : CollectionUtils.getPages(recordList, pageSize))
-         queryInput.setTableName(table.getName());
+         {
-         queryInput.setTransaction(transaction);
+            QueryInput queryInput = new QueryInput();
+            queryInput.setTableName(table.getName());
+            queryInput.setTransaction(transaction);
 
-         QQueryFilter filter = new QQueryFilter();
+            QQueryFilter filter = new QQueryFilter();
-         if(ukFieldNames.size() == 1)
+            if(ukFieldNames.size() == 1)
-         {
-            List<Serializable> values = recordList.stream()
-               .filter(r -> CollectionUtils.nullSafeIsEmpty(r.getErrors()))
-               .map(r -> r.getValue(ukFieldNames.get(0)))
-               .collect(Collectors.toList());
-            filter.addCriteria(new QFilterCriteria(ukFieldNames.get(0), QCriteriaOperator.IN, values));
-         }
-         else
-         {
-            filter.setBooleanOperator(QQueryFilter.BooleanOperator.OR);
-            for(QRecord record : recordList)
             {
-               if(CollectionUtils.nullSafeHasContents(record.getErrors()))
+               List<Serializable> values = page.stream()
+                  .filter(r -> CollectionUtils.nullSafeIsEmpty(r.getErrors()))
+                  .map(r -> r.getValue(ukFieldNames.get(0)))
+                  .collect(Collectors.toList());
+
+               if(values.isEmpty())
                {
                   continue;
                }
 
-               QQueryFilter subFilter = new QQueryFilter();
+               filter.addCriteria(new QFilterCriteria(ukFieldNames.get(0), QCriteriaOperator.IN, values));
-               filter.addSubFilter(subFilter);
+            }
-               for(String fieldName : ukFieldNames)
+            else
+            {
+               filter.setBooleanOperator(QQueryFilter.BooleanOperator.OR);
+               for(QRecord record : page)
                {
-                  Serializable value = record.getValue(fieldName);
+                  if(CollectionUtils.nullSafeHasContents(record.getErrors()))
-                  if(value == null)
                   {
-                     subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.IS_BLANK));
+                     continue;
                   }
-                  else
+
+                  QQueryFilter subFilter = new QQueryFilter();
+                  filter.addSubFilter(subFilter);
+                  for(String fieldName : ukFieldNames)
                   {
-                     subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.EQUALS, value));
+                     Serializable value = record.getValue(fieldName);
+                     if(value == null)
+                     {
+                        subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.IS_BLANK));
+                     }
+                     else
+                     {
+                        subFilter.addCriteria(new QFilterCriteria(fieldName, QCriteriaOperator.EQUALS, value));
+                     }
                   }
                }
+
+               if(CollectionUtils.nullSafeIsEmpty(filter.getSubFilters()))
+               {
+                  ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  // if we didn't build any sub-filters (because all records have errors in them), don't run a query w/ no clauses - continue to next page //
+                  ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  continue;
+               }
             }
 
-            if(CollectionUtils.nullSafeIsEmpty(filter.getSubFilters()))
+            queryInput.setFilter(filter);
+            QueryOutput queryOutput = new QueryAction().execute(queryInput);
+            for(QRecord record : queryOutput.getRecords())
             {
-               ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+               Optional<List<Serializable>> keyValues = getKeyValues(table, uniqueKey, record, allowNullKeyValuesToEqual);
-               // if we didn't build any sub-filters (because all records have errors in them), don't run a query w/ no clauses - rather - return early. //
+               if(keyValues.isPresent())
-               ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+               {
-               return (existingRecords);
+                  existingRecords.put(keyValues.get(), record.getValue(table.getPrimaryKeyField()));
-            }
+               }
-         }
-
-         queryInput.setFilter(filter);
-         QueryOutput queryOutput = new QueryAction().execute(queryInput);
-         for(QRecord record : queryOutput.getRecords())
-         {
-            Optional<List<Serializable>> keyValues = getKeyValues(table, uniqueKey, record, allowNullKeyValuesToEqual);
-            if(keyValues.isPresent())
-            {
-               existingRecords.put(keyValues.get(), record.getValue(table.getPrimaryKeyField()));
             }
          }
       }
@@ -200,4 +210,26 @@ public class UniqueKeyHelper
       }
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for pageSize
+    **
+    *******************************************************************************/
+   public static Integer getPageSize()
+   {
+      return pageSize;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for pageSize
+    **
+    *******************************************************************************/
+   public static void setPageSize(Integer pageSize)
+   {
+      UniqueKeyHelper.pageSize = pageSize;
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/QValueFormatter.java

@@ -529,10 +529,16 @@ public class QValueFormatter
                   }
                }
 
-               /////////////////////////////////////////////
+               ////////////////////////////////////////////////////////////////////////////////////////////////
-               // if field type is blob, update its value //
+               // if field type is blob OR if there's a supplemental process or code-ref that needs to run - //
-               /////////////////////////////////////////////
+               // then update its value to be a callback-url that'll give access to the bytes to download    //
-               if(QFieldType.BLOB.equals(field.getType()))
+               // implied here is that a String value (w/o supplemental code/proc) has its value stay as a   //
+               // URL, which is where the file is directly downloaded from.  And in the case of a String     //
+               // with code-to-run, then the code should run, followed by a redirect to the value URL.       //
+               ////////////////////////////////////////////////////////////////////////////////////////////////
+               if(QFieldType.BLOB.equals(field.getType())
+                  || adornmentValues.containsKey(AdornmentType.FileDownloadValues.SUPPLEMENTAL_CODE_REFERENCE)
+                  || adornmentValues.containsKey(AdornmentType.FileDownloadValues.SUPPLEMENTAL_PROCESS_NAME))
                {
                   record.setValue(field.getName(), "/data/" + table.getName() + "/" + primaryKey + "/" + field.getName() + "/" + fileName);
                }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidator.java

@@ -70,6 +70,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldBehavior;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppChildMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
@@ -780,14 +781,37 @@ public class QInstanceValidator
          if(assertCondition(StringUtils.hasContent(association.getName()), "missing a name for an Association on table " + table.getName()))
          {
             String messageSuffix = " for Association " + association.getName() + " on table " + table.getName();
+            boolean recognizedTable = false;
             if(assertCondition(StringUtils.hasContent(association.getAssociatedTableName()), "missing associatedTableName" + messageSuffix))
             {
-               assertCondition(qInstance.getTable(association.getAssociatedTableName()) != null, "unrecognized associatedTableName " + association.getAssociatedTableName() + messageSuffix);
+               if(assertCondition(qInstance.getTable(association.getAssociatedTableName()) != null, "unrecognized associatedTableName " + association.getAssociatedTableName() + messageSuffix))
+               {
+                  recognizedTable = true;
+               }
             }
 
             if(assertCondition(StringUtils.hasContent(association.getJoinName()), "missing joinName" + messageSuffix))
             {
-               assertCondition(qInstance.getJoin(association.getJoinName()) != null, "unrecognized joinName " + association.getJoinName() + messageSuffix);
+               QJoinMetaData join = qInstance.getJoin(association.getJoinName());
+               if(assertCondition(join != null, "unrecognized joinName " + association.getJoinName() + messageSuffix))
+               {
+                  assert join != null; // covered by the assertCondition
+
+                  if(recognizedTable)
+                  {
+                     boolean isLeftToRight = join.getLeftTable().equals(table.getName()) && join.getRightTable().equals(association.getAssociatedTableName());
+                     boolean isRightToLeft = join.getRightTable().equals(table.getName()) && join.getLeftTable().equals(association.getAssociatedTableName());
+                     assertCondition(isLeftToRight || isRightToLeft, "join [" + association.getJoinName() + "] does not connect tables [" + table.getName() + "] and [" + association.getAssociatedTableName() + "]" + messageSuffix);
+                     if(isLeftToRight)
+                     {
+                        assertCondition(join.getType().equals(JoinType.ONE_TO_MANY) || join.getType().equals(JoinType.ONE_TO_ONE), "Join type does not have 'one' on this table's side side (left)" + messageSuffix);
+                     }
+                     else if(isRightToLeft)
+                     {
+                        assertCondition(join.getType().equals(JoinType.MANY_TO_ONE) || join.getType().equals(JoinType.ONE_TO_ONE), "Join type does not have 'one' on this table's side (right)" + messageSuffix);
+                     }
+                  }
+               }
             }
          }
       }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QQueryFilter.java

@@ -55,6 +55,16 @@ public class QQueryFilter implements Serializable, Cloneable
    private BooleanOperator    booleanOperator = BooleanOperator.AND;
    private List<QQueryFilter> subFilters      = new ArrayList<>();
 
+   //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   // initial intent here was - put, e.g., UNION between multiple SELECT (with the individual selects being defined in subFilters) //
+   // but, actually SQL would let us do, e.g., SELECT UNION SELECT INTERSECT SELECT                                                //
+   // so - we could see a future implementation where we:                                                                          //
+   // - used the top-level subFilterSetOperator to indicate hat we are doing a multi-query set-operation query.                    //
+   // - looked within the subFilter, to see if it specified a subFilterSetOperator - and use that operator before that query       //
+   // but - in v0, just using the one at the top-level works                                                                       //
+   //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+   private SubFilterSetOperator subFilterSetOperator = null;
+
    ////////////////////////////////////////////////////////////////////////////////////////////////////////////
    // skip & limit are meant to only apply to QueryAction (at least at the initial time they are added here) //
    // e.g., they are ignored in CountAction, AggregateAction, etc, where their meanings may be less obvious  //
@@ -75,6 +85,19 @@ public class QQueryFilter implements Serializable, Cloneable
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public enum SubFilterSetOperator
+   {
+      UNION,
+      UNION_ALL,
+      INTERSECT,
+      EXCEPT
+   }
+
+
+
    /*******************************************************************************
     ** Constructor
     **
@@ -799,4 +822,35 @@ public class QQueryFilter implements Serializable, Cloneable
       }
    }
 
+
+   /*******************************************************************************
+    ** Getter for subFilterSetOperator
+    *******************************************************************************/
+   public SubFilterSetOperator getSubFilterSetOperator()
+   {
+      return (this.subFilterSetOperator);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for subFilterSetOperator
+    *******************************************************************************/
+   public void setSubFilterSetOperator(SubFilterSetOperator subFilterSetOperator)
+   {
+      this.subFilterSetOperator = subFilterSetOperator;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for subFilterSetOperator
+    *******************************************************************************/
+   public QQueryFilter withSubFilterSetOperator(SubFilterSetOperator subFilterSetOperator)
+   {
+      this.subFilterSetOperator = subFilterSetOperator;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducer.java

@@ -24,7 +24,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
 
 /*******************************************************************************
  ** Abstract class that knows how to produce meta data objects.  Useful with
- ** MetaDataProducerHelper, to put point at a package full of these, and populate
+ ** MetaDataProducerHelper, to point at a package full of these, and populate
  ** your whole QInstance.
  *******************************************************************************/
 public abstract class MetaDataProducer<T extends MetaDataProducerOutput> implements MetaDataProducerInterface<T>

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerHelper.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.model.metadata;
 
 
 import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
 import java.lang.reflect.Modifier;
 import java.util.ArrayList;
 import java.util.Comparator;
@@ -31,10 +32,23 @@ import java.util.List;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.ChildJoinFromRecordEntityGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.PossibleValueSourceOfEnumGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.PossibleValueSourceOfTableGenericMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildRecordListWidget;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildTable;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingPossibleValueEnum;
 import com.kingsrook.qqq.backend.core.utils.ClassPathUtils;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -90,6 +104,9 @@ public class MetaDataProducerHelper
       }
       List<MetaDataProducerInterface<?>> producers = new ArrayList<>();
 
+      ////////////////////////////////////////////////////////////////////////////////////////
+      // loop over classes, processing them based on either their type or their annotations //
+      ////////////////////////////////////////////////////////////////////////////////////////
       for(Class<?> aClass : classesInPackage)
       {
          try
@@ -101,23 +118,27 @@ public class MetaDataProducerHelper
 
             if(MetaDataProducerInterface.class.isAssignableFrom(aClass))
             {
-               boolean foundValidConstructor = false;
+               CollectionUtils.addIfNotNull(producers, processMetaDataProducer(aClass));
-               for(Constructor<?> constructor : aClass.getConstructors())
+            }
-               {
-                  if(constructor.getParameterCount() == 0)
-                  {
-                     Object o = constructor.newInstance();
-                     producers.add((MetaDataProducerInterface<?>) o);
-                     foundValidConstructor = true;
-                     break;
-                  }
-               }
 
-               if(!foundValidConstructor)
+            if(aClass.isAnnotationPresent(QMetaDataProducingEntity.class))
+            {
+               QMetaDataProducingEntity qMetaDataProducingEntity = aClass.getAnnotation(QMetaDataProducingEntity.class);
+               if(qMetaDataProducingEntity.producePossibleValueSource())
                {
-                  LOG.warn("Found a class which implements MetaDataProducerInterface, but it does not have a no-arg constructor, so it cannot be used.", logPair("class", aClass.getSimpleName()));
+                  producers.addAll(processMetaDataProducingEntity(aClass));
                }
             }
+
+            if(aClass.isAnnotationPresent(QMetaDataProducingPossibleValueEnum.class))
+            {
+               QMetaDataProducingPossibleValueEnum qMetaDataProducingPossibleValueEnum = aClass.getAnnotation(QMetaDataProducingPossibleValueEnum.class);
+               if(qMetaDataProducingPossibleValueEnum.producePossibleValueSource())
+               {
+                  CollectionUtils.addIfNotNull(producers, processMetaDataProducingPossibleValueEnum(aClass));
+               }
+            }
+
          }
          catch(Exception e)
          {
@@ -168,7 +189,176 @@ public class MetaDataProducerHelper
             LOG.debug("Not using producer which is not enabled", logPair("producer", producer.getClass().getSimpleName()));
          }
       }
-
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static <T extends PossibleValueEnum<T>> MetaDataProducerInterface<?> processMetaDataProducingPossibleValueEnum(Class<?> aClass)
+   {
+      String warningPrefix = "Found a class annotated as @" + QMetaDataProducingPossibleValueEnum.class.getSimpleName();
+      if(!PossibleValueEnum.class.isAssignableFrom(aClass))
+      {
+         LOG.warn(warningPrefix + ", but which is not a " + PossibleValueEnum.class.getSimpleName() + ", so it will not be used.", logPair("class", aClass.getSimpleName()));
+         return null;
+      }
+
+      PossibleValueEnum<?>[] values = (PossibleValueEnum<?>[]) aClass.getEnumConstants();
+      return (new PossibleValueSourceOfEnumGenericMetaDataProducer<T>(aClass.getSimpleName(), (PossibleValueEnum<T>[]) values));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static List<MetaDataProducerInterface<?>> processMetaDataProducingEntity(Class<?> aClass) throws Exception
+   {
+      List<MetaDataProducerInterface<?>> rs = new ArrayList<>();
+
+      String warningPrefix = "Found a class annotated as @" + QMetaDataProducingEntity.class.getSimpleName();
+      if(!QRecordEntity.class.isAssignableFrom(aClass))
+      {
+         LOG.warn(warningPrefix + ", but which is not a " + QRecordEntity.class.getSimpleName() + ", so it will not be used.", logPair("class", aClass.getSimpleName()));
+         return (rs);
+      }
+
+      Field tableNameField = aClass.getDeclaredField("TABLE_NAME");
+      if(!tableNameField.getType().equals(String.class))
+      {
+         LOG.warn(warningPrefix + ", but whose TABLE_NAME field is not a String, so it will not be used.", logPair("class", aClass.getSimpleName()));
+         return (rs);
+      }
+
+      String tableNameValue = (String) tableNameField.get(null);
+      rs.add(new PossibleValueSourceOfTableGenericMetaDataProducer(tableNameValue));
+
+      //////////////////////////
+      // process child tables //
+      //////////////////////////
+      QMetaDataProducingEntity qMetaDataProducingEntity = aClass.getAnnotation(QMetaDataProducingEntity.class);
+      for(ChildTable childTable : qMetaDataProducingEntity.childTables())
+      {
+         Class<? extends QRecordEntity> childEntityClass = childTable.childTableEntityClass();
+         if(childTable.childJoin().enabled())
+         {
+            CollectionUtils.addIfNotNull(rs, processChildJoin(aClass, childTable));
+
+            if(childTable.childRecordListWidget().enabled())
+            {
+               CollectionUtils.addIfNotNull(rs, processChildRecordListWidget(aClass, childTable));
+            }
+         }
+         else
+         {
+            if(childTable.childRecordListWidget().enabled())
+            {
+               //////////////////////////////////////////////////////////////////////////
+               // if not doing the join, can't do the child-widget, so warn about that //
+               //////////////////////////////////////////////////////////////////////////
+               LOG.warn(warningPrefix + " requested to produce a ChildRecordListWidget, but not produce a Join - which is not allowed (must do join to do widget). ", logPair("class", aClass.getSimpleName()), logPair("childEntityClass", childEntityClass.getSimpleName()));
+            }
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static MetaDataProducerInterface<?> processChildRecordListWidget(Class<?> aClass, ChildTable childTable) throws Exception
+   {
+      Class<? extends QRecordEntity> childEntityClass = childTable.childTableEntityClass();
+      String                         parentTableName  = getTableNameStaticFieldValue(aClass);
+      String                         childTableName   = getTableNameStaticFieldValue(childEntityClass);
+
+      ChildRecordListWidget childRecordListWidget = childTable.childRecordListWidget();
+      return (new ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer(childTableName, parentTableName, childRecordListWidget));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static String findPossibleValueField(Class<? extends QRecordEntity> entityClass, String possibleValueSourceName)
+   {
+      for(Field field : entityClass.getDeclaredFields())
+      {
+         if(field.isAnnotationPresent(QField.class))
+         {
+            QField qField = field.getAnnotation(QField.class);
+            if(qField.possibleValueSourceName().equals(possibleValueSourceName))
+            {
+               return field.getName();
+            }
+         }
+      }
+
+      return (null);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static MetaDataProducerInterface<?> processChildJoin(Class<?> aClass, ChildTable childTable) throws Exception
+   {
+      Class<? extends QRecordEntity> childEntityClass = childTable.childTableEntityClass();
+
+      String parentTableName        = getTableNameStaticFieldValue(aClass);
+      String childTableName         = getTableNameStaticFieldValue(childEntityClass);
+      String possibleValueFieldName = findPossibleValueField(childEntityClass, parentTableName);
+      if(!StringUtils.hasContent(possibleValueFieldName))
+      {
+         LOG.warn("Could not find field in [" + childEntityClass.getSimpleName() + "] with possibleValueSource referencing table [" + aClass.getSimpleName() + "]");
+         return (null);
+      }
+
+      return (new ChildJoinFromRecordEntityGenericMetaDataProducer(childTableName, parentTableName, possibleValueFieldName));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static MetaDataProducerInterface<?> processMetaDataProducer(Class<?> aClass) throws Exception
+   {
+      for(Constructor<?> constructor : aClass.getConstructors())
+      {
+         if(constructor.getParameterCount() == 0)
+         {
+            Object o = constructor.newInstance();
+            return (MetaDataProducerInterface<?>) o;
+         }
+      }
+
+      LOG.warn("Found a class which implements MetaDataProducerInterface, but it does not have a no-arg constructor, so it cannot be used.", logPair("class", aClass.getSimpleName()));
+      return null;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static String getTableNameStaticFieldValue(Class<?> aClass) throws NoSuchFieldException, IllegalAccessException
+   {
+      Field tableNameField = aClass.getDeclaredField("TABLE_NAME");
+      if(!tableNameField.getType().equals(String.class))
+      {
+         return (null);
+      }
+
+      String tableNameValue = (String) tableNameField.get(null);
+      return (tableNameValue);
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerInterface.java

@@ -27,12 +27,12 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
 
 /*******************************************************************************
  ** Interface for classes that know how to produce meta data objects.  Useful with
- ** MetaDataProducerHelper, to put point at a package full of these, and populate
+ ** MetaDataProducerHelper, to point at a package full of these, and populate
  ** your whole QInstance.
  **
  ** See also MetaDataProducer - an implementer of this interface, which actually
  ** came first, and is fine to extend if producing a meta-data class is all your
- ** clas means to do (nice and "Single-responsibility principle").
+ ** class means to do (nice and "Single-responsibility principle").
  **
  ** But, in some applications you may want to, for example, have one class that
  ** defines a process step, and also produces the meta-data for that process, so

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/AdornmentType.java

@@ -68,6 +68,9 @@ public enum AdornmentType
       String DEFAULT_EXTENSION = "defaultExtension";
       String DEFAULT_MIME_TYPE = "defaultMimeType";
 
+      String SUPPLEMENTAL_PROCESS_NAME   = "supplementalProcessName";
+      String SUPPLEMENTAL_CODE_REFERENCE = "supplementalCodeReference";
+
       ////////////////////////////////////////////////////
       // use these two together, as in:                 //
       // FILE_NAME_FORMAT = "Order %s Packing Slip.pdf" //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/possiblevalues/QPossibleValue.java

@@ -22,6 +22,9 @@
 package com.kingsrook.qqq.backend.core.model.metadata.possiblevalues;
 
 
+import java.util.Objects;
+
+
 /*******************************************************************************
  ** An actual possible value - an id and label.
  **
@@ -76,4 +79,37 @@ public class QPossibleValue<T>
    {
       return label;
    }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public boolean equals(Object o)
+   {
+      if(this == o)
+      {
+         return true;
+      }
+
+      if(o == null || getClass() != o.getClass())
+      {
+         return false;
+      }
+
+      QPossibleValue<?> that = (QPossibleValue<?>) o;
+      return Objects.equals(id, that.id) && Objects.equals(label, that.label);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public int hashCode()
+   {
+      return Objects.hash(id, label);
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/possiblevalues/QPossibleValueSource.java

@@ -23,7 +23,10 @@ package com.kingsrook.qqq.backend.core.model.metadata.possiblevalues;
 
 
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.TopLevelMetaDataInterface;
@@ -97,7 +100,7 @@ public class QPossibleValueSource implements TopLevelMetaDataInterface
     ** Create a new possible value source, for an enum, with default settings.
     ** e.g., type=ENUM; name from param values from the param; LABEL_ONLY format
     *******************************************************************************/
-   public static <T extends PossibleValueEnum<?>> QPossibleValueSource newForEnum(String name, T[] values)
+   public static <I, T extends PossibleValueEnum<I>> QPossibleValueSource newForEnum(String name, T[] values)
    {
       return new QPossibleValueSource()
          .withName(name)
@@ -553,11 +556,25 @@ public class QPossibleValueSource implements TopLevelMetaDataInterface
     **   myPossibleValueSource.withValuesFromEnum(MyEnum.values()));
     **
     *******************************************************************************/
-   public <T extends PossibleValueEnum<?>> QPossibleValueSource withValuesFromEnum(T[] values)
+   public <I, T extends PossibleValueEnum<I>> QPossibleValueSource withValuesFromEnum(T[] values)
    {
+      Set<I> usedIds = new HashSet<>();
+      List<I> duplicatedIds = new ArrayList<>();
+
       for(T t : values)
       {
+         if(usedIds.contains(t.getPossibleValueId()))
+         {
+            duplicatedIds.add(t.getPossibleValueId());
+         }
+
          addEnumValue(new QPossibleValue<>(t.getPossibleValueId(), t.getPossibleValueLabel()));
+         usedIds.add(t.getPossibleValueId());
+      }
+
+      if(!duplicatedIds.isEmpty())
+      {
+         throw (new QRuntimeException("Error:  Duplicated id(s) found in enum values: " + duplicatedIds));
       }
 
       return (this);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/ChildJoinFromRecordEntityGenericMetaDataProducer.java

@@ -0,0 +1,105 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a QJoinMetaData, based on a
+ ** QRecordEntity and a ChildTable sub-annotation.
+ **
+ ** e.g., Orders & LineItems - on the Order entity
+ ** <code>
+ @QMetaDataProducingEntity(
+    childTables = { @ChildTable(
+       childTableEntityClass = LineItem.class,
+       childJoin = @ChildJoin(enabled = true),
+       childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Order Lines"))
+    }
+ )
+ public class Order extends QRecordEntity
+ ** </code>
+ **
+ ** A Join will be made:
+ ** - left: Order
+ ** - right: LineItem
+ ** - type:  ONE_TO_MANY (one order (parent table) has mny lines (child table))
+ ** - joinOn:  order's primary key, lineItem's orderId field
+ ** - name:  inferred, based on the table names orderJoinLineItem)
+ *******************************************************************************/
+public class ChildJoinFromRecordEntityGenericMetaDataProducer implements MetaDataProducerInterface<QJoinMetaData>
+{
+   private String childTableName; // e.g., lineItem
+   private String parentTableName; // e.g., order
+   private String foreignKeyFieldName; // e.g., orderId
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public ChildJoinFromRecordEntityGenericMetaDataProducer(String childTableName, String parentTableName, String foreignKeyFieldName)
+   {
+      Objects.requireNonNull(childTableName, "childTableName cannot be null");
+      Objects.requireNonNull(parentTableName, "parentTableName cannot be null");
+      Objects.requireNonNull(foreignKeyFieldName, "foreignKeyFieldName cannot be null");
+
+      this.childTableName = childTableName;
+      this.parentTableName = parentTableName;
+      this.foreignKeyFieldName = foreignKeyFieldName;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QJoinMetaData produce(QInstance qInstance) throws QException
+   {
+      QTableMetaData possibleValueTable = qInstance.getTable(parentTableName);
+      if(possibleValueTable == null)
+      {
+         throw (new QException("Could not find tableMetaData " + parentTableName));
+      }
+
+      QJoinMetaData join = new QJoinMetaData()
+         .withLeftTable(parentTableName)
+         .withRightTable(childTableName)
+         .withInferredName()
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn(possibleValueTable.getPrimaryKeyField(), foreignKeyFieldName));
+
+      return (join);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer.java

@@ -0,0 +1,100 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildRecordListWidget;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a ChildRecordList QWidgetMetaData, to
+ ** produce a QJoinMetaData, based on a QRecordEntity and a ChildTable sub-annotation.
+ **
+ ** e.g., Orders & LineItems - on the Order entity
+ ** <code>
+ @QMetaDataProducingEntity( childTables = { @ChildTable(
+    childTableEntityClass = LineItem.class,
+    childJoin = @ChildJoin(enabled = true),
+    childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Order Lines"))
+ })
+ public class Order extends QRecordEntity
+ ** </code>
+ **
+ *******************************************************************************/
+public class ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer implements MetaDataProducerInterface<QWidgetMetaData>
+{
+   private String childTableName; // e.g., lineItem
+   private String parentTableName; // e.g., order
+
+   private ChildRecordListWidget childRecordListWidget;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public ChildRecordListWidgetFromRecordEntityGenericMetaDataProducer(String childTableName, String parentTableName, ChildRecordListWidget childRecordListWidget)
+   {
+      this.childTableName = childTableName;
+      this.parentTableName = parentTableName;
+      this.childRecordListWidget = childRecordListWidget;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QWidgetMetaData produce(QInstance qInstance) throws QException
+   {
+      String        name = QJoinMetaData.makeInferredJoinName(parentTableName, childTableName);
+      QJoinMetaData join = qInstance.getJoin(name);
+
+      QWidgetMetaData widget = ChildRecordListRenderer.widgetMetaDataBuilder(join)
+         .withName(name)
+         .withLabel(childRecordListWidget.label())
+         .withCanAddChildRecord(childRecordListWidget.canAddChildRecords())
+         .getWidgetMetaData();
+
+      if(StringUtils.hasContent(childRecordListWidget.manageAssociationName()))
+      {
+         widget.withDefaultValue("manageAssociationName", childRecordListWidget.manageAssociationName());
+      }
+
+      if(childRecordListWidget.maxRows() > 0)
+      {
+         widget.withDefaultValue("maxRows", childRecordListWidget.maxRows());
+      }
+
+      return (widget);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/PossibleValueSourceOfEnumGenericMetaDataProducer.java

@@ -0,0 +1,64 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+
+
+/***************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a QPossibleValueSource meta-data
+ ** based on a PossibleValueEnum
+ **
+ ***************************************************************************/
+public class PossibleValueSourceOfEnumGenericMetaDataProducer<T extends PossibleValueEnum<T>> implements MetaDataProducerInterface<QPossibleValueSource>
+{
+   private final String                 name;
+   private final PossibleValueEnum<T>[] values;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public PossibleValueSourceOfEnumGenericMetaDataProducer(String name, PossibleValueEnum<T>[] values)
+   {
+      this.name = name;
+      this.values = values;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QPossibleValueSource produce(QInstance qInstance)
+   {
+      return (QPossibleValueSource.newForEnum(name, values));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/PossibleValueSourceOfTableGenericMetaDataProducer.java

@@ -0,0 +1,61 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+
+
+/***************************************************************************
+ ** Generic meta-data-producer, which should be instantiated (e.g., by
+ ** MetaDataProducer Helper), to produce a QPossibleValueSource meta-data
+ ** based on a QRecordEntity class (which has corresponding QTableMetaData).
+ **
+ ***************************************************************************/
+public class PossibleValueSourceOfTableGenericMetaDataProducer implements MetaDataProducerInterface<QPossibleValueSource>
+{
+   private final String tableName;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public PossibleValueSourceOfTableGenericMetaDataProducer(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QPossibleValueSource produce(QInstance qInstance)
+   {
+      return (QPossibleValueSource.newForTable(tableName));
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/ChildJoin.java

@@ -0,0 +1,38 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+
+
+/***************************************************************************
+ ** value that goes inside a QMetadataProducingEntity annotation, to control
+ ** the generation of a QJoinMetaData
+ ***************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface ChildJoin
+{
+   boolean enabled();
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/ChildRecordListWidget.java

@@ -0,0 +1,46 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+
+
+/***************************************************************************
+ ** value that goes inside a QMetadataProducingEntity annotation, to control
+ ** the generation of a QWidgetMetaData - for a ChildRecordList widget.
+ ***************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface ChildRecordListWidget
+{
+   boolean enabled();
+
+   String label() default "";
+
+   int maxRows() default 20;
+
+   boolean canAddChildRecords() default false;
+
+   String manageAssociationName() default "";
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/ChildTable.java

@@ -0,0 +1,45 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/***************************************************************************
+ ** value that goes inside a QMetadataProducingEntity annotation, to define
+ ** child-tables, e.g., for producing joins and childRecordList widgets
+ ***************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface ChildTable
+{
+   Class<? extends QRecordEntity> childTableEntityClass();
+
+   String joinFieldName() default "";
+
+   ChildJoin childJoin() default @ChildJoin(enabled = false);
+
+   ChildRecordListWidget childRecordListWidget() default @ChildRecordListWidget(enabled = false);
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/QMetaDataProducingEntity.java

@@ -0,0 +1,48 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+
+/*******************************************************************************
+ ** annotation to go on a QRecordEntity class, which you would like to be
+ ** processed by MetaDataProducerHelper, to automatically produce some meta-data
+ ** objects.  Specifically supports:
+ **
+ ** - Making a possible-value-source out of the table.
+ ** - Processing child tables to create joins and childRecordList widgets
+ *******************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.TYPE)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface QMetaDataProducingEntity
+{
+   boolean producePossibleValueSource() default true;
+
+   ChildTable[] childTables() default { };
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/producers/annotations/QMetaDataProducingPossibleValueEnum.java

@@ -0,0 +1,42 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers.annotations;
+
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+
+/*******************************************************************************
+ ** annotation to go on a PossibleValueEnum class, which you would like to be
+ ** processed by MetaDataProducerHelper, to automatically produce possible-value-
+ ** source meta-data based on the enum.
+ *******************************************************************************/
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.TYPE)
+@SuppressWarnings("checkstyle:MissingJavadocMethod")
+public @interface QMetaDataProducingPossibleValueEnum
+{
+   boolean producePossibleValueSource() default true;
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/QTableMetaData.java

@@ -1329,6 +1329,21 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
 
 
 
+   /*******************************************************************************
+    ** Getter for an association by name
+    *******************************************************************************/
+   public Optional<Association> getAssociationByName(String name)
+   {
+      if(associations == null)
+      {
+         return (Optional.empty());
+      }
+
+      return (getAssociations().stream().filter(a -> a.getName().equals(name)).findFirst());
+   }
+
+
+
    /*******************************************************************************
     ** Setter for associations
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/CollectionUtils.java

@@ -700,4 +700,16 @@ public class CollectionUtils
       return (map.containsKey(key) && map.get(key) != null);
    }
 
+
+
+   /***************************************************************************
+    ** add an element to a collection, but, only if the element isn't null
+    ***************************************************************************/
+   public static <T, E extends T> void addIfNotNull(Collection<T> c, E element)
+   {
+      if(element != null)
+      {
+         c.add(element);
+      }
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/StringUtils.java

@@ -460,4 +460,19 @@ public class StringUtils
       return (Pattern.matches("[a-f0-9]{8}(?:-[a-f0-9]{4}){4}[a-f0-9]{8}", s));
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String emptyToNull(String s)
+   {
+      if(!hasContent(s))
+      {
+         return (null);
+      }
+
+      return (s);
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/UniqueKeyHelperTest.java

@@ -0,0 +1,123 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.tables.helpers;
+
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for UniqueKeyHelper 
+ *******************************************************************************/
+class UniqueKeyHelperTest extends BaseTest
+{
+   private static Integer originalPageSize;
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeAll
+   static void beforeAll()
+   {
+      originalPageSize = UniqueKeyHelper.getPageSize();
+      UniqueKeyHelper.setPageSize(5);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterAll
+   static void afterAll()
+   {
+      UniqueKeyHelper.setPageSize(originalPageSize);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeEach
+   @AfterEach
+   void beforeAndAfterEach()
+   {
+      MemoryRecordStore.fullReset();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testUniqueKey() throws QException
+   {
+      List<QRecord> recordsWithKey1Equals1AndKey2In1Through10 = List.of(
+         new QRecord().withValue("key1", 1).withValue("key2", 1),
+         new QRecord().withValue("key1", 1).withValue("key2", 2),
+         new QRecord().withValue("key1", 1).withValue("key2", 3),
+         new QRecord().withValue("key1", 1).withValue("key2", 4),
+         new QRecord().withValue("key1", 1).withValue("key2", 5),
+         new QRecord().withValue("key1", 1).withValue("key2", 6),
+         new QRecord().withValue("key1", 1).withValue("key2", 7),
+         new QRecord().withValue("key1", 1).withValue("key2", 8),
+         new QRecord().withValue("key1", 1).withValue("key2", 9),
+         new QRecord().withValue("key1", 1).withValue("key2", 10)
+      );
+
+      InsertInput insertInput = new InsertInput();
+      insertInput.setTableName(TestUtils.TABLE_NAME_TWO_KEYS);
+      insertInput.setRecords(recordsWithKey1Equals1AndKey2In1Through10);
+      InsertOutput insertOutput = new InsertAction().execute(insertInput);
+
+      MemoryRecordStore.resetStatistics();
+      MemoryRecordStore.setCollectStatistics(true);
+
+      QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_TWO_KEYS);
+      Map<List<Serializable>, Serializable> existingKeys = UniqueKeyHelper.getExistingKeys(null, table, recordsWithKey1Equals1AndKey2In1Through10, table.getUniqueKeys().get(0), false);
+      assertEquals(recordsWithKey1Equals1AndKey2In1Through10.size(), existingKeys.size());
+
+      assertEquals(2, MemoryRecordStore.getStatistics().get(MemoryRecordStore.STAT_QUERIES_RAN));
+   }
+
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidatorTest.java

@@ -2054,16 +2054,41 @@ public class QInstanceValidatorTest extends BaseTest
 
       assertValidationFailureReasons((qInstance -> qInstance.getTable(TestUtils.TABLE_NAME_ORDER).withAssociation(new Association().withName("myAssociation"))),
          "missing joinName for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER,
-         "missing associatedTableName for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER
+         "missing associatedTableName for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER);
-      );
 
       assertValidationFailureReasons((qInstance -> qInstance.getTable(TestUtils.TABLE_NAME_ORDER).withAssociation(new Association().withName("myAssociation").withJoinName("notAJoin").withAssociatedTableName(TestUtils.TABLE_NAME_LINE_ITEM))),
-         "unrecognized joinName notAJoin for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER
+         "unrecognized joinName notAJoin for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER);
-      );
 
       assertValidationFailureReasons((qInstance -> qInstance.getTable(TestUtils.TABLE_NAME_ORDER).withAssociation(new Association().withName("myAssociation").withJoinName("orderLineItem").withAssociatedTableName("notATable"))),
-         "unrecognized associatedTableName notATable for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER
+         "unrecognized associatedTableName notATable for Association myAssociation on table " + TestUtils.TABLE_NAME_ORDER);
-      );
+
+      //////////////////////////////////
+      // wrong join on an association //
+      //////////////////////////////////
+      assertValidationFailureReasons((qInstance ->
+      {
+         Association association = qInstance.getTable(TestUtils.TABLE_NAME_ORDER).getAssociationByName("orderLine").orElseThrow();
+         association.setJoinName("orderOrderExtrinsic");
+      }),
+         "join [orderOrderExtrinsic] does not connect tables [order] and [orderLine]");
+
+      //////////////////////////////////////////
+      // wrong table (doesn't match the join) //
+      //////////////////////////////////////////
+      assertValidationFailureReasons((qInstance ->
+      {
+         Association association = qInstance.getTable(TestUtils.TABLE_NAME_ORDER).getAssociationByName("orderLine").orElseThrow();
+         association.setAssociatedTableName(TestUtils.TABLE_NAME_ORDER_EXTRINSIC);
+      }),
+         "join [orderLineItem] does not connect tables [order] and [orderExtrinsic]");
+
+      //////////////////////////////
+      // invalid type on the join //
+      //////////////////////////////
+      assertValidationFailureReasons((qInstance -> qInstance.getJoin("orderLineItem").setType(JoinType.MANY_TO_MANY)),
+         "Join type does not have 'one' on this table's side side (left)");
+      assertValidationFailureReasons((qInstance -> qInstance.getJoin("orderLineItem").setType(JoinType.MANY_TO_ONE)),
+         "Join type does not have 'one' on this table's side side (left)");
    }
 
 
@@ -2323,7 +2348,7 @@ public class QInstanceValidatorTest extends BaseTest
       {
          int noOfReasons = actualReasons == null ? 0 : actualReasons.size();
          assertEquals(expectedReasons.length, noOfReasons, "Expected number of validation failure reasons.\nExpected reasons: " + String.join(",", expectedReasons)
-            + "\nActual reasons: " + (noOfReasons > 0 ? String.join("\n", actualReasons) : "--"));
+                                                           + "\nActual reasons: " + (noOfReasons > 0 ? String.join("\n", actualReasons) : "--"));
       }
 
       for(String reason : expectedReasons)
@@ -2451,6 +2476,7 @@ public class QInstanceValidatorTest extends BaseTest
    public static class ValidAuthCustomizer implements QAuthenticationModuleCustomizerInterface {}
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -2468,6 +2494,7 @@ public class QInstanceValidatorTest extends BaseTest
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerHelperTest.java

@@ -23,14 +23,26 @@ package com.kingsrook.qqq.backend.core.model.metadata;
 
 
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestAbstractMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestDisabledMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestImplementsMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducingChildEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducingEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.TestMetaDataProducingPossibleValueEnum;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestNoInterfacesExtendsObject;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.TestNoValidConstructorMetaDataProducer;
 import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
 
@@ -54,6 +66,48 @@ class MetaDataProducerHelperTest
       assertFalse(qInstance.getTables().containsKey(TestNoInterfacesExtendsObject.NAME));
       assertFalse(qInstance.getTables().containsKey(TestAbstractMetaDataProducer.NAME));
       assertFalse(qInstance.getTables().containsKey(TestDisabledMetaDataProducer.NAME));
+
+      /////////////////////////////////////////////
+      // annotation on PVS enum -> PVS meta data //
+      /////////////////////////////////////////////
+      assertTrue(qInstance.getPossibleValueSources().containsKey(TestMetaDataProducingPossibleValueEnum.class.getSimpleName()));
+      QPossibleValueSource enumPVS = qInstance.getPossibleValueSource(TestMetaDataProducingPossibleValueEnum.class.getSimpleName());
+      assertEquals(QPossibleValueSourceType.ENUM, enumPVS.getType());
+      assertEquals(2, enumPVS.getEnumValues().size());
+      assertEquals(new QPossibleValue<>(1, "One"), enumPVS.getEnumValues().get(0));
+
+      //////////////////////////////////////////////
+      // annotation on PVS table -> PVS meta data //
+      //////////////////////////////////////////////
+      assertTrue(qInstance.getPossibleValueSources().containsKey(TestMetaDataProducingEntity.TABLE_NAME));
+      QPossibleValueSource tablePVS = qInstance.getPossibleValueSource(TestMetaDataProducingEntity.TABLE_NAME);
+      assertEquals(QPossibleValueSourceType.TABLE, tablePVS.getType());
+      assertEquals(TestMetaDataProducingEntity.TABLE_NAME, tablePVS.getTableName());
+
+      //////////////////////////////////////////////////////////////////
+      // annotation on parent table w/ joined child -> join meta data //
+      //////////////////////////////////////////////////////////////////
+      String joinName = QJoinMetaData.makeInferredJoinName(TestMetaDataProducingEntity.TABLE_NAME, TestMetaDataProducingChildEntity.TABLE_NAME);
+      assertTrue(qInstance.getJoins().containsKey(joinName));
+      QJoinMetaData join = qInstance.getJoin(joinName);
+      assertEquals(TestMetaDataProducingEntity.TABLE_NAME, join.getLeftTable());
+      assertEquals(TestMetaDataProducingChildEntity.TABLE_NAME, join.getRightTable());
+      assertEquals(JoinType.ONE_TO_MANY, join.getType());
+      assertEquals("id", join.getJoinOns().get(0).getLeftField());
+      assertEquals("parentId", join.getJoinOns().get(0).getRightField());
+
+      //////////////////////////////////////////////////////////////////////////////////////
+      // annotation on parent table w/ joined child -> child record list widget meta data //
+      //////////////////////////////////////////////////////////////////////////////////////
+      assertTrue(qInstance.getWidgets().containsKey(joinName));
+      QWidgetMetaDataInterface widget = qInstance.getWidget(joinName);
+      assertEquals(WidgetType.CHILD_RECORD_LIST.getType(), widget.getType());
+      assertEquals("Test Children", widget.getLabel());
+      assertEquals(joinName, widget.getDefaultValues().get("joinName"));
+      assertEquals(false, widget.getDefaultValues().get("canAddChildRecord"));
+      assertNull(widget.getDefaultValues().get("manageAssociationName"));
+      assertEquals(15, widget.getDefaultValues().get("maxRows"));
+
    }
 
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/possiblevalues/QPossibleValueSourceTest.java

@@ -0,0 +1,96 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.possiblevalues;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Unit test for QPossibleValueSource 
+ *******************************************************************************/
+class QPossibleValueSourceTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testWithValuesFromEnum()
+   {
+      assertThatThrownBy(() -> new QPossibleValueSource().withValuesFromEnum(DupeIds.values()))
+         .isInstanceOf(QRuntimeException.class)
+         .hasMessageContaining("Duplicated id(s)")
+         .hasMessageMatching(".*: \\[1]$");
+   }
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private enum DupeIds implements PossibleValueEnum<Integer>
+   {
+      ONE_A(1, "A"),
+      TWO_B(2, "B"),
+      ONE_C(1, "C");
+
+
+      private final int id;
+      private final String label;
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      DupeIds(int id, String label)
+      {
+         this.id = id;
+         this.label = label;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public Integer getPossibleValueId()
+      {
+         return id;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public String getPossibleValueLabel()
+      {
+         return label;
+      }
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/producers/TestMetaDataProducingChildEntity.java

@@ -0,0 +1,140 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ ** QRecord Entity for TestMetaDataProducingEntity table
+ *******************************************************************************/
+public class TestMetaDataProducingChildEntity extends QRecordEntity implements MetaDataProducerInterface<QTableMetaData>
+{
+   public static final String TABLE_NAME = "testMetaDataProducingChildEntity";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(possibleValueSourceName = TestMetaDataProducingEntity.TABLE_NAME)
+   private Integer parentId;
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QTableMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME)
+         .withFieldsFromEntity(TestMetaDataProducingChildEntity.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Default constructor
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that takes a QRecord
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for parentId
+    *******************************************************************************/
+   public Integer getParentId()
+   {
+      return (this.parentId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for parentId
+    *******************************************************************************/
+   public void setParentId(Integer parentId)
+   {
+      this.parentId = parentId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for parentId
+    *******************************************************************************/
+   public TestMetaDataProducingChildEntity withParentId(Integer parentId)
+   {
+      this.parentId = parentId;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/producers/TestMetaDataProducingEntity.java

@@ -0,0 +1,119 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildJoin;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildRecordListWidget;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.ChildTable;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+
+
+/*******************************************************************************
+ ** QRecord Entity for TestMetaDataProducingEntity table
+ *******************************************************************************/
+@QMetaDataProducingEntity(producePossibleValueSource = true,
+   childTables =
+      {
+         @ChildTable(childTableEntityClass = TestMetaDataProducingChildEntity.class,
+            childJoin = @ChildJoin(enabled = true),
+            childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Test Children", maxRows = 15))
+      }
+)
+public class TestMetaDataProducingEntity extends QRecordEntity implements MetaDataProducerInterface<QTableMetaData>
+{
+   public static final String TABLE_NAME = "testMetaDataProducingEntity";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public QTableMetaData produce(QInstance qInstance) throws QException
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME)
+         .withFieldsFromEntity(TestMetaDataProducingEntity.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Default constructor
+    *******************************************************************************/
+   public TestMetaDataProducingEntity()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that takes a QRecord
+    *******************************************************************************/
+   public TestMetaDataProducingEntity(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public TestMetaDataProducingEntity withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/metadata/producers/TestMetaDataProducingPossibleValueEnum.java

@@ -0,0 +1,74 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.metadata.producers;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.producers.annotations.QMetaDataProducingPossibleValueEnum;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+@QMetaDataProducingPossibleValueEnum(producePossibleValueSource = true)
+public enum TestMetaDataProducingPossibleValueEnum implements PossibleValueEnum<Integer>
+{
+   ONE(1, "One"),
+   TWO(2, "Two");
+
+
+   private final int    id;
+   private final String label;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   TestMetaDataProducingPossibleValueEnum(int id, String label)
+   {
+      this.id = id;
+      this.label = label;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getPossibleValueLabel()
+   {
+      return label;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public Integer getPossibleValueId()
+   {
+      return id;
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/CollectionUtilsTest.java

@@ -26,10 +26,12 @@ import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Hashtable;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.Set;
 import java.util.TreeMap;
 import java.util.function.Function;
 import com.google.gson.reflect.TypeToken;
@@ -618,4 +620,23 @@ class CollectionUtilsTest extends BaseTest
          4, Map.of("B", "B4")), output);
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testAddIfNotNull()
+   {
+      HashSet<String> s = new HashSet<>();
+      CollectionUtils.addIfNotNull(s, null);
+      assertEquals(Set.of(), s);
+
+      CollectionUtils.addIfNotNull(s, "");
+      assertEquals(Set.of(""), s);
+
+      CollectionUtils.addIfNotNull(s, "1");
+      assertEquals(Set.of("", "1"), s);
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/StringUtilsTest.java

@@ -318,4 +318,19 @@ class StringUtilsTest extends BaseTest
       assertEquals("Apples were eaten", StringUtils.pluralFormat(2, "Apple{,s} {was,were} eaten"));
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testEmptyToNull()
+   {
+      assertNull(StringUtils.emptyToNull(null));
+      assertNull(StringUtils.emptyToNull(""));
+      assertNull(StringUtils.emptyToNull(" "));
+      assertNull(StringUtils.emptyToNull("  "));
+      assertEquals("a", StringUtils.emptyToNull("a"));
+   }
+
 }

qqq-backend-module-rdbms/src/main/java/com/kingsrook/qqq/backend/module/rdbms/actions/AbstractRDBMSAction.java

@@ -972,10 +972,15 @@ public abstract class AbstractRDBMSAction
             {
                sql = Objects.requireNonNullElse(sql, "").toString()
                   .replaceAll("FROM ", "\nFROM\n   ")
+                  .replaceAll("UNION ", "\nUNION\n   ")
+                  .replaceAll("INTERSECT ", "\nINTERSECT\n   ")
+                  .replaceAll("EXCEPT ", "\nEXCEPT\n   ")
                   .replaceAll("INNER", "\n   INNER")
                   .replaceAll("LEFT", "\n   LEFT")
                   .replaceAll("RIGHT", "\n   RIGHT")
-                  .replaceAll("WHERE", "\nWHERE\n   ");
+                  .replaceAll("WHERE", "\nWHERE\n   ")
+                  .replaceAll("ORDER BY", "\nORDER BY\n   ")
+                  .replaceAll("GROUP BY", "\nGROUP BY\n   ");
             }
 
             if(System.getProperty("qqq.rdbms.logSQL.output", "logger").equalsIgnoreCase("system.out"))

qqq-backend-module-rdbms/src/main/java/com/kingsrook/qqq/backend/module/rdbms/actions/RDBMSQueryAction.java

@@ -36,7 +36,6 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
 import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
 import com.kingsrook.qqq.backend.core.actions.tables.helpers.ActionTimeoutHelper;
 import com.kingsrook.qqq.backend.core.context.QContext;
@@ -46,6 +45,7 @@ import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.QueryHint;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.JoinsContext;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
@@ -95,35 +95,10 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
          QTableMetaData table     = queryInput.getTable();
          String         tableName = queryInput.getTableName();
 
-         Selection     selection = makeSelection(queryInput);
+         List<Serializable> params    = new ArrayList<>();
-         StringBuilder sql = new StringBuilder(selection.selectClause());
+         Selection          selection = makeSelection(queryInput);
 
-         QQueryFilter filter       = clonedOrNewFilter(queryInput.getFilter());
+         StringBuilder sql = makeSQL(queryInput, selection, tableName, params, table);
-         JoinsContext joinsContext = new JoinsContext(QContext.getQInstance(), tableName, queryInput.getQueryJoins(), filter);
-
-         List<Serializable> params = new ArrayList<>();
-         sql.append(" FROM ").append(makeFromClause(QContext.getQInstance(), tableName, joinsContext, params));
-         sql.append(" WHERE ").append(makeWhereClause(joinsContext, filter, params));
-
-         if(filter != null && CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
-         {
-            sql.append(" ORDER BY ").append(makeOrderByClause(table, filter.getOrderBys(), joinsContext));
-         }
-
-         if(filter != null && filter.getLimit() != null)
-         {
-            sql.append(" LIMIT ").append(filter.getLimit());
-
-            if(filter.getSkip() != null)
-            {
-               // todo - other sql grammars?
-               sql.append(" OFFSET ").append(filter.getSkip());
-            }
-         }
-
-         // todo sql customization - can edit sql and/or param list
-
-         setSqlAndJoinsInQueryStat(sql, joinsContext);
 
          Connection connection;
          boolean    needToCloseConnection = false;
@@ -258,6 +233,99 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private StringBuilder makeSQL(QueryInput queryInput, Selection selection, String tableName, List<Serializable> params, QTableMetaData table) throws QException
+   {
+      QQueryFilter filter       = clonedOrNewFilter(queryInput.getFilter());
+      JoinsContext joinsContext = new JoinsContext(QContext.getQInstance(), tableName, queryInput.getQueryJoins(), filter);
+
+      StringBuilder sql = new StringBuilder();
+
+      if(filter != null && filter.getSubFilterSetOperator() != null && CollectionUtils.nullSafeHasContents(filter.getSubFilters()))
+      {
+         for(QQueryFilter subFilter : filter.getSubFilters())
+         {
+            if(!sql.isEmpty())
+            {
+               sql.append(" ").append(filter.getSubFilterSetOperator().name().replace('_', ' ')).append(" ");
+            }
+
+            sql.append(" (");
+            sql.append(selection.selectClause());
+            sql.append(" FROM ").append(makeFromClause(QContext.getQInstance(), tableName, joinsContext, params));
+            sql.append(" WHERE ").append(makeWhereClause(joinsContext, subFilter, params));
+            sql.append(") ");
+         }
+
+         if(CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
+         {
+            /////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // the base version of makeOrderByClause uses `table`.`column` style references - which don't work for //
+            // these kinds of queries... so, use this version, which does index-based ones (maybe we could/should  //
+            // switch to always use those?                                                                         //
+            // the best here might be, to alias all columns, and then use those aliases in both versions...        //
+            /////////////////////////////////////////////////////////////////////////////////////////////////////////
+            sql.append(" ORDER BY ").append(makeOrderByClauseForSubFilterSetOperationQuery(table, filter.getOrderBys(), joinsContext, selection));
+         }
+      }
+      else
+      {
+         sql.append(selection.selectClause());
+         sql.append(" FROM ").append(makeFromClause(QContext.getQInstance(), tableName, joinsContext, params));
+         sql.append(" WHERE ").append(makeWhereClause(joinsContext, filter, params));
+
+         if(filter != null && CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
+         {
+            sql.append(" ORDER BY ").append(makeOrderByClause(table, filter.getOrderBys(), joinsContext));
+         }
+      }
+
+      if(filter != null && filter.getLimit() != null)
+      {
+         sql.append(" LIMIT ").append(filter.getLimit());
+
+         if(filter.getSkip() != null)
+         {
+            // todo - other sql grammars?
+            sql.append(" OFFSET ").append(filter.getSkip());
+         }
+      }
+
+      // todo sql customization - can edit sql and/or param list
+
+      setSqlAndJoinsInQueryStat(sql, joinsContext);
+      return sql;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private String makeOrderByClauseForSubFilterSetOperationQuery(QTableMetaData table, List<QFilterOrderBy> orderBys, JoinsContext joinsContext, Selection selection)
+   {
+      List<String> clauses = new ArrayList<>();
+
+      for(QFilterOrderBy orderBy : orderBys)
+      {
+         String                                ascOrDesc                     = orderBy.getIsAscending() ? "ASC" : "DESC";
+         JoinsContext.FieldAndTableNameOrAlias otherFieldAndTableNameOrAlias = joinsContext.getFieldAndTableNameOrAlias(orderBy.getFieldName());
+
+         QFieldMetaData field  = otherFieldAndTableNameOrAlias.field();
+         String         column = getColumnName(field);
+
+         String qualifiedColumn = escapeIdentifier(otherFieldAndTableNameOrAlias.tableNameOrAlias()) + "." + escapeIdentifier(column);
+         String columnNo        = String.valueOf(selection.qualifiedColumns.indexOf(qualifiedColumn) + 1);
+         clauses.add(columnNo + " " + ascOrDesc);
+
+      }
+      return (String.join(", ", clauses));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -282,10 +350,11 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
    /***************************************************************************
     ** output wrapper for makeSelection method.
     ** - selectClause is everything from SELECT up to (but not including) FROM
+    ** - qualifiedColumns is a list of the `table`.`column` strings
     ** - fields are those being selected, in the same order, and with mutated
     ** names for join fields.
     ***************************************************************************/
-   private record Selection(String selectClause, List<QFieldMetaData> fields)
+   private record Selection(String selectClause, List<String> qualifiedColumns, List<QFieldMetaData> fields)
    {
 
    }
@@ -318,10 +387,11 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
       // map those field names to columns, joined with ", ".                                                              //
       // if a field is heavy, and heavy fields aren't being selected, then replace that field name with a LENGTH function //
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      String columns = fieldList.stream()
+      List<String> qualifiedColumns = new ArrayList<>(fieldList.stream()
          .map(field -> Pair.of(field, escapeIdentifier(tableName) + "." + escapeIdentifier(getColumnName(field))))
          .map(pair -> wrapHeavyFieldsWithLengthFunctionIfNeeded(pair, queryInput.getShouldFetchHeavyFields()))
-         .collect(Collectors.joining(", "));
+         .toList());
+      String columns = String.join(", ", qualifiedColumns);
 
       ///////////////////////////////////////////////////////////////////////////////////////////////////////////
       // figure out if distinct is being used.  then start building the select clause with the table's columns //
@@ -360,10 +430,13 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
             /////////////////////////////////////////////////////
             // map to columns, wrapping heavy fields as needed //
             /////////////////////////////////////////////////////
-            String joinColumns = joinFieldList.stream()
+            List<String> qualifiedJoinColumns = joinFieldList.stream()
                .map(field -> Pair.of(field, escapeIdentifier(tableNameOrAlias) + "." + escapeIdentifier(getColumnName(field))))
                .map(pair -> wrapHeavyFieldsWithLengthFunctionIfNeeded(pair, queryInput.getShouldFetchHeavyFields()))
-               .collect(Collectors.joining(", "));
+               .toList();
+
+            qualifiedColumns.addAll(qualifiedJoinColumns);
+            String joinColumns = String.join(", ", qualifiedJoinColumns);
 
             ////////////////////////////////////////////////////////////////////////////////////////////////
             // append to output objects.                                                                  //
@@ -380,7 +453,7 @@ public class RDBMSQueryAction extends AbstractRDBMSAction implements QueryInterf
          }
       }
 
-      return (new Selection(selectClause.toString(), selectionFieldList));
+      return (new Selection(selectClause.toString(), qualifiedColumns, selectionFieldList));
    }
 
 

qqq-backend-module-rdbms/src/test/java/com/kingsrook/qqq/backend/module/rdbms/actions/RDBMSQueryActionSubFilterSetOperatorTest.java

@@ -0,0 +1,171 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.rdbms.actions;
+
+
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.module.rdbms.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** test for subfilter set
+ *******************************************************************************/
+public class RDBMSQueryActionSubFilterSetOperatorTest extends RDBMSActionTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeEach
+   public void beforeEach() throws Exception
+   {
+      super.primeTestDatabase();
+
+      // AbstractRDBMSAction.setLogSQL(true, true, "system.out");
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterEach
+   void afterEach()
+   {
+      AbstractRDBMSAction.setLogSQL(false);
+      QContext.getQSession().removeValue(QSession.VALUE_KEY_USER_TIMEZONE);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private QueryInput initQueryRequest()
+   {
+      QueryInput queryInput = new QueryInput();
+      queryInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+      return queryInput;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testUnion() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.UNION)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2, 3)))
+         .withOrderBy(new QFilterOrderBy("id", false))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(3, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(3, queryOutput.getRecords().get(0).getValueInteger("id"));
+      assertEquals(2, queryOutput.getRecords().get(1).getValueInteger("id"));
+      assertEquals(1, queryOutput.getRecords().get(2).getValueInteger("id"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testUnionAll() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.UNION_ALL)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2, 3)))
+         .withOrderBy(new QFilterOrderBy("id", false))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(4, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(3, queryOutput.getRecords().get(0).getValueInteger("id"));
+      assertEquals(2, queryOutput.getRecords().get(1).getValueInteger("id"));
+      assertEquals(2, queryOutput.getRecords().get(2).getValueInteger("id"));
+      assertEquals(1, queryOutput.getRecords().get(3).getValueInteger("id"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testIntersect() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.INTERSECT)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2, 3)))
+         .withOrderBy(new QFilterOrderBy("id", false))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(1, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(2, queryOutput.getRecords().get(0).getValueInteger("id"));
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void testExcept() throws QException
+   {
+      QueryInput queryInput = initQueryRequest();
+      queryInput.setFilter(new QQueryFilter()
+         .withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.EXCEPT)
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 1, 2, 3)))
+         .withSubFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN, 2)))
+         .withOrderBy(new QFilterOrderBy("id", true))
+      );
+
+      QueryOutput queryOutput = new RDBMSQueryAction().execute(queryInput);
+      assertEquals(2, queryOutput.getRecords().size(), "Expected # of rows");
+      assertEquals(1, queryOutput.getRecords().get(0).getValueInteger("id"));
+      assertEquals(3, queryOutput.getRecords().get(1).getValueInteger("id"));
+   }
+
+}

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/backend/javalin/QJavalinImplementation.java

@@ -44,6 +44,7 @@ import java.util.Optional;
 import java.util.function.Supplier;
 import com.fasterxml.jackson.core.type.TypeReference;
 import com.kingsrook.qqq.backend.core.actions.async.AsyncJobManager;
+import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
 import com.kingsrook.qqq.backend.core.actions.dashboard.RenderWidgetAction;
 import com.kingsrook.qqq.backend.core.actions.metadata.MetaDataAction;
 import com.kingsrook.qqq.backend.core.actions.metadata.ProcessMetaDataAction;
@@ -51,6 +52,8 @@ import com.kingsrook.qqq.backend.core.actions.metadata.TableMetaDataAction;
 import com.kingsrook.qqq.backend.core.actions.permissions.PermissionCheckResult;
 import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
 import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
+import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallbackFactory;
+import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
 import com.kingsrook.qqq.backend.core.actions.reporting.ExportAction;
 import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
@@ -77,6 +80,7 @@ import com.kingsrook.qqq.backend.core.model.actions.metadata.ProcessMetaDataInpu
 import com.kingsrook.qqq.backend.core.model.actions.metadata.ProcessMetaDataOutput;
 import com.kingsrook.qqq.backend.core.model.actions.metadata.TableMetaDataInput;
 import com.kingsrook.qqq.backend.core.model.actions.metadata.TableMetaDataOutput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportOutput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination;
@@ -106,6 +110,7 @@ import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
@@ -130,6 +135,7 @@ import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
 import com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeConsumer;
 import com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeFunction;
+import com.kingsrook.qqq.middleware.javalin.misc.DownloadFileSupplementalAction;
 import io.javalin.Javalin;
 import io.javalin.apibuilder.EndpointGroup;
 import io.javalin.http.Context;
@@ -1089,12 +1095,13 @@ public class QJavalinImplementation
             throw (new QNotFoundException("Could not find " + table.getLabel() + " with " + table.getFields().get(table.getPrimaryKeyField()).getLabel() + " of " + primaryKey));
          }
 
-         String                   mimeType              = null;
+         String                    mimeType              = null;
-         Optional<FieldAdornment> fileDownloadAdornment = fieldMetaData.getAdornments().stream().filter(a -> a.getType().equals(AdornmentType.FILE_DOWNLOAD)).findFirst();
+         Optional<FieldAdornment>  fileDownloadAdornment = fieldMetaData.getAdornments().stream().filter(a -> a.getType().equals(AdornmentType.FILE_DOWNLOAD)).findFirst();
+         Map<String, Serializable> adornmentValues       = null;
          if(fileDownloadAdornment.isPresent())
          {
-            Map<String, Serializable> values = fileDownloadAdornment.get().getValues();
+            adornmentValues = fileDownloadAdornment.get().getValues();
-            mimeType = ValueUtils.getValueAsString(values.get(AdornmentType.FileDownloadValues.DEFAULT_MIME_TYPE));
+            mimeType = ValueUtils.getValueAsString(adornmentValues.get(AdornmentType.FileDownloadValues.DEFAULT_MIME_TYPE));
          }
 
          if(mimeType != null)
@@ -1107,7 +1114,56 @@ public class QJavalinImplementation
             context.header("Content-Disposition", "attachment; filename=" + filename);
          }
 
-         context.result(getOutput.getRecord().getValueByteArray(fieldName));
+         //////////////////////////////////////////////////////////////////////////////////////////////
+         // if the adornment has a supplemental process name in it, or a supplemental code reference //
+         // then execute that custom code - e.g., to log that the file was downloaded.               //
+         //////////////////////////////////////////////////////////////////////////////////////////////
+         if(fileDownloadAdornment.isPresent())
+         {
+            String processName = ValueUtils.getValueAsString(adornmentValues.get(AdornmentType.FileDownloadValues.SUPPLEMENTAL_PROCESS_NAME));
+            if(StringUtils.hasContent(processName))
+            {
+               RunProcessInput input = new RunProcessInput();
+               input.setProcessName(processName);
+               input.setCallback(QProcessCallbackFactory.forRecord(getOutput.getRecord()));
+               input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
+               input.addValue("tableName", tableName);
+               input.addValue("primaryKey", primaryKey);
+               input.addValue("fieldName", fieldName);
+               input.addValue("filename", filename);
+               new RunProcessAction().execute(input);
+            }
+            else if(adornmentValues.containsKey(AdornmentType.FileDownloadValues.SUPPLEMENTAL_CODE_REFERENCE))
+            {
+               QCodeReference codeReference = (QCodeReference) adornmentValues.get(AdornmentType.FileDownloadValues.SUPPLEMENTAL_CODE_REFERENCE);
+
+               DownloadFileSupplementalAction action = QCodeLoader.getAdHoc(DownloadFileSupplementalAction.class, codeReference);
+
+               DownloadFileSupplementalAction.DownloadFileSupplementalActionInput input = new DownloadFileSupplementalAction.DownloadFileSupplementalActionInput()
+                  .withTableName(tableName)
+                  .withFieldName(fieldName)
+                  .withPrimaryKey(primaryKey)
+                  .withFileName(filename);
+
+               DownloadFileSupplementalAction.DownloadFileSupplementalActionOutput output = new DownloadFileSupplementalAction.DownloadFileSupplementalActionOutput();
+               action.run(input, output);
+            }
+         }
+
+         /////////////////////////////////////////////////////////
+         // if the field is a BLOB - send the bytes to the user //
+         /////////////////////////////////////////////////////////
+         if(QFieldType.BLOB.equals(fieldMetaData.getType()))
+         {
+            context.result(getOutput.getRecord().getValueByteArray(fieldName));
+         }
+         else
+         {
+            //////////////////////////////////////////////////////////////////
+            // else - assume a string is a URL - and issue a redirect to it //
+            //////////////////////////////////////////////////////////////////
+            context.redirect(getOutput.getRecord().getValueString(fieldName));
+         }
 
          QJavalinAccessLogger.logEndSuccess();
       }

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/misc/DownloadFileSupplementalAction.java

@@ -0,0 +1,198 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.middleware.javalin.misc;
+
+
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
+/*******************************************************************************
+ ** custom code that can run when user downloads a file.  Set as a code-reference
+ ** on a field adornment.
+ *******************************************************************************/
+public interface DownloadFileSupplementalAction
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   void run(DownloadFileSupplementalActionInput input, DownloadFileSupplementalActionOutput output) throws QException;
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   class DownloadFileSupplementalActionInput
+   {
+      private String tableName;
+      private String primaryKey;
+      private String fieldName;
+      private String fileName;
+
+
+
+      /*******************************************************************************
+       ** Getter for tableName
+       **
+       *******************************************************************************/
+      public String getTableName()
+      {
+         return tableName;
+      }
+
+
+
+      /*******************************************************************************
+       ** Setter for tableName
+       **
+       *******************************************************************************/
+      public void setTableName(String tableName)
+      {
+         this.tableName = tableName;
+      }
+
+
+
+      /*******************************************************************************
+       ** Fluent setter for tableName
+       **
+       *******************************************************************************/
+      public DownloadFileSupplementalActionInput withTableName(String tableName)
+      {
+         this.tableName = tableName;
+         return (this);
+      }
+
+
+
+      /*******************************************************************************
+       ** Getter for primaryKey
+       **
+       *******************************************************************************/
+      public String getPrimaryKey()
+      {
+         return primaryKey;
+      }
+
+
+
+      /*******************************************************************************
+       ** Setter for primaryKey
+       **
+       *******************************************************************************/
+      public void setPrimaryKey(String primaryKey)
+      {
+         this.primaryKey = primaryKey;
+      }
+
+
+
+      /*******************************************************************************
+       ** Fluent setter for primaryKey
+       **
+       *******************************************************************************/
+      public DownloadFileSupplementalActionInput withPrimaryKey(String primaryKey)
+      {
+         this.primaryKey = primaryKey;
+         return (this);
+      }
+
+
+
+      /*******************************************************************************
+       ** Getter for fieldName
+       **
+       *******************************************************************************/
+      public String getFieldName()
+      {
+         return fieldName;
+      }
+
+
+
+      /*******************************************************************************
+       ** Setter for fieldName
+       **
+       *******************************************************************************/
+      public void setFieldName(String fieldName)
+      {
+         this.fieldName = fieldName;
+      }
+
+
+
+      /*******************************************************************************
+       ** Fluent setter for fieldName
+       **
+       *******************************************************************************/
+      public DownloadFileSupplementalActionInput withFieldName(String fieldName)
+      {
+         this.fieldName = fieldName;
+         return (this);
+      }
+
+
+
+      /*******************************************************************************
+       ** Getter for fileName
+       **
+       *******************************************************************************/
+      public String getFileName()
+      {
+         return fileName;
+      }
+
+
+
+      /*******************************************************************************
+       ** Setter for fileName
+       **
+       *******************************************************************************/
+      public void setFileName(String fileName)
+      {
+         this.fileName = fileName;
+      }
+
+
+
+      /*******************************************************************************
+       ** Fluent setter for fileName
+       **
+       *******************************************************************************/
+      public DownloadFileSupplementalActionInput withFileName(String fileName)
+      {
+         this.fileName = fileName;
+         return (this);
+      }
+
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   class DownloadFileSupplementalActionOutput
+   {
+
+   }
+}

qqq-middleware-javalin/src/test/java/com/kingsrook/qqq/backend/javalin/QJavalinImplementationTest.java

@@ -29,8 +29,11 @@ import java.net.URLEncoder;
 import java.nio.charset.StandardCharsets;
 import java.util.HashMap;
 import java.util.Map;
+import java.util.Optional;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Function;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.exceptions.QInstanceValidationException;
 import com.kingsrook.qqq.backend.core.logging.QCollectingLogger;
@@ -39,12 +42,19 @@ import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
 import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceLambda;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.modules.backend.implementations.mock.MockBackendModule;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
 import com.kingsrook.qqq.backend.core.utils.SleepUtils;
+import com.kingsrook.qqq.middleware.javalin.misc.DownloadFileSupplementalAction;
 import kong.unirest.HttpResponse;
 import kong.unirest.Unirest;
 import org.apache.logging.log4j.Level;
@@ -282,6 +292,70 @@ class QJavalinImplementationTest extends QJavalinTestBase
 
 
 
+   /*******************************************************************************
+    ** test downloading from a URL field
+    **
+    *******************************************************************************/
+   @Test
+   public void test_dataDownloadRecordFieldUrl()
+   {
+      try
+      {
+         TestDownloadFileSupplementalAction.callCount = 0;
+
+         Unirest.config().reset();
+         Unirest.config().followRedirects(false);
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // first request - has no custom code - should just give us back a redirect to the value in the field //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////
+         HttpResponse<String> response = Unirest.get(BASE_URL + "/data/person/1/licenseScanPdfUrl/License-1.pdf").asString();
+         assertEquals(302, response.getStatus());
+         assertThat(response.getHeaders().get("location").get(0)).contains("https://");
+
+         ////////////////////////////////////////////////////
+         // set a code-reference on the download adornment //
+         ////////////////////////////////////////////////////
+         Optional<FieldAdornment> fileDownloadAdornment = QJavalinImplementation.getQInstance().getTable(TestUtils.TABLE_NAME_PERSON)
+            .getField("licenseScanPdfUrl")
+            .getAdornment(AdornmentType.FILE_DOWNLOAD);
+         fileDownloadAdornment.get().withValue(AdornmentType.FileDownloadValues.SUPPLEMENTAL_CODE_REFERENCE, new QCodeReference(TestDownloadFileSupplementalAction.class));
+
+         /////////////////////////////////////////
+         // request again - assert the code ran //
+         /////////////////////////////////////////
+         assertEquals(0, TestDownloadFileSupplementalAction.callCount);
+         response = Unirest.get(BASE_URL + "/data/person/1/licenseScanPdfUrl/License-1.pdf").asString();
+         assertEquals(302, response.getStatus());
+         assertThat(response.getHeaders().get("location").get(0)).contains("https://");
+         assertEquals(1, TestDownloadFileSupplementalAction.callCount);
+
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // set adornment to run process (note, leaving the code-ref - this demonstrates that process "trumps" if both exist) //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         AtomicInteger processRunCount = new AtomicInteger(0);
+         QJavalinImplementation.getQInstance().addProcess(new QProcessMetaData().withName("testDownloadProcess").withStep(
+            new QBackendStepMetaData().withName("execute").withCode(new QCodeReferenceLambda<BackendStep>((input, output) -> processRunCount.incrementAndGet()))
+         ));
+         fileDownloadAdornment.get().withValue(AdornmentType.FileDownloadValues.SUPPLEMENTAL_PROCESS_NAME, "testDownloadProcess");
+
+         /////////////////////////////////////////
+         // request again - assert the code ran //
+         /////////////////////////////////////////
+         response = Unirest.get(BASE_URL + "/data/person/1/licenseScanPdfUrl/License-1.pdf").asString();
+         assertEquals(302, response.getStatus());
+         assertThat(response.getHeaders().get("location").get(0)).contains("https://");
+         assertEquals(1, TestDownloadFileSupplementalAction.callCount);
+         assertEquals(1, processRunCount.get());
+      }
+      finally
+      {
+         Unirest.config().reset();
+      }
+   }
+
+
+
    /*******************************************************************************
     ** test a table get (single record) for an id that isn't found
     **
@@ -1395,4 +1469,19 @@ class QJavalinImplementationTest extends QJavalinTestBase
       }
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static class TestDownloadFileSupplementalAction implements DownloadFileSupplementalAction
+   {
+      static int callCount = 0;
+
+      @Override
+      public void run(DownloadFileSupplementalActionInput input, DownloadFileSupplementalActionOutput output) throws QException
+      {
+         callCount++;
+      }
+   }
 }

qqq-middleware-javalin/src/test/java/com/kingsrook/qqq/backend/javalin/TestUtils.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.javalin;
 import java.io.InputStream;
 import java.nio.charset.StandardCharsets;
 import java.sql.Connection;
+import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Objects;
@@ -316,6 +317,8 @@ public class TestUtils
          .withField(new QFieldMetaData("testScriptId", QFieldType.INTEGER).withBackendName("test_script_id"))
          .withField(new QFieldMetaData("photo", QFieldType.BLOB).withBackendName("photo"))
          .withField(new QFieldMetaData("photoFileName", QFieldType.STRING).withBackendName("photo_file_name"))
+         .withField(new QFieldMetaData("licenseScanPdfUrl", QFieldType.STRING).withBackendName("license_scan_pdf_url"))
+
          .withAssociation(new Association().withName("pets").withJoinName("personJoinPet").withAssociatedTableName(TABLE_NAME_PET))
          .withAssociatedScript(new AssociatedScript()
             .withFieldName("testScriptId")
@@ -331,6 +334,11 @@ public class TestUtils
             .withValue(AdornmentType.FileDownloadValues.DEFAULT_MIME_TYPE, "image")
             .withValue(AdornmentType.FileDownloadValues.FILE_NAME_FIELD, "photoFileName"));
 
+      qTableMetaData.getField("licenseScanPdfUrl")
+         .withFieldAdornment(new FieldAdornment(AdornmentType.FILE_DOWNLOAD)
+            .withValue(AdornmentType.FileDownloadValues.FILE_NAME_FORMAT, "License-%s.pdf")
+            .withValue(AdornmentType.FileDownloadValues.FILE_NAME_FORMAT_FIELDS, new ArrayList<>(List.of("id"))));
+
       return (qTableMetaData);
    }
 

qqq-middleware-javalin/src/test/resources/prime-test-database.sql

@@ -33,10 +33,11 @@ CREATE TABLE person
    partner_person_id INT,
    test_script_id INT,
    photo BLOB,
-   photo_file_name VARCHAR(50)
+   photo_file_name VARCHAR(50),
+   license_scan_pdf_url VARCHAR(250)
 );
 
-INSERT INTO person (id, first_name, last_name, birth_date, email, partner_person_id, photo, photo_file_name) VALUES (1, 'Darin', 'Kelkhoff', '1980-05-31', 'darin.kelkhoff@gmail.com', 6, '12345', 'darin-photo.png');
+INSERT INTO person (id, first_name, last_name, birth_date, email, partner_person_id, photo, photo_file_name, license_scan_pdf_url) VALUES (1, 'Darin', 'Kelkhoff', '1980-05-31', 'darin.kelkhoff@gmail.com', 6, '12345', 'darin-photo.png', 'https://somedomain/somepath.pdf');
 INSERT INTO person (id, first_name, last_name, birth_date, email) VALUES (2, 'James', 'Maes', '1980-05-15', 'jmaes@mmltholdings.com');
 INSERT INTO person (id, first_name, last_name, birth_date, email) VALUES (3, 'Tim', 'Chamberlain', '1976-05-28', 'tchamberlain@mmltholdings.com');
 INSERT INTO person (id, first_name, last_name, birth_date, email) VALUES (4, 'Tyler', 'Samples', '1990-01-01', 'tsamples@mmltholdings.com');