Merge branch 'dev' into feature/CE-779-order-level-ship-date

docs/Introduction.adoc

@@ -1,8 +1,176 @@
 = Introduction
+include::variables.adoc[]
 
-QQQ is ...
+QQQ is a Low-code Application Framework for Engineers.
+Its purpose is to provide the basic structural elements of an application - things that every application needs - so that the engineers building an application on top of QQQ don't need to worry about those pieces, and can instead focus on the unique needs of the application that they are building.
 
-- Framework
-- Declarative
-- Easy thing easy; Hard thing possible
-- Customizable
+== What makes QQQ special?
+
+The scope of what QQQ provides is far-reaching, and most likely goes beyond what you may initially be thinking.
+That is to say, QQQ includes code all the way from the backend of an application, through its middleware layer, and including its frontend.
+For example, a common set of modules deployed in a QQQ application will provide:
+
+* Backend RDBMS/Database connectivity and access.
+* Frontend web UI (e.g., a React application)
+* A java web server acting as middleware between the frontend web UI and the backend
+
+That is to say - as an engineer deploying a QQQ application - you do not need to write a single line of code that is concerned with any of those things.
+
+* You do not need to write code to connect to your database.
+* You do not write any web UI code.
+* You do not write any middleware code to tie together the frontend and backend.
+
+Instead, QQQ includes *all* of these pieces.
+QQQ knows how to connect to databases (and actually, several other kinds of backend systems - but ignore that for now).
+Plus it knows how to do most of what an application needs to do with a database (single-record lookups, complex queries, joins, aggregates, bulk inserts, updates, deletes).
+QQQ also knows how to present the data from a database - in table views, or single-record views, or exports, or reports or widgets.
+And it knows how to present a powerful ad-hoc query interface to users, and how to show screens where users can create, update, and delete records.
+It also provides the connective tissue (middleware) between those backend layers where data is stored, and frontend layers were users interact with data.
+
+== What makes your application special?
+
+I've said a lot about what does QQQ knows - but let's dig a little deeper.
+What does QQQ know, and what does it not know?
+Well - what it doesn't know is, it doesn't know the special or unique aspects of the application that you are building.
+
+So, what makes your application unique?
+
+Is your application unique because it needs to have screens where users can search for records in a table?
+
+No!
+
+QQQ assumes (as does the author of this document) that all applications (at least of the nature that QQQ supports) need what we call Query Screens.
+So QQQ gives you a Query Screen for all of your tables - with zero code from you.
+
+Is your application unique because you want users to be able to view, create, edit, and delete records from tables?
+
+No!
+
+QQQ assumes that all applications need these basic https://en.wikipedia.org/wiki/Create,_read,_update_and_delete[CRUD] capabilities.
+So QQQ provides all of these UI screens - for view, create, edit, delete - along with the supporting middleware and backend code - all the way down to the SQL that selects & manages the data.
+You get it all for free - zero code.
+
+Is your application unique because you have a `fiz_bar` table, with 47 columns, and a `whoz_zat` table, with 42 columns of its own, that joins to `fiz_bar` on the `zizzy_ziz_id` field?
+
+Yes!
+
+OK - we found some of it - what makes your application unique is the data that you're working with.
+Your tables - their fields - the connection info for your database.
+QQQ doesn't know those details.
+So - that's your first job as a QQQ application engineer - to describe your data to QQQ.
+
+For the example above - you need to tell QQQ that you have a `fiz_bar` table, and that you have a `whoz_zat` table - and you need to tell QQQ what the fields or columns in thsoe tables are.
+You can even tell QQQ how to join those tables (on that `zizzy_ziz_id` field).
+And then that's it.
+Once QQQ has this {link-table} meta data, it can then provide its Query screens, and full CRUD screens to your tables, in your database, with your fields.
+
+And at the risk of repeating myself - you can do this (get a full Query & CRUD application) with zero lines of actual procedural code.
+You only need to supply meta-data (which, at the time of this writing, is done in Java, but it's just creating objects - and in the future could be done in YAML files, for example).
+
+== Beyond Basics
+Going beyond the basic wiring as described above, QQQ also provides some of the more advanced elements needed in a modern data-driven web application, including:
+
+* Authentication & Authorization
+* ad-hoc Query engine for access to data tables
+* Full CRUD (Create, Read, Update, Delete) capabilities
+* Multistep custom workflows ("Processes" in QQQ parlance)
+* Scheduled jobs
+* Enterprise Service Bus
+
+So what do we mean by all of this?
+We said that basically every application needs, for example, Authentication & Authorization.
+Login screens.
+User & Role tables.
+Permissions.
+So, when it's time for you to build a new application for your _Big Tall Floor Lamp_ manufacturing business, do you need to start by writing a login screen?
+And a Permissions scheme?
+And throwing HTTP 401 errors?
+And managing user-role relationships?
+And then having a bug in the check permission logic on the _Light Bulb Inventory Edit_ screen, so Jim is always keying in bad quantities, even though he isn't supposed to have permission there?
+
+No!
+
+All of the (really important, even though application developers hate doing it) aspects of security - you don't need to write ANY code for dealing with that.
+Just tell QQQ what Authentication provider you want to use (e.g., https://auth0.com/[Auth0]), and - to paraphrase the old https://www.youtube.com/watch?v=YHzM4avGrKI[iMac ad] - there's no step 2.
+QQQ just does it.
+
+''''
+
+QQQ can provide this type of application using a variety and/or combination of backend data storage types.
+And, whichever type of backend is used, QQQ gives a common interface (both user-facing and programmer-facing).
+Backend types include:
+
+* Relational Databases (RDBMS)
+* File Systems
+* Web APIs
+* NoSQL/Document Databases (_Future_)
+
+In addition, out-of-the-box, QQQ also goes beyond these basics, delivering:
+
+* Bulk versions of all CRUD operations.
+* Automatically generated JSON APIs.
+* Auditing of data changes.
+* End-user (e.g., non-engineer) customization via dynamic scripting capabilities
+
+#todo say much more#
+
+== QQQ Architecture
+
+Like a house!
+
+== Developing a QQQ Application
+In developing an application with QQQ, engineers will generally have to define two types of code:
+
+. *Meta Data* - This is the code that you use to tell QQQ the shape of your application - your unique Tables, Processes, Apps, Reports, Widgets, etc.
+In general, this code is 100% declarative in nature (as opposed to procedural or functional).
+That is to say - it has no logic.
+It is just a definition of what exists - but it has no instructions, algorithms, or business logic.
+
+* _In a future version of QQQ, we anticipate being able to define meta-data in a format such as YAML or JSON, or to even load it from a relational or document database.
+This speaks to the fact that this "code" is not executable code - but rather is a simple declaration of (meta) data._
+* A key function of QQQ then is to drive all of its layers of functionality - frontend UIs, middleware, and core backend actions (e.g., ORM operations) - based on this meta-data.
+** For example:
+... You define the meta-data for a table in your application - including its fields and their data types, as well as what backend system the table exists within.
+... Then, the QQQ Frontend Material Dashboard UI's Query Screen loads that table's meta-data, and uses it to control the screen that is presented.  Including:
+**** The data grid shown on the screen will have columns for each field in the table.
+**** The Filter button in the Query Screen will present a menu listing all fields from the table for the user to build ad-hoc queries against the table.
+The data-types specified for the fields (in the meta-data) dictate what operators QQQ allows the user to use against fields (e.g., Strings offer "contains" vs Numbers offer "greater than").
+**** Values for records from the table will be formatted for presentation based on the meta-data (such as a numeric field being shown with commas if it represents a quantity, or formatted as currency).
+...
+
+[start=2]
+. *Meta Data* - declarative code - java object instances (potentially which could be read from `.yaml` files or other data sources in a future version of QQQ), which tell QQQ about the backend systems, tables, processes, reports, widgets, etc, that make up the application.
+For example:
+* Details about the database you are using, and how to connect to it.
+* A database table's name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).
+* The description of web API - its URL and authentication mechanism.
+* A table/path within a web API, and the fields returned in the JSON at that endpoint.
+* The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.
+* Details about a chart that summarizes data from a table for presentation as a dashboard widget.
+// the section below is kinda dumb.  like, it says you have to write application code, but
+// then it just talks about how your app code gets for-free the same shit that QQQ does.
+// it should instead say more about what your custom app code is or does.
+// 2. *Application code* - to customize beyond what the framework does out-of-the box, and to provide application-specific business-logic.
+// QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model.
+// For example:
+// * The same record-security model that is enforced for ad-hoc user queries through the frontend is applied to custom application code.
+// ** So if your table has a security key defined, which says that users can only see Order records that are associated with the user's assigned Store, then QQQ's order Query Screen will enforce that rule.
+// ** And at the same time - any custom processes ran by a user will have the same security applied to any queries that they run against the Order table.
+// ** And any custom dashboard widgets - will only include data that the user is allowed to see.
+// * Record audits are performed in custom code the same as they are in framework-driven actions.
+// ** So if a custom process edits a record, details of the changed fields show up in the record's audit, the same as if the record was edited using the standard QQQ edit action.
+// * Changed records are sent through the ESB automatically regardless of whether they are updated by custom application code or standard framework code.
+// ** Meaning record automations are triggered regardless of how a record is created or edited - without you, as an application engineering, needing to send records through the bus.
+// * The multi-threaded, paged producer/consumer pattern used in standard framework actions is how all custom application actions are also invoked.
+// ** For example, the standard QQQ Bulk Edit action uses the same streamed-ETL process that custom application processes can use.
+// Meaning your custom processes can take full advantage of the same complex frontend, middleware, and backend structural pieces, and you can just focus on your unique busines logic needs.
+2. *Application code* - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic.
+QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model.
+For example:
+
+== Lifecycle?
+* define meta data
+** enrichment
+** validation
+* start application
+* for dev - hotSwap

docs/actions/QueryAction.adoc

@@ -1,34 +1,45 @@
 == QueryAction
 include::../variables.adoc[]
 
-The `*QueryAction*` is the basic action that is used to get records from a {link-table}.
+The `*QueryAction*` is the basic action that is used to get records from a {link-table}, generally according to a <<QQueryFilter,Filter>>.
 In SQL/RDBMS terms, it is analogous to a `SELECT` statement, where 0 or more records may be found and returned.
 
 === Examples
 ==== Basic Form
 [source,java]
 ----
-QueryInput input = new QueryInput(qInstance);
-input.setSession(session);
+QueryInput input = new QueryInput();
 input.setTableName("orders");
 input.setFilter(new QQueryFilter()
    .withCriteria(new QFilterCriteria("total", GREATER_THAN, new BigDecimal("3.50")))
-   .withOrderBy(new QFilterOrderBy("orderDate", false))
-);
+   .withOrderBy(new QFilterOrderBy("orderDate", false)));
 QueryOutput output = new QueryAction.execute(input);
 List<QRecord> records = output.getRecords();
 ----
 
+=== Details
+`QueryAction`, in general, can be called in two different modes:
+
+. The most common use-case case, and default, fetches all records synchronously, does any post-processing (as requested in the <<QueryInput>>), and returns all records as a list in the <<QueryOutput>>).
+. The alternative use-case is meant for larger operations, where one wouldn't want all records matching a query in-memory.
+For this scenario, a `RecordPipe` object can be passed in to the <<QueryInput>>.
+This causes `QueryAction` to run its post-processing action on records as they are placed into the pipe, and to potentially block (per the pipe's settings).
+This method of usage needs to be done on a separate thread from another thread which would be consuming records from the pipe.
+QQQ's `AsyncRecordPipeLoop` class provides an implementation of doing such a dual-threaded job.
+
+If the {link-table} has a `POST_QUERY_CUSTOMIZER` defined, then after records are fetched from the backend, that code is executed on the records before they leave the `QueryAction` (either through its `QueryOutput` or `RecordPipe`).
+
 === QueryInput
 * `table` - *String, Required* - Name of the table being queried against.
-* `filter` - *QQueryFilter object* - Specification for what records should be returned, based on *QFilterCriteria* objects, and how they should be sorted, based on *QFilterOrderBy* objects.
+* `filter` - *<<QQueryFilter>> object* - Specification for what records should be returned, based on *<<QFilterCriteria>>* objects, and how they should be sorted, based on *<<QFilterOrderBy>>* objects.
+If a `filter` is not given, then all rows in the table will be returned by the query.
 * `skip` - *Integer* - Optional number of records to be skipped at the beginning of the result set.
 e.g., for implementing pagination.
 * `limit` - *Integer* - Optional maximum number of records to be returned by the query.
 * `transaction` - *QBackendTransaction object* - Optional transaction object.
 ** Behavior for this object is backend-dependant.
 In an RDBMS backend, this object is generally needed if you want your query to see data that may have been modified within the same transaction.
-* `recordPipe` - *RecordPipe object* - Optional object that records are placed into, for asynchronous processing.
+* `recordPipe` - *RecordPipe object* - Optional pipe object that records are placed into, for asynchronous processing.
 ** If a *recordPipe* is used, then records cannot be retrieved from the *QueryOutput*.
 Rather, such records must be read from the pipe's `consumeAvailableRecords()` method.
 ** A *recordPipe* should only be used when a *QueryAction* is running in a separate Thread from the record's consumer.
@@ -36,18 +47,21 @@ Rather, such records must be read from the pipe's `consumeAvailableRecords()` me
 (e.g., to provide text translations in the generated records' `displayValues` map).
 ** For example, if running a query to present results to a user, this would generally need to be *true*.
 But if running a query to provide data as part of a process, then this can generally be left as *false*.
-* `shouldGenerateDisplayValues` - *boolean, default: false* - Controls whether if field level *displayFormats* should be used to populate the generated records' `displayValues` map.
+* `shouldGenerateDisplayValues` - *boolean, default: false* - Controls whether field level *displayFormats* should be used to populate the generated records' `displayValues` map.
 ** For example, if running a query to present results to a user, this would generally need to be *true*.
 But if running a query to provide data as part of a process, then this can generally be left as *false*.
-* `queryJoins` - *List of QueryJoin objects* - Optional list of tables to be joined with the main *table* specified in the *QueryInput*.
+* `shouldFetchHeavyFields` - *boolean, default: true* - Controls whether or not fields marked as `isHeavy` should be fetched & returned or not.
+* `shouldOmitHiddenFields` - *boolean, default: true* - Controls whether or not fields marked as `isHidden` should be included in the result or not.
+* `shouldMaskPassword` - *boolean, default: true* - Controls whether or not fields with `type` = `PASSWORD` should be masked, or if their actual values should be returned.
+* `queryJoins` - *List of <<QueryJoin>> objects* - Optional list of tables to be joined with the main table being queried.
 See QueryJoin below for further details.
 
 ==== QQueryFilter
-A key component of *QueryInput*, a *QQueryFilter* defines both what records should be included in a query's results (e.g., an SQL `WHERE`), as well as how those results should be sorted (SQL `ORDER BY`).
+A key component of *<<QueryInput>>*, a *QQueryFilter* defines both what records should be included in a query's results (e.g., an SQL `WHERE`), as well as how those results should be sorted (SQL `ORDER BY`).
 
-* `criteria` - *List of QFilterCriteria* - Individual conditions or clauses to filter records.
+* `criteria` - *List of <<QFilterCriteria>>* - Individual conditions or clauses to filter records.
 They are combined using the *booleanOperator* specified in the *QQueryFilter*.  See below for further details.
-* `orderBys` - *List of QFilterOrderBy* - List of fields (and directions) to control the sorting of query results.
+* `orderBys` - *List of <<QFilterOrderBy>>* - List of fields (and directions) to control the sorting of query results.
 In general, multiple *orderBys* can be given (depending on backend implementations).
 * `booleanOperator` - *Enum of AND, OR, default: AND* - Specifies the logical joining operator used among individual criteria.
 * `subFilters` - *List of QQueryFilter* - To build arbitrarily complex queries, with nested boolean logic, 0 or more *subFilters* may be provided.
@@ -69,7 +83,7 @@ In general, multiple *orderBys* can be given (depending on backend implementatio
     )));
 
 // which would generate the following WHERE clause in an RDBMS backend:
-   WHERE (first_name='James' AND last_name='Maes') OR (first_name='Darin' AND last_name='Kelkhoff')
+// WHERE (first_name='James' AND last_name='Maes') OR (first_name='Darin' AND last_name='Kelkhoff')
 ----
 
 ===== QFilterCriteria
@@ -79,19 +93,19 @@ In general, multiple *orderBys* can be given (depending on backend implementatio
 * `operator` - *Enum of QCriteriaOperator, required* - Comparison operation to be applied to the field specified as *fieldName* and the *values* or *otherFieldName*.
 ** e.g., `EQUALS`, `NOT_IN`, `GREATER_THAN`, `BETWEEN`, `IS_BLANK`, etc.
 * `values` - *List of values, conditional* - Provides the value(s) that the field is compared against.
-The number of values (0, 1, 2, or more) be driven based on the *operator* being used.
+The number of values (0, 1, 2, or more) required are based on the *operator* being used.
 If an *otherFieldName* is given, and the *operator* expects 1 value, then *values* is ignored, and *otherFieldName* is used.
 * `otherFieldName` - *String, conditional* - Specifies that the *fieldName* should be compared against another field in the records, rather than the values in the *values* property.
 Only used for *operators* that expect 1 value (e.g., `EQUALS` or `LESS_THAN_OR_EQUALS` - not `IS_NOT_BLANK` or `IN`).
 
-QFilterCriteria definition examples:
 [source,java]
+.QFilterCriteria definition examples:
 ----
-// one-liners, via constructors that take (List<Serializable> values) or (Serializable... values) in 3rd position
-new QFilterCriteria("id", IN, List.of(1, 2, 3))
+// in-line, via constructors that take (List<Serializable> values) or (Serializable... values) as 3rd arg
+new QFilterCriteria("id", IN, 1, 2, 3)
 new QFilterCriteria("name", IS_BLANK)
 new QFilterCriteria("orderNo", IN, orderNoList)
-new QFilterCriteria("state", EQUALS, "MO");
+new QFilterCriteria("state", EQUALS, "MO")
 
 // long-form, with fluent setters
 new QFilterCriteria()
@@ -105,7 +119,7 @@ new QFilterCriteria()
    .withOpeartor(QCriteriaOperator.EQUALS)
    .withOtherFieldName("lastName");
 
-// using otherFieldName to build a criterion that looks at two fields from join tables
+// using otherFieldName to build a criterion that looks at two fields from two different join tables
 new QFilterCriteria()
    .withFieldName("billToCustomer.lastName")
    .withOpeartor(QCriteriaOperator.NOT_EQUALS)
@@ -118,9 +132,9 @@ new QFilterCriteria()
 ** Or, in the case of a query with *queryJoins*, a qualified name of a field from a join-table (where the qualifier would be the joined table's name or alias, followed by a dot)
 * `isAscending` - *boolean, default: true* - Specify if the sort is ascending or descending.
 
-QFilterCriteria definition examples:
 
 [source,java]
+.QFilterOrderBy definition examples:
 ----
 // short-form, via constructors
 new QFilterOrderBy("id") // isAscending defaults to true.
@@ -129,7 +143,7 @@ new QFilterOrderBy("name", false)
 // long-form, with fluent setters
 new QFilterOrderBy()
    .withFieldName("birthDate")
-   .withIsAscending(true);
+   .withIsAscending(false);
 ----
 
 ==== QueryJoin
@@ -147,9 +161,8 @@ If given, must be used as the part before the dot in field name specifications t
 If *true*, then the `QRecord` objects returned by this query will have values with corresponding to the (table-or-alias `.` field-name) form.
 * `type` - *Enum of INNER, LEFT, RIGHT, FULL, default: INNER* - specifies the SQL-style type of join being performed.
 
-QueryJoin definition examples:
-
 [source,java]
+.QueryJoin definition examples:
 ----
 // selecting from an "orderLine" table - then join to its corresponding "order" table
 queryInput.withTableName("orderLine");

docs/docinfo.html

@@ -1,27 +1,6 @@
-<!--
-  ~ QQQ - Low-code Application Framework for Engineers.
-  ~ Copyright (C) 2021-2022.  Kingsrook, LLC
-  ~ 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
-  ~ contact@kingsrook.com
-  ~ https://github.com/Kingsrook/
-  ~
-  ~ This program is free software: you can redistribute it and/or modify
-  ~ it under the terms of the GNU Affero General Public License as
-  ~ published by the Free Software Foundation, either version 3 of the
-  ~ License, or (at your option) any later version.
-  ~
-  ~ This program is distributed in the hope that it will be useful,
-  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of
-  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-  ~ GNU Affero General Public License for more details.
-  ~
-  ~ You should have received a copy of the GNU Affero General Public License
-  ~ along with this program.  If not, see <https://www.gnu.org/licenses/>.
-  -->
-
 <style>
-   pre
+   .sect2 + .sect2
    {
-      white-space: pre;
+      border-top: 1px solid #e7e7e9;
    }
 </style>

docs/implementations/TableSync.adoc

@@ -0,0 +1,406 @@
+== TableSyncProcess
+include::../variables.adoc[]
+
+The `TableSyncProcess` is designed to help solve the common use-case where you have a set of records in one table,
+you want to apply a transformation to them, and then you want them to result in records in a different table.
+
+=== Sample Scenario
+For example, you may be receiving an import file feed from a partner - say, a CSV file of order records
+- that you need to synchronize into your local database's orders table.
+
+=== High Level Steps
+At a high-level, the steps of this task are:
+
+1. Get the records from the partner's feed.
+2. Decide Insert/Update - e.g., if you already have any of these orders in your database table, in which case, they need updated, versus ones
+you don't have, which need inserted.
+3. Map fields from the partner's order record to your own fields.
+4. Store the necessary records in your database (insert or update).
+
+=== What you need to tell QQQ
+`TableSyncProcess`, as defined by QQQ, knows how to do everything for a job like this, other than the part that's unique to your business.
+Those unique parts that you need to tell QQQ are:
+
+* What are the source & destination tables?
+* What are the criteria to identify source records to be processed?
+* What are the key fields are that "link" together records from the source & destination tables?
+** For the example above, imagine you have a "partnerOrderNo" field in your database's order table - then you'd need
+to tell QQQ what field in the partner's table provides the value for that field in your table.
+* How do fields / values from the source table map to fields & values in the destination table?
+
+=== Detailed Steps
+To get more specific, in a QQQ `TableSyncProcess`, here's what happens for each of the high-level steps given above:
+
+* Getting records from the partner's feed (done by QQQ):
+** Records from the source will be fetched via an Extract step, which runs a query to find the records that need processing.
+** Depending on your use-case, you may use an `ExtractViaQueryStep` (maybe with a Table Automation) or `ExtractViaBasepullQueryStep`
+(e.g., if you are polling a remote data source).
+* Deciding Insert/Update (done by QQQ):
+** Given  a set of records from the source table (e.g., output of the Extract step mentioned above), get values from the "key" field in that table.
+** Do a lookup in the destination table, where its corresponding "key" field has the values extracted from the source records.
+** For each source record, if its "key" was found in the destination table, then plan an update to that existing corresponding
+destination record; else, plan to insert a new record in the destination table.
+* Mapping values (done by your custom Application code):
+** Specifically, mapping is done in a subclass of `AbstractTableSyncTransformStep`, in the `populateRecordToStore` method.
+Of particular interest are these two parameters to that method:
+*** `QRecord destinationRecord` - as determined by the insert/update logic above, this will either be a
+new empty record (e.g., for inserting), or a fully populated record from the destination table (for updating).
+*** `QRecord sourceRecord` - this is the record being processed, from the source table.
+** This method is responsible for setting values in the `destinationRecord`, and returning that record
+(unless it has decided that for some reason the record should _not_ be stored, in which case it may return `null`).
+* Storing the records (done by QQQ):
+** This is typically done with the `LoadViaInsertOrUpdateStep`, though it is customizable if additional work is needed (e.g., via a
+subclass of `LoadViaInsertOrUpdateStep`, or a more custom subclass of `AbstractLoadStep`).
+
+=== Bare-bones Example
+For this example, let's assume we're setting up a partner-order-feed as described above, with the following details:
+
+* We have records from a partner in a table named `"partnerOrderImport"` (let's assume the records may have been created
+using the QQQ `FilesystemImporter` process).
+These records have the following fields:
+** `orderNo, date, city, state, postal, whseNo`
+* We need to synchronize those records with table in our database named `"order"`, with the following fields corresponding to those from the partner:
+** `partnerOrderNo, orderDate, shipToCity, shipToState, shipToZipCode, warehouseId`
+* The same conceptual order may appear in the `"partnerOrderImport"` multiple times, e.g., if they update some data on the order and re-transmit it to us.
+Meaning, we need to update our `"order"` records when we receive a new version an existing order.
+
+To use `*TableSyncProcess*` for solving this use-case, we'll need to create 2 things:
+
+1. A `QProcessMetaData` object, which we can create using the builder object provided by class `TableSyncProcess`.
+Note that this type of process is specialization of the standard QQQ `StreamedETLWithFrontendProcess`, as described elsewhere in this documentation.
+2. A subclass of `AbstractTableSyncTransformStep`, where we implement our mapping logic.
+Again, note that `AbstractTableSyncTransformStep` is a subclass of `AbstractTransformStep`, as used by `StreamedETLWithFrontendProcess`.
+
+And to be good programmers, we'll actually create a 3rd thing:
+
+[start=3]
+. A unit test for our Transform step.
+
+Here are examples of these pieces of code:
+
+[source,java]
+.Example of building process using the TableSyncProcess builder:
+----
+// the false argument below tells the build we are not a basepull-style process
+QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)
+
+    // give our process a unique name within our QInstance
+    .withName("partnerOrderToLocalOrderProcess")
+
+    // tell the process to what class to use for transforming records from source to destination
+    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)
+
+    .getProcessMetaData();
+----
+
+[source,java]
+.Example implementation of an AbstractTableSyncTransformStep
+----
+public class PartnerOrderToOrderTransformStep extends AbstractTableSyncTransformStep
+{
+   @Override
+   protected SyncProcessConfig getSyncProcessConfig()
+   {
+      return (new SyncProcessConfig(
+         "partnerOrderImport", // source tableName
+         "orderNo",            // source table key fieldName
+         "order",              // destination tableName
+         "partnerOrderNo"      // destination table foreign key fieldName
+      ));
+   }
+
+   @Override
+   public QRecord populateRecordToStore(RunBackendStepInput runBackendStepInput, QRecord destinationRecord, QRecord sourceRecord) throws QException
+   {
+      // map simple values from source table to destination table
+      destinationRecord.setValue("orderDate", sourceRecord.get("date"));
+      destinationRecord.setValue("shipToAddressCity", sourceRecord.get("city"));
+      destinationRecord.setValue("shipToAddressState", sourceRecord.get("state"));
+      destinationRecord.setValue("shipToAddressZipCode", sourceRecord.get("postal"));
+      return (destinationRecord);
+   }
+
+}
+----
+
+[source,java]
+.Example Unit Test for a transform step
+----
+   @Test
+   void testTransformStep()
+   {
+      // insert 1 test order, that will be updated by the transform step
+      Integer existingId = new InsertAction().execute(new InsertInput("order").withRecords(List.of(
+         new QRecord().withValue("partnerOrderNo", 101).withValue("shipToState", "IL")
+      ))).getRecords().get(0).getValueInteger("id");
+
+      // set up input for the step - a list of 2 of the partner's orders
+      RunBackendStepInput input = new RunBackendStepInput();
+      input.setRecords(List.of(
+         new QRecord().withValue("orderNo", 101).withValue("state", "NY"), // will update the order above
+         new QRecord().withValue("orderNo", 102).withValue("state", "CA")  // will insert a new order
+      ));
+      RunBackendStepOutput output = new RunBackendStepOutput();
+
+      // run the code under test - our transform step
+      new PartnerOrderToOrderTransformStep().run(input, output);
+
+      // Note that by just running the transform step, no records have been stored.
+      // We can assert against the output of this step.
+
+      assertEquals(existingId, output.getRecords().get(0).getValue("id"));
+      assertEquals(101, output.getRecords().get(0).getValue("partnerOrderNo"));
+      assertEquals("NY", output.getRecords().get(0).getValue("shipToState"));
+
+      assertNull(output.getRecords().get(1).getValue("id"));
+      assertEquals(102, output.getRecords().get(1).getValue("partnerOrderNo"));
+      assertEquals("CA", output.getRecords().get(1).getValue("shipToState"));
+   }
+
+   @Test
+   void testFullProcess()
+   {
+      // todo! :)
+   }
+----
+
+=== Pseudocode process flow
+Now that we've seen the bare-bones example, let's see an even more detailed breakdown of how a full `TableSyncProcess` works,
+by looking at its 3 "ETL" steps in pseudocode:
+
+==== ExtractStep (Producer Thread)
+
+* Queries source table for records
+** Often based on Table Automations (e.g., for all newly inserted records) or Basepull pattern (polling for new/updated records).
+
+==== TransformStep (Consumer Thread)
+
+* Receives pages of records from `ExtractStep` in the `run` method.
+* Makes `sourceKeyList` by getting `sourceTableKeyField` values from the records.
+* Calls `initializeRecordLookupHelper(runBackendStepInput, sourceRecordList)`
+** Calls `getLookupsToPreLoad` to control which lookups are performed.
+* Calls `getExistingRecordsByForeignKey(runBackendStepInput, destinationTableForeignKeyField, destinationTableName, sourceKeyList);`
+** Calls `getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList)` as part of querying the `destinationTable`
+** Returns the output of `buildExistingRecordsMap(destinationTableForeignKeyField, queryOutput.getRecords())`
+* foreach input record (from `sourceTable`):
+** Calls `getExistingRecord(existingRecordsByForeignKey, destinationForeignKeyField, sourceKeyValue)`
+** if an existing record was returned (and if the syncConfig says `performUpdates`), this record is set as `recordToStore`
+** else if no existing record was returned (and if the syncConfig says `performInserts`), a new record is set as `recordToStore`
+** else continue the foreach.
+** call `populateRecordToStore(runBackendStepInput, recordToStore, sourceRecord)`
+** if a record is returned it is added to the process step output (to be stored in the LoadStep)
+
+==== LoadStep (Consumer Thread)
+
+* Receives records from the output of the `TransformStep`.
+* Inserts and/or Updates `destinationTable`, with records returned by `populateRecordToStore`
+
+=== Additional Process Configuration Examples
+The following examples show how to use additional settings in the `TableSyncProcess` builder.
+
+==== UI
+While a `TableSyncProcess` will often run via a schedule and/or automation, we may also want to allow users
+to manually run it in a UI.
+
+[source,java]
+.Making our process available for a UI
+----
+QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(true)
+    .withName("partnerOrderToLocalOrderProcess")
+    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)
+
+    // attach our process to its source table, to show up in UI
+    .withTableName("partnerOrderImport")
+
+    // add some fields to display on the review screen, in UI
+    .withReviewStepRecordFields(List.of(
+       new QFieldMetaData("clientId", QFieldType.STRING).withLabel("Client"),
+       new QFieldMetaData("warehouseId", QFieldType.STRING).withLabel("Warehouse"),
+       new QFieldMetaData("partnerOrderNo", QFieldType.STRING)))
+    .getProcessMetaData();
+----
+
+==== Basepull
+The previous example would work as a Table Automation (e.g., where the list of records identified in the
+Extract step were determined by the Automation system).
+However, a second common pattern is to use `Basepull` (e.g., if polling for updated records from a partner API endpoint).
+
+[source,java]
+.Configuring our process as a Basepull
+----
+// the true argument below tells the build we ARE a basepull-style process
+// this changes the default extract-step.
+QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)
+    .withName("partnerOrderToLocalOrderProcess")
+    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)
+
+    // See Basepull documentation for details
+    .withBasepullConfiguration(new BasepullConfiguration())
+
+    // schedule our process to run automatically every minute
+    .withSchedule(new QScheduleMetaData().withRepeatSeconds(60))
+
+    .getProcessMetaData();
+----
+
+=== Additional Options in the Transform Step
+
+==== Specifying to not perform Inserts or not perform Updates
+We may have a scenario where we want our sync process to never update records if the key is already found in the destination table.
+We can configure this with an additional optional parameter to the `SyncProcessConfig` constructor:
+
+[source,java]
+.Specifying to not do updates in a TableSyncProcess
+----
+   @Override
+   protected SyncProcessConfig getSyncProcessConfig()
+   {
+      return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
+         true,  // performInserts
+         false  // performUpdates
+      ));
+   }
+----
+
+Similarly, we may want to disallow inserts from a particular sync process.
+The `performInserts` argument to the `SyncProcessConfig` constructor lets us do that:
+
+[source,java]
+.Specifying to not do inserts in a TableSyncProcess
+----
+   @Override
+   protected SyncProcessConfig getSyncProcessConfig()
+   {
+      return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
+         false, // performInserts
+         true   // performUpdates
+      ));
+   }
+----
+
+==== Customizing the query for existing records
+
+In some cases, a specific Table Sync process may need to refine the query filter that is used
+to lookup existing records in the destination table (e.g. for determining insert vs. update).
+
+For example, in our orders-from-a-partner scenario, if we have more than 1 partner sending us orders,
+where there could be overlapping orderNo values among them - we may have an additional field in our
+orders table to identify which partner an order came from.
+So then when we're looking up orders by `partnerOrderNo`, we would need to also include the `partnerId` field
+in our query, so that we only update orders from the specific partner that we're dealing with.
+
+To do this (to customize the existing record query filter), we need can just override the method `getExistingRecordQueryFilter`.
+Generally we would start by calling the `super` version of the method, and then add to it additional criteria.
+
+[source,java]
+.Customizing the query filter used to look for existing records
+----
+   /*******************************************************************************
+    ** Define the query filter to find existing records.  e.g., for determining
+    ** insert vs. update.  Subclasses may override this to customize the behavior,
+    ** e.g., in case an additional field is needed in the query.
+    *******************************************************************************/
+   protected QQueryFilter getExistingRecordQueryFilter(RunBackendStepInput runBackendStepInput, List<Serializable> sourceKeyList)
+   {
+      QQueryFilter filter = super.getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList);
+      filter.addCriteria(new QFilterCriteria("partnerId", EQUALS, PARTNER_ID));
+      return (filter);
+   }
+----
+
+==== More efficient additional record lookups
+
+It is a common use-case to need to map various ids from a partner's system to ids in your own system.
+For the orders example, we might need to know what warehouse the order is shipping from.
+The customer may send their identifier for the warehouse, and we may need to map those identifiers to our own warehouse ids.
+
+The QQQ-provided class `RecordLookupHelper` exists to help with performing lookups like this,
+and in particular, it can be used to execute one query to fetch a full table, storing records
+by a key field, then returning those records without performing additional queries.
+
+`AbstractTableSyncTransformStep` has a protected `recordLookupHelper` member.
+If we override the method `getLookupsToPreLoad()`, then this object is
+populated by calling its `preloadRecords` method with each specified pair of tableNames and fieldNames.
+
+[source,java]
+.Specifying tables to pre-load using a RecordLookupHelper
+----
+   /*******************************************************************************
+    ** Specify a list of tableName/keyColumnName pairs to run through
+    ** the preloadRecords method of the recordLookupHelper.
+    *******************************************************************************/
+   @Override
+   protected List<Pair<String, String>> getLookupsToPreLoad()
+   {
+      return (List.of(
+         Pair.of("warehouse", "partnerWarehouseNo")
+      ));
+   }
+----
+
+If we have preloaded some lookups, we can then use them in our `populateRecordToStore` method as follows:
+[source,java]
+.Using the recordLookupHelper in populateRecordToStore
+----
+   // lookup warehouse with partnerWarehouseNo=whseNo from partner, and use our id in destination record
+   String partnerWarehouseNo = sourceRecord.getValue("whseNo");
+   Integer warehouseId = recordLookupHelper.getRecordId("warehouse", "partnerWarehouseNo", whseNo, Integer.class);
+   destinationRecord.setValue("warehouseId", warehouseId);
+----
+
+==== Additional override points
+
+There are more methods which can be overridden in your `AbstractTableSyncTransformStep` subclass,
+to provide further customizations of behaviors, specifically in the area of dealing with existing
+records (e.g., the insert/update use-case).
+[source,java]
+.Additional AbstractTableSyncTransformStep overrides
+----
+
+   /*******************************************************************************
+    ** Run the existingRecordQueryFilter - to look in the destinationTable for
+    ** any records that may need an update (rather than an insert).
+    **
+    ** Generally returns a Map, keyed by a Pair of the destinationTableForeignKeyField
+    ** and the value in that field.  But, for more complex use-cases, one can override
+    ** the buildExistingRecordsMap method, to make different keys (e.g., if there are
+    ** two possible destinationTableForeignKeyFields).
+    *******************************************************************************/
+   protected Map<Pair<String, Serializable>, QRecord> getExistingRecordsByForeignKey
+   (
+      RunBackendStepInput runBackendStepInput,
+      String destinationTableForeignKeyField,
+      String destinationTableName,
+      List<Serializable> sourceKeyList
+   ) throws QException;
+
+
+   /*******************************************************************************
+    ** Overridable point where you can, for example, keys in the existingRecordsMap
+    ** with different fieldNames from the destinationTable.
+    **
+    ** Note, if you're overriding this method, you'll likely also want & need to
+    ** override getExistingRecord.
+    *******************************************************************************/
+   protected Map<Pair<String, Serializable>, QRecord> buildExistingRecordsMap
+   (
+      String destinationTableForeignKeyField,
+      List<QRecord> existingRecordList
+   );
+
+   /*******************************************************************************
+    ** Given the map of existingRecordsByForeignKey (as built by
+    ** getExistingRecordsByForeignKey which calls buildExistingRecordsMap),
+    ** get one record from that map, for a given key-value from a source record.
+    **
+    ** The destinationForeignKeyField is given as advice if needed (e.g., to see its type)
+    *******************************************************************************/
+   protected QRecord getExistingRecord
+   (
+      Map<Pair<String, Serializable>, QRecord> existingRecordsByForeignKey,
+      QFieldMetaData destinationForeignKeyField,
+      Serializable sourceKeyValue
+   );
+
+----
+

docs/index.adoc

@@ -1,17 +1,63 @@
 = QQQ
-:doctype: book
+:doctype: article
 :toc: left
+:toclevels: 2
 :source-highlighter: coderay
 
 include::Introduction.adoc[leveloffset=+1]
 
 == Meta Data
-include::metaData/Tables.adoc[leveloffset=+1]
-''''
-include::metaData/Reports.adoc[leveloffset=+1]
+// Organizational units
+include::metaData/QInstance.adoc[leveloffset=+1]
+include::metaData/Backends.adoc[leveloffset=+1]
+include::metaData/Apps.adoc[leveloffset=+1]
 
-== Actions
+// Primary meta-data types
+include::metaData/Tables.adoc[leveloffset=+1]
+include::metaData/Processes.adoc[leveloffset=+1]
+include::metaData/Widgets.adoc[leveloffset=+1]
+
+// Helper meta-data types
+include::metaData/Fields.adoc[leveloffset=+1]
+include::metaData/PossibleValueSources.adoc[leveloffset=+1]
+include::metaData/Joins.adoc[leveloffset=+1]
+include::metaData/SecurtiyKeyTypes.adoc[leveloffset=+1]
+include::metaData/Reports.adoc[leveloffset=+1]
+include::metaData/Icons.adoc[leveloffset=+1]
+include::metaData/PermissionRules.adoc[leveloffset=+1]
+
+
+== Custom Application Code
+include::misc/QContext.adoc[leveloffset=+1]
+include::misc/QRecords.adoc[leveloffset=+1]
+include::misc/QRecordEntities.adoc[leveloffset=+1]
+include::misc/ProcessBackendSteps.adoc[leveloffset=+1]
+
+=== Table Customizers
+#todo#
+
+== QQQ Core Actions
 include::actions/QueryAction.adoc[leveloffset=+1]
-''''
-include::actions/RenderTemplateAction.adoc[leveloffset=+1]
-''''
+
+include::actions/GetAction.adoc[leveloffset=+1]
+
+=== CountAction
+#todo#
+
+=== AggregateAction
+#todo#
+
+include::actions/InsertAction.adoc[leveloffset=+1]
+
+=== UpdateAction
+#todo#
+
+=== DeleteAction
+#todo#
+
+=== AuditAction
+#todo#
+
+== QQQ Default Implementations
+include::implementations/TableSync.adoc[leveloffset=+1]
+// later... include::actions/RenderTemplateAction.adoc[leveloffset=+1]

docs/metaData/Apps.adoc

@@ -0,0 +1,92 @@
+[#Apps]
+== Apps
+include::../variables.adoc[]
+
+QQQ User Interfaces (e.g., Material Dashboard) generally organize their contents via *Apps*.
+Apps are a lightweight construct in QQQ - basically just containers for other objects.
+
+Specifically, Apps can contain:
+
+* {link-widgets}
+* {link-tables}
+* {link-process}
+* {link-reports}
+* Other {link-apps} - to create a multi-tiered navigational hierarchy.
+
+=== QAppMetaData
+Apps are defined in a QQQ Instance in `*QAppMetaData*` objects.  Apps must consist of either 1 or more {link-widgets}, or 1 or more *Sections*, which are expected to contain 1 or more {link-tables}, {link-processes}, or {link-reports}.
+
+*QAppMetaData Properties:*
+
+* `name` - *String, Required* - Unique name for the app within the QQQ Instance.
+* `label` - *String* - User-facing label for the app, presented in User Interfaces.
+Inferred from `name` if not set.
+* `permissionRules` - *QPermissionRules* - Permissions to apply to the app.  See {link-permissionRules} for details.
+* `children` - *List of QAppChildMetaData* - Objects contained within the app. These can be {link-tables}, {link-processes}, {link-reports} or other {link-apps}.
+** See the example below for some common patterns for how these child-meta data objects are added to an App.
+* `parentAppName` - *String* - For an app which is a child of another app, the parent app's name is referenced in this field.
+** Note that this is generally automatically set when the child is added to its parent, in the `addChild` method.
+* `icon` - *QIcon* - An icon to display in a UI for the app.  See {link-icons}.
+* `widgets` - *List of String* - A list of names of {link-widgets} to include in the app.
+* `sections` - *List of <<QAppSection>>* - A list of `QAppSection` objects, to create organizational subdivisions within the app.
+** As shown in the example below, the method `withSectionOfChildren` can be used to fluently add a new `QAppSection`, along with its child objects, to both an app and a section all at once.
+
+==== QAppSection
+A `QAppSection` is an organizational subsection of a {link-app}.
+
+* `name` - *String, Required* - Unique name for the section within its app.
+* `label` - *String* - User-facing label for the section, presented in User Interfaces.
+Inferred from `name` if not set.
+* `icon` - *QIcon* - An icon to display in a UI for the section.  See {link-icons}.
+* `tables` - *List of String* - A list of names of {link-tables} to include in the section.
+* `processes` - *List of String* - A list of names of {link-processes} to include in the section.
+* `reports` - *List of String* - A list of names of {link-reports} to include in the section.
+
+*Examples*
+[source,java]
+----
+/*******************************************************************************
+** Full example of constructing a QAppMetaData object.
+*******************************************************************************/
+public class ExampleAppMetaDataProducer extends MetaDataProducer<QAppMetaData>
+{
+
+   /*******************************************************************************
+    ** Produce the QAppMetaData
+    *******************************************************************************/
+   @Override
+   public QAppMetaData produce(QInstance qInstance) throws QException
+   {
+      return (new QAppMetaData()
+         .withName("sample")
+         .withLabel("My Sample App")
+         .withIcon(new QIcon().withName("thumb_up"))
+         .withWidgets(List.of(
+            UserWelcomeWidget.NAME,
+            SystemHealthLineChartWidget.NAME))
+         .withSectionOfChildren(new QAppSection().withName("peoplePlacesAndThings"),
+            qInstance.getTable(People.TABLE_NAME),
+            qInstance.getTable(Places.TABLE_NAME),
+            qInstance.getTable(Things.TABLE_NAME),
+            qInstance.getProcess(AssociatePeopleWithPlacesProcess.NAME))
+         .withSectionOfChildren(new QAppSection().withName("math").withLabel("Mathematics"),
+            qInstance.getProcess(ComputePiProcess.NAME),
+            qInstance.getReport(PrimeNumbersReport.NAME),
+            qInstance.getReport(PolygonReport.NAME)));
+   }
+
+
+   /*******************************************************************************
+    ** Since this meta-data producer expects to find other meta-data objects in the
+    ** QInstance, give it a sortOrder higher than the default (which we'll expect
+    ** the other objects used).
+    *******************************************************************************/
+   @Override
+   public int getSortOrder()
+   {
+      return (Integer.MAX_VALUE);
+   }
+
+}
+----
+

docs/metaData/Backends.adoc

@@ -0,0 +1,150 @@
+[#Backends]
+== Backends
+include::../variables.adoc[]
+
+A key component of QQQ is its ability to connect to various backend data stores, while providing the same interfaces to those backends - both User Interfaces, and Programming Interfaces.
+
+For example, out-of-the-box, QQQ can connect to:
+
+* <<RDBMSBackendMetaData,RDBMS>> (Relational Database Management Systems, such as MySQL)
+* File Systems (<<S3BackendMetaData,Amazon S3>> or <<FilesystemBackendMetaData,local disk>>)
+* <<APIBackendMetaData,JSON Web APIs>> (_using a custom mapping class per-API backend_).
+* In-Memory data stores
+
+All {link-tables} in a QQQ instance must belong to a backend. As such, any instance using tables (which would be almost all instances) must define 1 or more backends.
+
+=== QBackendMetaData
+Backends are defined in a QQQ Instance in a `*QBackendMetaData*` object.
+These objects will have some common attributes, but many different attributes based on the type of backend being used.
+
+*QBackendMetaData Properties:*
+
+* `name` - *String, Required* - Unique name for the backend within the QQQ Instance.
+* `backendType` - *String, Required* - Identifier for the backend type being defined.
+** This attribute is typically set in the constructor of a `QBackendMetaData` subclass, and as such does not need to be set when defining a backend meta data object.
+* `enabledCapabilities` and `disabledCapability` - *Sets*, containing *Capability* enum values.
+Basic rules that apply to all tables in the backend, describing what actions (such as Delete, or Count) are supported in the backend.
+** By default, QQQ assumes that a backend supports _most_ capabilities, with one exception being `QUERY_STATS`.
+** #TODO# fully explain rules here
+* `usesVariants` - *Boolean, default false* - Control whether or not the backend supports the concept of "Variants".
+** Supporting variants means that tables within the backend can exist in alternative "variants" of the backend.
+For example, this might mean a sharded or multi-tenant database backend (perhaps a different server or database name per-client).
+Or this might mean using more than one set of credentials for connecting to an API backend - each of those credential sets would be a "variant".
+** A backend that uses variants requires additional properties to be set.  #TODO complete variant documentation#
+
+In a QQQ application, one will typically not create an instance of `QBackendMetaData` directly, but instead will create an instance of one of its subclasses, specific to the type of backend being used.
+The currently available list of such classes are:
+
+==== RDBMSBackendMetaData
+The meta data required for working with tables in an RDBMS (relational database) backend are defined in an instance of the `*RDBMSBackendMetaData*` class.
+
+*RDBMSBackendMetaData Properties:*
+
+* `vendor` - *String, Required* - Database vendor.  Currently supported values are: `aurora`, `mysql`, `h2`.
+* `jdbcUrl` - *String, Optional* - Full JDBC URL for connecting to the database.
+** If this property is provided, then following properties (which are the components of a JDBC URL) are ignored.
+In other words, you can either provide the `jdbcUrl`, or the individual components that make it up.
+* `hostName` - *String, Conditionally Required* - Hostname or ip address of the RDBMS server.
+* `port` - *Integer, Conditionally Required* - Port used to connect to the RDBMS server.
+* `databaseName` - *String, Conditionally Required* - Name of the database being connected to, within the RDBMS server.
+* `username` - *String, Conditionally Required* - Username for authenticating in the database server.
+* `password` - *String, Conditionally Required* - Password for authenticating in the database server.
+
+*Examples*
+[source,java]
+----
+/*******************************************************************************
+** Full example of constructing an RDBMSBackendMetaData
+*******************************************************************************/
+public class ExampleDatabaseBackendMetaDataProducer extends MetaDataProducer<QBackendMetaData>
+{
+   public static final String NAME = "rdbmsBackend";
+
+   /*******************************************************************************
+    ** Produce the QBackendMetaData
+    *******************************************************************************/
+   @Override
+   public QBackendMetaData produce(QInstance qInstance)
+   {
+      ///////////////////////////////////////////////////////////////////////
+      // read settings from either a .env file or the system's environment //
+      ///////////////////////////////////////////////////////////////////////
+      QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter();
+      String vendor       = interpreter.interpret("${env.RDBMS_VENDOR}");
+      String hostname     = interpreter.interpret("${env.RDBMS_HOSTNAME}");
+      String port         = interpreter.interpret("${env.RDBMS_PORT}");
+      String databaseName = interpreter.interpret("${env.RDBMS_DATABASE_NAME}");
+      String username     = interpreter.interpret("${env.RDBMS_USERNAME}");
+      String password     = interpreter.interpret("${env.RDBMS_PASSWORD}");
+
+      return (new RDBMSBackendMetaData()
+         .withName(NAME)
+         .withVendor(vendor)
+         .withHostName(hostname)
+         .withPort(ValueUtils.getValueAsInteger(port))
+         .withDatabaseName(databaseName)
+         .withUsername(username)
+         .withPassword(password)
+         .withCapability(Capability.QUERY_STATS));
+   }
+}
+----
+
+==== S3BackendMetaData
+The meta data required for working with tables in an Amazon S3 backend are defined in an instance of the `*S3BackendMetaData*` class.
+
+*S3BackendMetaData Properties:*
+
+* `bucketName` - *String, Required* - Bucket name to connect to inside AWS S3.
+* `accessKey` - *String, Required* - Access key for connecting to S3 inside AWS S3.
+* `secretKey` - *String, Required* - Secret key for connecting to S3 inside AWS S3.
+* `region` - *String, Required* - AWS region containing the Bucket in S3.
+* `basePath` - *String, Required* - Base path to the files within the S3 bucket.
+
+==== FilesystemBackendMetaData
+The meta data required for working with tables in a (local) filesystem backend are defined in an instance of the `*FilesystemBackendMetaData*` class.
+
+*FilesystemBackendMetaData Properties:*
+
+* `basePath` - *String, Required* - Base path to the backend's files.
+
+==== APIBackendMetaData
+The meta data required for working with tables in a web API are defined in an instance of the `*APIBackendMetaData*` class.
+
+QQQ provides a minimal, reasonable default implementation for working with web APIs, making assumptions such as using `POST` to insert records, and `GET` with a primary key in the URL to get a single record.
+However, in our experience, almost all APIs are implemented differently enough, that a layer of custom code is required.
+For example, query/search endpoints are almost always unique in how they take their search parameters, and how they wrap their output.
+
+To deal with this, a QQQ API Backend can define a custom class in the `actionUtil` property of the backend meta-data, as a subclass of `BaseAPIActionUtil`, which ideally can override methods at the level where unique functionality is needed.
+For example, an application need not define the full method for executing a Query against an API backend (which would need to make the HTTP request (actually multiple, to deal with pagination)).
+Instead, one can just override the `buildQueryStringForGet` method, where the unique details of making the request are defined, and maybe the `jsonObjectToRecord` method, where records are mapped from the API's response to a QRecord.
+
+#todo - full reference and examples for `BaseAPIActionUtil`#
+
+*APIBackendMetaData Properties:*
+
+* `baseUrl` - *String, Required* - Base URL for connecting to the API.
+* `contentType` - *String, Required* - value of `Content-type` header included in all requests to the API.
+* `actionUtil` - *QCodeReference, Required* - Reference to a class that extends `BaseAPIActionUtil`, where custom code for working with this API backend is defined.
+* `customValues` - *Map of String → Serializable* - Application-defined additional name=value pairs that can
+* `authorizationType` - *Enum, Required* - Specifies how authentication is provided for the API.
+The value here, dictates which other authentication-related properties are required.
+Possible values are:
+** `API_KEY_HEADER` - Uses the `apiKey` property in an HTTP header named `API-Key`.
+_In the future, when needed, QQQ will add a property, tentatively named `apiKeyHeaderName`, to allow customization of this header name._
+** `API_TOKEN` - Uses the `apiKey` property in an `Authroization` header, as: `"Token " + apiKey`
+** `BASIC_AUTH_API_KEY` - Uses the `apiKey` property, Base64-encoded, in an `Authroization` header, as `"Basic " + base64(apiKey)`
+** `BASIC_AUTH_USERNAME_PASSWORD` - Combines the `username` and `password` properties, Base64-encoded, in an `Authroization` header, as `"Basic " + base64(username + ":" + password)`
+** `OAUTH2` - Implements an OAuth2 client credentials token grant flow, using the properties `clientId` and `clientSecret`.
+By default, the id & secret are sent as query-string parameters to the API's `oauth/token` endpoint.
+Alternatively, if the meta-data has a `customValue` of `setCredentialsInHeader=true`, then the id & secret are posted in an `Authorization` header (base-64 encoded, and concatenated with `":"`).
+** `API_KEY_QUERY_PARAM` - Uses the `apiKey` property as a query-string parameter, with its name taken from the `apiKeyQueryParamName` property.
+** `CUSTOM` - Has a no-op implementation at the QQQ layer.
+Assumes that an override of `protected void handleCustomAuthorization(HttpRequestBase request)` be implemented in the backend's `actionUtil` class.
+This would be
+* `apiKey` - *String, Conditional* - See `authorizationType` above for details.
+* `clientId` - *String, Conditional* - See `authorizationType` above for details.
+* `clientSecret` - *String, Conditional* - See `authorizationType` above for details.
+* `username` - *String, Conditional* - See `authorizationType` above for details.
+* `password` - *String, Conditional* - See `authorizationType` above for details.
+* `apiKeyQueryParamName` - *String, Conditional* - See `authorizationType` above for details.

docs/metaData/Fields.adoc

@@ -1,4 +1,5 @@
-== QQQ Fields
+[#Fields]
+== Fields
 include::../variables.adoc[]
 
 QQQ Fields define

docs/metaData/Joins.adoc

@@ -0,0 +1,17 @@
+[#Joins]
+== Joins
+include::../variables.adoc[]
+
+#TODO#
+
+=== QJoinMetaData
+Joins are defined in a QQQ Instance in a `*QJoinMetaData*` object.
+
+#TODO#
+
+*QJoinMetaData Properties:*
+
+* `name` - *String, Required* - Unique name for the join within the QQQ Instance. #todo infererences or conventions?#
+
+#TODO#
+

docs/metaData/PossibleValueSources.adoc

@@ -0,0 +1,17 @@
+[#PossibleValueSources]
+== Possible Value Sources
+include::../variables.adoc[]
+
+#TODO#
+
+=== QPossibleValueSource
+A Possible Value Source is defined in a QQQ Instance in a `*QPossibleValueSource*` object.
+
+#TODO#
+
+*QPossibleValueSource Properties:*
+
+* `name` - *String, Required* - Unique name for the possible value source within the QQQ Instance.
+
+#TODO#
+

docs/metaData/Processes.adoc

@@ -0,0 +1,216 @@
+[#Processes]
+== Processes
+
+include::../variables.adoc[]
+
+Besides {link-tables}, the other most common type of object in a QQQ Instance is the Process.
+Processes are "custom" actions (e.g., defined by the application developers, rather than QQQ) that users and/or the system can execute against records.
+Processes generally are made up of two types of sub-objects:
+
+* *Screens* - i.e., User Interfaces (e.g., for gathering input and/or showing output).
+* *BackendSteps* - Java classes (of type `BackendStep`) that execute the logic of the process.
+
+=== QProcessMetaData
+
+Processes are defined in a QQQ Instance in a `*QProcessMetaData*` object.
+In addition to directly building a `QProcessMetaData` object setting its properties directly, there are a few common process patterns that provide *Builder* objects for ease-of-use.
+See StreamedETLWithFrontendProcess below for a common example
+
+*QProcessMetaData Properties:*
+
+* `name` - *String, Required* - Unique name for the process within the QQQ Instance.
+* `label` - *String* - User-facing label for the process, presented in User Interfaces.
+Inferred from `name` if not set.
+* `icon` - *QIcon* - Icon associated with this process in certain user interfaces.
+See {link-icons}.
+* `tableName` - *String* - Name of a {link-table} that the process is associated with in User Interfaces (e.g., Action menu).
+* `isHidden` - *Boolean, default false* - Option to hide the process from all User Interfaces.
+* `basepullConfiguration` - *<<BasepullConfiguration>>* - config for the common "Basepull" pattern, of identifying records with a timestamp greater than the last time the process was ran.
+See below for details.
+* `permissionRules` - *QPermissionRules object* - define the permission/access rules for the process.
+See {link-permissionRules} for details.
+* `steps` and `stepList` - *Map of String → <<QStepMetaData>>* and *List of QStepMetaData* - Defines the <<QFrontendStepMetaData,screens>> and <<QBackendStepMetaData,backend code>> that makes up the process.
+** `stepList` is the list of steps in the order that they will by default be executed.
+** `steps` is a map, including all steps from `stepList`, but which may also include steps which can used by the process if its backend steps make the decision to do so, at run-time.
+** A process's steps are normally defined in one of two was:
+*** 1) by a single call to `.withStepList(List<QStepMetaData>)`, which internally adds each step into the `steps` map.
+*** 2) by multiple calls to `.addStep(QStepMetaData)`, which adds a step to both the `stepList` and `steps` map.
+** If a process also needs optional steps, they should be added by a call to `.addOptionalStep(QStepMetaData)`, which only places them in the `steps` map.
+* `schedule` - *<<QScheduleMetaData>>* - set up the process to run automatically on the specified schedule.
+See below for details.
+* `minInputRecords` - *Integer* - #not used...#
+* `maxInputRecords` - *Integer* - #not used...#
+
+#todo: supplementalMetaData (API)#
+
+==== QStepMetaData
+
+This is the base class for the two types of steps in a process - <<QFrontendStepMetaData,screens>> and <<QBackendStepMetaData,backend code>>.
+There are some shared attributes of both of them, defined here.
+
+*QStepMetaData Properties:*
+
+* `name` - *String, Required* - Unique name for the step within the process.
+* `label` - *String* - User-facing label for the step, presented in User Interfaces.
+Inferred from `name` if not set.
+* `stepType` - *String* - _Deprecated._
+
+==== QFrontendStepMetaData
+
+For processes with a user-interface, they must define one or more "screens" in the form of `QFrontendStepMetaData` objects.
+
+*QFrontendStepMetaData Properties:*
+
+* `components` - *List of <<QFrontendComponentMetaData>>* - a list of components to be rendered on the screen.
+* `formFields` - *List of String* - list of field names used by the screen as form-inputs.
+* `viewFields` - *List of String* - list of field names used by the screen as visible outputs.
+* `recordListFields` - *List of String* - list of field names used by the screen in a record listing.
+
+==== QFrontendComponentMetaData
+
+A screen in a process may consist of multiple "components" - e.g., help text, and a form, and a list of records.
+Each of these components are defined in a `QFrontendComponentMetaData`.
+
+*QFrontendComponentMetaData Properties:*
+
+* `type` - *enum, Required* - The type of component to display.
+Each component type works with different keys in the `values` map.
+Possible values for `type` are:
+** `EDIT_FORM` - Displays a list of fields for editing, similar to a record edit screen.
+Requires that `formFields` be populated in the step.
+** `VIEW_FORM` - Displays a list of fields for viewing (not editing), similar to a record view screen.
+Requires that `viewFields` be populated in the step.
+** `HELP_TEXT` - Block of help text to be display on the screen.
+Requires an entry in the component's `values` map named `"text"`.
+** `HTML` - Block of custom HTML, generated by the process backend.
+Expects a process value named `html`.
+** `DOWNLOAD_FORM` - Presentation of a link to download a file generated by the process.
+Expects process values named `downloadFileName` and `serverFilePath`.
+** `GOOGLE_DRIVE_SELECT_FOLDER` - Special form that presents a UI from Google Drive, where the user can select a folder (e.g., as a target for uploading files in a subsequent backend step).
+** `BULK_EDIT_FORM` - For use by the standard QQQ Bulk Edit process.
+** `VALIDATION_REVIEW_SCREEN` - For use by the QQQ Streamed ETL With Frontend process family of processes.
+Displays a component prompting the user to run full validation or to skip it, or, if full validation has been ran, then showing the results of that validation.
+** `PROCESS_SUMMARY_RESULTS` - For use by the QQQ Streamed ETL With Frontend process family of processes.
+Displays the summary results of running the process.
+** `RECORD_LIST` - _Deprecated.
+Showed a grid with a list of records as populated by the process._
+* `values` - *Map of String → Serializable* - Key=value pairs, with different expectations based on the component's `type`.
+See above for details.
+
+==== QBackendStepMetaData
+
+Process Backend Steps are where custom (at this time, Java, but in the future, potentially, from any supported language) code is executed, to provide the logic of the process.
+QQQ comes with several common backend steps, such as for extracting lists of records, storing lists of records, etc.
+
+*QBackendStepMetaData Properties:*
+
+* `code` - *QCodeReference, Required* - Reference to the code to be executed for the step.
+The referenced code must implement the `BackendStep` interface.
+* `inputMetaData` - *QFunctionInputMetaData* - Definition of the data that the backend step expects and/or requires.
+Sub-properties are:
+** `fieldList` - *List of {link-fields}* - Optional list of fields used by the process step.
+In general, a process does not _have to_ specify the fields that its steps use.
+It can be used, however, for example, to cause a `defaultValue` to be applied to a field if it isn't specified in the process's input.
+It can also be used to cause the process to throw an error, if a field is marked as `isRequired`, but a value is not present.
+** `recordListMetaData` - *RecordListMetaData object* - _Not used at this time._
+
+==== BasepullConfiguration
+
+A "Basepull" process is a common pattern where an application needs to perform some action on all new (or updated) records from a particular data source.
+To implement this pattern, one needs to store the timestamp of when the action runs, then query the source for records where a date-time field is after that timestamp.
+
+QQQ helps facilitate this pattern by automatically retrieving and updating that timestamp field, and by building a default query filter based on that timestamp.
+
+This is done by adding a `BasepullConfiguration` object to a process's meta-data.
+
+*BasepullConfiguration Properties:*
+
+* `tableName` - *String, Required* - Name of a {link-table} in the QQQ Instance where the basepull timestamps are stored.
+* `keyField` - *String, Required* - Name of a {link-field} in the basepull table that stores a unique identifier for the process.
+* `keyValue` - *String* - Optional value to be stored in the `keyField` of the basepull table as the unique identifier for the process.
+If not set, then the process's `name` is used.
+* `lastRunTimeFieldName` - *String, Required* - Name of a {link-field} in the basepull table that stores the last-run time for the process.
+* `hoursBackForInitialTimestamp` - *Integer* - Optional number of hours to go back in time (from `now`) for the first time that the process is executed (i.e., if there is no timestamp stored in the basepull table).
+* `timestampField` - *String, Required* - Name of a {link-field} in the table being queried against the last-run timestamp.
+
+==== QScheduleMetaData
+
+QQQ can automatically run processes using an internal scheduler, if they are configured with a `QScheduleMetaData` object.
+
+*QScheduleMetaData Properties*
+
+* `repeatSeconds` - *Integer, Conditional* - How often the process should be executed, in seconds.
+* `repeatMillis` - *Integer, Conditional* - How often the process should be executed, in milliseconds.
+Mutually exclusive with `repeatSeconds`.
+* `initialDelaySeconds` - *Integer, Conditional* - How long between when the scheduler starts and the process should first run, in seconds.
+* `initialDelayMillis` - *Integer, Conditional* - How long between when the scheduler starts and the process should first run, in milliseconds.
+Mutually exclusive with `initialDelaySeconds`.
+* `variantRunStrategory` - *enum, Conditional* - For processes than run against {link-tables} that use a {link-backend} with Variants enabled, this property controls if the various instances of the process should run in `PARALLEL` or in `SERIAL`.
+* `variantBackend` - *enum, Conditional* - For processes than run against {link-tables} that use a {link-backend} with Variants enabled, this property specifies the name of the {link-backend}.
+
+==== StreamedETLWithFrontendProcess
+
+A common pattern for QQQ processes to exhibit is called the "Streamed ETL With Frontend" process pattern.
+This pattern is to do an "Extract, Transform, Load" job on a potentially large set of records.
+
+The records are Streamed through the process's steps, meaning, QQQ runs multiple threads - a producer, which is selecting records, and a consumer, which is processing records.
+As such, in general, an unlimited number of records can be processed by a process, without worrying about exhausting server resources (e.g., OutOfMemory).
+
+These processes also have a standard user-interface for displaying a summary of what the process will do (and has done), with a small number of records as a preview.
+The goal of the summary is to give the user the big-picture of what the process will do (e.g., X records will be inserted; Y records will be updated), along with a small view of some details on the records that will be stored (e.g., on record A field B will be set to C).
+
+This type of process uses 3 backend steps, and 2 frontend steps, as follows:
+
+* *preview* (backend) - does just a little work (limited # of rows), to give the user a preview of what the final result will be - e.g., some data to seed the review screen.
+* *review* (frontend) - a review screen, which after the preview step does not have a full process summary, but can generally tell the user how many records are input to the process, and can show a preview of a small number of the records.
+* *validate* (backend) - optionally (per input on review screen), does like the preview step, but does it for all records from the extract step.
+* *review* (frontend) - a second view of the review screen, if the validate step was executed.
+Now that the full validation was performed, a full process summary can be shown, along with a some preview records.
+* *execute* (backend) - processes all the rows - does all the work - stores data in the backend.
+* *result* (frontend) - a result screen, showing a "past-tense" version of the process summary.
+
+These backend steps are defined within QQQ, meaning they themselves do not execute any application-defined custom code.
+Instead, these steps use the following secondary <<QBackendStepMetaData,backend steps>>:
+
+* *Extract* - Fetch the rows to be processed.
+Used by preview (but only for a limited number of rows), validate (without limit), and execute (without limit).
+* *Transform* - Do whatever transformation is needed to the rows.
+Done on preview, validate, and execute.
+Always works with a page of records at a time.
+Since it is called on the preview & validate steps, it should *NOT* ever store any data (unless it does a specific check to confirm that it is being used on an *execute* step).
+* *Load* - Store the records into the backend, as appropriate.
+Only called by the execute step.
+Always works with a page of records at a time.
+
+The meta-data for a `StreamedETLWithFrontendProcess` uses several input fields on its steps.
+As such, it can be somewhat clumsy and error-prone to fully define a `StreamedETLWithFrontendProcess`.
+To improve this programmer-interface, an inner `Builder` class exists within `StreamedETLWithFrontendProcess` (generated by a call to `StreamedETLWithFrontendProcess.processMetaDataBuilder()`).
+
+*StreamedETLWithFrontendProcess.Builder methods:*
+
+* `withName(String name) - Set the name for the process.
+* `withLabel(String label) - Set the label for the process.
+* `withIcon(QIcon icon)` - Set an {link-icon} to be display with the process in the UI.
+* `withExtractStepClass(Class<? extends AbstractExtractStep>)` - Define the Extract step for the process.
+If no special extraction logic is needed, `ExtractViaQuery.class` is often a reasonable default.
+In other cases, `ExtractViaQuery` can be a reasonable class to extend for a custom extract step.
+* `withTransformStepClass(Class<? extends AbstractTransformStep>)` - Define the Transform step for the process.
+If no transformation logic is needed, `NoopTransformStep.class` can be used (though this is not very common).
+* `withLoadStepClass(Class<? extends AbstractLoadStep>)` - Define the Load step for the process.
+Several standard implementations exist, such as: `LoadViaInsertStep.class`, `LoadViaUpdateStep.class`, and `LoadViaDeleteStep.class`.
+* `withTableName(String tableName)` - Specify the name of the {link-table} that the process should be associated with in the UI.
+* `withSourceTable(String sourceTable)` - Specify the name of the {link-table} to be used as the source of records for the process.
+* `withDestinationTable(String destinationTable)` - Specify the name of the {link-table} to be used as the destination for records from the process.
+* `withSupportsFullValidation(Boolean supportsFullValidation)` - By default, all StreamedETLWithFrontendProcesses do allow the user to choose to run the full validation step.
+However, in case cases it may not make sense to do so - so this method can be used to turn off that option.
+* `withDoFullValidation(Boolean doFullValidation)` - By default, all StreamedETLWithFrontendProcesses will prompt the user if they want to run the full validation step or not.
+However, in case cases you may want to enforce that the validation step always be executed.
+Calling this method will remove the option from the user, and always run a full validation.
+* `withTransactionLevelAutoCommit()`, `withTransactionLevelPage()`, and `withTransactionLevelProcess()` - Change the transaction-level used by the process.
+By default, these processes are ran with a single transaction for all pages of their execute step.
+But for some cases, doing page-level transactions can reduce long-transactions and locking within the system.
+* `withPreviewMessage(String previewMessage)` - Sets the message shown on the validation review screen(s) above the preview records.
+* `withReviewStepRecordFields(List<QFieldMetaData> fieldList)` -
+* `withFields(List<QFieldMetaData> fieldList)` - Adds additional input fields to the preview step of the process.
+* `withBasepullConfiguration(BasepullConfiguration basepullConfiguration)` - Add a <<BasepullConfiguration>> to the process.
+* `withSchedule(QScheduleMetaData schedule)` - Add a <<QScheduleMetaData>> to the process.

docs/metaData/Reports.adoc

@@ -1,4 +1,5 @@
-== QQQ Reports
+[#Reports]
+== Reports
 include::../variables.adoc[]
 
 QQQ can generate reports based on {link-tables} defined within a QQQ Instance.

docs/metaData/SecurtiyKeyTypes.adoc

@@ -0,0 +1,17 @@
+[#SecurityKeyTypes]
+== Security Key Types
+include::../variables.adoc[]
+
+#TODO#
+
+=== QSecurityKeyType
+A Security Key Type is defined in a QQQ Instance in a `*QSecurityKeyType*` object.
+
+#TODO#
+
+*QSecurityKeyType Properties:*
+
+* `name` - *String, Required* - Unique name for the security key type within the QQQ Instance.
+
+#TODO#
+

docs/metaData/Tables.adoc

@@ -1,15 +1,16 @@
-== QQQ Tables
+[#Tables]
+== Tables
 include::../variables.adoc[]
 
-The core type of object in a QQQ Instance is the Table.
+One of the most common types of object in a QQQ Instance is the Table.
 In the most common use-case, a QQQ Table may be the in-app representation of a Database table.
 That is, it is a collection of records (or rows) of data, each of which has a set of fields (or columns).
 
-QQQ also allows other types of data sources ({link-backends}) to be used as tables, such as File systems, API's, Java enums or objects, etc.
+QQQ also allows other types of data sources ({link-backends}) to be used as tables, such as File systems, API's, etc.
 All of these backend types present the same interfaces (both user-interfaces, and application programming interfaces), regardless of their backend type.
 
 === QTableMetaData
-Tables are defined in a QQQ Instance in a `*QTableMetaData*` object.
+Tables are defined in a QQQ Instance in `*QTableMetaData*` objects.
 All tables must reference a {link-backend}, a list of fields that define the shape of records in the table, and additional data to describe how to work with the table within its backend.
 
 *QTableMetaData Properties:*
@@ -19,14 +20,37 @@ All tables must reference a {link-backend}, a list of fields that define the sha
 Inferred from `name` if not set.
 * `backendName` - *String, Required* - Name of a {link-backend} in which this table's data is managed.
 * `fields` - *Map of String → {link-field}, Required* - The columns of data that make up all records in this table.
-* `primaryKeyField` - *String, Conditional* - Name of a {link-field} that serves as the primary key (e.g., unique identifier) for records in this table.
-* `uniqueKeys` - *List of UniqueKey* - Definition of additional unique constraints (from an RDBMS point of view) from the table.
+* `primaryKeyField` - *String, Conditional* - Name of a {link-field} that serves as the primary key (unique identifier) for records in this table.
+** Whether a primary key field is required or not depends on the backend type that the table belongs to.
+* `uniqueKeys` - *List of UniqueKey* - Definition of additional unique keys or constraints (from an RDBMS point of view) from the table.
 e.g., sets of columns which must have unique values for each record in the table.
+The properties of the `UniqueKey` object are:
+** `fieldNames` - *List of String, Required* - List of field names from this table.
+** `label` - *String* - Optional label to be shown to users with error messages (e.g., for violation of this unique key).
 * `backendDetails` - *QTableBackendDetails or subclass* - Additional data to configure the table within its {link-backend}.
-* `automationDetails` - *QTableAutomationDetails* - Configuration of automated jobs that run against records in the table, e.g., upon insert or update.
+** For example, for an RDBMS-type backend, the name of the table within the database.
+** vs. a FileSystem backend, this may be the sub-path where files for the table are stored.
+** #todo - details on these#
+* `automationDetails` - *<<QTableAutomationDetails>>* - Configuration of automated jobs that run against records in the table, e.g., upon insert or update.
 * `customizers` - *Map of String → QCodeReference* - References to custom code that are injected into standard table actions, that allow applications to customize certain parts of how the table works.
+** Allowed values for keys in this map come from the `role` property of the `TableCustomizers` enum.
+** Based on the key in this map, the `QCodeReference` used as the value must be of the appropriate java type, as specified in the `expectedType` property of the `TableCustomizers` enum value corresponding to the key.
+** Example:
+
+[source,java]
+----
+// in defining a QTableMetaData, a customizer can be added as:
+.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(MyPreInsCustomizer.class))
+
+// where MyPreInsCustomizer would be defined as:
+public class MyPreInsCustomizer extends AbstractPreInsertCustomizer
+----
+
+
+* `isHidden` - *Boolean, default false* - Option to hide the table from all User Interfaces.
 * `parentAppName` - *String* - Name of a {link-app} that this table exists within.
-* `icon` - *QIcon* - Icon associated with this table in certain user interfaces.
+** This field generally does not need to be set on the table when it is defined, but rather, is set when the table gets placed within an app.
+* `icon` - *QIcon* - Icon associated with this table in certain user interfaces.  See {link-icons}.
 * `recordLabelFormat` - *String* - Java Format String, used with `recordLabelFields` to produce a label shown for records from the table.
 * `recordLabelFields` - *List of String, Conditional* - Used with `recordLabelFormat` to provide values for any format specifiers in the format string.
 These strings must be field names within the table.
@@ -42,8 +66,181 @@ new QFieldMetaData("birthDate", QFieldType.DATE)
 .withRecordLabelFormat("%s (%s)")
 .withRecordLabelFields(List.of("name", "birthDate"))
 ----
-* `sections` - *List of QFieldSection* - Mechanism to organize fields within user interfaces, into logical sections.
+
+* `sections` - *List of <<QFieldSection>>* - Mechanism to organize fields within user interfaces, into logical sections.
 If any sections are present in the table meta data, then all fields in the table must be listed in exactly 1 section.
 If no sections are defined, then instance enrichment will define default sections.
-* `associatedScripts` - *List of AssociatedScript* - Definition of user-defined scripts that can be associated with records within the table.
+* `associatedScripts` - *List of <<AssociatedScript>>* - Definition of user-defined scripts that can be associated with records within the table.
 * `enabledCapabilities` and `disabledCapabilities` - *Set of Capability enum values* - Overrides from the backend level, for capabilities that this table does or does not possess.
+* `associations` - *List of <<Association>>* - tables whose records can be managed along with records from this table.  See below for details.
+* `recordSecurityLocks` - *List of <<RecordSecurityLock>>* - locks that apply to records in the table - e.g., to control what users can or cannot access records in the table.
+See RecordSecurityLock below for details.
+* `permissionRules` - *QPermissionRules object* - define the permission/access rules for the table.
+See {link-permissionRules} for details.
+* `auditRules` - *<<QAuditRules>> object* - define the audit rules for the table.
+See QAuditRules below for details.
+* `cacheOf` - *<<CacheOf>> object* - specify that this table serves as a "cache of" another table.
+See CacheOf object below for details.
+* `exposedJoins` - *List of <<ExposedJoin>> objects* - optional list of joined tables that are to be exposed in User Interfaces.
+See ExposedJoin object below for details.
+
+#todo: supplementalMetaData (API)#
+
+
+==== QFieldSection
+When users view records from a QQQ Table in a UI, fields are organized on the screen based on the `QFieldSection` objects in the table's meta-data.
+
+*QFieldSection Properties:*
+
+* `name` - *String, Required* - unique identifier for the section within its table.
+* `label` - *String* - User-facing label for the section, presented in User Interfaces.
+Inferred from `name` if not set.
+* `tier` - *enum* - importance of the fields in section for the table.
+Different tiers may be presented differently in UI's.
+Only a single `T1` section is allowed per-table.  Possible values are: `T1`, `T2`, and `T3`.
+* `icon` - *QIcon* - Icon associated with this section in certain user interfaces.  See {link-icons}.
+* `isHidden` - *Boolean, default false* - Option to hide the table from all User Interfaces.
+* `gridColumns` - *Integer* - Option to specify how many columns in a grid layout the section should use.
+For the Material-Dashboard frontend, this is a grid of 12.
+* `fieldNames` - *List of String, Conditional* - List of names of {link-fields} from this table to be included in this section.
+* `widgetName` - *String, Conditional* - Name of a {link-widget} to be displayed in this section.
+** Note that exactly one of `fieldNames` or `widgetName` must be used.
+
+==== QTableAutomationDetails
+Records in QQQ can have application-defined custom actions automatically asynchronously executed against them after they are inserted or updated.
+The configuration to enable this functionality is assigned to a table in a `QTableAutomationDetails` object.
+
+*QTableAutomationDetails Properties:*
+
+* `statusTracking` - *AutomationStatusTracking object, Required* - define how QQQ should keep track, per record, its status (e.g., pending-insert-automations, running-update-automations, etc).
+Properties of `AutomationStatusTracking` object are:
+** `type` - *enum, Required* - what type of tracking is used for the table.
+Possible values are:
+*** `FIELD_IN_TABLE` - specifies that the table has a field which stores an `AutomationStatus` id.
+*** _Additional types may be defined in the future, such as ONE_TO_ONE_TABLE or SHARED_TABLE._
+** `fieldName` - *String, Conditional* - for `type=FIELD_IN_TABLE`, this property specifies the name of the {link-field} in the table that stores the `AutomationStatus` id.
+* `providerName` - *String, Required* - name of an Automation Provider within the QQQ Instance, which is responsible for running the automations on this table.
+* `overrideBatchSize` - *Integer* - optional control over how many records from the table are processed in a single batch/page.
+For tables with "slow" actions (e.g., one that may need to make an API call per-record), using a smaller batch size (say, 50) may be required to avoid timeout errors.
+* `actions` - *List of TableAutomationAction* - list of the actions to perform on new and updated records in the table.
+Properties are:
+** `name` - *String, Required* - unique identifier for the action within its table.
+** `triggerEvent` - *enum, Required* - indicate which event type (`POST_INSERT`, `POST_UPDATE`, or `PRE_DELETE` (which is not yet implemented)) the action applies to.
+** `priority` - *Integer, default 500* - mechanism to control the order in which actions on a table are executed, if there are more than one.
+Actions with a smaller value for `priority` are executed first.  Ties are broken in an undefined manner.
+** `filter` - *QQueryFilter* - optional filter that gets applied to records when they match the `triggerEvent`, to control which records have the action ran against them.
+** `includeRecordAssociations` - *Boolean, default false* - for tables that have associations, control whether or not a record's associated records are loaded when records are fetched and passed into the action's custom code.
+** `values` - *Map of String → Serializable* - optional application-defined map of name=value pairs that can be passed into the action's custom code.
+** `processName` - *String, Conditional* - name of a {link-processes} in the QQQ Instance which is executed as the custom-code of the action.
+** `codeReference` - *QCodeReference, Conditional* - reference to a class that extends `RecordAutomationHandler`, to be executed as the custom-code of the action.
+*** Note, exactly one of `processName` or `codeReference` must be provided.
+
+
+
+==== Association
+An `Association` is a way to define a relationship between tables, that facilitates, for example, a parent record having a list of its child records included in it when it is returned from a Query.
+Similarly, associated records can automatically be inserted/updated/deleted if they are included in a parent record when it is stored.
+
+*Association Properties:*
+
+* `name` - *String, Required* - unique name for the association within this table.
+Used as the key in the `associatedRecords` map within `QRecord` objects for this table.
+* `associatedTableName` - *String, Required* - name of a {link-table}, which is the associated table.
+* `joinName` - *String, Required* - name of a {link-join} in the instance, which defines how the tables are joined.
+
+
+
+==== RecordSecurityLock
+A `RecordSecurityLock` is the mechanism through which users can be allowed or denied access to read and/or write records, based on values in the record, and values in the user's session.
+Record security locks must correspond to a {link-securityKeyType}.
+
+For example:
+
+* An instance may have a security key type called `clientId`.
+* Users may have 1 or more `clientId` values in their Session, or, they may have an "All Clients" key in their session (e.g., for internal/admin users).
+* For some tables, it may be required to limit visibility to records based on a user's `clientId` key.
+To do this. a *RecordSecurityLock* would be applied to the table, specifying the `clientId` field corresponds to the `clientId` security key.
+* With these settings in place, QQQ will prevent users from viewing records from this table that do not have a matching key, and will similarly prevent users from writing records with an invalid key value.
+** For example, in an RDBMS backend, all `SELECT` statements generated against such a table will have an implicit filter, such as `AND client_id = ?` based on the user's security key values.
+
+*RecordSecurityLock Properties:*
+
+* `securityKeyType` - *String, Required* - name of a {link-securityKeyType} in the Instance.
+* `fieldName` - *String, Required* - name of a {link-field} in this table (or a joined table, if `joinNameChain` is set), where the value for the lock is stored.
+* `joinNameChain` - *List of String* - if the lock value is not stored in this table, but rather comes from a joined table, then this property defines the path of joins from this table to the table with the lock field.
+* `nullValueBehavior` - *enum, default: DENY* - control how records with a `null` value in the lock field should behave.
+Possible values are:
+** `DENY` - deny all users access to a record with a `null` value in the lock field (unless the user has an all-access key - see {link-securityKeyType})
+** `ALLOW` - allow all users access to a record with a `null` value in the lock field.
+** `ALLOW_WRITE_ONLY` - allow all users to write records with `null` in the lock field, but deny reads on records with `null` in the lock field (also excepted by all-access keys).
+* `lockScope` - *enum, default: READ_AND_WRITE* - control what types of operations the lock applies to.
+Possible values are:
+** `READ_AND_WRITE` - control both reading and writing records based on the user having an appropriate security key.
+** `WRITE` - allow all users to read the record, but limit writes to users with an appropriate security key.
+
+
+
+==== QAuditRules
+The audit rules on a table define the level of detail that is automatically stored in the audit table (if any) for DML actions (Insert, Update, Delete).
+
+*QAuditRules Properties:*
+
+* `auditLevel` - *enum, Required* - level of details that are audited.
+Possible values are:
+** `NONE` - no automatic audits are stored for the table.
+** `RECORD` - only record-level audits are stored for the table (e.g., a message such as "record was edited", but without field-level details)
+** `FIELD` - full field-level audits are stored (e.g., including all old & new values as audit details).
+
+
+
+==== CacheOf
+One QQQ Table can be defined as a "cache of" another QQQ Table by assigning a `CacheOf` object to the table which will function as the cache.
+_Note, at this time, only limited use-cases are supported._
+
+*CacheOf Properties:*
+
+* `sourceTable` - *String, Required* - name of the other QQQ Table that is the source of data in this cache.
+* `expirationSeconds` - *Integer* - optional number of seconds that a cached record is allowed to exist before it is considered expired, and must be re-fetched from the source table.
+* `cachedDateFieldName` - *String, Conditional* - used with `expirationSeconds` to define the field in this table that is used for storing the timestamp for when the record was cached.
+* `useCases` - *List of CacheUseCase* - what caching use-cases are to be implemented.
+
+Properties of *CacheUseCase* are:
+
+* `type` - *Enum, Required* - the type of use-case.  Possible values are:
+** `PRIMARY_KEY_TO_PRIMARY_KEY` - the primary key in the cache table equals the primary key in the source table.
+** `UNIQUE_KEY_TO_PRIMARY_KEY` - a unique key in the cache table equals the primary key in the source table.
+** `UNIQUE_KEY_TO_UNIQUE_KEY` - a unique key in the cache table equals a unique key in the source table.
+* `cacheSourceMisses` - *Boolean, default false* - whether or not, if a "miss" happens in the source, if that fact gets cached.
+* `cacheUniqueKey` - *UniqueKey, conditional* - define the fields in the cache table that define the unique key being used as the cache key.
+* `sourceUniqueKey` - *UniqueKey, conditional* - define the fields in the source table that define the unique key being used as the cache key.
+* `doCopySourcePrimaryKeyToCache` - *Boolean, default false* - specify whether or not the value of the primary key in the source table should be copied into records built in the cache table.
+* `excludeRecordsMatching` - *List of QQueryFilter* - optional filter to be applied to records before they are cached.
+If a record matches the filter, then it will not be cached.
+
+
+==== ExposedJoin
+Query screens in QQQ applications can potentially allow users to both display fields from joined tables, and filter by fields from joined tables, for any {link-join} explicitly defined as an *Exposed Join*.
+
+_The reasoning why not all joins are implicitly exposed is that in many applications, the full join-graph can sometimes be overwhelming, surprisingly broad, and not necessarily practically useful.
+This could be subject to change in the future, e.g., given a UI that allowed users to more explicitly add additional join tables..._
+
+*ExposedJoin Properties:*
+
+* `label` - *String, Required* - how the joined table should be presented in the UI.
+* `joinTable` - *String, Required* - name of the QQQ Table that is joined to this table, and is being exposed as a join in the UI.
+* `joinPath` - *List of String, Required* - names of 1 or more QQQ Joins that describe how to get from this table to the join table.
+
+
+
+==== AssociatedScript
+A QQQ Table can have end-user defined Script records associated with individual records in the table by use of the `associatedScripts` property of the table's meta-data.
+
+The "types" of these scripts (e.g., how they are used in an application) are wholly application-designed & managed.
+QQQ provides the mechanism for UI's to present and manage such scripts (e.g., the *Developer Mode* screen in the Material Dashboard), as well as an interface to load & execute such scripts `RunAssociatedScriptAction`).
+
+*AssociatedScript Properties:*
+
+* `fieldName` - *String, Required* - name of a {link-field} in the table which stores the id of the associated script record.
+* `scriptTypeId` - *Serializable (typically Integer), Required* - primary key value from the `"scriptType"` table in the instance, to designate the type of the Script.
+* `scriptTester` - *QCodeReference* - reference to a class which implements `TestScriptActionInterface`, that can be used by UI's for running an associated script to test it.
+

docs/metaData/Widgets.adoc

@@ -0,0 +1,17 @@
+[#Widgets]
+== Widgets
+include::../variables.adoc[]
+
+#TODO#
+
+=== QWidgetMetaData
+A Widget is defined in a QQQ Instance in a `*QWidgetMetaData*` object.
+
+#TODO#
+
+*QWidgetMetaData Properties:*
+
+* `name` - *String, Required* - Unique name for the widget within the QQQ Instance.
+
+#TODO#
+

docs/variables.adoc

@@ -1,13 +1,25 @@
-ifdef::env-name[:relfilesuffix: .adoc]
-:link-backend: link:Backends{relfilesuffix}[QQQ Backend]
-:link-backends: link:Backends{relfilesuffix}[QQQ Backends]
-:link-table: link:Tables{relfilesuffix}[QQQ Table]
-:link-tables: link:Tables{relfilesuffix}[QQQ Tables]
-:link-join: link:Joins{relfilesuffix}[QQQ Join]
-:link-joins: link:Joins{relfilesuffix}[QQQ Joins]
-:link-field: link:Fields{relfilesuffix}[QQQ Field]
-:link-fields: link:Fields{relfilesuffix}[QQQ Fields]
-:link-process: link:Processes{relfilesuffix}[QQQ Process]
-:link-processes: link:Processes{relfilesuffix}[QQQ Processes]
-:link-app: link:Apps{relfilesuffix}[QQQ App]
-:link-apps: link:Apps{relfilesuffix}[QQQ Apps]
+:link-app: <<Apps,QQQ App>>
+:link-apps: <<Apps,QQQ Apps>>
+:link-backend: <<Backends,QQQ Backend>>
+:link-backends: <<Backends,QQQ Backends>>
+:link-field: <<Fields,QQQ Field>>
+:link-fields: <<Fields,QQQ Fields>>
+:link-icon: <<Icons,Icon>>
+:link-icons: <<Icons,Icons>>
+:link-instance: <<QInstance,QInstance>>
+:link-join: <<Joins,QQQ Join>>
+:link-joins: <<Joins,QQQ Joins>>
+:link-permissionRule: <<PermissionRules,Permission Rule>>
+:link-permissionRules: <<PermissionRules,Permission Rules>>
+:link-possibleValueSource: <<PossibleValueSources,QQQ Possible Value Source>>
+:link-possibleValueSources: <<PossibleValueSources,QQQ Possible Value Sources>>
+:link-process: <<Processes,QQQ Process>>
+:link-processes: <<Processes,QQQ Processes>>
+:link-report: <<Reports,QQQ Report>>
+:link-reports: <<Reports,QQQ Reports>>
+:link-securityKeyType: <<SecurityKeyTypes,Security Key Type>>
+:link-securityKeyTypes: <<SecurityKeyTypes,Security Key Types>>
+:link-table: <<Tables,QQQ Table>>
+:link-tables: <<Tables,QQQ Tables>>
+:link-widget: <<Widgets,QQQ Widget>>
+:link-widgets: <<Widgets,QQQ Widgets>>

pom.xml

@@ -33,6 +33,7 @@
       <module>qqq-backend-module-api</module>
       <module>qqq-backend-module-filesystem</module>
       <module>qqq-backend-module-rdbms</module>
+      <module>qqq-backend-module-mongodb</module>
       <module>qqq-language-support-javascript</module>
       <module>qqq-middleware-picocli</module>
       <module>qqq-middleware-javalin</module>
@@ -108,6 +109,16 @@
    </dependencyManagement>
 
    <build>
+      <resources>
+         <resource>
+            <directory>src/main/java</directory>
+            <filtering>false</filtering>
+         </resource>
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>false</filtering>
+         </resource>
+      </resources>
       <plugins>
          <!-- plugins specifically for this module -->
          <!-- none at this time -->
@@ -317,6 +328,32 @@ fi
       </plugins>
    </build>
 
+   <!-- mvn javadoc:aggregate -->
+   <reporting>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-javadoc-plugin</artifactId>
+        <version>3.6.2</version>
+        <reportSets>
+          <reportSet>
+            <id>aggregate</id>
+            <inherited>false</inherited>
+            <reports>
+              <report>aggregate</report>
+            </reports>
+          </reportSet>
+          <reportSet>
+            <id>default</id>
+            <reports>
+              <report>javadoc</report>
+            </reports>
+          </reportSet>
+        </reportSets>
+      </plugin>
+    </plugins>
+   </reporting>
+
    <repositories>
       <repository>
          <id>github-qqq-maven-registry</id>

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/QBackendTransaction.java

@@ -23,6 +23,9 @@ package com.kingsrook.qqq.backend.core.actions;
 
 
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
+import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
+import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
 
 
 /*******************************************************************************
@@ -30,12 +33,26 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
  ** part of a transaction.
  **
  ** Most obvious use-case would be a JDBC Connection.  See subclass in rdbms module.
+ ** Ditto MongoDB.
  **
  ** Note:  One would imagine that this class shouldn't ever implement Serializable...
  *******************************************************************************/
 public class QBackendTransaction
 {
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QBackendTransaction openFor(AbstractTableActionInput input) throws QException
+   {
+      QBackendModuleDispatcher qBackendModuleDispatcher = new QBackendModuleDispatcher();
+      QBackendModuleInterface  qModule                  = qBackendModuleDispatcher.getQBackendModule(input.getBackend());
+      QBackendTransaction      transaction              = qModule.openTransaction(input);
+      return (transaction);
+   }
+
+
+
    /*******************************************************************************
     ** Commit the transaction.
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/audits/DMLAuditAction.java

@@ -30,10 +30,12 @@ import java.time.temporal.ChronoUnit;
 import java.util.ArrayList;
 import java.util.Comparator;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
+import java.util.Set;
 import com.kingsrook.qqq.backend.core.actions.AbstractQActionFunction;
 import com.kingsrook.qqq.backend.core.actions.values.QPossibleValueTranslator;
 import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
@@ -73,6 +75,7 @@ public class DMLAuditAction extends AbstractQActionFunction<DMLAuditInput, DMLAu
 
    public static final String AUDIT_CONTEXT_FIELD_NAME = "auditContext";
 
+   private static Set<String> loggedUnauditableTableNames = new HashSet<>();
 
 
    /*******************************************************************************
@@ -102,6 +105,21 @@ public class DMLAuditAction extends AbstractQActionFunction<DMLAuditInput, DMLAu
             return (output);
          }
 
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////
+         // currently, the table's primary key must be integer... so, log (once) and return early if not that //
+         // (or, if no primary key!)                                                                          //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////
+         QFieldMetaData field = table.getFields().get(table.getPrimaryKeyField());
+         if(field == null || !QFieldType.INTEGER.equals(field.getType()))
+         {
+            if(!loggedUnauditableTableNames.contains(table.getName()))
+            {
+               LOG.info("Cannot audit table without integer as its primary key", logPair("tableName", table.getName()));
+               loggedUnauditableTableNames.add(table.getName());
+            }
+            return (output);
+         }
+
          String contextSuffix = getContentSuffix(input);
 
          AuditInput auditInput = new AuditInput();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/automation/polling/PollingAutomationPerTableRunner.java

@@ -43,6 +43,7 @@ import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.LogPair;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
@@ -87,8 +88,9 @@ public class PollingAutomationPerTableRunner implements Runnable
 {
    private static final QLogger LOG = QLogger.getLogger(PollingAutomationPerTableRunner.class);
 
-   private final TableActions tableActions;
-   private final String       name;
+   private final TableActionsInterface tableActions;
+
+   private String name;
 
    private QInstance          instance;
    private Supplier<QSession> sessionSupplier;
@@ -115,11 +117,52 @@ public class PollingAutomationPerTableRunner implements Runnable
 
 
 
+   /*******************************************************************************
+    ** Interface to be used by 2 records in this class - normal TableActions, and
+    ** ShardedTableActions.
+    *******************************************************************************/
+   public interface TableActionsInterface
+   {
       /*******************************************************************************
        **
        *******************************************************************************/
-   public record TableActions(String tableName, AutomationStatus status)
+      String tableName();
+
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      AutomationStatus status();
+   }
+
+
+
+   /*******************************************************************************
+    ** Wrapper for a pair of (tableName, automationStatus)
+    *******************************************************************************/
+   public record TableActions(String tableName, AutomationStatus status) implements TableActionsInterface
    {
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      public void noopToFakeTestCoverage()
+      {
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** extended version of TableAction, for sharding use-case - adds the shard
+    ** details.
+    *******************************************************************************/
+   public record ShardedTableActions(String tableName, AutomationStatus status, String shardByFieldName, Serializable shardValue, String shardLabel) implements TableActionsInterface
+   {
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      public void noopToFakeTestCoverage()
+      {
+      }
    }
 
 
@@ -128,18 +171,48 @@ public class PollingAutomationPerTableRunner implements Runnable
     ** basically just get a list of tables which at least *could* have automations
     ** run - either meta-data automations, or table-triggers (data/user defined).
     *******************************************************************************/
-   public static List<TableActions> getTableActions(QInstance instance, String providerName)
+   public static List<TableActionsInterface> getTableActions(QInstance instance, String providerName)
    {
-      List<TableActions> tableActionList = new ArrayList<>();
+      List<TableActionsInterface> tableActionList = new ArrayList<>();
 
       for(QTableMetaData table : instance.getTables().values())
       {
-         if(table.getAutomationDetails() != null && providerName.equals(table.getAutomationDetails().getProviderName()))
+         QTableAutomationDetails automationDetails = table.getAutomationDetails();
+         if(automationDetails != null && providerName.equals(automationDetails.getProviderName()))
          {
+            if(StringUtils.hasContent(automationDetails.getShardByFieldName()))
+            {
+               //////////////////////////////////////////////////////////////////////////////////////////////
+               // for sharded automations, add a tableAction (of the sharded subtype) for each shard-value //
+               //////////////////////////////////////////////////////////////////////////////////////////////
+               try
+               {
+                  QueryInput queryInput = new QueryInput();
+                  queryInput.setTableName(automationDetails.getShardSourceTableName());
+                  QueryOutput queryOutput = new QueryAction().execute(queryInput);
+                  for(QRecord record : queryOutput.getRecords())
+                  {
+                     Serializable shardId = record.getValue(automationDetails.getShardIdFieldName());
+                     String       label   = record.getValueString(automationDetails.getShardLabelFieldName());
+                     tableActionList.add(new ShardedTableActions(table.getName(), AutomationStatus.PENDING_INSERT_AUTOMATIONS, automationDetails.getShardByFieldName(), shardId, label));
+                     tableActionList.add(new ShardedTableActions(table.getName(), AutomationStatus.PENDING_UPDATE_AUTOMATIONS, automationDetails.getShardByFieldName(), shardId, label));
+                  }
+               }
+               catch(Exception e)
+               {
+                  LOG.error("Error getting sharded table automation actions for a table", e, new LogPair("tableName", table.getName()));
+               }
+            }
+            else
+            {
+               ///////////////////////////////////////////////////////////////////
+               // for non-sharded, we just need tabler name & automation status //
+               ///////////////////////////////////////////////////////////////////
                tableActionList.add(new TableActions(table.getName(), AutomationStatus.PENDING_INSERT_AUTOMATIONS));
                tableActionList.add(new TableActions(table.getName(), AutomationStatus.PENDING_UPDATE_AUTOMATIONS));
             }
          }
+      }
 
       return (tableActionList);
    }
@@ -149,12 +222,17 @@ public class PollingAutomationPerTableRunner implements Runnable
    /*******************************************************************************
     **
     *******************************************************************************/
-   public PollingAutomationPerTableRunner(QInstance instance, String providerName, Supplier<QSession> sessionSupplier, TableActions tableActions)
+   public PollingAutomationPerTableRunner(QInstance instance, String providerName, Supplier<QSession> sessionSupplier, TableActionsInterface tableActions)
    {
       this.instance = instance;
       this.sessionSupplier = sessionSupplier;
       this.tableActions = tableActions;
       this.name = providerName + ">" + tableActions.tableName() + ">" + tableActions.status().getInsertOrUpdate();
+
+      if(tableActions instanceof ShardedTableActions shardedTableActions)
+      {
+         this.name += ":" + shardedTableActions.shardLabel();
+      }
    }
 
 
@@ -229,6 +307,15 @@ public class PollingAutomationPerTableRunner implements Runnable
                throw (new NotImplementedException("Automation Status Tracking type [" + statusTrackingType + "] is not yet implemented in here."));
             }
 
+            if(tableActions instanceof ShardedTableActions shardedTableActions)
+            {
+               //////////////////////////////////////////////////////////////
+               // for sharded actions, add the shardBy field as a criteria //
+               //////////////////////////////////////////////////////////////
+               QQueryFilter filter = queryInput.getFilter();
+               filter.addCriteria(new QFilterCriteria(shardedTableActions.shardByFieldName(), QCriteriaOperator.EQUALS, shardedTableActions.shardValue()));
+            }
+
             queryInput.setRecordPipe(recordPipe);
             return (new QueryAction().execute(queryInput));
          }, () ->
@@ -257,10 +344,26 @@ public class PollingAutomationPerTableRunner implements Runnable
       for(TableAutomationAction action : table.getAutomationDetails().getActions())
       {
          if(action.getTriggerEvent().equals(triggerEvent))
+         {
+            ///////////////////////////////////////////////////////////
+            // for sharded configs, only run if the shard id matches //
+            ///////////////////////////////////////////////////////////
+            if(tableActions instanceof ShardedTableActions shardedTableActions)
+            {
+               if(shardedTableActions.shardValue().equals(action.getShardId()))
                {
                   rs.add(action);
                }
             }
+            else
+            {
+               ////////////////////////////////////////////
+               // for non-sharded, always add the action //
+               ////////////////////////////////////////////
+               rs.add(action);
+            }
+         }
+      }
 
       /////////////////////////////////////////////////
       // next add any tableTriggers, defined in data //
@@ -471,7 +574,7 @@ public class PollingAutomationPerTableRunner implements Runnable
             @Override
             public QQueryFilter getQueryFilter()
             {
-               List<Serializable> recordIds = records.stream().map(r -> r.getValueInteger(table.getPrimaryKeyField())).collect(Collectors.toList());
+               List<Serializable> recordIds = records.stream().map(r -> r.getValue(table.getPrimaryKeyField())).collect(Collectors.toList());
                return (new QQueryFilter().withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.IN, recordIds)));
             }
          });

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/interfaces/InsertInterface.java

@@ -31,7 +31,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
  ** Interface for the Insert action.
  **
  *******************************************************************************/
-public interface InsertInterface extends QActionInterface
+public interface InsertInterface
 {
    /*******************************************************************************
     **

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/metadata/MetaDataAction.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.actions.metadata;
 
 
 import java.util.ArrayList;
+import java.util.Comparator;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
@@ -150,14 +151,21 @@ public class MetaDataAction
       }
       metaDataOutput.setWidgets(widgets);
 
+      ///////////////////////////////////////////////////////
+      // sort apps - by sortOrder (integer), then by label //
+      ///////////////////////////////////////////////////////
+      List<QAppMetaData> sortedApps = metaDataInput.getInstance().getApps().values().stream()
+         .sorted(Comparator.comparing((QAppMetaData a) -> a.getSortOrder())
+            .thenComparing((QAppMetaData a) -> a.getLabel()))
+         .toList();
+
       ///////////////////////////////////
       // map apps to frontend metadata //
       ///////////////////////////////////
       Map<String, QFrontendAppMetaData> apps = new LinkedHashMap<>();
-      for(Map.Entry<String, QAppMetaData> entry : metaDataInput.getInstance().getApps().entrySet())
+      for(QAppMetaData app : sortedApps)
       {
-         String       appName = entry.getKey();
-         QAppMetaData app     = entry.getValue();
+         String appName = app.getName();
 
          PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, app);
          if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
@@ -191,7 +199,7 @@ public class MetaDataAction
       // organize app tree nodes by their hierarchy //
       ////////////////////////////////////////////////
       List<AppTreeNode> appTree = new ArrayList<>();
-      for(QAppMetaData appMetaData : metaDataInput.getInstance().getApps().values())
+      for(QAppMetaData appMetaData : sortedApps)
       {
          if(appMetaData.getParentAppName() == null)
          {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/processes/QProcessCallbackFactory.java

@@ -0,0 +1,68 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.processes;
+
+
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+
+
+/*******************************************************************************
+ ** Constructor for commonly used QProcessCallback's
+ *******************************************************************************/
+public class QProcessCallbackFactory
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QProcessCallback forFilter(QQueryFilter filter)
+   {
+      return new QProcessCallback()
+      {
+         /*******************************************************************************
+          **
+          *******************************************************************************/
+         @Override
+         public QQueryFilter getQueryFilter()
+         {
+            return (filter);
+         }
+
+
+
+         /*******************************************************************************
+          **
+          *******************************************************************************/
+         @Override
+         public Map<String, Serializable> getFieldValues(List<QFieldMetaData> fields)
+         {
+            return (Collections.emptyMap());
+         }
+      };
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/InsertAction.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.actions.tables;
 
 
 import java.io.Serializable;
+import java.time.Instant;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -52,6 +53,7 @@ import com.kingsrook.qqq.backend.core.model.actions.audits.DMLAuditInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
@@ -116,21 +118,15 @@ public class InsertAction extends AbstractQActionFunction<InsertInput, InsertOut
 
       setAutomationStatusField(insertInput);
 
-      //////////////////////////////////////////////////////
-      // load the backend module and its insert interface //
-      //////////////////////////////////////////////////////
-      QBackendModuleInterface qModule         = getBackendModuleInterface(insertInput);
-      InsertInterface         insertInterface = qModule.getInsertInterface();
-
       /////////////////////////////
       // run standard validators //
       /////////////////////////////
       performValidations(insertInput, false);
 
-      ////////////////////////////////////
-      // have the backend do the insert //
-      ////////////////////////////////////
-      InsertOutput insertOutput = insertInterface.execute(insertInput);
+      //////////////////////////////////////////////////////
+      // use the backend module to actually do the insert //
+      //////////////////////////////////////////////////////
+      InsertOutput insertOutput = runInsertInBackend(insertInput);
 
       if(insertOutput.getRecords() == null)
       {
@@ -195,6 +191,71 @@ public class InsertAction extends AbstractQActionFunction<InsertInput, InsertOut
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private InsertOutput runInsertInBackend(InsertInput insertInput) throws QException
+   {
+      ///////////////////////////////////
+      // exit early if 0 input records //
+      ///////////////////////////////////
+      if(CollectionUtils.nullSafeIsEmpty(insertInput.getRecords()))
+      {
+         LOG.debug("Insert request called with 0 records.  Returning with no-op", logPair("tableName", insertInput.getTableName()));
+         InsertOutput rs = new InsertOutput();
+         rs.setRecords(new ArrayList<>());
+         return (rs);
+      }
+
+      /////////////////////////////////////////////
+      // set values in create date & modify date //
+      // todo .. better (not hard-coded names)   //
+      /////////////////////////////////////////////
+      Instant now = Instant.now();
+      for(QRecord record : insertInput.getRecords())
+      {
+         setValueIfTableHasField(record, insertInput.getTable(), "createDate", now);
+         setValueIfTableHasField(record, insertInput.getTable(), "modifyDate", now);
+      }
+
+      //////////////////////////////////////////////////////
+      // load the backend module and its insert interface //
+      //////////////////////////////////////////////////////
+      QBackendModuleInterface qModule         = getBackendModuleInterface(insertInput.getBackend());
+      InsertInterface         insertInterface = qModule.getInsertInterface();
+
+      ////////////////////////////////////
+      // have the backend do the insert //
+      ////////////////////////////////////
+      InsertOutput insertOutput = insertInterface.execute(insertInput);
+      return insertOutput;
+   }
+
+
+
+   /*******************************************************************************
+    ** If the table has a field with the given name, then set the given value in the
+    ** given record.
+    *******************************************************************************/
+   private static void setValueIfTableHasField(QRecord record, QTableMetaData table, String fieldName, Serializable value)
+   {
+      try
+      {
+         if(table.getFields().containsKey(fieldName))
+         {
+            record.setValue(fieldName, value);
+         }
+      }
+      catch(Exception e)
+      {
+         /////////////////////////////////////////////////
+         // this means field doesn't exist, so, ignore. //
+         /////////////////////////////////////////////////
+      }
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -426,23 +487,11 @@ public class InsertAction extends AbstractQActionFunction<InsertInput, InsertOut
    /*******************************************************************************
     **
     *******************************************************************************/
-   private QBackendModuleInterface getBackendModuleInterface(InsertInput insertInput) throws QException
+   private QBackendModuleInterface getBackendModuleInterface(QBackendMetaData backend) throws QException
    {
-      ActionHelper.validateSession(insertInput);
       QBackendModuleDispatcher qBackendModuleDispatcher = new QBackendModuleDispatcher();
-      QBackendModuleInterface  qModule                  = qBackendModuleDispatcher.getQBackendModule(insertInput.getBackend());
+      QBackendModuleInterface qModule = qBackendModuleDispatcher.getQBackendModule(backend);
       return (qModule);
    }
 
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   public QBackendTransaction openTransaction(InsertInput insertInput) throws QException
-   {
-      QBackendModuleInterface qModule = getBackendModuleInterface(insertInput);
-      return (qModule.getInsertInterface().openTransaction(insertInput));
-   }
-
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/ReplaceAction.java

@@ -84,9 +84,7 @@ public class ReplaceAction extends AbstractQActionFunction<ReplaceInput, Replace
          String         primaryKeyField = table.getPrimaryKeyField();
          if(transaction == null)
          {
-            InsertInput insertInput = new InsertInput();
-            insertInput.setTableName(input.getTableName());
-            transaction = new InsertAction().openTransaction(insertInput);
+            transaction = QBackendTransaction.openFor(new InsertInput(input.getTableName()));
             weOwnTheTransaction = true;
          }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/UpdateAction.java

@@ -135,7 +135,7 @@ public class UpdateAction
       ////////////////////////////////////
       // have the backend do the update //
       ////////////////////////////////////
-      UpdateOutput updateOutput = updateInterface.execute(updateInput);
+      UpdateOutput updateOutput = runUpdateInBackend(updateInput, updateInterface);
 
       if(updateOutput.getRecords() == null)
       {
@@ -203,6 +203,28 @@ public class UpdateAction
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private UpdateOutput runUpdateInBackend(UpdateInput updateInput, UpdateInterface updateInterface) throws QException
+   {
+      ///////////////////////////////////
+      // exit early if 0 input records //
+      ///////////////////////////////////
+      if(CollectionUtils.nullSafeIsEmpty(updateInput.getRecords()))
+      {
+         LOG.debug("Update request called with 0 records.  Returning with no-op", logPair("tableName", updateInput.getTableName()));
+         UpdateOutput rs = new UpdateOutput();
+         rs.setRecords(new ArrayList<>());
+         return (rs);
+      }
+
+      UpdateOutput updateOutput = updateInterface.execute(updateInput);
+      return updateOutput;
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/UpdateActionRecordSplitHelper.java

@@ -0,0 +1,203 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.tables.helpers;
+
+
+import java.io.Serializable;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ListingHash;
+
+
+/*******************************************************************************
+ ** Helper for backends that want to do their updates on records grouped by the
+ ** set of fields that are being changed, and/or by the values those fields are
+ ** being set to.
+ **
+ ** e.g., RDBMS, for n records where some sub-set of fields are all having values
+ ** set the same (say, a status=x), we can do that as 1 query where id in (?,?,...,?).
+ *******************************************************************************/
+public class UpdateActionRecordSplitHelper
+{
+   private ListingHash<List<String>, QRecord> recordsByFieldBeingUpdated = new ListingHash<>();
+   private boolean                            haveAnyWithoutErrors       = false;
+   private List<QRecord>                      outputRecords              = new ArrayList<>();
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void init(UpdateInput updateInput)
+   {
+      QTableMetaData table = updateInput.getTable();
+      Instant        now   = Instant.now();
+
+      for(QRecord record : updateInput.getRecords())
+      {
+         ////////////////////////////////////////////
+         // todo .. better (not a hard-coded name) //
+         ////////////////////////////////////////////
+         setValueIfTableHasField(record, table, "modifyDate", now);
+
+         List<String> updatableFields = table.getFields().values().stream()
+            .map(QFieldMetaData::getName)
+            // todo - intent here is to avoid non-updateable fields - but this
+            //  should be like based on field.isUpdatable once that attribute exists
+            .filter(name -> !name.equals("id"))
+            .filter(name -> record.getValues().containsKey(name))
+            .toList();
+         recordsByFieldBeingUpdated.add(updatableFields, record);
+
+         if(CollectionUtils.nullSafeIsEmpty(record.getErrors()))
+         {
+            haveAnyWithoutErrors = true;
+         }
+
+         //////////////////////////////////////////////////////////////////////////////
+         // go ahead and put the record into the output list at this point in time,  //
+         // so that the output list's order matches the input list order             //
+         // note that if we want to capture updated values (like modify dates), then //
+         // we may want a map of primary key to output record, for easy updating.    //
+         //////////////////////////////////////////////////////////////////////////////
+         QRecord outputRecord = new QRecord(record);
+         outputRecords.add(outputRecord);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static boolean areAllValuesBeingUpdatedTheSame(UpdateInput updateInput, List<QRecord> recordList, List<String> fieldsBeingUpdated)
+   {
+      if(updateInput.getAreAllValuesBeingUpdatedTheSame() != null)
+      {
+         ////////////////////////////////////////////////////////////
+         // if input told us what value to use here, then trust it //
+         ////////////////////////////////////////////////////////////
+         return (updateInput.getAreAllValuesBeingUpdatedTheSame());
+      }
+      else
+      {
+         if(recordList.size() == 1)
+         {
+            //////////////////////////////////////////////////////
+            // if a single record, then yes, that always counts //
+            //////////////////////////////////////////////////////
+            return (true);
+         }
+
+         ///////////////////////////////////////////////////////////////////////
+         // else iterate over the records, comparing them to the first record //
+         // return a false if any diffs are found. if no diffs, return true.  //
+         ///////////////////////////////////////////////////////////////////////
+         QRecord firstRecord = recordList.get(0);
+         for(int i = 1; i < recordList.size(); i++)
+         {
+            QRecord record = recordList.get(i);
+
+            if(CollectionUtils.nullSafeHasContents(record.getErrors()))
+            {
+               ///////////////////////////////////////////////////////
+               // skip records w/ errors (that we won't be updating //
+               ///////////////////////////////////////////////////////
+               continue;
+            }
+
+            for(String fieldName : fieldsBeingUpdated)
+            {
+               if(!Objects.equals(firstRecord.getValue(fieldName), record.getValue(fieldName)))
+               {
+                  return (false);
+               }
+            }
+         }
+
+         return (true);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** If the table has a field with the given name, then set the given value in the
+    ** given record.
+    *******************************************************************************/
+   protected void setValueIfTableHasField(QRecord record, QTableMetaData table, String fieldName, Serializable value)
+   {
+      try
+      {
+         if(table.getFields().containsKey(fieldName))
+         {
+            record.setValue(fieldName, value);
+         }
+      }
+      catch(Exception e)
+      {
+         /////////////////////////////////////////////////
+         // this means field doesn't exist, so, ignore. //
+         /////////////////////////////////////////////////
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for haveAnyWithoutErrors
+    **
+    *******************************************************************************/
+   public boolean getHaveAnyWithoutErrors()
+   {
+      return haveAnyWithoutErrors;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for recordsByFieldBeingUpdated
+    **
+    *******************************************************************************/
+   public ListingHash<List<String>, QRecord> getRecordsByFieldBeingUpdated()
+   {
+      return recordsByFieldBeingUpdated;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for outputRecords
+    **
+    *******************************************************************************/
+   public List<QRecord> getOutputRecords()
+   {
+      return outputRecords;
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/QPossibleValueTranslator.java

@@ -34,7 +34,6 @@ import java.util.Objects;
 import java.util.Set;
 import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
-import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QValueException;
@@ -104,7 +103,7 @@ public class QPossibleValueTranslator
          {
             if(!transactionsPerTable.containsKey(tableName))
             {
-               transactionsPerTable.put(tableName, new InsertAction().openTransaction(new InsertInput(tableName)));
+               transactionsPerTable.put(tableName, QBackendTransaction.openFor(new InsertInput(tableName)));
             }
 
             return (transactionsPerTable.get(tableName));

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/QValueFormatter.java

@@ -334,24 +334,34 @@ public class QValueFormatter
                         if(exposedJoin.getJoinTable().equals(nameParts[0]))
                         {
                            QTableMetaData joinTable = QContext.getQInstance().getTable(nameParts[0]);
+                           if(joinTable.getFields().containsKey(nameParts[1]))
+                           {
                               fieldMap.put(fieldName, joinTable.getField(nameParts[1]));
                            }
                         }
                      }
+                  }
                   else
+                  {
+                     if(table.getFields().containsKey(fieldName))
                      {
                         fieldMap.put(fieldName, table.getField(fieldName));
                      }
                   }
+               }
                catch(Exception e)
                {
-                  ///////////////////////////////////////////////////////////
-                  // put an empty field in - so no formatting will be done //
-                  ///////////////////////////////////////////////////////////
-                  LOG.info("Error getting field for setting display value", e, logPair("fieldName", fieldName), logPair("tableName", table.getName()));
-                  fieldMap.put(fieldName, new QFieldMetaData());
+                  LOG.warn("Error getting field for setting display value", e, logPair("fieldName", fieldName), logPair("tableName", table.getName()));
                }
             }
+
+            //////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // if we didn't find the field definition, put an empty field in the map, so no formatting will be done //
+            //////////////////////////////////////////////////////////////////////////////////////////////////////////
+            if(!fieldMap.containsKey(fieldName))
+            {
+               fieldMap.put(fieldName, new QFieldMetaData());
+            }
          }
 
          setDisplayValuesInRecord(fieldMap, record);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/adapters/CsvToQRecordAdapter.java

@@ -35,6 +35,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.shared.mapping.AbstractQFieldMapping;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
@@ -133,11 +134,17 @@ public class CsvToQRecordAdapter
                CSVFormat.DEFAULT
                   .withFirstRecordAsHeader()
                   .withIgnoreHeaderCase()
+                  .withIgnoreEmptyLines()
                   .withTrim());
 
             List<String> headers = csvParser.getHeaderNames();
             headers = makeHeadersUnique(headers);
 
+            ////////////////////////////////////////
+            // used by csv-headers-as-field-names //
+            ////////////////////////////////////////
+            Map<String, QFieldMetaData> csvHeaderFieldMapping = buildCsvHeaderFieldMappingIfNeeded(inputWrapper, headers);
+
             Iterator<CSVRecord> csvIterator = csvParser.iterator();
             int                 recordCount = 0;
             while(csvIterator.hasNext())
@@ -160,12 +167,28 @@ public class CsvToQRecordAdapter
                QRecord qRecord = new QRecord();
                try
                {
+                  if(inputWrapper.getCsvHeadersAsFieldNames())
+                  {
+                     /////////////////////////////////////////////////////////////////////////////////////////
+                     // in csv-headers-as-field-names mode, don't mess with table, and don't do any mapping //
+                     /////////////////////////////////////////////////////////////////////////////////////////
+                     for(Map.Entry<String, String> entry : csvValues.entrySet())
+                     {
+                        setValue(inputWrapper, qRecord, csvHeaderFieldMapping.get(entry.getKey()), entry.getValue());
+                     }
+                  }
+                  else
+                  {
+                     ///////////////////////////////////////
+                     // otherwise, fields come from table //
+                     ///////////////////////////////////////
                      for(QFieldMetaData field : table.getFields().values())
                      {
                         String fieldSource = mapping == null ? field.getName() : String.valueOf(mapping.getFieldSource(field.getName()));
                         fieldSource = adjustHeaderCase(fieldSource, inputWrapper);
                         setValue(inputWrapper, qRecord, field, csvValues.get(fieldSource));
                      }
+                  }
 
                   runRecordCustomizer(recordCustomizer, qRecord);
                }
@@ -247,6 +270,26 @@ public class CsvToQRecordAdapter
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private Map<String, QFieldMetaData> buildCsvHeaderFieldMappingIfNeeded(InputWrapper inputWrapper, List<String> headers)
+   {
+      Map<String, QFieldMetaData> csvHeaderFieldMapping = null;
+      if(inputWrapper.getCsvHeadersAsFieldNames())
+      {
+         csvHeaderFieldMapping = new HashMap<>();
+         for(String header : headers)
+         {
+            header = adjustHeaderCase(header, inputWrapper);
+            csvHeaderFieldMapping.put(header, new QFieldMetaData(header, QFieldType.STRING));
+         }
+      }
+      return csvHeaderFieldMapping;
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -377,6 +420,7 @@ public class CsvToQRecordAdapter
       private boolean                  doCorrectValueTypes = false;
 
       private boolean caseSensitiveHeaders   = false;
+      private boolean csvHeadersAsFieldNames = false;
 
 
 
@@ -618,6 +662,40 @@ public class CsvToQRecordAdapter
 
 
 
+      /*******************************************************************************
+       ** Getter for csvHeadersAsFieldNames
+       **
+       *******************************************************************************/
+      public boolean getCsvHeadersAsFieldNames()
+      {
+         return csvHeadersAsFieldNames;
+      }
+
+
+
+      /*******************************************************************************
+       ** Setter for csvHeadersAsFieldNames
+       **
+       *******************************************************************************/
+      public void setCsvHeadersAsFieldNames(boolean csvHeadersAsFieldNames)
+      {
+         this.csvHeadersAsFieldNames = csvHeadersAsFieldNames;
+      }
+
+
+
+      /*******************************************************************************
+       ** Fluent setter for csvHeadersAsFieldNames
+       **
+       *******************************************************************************/
+      public InputWrapper withCsvHeadersAsFieldNames(boolean csvHeadersAsFieldNames)
+      {
+         this.csvHeadersAsFieldNames = csvHeadersAsFieldNames;
+         return (this);
+      }
+
+
+
       /*******************************************************************************
        ** Getter for doCorrectValueTypes
        **

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/adapters/JsonToQRecordAdapter.java

@@ -103,6 +103,15 @@ public class JsonToQRecordAdapter
    {
       QRecord record = new QRecord();
 
+      if(table == null)
+      {
+         jsonObject.keys().forEachRemaining(key ->
+         {
+            record.setValue(key, jsonObject.optString(key));
+         });
+      }
+      else
+      {
          for(QFieldMetaData field : table.getFields().values())
          {
             String fieldSource = mapping == null ? field.getName() : String.valueOf(mapping.getFieldSource(field.getName()));
@@ -112,6 +121,7 @@ public class JsonToQRecordAdapter
                record.setValue(field.getName(), (Serializable) jsonObject.get(fieldSource));
             }
          }
+      }
 
       return (record);
    }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidator.java

@@ -31,6 +31,7 @@ import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
+import java.util.Optional;
 import java.util.Set;
 import java.util.function.Supplier;
 import java.util.stream.Stream;
@@ -1244,7 +1245,25 @@ public class QInstanceValidator
                         {
                            if(fieldMetaData.getDefaultValue() != null && fieldMetaData.getDefaultValue() instanceof QCodeReference codeReference)
                            {
-                              validateSimpleCodeReference("Process " + processName + " backend step code reference: ", codeReference, BackendStep.class);
+                              ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                              // by default, assume that any process field which is a QCodeReference should be a reference to a BackendStep... //
+                              // but... allow a secondary field name to be set, to tell us what class to *actually* expect here...             //
+                              ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                              Class<?> expectedClass = BackendStep.class;
+                              try
+                              {
+                                 Optional<QFieldMetaData> expectedTypeField = backendStepMetaData.getInputMetaData().getField(fieldMetaData.getName() + "_expectedType");
+                                 if(expectedTypeField.isPresent() && expectedTypeField.get().getDefaultValue() != null)
+                                 {
+                                    expectedClass = Class.forName(ValueUtils.getValueAsString(expectedTypeField.get().getDefaultValue()));
+                                 }
+                              }
+                              catch(Exception e)
+                              {
+                                 warn("Error loading expectedType for field [" + fieldMetaData.getName() + "] in process [" + processName + "]: " + e.getMessage());
+                              }
+
+                              validateSimpleCodeReference("Process " + processName + " code reference: ", codeReference, expectedClass);
                            }
                         }
                      }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QFilterCriteria.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.actions.tables.query;
 import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collection;
 import java.util.List;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
@@ -93,11 +94,37 @@ public class QFilterCriteria implements Serializable, Cloneable
    /*******************************************************************************
     **
     *******************************************************************************/
-   public QFilterCriteria(String fieldName, QCriteriaOperator operator, List<Serializable> values)
+   @SuppressWarnings("unchecked")
+   public QFilterCriteria(String fieldName, QCriteriaOperator operator, List<? extends Serializable> values)
    {
       this.fieldName = fieldName;
       this.operator = operator;
-      this.values = values == null ? new ArrayList<>() : values;
+      this.values = values == null ? new ArrayList<>() : (List<Serializable>) values;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings({ "rawtypes", "unchecked" })
+   public QFilterCriteria(String fieldName, QCriteriaOperator operator, Collection<? extends Serializable> values)
+   {
+      this.fieldName = fieldName;
+      this.operator = operator;
+
+      if(values == null)
+      {
+         this.values = new ArrayList<>();
+      }
+      else if(values instanceof List list)
+      {
+         this.values = list;
+      }
+      else
+      {
+         this.values = new ArrayList<>(values);
+      }
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecord.java

@@ -167,6 +167,16 @@ public class QRecord implements Serializable
             ArrayList<?> cloneList = new ArrayList<>(arrayList);
             clone.put(entry.getKey(), (V) cloneList);
          }
+         else if(entry.getValue() instanceof LinkedHashMap<?, ?> linkedHashMap)
+         {
+            LinkedHashMap<?, ?> cloneMap = new LinkedHashMap<>(linkedHashMap);
+            clone.put(entry.getKey(), (V) cloneMap);
+         }
+         else if(entry.getValue() instanceof HashMap<?, ?> hashMap)
+         {
+            HashMap<?, ?> cloneMap = new HashMap<>(hashMap);
+            clone.put(entry.getKey(), (V) cloneMap);
+         }
          else if(entry.getValue() instanceof QRecord otherQRecord)
          {
             clone.put(entry.getKey(), (V) new QRecord(otherQRecord));

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/MetaDataProducerHelper.java

@@ -95,6 +95,7 @@ public class MetaDataProducerHelper
       ////////////////////////////////////////////////////////////////////////////////////////////
       // sort them by sort order, then by the type that they return - specifically - doing apps //
       // after all other types (as apps often try to get other types from the instance)         //
+      // also - do backends earlier than others (e.g., tables may expect backends to exist)     //
       ////////////////////////////////////////////////////////////////////////////////////////////
       producers.sort(Comparator
          .comparing((MetaDataProducerInterface<?> p) -> p.getSortOrder())
@@ -105,11 +106,15 @@ public class MetaDataProducerHelper
                Class<?> outputType = p.getClass().getMethod("produce", QInstance.class).getReturnType();
                if(outputType.equals(QAppMetaData.class))
                {
-                  return (1);
+                  return (2);
+               }
+               else if(outputType.equals(QBackendMetaData.class))
+               {
+                  return (0);
                }
                else
                {
-                  return (0);
+                  return (1);
                }
             }
             catch(Exception e)

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/QInstance.java

@@ -746,13 +746,23 @@ public class QInstance
 
 
    /*******************************************************************************
-    ** Setter for hasBeenValidated
+    ** If pass a QInstanceValidationKey (which can only be instantiated by the validator),
+    ** then the hasBeenValidated field will be set to true.
     **
+    ** Else, if passed a null, hasBeenValidated will be reset to false - e.g., to
+    ** re-trigger validation (can be useful in tests).
     *******************************************************************************/
    public void setHasBeenValidated(QInstanceValidationKey key)
+   {
+      if(key == null)
+      {
+         this.hasBeenValidated = false;
+      }
+      else
       {
          this.hasBeenValidated = true;
       }
+   }
 
 
 
@@ -1198,4 +1208,14 @@ public class QInstance
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void add(TopLevelMetaDataInterface metaData)
+   {
+      metaData.addSelfToInstance(this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/layout/QAppMetaData.java

@@ -43,6 +43,8 @@ public class QAppMetaData implements QAppChildMetaData, MetaDataWithPermissionRu
    private String name;
    private String label;
 
+   private Integer sortOrder = 500;
+
    private QPermissionRules permissionRules;
 
    private List<QAppChildMetaData> children;
@@ -426,4 +428,36 @@ public class QAppMetaData implements QAppChildMetaData, MetaDataWithPermissionRu
    {
       qInstance.addApp(this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for sortOrder
+    *******************************************************************************/
+   public Integer getSortOrder()
+   {
+      return (this.sortOrder);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for sortOrder
+    *******************************************************************************/
+   public void setSortOrder(Integer sortOrder)
+   {
+      this.sortOrder = sortOrder;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for sortOrder
+    *******************************************************************************/
+   public QAppMetaData withSortOrder(Integer sortOrder)
+   {
+      this.sortOrder = sortOrder;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/AbstractProcessMetaDataBuilder.java

@@ -23,8 +23,11 @@ package com.kingsrook.qqq.backend.core.model.metadata.processes;
 
 
 import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
 import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
 import com.kingsrook.qqq.backend.core.processes.implementations.basepull.BasepullConfiguration;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
 /*******************************************************************************
@@ -32,6 +35,8 @@ import com.kingsrook.qqq.backend.core.processes.implementations.basepull.Basepul
  *******************************************************************************/
 public class AbstractProcessMetaDataBuilder
 {
+   private static final QLogger LOG = QLogger.getLogger(AbstractProcessMetaDataBuilder.class);
+
    protected QProcessMetaData processMetaData;
 
 
@@ -58,6 +63,54 @@ public class AbstractProcessMetaDataBuilder
 
 
 
+   /*******************************************************************************
+    ** Fluent setter for name
+    **
+    *******************************************************************************/
+   public AbstractProcessMetaDataBuilder withName(String name)
+   {
+      processMetaData.setName(name);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for label
+    **
+    *******************************************************************************/
+   public AbstractProcessMetaDataBuilder withLabel(String name)
+   {
+      processMetaData.setLabel(name);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableName
+    **
+    *******************************************************************************/
+   public AbstractProcessMetaDataBuilder withTableName(String tableName)
+   {
+      processMetaData.setTableName(tableName);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for icon
+    **
+    *******************************************************************************/
+   public AbstractProcessMetaDataBuilder withIcon(QIcon icon)
+   {
+      processMetaData.setIcon(icon);
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -65,7 +118,8 @@ public class AbstractProcessMetaDataBuilder
    {
       processMetaData.getInputFields().stream()
          .filter(f -> f.getName().equals(fieldName)).findFirst()
-         .ifPresent(f -> f.setDefaultValue(value));
+         .ifPresentOrElse(f -> f.setDefaultValue(value),
+            () -> LOG.warn("Could not find process input field for setting default value", logPair("processName", () -> processMetaData.getName()), logPair("fieldName", fieldName)));
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/QTableMetaData.java

@@ -34,6 +34,7 @@ import java.util.Optional;
 import java.util.Set;
 import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
 import com.kingsrook.qqq.backend.core.model.data.QRecordEntityField;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
@@ -49,6 +50,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRule
 import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.QTableAutomationDetails;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.cache.CacheOf;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
 /*******************************************************************************
@@ -57,6 +59,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.cache.CacheOf;
  *******************************************************************************/
 public class QTableMetaData implements QAppChildMetaData, Serializable, MetaDataWithPermissionRules, TopLevelMetaDataInterface
 {
+   private static final QLogger LOG = QLogger.getLogger(QTableMetaData.class);
+
    private String name;
    private String label;
 
@@ -813,6 +817,15 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
     *******************************************************************************/
    public QTableMetaData withUniqueKey(UniqueKey uniqueKey)
    {
+      ////////////////////////////////////////////////////////////////////////////////////
+      // you can't add a null key, so, if someone tried, just gracefully return w/ noop //
+      ////////////////////////////////////////////////////////////////////////////////////
+      if(uniqueKey == null)
+      {
+         LOG.debug("Skipping request to add null uniqueKey", logPair("tableName", name));
+         return (this);
+      }
+
       if(this.uniqueKeys == null)
       {
          this.uniqueKeys = new ArrayList<>();
@@ -1130,6 +1143,15 @@ public class QTableMetaData implements QAppChildMetaData, Serializable, MetaData
     *******************************************************************************/
    public QTableMetaData withRecordSecurityLock(RecordSecurityLock recordSecurityLock)
    {
+      /////////////////////////////////////////////////////////////////////////////////////
+      // you can't add a null lock, so, if someone tried, just gracefully return w/ noop //
+      /////////////////////////////////////////////////////////////////////////////////////
+      if(recordSecurityLock == null)
+      {
+         LOG.debug("Skipping request to add null recordSecurityLock", logPair("tableName", name));
+         return (this);
+      }
+
       if(this.recordSecurityLocks == null)
       {
          this.recordSecurityLocks = new ArrayList<>();

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/QTableAutomationDetails.java

@@ -37,6 +37,11 @@ public class QTableAutomationDetails
 
    private Integer overrideBatchSize;
 
+   private String shardByFieldName; // field in "this" table, to use for sharding
+   private String shardSourceTableName; // name of the table where the shards are defined as rows
+   private String shardLabelFieldName; // field in shard-source-table to use for labeling shards
+   private String shardIdFieldName; // field in shard-source-table to identify shards (e.g., joins to this table's shardByFieldName)
+
 
 
    /*******************************************************************************
@@ -188,4 +193,128 @@ public class QTableAutomationDetails
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for shardByFieldName
+    *******************************************************************************/
+   public String getShardByFieldName()
+   {
+      return (this.shardByFieldName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for shardByFieldName
+    *******************************************************************************/
+   public void setShardByFieldName(String shardByFieldName)
+   {
+      this.shardByFieldName = shardByFieldName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for shardByFieldName
+    *******************************************************************************/
+   public QTableAutomationDetails withShardByFieldName(String shardByFieldName)
+   {
+      this.shardByFieldName = shardByFieldName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for shardSourceTableName
+    *******************************************************************************/
+   public String getShardSourceTableName()
+   {
+      return (this.shardSourceTableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for shardSourceTableName
+    *******************************************************************************/
+   public void setShardSourceTableName(String shardSourceTableName)
+   {
+      this.shardSourceTableName = shardSourceTableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for shardSourceTableName
+    *******************************************************************************/
+   public QTableAutomationDetails withShardSourceTableName(String shardSourceTableName)
+   {
+      this.shardSourceTableName = shardSourceTableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for shardLabelFieldName
+    *******************************************************************************/
+   public String getShardLabelFieldName()
+   {
+      return (this.shardLabelFieldName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for shardLabelFieldName
+    *******************************************************************************/
+   public void setShardLabelFieldName(String shardLabelFieldName)
+   {
+      this.shardLabelFieldName = shardLabelFieldName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for shardLabelFieldName
+    *******************************************************************************/
+   public QTableAutomationDetails withShardLabelFieldName(String shardLabelFieldName)
+   {
+      this.shardLabelFieldName = shardLabelFieldName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for shardIdFieldName
+    *******************************************************************************/
+   public String getShardIdFieldName()
+   {
+      return (this.shardIdFieldName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for shardIdFieldName
+    *******************************************************************************/
+   public void setShardIdFieldName(String shardIdFieldName)
+   {
+      this.shardIdFieldName = shardIdFieldName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for shardIdFieldName
+    *******************************************************************************/
+   public QTableAutomationDetails withShardIdFieldName(String shardIdFieldName)
+   {
+      this.shardIdFieldName = shardIdFieldName;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/tables/automation/TableAutomationAction.java

@@ -37,6 +37,7 @@ public class TableAutomationAction
    private TriggerEvent triggerEvent;
    private Integer      priority = 500;
    private QQueryFilter filter;
+   private Serializable shardId;
 
    ////////////////////////////////////////////////////////////////////////
    // flag that will cause the records to cause their associations to be //
@@ -329,4 +330,35 @@ public class TableAutomationAction
       return (this);
    }
 
+
+
+   /*******************************************************************************
+    ** Getter for shardId
+    *******************************************************************************/
+   public Serializable getShardId()
+   {
+      return (this.shardId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for shardId
+    *******************************************************************************/
+   public void setShardId(Serializable shardId)
+   {
+      this.shardId = shardId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for shardId
+    *******************************************************************************/
+   public TableAutomationAction withShardId(Serializable shardId)
+   {
+      this.shardId = shardId;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/QBackendModuleInterface.java

@@ -22,6 +22,7 @@
 package com.kingsrook.qqq.backend.core.modules.backend;
 
 
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.interfaces.AggregateInterface;
 import com.kingsrook.qqq.backend.core.actions.interfaces.CountInterface;
 import com.kingsrook.qqq.backend.core.actions.interfaces.DeleteInterface;
@@ -29,6 +30,8 @@ import com.kingsrook.qqq.backend.core.actions.interfaces.GetInterface;
 import com.kingsrook.qqq.backend.core.actions.interfaces.InsertInterface;
 import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
 import com.kingsrook.qqq.backend.core.actions.interfaces.UpdateInterface;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableBackendDetails;
 
@@ -126,6 +129,14 @@ public interface QBackendModuleInterface
       return null;
    }
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   default QBackendTransaction openTransaction(AbstractTableActionInput input) throws QException
+   {
+      return (new QBackendTransaction());
+   }
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/memory/AbstractMemoryAction.java

@@ -23,7 +23,6 @@ package com.kingsrook.qqq.backend.core.modules.backend.implementations.memory;
 
 
 import java.io.Serializable;
-import com.kingsrook.qqq.backend.core.actions.interfaces.QActionInterface;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
@@ -32,7 +31,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
 /*******************************************************************************
  ** Base class for all core actions in the Memory backend module.
  *******************************************************************************/
-public abstract class AbstractMemoryAction implements QActionInterface
+public abstract class AbstractMemoryAction
 {
 
    /*******************************************************************************

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/memory/MemoryInsertAction.java

@@ -22,13 +22,10 @@
 package com.kingsrook.qqq.backend.core.modules.backend.implementations.memory;
 
 
-import java.time.Instant;
 import com.kingsrook.qqq.backend.core.actions.interfaces.InsertInterface;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
-import com.kingsrook.qqq.backend.core.model.data.QRecord;
-import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 
 
 /*******************************************************************************
@@ -45,18 +42,6 @@ public class MemoryInsertAction extends AbstractMemoryAction implements InsertIn
    {
       try
       {
-         QTableMetaData table = insertInput.getTable();
-         Instant        now   = Instant.now();
-
-         for(QRecord record : insertInput.getRecords())
-         {
-            ///////////////////////////////////////////
-            // todo .. better (not hard-coded names) //
-            ///////////////////////////////////////////
-            setValueIfTableHasField(record, table, "createDate", now, false);
-            setValueIfTableHasField(record, table, "modifyDate", now, false);
-         }
-
          InsertOutput insertOutput = new InsertOutput();
          insertOutput.setRecords(MemoryRecordStore.getInstance().insert(insertInput, true));
          return (insertOutput);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamed/StreamedETLBackendStep.java

@@ -27,7 +27,6 @@ import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.async.AsyncRecordPipeLoop;
 import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
 import com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe;
-import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
@@ -116,7 +115,7 @@ public class StreamedETLBackendStep implements BackendStep
 
       insertInput.setTableName(runBackendStepInput.getValueString(BasicETLProcess.FIELD_DESTINATION_TABLE));
 
-      return new InsertAction().openTransaction(insertInput);
+      return QBackendTransaction.openFor(insertInput);
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaDeleteStep.java

@@ -26,7 +26,6 @@ import java.util.Optional;
 import java.util.stream.Collectors;
 import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
-import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
@@ -34,7 +33,6 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.InputSource;
 import com.kingsrook.qqq.backend.core.model.actions.tables.QInputSource;
 import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
-import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 
 
@@ -87,9 +85,8 @@ public class LoadViaDeleteStep extends AbstractLoadStep
    @Override
    public Optional<QBackendTransaction> openTransaction(RunBackendStepInput runBackendStepInput) throws QException
    {
-      InsertInput insertInput = new InsertInput();
-      insertInput.setTableName(runBackendStepInput.getValueString(FIELD_DESTINATION_TABLE));
-
-      return (Optional.of(new InsertAction().openTransaction(insertInput)));
+      DeleteInput deleteInput = new DeleteInput();
+      deleteInput.setTableName(runBackendStepInput.getValueString(FIELD_DESTINATION_TABLE));
+      return (Optional.of(QBackendTransaction.openFor(deleteInput)));
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaInsertOrUpdateStep.java

@@ -129,8 +129,7 @@ public class LoadViaInsertOrUpdateStep extends AbstractLoadStep
    {
       InsertInput insertInput = new InsertInput();
       insertInput.setTableName(runBackendStepInput.getValueString(FIELD_DESTINATION_TABLE));
-
-      return (Optional.of(new InsertAction().openTransaction(insertInput)));
+      return (Optional.of(QBackendTransaction.openFor(insertInput)));
    }
 
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaInsertStep.java

@@ -88,7 +88,6 @@ public class LoadViaInsertStep extends AbstractLoadStep
    {
       InsertInput insertInput = new InsertInput();
       insertInput.setTableName(runBackendStepInput.getValueString(FIELD_DESTINATION_TABLE));
-
-      return (Optional.of(new InsertAction().openTransaction(insertInput)));
+      return (Optional.of(QBackendTransaction.openFor(insertInput)));
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/LoadViaUpdateStep.java

@@ -24,14 +24,12 @@ package com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwit
 
 import java.util.Optional;
 import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
-import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.InputSource;
 import com.kingsrook.qqq.backend.core.model.actions.tables.QInputSource;
-import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateOutput;
 
@@ -81,9 +79,8 @@ public class LoadViaUpdateStep extends AbstractLoadStep
    @Override
    public Optional<QBackendTransaction> openTransaction(RunBackendStepInput runBackendStepInput) throws QException
    {
-      InsertInput insertInput = new InsertInput();
-      insertInput.setTableName(runBackendStepInput.getValueString(FIELD_DESTINATION_TABLE));
-
-      return (Optional.of(new InsertAction().openTransaction(insertInput)));
+      UpdateInput updateInput = new UpdateInput();
+      updateInput.setTableName(runBackendStepInput.getValueString(FIELD_DESTINATION_TABLE));
+      return (Optional.of(QBackendTransaction.openFor(updateInput)));
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/StreamedETLExecuteStep.java

@@ -34,7 +34,6 @@ import com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.audits.AuditInput;
-import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/scripts/StoreScriptRevisionProcessStep.java

@@ -75,7 +75,7 @@ public class StoreScriptRevisionProcessStep implements BackendStep
       InsertAction insertAction = new InsertAction();
       InsertInput  insertInput  = new InsertInput();
       insertInput.setTableName(ScriptRevision.TABLE_NAME);
-      QBackendTransaction transaction = insertAction.openTransaction(insertInput);
+      QBackendTransaction transaction = QBackendTransaction.openFor(insertInput);
       insertInput.setTransaction(transaction);
 
       try

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/tablesync/AbstractTableSyncTransformStep.java

@@ -182,6 +182,14 @@ public abstract class AbstractTableSyncTransformStep extends AbstractTransformSt
    public record SyncProcessConfig(String sourceTable, String sourceTableKeyField, String destinationTable, String destinationTableForeignKey, boolean performInserts, boolean performUpdates)
    {
 
+      /*******************************************************************************
+       ** Overloaded constructor - defaults both performInserts & performUpdates to true.
+       *******************************************************************************/
+      public SyncProcessConfig(String sourceTable, String sourceTableKeyField, String destinationTable, String destinationTableForeignKey)
+      {
+         this(sourceTable, sourceTableKeyField, destinationTable, destinationTableForeignKey, true, true);
+      }
+
       /*******************************************************************************
        ** artificial method, here to make jacoco see that this class is indeed
        ** included in test coverage...
@@ -268,6 +276,7 @@ public abstract class AbstractTableSyncTransformStep extends AbstractTransformSt
 
          if(sourceKeyValue == null || "".equals(sourceKeyValue))
          {
+            LOG.debug("Skipping record without a value in the sourceKeyField", logPair("keyField", sourceTableKeyField));
             errorMissingKeyField.incrementCountAndAddPrimaryKey(sourcePrimaryKey);
 
             try

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/scheduler/ScheduleManager.java

@@ -120,6 +120,15 @@ public class ScheduleManager
          return;
       }
 
+      boolean needToClearContext = false;
+      try
+      {
+         if(QContext.getQInstance() == null)
+         {
+            needToClearContext = true;
+            QContext.init(qInstance, sessionSupplier.get());
+         }
+
          for(QQueueProviderMetaData queueProvider : qInstance.getQueueProviders().values())
          {
             startQueueProvider(queueProvider);
@@ -148,7 +157,6 @@ public class ScheduleManager
                   // if this a "parallel", which for example means we want to have a thread for each backend variant //
                   // running at the same time, get the variant records and schedule each separately                  //
                   /////////////////////////////////////////////////////////////////////////////////////////////////////
-               QContext.init(qInstance, sessionSupplier.get());
                   QBackendMetaData backendMetaData = qInstance.getBackend(scheduleMetaData.getVariantBackend());
                   for(QRecord qRecord : CollectionUtils.nonNullList(getBackendVariantFilteredRecords(process)))
                   {
@@ -169,6 +177,14 @@ public class ScheduleManager
             }
          }
       }
+      finally
+      {
+         if(needToClearContext)
+         {
+            QContext.clear();
+         }
+      }
+   }
 
 
 
@@ -210,8 +226,8 @@ public class ScheduleManager
       // ask the PollingAutomationPerTableRunner how many threads of itself need setup //
       // then start a scheduled executor foreach one                                   //
       ///////////////////////////////////////////////////////////////////////////////////
-      List<PollingAutomationPerTableRunner.TableActions> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, automationProvider.getName());
-      for(PollingAutomationPerTableRunner.TableActions tableAction : tableActions)
+      List<PollingAutomationPerTableRunner.TableActionsInterface> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, automationProvider.getName());
+      for(PollingAutomationPerTableRunner.TableActionsInterface tableAction : tableActions)
       {
          if(allowedToStart(tableAction.tableName()))
          {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/utils/ValueUtils.java

@@ -574,7 +574,7 @@ public class ValueUtils
    /*******************************************************************************
     **
     *******************************************************************************/
-   public static LocalTime getValueAsLocalTime(Serializable value)
+   public static LocalTime getValueAsLocalTime(Object value)
    {
       try
       {
@@ -615,7 +615,7 @@ public class ValueUtils
    /*******************************************************************************
     **
     *******************************************************************************/
-   public static byte[] getValueAsByteArray(Serializable value)
+   public static byte[] getValueAsByteArray(Object value)
    {
       if(value == null)
       {
@@ -641,7 +641,7 @@ public class ValueUtils
     **
     *******************************************************************************/
    @SuppressWarnings("unchecked")
-   public static <T extends Serializable> T getValueAsType(Class<T> type, Serializable value)
+   public static <T extends Serializable> T getValueAsType(Class<T> type, Object value)
    {
       if(type.equals(Integer.class))
       {
@@ -687,7 +687,7 @@ public class ValueUtils
     **
     *******************************************************************************/
    @SuppressWarnings("checkstyle:indentation")
-   public static Serializable getValueAsFieldType(QFieldType type, Serializable value)
+   public static Serializable getValueAsFieldType(QFieldType type, Object value)
    {
       return switch(type)
       {

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/audits/DMLAuditActionTest.java

@@ -400,4 +400,49 @@ class DMLAuditActionTest extends BaseTest
       QContext.popAction();
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testTableWithoutIntegerPrimaryKey() throws QException
+   {
+      QInstance qInstance = QContext.getQInstance();
+      new AuditsMetaDataProvider().defineAll(qInstance, TestUtils.MEMORY_BACKEND_NAME, null);
+
+      ////////////////////////////////////////////////////////////////////////////////////////////////////
+      // we used to throw if table had no primary key.  first, assert that we do not throw in that case //
+      ////////////////////////////////////////////////////////////////////////////////////////////////////
+      QContext.getQInstance().addTable(
+         new QTableMetaData()
+            .withName("nullPkey")
+            .withField(new QFieldMetaData("foo", QFieldType.STRING))
+            .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD)));
+
+      new DMLAuditAction().execute(new DMLAuditInput()
+         .withTableActionInput(new InsertInput("nullPkey"))
+         .withRecordList(List.of(new QRecord())));
+
+      //////////////////////////////////////////////////////////////////////////////////////////////
+      // next, make sure we don't throw (and don't record anything) if table's pkey isn't integer //
+      //////////////////////////////////////////////////////////////////////////////////////////////
+      QContext.getQInstance().addTable(
+         new QTableMetaData()
+            .withName("stringPkey")
+            .withField(new QFieldMetaData("idString", QFieldType.STRING))
+            .withPrimaryKeyField("idString")
+            .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD)));
+
+      new DMLAuditAction().execute(new DMLAuditInput()
+         .withTableActionInput(new InsertInput("stringPkey"))
+         .withRecordList(List.of(new QRecord())));
+
+      //////////////////////////////////
+      // make sure no audits happened //
+      //////////////////////////////////
+      List<QRecord> auditList = TestUtils.queryTable("audit");
+      assertTrue(auditList.isEmpty());
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/automation/polling/PollingAutomationPerTableRunnerTest.java

@@ -202,8 +202,8 @@ class PollingAutomationPerTableRunnerTest extends BaseTest
     *******************************************************************************/
    private void runAllTableActions(QInstance qInstance) throws QException
    {
-      List<PollingAutomationPerTableRunner.TableActions> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, TestUtils.POLLING_AUTOMATION);
-      for(PollingAutomationPerTableRunner.TableActions tableAction : tableActions)
+      List<PollingAutomationPerTableRunner.TableActionsInterface> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, TestUtils.POLLING_AUTOMATION);
+      for(PollingAutomationPerTableRunner.TableActionsInterface tableAction : tableActions)
       {
          PollingAutomationPerTableRunner pollingAutomationPerTableRunner = new PollingAutomationPerTableRunner(qInstance, TestUtils.POLLING_AUTOMATION, QSession::new, tableAction);
 
@@ -504,8 +504,8 @@ class PollingAutomationPerTableRunnerTest extends BaseTest
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       assertThatThrownBy(() ->
       {
-         List<PollingAutomationPerTableRunner.TableActions> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, TestUtils.POLLING_AUTOMATION);
-         for(PollingAutomationPerTableRunner.TableActions tableAction : tableActions)
+         List<PollingAutomationPerTableRunner.TableActionsInterface> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, TestUtils.POLLING_AUTOMATION);
+         for(PollingAutomationPerTableRunner.TableActionsInterface tableAction : tableActions)
          {
             PollingAutomationPerTableRunner pollingAutomationPerTableRunner = new PollingAutomationPerTableRunnerThatShouldSimulateServerShutdownMidRun(qInstance, TestUtils.POLLING_AUTOMATION, QSession::new, tableAction);
 
@@ -564,7 +564,7 @@ class PollingAutomationPerTableRunnerTest extends BaseTest
       /*******************************************************************************
        **
        *******************************************************************************/
-      public PollingAutomationPerTableRunnerThatShouldSimulateServerShutdownMidRun(QInstance instance, String providerName, Supplier<QSession> sessionSupplier, TableActions tableActions)
+      public PollingAutomationPerTableRunnerThatShouldSimulateServerShutdownMidRun(QInstance instance, String providerName, Supplier<QSession> sessionSupplier, TableActionsInterface tableActions)
       {
          super(instance, providerName, sessionSupplier, tableActions);
       }
@@ -581,4 +581,16 @@ class PollingAutomationPerTableRunnerTest extends BaseTest
       }
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testLoadingRecordTypesToEnsureClassCoverage()
+   {
+      new PollingAutomationPerTableRunner.TableActions(null, null).noopToFakeTestCoverage();
+      new PollingAutomationPerTableRunner.ShardedTableActions(null, null, null, null, null).noopToFakeTestCoverage();
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/automation/polling/StandardScheduledExecutorTest.java

@@ -186,9 +186,9 @@ class StandardScheduledExecutorTest extends BaseTest
     *******************************************************************************/
    private void runPollingAutomationExecutorForAwhile(QInstance qInstance, Supplier<QSession> sessionSupplier)
    {
-      List<PollingAutomationPerTableRunner.TableActions> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, TestUtils.POLLING_AUTOMATION);
+      List<PollingAutomationPerTableRunner.TableActionsInterface> tableActions = PollingAutomationPerTableRunner.getTableActions(qInstance, TestUtils.POLLING_AUTOMATION);
       List<StandardScheduledExecutor>                    executors    = new ArrayList<>();
-      for(PollingAutomationPerTableRunner.TableActions tableAction : tableActions)
+      for(PollingAutomationPerTableRunner.TableActionsInterface tableAction : tableActions)
       {
          PollingAutomationPerTableRunner pollingAutomationPerTableRunner = new PollingAutomationPerTableRunner(qInstance, TestUtils.POLLING_AUTOMATION, sessionSupplier, tableAction);
          StandardScheduledExecutor       pollingAutomationExecutor       = new StandardScheduledExecutor(pollingAutomationPerTableRunner);

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/processes/QProcessCallbackFactoryTest.java

@@ -0,0 +1,58 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.processes;
+
+
+import java.util.ArrayList;
+import java.util.Collections;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for QProcessCallbackFactory 
+ *******************************************************************************/
+class QProcessCallbackFactoryTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test()
+   {
+      QProcessCallback qProcessCallback = QProcessCallbackFactory.forFilter(new QQueryFilter(new QFilterCriteria("foo", QCriteriaOperator.EQUALS, "bar")));
+
+      QQueryFilter queryFilter = qProcessCallback.getQueryFilter();
+      assertEquals(1, queryFilter.getCriteria().size());
+      assertEquals("foo", queryFilter.getCriteria().get(0).getFieldName());
+      assertEquals(QCriteriaOperator.EQUALS, queryFilter.getCriteria().get(0).getOperator());
+      assertEquals("bar", queryFilter.getCriteria().get(0).getValues().get(0));
+
+      assertEquals(Collections.emptyMap(), qProcessCallback.getFieldValues(new ArrayList<>()));
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/scripts/RunAssociatedScriptActionTest.java

@@ -156,10 +156,10 @@ class RunAssociatedScriptActionTest extends BaseTest
       /////////////////////////////////////
       assertEquals(N, TestUtils.queryTable(ScriptLog.TABLE_NAME).size());
 
-      ////////////////////////////////////////////////////////////////////////////////////////
-      // and we should have just ran 2 inserts - for the log & logLines (even though empty) //
-      ////////////////////////////////////////////////////////////////////////////////////////
-      assertEquals(2, MemoryRecordStore.getStatistics().get(MemoryRecordStore.STAT_INSERTS_RAN));
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // and we should have just ran 1 inserts - for the log (no longer run one for empty insert of 0 log-lines) //
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      assertEquals(1, MemoryRecordStore.getStatistics().get(MemoryRecordStore.STAT_INSERTS_RAN));
 
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       // and we shouldn't have run N queries (which we would have (at least), if we would have built a new Action object inside the loop) //

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/UpdateActionRecordSplitHelperTest.java

@@ -0,0 +1,147 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.actions.tables.helpers;
+
+
+import java.time.Instant;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.statusmessages.SystemErrorStatusMessage;
+import com.kingsrook.qqq.backend.core.utils.ListingHash;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for UpdateActionRecordSplitHelper 
+ *******************************************************************************/
+class UpdateActionRecordSplitHelperTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test()
+   {
+      String tableName = getClass().getSimpleName();
+      QContext.getQInstance().addTable(new QTableMetaData()
+         .withName(tableName)
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("A", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("B", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
+
+      UpdateInput updateInput = new UpdateInput(tableName)
+         .withRecord(new QRecord().withValue("id", 1).withValue("A", 1))
+         .withRecord(new QRecord().withValue("id", 2).withValue("A", 2))
+         .withRecord(new QRecord().withValue("id", 3).withValue("B", 3))
+         .withRecord(new QRecord().withValue("id", 4).withValue("B", 3))
+         .withRecord(new QRecord().withValue("id", 5).withValue("B", 3))
+         .withRecord(new QRecord().withValue("id", 6).withValue("A", 4).withValue("B", 5));
+      UpdateActionRecordSplitHelper updateActionRecordSplitHelper = new UpdateActionRecordSplitHelper();
+      updateActionRecordSplitHelper.init(updateInput);
+      ListingHash<List<String>, QRecord> recordsByFieldBeingUpdated = updateActionRecordSplitHelper.getRecordsByFieldBeingUpdated();
+
+      Function<Collection<QRecord>, Set<Integer>> extractIds = (records) ->
+         records.stream().map(r -> r.getValueInteger("id")).collect(Collectors.toSet());
+
+      ////////////////////////////////////////
+      // validate that modify dates got set //
+      ////////////////////////////////////////
+      updateInput.getRecords().forEach(r ->
+         assertThat(r.getValue("modifyDate")).isInstanceOf(Instant.class));
+
+      //////////////////////////////////////////////////////////////
+      // validate the grouping of records by fields-being-updated //
+      //////////////////////////////////////////////////////////////
+      assertEquals(3, recordsByFieldBeingUpdated.size());
+      assertEquals(Set.of(1, 2), extractIds.apply(recordsByFieldBeingUpdated.get(List.of("A", "modifyDate"))));
+      assertEquals(Set.of(3, 4, 5), extractIds.apply(recordsByFieldBeingUpdated.get(List.of("B", "modifyDate"))));
+      assertEquals(Set.of(6), extractIds.apply(recordsByFieldBeingUpdated.get(List.of("A", "B", "modifyDate"))));
+
+      ///////////////////////////////////////////////////////////////////
+      // validate the output records were built, in the order expected //
+      ///////////////////////////////////////////////////////////////////
+      List<QRecord> outputRecords = updateActionRecordSplitHelper.getOutputRecords();
+      for(int i = 0; i < outputRecords.size(); i++)
+      {
+         assertEquals(i + 1, outputRecords.get(i).getValueInteger("id"));
+      }
+
+      /////////////////////////////////////////////////////
+      // test the areAllValuesBeingUpdatedTheSame method //
+      /////////////////////////////////////////////////////
+      Function<List<String>, Boolean> runAreAllValuesBeingUpdatedTheSame = (fields) ->
+         UpdateActionRecordSplitHelper.areAllValuesBeingUpdatedTheSame(updateInput, recordsByFieldBeingUpdated.get(fields), fields);
+
+      assertFalse(runAreAllValuesBeingUpdatedTheSame.apply(List.of("A", "modifyDate")));
+      assertTrue(runAreAllValuesBeingUpdatedTheSame.apply(List.of("B", "modifyDate")));
+      assertTrue(runAreAllValuesBeingUpdatedTheSame.apply(List.of("A", "B", "modifyDate")));
+
+      ////////////////////////////////////////////////////////////////////
+      // make sure that the override of the logic for this method works //
+      ////////////////////////////////////////////////////////////////////
+      updateInput.setAreAllValuesBeingUpdatedTheSame(true);
+      assertTrue(runAreAllValuesBeingUpdatedTheSame.apply(List.of("A", "modifyDate")));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testRecordsWithErrors()
+   {
+      String tableName = getClass().getSimpleName() + "WithErrors";
+      QContext.getQInstance().addTable(new QTableMetaData()
+         .withName(tableName)
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("A", QFieldType.INTEGER)));
+
+      {
+         UpdateInput updateInput = new UpdateInput(tableName)
+            .withRecord(new QRecord().withValue("id", 1).withValue("A", 1).withError(new SystemErrorStatusMessage("error")))
+            .withRecord(new QRecord().withValue("id", 2).withValue("A", 2).withError(new SystemErrorStatusMessage("error")))
+            .withRecord(new QRecord().withValue("id", 2).withValue("A", 3).withError(new SystemErrorStatusMessage("error")));
+         UpdateActionRecordSplitHelper updateActionRecordSplitHelper = new UpdateActionRecordSplitHelper();
+         updateActionRecordSplitHelper.init(updateInput);
+         assertFalse(updateActionRecordSplitHelper.getHaveAnyWithoutErrors());
+      }
+
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/adapters/CsvToQRecordAdapterTest.java

@@ -465,4 +465,60 @@ class CsvToQRecordAdapterTest extends BaseTest
       assertThat(qRecord.getErrors().get(0).toString()).isEqualTo("Error parsing line #2: Value [green] could not be converted to an Integer.");
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testCsvHeadersAsFields() throws QException
+   {
+      CsvToQRecordAdapter csvToQRecordAdapter = new CsvToQRecordAdapter();
+      csvToQRecordAdapter.buildRecordsFromCsv(new CsvToQRecordAdapter.InputWrapper()
+         .withCsvHeadersAsFieldNames(true)
+         .withCaseSensitiveHeaders(true)
+         .withCsv("""
+            firstName,birthDate,favoriteShapeId
+            John,1980,1
+            Paul,1970-06-15,green
+            """));
+
+      List<QRecord> qRecords = csvToQRecordAdapter.getRecordList();
+
+      QRecord qRecord = qRecords.get(0);
+      assertEquals("John", qRecord.getValue("firstName"));
+      assertEquals("1980", qRecord.getValue("birthDate"));
+      assertEquals("1", qRecord.getValue("favoriteShapeId"));
+
+      qRecord = qRecords.get(1);
+      assertEquals("Paul", qRecord.getValue("firstName"));
+      assertEquals("1970-06-15", qRecord.getValue("birthDate"));
+      assertEquals("green", qRecord.getValue("favoriteShapeId"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testCsvHeadersAsFieldsDuplicatedNames() throws QException
+   {
+      CsvToQRecordAdapter csvToQRecordAdapter = new CsvToQRecordAdapter();
+      csvToQRecordAdapter.buildRecordsFromCsv(new CsvToQRecordAdapter.InputWrapper()
+         .withCsvHeadersAsFieldNames(true)
+         .withCaseSensitiveHeaders(true)
+         .withCsv("""
+            orderId,sku,sku
+            10001,BASIC1,BASIC2
+            """));
+
+      List<QRecord> qRecords = csvToQRecordAdapter.getRecordList();
+
+      QRecord qRecord = qRecords.get(0);
+      assertEquals("10001", qRecord.getValue("orderId"));
+      assertEquals("BASIC1", qRecord.getValue("sku"));
+      assertEquals("BASIC2", qRecord.getValue("sku 2"));
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/adapters/JsonToQRecordAdapterTest.java

@@ -164,6 +164,29 @@ class JsonToQRecordAdapterTest extends BaseTest
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   public void test_buildRecordsFromJsonWithoutTable_inputList()
+   {
+      JsonToQRecordAdapter jsonToQRecordAdapter = new JsonToQRecordAdapter();
+      List<QRecord> qRecords = jsonToQRecordAdapter.buildRecordsFromJson("""
+         [
+            { "firstName":"Tyler", "last":"Samples" },
+            { "firstName":"Tim", "lastName":"Chamberlain" }
+         ]
+         """, null, null);
+      assertNotNull(qRecords);
+      assertEquals(2, qRecords.size());
+      assertEquals("Tyler", qRecords.get(0).getValue("firstName"));
+      assertEquals("Samples", qRecords.get(0).getValue("last"));
+      assertEquals("Tim", qRecords.get(1).getValue("firstName"));
+      assertEquals("Chamberlain", qRecords.get(1).getValue("lastName"));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/data/QRecordTest.java

@@ -25,6 +25,7 @@ package com.kingsrook.qqq.backend.core.model.data;
 import java.math.BigDecimal;
 import java.util.ArrayList;
 import java.util.HashMap;
+import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.BaseTest;
@@ -33,6 +34,7 @@ import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
 import org.junit.jupiter.api.Test;
 import static com.kingsrook.qqq.backend.core.model.data.QRecord.BACKEND_DETAILS_TYPE_HEAVY_FIELD_LENGTHS;
 import static com.kingsrook.qqq.backend.core.model.data.QRecord.BACKEND_DETAILS_TYPE_JSON_SOURCE_OBJECT;
+import static org.assertj.core.api.Assertions.assertThat;
 import static org.junit.jupiter.api.Assertions.assertArrayEquals;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertNotEquals;
@@ -147,10 +149,6 @@ class QRecordTest extends BaseTest
       QRecord byteArrayValue = new QRecord().withValue("myBytes", new byte[] { 65, 66, 67, 68 });
       assertArrayEquals(new byte[] { 65, 66, 67, 68 }, new QRecord(byteArrayValue).getValueByteArray("myBytes"));
 
-      ArrayList<Integer> originalArrayList        = new ArrayList<>(List.of(1, 2, 3));
-      QRecord            recordWithArrayListValue = new QRecord().withValue("myList", originalArrayList);
-      QRecord            cloneWithArrayListValue  = new QRecord(recordWithArrayListValue);
-
       ////////////////////////////////////////////
       // qrecord as a value inside another (!?) //
       ////////////////////////////////////////////
@@ -159,18 +157,6 @@ class QRecordTest extends BaseTest
       assertEquals(1, ((QRecord) cloneWithNestedQRecord.getValue("myRecord")).getValueInteger("A"));
       assertNotSame(cloneWithNestedQRecord.getValue("myRecord"), nestedQRecordValue.getValue("myRecord"));
 
-      ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      // the clone list and original list should be equals (have contents that are equals), but not be the same (reference) //
-      ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      assertEquals(List.of(1, 2, 3), cloneWithArrayListValue.getValue("myList"));
-      assertNotSame(originalArrayList, cloneWithArrayListValue.getValue("myList"));
-
-      //////////////////////////////////////////////////////////////////////////////////////////////////////
-      // make sure a change to the original list doesn't change the cloned list (as it was cloned deeply) //
-      //////////////////////////////////////////////////////////////////////////////////////////////////////
-      originalArrayList.add(4);
-      assertNotEquals(originalArrayList, cloneWithArrayListValue.getValue("myList"));
-
       QRecord emptyRecord = new QRecord();
       QRecord emptyClone  = new QRecord(emptyRecord);
       assertNull(emptyClone.getTableName());
@@ -183,4 +169,59 @@ class QRecordTest extends BaseTest
       assertEquals(0, emptyClone.getAssociatedRecords().size());
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testListAsValue()
+   {
+      ArrayList<Integer> originalArrayList        = new ArrayList<>(List.of(1, 2, 3));
+      QRecord            recordWithArrayListValue = new QRecord().withValue("myList", originalArrayList);
+      QRecord            cloneWithArrayListValue  = new QRecord(recordWithArrayListValue);
+
+      ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // the clone list and original list should be equals (have contents that are equals), but not be the same (reference) //
+      ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      assertEquals(List.of(1, 2, 3), cloneWithArrayListValue.getValue("myList"));
+      assertNotSame(originalArrayList, cloneWithArrayListValue.getValue("myList"));
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////
+      // make sure a change to the original list doesn't change the cloned list (as it was cloned deeply) //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////
+      originalArrayList.add(4);
+      assertNotEquals(originalArrayList, cloneWithArrayListValue.getValue("myList"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testMapAsValue()
+   {
+      LinkedHashMap<String, Integer> originalMap        = new LinkedHashMap<>(Map.of("one", 1, "two", 2, "three", 3));
+      QRecord                        recordWithMapValue = new QRecord().withValue("myMap", originalMap);
+      QRecord                        cloneWithMapValue  = new QRecord(recordWithMapValue);
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // the clone map and original map should be equals (have contents that are equals), but not be the same (reference) //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      assertEquals(originalMap, cloneWithMapValue.getValue("myMap"));
+      assertNotSame(originalMap, cloneWithMapValue.getValue("myMap"));
+
+      //////////////////////////////////////////////////////////
+      // make sure we re-created it as the same subtype (LHM) //
+      //////////////////////////////////////////////////////////
+      assertThat(cloneWithMapValue.getValue("myMap")).isInstanceOf(LinkedHashMap.class);
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////
+      // make sure a change to the original list doesn't change the cloned list (as it was cloned deeply) //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////
+      originalMap.put("four", 4);
+      assertNotEquals(originalMap, cloneWithMapValue.getValue("myMap"));
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/garbagecollector/GarbageCollectorTest.java

@@ -74,7 +74,7 @@ class GarbageCollectorTest extends BaseTest
    @Test
    void testBasic() throws QException
    {
-      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_PERSON_MEMORY, "createDate", NowWithOffset.minus(30, ChronoUnit.DAYS), null);
+      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_PERSON_MEMORY, "timestamp", NowWithOffset.minus(30, ChronoUnit.DAYS), null);
       QContext.getQInstance().addProcess(process);
 
       new InsertAction().execute(new InsertInput(TestUtils.TABLE_NAME_PERSON_MEMORY).withRecords(getPersonRecords()));
@@ -97,11 +97,11 @@ class GarbageCollectorTest extends BaseTest
    private static List<QRecord> getPersonRecords()
    {
       List<QRecord> records = List.of(
-         new QRecord().withValue("id", 1).withValue("createDate", Instant.now().minus(90, ChronoUnit.DAYS)),
-         new QRecord().withValue("id", 2).withValue("createDate", Instant.now().minus(31, ChronoUnit.DAYS)),
-         new QRecord().withValue("id", 3).withValue("createDate", Instant.now().minus(30, ChronoUnit.DAYS).minus(5, ChronoUnit.MINUTES)),
-         new QRecord().withValue("id", 4).withValue("createDate", Instant.now().minus(29, ChronoUnit.DAYS).minus(23, ChronoUnit.HOURS)),
-         new QRecord().withValue("id", 5).withValue("createDate", Instant.now().minus(5, ChronoUnit.DAYS)));
+         new QRecord().withValue("id", 1).withValue("timestamp", Instant.now().minus(90, ChronoUnit.DAYS)),
+         new QRecord().withValue("id", 2).withValue("timestamp", Instant.now().minus(31, ChronoUnit.DAYS)),
+         new QRecord().withValue("id", 3).withValue("timestamp", Instant.now().minus(30, ChronoUnit.DAYS).minus(5, ChronoUnit.MINUTES)),
+         new QRecord().withValue("id", 4).withValue("timestamp", Instant.now().minus(29, ChronoUnit.DAYS).minus(23, ChronoUnit.HOURS)),
+         new QRecord().withValue("id", 5).withValue("timestamp", Instant.now().minus(5, ChronoUnit.DAYS)));
       return records;
    }
 
@@ -113,7 +113,7 @@ class GarbageCollectorTest extends BaseTest
    @Test
    void testOverrideDate() throws QException
    {
-      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_PERSON_MEMORY, "createDate", NowWithOffset.minus(30, ChronoUnit.DAYS), null);
+      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_PERSON_MEMORY, "timestamp", NowWithOffset.minus(30, ChronoUnit.DAYS), null);
       QContext.getQInstance().addProcess(process);
 
       new InsertAction().execute(new InsertInput(TestUtils.TABLE_NAME_PERSON_MEMORY).withRecords(getPersonRecords()));
@@ -157,7 +157,7 @@ class GarbageCollectorTest extends BaseTest
    @Test
    void testWithDeleteAllJoins() throws QException
    {
-      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_ORDER, "createDate", NowWithOffset.minus(30, ChronoUnit.DAYS), "*");
+      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_ORDER, "timestamp", NowWithOffset.minus(30, ChronoUnit.DAYS), "*");
       QContext.getQInstance().addProcess(process);
 
       QContext.getQSession().withSecurityKeyValue(TestUtils.SECURITY_KEY_TYPE_STORE_ALL_ACCESS, true);
@@ -192,7 +192,7 @@ class GarbageCollectorTest extends BaseTest
    @Test
    void testWithDeleteSomeJoins() throws QException
    {
-      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_ORDER, "createDate", NowWithOffset.minus(30, ChronoUnit.DAYS), TestUtils.TABLE_NAME_LINE_ITEM);
+      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_ORDER, "timestamp", NowWithOffset.minus(30, ChronoUnit.DAYS), TestUtils.TABLE_NAME_LINE_ITEM);
       QContext.getQInstance().addProcess(process);
 
       //////////////////////////////////////////////////////////////////////////
@@ -232,7 +232,7 @@ class GarbageCollectorTest extends BaseTest
    @Test
    void testWithDeleteNoJoins() throws QException
    {
-      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_ORDER, "createDate", NowWithOffset.minus(30, ChronoUnit.DAYS), null);
+      QProcessMetaData process = GarbageCollectorProcessMetaDataProducer.createProcess(TestUtils.TABLE_NAME_ORDER, "timestamp", NowWithOffset.minus(30, ChronoUnit.DAYS), null);
       QContext.getQInstance().addProcess(process);
 
       ////////////////////////////////////////////////////////////////////////////////
@@ -270,11 +270,11 @@ class GarbageCollectorTest extends BaseTest
    private static List<QRecord> getOrderRecords()
    {
       List<QRecord> records = List.of(
-         new QRecord().withValue("id", 1).withValue("createDate", Instant.now().minus(90, ChronoUnit.DAYS)),
-         new QRecord().withValue("id", 2).withValue("createDate", Instant.now().minus(31, ChronoUnit.DAYS)),
-         new QRecord().withValue("id", 3).withValue("createDate", Instant.now().minus(30, ChronoUnit.DAYS).minus(5, ChronoUnit.MINUTES)),
-         new QRecord().withValue("id", 4).withValue("createDate", Instant.now().minus(29, ChronoUnit.DAYS).minus(23, ChronoUnit.HOURS)),
-         new QRecord().withValue("id", 5).withValue("createDate", Instant.now().minus(5, ChronoUnit.DAYS)));
+         new QRecord().withValue("id", 1).withValue("timestamp", Instant.now().minus(90, ChronoUnit.DAYS)),
+         new QRecord().withValue("id", 2).withValue("timestamp", Instant.now().minus(31, ChronoUnit.DAYS)),
+         new QRecord().withValue("id", 3).withValue("timestamp", Instant.now().minus(30, ChronoUnit.DAYS).minus(5, ChronoUnit.MINUTES)),
+         new QRecord().withValue("id", 4).withValue("timestamp", Instant.now().minus(29, ChronoUnit.DAYS).minus(23, ChronoUnit.HOURS)),
+         new QRecord().withValue("id", 5).withValue("timestamp", Instant.now().minus(5, ChronoUnit.DAYS)));
       return records;
    }
 

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/TestUtils.java

@@ -549,7 +549,9 @@ public class TestUtils
          .withField(new QFieldMetaData("cost", QFieldType.DECIMAL).withDisplayFormat(DisplayFormat.CURRENCY))
          .withField(new QFieldMetaData("price", QFieldType.DECIMAL).withDisplayFormat(DisplayFormat.CURRENCY))
          .withField(new QFieldMetaData("ssn", QFieldType.STRING).withType(QFieldType.PASSWORD))
-         .withField(new QFieldMetaData("superSecret", QFieldType.STRING).withType(QFieldType.PASSWORD).withIsHidden(true));
+         .withField(new QFieldMetaData("superSecret", QFieldType.STRING).withType(QFieldType.PASSWORD).withIsHidden(true))
+         .withField(new QFieldMetaData("timestamp", QFieldType.DATE_TIME)) // adding this for GC tests, so we can set a date-time (since CD & MD are owned by system)
+         ;
    }
 
 
@@ -602,6 +604,7 @@ public class TestUtils
          .withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
          .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
          .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
+         .withField(new QFieldMetaData("timestamp", QFieldType.DATE_TIME)) // adding this for GC tests, so we can set a date-time (since CD & MD are owned by system)
          .withField(new QFieldMetaData("orderId", QFieldType.INTEGER))
          .withField(new QFieldMetaData("lineNumber", QFieldType.STRING))
          .withField(new QFieldMetaData("sku", QFieldType.STRING).withLabel("SKU"))

qqq-backend-module-api/src/main/java/com/kingsrook/qqq/backend/module/api/actions/BaseAPIActionUtil.java

@@ -189,12 +189,6 @@ public class BaseAPIActionUtil
       InsertOutput insertOutput = new InsertOutput();
       insertOutput.setRecords(new ArrayList<>());
 
-      if(CollectionUtils.nullSafeIsEmpty(insertInput.getRecords()))
-      {
-         LOG.debug("Insert request called with 0 records.  Returning with no-op");
-         return (insertOutput);
-      }
-
       try
       {
          // todo - supports bulk post?

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/base/model/metadata/FilesystemTableMetaDataBuilder.java

@@ -0,0 +1,210 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.base.model.metadata;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.module.filesystem.local.FilesystemBackendModule;
+import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemTableBackendDetails;
+import com.kingsrook.qqq.backend.module.filesystem.s3.S3BackendModule;
+import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3TableBackendDetails;
+
+
+/*******************************************************************************
+ ** Builder class to create standard style QTableMetaData for tables in filesystem
+ ** modules (avoid some boilerplate).
+ **
+ ** e.g., lets us create a file-based table like so:
+ <pre>
+ QTableMetaData table = new FilesystemTableMetaDataBuilder()
+ .withName("myTableName")
+ .withBackend(qInstance.getBackend("myBackendName"))
+ .withGlob("*.csv")
+ .withBasePath("/")
+ .buildStandardCardinalityOneTable();
+ </pre>
+ *******************************************************************************/
+public class FilesystemTableMetaDataBuilder
+{
+   private String           name;
+   private QBackendMetaData backend;
+   private String           basePath;
+   private String           glob;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("checkstyle:Indentation")
+   public QTableMetaData buildStandardCardinalityOneTable()
+   {
+      AbstractFilesystemTableBackendDetails tableBackendDetails = switch(backend.getBackendType())
+      {
+         case S3BackendModule.BACKEND_TYPE -> new S3TableBackendDetails();
+         case FilesystemBackendModule.BACKEND_TYPE -> new FilesystemTableBackendDetails();
+         default -> throw new IllegalStateException("Unexpected value: " + backend.getBackendType());
+      };
+
+      return new QTableMetaData()
+         .withName(name)
+         .withIsHidden(true)
+         .withBackendName(backend.getName())
+         .withPrimaryKeyField("fileName")
+         .withField(new QFieldMetaData("fileName", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("contents", QFieldType.STRING))
+         .withBackendDetails(tableBackendDetails
+            .withCardinality(Cardinality.ONE)
+            .withFileNameFieldName("fileName")
+            .withContentsFieldName("contents")
+            .withBasePath(basePath)
+            .withGlob(glob));
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for backend
+    *******************************************************************************/
+   public QBackendMetaData getBackend()
+   {
+      return (this.backend);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for backend
+    *******************************************************************************/
+   public void setBackend(QBackendMetaData backend)
+   {
+      this.backend = backend;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for backend
+    *******************************************************************************/
+   public FilesystemTableMetaDataBuilder withBackend(QBackendMetaData backend)
+   {
+      this.backend = backend;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    *******************************************************************************/
+   public String getName()
+   {
+      return (this.name);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    *******************************************************************************/
+   public void setName(String name)
+   {
+      this.name = name;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for name
+    *******************************************************************************/
+   public FilesystemTableMetaDataBuilder withName(String name)
+   {
+      this.name = name;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for basePath
+    *******************************************************************************/
+   public String getBasePath()
+   {
+      return (this.basePath);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for basePath
+    *******************************************************************************/
+   public void setBasePath(String basePath)
+   {
+      this.basePath = basePath;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for basePath
+    *******************************************************************************/
+   public FilesystemTableMetaDataBuilder withBasePath(String basePath)
+   {
+      this.basePath = basePath;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for glob
+    *******************************************************************************/
+   public String getGlob()
+   {
+      return (this.glob);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for glob
+    *******************************************************************************/
+   public void setGlob(String glob)
+   {
+      this.glob = glob;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for glob
+    *******************************************************************************/
+   public FilesystemTableMetaDataBuilder withGlob(String glob)
+   {
+      this.glob = glob;
+      return (this);
+   }
+
+}

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/local/FilesystemBackendModule.java

@@ -51,6 +51,7 @@ public class FilesystemBackendModule implements QBackendModuleInterface, Filesys
 {
    private static final QLogger LOG = QLogger.getLogger(FilesystemBackendModule.class);
 
+   public static final String BACKEND_TYPE = "filesystem";
 
 
    /*******************************************************************************
@@ -71,7 +72,7 @@ public class FilesystemBackendModule implements QBackendModuleInterface, Filesys
    @Override
    public String getBackendType()
    {
-      return ("filesystem");
+      return (BACKEND_TYPE);
    }
 
 

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/processes/implementations/filesystem/importer/FilesystemImporterMetaDataTemplate.java

@@ -0,0 +1,514 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer;
+
+
+import java.util.List;
+import java.util.function.Consumer;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
+import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.QTableAutomationDetails;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TableAutomationAction;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TriggerEvent;
+
+
+/*******************************************************************************
+ ** Class to serve as a template for producing an instance of a process & tables
+ ** that provide the QQQ service to manage importing files (e.g., partner feeds on S3).
+ **
+ ** The template contains the following components:
+ ** - A process that loads files from a source-table (e.g., of filesystem, cardinality=ONE)
+ **   and stores them in the following tables:
+ ** - {baseName}importFile table - simple header for imported files.
+ ** - {baseName}importRecord table - a record foreach record in an imported file.
+ ** - PVS for the importFile table
+ ** - Join & Widget (to show importRecords on importFile view screen)
+ **
+ ** Most likely one would add all the meta-data objects in an instance of this
+ ** template, then either use tableAutomations or a basepull process against records
+ ** in the importRecord table, to run through a process (e.g., an AbstractTableSync)
+ ** to result in final values for your business case.
+ **
+ ** A typical usage may look like:
+ **
+ ** <pre>
+ // set up the process that'll be used to import the files.
+ FilesystemImporterProcessMetaDataBuilder importerProcessBuilder = (FilesystemImporterProcessMetaDataBuilder) new FilesystemImporterProcessMetaDataBuilder()
+ .withFileFormat("csv")
+ .withSourceTableName(MyFeedSourceTableMetaDataProducer.NAME)
+ .withRemoveFileAfterImport(true)
+ .withUpdateFileIfNameExists(false)
+ .withName("myFeedImporter")
+ .withSchedule(new QScheduleMetaData().withRepeatSeconds(300));
+
+ FilesystemImporterMetaDataTemplate template = new FilesystemImporterMetaDataTemplate(qInstance, "myFeed", MongoDBMetaDataProducer.NAME, importerProcessBuilder, table ->
+ {
+ // whatever customizations you may need on the tables
+ table.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED));
+ });
+
+ // set up automations on the table
+ template.addAutomationStatusField(template.getImportRecordTable(), getStandardAutomationStatusField().withBackendName("metaData.automationStatus"));
+ template.addStandardPostInsertAutomation(template.getImportRecordTable(), getBasicTableAutomationDetails(), "myFeedTableSyncProcess");
+
+ // finally, add all the meta-data from the template to a QInstance
+ template.addToInstance(qInstance);
+ </pre>
+ **
+ *******************************************************************************/
+public class FilesystemImporterMetaDataTemplate
+{
+   public static final String IMPORT_FILE_TABLE_SUFFIX       = "ImportFile";
+   public static final String IMPORT_RECORD_TABLE_SUFFIX     = "ImportRecord";
+   public static final String IMPORT_FILE_RECORD_JOIN_SUFFIX = "ImportFileImportRecordJoin";
+
+   private QTableMetaData                           importFileTable;
+   private QTableMetaData                           importRecordTable;
+   private QPossibleValueSource                     importFilePVS;
+   private QJoinMetaData                            importFileImportRecordJoin;
+   private QWidgetMetaDataInterface                 importFileImportRecordJoinWidget;
+   private FilesystemImporterProcessMetaDataBuilder importerProcessMetaDataBuilder;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate(QInstance qInstance, String importBaseName, String backendName, FilesystemImporterProcessMetaDataBuilder importerProcessMetaDataBuilder, Consumer<QTableMetaData> tableEnricher)
+   {
+      QBackendMetaData backend = qInstance.getBackend(backendName);
+
+      this.importFileTable = defineTableImportFile(backend, importBaseName);
+      this.importRecordTable = defineTableImportRecord(backend, importBaseName);
+
+      for(QTableMetaData table : List.of(this.importFileTable, this.importRecordTable))
+      {
+         table.setBackendName(backendName);
+         if(tableEnricher != null)
+         {
+            tableEnricher.accept(table);
+         }
+      }
+
+      this.importFilePVS = QPossibleValueSource.newForTable(this.importFileTable.getName());
+
+      this.importFileImportRecordJoin = defineImportFileImportRecordJoin(importBaseName);
+      this.importFileImportRecordJoinWidget = defineImportFileImportRecordChildWidget(this.importFileImportRecordJoin);
+
+      this.importerProcessMetaDataBuilder = importerProcessMetaDataBuilder
+         .withImportFileTable(this.importFileTable.getName())
+         .withImportRecordTable(this.importRecordTable.getName());
+   }
+
+
+
+   /*******************************************************************************
+    ** Set up importRecord table being built by this template to hve an automation-
+    ** status field on it, and an automation details object attached to it.
+    *******************************************************************************/
+   public void addImportRecordAutomations(QFieldMetaData automationStatusField, QTableAutomationDetails automationDetails)
+   {
+      getImportRecordTable().addField(automationStatusField);
+      getImportRecordTable().getSections().get(1).getFieldNames().add(0, automationStatusField.getName());
+      getImportRecordTable().withAutomationDetails(automationDetails);
+   }
+
+
+
+   /*******************************************************************************
+    ** Add 1 process as a post-insert automation-action on this template's importRecord
+    ** table.
+    **
+    ** The automation action is returned - which you may want for changing things, e.g.,
+    ** its priority (e.g., addImportRecordPostInsertAutomationAction(...).withPriority(1);
+    *******************************************************************************/
+   public TableAutomationAction addImportRecordPostInsertAutomationAction(String processName)
+   {
+      if(getImportRecordTable().getAutomationDetails() == null)
+      {
+         throw (new IllegalStateException(getImportRecordTable().getName() + " does not have automationDetails - do you need to call addAutomations first?"));
+      }
+
+      TableAutomationAction action = new TableAutomationAction()
+         .withName(processName)
+         .withTriggerEvent(TriggerEvent.POST_INSERT)
+         .withProcessName(processName);
+
+      getImportRecordTable().getAutomationDetails().withAction(action);
+
+      return (action);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QWidgetMetaDataInterface defineImportFileImportRecordChildWidget(QJoinMetaData join)
+   {
+      return ChildRecordListRenderer.widgetMetaDataBuilder(join)
+         .withName(join.getName())
+         .withLabel("Import Records")
+         .withMaxRows(100)
+         .withCanAddChildRecord(false)
+         .getWidgetMetaData();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QJoinMetaData defineImportFileImportRecordJoin(String importBaseName)
+   {
+      return new QJoinMetaData()
+         .withLeftTable(importBaseName + IMPORT_FILE_TABLE_SUFFIX)
+         .withRightTable(importBaseName + IMPORT_RECORD_TABLE_SUFFIX)
+         .withName(importBaseName + IMPORT_FILE_RECORD_JOIN_SUFFIX)
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn("id", "importFileId"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QTableMetaData defineTableImportFile(QBackendMetaData backend, String importBaseName)
+   {
+      QFieldType idType = getIdFieldType(backend);
+
+      QTableMetaData qTableMetaData = new QTableMetaData()
+         .withName(importBaseName + IMPORT_FILE_TABLE_SUFFIX)
+         .withIcon(new QIcon().withName("upload_file"))
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("sourceFileName")
+         .withPrimaryKeyField("id")
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.RECORD))
+
+         .withField(new QFieldMetaData("id", idType).withIsEditable(false).withBackendName(getIdFieldBackendName(backend)))
+         .withField(new QFieldMetaData("sourceFileName", QFieldType.STRING))
+         .withField(new QFieldMetaData("archivedPath", QFieldType.STRING))
+         .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
+         .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
+
+         .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1, List.of("id", "sourceFileName", "archivedPath")))
+         .withSection(new QFieldSection("records", new QIcon().withName("power_input"), Tier.T2).withWidgetName(importBaseName + IMPORT_FILE_RECORD_JOIN_SUFFIX))
+         .withSection(new QFieldSection("dates", new QIcon().withName("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")))
+
+         .withAssociation(new Association().withName("importRecords").withJoinName(importBaseName + IMPORT_FILE_RECORD_JOIN_SUFFIX).withAssociatedTableName(importBaseName + IMPORT_RECORD_TABLE_SUFFIX));
+
+      return (qTableMetaData);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QFieldType getIdFieldType(QBackendMetaData backend)
+   {
+      QFieldType idType = QFieldType.INTEGER;
+      if("mongodb".equals(backend.getBackendType()))
+      {
+         idType = QFieldType.STRING;
+      }
+      return idType;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public String getIdFieldBackendName(QBackendMetaData backend)
+   {
+      if("mongodb".equals(backend.getBackendType()))
+      {
+         return ("_id");
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QTableMetaData defineTableImportRecord(QBackendMetaData backend, String importBaseName)
+   {
+      QFieldType idType = getIdFieldType(backend);
+
+      QTableMetaData qTableMetaData = new QTableMetaData()
+         .withName(importBaseName + IMPORT_RECORD_TABLE_SUFFIX)
+         .withIcon(new QIcon().withName("power_input"))
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("importFileId", "recordNo")
+         .withPrimaryKeyField("id")
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.RECORD))
+         .withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(ImportRecordPostQueryCustomizer.class))
+
+         .withField(new QFieldMetaData("id", idType).withIsEditable(false).withBackendName(getIdFieldBackendName(backend)))
+
+         .withField(new QFieldMetaData("importFileId", idType).withBackendName("metaData.importFileId")
+            .withPossibleValueSourceName(importBaseName + IMPORT_FILE_TABLE_SUFFIX))
+         .withField(new QFieldMetaData("recordNo", QFieldType.INTEGER).withBackendName("metaData.recordNo"))
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // so, we'll use this field as a "virtual" field, e.g., populated with JSON in table post-query customizer, with all un-structured values //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         .withField(new QFieldMetaData("values", QFieldType.TEXT)
+            .withIsEditable(false)
+            .withFieldAdornment(new FieldAdornment(AdornmentType.CODE_EDITOR)
+               .withValue(AdornmentType.CodeEditorValues.languageMode("json"))))
+
+         .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("metaData.createDate").withIsEditable(false))
+         .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBackendName("metaData.modifyDate").withIsEditable(false))
+
+         .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1, List.of("id", "importFileId", "recordNo")))
+         .withSection(new QFieldSection("data", new QIcon().withName("text_snippet"), Tier.T2, List.of("values")))
+         .withSection(new QFieldSection("dates", new QIcon().withName("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+
+      return (qTableMetaData);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void addToInstance(QInstance instance)
+   {
+      instance.add(importFileTable);
+      instance.add(importRecordTable);
+      instance.add(importFilePVS);
+      instance.add(importFileImportRecordJoin);
+      instance.add(importFileImportRecordJoinWidget);
+      instance.add(importerProcessMetaDataBuilder.getProcessMetaData());
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for importFileTable
+    *******************************************************************************/
+   public QTableMetaData getImportFileTable()
+   {
+      return (this.importFileTable);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for importFileTable
+    *******************************************************************************/
+   public void setImportFileTable(QTableMetaData importFileTable)
+   {
+      this.importFileTable = importFileTable;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for importFileTable
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate withImportFileTable(QTableMetaData importFileTable)
+   {
+      this.importFileTable = importFileTable;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for importRecordTable
+    *******************************************************************************/
+   public QTableMetaData getImportRecordTable()
+   {
+      return (this.importRecordTable);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for importRecordTable
+    *******************************************************************************/
+   public void setImportRecordTable(QTableMetaData importRecordTable)
+   {
+      this.importRecordTable = importRecordTable;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for importRecordTable
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate withImportRecordTable(QTableMetaData importRecordTable)
+   {
+      this.importRecordTable = importRecordTable;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for importFilePVS
+    *******************************************************************************/
+   public QPossibleValueSource getImportFilePVS()
+   {
+      return (this.importFilePVS);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for importFilePVS
+    *******************************************************************************/
+   public void setImportFilePVS(QPossibleValueSource importFilePVS)
+   {
+      this.importFilePVS = importFilePVS;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for importFilePVS
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate withImportFilePVS(QPossibleValueSource importFilePVS)
+   {
+      this.importFilePVS = importFilePVS;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for importFileImportRecordJoin
+    *******************************************************************************/
+   public QJoinMetaData getImportFileImportRecordJoin()
+   {
+      return (this.importFileImportRecordJoin);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for importFileImportRecordJoin
+    *******************************************************************************/
+   public void setImportFileImportRecordJoin(QJoinMetaData importFileImportRecordJoin)
+   {
+      this.importFileImportRecordJoin = importFileImportRecordJoin;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for importFileImportRecordJoin
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate withImportFileImportRecordJoin(QJoinMetaData importFileImportRecordJoin)
+   {
+      this.importFileImportRecordJoin = importFileImportRecordJoin;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for importFileImportRecordJoinWidget
+    *******************************************************************************/
+   public QWidgetMetaDataInterface getImportFileImportRecordJoinWidget()
+   {
+      return (this.importFileImportRecordJoinWidget);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for importFileImportRecordJoinWidget
+    *******************************************************************************/
+   public void setImportFileImportRecordJoinWidget(QWidgetMetaDataInterface importFileImportRecordJoinWidget)
+   {
+      this.importFileImportRecordJoinWidget = importFileImportRecordJoinWidget;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for importFileImportRecordJoinWidget
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate withImportFileImportRecordJoinWidget(QWidgetMetaDataInterface importFileImportRecordJoinWidget)
+   {
+      this.importFileImportRecordJoinWidget = importFileImportRecordJoinWidget;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for importerProcessMetaDataBuilder
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder getImporterProcessMetaDataBuilder()
+   {
+      return (this.importerProcessMetaDataBuilder);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for importerProcessMetaDataBuilder
+    *******************************************************************************/
+   public void setImporterProcessMetaDataBuilder(FilesystemImporterProcessMetaDataBuilder importerProcessMetaDataBuilder)
+   {
+      this.importerProcessMetaDataBuilder = importerProcessMetaDataBuilder;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for importerProcessMetaDataBuilder
+    *******************************************************************************/
+   public FilesystemImporterMetaDataTemplate withImporterProcessMetaDataBuilder(FilesystemImporterProcessMetaDataBuilder importerProcessMetaDataBuilder)
+   {
+      this.importerProcessMetaDataBuilder = importerProcessMetaDataBuilder;
+      return (this);
+   }
+
+}

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/processes/implementations/filesystem/importer/FilesystemImporterProcessMetaDataBuilder.java

@@ -0,0 +1,211 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer;
+
+
+import java.io.Serializable;
+import java.util.function.Function;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.AbstractProcessMetaDataBuilder;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+
+
+/*******************************************************************************
+ ** Process MetaData Builder for FilesystemImporter process.
+ ** Meant to be used with (and actually is a parameter to the constructor of)
+ ** {@link FilesystemImporterMetaDataTemplate}
+ *******************************************************************************/
+public class FilesystemImporterProcessMetaDataBuilder extends AbstractProcessMetaDataBuilder
+{
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder()
+   {
+      super(new QProcessMetaData()
+         .addStep(new QBackendStepMetaData()
+            .withName("sync")
+            .withCode(new QCodeReference(FilesystemImporterStep.class))
+            .withInputData(new QFunctionInputMetaData()
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_SOURCE_TABLE, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_FILE_FORMAT, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_IMPORT_FILE_TABLE, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_IMPORT_RECORD_TABLE, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_REMOVE_FILE_AFTER_IMPORT, QFieldType.BOOLEAN).withDefaultValue(true))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_UPDATE_FILE_IF_NAME_EXISTS, QFieldType.BOOLEAN).withDefaultValue(false))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_ARCHIVE_FILE_ENABLED, QFieldType.BOOLEAN).withDefaultValue(false))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_ARCHIVE_TABLE_NAME, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_ARCHIVE_PATH, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_NAME, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_VALUE, QFieldType.STRING))
+
+               //////////////////////////////////////////////////////////////////////////////////////
+               // define a QCodeReference - expected to be of type Function<QRecord, Serializable> //
+               // make sure the QInstanceValidator knows that the QCodeReference should be a       //
+               // Function (not a BackendStep, which is the default for process fields)            //
+               //////////////////////////////////////////////////////////////////////////////////////
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_IMPORT_SECURITY_VALUE_SUPPLIER, QFieldType.STRING))
+               .withField(new QFieldMetaData(FilesystemImporterStep.FIELD_IMPORT_SECURITY_VALUE_SUPPLIER + "_expectedType", QFieldType.STRING)
+                  .withDefaultValue(Function.class.getName()))
+            )));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withSourceTableName(String sourceTableName)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_SOURCE_TABLE, sourceTableName);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withFileFormat(String fileFormat)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_FILE_FORMAT, fileFormat);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withImportFileTable(String importFileTable)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_IMPORT_FILE_TABLE, importFileTable);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withImportRecordTable(String importRecordTable)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_IMPORT_RECORD_TABLE, importRecordTable);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withRemoveFileAfterImport(boolean removeFileAfterImport)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_REMOVE_FILE_AFTER_IMPORT, removeFileAfterImport);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withUpdateFileIfNameExists(boolean updateFileIfNameExists)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_UPDATE_FILE_IF_NAME_EXISTS, updateFileIfNameExists);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withArchiveFileEnabled(boolean archiveFileEnabled)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_ARCHIVE_FILE_ENABLED, archiveFileEnabled);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withArchiveTableName(String archiveTableName)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_ARCHIVE_TABLE_NAME, archiveTableName);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withArchivePath(String archivePath)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_ARCHIVE_PATH, archivePath);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withImportSecurityFieldName(String securityFieldName)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_NAME, securityFieldName);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withImportSecurityFieldValue(Serializable securityFieldValue)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_VALUE, securityFieldValue);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FilesystemImporterProcessMetaDataBuilder withImportSecurityValueSupplierFunction(Class<? extends Function<QRecord, Serializable>> supplierFunction)
+   {
+      setInputFieldDefaultValue(FilesystemImporterStep.FIELD_IMPORT_SECURITY_VALUE_SUPPLIER, new QCodeReference(supplierFunction));
+      return (this);
+   }
+
+}

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/processes/implementations/filesystem/importer/FilesystemImporterStep.java

@@ -0,0 +1,434 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer;
+
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Serializable;
+import java.time.LocalDateTime;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.UUID;
+import java.util.function.Function;
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
+import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.adapters.CsvToQRecordAdapter;
+import com.kingsrook.qqq.backend.core.adapters.JsonToQRecordAdapter;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.module.filesystem.base.FilesystemBackendModuleInterface;
+import com.kingsrook.qqq.backend.module.filesystem.base.actions.AbstractBaseFilesystemAction;
+import com.kingsrook.qqq.backend.module.filesystem.exceptions.FilesystemException;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** BackendStep for FilesystemImporter process
+ **
+ ** Job is to:
+ ** - foreach file in the `source` table (e.g., a ONE-type filesystem table):
+ **   - optionally create an archive/backup copy of the file
+ **   - create a record in the `importFile` table
+ **   - parse the file, creating many records in the `importRecord` table
+ **   - remove the file from the `source` (if so configured (e.g., may turn off for Read-only FS))
+ *******************************************************************************/
+@SuppressWarnings("unchecked")
+public class FilesystemImporterStep implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(FilesystemImporterStep.class);
+
+   public static final String FIELD_SOURCE_TABLE        = "sourceTable";
+   public static final String FIELD_FILE_FORMAT         = "fileFormat";
+   public static final String FIELD_IMPORT_FILE_TABLE   = "importFileTable";
+   public static final String FIELD_IMPORT_RECORD_TABLE = "importRecordTable";
+
+   public static final String FIELD_IMPORT_SECURITY_FIELD_NAME     = "importSecurityFieldName";
+   public static final String FIELD_IMPORT_SECURITY_FIELD_VALUE    = "importSecurityFieldValue";
+   public static final String FIELD_IMPORT_SECURITY_VALUE_SUPPLIER = "importSecurityFieldSupplier";
+
+   public static final String FIELD_ARCHIVE_FILE_ENABLED     = "archiveFileEnabled";
+   public static final String FIELD_ARCHIVE_TABLE_NAME       = "archiveTableName";
+   public static final String FIELD_ARCHIVE_PATH             = "archivePath";
+   public static final String FIELD_REMOVE_FILE_AFTER_IMPORT = "removeFileAfterImport";
+
+   public static final String FIELD_UPDATE_FILE_IF_NAME_EXISTS = "updateFileIfNameExists";
+
+   private Function<QRecord, Serializable> securitySupplier = null;
+
+
+   /*******************************************************************************
+    ** Execute the step - using the request as input, and the result as output.
+    *******************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      ////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // defer to a private method here, so we can add a type-parameter for that method to use              //
+      // would think we could do that here, but get compiler error, since this method comes from base class //
+      ////////////////////////////////////////////////////////////////////////////////////////////////////////
+      doRun(runBackendStepInput, runBackendStepOutput);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private <F> void doRun(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      String  fileFormat             = runBackendStepInput.getValueString(FIELD_FILE_FORMAT);
+      Boolean removeFileAfterImport  = runBackendStepInput.getValueBoolean(FIELD_REMOVE_FILE_AFTER_IMPORT);
+      Boolean updateFileIfNameExists = runBackendStepInput.getValueBoolean(FIELD_UPDATE_FILE_IF_NAME_EXISTS);
+      Boolean archiveFileEnabled     = runBackendStepInput.getValueBoolean(FIELD_ARCHIVE_FILE_ENABLED);
+
+      QTableMetaData sourceTable     = runBackendStepInput.getInstance().getTable(runBackendStepInput.getValueString(FIELD_SOURCE_TABLE));
+      QTableMetaData importFileTable = runBackendStepInput.getInstance().getTable(runBackendStepInput.getValueString(FIELD_IMPORT_FILE_TABLE));
+
+      String missingFieldErrorPrefix = "Process " + runBackendStepInput.getProcessName() + " was misconfigured - missing value in field: ";
+      Objects.requireNonNull(fileFormat, missingFieldErrorPrefix + FIELD_FILE_FORMAT);
+
+      ///////////////////////////////////////////////////////////////////////////////////
+      // list files in the backend system                                              //
+      // todo - can we do this using query action, with this being a "ONE" type table? //
+      ///////////////////////////////////////////////////////////////////////////////////
+      QBackendMetaData                    sourceBackend    = runBackendStepInput.getInstance().getBackendForTable(sourceTable.getName());
+      FilesystemBackendModuleInterface<F> sourceModule     = (FilesystemBackendModuleInterface<F>) new QBackendModuleDispatcher().getQBackendModule(sourceBackend);
+      AbstractBaseFilesystemAction<F>     sourceActionBase = sourceModule.getActionBase();
+      sourceActionBase.preAction(sourceBackend);
+      Map<String, F> sourceFiles = getFileNames(sourceActionBase, sourceTable, sourceBackend);
+
+      if(CollectionUtils.nullSafeIsEmpty(sourceFiles))
+      {
+         LOG.debug("No files found in import filesystem", logPair("sourceTable", sourceTable));
+         return;
+      }
+
+      ////////////////////////////////////////////////////////
+      // look up any existing file records with those names //
+      ////////////////////////////////////////////////////////
+      QueryInput queryInput = new QueryInput();
+      queryInput.setTableName(importFileTable.getName());
+      queryInput.setFilter(new QQueryFilter(new QFilterCriteria("sourceFileName", QCriteriaOperator.IN, sourceFiles.keySet())));
+
+      QueryOutput               queryOutput           = new QueryAction().execute(queryInput);
+      Map<String, Serializable> existingImportedFiles = CollectionUtils.listToMap(queryOutput.getRecords(), r -> r.getValueString("sourceFileName"), r -> r.getValue("id"));
+
+      for(Map.Entry<String, F> sourceEntry : sourceFiles.entrySet())
+      {
+         QBackendTransaction transaction = null;
+         try
+         {
+            String sourceFileName = sourceEntry.getKey();
+
+            /////////////////////////////////////////////////////////
+            // if filename was already imported, decide what to do //
+            /////////////////////////////////////////////////////////
+            boolean      alreadyImported = existingImportedFiles.containsKey(sourceFileName);
+            Serializable idToUpdate      = null;
+            if(alreadyImported)
+            {
+               //////////////////////////////////////////////////////////////////////////////////
+               // todo - would we want to support importing multiple-times the same file name? //
+               // possibly - if so, add it here, presumably w/ another boolean field           //
+               //////////////////////////////////////////////////////////////////////////////////
+               if(updateFileIfNameExists)
+               {
+                  LOG.info("Updating already-imported file", logPair("fileName", sourceFileName), logPair("id", idToUpdate));
+                  idToUpdate = existingImportedFiles.get(sourceFileName);
+               }
+               else
+               {
+                  LOG.debug("Skipping already-imported file", logPair("fileName", sourceFileName));
+                  removeSourceFileIfSoConfigured(removeFileAfterImport, sourceActionBase, sourceTable, sourceBackend, sourceFileName);
+                  continue;
+               }
+            }
+
+            ///////////////////////////////////
+            // read the file as input stream //
+            ///////////////////////////////////
+            try(InputStream inputStream = sourceActionBase.readFile(sourceEntry.getValue()))
+            {
+               byte[] bytes = inputStream.readAllBytes();
+
+               //////////////////////////////////////
+               // archive the file, if so directed //
+               //////////////////////////////////////
+               String archivedPath = null;
+               if(archiveFileEnabled)
+               {
+                  archivedPath = archiveFile(runBackendStepInput, sourceFileName, bytes);
+               }
+
+               /////////////////////////////////
+               // build record for importFile //
+               /////////////////////////////////
+               LOG.info("Syncing file [" + sourceFileName + "]");
+               QRecord importFileRecord = new QRecord()
+                  .withValue("id", idToUpdate)
+                  .withValue("sourceFileName", sourceFileName)
+                  .withValue("archivedPath", archivedPath);
+
+               addSecurityValue(runBackendStepInput, importFileRecord);
+
+               //////////////////////////////////////
+               // build child importRecord records //
+               //////////////////////////////////////
+               String content = new String(bytes);
+               importFileRecord.withAssociatedRecords("importRecords", parseFileIntoRecords(runBackendStepInput, content));
+
+               ///////////////////////////////////////////////////////////////////
+               // insert the file & records (records as association under file) //
+               ///////////////////////////////////////////////////////////////////
+               InsertAction insertAction = new InsertAction();
+               InsertInput  insertInput  = new InsertInput();
+               insertInput.setTableName(importFileTable.getName());
+               insertInput.setRecords(List.of(importFileRecord));
+
+               transaction = QBackendTransaction.openFor(insertInput);
+               insertInput.setTransaction(transaction);
+
+               InsertOutput insertOutput = insertAction.execute(insertInput);
+
+               LOG.info("Inserted insertFile & records", logPair("id", insertOutput.getRecords().get(0).getValue("id")));
+
+               transaction.commit();
+            }
+
+            ///////////////////////////////////////////////////////////////////////////////////////////////
+            // after the records are built, we can delete the file                                       //
+            // if we are interrupted between the commit & the delete, then the file will be found again, //
+            // and we'll either skip it or do an update, based on FIELD_UPDATE_FILE_IF_NAME_EXISTS flag  //
+            ///////////////////////////////////////////////////////////////////////////////////////////////
+            removeSourceFileIfSoConfigured(removeFileAfterImport, sourceActionBase, sourceTable, sourceBackend, sourceFileName);
+         }
+         catch(Exception e)
+         {
+            LOG.error("Error processing file: " + sourceEntry, e);
+            if(transaction != null)
+            {
+               transaction.rollback();
+            }
+         }
+         finally
+         {
+            if(transaction != null)
+            {
+               transaction.close();
+            }
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** if the process is configured w/ a security field & value, set it on the import
+    ** File & Record records.
+    *******************************************************************************/
+   private void addSecurityValue(RunBackendStepInput runBackendStepInput, QRecord record)
+   {
+      String securityField = runBackendStepInput.getValueString(FIELD_IMPORT_SECURITY_FIELD_NAME);
+
+      /////////////////////////////////////////////////////////////
+      // if we're using a security supplier function, load it up //
+      /////////////////////////////////////////////////////////////
+      QCodeReference securitySupplierReference = (QCodeReference) runBackendStepInput.getValue(FIELD_IMPORT_SECURITY_VALUE_SUPPLIER);
+      try
+      {
+         if(securitySupplierReference != null && securitySupplier == null)
+         {
+            securitySupplier = QCodeLoader.getAdHoc(Function.class, securitySupplierReference);
+         }
+      }
+      catch(Exception e)
+      {
+         throw (new QRuntimeException("Error loading Security Supplier Function from QCodeReference [" + securitySupplierReference + "]", e));
+      }
+
+      ///////////////////////////////////////////////////////////////////////////////////////
+      // either get the security value from the supplier, or the field value field's value //
+      ///////////////////////////////////////////////////////////////////////////////////////
+      Serializable securityValue = securitySupplier != null
+         ? securitySupplier.apply(record)
+         : runBackendStepInput.getValue(FIELD_IMPORT_SECURITY_FIELD_VALUE);
+
+      ////////////////////////////////////////////////////////////////////
+      // if we have a field name and a value, then add it to the record //
+      ////////////////////////////////////////////////////////////////////
+      if(StringUtils.hasContent(securityField) && securityValue != null)
+      {
+         record.setValue(securityField, securityValue);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static <F> void removeSourceFileIfSoConfigured(Boolean removeFileAfterImport, AbstractBaseFilesystemAction<F> sourceActionBase, QTableMetaData sourceTable, QBackendMetaData sourceBackend, String sourceFileName) throws FilesystemException
+   {
+      if(removeFileAfterImport)
+      {
+         String fullBasePath = sourceActionBase.getFullBasePath(sourceTable, sourceBackend);
+         sourceActionBase.deleteFile(QContext.getQInstance(), sourceTable, fullBasePath + "/" + sourceFileName);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private String archiveFile(RunBackendStepInput runBackendStepInput, String sourceFileName, byte[] bytes) throws QException, IOException
+   {
+      String         archiveTableName = runBackendStepInput.getValueString(FIELD_ARCHIVE_TABLE_NAME);
+      QTableMetaData archiveTable;
+      try
+      {
+         archiveTable = runBackendStepInput.getInstance().getTable(archiveTableName);
+      }
+      catch(Exception e)
+      {
+         throw (new QException("Error getting archive table [" + archiveTableName + "]", e));
+      }
+
+      String archivePath = Objects.requireNonNullElse(runBackendStepInput.getValueString(FIELD_ARCHIVE_PATH), "");
+
+      QBackendMetaData                    archiveBackend    = runBackendStepInput.getInstance().getBackendForTable(archiveTable.getName());
+      FilesystemBackendModuleInterface<?> archiveModule     = (FilesystemBackendModuleInterface<?>) new QBackendModuleDispatcher().getQBackendModule(archiveBackend);
+      AbstractBaseFilesystemAction<?>     archiveActionBase = archiveModule.getActionBase();
+      archiveActionBase.preAction(archiveBackend);
+
+      LocalDateTime now = LocalDateTime.now();
+      String path = archiveActionBase.getFullBasePath(archiveTable, archiveBackend)
+         + File.separator + archivePath
+         + File.separator + now.getYear()
+         + File.separator + now.getMonth()
+         + File.separator + UUID.randomUUID()
+         + "-" + sourceFileName.replaceAll(".*" + File.separator, "");
+      path = AbstractBaseFilesystemAction.stripDuplicatedSlashes(path);
+
+      LOG.info("Archiving file", logPair("path", path));
+      archiveActionBase.writeFile(archiveBackend, path, bytes);
+
+      return (path);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("checkstyle:Indentation")
+   List<QRecord> parseFileIntoRecords(RunBackendStepInput runBackendStepInput, String content) throws QException
+   {
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // first, parse the content into records, w/ unknown field names - just whatever is in the CSV or JSON //
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////
+      String fileFormat = runBackendStepInput.getValueString(FIELD_FILE_FORMAT);
+
+      List<QRecord> contentRecords = switch(fileFormat.toLowerCase())
+      {
+         case "csv" ->
+         {
+            CsvToQRecordAdapter csvToQRecordAdapter = new CsvToQRecordAdapter();
+            csvToQRecordAdapter.buildRecordsFromCsv(new CsvToQRecordAdapter.InputWrapper()
+               .withCsv(content)
+               .withCaseSensitiveHeaders(true)
+               .withCsvHeadersAsFieldNames(true)
+            );
+            yield (csvToQRecordAdapter.getRecordList());
+         }
+
+         case "json" -> new JsonToQRecordAdapter().buildRecordsFromJson(content, null, null);
+
+         default -> throw (new QException("Unexpected file format: " + fileFormat));
+      };
+
+      /////////////////////////////////////////////////////////////////////////////
+      // now, add some fields that we know about to those records, for returning //
+      /////////////////////////////////////////////////////////////////////////////
+      List<QRecord> importRecordList = new ArrayList<>();
+      int           recordNo         = 1;
+      for(QRecord record : contentRecords)
+      {
+         record.setValue("recordNo", recordNo++);
+         addSecurityValue(runBackendStepInput, record);
+
+         importRecordList.add(record);
+      }
+
+      return (importRecordList);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private <F> Map<String, F> getFileNames(AbstractBaseFilesystemAction<F> actionBase, QTableMetaData table, QBackendMetaData backend) throws QException
+   {
+      List<F> files = actionBase.listFiles(table, backend);
+
+      /////////////////////////////////////////////////////
+      // use a tree map, so files will be sorted by name //
+      /////////////////////////////////////////////////////
+      Map<String, F> rs = new TreeMap<>();
+
+      for(F file : files)
+      {
+         String fileName = actionBase.stripBackendAndTableBasePathsFromFileName(actionBase.getFullPathForFile(file), backend, table);
+         rs.put(fileName, file);
+      }
+
+      return (rs);
+   }
+
+}

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/processes/implementations/filesystem/importer/ImportRecordPostQueryCustomizer.java

@@ -0,0 +1,82 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer;
+
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.customizers.AbstractPostQueryCustomizer;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ ** combine all unstructured fields of the record into a JSON blob in the "values" field.
+ *******************************************************************************/
+public class ImportRecordPostQueryCustomizer extends AbstractPostQueryCustomizer
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public List<QRecord> apply(List<QRecord> records)
+   {
+      if(CollectionUtils.nullSafeHasContents(records))
+      {
+         QTableMetaData table = null;
+         if(StringUtils.hasContent(records.get(0).getTableName()))
+         {
+            table = QContext.getQInstance().getTable(records.get(0).getTableName());
+         }
+
+         for(QRecord record : records)
+         {
+            Map<String, Serializable> values = record.getValues();
+
+            if(table != null)
+            {
+               ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+               // remove known values from a clone of the values map - then only put the un-structured values in a JSON document in the values field //
+               ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+               values = new HashMap<>(values);
+               for(String fieldName : table.getFields().keySet())
+               {
+                  values.remove(fieldName);
+               }
+            }
+
+            String valuesJson = JsonUtils.toJson(values);
+            record.setValue("values", valuesJson);
+         }
+      }
+
+      return (records);
+   }
+
+}

qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/s3/S3BackendModule.java

@@ -46,6 +46,7 @@ import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3TableBack
  *******************************************************************************/
 public class S3BackendModule implements QBackendModuleInterface, FilesystemBackendModuleInterface
 {
+   public static final String BACKEND_TYPE = "s3";
 
 
    /*******************************************************************************
@@ -66,7 +67,7 @@ public class S3BackendModule implements QBackendModuleInterface, FilesystemBacke
    @Override
    public String getBackendType()
    {
-      return ("s3");
+      return (BACKEND_TYPE);
    }
 
 

qqq-backend-module-filesystem/src/test/java/com/kingsrook/qqq/backend/module/filesystem/TestUtils.java

@@ -29,6 +29,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
 import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
@@ -38,12 +39,15 @@ import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.modules.authentication.implementations.MockAuthenticationModule;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamed.StreamedETLProcess;
 import com.kingsrook.qqq.backend.module.filesystem.base.model.metadata.Cardinality;
 import com.kingsrook.qqq.backend.module.filesystem.base.model.metadata.RecordFormat;
 import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemBackendMetaData;
 import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemTableBackendDetails;
 import com.kingsrook.qqq.backend.module.filesystem.processes.implementations.etl.streamed.StreamedETLFilesystemBackendStep;
+import com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer.FilesystemImporterMetaDataTemplate;
+import com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer.FilesystemImporterProcessMetaDataBuilder;
 import com.kingsrook.qqq.backend.module.filesystem.s3.BaseS3Test;
 import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3BackendMetaData;
 import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3TableBackendDetails;
@@ -59,16 +63,19 @@ public class TestUtils
    public static final String BACKEND_NAME_S3             = "s3";
    public static final String BACKEND_NAME_S3_SANS_PREFIX = "s3sansPrefix";
    public static final String BACKEND_NAME_MOCK           = "mock";
+   public static final String BACKEND_NAME_MEMORY = "memory";
 
    public static final String TABLE_NAME_PERSON_LOCAL_FS_JSON = "person-local-json";
    public static final String TABLE_NAME_PERSON_LOCAL_FS_CSV  = "person-local-csv";
    public static final String TABLE_NAME_BLOB_LOCAL_FS        = "local-blob";
+   public static final String TABLE_NAME_ARCHIVE_LOCAL_FS    = "local-archive";
    public static final String TABLE_NAME_PERSON_S3            = "person-s3";
    public static final String TABLE_NAME_BLOB_S3              = "s3-blob";
    public static final String TABLE_NAME_PERSON_MOCK          = "person-mock";
    public static final String TABLE_NAME_BLOB_S3_SANS_PREFIX = "s3-blob-sans-prefix";
 
    public static final String PROCESS_NAME_STREAMED_ETL                   = "etl.streamed";
+   public static final String LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME = "localPersonCsvFileImporter";
 
    ///////////////////////////////////////////////////////////////////
    // shouldn't be accessed directly, as we append a counter to it. //
@@ -136,20 +143,44 @@ public class TestUtils
       qInstance.addTable(defineLocalFilesystemJSONPersonTable());
       qInstance.addTable(defineLocalFilesystemCSVPersonTable());
       qInstance.addTable(defineLocalFilesystemBlobTable());
+      qInstance.addTable(defineLocalFilesystemArchiveTable());
       qInstance.addBackend(defineS3Backend());
       qInstance.addBackend(defineS3BackendSansPrefix());
       qInstance.addTable(defineS3CSVPersonTable());
       qInstance.addTable(defineS3BlobTable());
       qInstance.addTable(defineS3BlobSansPrefixTable());
       qInstance.addBackend(defineMockBackend());
+      qInstance.addBackend(defineMemoryBackend());
       qInstance.addTable(defineMockPersonTable());
       qInstance.addProcess(defineStreamedLocalCsvToMockETLProcess());
 
+      definePersonCsvImporter(qInstance);
+
       return (qInstance);
    }
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void definePersonCsvImporter(QInstance qInstance)
+   {
+      String importBaseName = "personImporter";
+      FilesystemImporterProcessMetaDataBuilder filesystemImporterProcessMetaDataBuilder = (FilesystemImporterProcessMetaDataBuilder) new FilesystemImporterProcessMetaDataBuilder()
+         .withSourceTableName(TABLE_NAME_PERSON_LOCAL_FS_CSV)
+         .withFileFormat("csv")
+         .withArchiveFileEnabled(true)
+         .withArchiveTableName(TABLE_NAME_ARCHIVE_LOCAL_FS)
+         .withArchivePath("archive-of/personImporterFiles")
+         .withName(LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+
+      FilesystemImporterMetaDataTemplate filesystemImporterMetaDataTemplate = new FilesystemImporterMetaDataTemplate(qInstance, importBaseName, BACKEND_NAME_MEMORY, filesystemImporterProcessMetaDataBuilder, table -> table.withAuditRules(QAuditRules.defaultInstanceLevelNone()));
+      filesystemImporterMetaDataTemplate.addToInstance(qInstance);
+   }
+
+
+
    /*******************************************************************************
     ** Define the authentication used in standard tests - using 'mock' type.
     **
@@ -257,6 +288,28 @@ public class TestUtils
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QTableMetaData defineLocalFilesystemArchiveTable()
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME_ARCHIVE_LOCAL_FS)
+         .withLabel("Archive")
+         .withBackendName(defineLocalFilesystemBackend().getName())
+         .withPrimaryKeyField("fileName")
+         .withField(new QFieldMetaData("fileName", QFieldType.STRING))
+         .withField(new QFieldMetaData("contents", QFieldType.BLOB))
+         .withBackendDetails(new FilesystemTableBackendDetails()
+            .withBasePath("archive")
+            .withCardinality(Cardinality.ONE)
+            .withFileNameFieldName("fileName")
+            .withContentsFieldName("contents")
+         );
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -356,6 +409,18 @@ public class TestUtils
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QBackendMetaData defineMemoryBackend()
+   {
+      return (new QBackendMetaData()
+         .withBackendType(MemoryBackendModule.class)
+         .withName(BACKEND_NAME_MEMORY));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-module-filesystem/src/test/java/com/kingsrook/qqq/backend/module/filesystem/local/actions/FilesystemActionTest.java

@@ -49,7 +49,7 @@ public class FilesystemActionTest extends BaseTest
     **
     *******************************************************************************/
    @BeforeEach
-   public void beforeEach() throws Exception
+   public void filesystemBaseBeforeEach() throws Exception
    {
       primeFilesystem();
       QContext.init(TestUtils.defineInstance(), new QSession());
@@ -61,7 +61,7 @@ public class FilesystemActionTest extends BaseTest
     **
     *******************************************************************************/
    @AfterEach
-   public void afterEach() throws Exception
+   public void filesystemBaseAfterEach() throws Exception
    {
       cleanFilesystem();
    }
@@ -130,7 +130,7 @@ public class FilesystemActionTest extends BaseTest
    /*******************************************************************************
     ** Write some data files into the directory for the filesystem module.
     *******************************************************************************/
-   private void writePersonCSVFiles(File baseDirectory) throws IOException
+   protected void writePersonCSVFiles(File baseDirectory) throws IOException
    {
       String fullPath = baseDirectory.getAbsolutePath();
       if(TestUtils.defineLocalFilesystemCSVPersonTable().getBackendDetails() instanceof FilesystemTableBackendDetails details)

qqq-backend-module-filesystem/src/test/java/com/kingsrook/qqq/backend/module/filesystem/processes/implementations/filesystem/importer/FilesystemImporterStepTest.java

@@ -0,0 +1,321 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer;
+
+
+import java.io.File;
+import java.io.Serializable;
+import java.time.LocalDateTime;
+import java.util.function.Function;
+import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
+import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
+import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import com.kingsrook.qqq.backend.module.filesystem.TestUtils;
+import com.kingsrook.qqq.backend.module.filesystem.local.actions.FilesystemActionTest;
+import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemBackendMetaData;
+import org.json.JSONObject;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.fail;
+
+
+/*******************************************************************************
+ ** Unit test for FilesystemImporterStep 
+ *******************************************************************************/
+class FilesystemImporterStepTest extends FilesystemActionTest
+{
+
+   //////////////////////////////////////////////////////////////////////////
+   // note - we take advantage of the @BeforeEach and @AfterEach to set up //
+   // and clean up files on disk for this test.                            //
+   //////////////////////////////////////////////////////////////////////////
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterEach
+   public void afterEach() throws Exception
+   {
+      MemoryRecordStore.getInstance().reset();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      /////////////////////////////////////////////////////
+      // make sure we see 2 source files before we begin //
+      /////////////////////////////////////////////////////
+      FilesystemBackendMetaData backend   = (FilesystemBackendMetaData) QContext.getQInstance().getBackend(TestUtils.BACKEND_NAME_LOCAL_FS);
+      String                    basePath  = backend.getBasePath();
+      File                      sourceDir = new File(basePath + "/persons-csv/");
+      assertEquals(2, listOrFail(sourceDir).length);
+
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      String importBaseName = "personImporter";
+      assertEquals(2, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX)).getCount());
+      assertEquals(5, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX)).getCount());
+
+      QRecord record = new GetAction().executeForRecord(new GetInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX).withPrimaryKey(1));
+      assertEquals(1, record.getValue("importFileId"));
+      assertEquals("John", record.getValue("firstName"));
+      assertThat(record.getValue("values")).isInstanceOf(String.class);
+      JSONObject values = new JSONObject(record.getValueString("values"));
+      assertEquals("John", values.get("firstName"));
+
+      ///////////////////////////////////////////
+      // make sure 2 archive files got created //
+      ///////////////////////////////////////////
+      LocalDateTime now = LocalDateTime.now();
+      assertEquals(2, listOrFail(new File(basePath + "/archive/archive-of/personImporterFiles/" + now.getYear() + "/" + now.getMonth())).length);
+
+      ////////////////////////////////////////////
+      // make sure the source files got deleted //
+      ////////////////////////////////////////////
+      assertEquals(0, listOrFail(sourceDir).length);
+   }
+
+
+
+   /*******************************************************************************
+    ** do a listFiles, but fail properly if it returns null (so IJ won't warn all the time)
+    *******************************************************************************/
+   private static File[] listOrFail(File dir)
+   {
+      File[] files = dir.listFiles();
+      if(files == null)
+      {
+         fail("Null result when listing directory: " + dir);
+      }
+      return (files);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testJSON() throws QException
+   {
+      ////////////////////////////////////////////////////////////////////
+      // adjust the process to use the JSON file table, and JSON format //
+      ////////////////////////////////////////////////////////////////////
+      QProcessMetaData process = QContext.getQInstance().getProcess(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      process.getInputFields().stream().filter(f -> f.getName().equals(FilesystemImporterStep.FIELD_SOURCE_TABLE)).findFirst().get().setDefaultValue(TestUtils.TABLE_NAME_PERSON_LOCAL_FS_JSON);
+      process.getInputFields().stream().filter(f -> f.getName().equals(FilesystemImporterStep.FIELD_FILE_FORMAT)).findFirst().get().setDefaultValue("json");
+
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      String importBaseName = "personImporter";
+      assertEquals(2, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX)).getCount());
+      assertEquals(3, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX)).getCount());
+
+      QRecord record = new GetAction().executeForRecord(new GetInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX).withPrimaryKey(1));
+      assertEquals(1, record.getValue("importFileId"));
+      assertEquals("John", record.getValue("firstName"));
+      assertThat(record.getValue("values")).isInstanceOf(String.class);
+      JSONObject values = new JSONObject(record.getValueString("values"));
+      assertEquals("John", values.get("firstName"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testNoFilesFound() throws Exception
+   {
+      cleanFilesystem();
+
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      String importBaseName = "personImporter";
+      assertEquals(0, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX)).getCount());
+      assertEquals(0, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX)).getCount());
+   }
+
+   // todo - updates?
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testDuplicateFileNameNonUpdate() throws Exception
+   {
+      FilesystemBackendMetaData backend   = (FilesystemBackendMetaData) QContext.getQInstance().getBackend(TestUtils.BACKEND_NAME_LOCAL_FS);
+      String                    basePath  = backend.getBasePath();
+      File                      sourceDir = new File(basePath + "/persons-csv/");
+
+      /////////////////////////////////////////////////////////////////
+      // run the process once - assert how many records got inserted //
+      /////////////////////////////////////////////////////////////////
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      String importBaseName = "personImporter";
+      assertEquals(2, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX)).getCount());
+      assertEquals(5, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX)).getCount());
+
+      ///////////////////////////////////////////////////////
+      // put the source files back - assert they are there //
+      ///////////////////////////////////////////////////////
+      writePersonCSVFiles(new File(basePath));
+      assertEquals(2, listOrFail(sourceDir).length);
+
+      ////////////////////////
+      // re-run the process //
+      ////////////////////////
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      ////////////////////////////////////////
+      // make sure no new records are built //
+      ////////////////////////////////////////
+      assertEquals(2, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX)).getCount());
+      assertEquals(5, new CountAction().execute(new CountInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX)).getCount());
+
+      /////////////////////////////////////////////////
+      // make sure no new archive files were created //
+      /////////////////////////////////////////////////
+      LocalDateTime now = LocalDateTime.now();
+      assertEquals(2, listOrFail(new File(basePath + "/archive/archive-of/personImporterFiles/" + now.getYear() + "/" + now.getMonth())).length);
+
+      ////////////////////////////////////////////
+      // make sure the source files got deleted //
+      ////////////////////////////////////////////
+      assertEquals(0, listOrFail(sourceDir).length);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSecurityKey() throws QException
+   {
+      //////////////////////////////////////////////
+      // Add a security name/value to our process //
+      //////////////////////////////////////////////
+      QProcessMetaData process = QContext.getQInstance().getProcess(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      process.getInputFields().stream().filter(f -> f.getName().equals(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_NAME)).findFirst().get().setDefaultValue("customerId");
+      process.getInputFields().stream().filter(f -> f.getName().equals(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_VALUE)).findFirst().get().setDefaultValue(47);
+
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      ////////////////////////////////////////////////////////////////////////////////////////////
+      // assert the security field gets its value on both the importFile & importRecord records //
+      ////////////////////////////////////////////////////////////////////////////////////////////
+      String  importBaseName = "personImporter";
+      QRecord fileRecord     = new GetAction().executeForRecord(new GetInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX).withPrimaryKey(1));
+      assertEquals(47, fileRecord.getValue("customerId"));
+
+      QRecord recordRecord = new GetAction().executeForRecord(new GetInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX).withPrimaryKey(1));
+      assertEquals(47, recordRecord.getValue("customerId"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSecuritySupplier() throws QException
+   {
+      //////////////////////////////////////////////
+      // Add a security name/value to our process //
+      //////////////////////////////////////////////
+      QProcessMetaData process = QContext.getQInstance().getProcess(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      process.getInputFields().stream().filter(f -> f.getName().equals(FilesystemImporterStep.FIELD_IMPORT_SECURITY_FIELD_NAME)).findFirst().get().setDefaultValue("customerId");
+      process.getInputFields().stream().filter(f -> f.getName().equals(FilesystemImporterStep.FIELD_IMPORT_SECURITY_VALUE_SUPPLIER)).findFirst().get().setDefaultValue(new QCodeReference(SecuritySupplier.class));
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////
+      // re-validate our instance now that we have that code-reference in place for the security supplier //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////
+      QContext.getQInstance().setHasBeenValidated(null);
+      new QInstanceValidator().validate(QContext.getQInstance());
+
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(TestUtils.LOCAL_PERSON_CSV_FILE_IMPORTER_PROCESS_NAME);
+      new RunProcessAction().execute(runProcessInput);
+
+      ////////////////////////////////////////////////////////////////////////////////////////////
+      // assert the security field gets its value on both the importFile & importRecord records //
+      ////////////////////////////////////////////////////////////////////////////////////////////
+      String  importBaseName = "personImporter";
+      QRecord fileRecord     = new GetAction().executeForRecord(new GetInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_FILE_TABLE_SUFFIX).withPrimaryKey(1));
+      assertEquals(1701, fileRecord.getValue("customerId"));
+
+      QRecord recordRecord = new GetAction().executeForRecord(new GetInput(importBaseName + FilesystemImporterMetaDataTemplate.IMPORT_RECORD_TABLE_SUFFIX).withPrimaryKey(1));
+      assertEquals(1701, recordRecord.getValue("customerId"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static class SecuritySupplier implements Function<QRecord, Serializable>
+   {
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      @Override
+      public Serializable apply(QRecord qRecord)
+      {
+         return (1701);
+      }
+   }
+
+}

qqq-backend-module-filesystem/src/test/java/com/kingsrook/qqq/backend/module/filesystem/processes/implementations/filesystem/importer/ImportRecordPostQueryCustomizerTest.java

@@ -0,0 +1,85 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.filesystem.processes.implementations.filesystem.importer;
+
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
+import com.kingsrook.qqq.backend.module.filesystem.BaseTest;
+import org.json.JSONObject;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for ImportRecordPostQueryCustomizer 
+ *******************************************************************************/
+class ImportRecordPostQueryCustomizerTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test()
+   {
+      Instant createDate = Instant.parse("2024-01-08T20:07:21Z");
+
+      List<QRecord> output = new ImportRecordPostQueryCustomizer().apply(List.of(
+         new QRecord()
+            .withTableName("personImporterImportRecord")
+            .withValue("importFileId", 1)
+            .withValue("unmapped", 2)
+            .withValue("unstructured", 3)
+            .withValue("nosqlObject", MapBuilder.of(HashMap::new).with("foo", "bar").with("createDate", createDate).build())
+      ));
+
+      assertEquals(1, output.get(0).getValue("importFileId"));
+      assertEquals(2, output.get(0).getValue("unmapped"));
+      assertEquals(3, output.get(0).getValue("unstructured"));
+      assertEquals(Map.of("foo", "bar", "createDate", createDate), output.get(0).getValue("nosqlObject"));
+
+      ///////////////////////////////////////////////////////////////////////////////////////////
+      // make sure all un-structured fields get put in the "values" field as a JSON string     //
+      // compare as maps, beacuse JSONObject seems to care about the ordering, which, we don't //
+      ///////////////////////////////////////////////////////////////////////////////////////////
+      Map<String, Object> expectedMap = new JSONObject("""
+         {
+            "unmapped": 2,
+            "unstructured": 3,
+            "nosqlObject":
+            {
+               "foo": "bar",
+               "createDate": "%s"
+            }
+         }
+         """.formatted(createDate)).toMap();
+      Map<String, Object> actualMap = new JSONObject(output.get(0).getValueString("values")).toMap();
+      assertThat(actualMap).isEqualTo(expectedMap);
+   }
+
+}

qqq-backend-module-mongodb/pom.xml

@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ QQQ - Low-code Application Framework for Engineers.
+  ~ Copyright (C) 2021-2024.  Kingsrook, LLC
+  ~ 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+  ~ contact@kingsrook.com
+  ~ https://github.com/Kingsrook/
+  ~
+  ~ This program is free software: you can redistribute it and/or modify
+  ~ it under the terms of the GNU Affero General Public License as
+  ~ published by the Free Software Foundation, either version 3 of the
+  ~ License, or (at your option) any later version.
+  ~
+  ~ This program is distributed in the hope that it will be useful,
+  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of
+  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  ~ GNU Affero General Public License for more details.
+  ~
+  ~ You should have received a copy of the GNU Affero General Public License
+  ~ along with this program.  If not, see <https://www.gnu.org/licenses/>.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+   <modelVersion>4.0.0</modelVersion>
+
+   <artifactId>qqq-backend-module-mongodb</artifactId>
+
+   <parent>
+      <groupId>com.kingsrook.qqq</groupId>
+      <artifactId>qqq-parent-project</artifactId>
+      <version>${revision}</version>
+   </parent>
+
+   <properties>
+      <!-- props specifically to this module -->
+      <!-- none at this time -->
+   </properties>
+
+   <dependencies>
+      <!-- other qqq modules deps -->
+      <dependency>
+         <groupId>com.kingsrook.qqq</groupId>
+         <artifactId>qqq-backend-core</artifactId>
+         <version>${revision}</version>
+      </dependency>
+
+      <!-- 3rd party deps specifically for this module -->
+      <dependency>
+         <groupId>org.mongodb</groupId>
+         <artifactId>mongodb-driver-sync</artifactId>
+         <version>4.11.1</version>
+      </dependency>
+      <dependency>
+         <groupId>org.apache.logging.log4j</groupId>
+         <artifactId>log4j-slf4j-impl</artifactId>
+         <version>2.17.1</version>
+      </dependency>
+
+      <dependency>
+         <groupId>org.testcontainers</groupId>
+         <artifactId>mongodb</artifactId>
+         <version>1.19.3</version>
+         <scope>test</scope>
+      </dependency>
+
+      <!-- Common deps for all qqq modules -->
+      <dependency>
+         <groupId>org.apache.maven.plugins</groupId>
+         <artifactId>maven-checkstyle-plugin</artifactId>
+      </dependency>
+      <dependency>
+         <groupId>org.apache.logging.log4j</groupId>
+         <artifactId>log4j-api</artifactId>
+      </dependency>
+      <dependency>
+         <groupId>org.apache.logging.log4j</groupId>
+         <artifactId>log4j-core</artifactId>
+      </dependency>
+      <dependency>
+         <groupId>org.junit.jupiter</groupId>
+         <artifactId>junit-jupiter-engine</artifactId>
+         <scope>test</scope>
+      </dependency>
+      <dependency>
+         <groupId>org.assertj</groupId>
+         <artifactId>assertj-core</artifactId>
+         <scope>test</scope>
+      </dependency>
+   </dependencies>
+
+   <build>
+      <plugins>
+         <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-shade-plugin</artifactId>
+            <version>2.4.3</version>
+            <configuration>
+               <createDependencyReducedPom>false</createDependencyReducedPom>
+               <filters>
+                  <filter>
+                     <artifact>*:*</artifact>
+                     <excludes>
+                        <exclude>META-INF/*</exclude>
+                     </excludes>
+                  </filter>
+               </filters>
+            </configuration>
+            <executions>
+               <execution>
+                  <phase>${plugin.shade.phase}</phase>
+                  <goals>
+                     <goal>shade</goal>
+                  </goals>
+               </execution>
+            </executions>
+         </plugin>
+      </plugins>
+   </build>
+
+</project>

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/MongoDBBackendModule.java

@@ -0,0 +1,168 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb;
+
+
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
+import com.kingsrook.qqq.backend.core.actions.interfaces.AggregateInterface;
+import com.kingsrook.qqq.backend.core.actions.interfaces.CountInterface;
+import com.kingsrook.qqq.backend.core.actions.interfaces.DeleteInterface;
+import com.kingsrook.qqq.backend.core.actions.interfaces.InsertInterface;
+import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
+import com.kingsrook.qqq.backend.core.actions.interfaces.UpdateInterface;
+import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableBackendDetails;
+import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
+import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
+import com.kingsrook.qqq.backend.module.mongodb.actions.AbstractMongoDBAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoClientContainer;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBAggregateAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBCountAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBDeleteAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBInsertAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBQueryAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBTransaction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoDBUpdateAction;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBTableBackendDetails;
+
+
+/*******************************************************************************
+ ** QQQ Backend module for working with MongoDB
+ *******************************************************************************/
+public class MongoDBBackendModule implements QBackendModuleInterface
+{
+   static
+   {
+      QBackendModuleDispatcher.registerBackendModule(new MongoDBBackendModule());
+   }
+
+   /*******************************************************************************
+    ** Method where a backend module must be able to provide its type (name).
+    *******************************************************************************/
+   public String getBackendType()
+   {
+      return ("mongodb");
+   }
+
+
+
+   /*******************************************************************************
+    ** Method to identify the class used for backend meta data for this module.
+    *******************************************************************************/
+   @Override
+   public Class<? extends QBackendMetaData> getBackendMetaDataClass()
+   {
+      return (MongoDBBackendMetaData.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Method to identify the class used for table-backend details for this module.
+    *******************************************************************************/
+   @Override
+   public Class<? extends QTableBackendDetails> getTableBackendDetailsClass()
+   {
+      return (MongoDBTableBackendDetails.class);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public CountInterface getCountInterface()
+   {
+      return (new MongoDBCountAction());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public QueryInterface getQueryInterface()
+   {
+      return (new MongoDBQueryAction());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public InsertInterface getInsertInterface()
+   {
+      return (new MongoDBInsertAction());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public UpdateInterface getUpdateInterface()
+   {
+      return (new MongoDBUpdateAction());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public DeleteInterface getDeleteInterface()
+   {
+      return (new MongoDBDeleteAction());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public AggregateInterface getAggregateInterface()
+   {
+      return (new MongoDBAggregateAction());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public QBackendTransaction openTransaction(AbstractTableActionInput input)
+   {
+      MongoDBBackendMetaData backend              = (MongoDBBackendMetaData) input.getBackend();
+      MongoClientContainer   mongoClientContainer = new AbstractMongoDBAction().openClient(backend, null);
+      return (new MongoDBTransaction(backend, mongoClientContainer.getMongoClient()));
+   }
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/AbstractMongoDBAction.java

@@ -0,0 +1,788 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
+import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.JoinsContext;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.AbstractFilterExpression;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType;
+import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock;
+import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLockFilters;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.querystats.QueryStat;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBTableBackendDetails;
+import com.mongodb.ConnectionString;
+import com.mongodb.MongoClientSettings;
+import com.mongodb.MongoCredential;
+import com.mongodb.client.MongoClient;
+import com.mongodb.client.MongoClients;
+import com.mongodb.client.model.Filters;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.bson.types.ObjectId;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** Base class for all mongoDB module actions.
+ *******************************************************************************/
+public class AbstractMongoDBAction
+{
+   private static final QLogger LOG = QLogger.getLogger(AbstractMongoDBAction.class);
+
+   protected QueryStat queryStat;
+
+
+
+   /*******************************************************************************
+    ** Open a MongoDB Client / session -- re-using the one in the input transaction
+    ** if it is present.
+    *******************************************************************************/
+   public MongoClientContainer openClient(MongoDBBackendMetaData backend, QBackendTransaction transaction)
+   {
+      if(transaction instanceof MongoDBTransaction mongoDBTransaction)
+      {
+         //////////////////////////////////////////////////////////////////////////////////////////
+         // re-use the connection from the transaction (indicating false in last parameter here) //
+         //////////////////////////////////////////////////////////////////////////////////////////
+         return (new MongoClientContainer(mongoDBTransaction.getMongoClient(), mongoDBTransaction.getClientSession(), false));
+      }
+
+      String           suffix           = StringUtils.hasContent(backend.getUrlSuffix()) ? "?" + backend.getUrlSuffix() : "";
+      ConnectionString connectionString = new ConnectionString("mongodb://" + backend.getHost() + ":" + backend.getPort() + "/" + suffix);
+
+      MongoCredential credential = MongoCredential.createCredential(backend.getUsername(), backend.getAuthSourceDatabase(), backend.getPassword().toCharArray());
+
+      MongoClientSettings settings = MongoClientSettings.builder()
+
+         ////////////////////////////////////////////////
+         // is this needed, what, for a cluster maybe? //
+         ////////////////////////////////////////////////
+         // .applyToClusterSettings(builder -> builder.hosts(seeds))
+
+         .applyConnectionString(connectionString)
+         .credential(credential)
+         .build();
+
+      MongoClient mongoClient = MongoClients.create(settings);
+
+      ////////////////////////////////////////////////////////////////////////////
+      // indicate that this connection was newly opened via the true param here //
+      ////////////////////////////////////////////////////////////////////////////
+      return (new MongoClientContainer(mongoClient, mongoClient.startSession(), true));
+   }
+
+
+
+   /*******************************************************************************
+    ** Get the name to use for a field in the mongoDB, from the fieldMetaData.
+    **
+    ** That is, field.backendName if set -- else, field.name
+    *******************************************************************************/
+   protected String getFieldBackendName(QFieldMetaData field)
+   {
+      if(field.getBackendName() != null)
+      {
+         return (field.getBackendName());
+      }
+      return (field.getName());
+   }
+
+
+
+   /*******************************************************************************
+    ** Get the name to use for a table in the mongoDB, from the table's backendDetails.
+    **
+    ** else, the table's name.
+    *******************************************************************************/
+   protected String getBackendTableName(QTableMetaData table)
+   {
+      if(table == null)
+      {
+         return (null);
+      }
+
+      if(table.getBackendDetails() != null)
+      {
+         String backendTableName = ((MongoDBTableBackendDetails) table.getBackendDetails()).getTableName();
+         if(StringUtils.hasContent(backendTableName))
+         {
+            return (backendTableName);
+         }
+      }
+      return table.getName();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected int getPageSize()
+   {
+      return (1000);
+   }
+
+
+
+   /*******************************************************************************
+    ** Convert a mongodb document to a QRecord.
+    *******************************************************************************/
+   protected QRecord documentToRecord(QTableMetaData table, Document document)
+   {
+      QRecord record = new QRecord();
+      record.setTableName(table.getName());
+
+      //////////////////////////////////////////////////////////////////////////////////////////////
+      // first iterate over the table's fields, looking for them (at their backend name (path,    //
+      // if it has dots) inside the document note that we'll remove values from the document      //
+      // as we go - then after this loop, will handle all remaining values as unstructured fields //
+      //////////////////////////////////////////////////////////////////////////////////////////////
+      Map<String, Serializable> values = record.getValues();
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         String fieldName = field.getName();
+         String fieldBackendName = getFieldBackendName(field);
+
+         if(fieldBackendName.contains("."))
+         {
+            /////////////////////////////////////////////////////////////
+            // process backend-names with dots as hierarchical objects //
+            /////////////////////////////////////////////////////////////
+            String[] parts       = fieldBackendName.split("\\.");
+            Document tmpDocument = document;
+            for(int i = 0; i < parts.length - 1; i++)
+            {
+               if(!tmpDocument.containsKey(parts[i]))
+               {
+                  ///////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  // if we can't find the sub-document, break, and we won't have a value for this field (do we want null?) //
+                  ///////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  setValue(values, fieldName, null);
+                  break;
+               }
+               else
+               {
+                  if(tmpDocument.get(parts[i]) instanceof Document subDocument)
+                  {
+                     tmpDocument = subDocument;
+                  }
+                  else
+                  {
+                     LOG.warn("Unexpected - In table [" + table.getName() + "] found a non-document at sub-key [" + parts[i] + "] for field [" + field.getName() + "]");
+                  }
+               }
+            }
+
+            Object value = tmpDocument.remove(parts[parts.length - 1]);
+            setValue(values, fieldName, value);
+         }
+         else
+         {
+            Object value = document.remove(fieldBackendName);
+            setValue(values, fieldName, value);
+         }
+      }
+
+      //////////////////////////////////////////////////////////////
+      // handle remaining values in the document as un-structured //
+      //////////////////////////////////////////////////////////////
+      for(String subFieldName : document.keySet())
+      {
+         Object subValue = document.get(subFieldName);
+         setValue(values, subFieldName, subValue);
+      }
+
+      return (record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Recursive helper method to put a value in a map - where mongodb documents
+    ** are recursively expanded, and types are mapped to QQQ expectations.
+    *******************************************************************************/
+   private void setValue(Map<String, Serializable> values, String fieldName, Object value)
+   {
+      if(value instanceof ObjectId objectId)
+      {
+         values.put(fieldName, objectId.toString());
+      }
+      else if(value instanceof java.util.Date date)
+      {
+         values.put(fieldName, date.toInstant());
+      }
+      else if(value instanceof Document document)
+      {
+         LinkedHashMap<String, Serializable> subValues = new LinkedHashMap<>();
+         values.put(fieldName, subValues);
+
+         for(String subFieldName : document.keySet())
+         {
+            Object subValue = document.get(subFieldName);
+            setValue(subValues, subFieldName, subValue);
+         }
+      }
+      else if(value instanceof Serializable s)
+      {
+         values.put(fieldName, s);
+      }
+      else if(value != null)
+      {
+         values.put(fieldName, String.valueOf(value));
+      }
+      else
+      {
+         values.put(fieldName, null);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Convert a QRecord to a mongodb document.
+    *******************************************************************************/
+   protected Document recordToDocument(QTableMetaData table, QRecord record) throws QException
+   {
+      Document document = new Document();
+
+      ////////////////////////////////////////////////////////////////////////////////////////////////
+      // first iterate over fields defined in the table - put them in the document for mongo first. //
+      // track the names that we've processed in a set. then later we'll go over all values in the  //
+      // record and send them all to mongo (skipping ones we knew about from the table definition)  //
+      ////////////////////////////////////////////////////////////////////////////////////////////////
+      Set<String> processedFields = new HashSet<>();
+
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         Serializable value = record.getValue(field.getName());
+         processedFields.add(field.getName());
+
+         if(field.getName().equals(table.getPrimaryKeyField()) && value == null)
+         {
+            ////////////////////////////////////
+            // let mongodb client generate id //
+            ////////////////////////////////////
+            continue;
+         }
+
+         String fieldBackendName = getFieldBackendName(field);
+         if(fieldBackendName.contains("."))
+         {
+            /////////////////////////////////////////////////////////////
+            // process backend-names with dots as hierarchical objects //
+            /////////////////////////////////////////////////////////////
+            String[] parts       = fieldBackendName.split("\\.");
+            Document tmpDocument = document;
+            for(int i = 0; i < parts.length - 1; i++)
+            {
+               if(!tmpDocument.containsKey(parts[i]))
+               {
+                  Document subDocument = new Document();
+                  tmpDocument.put(parts[i], subDocument);
+                  tmpDocument = subDocument;
+               }
+               else
+               {
+                  if(tmpDocument.get(parts[i]) instanceof Document subDocument)
+                  {
+                     tmpDocument = subDocument;
+                  }
+                  else
+                  {
+                     throw (new QException("Fields in table [" + table.getName() + "] specify both a sub-object and a field at the key: " + parts[i]));
+                  }
+               }
+            }
+            tmpDocument.append(parts[parts.length - 1], value);
+         }
+         else
+         {
+            document.append(fieldBackendName, value);
+         }
+      }
+
+      /////////////////////////
+      // do remaining values //
+      /////////////////////////
+      for(Map.Entry<String, Serializable> entry : record.getValues().entrySet())
+      {
+         if(!processedFields.contains(entry.getKey()))
+         {
+            document.append(entry.getKey(), entry.getValue());
+         }
+      }
+
+      return (document);
+   }
+
+
+
+   /*******************************************************************************
+    ** Convert QQueryFilter to Bson search query document - including security
+    ** for the table if needed.
+    *******************************************************************************/
+   protected Bson makeSearchQueryDocument(QTableMetaData table, QQueryFilter filter) throws QException
+   {
+      Bson         searchQueryWithoutSecurity = makeSearchQueryDocumentWithoutSecurity(table, filter);
+      QQueryFilter securityFilter             = makeSecurityQueryFilter(table);
+      if(!securityFilter.hasAnyCriteria())
+      {
+         return (searchQueryWithoutSecurity);
+      }
+
+      Bson searchQueryForSecurity = makeSearchQueryDocumentWithoutSecurity(table, securityFilter);
+
+      if(searchQueryWithoutSecurity.toBsonDocument().isEmpty())
+      {
+         return (searchQueryForSecurity);
+      }
+      else
+      {
+         return (Filters.and(searchQueryWithoutSecurity, searchQueryForSecurity));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Build a QQueryFilter to apply record-level security to the query.
+    ** Note, it may be empty, if there are no lock fields, or all are all-access.
+    **
+    ** Originally copied from RDBMS module... should this be shared?
+    ** and/or, how big of a re-write did that get in the joins-enhancements branch...
+    *******************************************************************************/
+   private QQueryFilter makeSecurityQueryFilter(QTableMetaData table) throws QException
+   {
+      QQueryFilter securityFilter = new QQueryFilter();
+      securityFilter.setBooleanOperator(QQueryFilter.BooleanOperator.AND);
+
+      for(RecordSecurityLock recordSecurityLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks())))
+      {
+         addSubFilterForRecordSecurityLock(QContext.getQInstance(), QContext.getQSession(), table, securityFilter, recordSecurityLock, null, table.getName(), false);
+      }
+
+      return (securityFilter);
+   }
+
+
+
+   /*******************************************************************************
+    ** Helper for makeSecuritySearchQuery.
+    **
+    ** Originally copied from RDBMS module... should this be shared?
+    ** and/or, how big of a re-write did that get in the joins-enhancements branch...
+    *******************************************************************************/
+   private static void addSubFilterForRecordSecurityLock(QInstance instance, QSession session, QTableMetaData table, QQueryFilter securityFilter, RecordSecurityLock recordSecurityLock, JoinsContext joinsContext, String tableNameOrAlias, boolean isOuter) throws QException
+   {
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // check if the key type has an all-access key, and if so, if it's set to true for the current user/session //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      QSecurityKeyType securityKeyType = instance.getSecurityKeyType(recordSecurityLock.getSecurityKeyType());
+      if(StringUtils.hasContent(securityKeyType.getAllAccessKeyName()))
+      {
+         if(session.hasSecurityKeyValue(securityKeyType.getAllAccessKeyName(), true, QFieldType.BOOLEAN))
+         {
+            ///////////////////////////////////////////////////////////////////////////////
+            // if we have all-access on this key, then we don't need a criterion for it. //
+            ///////////////////////////////////////////////////////////////////////////////
+            return;
+         }
+      }
+
+      ///////////////////////////////////////////////////////////////////////////////////////
+      // some differences from RDBMS here, due to not yet having joins support in mongo... //
+      ///////////////////////////////////////////////////////////////////////////////////////
+      // String fieldName = tableNameOrAlias + "." + recordSecurityLock.getFieldName();
+      String fieldName = recordSecurityLock.getFieldName();
+      if(CollectionUtils.nullSafeHasContents(recordSecurityLock.getJoinNameChain()))
+      {
+         throw (new QException("Security locks in mongodb with joinNameChain is not yet supported"));
+         // fieldName = recordSecurityLock.getFieldName();
+      }
+
+      ///////////////////////////////////////////////////////////////////////////////////////////
+      // else - get the key values from the session and decide what kind of criterion to build //
+      ///////////////////////////////////////////////////////////////////////////////////////////
+      QQueryFilter          lockFilter   = new QQueryFilter();
+      List<QFilterCriteria> lockCriteria = new ArrayList<>();
+      lockFilter.setCriteria(lockCriteria);
+
+      QFieldType type = QFieldType.INTEGER;
+      try
+      {
+         if(joinsContext == null)
+         {
+            type = table.getField(fieldName).getType();
+         }
+         else
+         {
+            JoinsContext.FieldAndTableNameOrAlias fieldAndTableNameOrAlias = joinsContext.getFieldAndTableNameOrAlias(fieldName);
+            type = fieldAndTableNameOrAlias.field().getType();
+         }
+      }
+      catch(Exception e)
+      {
+         LOG.debug("Error getting field type...  Trying Integer", e);
+      }
+
+      List<Serializable> securityKeyValues = session.getSecurityKeyValues(recordSecurityLock.getSecurityKeyType(), type);
+      if(CollectionUtils.nullSafeIsEmpty(securityKeyValues))
+      {
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // handle user with no values -- they can only see null values, and only iff the lock's null-value behavior is ALLOW //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         if(RecordSecurityLock.NullValueBehavior.ALLOW.equals(recordSecurityLock.getNullValueBehavior()))
+         {
+            lockCriteria.add(new QFilterCriteria(fieldName, QCriteriaOperator.IS_BLANK));
+         }
+         else
+         {
+            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // else, if no user/session values, and null-value behavior is deny, then setup a FALSE condition, to allow no rows.           //
+            // todo - make some explicit contradiction here - maybe even avoid running the whole query - as you're not allowed ANY records //
+            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+            lockCriteria.add(new QFilterCriteria(fieldName, QCriteriaOperator.IN, Collections.emptyList()));
+         }
+      }
+      else
+      {
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // else, if user/session has some values, build an IN rule -                                                //
+         // noting that if the lock's null-value behavior is ALLOW, then we actually want IS_NULL_OR_IN, not just IN //
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         if(RecordSecurityLock.NullValueBehavior.ALLOW.equals(recordSecurityLock.getNullValueBehavior()))
+         {
+            lockCriteria.add(new QFilterCriteria(fieldName, QCriteriaOperator.IS_NULL_OR_IN, securityKeyValues));
+         }
+         else
+         {
+            lockCriteria.add(new QFilterCriteria(fieldName, QCriteriaOperator.IN, securityKeyValues));
+         }
+      }
+
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // if this field is on the outer side of an outer join, then if we do a straight filter on it, then we're basically      //
+      // nullifying the outer join... so for an outer join use-case, OR the security field criteria with a primary-key IS NULL //
+      // which will make missing rows from the join be found.                                                                  //
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      if(isOuter)
+      {
+         lockFilter.setBooleanOperator(QQueryFilter.BooleanOperator.OR);
+         lockFilter.addCriteria(new QFilterCriteria(tableNameOrAlias + "." + table.getPrimaryKeyField(), QCriteriaOperator.IS_BLANK));
+      }
+
+      securityFilter.addSubFilter(lockFilter);
+   }
+
+
+
+   /*******************************************************************************
+    ** w/o considering security, just map a QQueryFilter to a Bson searchQuery.
+    *******************************************************************************/
+   @SuppressWarnings("checkstyle:Indentation")
+   private Bson makeSearchQueryDocumentWithoutSecurity(QTableMetaData table, QQueryFilter filter)
+   {
+      if(filter == null || !filter.hasAnyCriteria())
+      {
+         return (new Document());
+      }
+
+      List<Bson> criteriaFilters = new ArrayList<>();
+
+      for(QFilterCriteria criteria : CollectionUtils.nonNullList(filter.getCriteria()))
+      {
+         List<Serializable> values           = criteria.getValues() == null ? new ArrayList<>() : new ArrayList<>(criteria.getValues());
+         QFieldMetaData     field            = table.getField(criteria.getFieldName());
+         String             fieldBackendName = getFieldBackendName(field);
+
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // replace any expression-type values with their evaluation                                                                         //
+         // also, "scrub" non-expression values, which type-converts them (e.g., strings in various supported date formats become LocalDate) //
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         ListIterator<Serializable> valueListIterator = values.listIterator();
+         while(valueListIterator.hasNext())
+         {
+            Serializable value = valueListIterator.next();
+            if(value instanceof AbstractFilterExpression<?> expression)
+            {
+               valueListIterator.set(expression.evaluate());
+            }
+            /*
+            todo - is this needed??
+            else
+            {
+               Serializable scrubbedValue = scrubValue(field, value);
+               valueListIterator.set(scrubbedValue);
+            }
+            */
+         }
+
+         /////////////////////////////////////////////////////////////////////////////////////////
+         // make sure any values we're going to run against the primary key (_id) are ObjectIds //
+         /////////////////////////////////////////////////////////////////////////////////////////
+         if(field.getName().equals(table.getPrimaryKeyField()))
+         {
+            ListIterator<Serializable> iterator = values.listIterator();
+            while(iterator.hasNext())
+            {
+               Serializable value = iterator.next();
+               iterator.set(new ObjectId(String.valueOf(value)));
+            }
+         }
+
+         ////////
+         // :( //
+         ////////
+         if(StringUtils.hasContent(criteria.getOtherFieldName()))
+         {
+            throw (new IllegalArgumentException("A mongodb query with an 'otherFieldName' specified is not currently supported."));
+         }
+
+         criteriaFilters.add(switch(criteria.getOperator())
+         {
+            case EQUALS -> Filters.eq(fieldBackendName, getValue(values, 0));
+
+            case NOT_EQUALS -> Filters.and(
+               Filters.ne(fieldBackendName, getValue(values, 0)),
+
+               ////////////////////////////////////////////////////////////////////////////////////////////
+               // to match RDBMS and other QQQ backends, consider a null to not match a not-equals query //
+               ////////////////////////////////////////////////////////////////////////////////////////////
+               Filters.not(Filters.eq(fieldBackendName, null))
+            );
+
+            case NOT_EQUALS_OR_IS_NULL -> Filters.or(
+               Filters.eq(fieldBackendName, null),
+               Filters.ne(fieldBackendName, getValue(values, 0))
+            );
+            case IN -> filterIn(fieldBackendName, values);
+            case NOT_IN -> Filters.nor(filterIn(fieldBackendName, values));
+            case IS_NULL_OR_IN -> Filters.or(
+               Filters.eq(fieldBackendName, null),
+               filterIn(fieldBackendName, values)
+            );
+            case LIKE -> filterRegex(fieldBackendName, null, ValueUtils.getValueAsString(getValue(values, 0)).replaceAll("%", ".*"), null);
+            case NOT_LIKE -> Filters.nor(filterRegex(fieldBackendName, null, ValueUtils.getValueAsString(getValue(values, 0)).replaceAll("%", ".*"), null));
+            case STARTS_WITH -> filterRegex(fieldBackendName, null, getValue(values, 0), ".*");
+            case ENDS_WITH -> filterRegex(fieldBackendName, ".*", getValue(values, 0), null);
+            case CONTAINS -> filterRegex(fieldBackendName, ".*", getValue(values, 0), ".*");
+            case NOT_STARTS_WITH -> Filters.nor(filterRegex(fieldBackendName, null, getValue(values, 0), ".*"));
+            case NOT_ENDS_WITH -> Filters.nor(filterRegex(fieldBackendName, ".*", getValue(values, 0), null));
+            case NOT_CONTAINS -> Filters.nor(filterRegex(fieldBackendName, ".*", getValue(values, 0), ".*"));
+            case LESS_THAN -> Filters.lt(fieldBackendName, getValue(values, 0));
+            case LESS_THAN_OR_EQUALS -> Filters.lte(fieldBackendName, getValue(values, 0));
+            case GREATER_THAN -> Filters.gt(fieldBackendName, getValue(values, 0));
+            case GREATER_THAN_OR_EQUALS -> Filters.gte(fieldBackendName, getValue(values, 0));
+            case IS_BLANK -> filterIsBlank(fieldBackendName);
+            case IS_NOT_BLANK -> Filters.nor(filterIsBlank(fieldBackendName));
+            case BETWEEN -> filterBetween(fieldBackendName, values);
+            case NOT_BETWEEN -> Filters.nor(filterBetween(fieldBackendName, values));
+         });
+      }
+
+      /////////////////////////////////////
+      // recursively process sub-filters //
+      /////////////////////////////////////
+      if(CollectionUtils.nullSafeHasContents(filter.getSubFilters()))
+      {
+         for(QQueryFilter subFilter : filter.getSubFilters())
+         {
+            criteriaFilters.add(makeSearchQueryDocumentWithoutSecurity(table, subFilter));
+         }
+      }
+
+      Bson bson = QQueryFilter.BooleanOperator.AND.equals(filter.getBooleanOperator()) ? Filters.and(criteriaFilters) : Filters.or(criteriaFilters);
+      return bson;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static Serializable getValue(List<Serializable> values, int i)
+   {
+      if(values == null || values.size() <= i)
+      {
+         throw new IllegalArgumentException("Incorrect number of values given for criteria");
+      }
+
+      return (values.get(i));
+   }
+
+
+
+   /*******************************************************************************
+    ** build a bson filter doing a regex (e.g., for LIKE, STARTS_WITH, etc)
+    *******************************************************************************/
+   private Bson filterRegex(String fieldBackendName, String prefix, Serializable mainRegex, String suffix)
+   {
+      if(prefix == null)
+      {
+         prefix = "";
+      }
+
+      if(suffix == null)
+      {
+         suffix = "";
+      }
+
+      String fullRegex = prefix + ValueUtils.getValueAsString(mainRegex + suffix);
+      return (Filters.regex(fieldBackendName, Pattern.compile(fullRegex)));
+   }
+
+
+
+   /*******************************************************************************
+    ** build a bson filter doing IN
+    *******************************************************************************/
+   private static Bson filterIn(String fieldBackendName, List<Serializable> values)
+   {
+      return Filters.in(fieldBackendName, values);
+   }
+
+
+
+   /*******************************************************************************
+    ** build a bson filter doing BETWEEN
+    *******************************************************************************/
+   private static Bson filterBetween(String fieldBackendName, List<Serializable> values)
+   {
+      return Filters.and(
+         Filters.gte(fieldBackendName, getValue(values, 0)),
+         Filters.lte(fieldBackendName, getValue(values, 1))
+      );
+   }
+
+
+
+   /*******************************************************************************
+    ** build a bson filter doing BLANK (null or == "")
+    *******************************************************************************/
+   private static Bson filterIsBlank(String fieldBackendName)
+   {
+      return Filters.or(
+         Filters.eq(fieldBackendName, null),
+         Filters.eq(fieldBackendName, "")
+      );
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for queryStat
+    *******************************************************************************/
+   public QueryStat getQueryStat()
+   {
+      return (this.queryStat);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for queryStat
+    *******************************************************************************/
+   public void setQueryStat(QueryStat queryStat)
+   {
+      this.queryStat = queryStat;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected void setQueryInQueryStat(Bson query)
+   {
+      if(queryStat != null && query != null)
+      {
+         queryStat.setQueryText(query.toString());
+
+         ////////////////////////////////////////////////////////////////
+         // todo - if we support joins in the future, do them here too //
+         ////////////////////////////////////////////////////////////////
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected void logQuery(String tableName, String actionName, List<Bson> query, Long queryStartTime)
+   {
+
+      if(System.getProperty("qqq.mongodb.logQueries", "false").equals("true"))
+      {
+         try
+         {
+            if(System.getProperty("qqq.mongodb.logQueries.output", "logger").equalsIgnoreCase("system.out"))
+            {
+               System.out.println("Table: " + tableName + ", Action: " + actionName + ", Query: " + query);
+
+               if(queryStartTime != null)
+               {
+                  System.out.println("Query Took [" + QValueFormatter.formatValue(DisplayFormat.COMMAS, (System.currentTimeMillis() - queryStartTime)) + "] ms");
+               }
+            }
+            else
+            {
+               LOG.debug("Running Query", logPair("table", tableName), logPair("action", actionName), logPair("query", query), logPair("millis", queryStartTime == null ? null : (System.currentTimeMillis() - queryStartTime)));
+            }
+         }
+         catch(Exception e)
+         {
+            LOG.debug("Error logging query...", e);
+         }
+      }
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoClientContainer.java

@@ -0,0 +1,158 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import com.mongodb.client.ClientSession;
+import com.mongodb.client.MongoClient;
+
+
+/*******************************************************************************
+ ** Wrapper around a MongoClient, ClientSession, and a boolean to help signal
+ ** where it was opened (e.g., so you know if you need to close it yourself, or
+ ** if it came from someone else (e.g., via an input transaction)).
+ *******************************************************************************/
+public class MongoClientContainer
+{
+   private MongoClient   mongoClient;
+   private ClientSession mongoSession;
+   private boolean       needToClose;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public MongoClientContainer(MongoClient mongoClient, ClientSession mongoSession, boolean needToClose)
+   {
+      this.mongoClient = mongoClient;
+      this.mongoSession = mongoSession;
+      this.needToClose = needToClose;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for mongoClient
+    *******************************************************************************/
+   public MongoClient getMongoClient()
+   {
+      return (this.mongoClient);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for mongoClient
+    *******************************************************************************/
+   public void setMongoClient(MongoClient mongoClient)
+   {
+      this.mongoClient = mongoClient;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for mongoClient
+    *******************************************************************************/
+   public MongoClientContainer withMongoClient(MongoClient mongoClient)
+   {
+      this.mongoClient = mongoClient;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for mongoSession
+    *******************************************************************************/
+   public ClientSession getMongoSession()
+   {
+      return (this.mongoSession);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for mongoSession
+    *******************************************************************************/
+   public void setMongoSession(ClientSession mongoSession)
+   {
+      this.mongoSession = mongoSession;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for mongoSession
+    *******************************************************************************/
+   public MongoClientContainer withMongoSession(ClientSession mongoSession)
+   {
+      this.mongoSession = mongoSession;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for needToClose
+    *******************************************************************************/
+   public boolean getNeedToClose()
+   {
+      return (this.needToClose);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for needToClose
+    *******************************************************************************/
+   public void setNeedToClose(boolean needToClose)
+   {
+      this.needToClose = needToClose;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for needToClose
+    *******************************************************************************/
+   public MongoClientContainer withNeedToClose(boolean needToClose)
+   {
+      this.needToClose = needToClose;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void closeIfNeeded()
+   {
+      if(needToClose)
+      {
+         mongoSession.close();
+         mongoClient.close();
+      }
+   }
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBAggregateAction.java

@@ -0,0 +1,274 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import com.kingsrook.qqq.backend.core.actions.interfaces.AggregateInterface;
+import com.kingsrook.qqq.backend.core.actions.tables.helpers.ActionTimeoutHelper;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.Aggregate;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateResult;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.GroupBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.QFilterOrderByAggregate;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.QFilterOrderByGroupBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import com.mongodb.client.model.Accumulators;
+import com.mongodb.client.model.Aggregates;
+import com.mongodb.client.model.BsonField;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MongoDBAggregateAction extends AbstractMongoDBAction implements AggregateInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBAggregateAction.class);
+
+   private ActionTimeoutHelper actionTimeoutHelper;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("checkstyle:indentation")
+   public AggregateOutput execute(AggregateInput aggregateInput) throws QException
+   {
+      MongoClientContainer mongoClientContainer = null;
+
+      Long       queryStartTime = System.currentTimeMillis();
+      List<Bson> queryToLog     = new ArrayList<>();
+
+      try
+      {
+         AggregateOutput        aggregateOutput  = new AggregateOutput();
+         QTableMetaData         table            = aggregateInput.getTable();
+         String                 backendTableName = getBackendTableName(table);
+         MongoDBBackendMetaData backend          = (MongoDBBackendMetaData) aggregateInput.getBackend();
+
+         mongoClientContainer = openClient(backend, null); // todo - aggregate input has no transaction!?
+         MongoDatabase             database   = mongoClientContainer.getMongoClient().getDatabase(backend.getDatabaseName());
+         MongoCollection<Document> collection = database.getCollection(backendTableName);
+
+         QQueryFilter filter      = aggregateInput.getFilter();
+         Bson         searchQuery = makeSearchQueryDocument(table, filter);
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // set up & start an actionTimeoutHelper (note, internally it'll deal with the time being null or negative as meaning not to timeout) //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         actionTimeoutHelper = new ActionTimeoutHelper(aggregateInput.getTimeoutSeconds(), TimeUnit.SECONDS, new TimeoutCanceller(mongoClientContainer));
+         actionTimeoutHelper.start();
+
+         /////////////////////////////////////////////////////////////////////////
+         // we have to submit a list of BSON objects to the aggregate function. //
+         // the first one is the search query                                   //
+         // second is the group-by stuff, which we'll explain as we build it    //
+         /////////////////////////////////////////////////////////////////////////
+         List<Bson> bsonList = new ArrayList<>();
+         bsonList.add(Aggregates.match(searchQuery));
+         setQueryInQueryStat(searchQuery);
+         queryToLog = bsonList;
+
+         //////////////////////////////////////////////////////////////////////////////////////
+         // if there are group-by fields, then we need to build a document with those fields //
+         // not sure what the whole name, $name is, but, go go mongo                         //
+         //////////////////////////////////////////////////////////////////////////////////////
+         Document groupValueDocument = new Document();
+         if(CollectionUtils.nullSafeHasContents(aggregateInput.getGroupBys()))
+         {
+            for(GroupBy groupBy : aggregateInput.getGroupBys())
+            {
+               String name = getFieldBackendName(table.getField(groupBy.getFieldName()));
+               groupValueDocument.append(name, "$" + name);
+            }
+         }
+
+         ////////////////////////////////////////////////////////////////////
+         // next build a list of accumulator fields - for aggregate values //
+         ////////////////////////////////////////////////////////////////////
+         List<BsonField> bsonFields = new ArrayList<>();
+         for(Aggregate aggregate : aggregateInput.getAggregates())
+         {
+            String fieldName  = aggregate.getFieldName() + "_" + aggregate.getOperator().toString().toLowerCase();
+            String expression = "$" + getFieldBackendName(table.getField(aggregate.getFieldName()));
+
+            bsonFields.add(switch(aggregate.getOperator())
+            {
+               case COUNT -> Accumulators.sum(fieldName, 1); // count... do a sum of 1's
+               case COUNT_DISTINCT -> throw new QException("Count Distinct is not supported for MongoDB tables at this time.");
+               case SUM -> Accumulators.sum(fieldName, expression);
+               case MIN -> Accumulators.min(fieldName, expression);
+               case MAX -> Accumulators.max(fieldName, expression);
+               case AVG -> Accumulators.avg(fieldName, expression);
+            });
+         }
+
+         ///////////////////////////////////////////////////////////////////////////////////
+         // add the group-by fields and the aggregates in the group stage of the pipeline //
+         ///////////////////////////////////////////////////////////////////////////////////
+         bsonList.add(Aggregates.group(groupValueDocument, bsonFields));
+
+         //////////////////////////////////////////////
+         // if there are any order-bys, add them too //
+         //////////////////////////////////////////////
+         if(filter != null && CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
+         {
+            Document sortValue = new Document();
+            for(QFilterOrderBy orderBy : filter.getOrderBys())
+            {
+               String fieldName;
+               if(orderBy instanceof QFilterOrderByAggregate orderByAggregate)
+               {
+                  Aggregate aggregate = orderByAggregate.getAggregate();
+                  fieldName = aggregate.getFieldName() + "_" + aggregate.getOperator().toString().toLowerCase();
+               }
+               else if(orderBy instanceof QFilterOrderByGroupBy orderByGroupBy)
+               {
+                  fieldName = "_id." + getFieldBackendName(table.getField(orderByGroupBy.getGroupBy().getFieldName()));
+               }
+               else
+               {
+                  ///////////////////////////////////////////////////
+                  // does this happen?  should it be "_id." if so? //
+                  ///////////////////////////////////////////////////
+                  fieldName = getFieldBackendName(table.getField(orderBy.getFieldName()));
+               }
+
+               sortValue.append(fieldName, orderBy.getIsAscending() ? 1 : -1);
+            }
+
+            bsonList.add(new Document("$sort", sortValue));
+         }
+
+         ////////////////////////////////////////////////////////
+         // todo - system property to control (like print-sql) //
+         ////////////////////////////////////////////////////////
+         // LOG.debug(bsonList.toString());
+
+         ///////////////////////////
+         // execute the aggregate //
+         ///////////////////////////
+         AggregateIterable<Document> aggregates = collection.aggregate(mongoClientContainer.getMongoSession(), bsonList);
+
+         List<AggregateResult> results = new ArrayList<>();
+         aggregateOutput.setResults(results);
+
+         /////////////////////
+         // process results //
+         /////////////////////
+         for(Document document : aggregates)
+         {
+            /////////////////////////////////////////////////////////////////////////
+            // once we've started getting results, go ahead and cancel the timeout //
+            /////////////////////////////////////////////////////////////////////////
+            actionTimeoutHelper.cancel();
+            setQueryStatFirstResultTime();
+
+            AggregateResult result = new AggregateResult();
+            results.add(result);
+
+            ////////////////////////////////////////////////////////////////
+            // get group by values (if there are any) out of the document //
+            ////////////////////////////////////////////////////////////////
+            for(GroupBy groupBy : CollectionUtils.nonNullList(aggregateInput.getGroupBys()))
+            {
+               Document idDocument = (Document) document.get("_id");
+               Object   value      = idDocument.get(groupBy.getFieldName());
+               result.withGroupByValue(groupBy, ValueUtils.getValueAsFieldType(groupBy.getType(), value));
+            }
+
+            //////////////////////////////////////////
+            // get aggregate values out of document //
+            //////////////////////////////////////////
+            for(Aggregate aggregate : aggregateInput.getAggregates())
+            {
+               QFieldMetaData field     = table.getField(aggregate.getFieldName());
+               QFieldType     fieldType = aggregate.getFieldType();
+               if(fieldType == null)
+               {
+                  fieldType = field.getType();
+               }
+               if(fieldType.equals(QFieldType.INTEGER) && (aggregate.getOperator().equals(AggregateOperator.AVG)))
+               {
+                  fieldType = QFieldType.DECIMAL;
+               }
+
+               Object value = document.get(aggregate.getFieldName() + "_" + aggregate.getOperator().toString().toLowerCase());
+               result.withAggregateValue(aggregate, ValueUtils.getValueAsFieldType(fieldType, value));
+            }
+         }
+
+         return (aggregateOutput);
+      }
+      catch(Exception e)
+      {
+         if(actionTimeoutHelper != null && actionTimeoutHelper.getDidTimeout())
+         {
+            setQueryStatFirstResultTime();
+            throw (new QUserFacingException("Aggregate timed out."));
+         }
+
+         /*
+         /////////////////////////////////////////////////////////////////////////////////////
+         // this was copied from RDBMS - not sure where/how/if it's being used there though //
+         /////////////////////////////////////////////////////////////////////////////////////
+         if(isCancelled)
+         {
+            throw (new QUserFacingException("Aggregate was cancelled."));
+         }
+         */
+
+         LOG.warn("Error executing aggregate", e);
+         throw new QException("Error executing aggregate", e);
+      }
+      finally
+      {
+         logQuery(getBackendTableName(aggregateInput.getTable()), "aggregate", queryToLog, queryStartTime);
+
+         if(mongoClientContainer != null)
+         {
+            mongoClientContainer.closeIfNeeded();
+         }
+      }
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBCountAction.java

@@ -0,0 +1,136 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import com.kingsrook.qqq.backend.core.actions.interfaces.CountInterface;
+import com.kingsrook.qqq.backend.core.actions.tables.helpers.ActionTimeoutHelper;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import com.mongodb.client.model.Accumulators;
+import com.mongodb.client.model.Aggregates;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MongoDBCountAction extends AbstractMongoDBAction implements CountInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBCountAction.class);
+
+   private ActionTimeoutHelper actionTimeoutHelper;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public CountOutput execute(CountInput countInput) throws QException
+   {
+      MongoClientContainer mongoClientContainer = null;
+
+      Long       queryStartTime = System.currentTimeMillis();
+      List<Bson> queryToLog     = new ArrayList<>();
+
+      try
+      {
+         CountOutput            countOutput      = new CountOutput();
+         QTableMetaData         table            = countInput.getTable();
+         String                 backendTableName = getBackendTableName(table);
+         MongoDBBackendMetaData backend          = (MongoDBBackendMetaData) countInput.getBackend();
+
+         mongoClientContainer = openClient(backend, null); // todo - count input has no transaction!?
+         MongoDatabase             database   = mongoClientContainer.getMongoClient().getDatabase(backend.getDatabaseName());
+         MongoCollection<Document> collection = database.getCollection(backendTableName);
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // set up & start an actionTimeoutHelper (note, internally it'll deal with the time being null or negative as meaning not to timeout) //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         actionTimeoutHelper = new ActionTimeoutHelper(countInput.getTimeoutSeconds(), TimeUnit.SECONDS, new TimeoutCanceller(mongoClientContainer));
+         actionTimeoutHelper.start();
+
+         QQueryFilter filter      = countInput.getFilter();
+         Bson         searchQuery = makeSearchQueryDocument(table, filter);
+         queryToLog.add(searchQuery);
+         setQueryInQueryStat(searchQuery);
+
+         List<Bson> bsonList = List.of(
+            Aggregates.match(searchQuery),
+            Aggregates.group("_id", Accumulators.sum("count", 1)));
+
+         AggregateIterable<Document> aggregate = collection.aggregate(mongoClientContainer.getMongoSession(), bsonList);
+
+         Document document = aggregate.first();
+         countOutput.setCount(document == null ? 0 : document.get("count", Integer.class));
+
+         actionTimeoutHelper.cancel();
+         setQueryStatFirstResultTime();
+
+         return (countOutput);
+      }
+      catch(Exception e)
+      {
+         if(actionTimeoutHelper != null && actionTimeoutHelper.getDidTimeout())
+         {
+            setQueryStatFirstResultTime();
+            throw (new QUserFacingException("Count timed out."));
+         }
+
+         /*
+         /////////////////////////////////////////////////////////////////////////////////////
+         // this was copied from RDBMS - not sure where/how/if it's being used there though //
+         /////////////////////////////////////////////////////////////////////////////////////
+         if(isCancelled)
+         {
+            throw (new QUserFacingException("Count was cancelled."));
+         }
+         */
+
+         LOG.warn("Error executing count", e);
+         throw new QException("Error executing count", e);
+      }
+      finally
+      {
+         logQuery(getBackendTableName(countInput.getTable()), "count", queryToLog, queryStartTime);
+
+         if(mongoClientContainer != null)
+         {
+            mongoClientContainer.closeIfNeeded();
+         }
+      }
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBDeleteAction.java

@@ -0,0 +1,137 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.interfaces.DeleteInterface;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.result.DeleteResult;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.bson.types.ObjectId;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MongoDBDeleteAction extends AbstractMongoDBAction implements DeleteInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBDeleteAction.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public boolean supportsQueryFilterInput()
+   {
+      return (true);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public DeleteOutput execute(DeleteInput deleteInput) throws QException
+   {
+      MongoClientContainer mongoClientContainer = null;
+
+      Long       queryStartTime = System.currentTimeMillis();
+      List<Bson> queryToLog     = new ArrayList<>();
+
+      try
+      {
+         DeleteOutput           deleteOutput     = new DeleteOutput();
+         QTableMetaData         table            = deleteInput.getTable();
+         String                 backendTableName = getBackendTableName(table);
+         MongoDBBackendMetaData backend          = (MongoDBBackendMetaData) deleteInput.getBackend();
+
+         mongoClientContainer = openClient(backend, deleteInput.getTransaction());
+         MongoDatabase             database   = mongoClientContainer.getMongoClient().getDatabase(backend.getDatabaseName());
+         MongoCollection<Document> collection = database.getCollection(backendTableName);
+
+         QQueryFilter queryFilter = deleteInput.getQueryFilter();
+         Bson         searchQuery;
+         if(CollectionUtils.nullSafeHasContents(deleteInput.getPrimaryKeys()))
+         {
+            searchQuery = Filters.in("_id", deleteInput.getPrimaryKeys().stream().map(id -> new ObjectId(ValueUtils.getValueAsString(id))).toList());
+         }
+         else if(queryFilter != null && queryFilter.hasAnyCriteria())
+         {
+            QQueryFilter filter = queryFilter;
+            searchQuery = makeSearchQueryDocument(table, filter);
+         }
+         else
+         {
+            LOG.info("Missing both primary keys and a search filter in delete request - exiting with noop", logPair("tableName", table.getName()));
+            return (deleteOutput);
+         }
+
+         queryToLog.add(searchQuery);
+
+         ////////////////////////////////////////////////////////
+         // todo - system property to control (like print-sql) //
+         ////////////////////////////////////////////////////////
+         // LOG.debug(searchQuery);
+
+         DeleteResult deleteResult = collection.deleteMany(mongoClientContainer.getMongoSession(), searchQuery);
+         deleteOutput.setDeletedRecordCount((int) deleteResult.getDeletedCount());
+
+         //////////////////////////////////////////////////////////////////////////
+         // todo any way to get records with errors or warnings for deleteOutput //
+         //////////////////////////////////////////////////////////////////////////
+
+         return (deleteOutput);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error executing delete", e);
+         throw new QException("Error executing delete", e);
+      }
+      finally
+      {
+         logQuery(getBackendTableName(deleteInput.getTable()), "delete", queryToLog, queryStartTime);
+
+         if(mongoClientContainer != null)
+         {
+            mongoClientContainer.closeIfNeeded();
+         }
+      }
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBInsertAction.java

@@ -0,0 +1,146 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.interfaces.InsertInterface;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import com.mongodb.client.result.InsertManyResult;
+import org.bson.BsonValue;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MongoDBInsertAction extends AbstractMongoDBAction implements InsertInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBInsertAction.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public InsertOutput execute(InsertInput insertInput) throws QException
+   {
+      MongoClientContainer mongoClientContainer = null;
+      InsertOutput         rs                   = new InsertOutput();
+      List<QRecord>        outputRecords        = new ArrayList<>();
+      rs.setRecords(outputRecords);
+
+      Long       queryStartTime = System.currentTimeMillis();
+      List<Bson> queryToLog     = new ArrayList<>();
+
+      try
+      {
+         QTableMetaData         table            = insertInput.getTable();
+         String                 backendTableName = getBackendTableName(table);
+         MongoDBBackendMetaData backend          = (MongoDBBackendMetaData) insertInput.getBackend();
+
+         mongoClientContainer = openClient(backend, insertInput.getTransaction());
+         MongoDatabase             database   = mongoClientContainer.getMongoClient().getDatabase(backend.getDatabaseName());
+         MongoCollection<Document> collection = database.getCollection(backendTableName);
+
+         ///////////////////////////////////////////////////////////////////////////
+         // page over input record list (assuming some size of batch is too big?) //
+         ///////////////////////////////////////////////////////////////////////////
+         for(List<QRecord> page : CollectionUtils.getPages(insertInput.getRecords(), getPageSize()))
+         {
+            //////////////////////////////////////////////////////////////////
+            // build list of documents from records w/o errors in this page //
+            //////////////////////////////////////////////////////////////////
+            List<Document> documentList = new ArrayList<>();
+            for(QRecord record : page)
+            {
+               if(CollectionUtils.nullSafeHasContents(record.getErrors()))
+               {
+                  continue;
+               }
+
+               Document document = recordToDocument(table, record);
+               documentList.add(document);
+               queryToLog.add(document);
+            }
+
+            /////////////////////////////////////
+            // skip pages that were all errors //
+            /////////////////////////////////////
+            if(documentList.isEmpty())
+            {
+               continue;
+            }
+
+            ///////////////////////////////////////////////
+            // actually do the insert                    //
+            // todo - how are errors returned by mongo?? //
+            ///////////////////////////////////////////////
+            InsertManyResult insertManyResult = collection.insertMany(mongoClientContainer.getMongoSession(), documentList);
+
+            /////////////////////////////////
+            // put ids on inserted records //
+            /////////////////////////////////
+            int index = 0;
+            for(QRecord record : page)
+            {
+               QRecord outputRecord = new QRecord(record);
+               rs.addRecord(outputRecord);
+
+               if(CollectionUtils.nullSafeIsEmpty(record.getErrors()))
+               {
+                  BsonValue insertedId = insertManyResult.getInsertedIds().get(index++);
+                  String    idString   = insertedId.asObjectId().getValue().toString();
+                  outputRecord.setValue(table.getPrimaryKeyField(), idString);
+               }
+            }
+         }
+      }
+      catch(Exception e)
+      {
+         throw new QException("Error executing insert: " + e.getMessage(), e);
+      }
+      finally
+      {
+         logQuery(getBackendTableName(insertInput.getTable()), "insert", queryToLog, queryStartTime);
+
+         if(mongoClientContainer != null)
+         {
+            mongoClientContainer.closeIfNeeded();
+         }
+      }
+
+      return (rs);
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBQueryAction.java

@@ -0,0 +1,185 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
+import com.kingsrook.qqq.backend.core.actions.tables.helpers.ActionTimeoutHelper;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MongoDBQueryAction extends AbstractMongoDBAction implements QueryInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBQueryAction.class);
+
+   private ActionTimeoutHelper actionTimeoutHelper;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QueryOutput execute(QueryInput queryInput) throws QException
+   {
+      MongoClientContainer mongoClientContainer = null;
+
+      Long       queryStartTime = System.currentTimeMillis();
+      List<Bson> queryToLog     = new ArrayList<>();
+
+      try
+      {
+         QueryOutput            queryOutput      = new QueryOutput(queryInput);
+         QTableMetaData         table            = queryInput.getTable();
+         String                 backendTableName = getBackendTableName(table);
+         MongoDBBackendMetaData backend          = (MongoDBBackendMetaData) queryInput.getBackend();
+
+         mongoClientContainer = openClient(backend, queryInput.getTransaction());
+         MongoDatabase             database   = mongoClientContainer.getMongoClient().getDatabase(backend.getDatabaseName());
+         MongoCollection<Document> collection = database.getCollection(backendTableName);
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // set up & start an actionTimeoutHelper (note, internally it'll deal with the time being null or negative as meaning not to timeout) //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         actionTimeoutHelper = new ActionTimeoutHelper(queryInput.getTimeoutSeconds(), TimeUnit.SECONDS, new TimeoutCanceller(mongoClientContainer));
+         actionTimeoutHelper.start();
+
+         /////////////////////////
+         // set up filter/query //
+         /////////////////////////
+         QQueryFilter filter      = queryInput.getFilter();
+         Bson         searchQuery = makeSearchQueryDocument(table, filter);
+         queryToLog.add(searchQuery);
+         setQueryInQueryStat(searchQuery);
+
+         ////////////////////////////////////////////////////////////
+         // create cursor - further adjustments to it still follow //
+         ////////////////////////////////////////////////////////////
+         FindIterable<Document> cursor = collection.find(mongoClientContainer.getMongoSession(), searchQuery);
+
+         ///////////////////////////////////
+         // add a sort operator if needed //
+         ///////////////////////////////////
+         if(filter != null && CollectionUtils.nullSafeHasContents(filter.getOrderBys()))
+         {
+            Document sortDocument = new Document();
+            queryToLog.add(sortDocument);
+            for(QFilterOrderBy orderBy : filter.getOrderBys())
+            {
+               String fieldBackendName = getFieldBackendName(table.getField(orderBy.getFieldName()));
+               sortDocument.put(fieldBackendName, orderBy.getIsAscending() ? 1 : -1);
+            }
+            cursor.sort(sortDocument);
+         }
+
+         ////////////////////////
+         // apply skip & limit //
+         ////////////////////////
+         if(filter != null)
+         {
+            if(filter.getSkip() != null)
+            {
+               cursor.skip(filter.getSkip());
+            }
+
+            if(filter.getLimit() != null)
+            {
+               cursor.limit(filter.getLimit());
+            }
+         }
+
+         ////////////////////////////////////////////
+         // iterate over results, building records //
+         ////////////////////////////////////////////
+         for(Document document : cursor)
+         {
+            /////////////////////////////////////////////////////////////////////////
+            // once we've started getting results, go ahead and cancel the timeout //
+            /////////////////////////////////////////////////////////////////////////
+            actionTimeoutHelper.cancel();
+            setQueryStatFirstResultTime();
+
+            QRecord record = documentToRecord(table, document);
+            queryOutput.addRecord(record);
+
+            if(queryInput.getAsyncJobCallback().wasCancelRequested())
+            {
+               LOG.info("Breaking query job, as requested.");
+               break;
+            }
+         }
+
+         return (queryOutput);
+      }
+      catch(Exception e)
+      {
+         if(actionTimeoutHelper != null && actionTimeoutHelper.getDidTimeout())
+         {
+            setQueryStatFirstResultTime();
+            throw (new QUserFacingException("Query timed out."));
+         }
+
+         /*
+         /////////////////////////////////////////////////////////////////////////////////////
+         // this was copied from RDBMS - not sure where/how/if it's being used there though //
+         /////////////////////////////////////////////////////////////////////////////////////
+         if(isCancelled)
+         {
+            throw (new QUserFacingException("Query was cancelled."));
+         }
+         */
+
+         LOG.warn("Error executing query", e);
+         throw new QException("Error executing query", e);
+      }
+      finally
+      {
+         logQuery(getBackendTableName(queryInput.getTable()), "query", queryToLog, queryStartTime);
+
+         if(mongoClientContainer != null)
+         {
+            mongoClientContainer.closeIfNeeded();
+         }
+      }
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBTransaction.java

@@ -0,0 +1,215 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.time.Duration;
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.ClientSession;
+import com.mongodb.client.MongoClient;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** MongoDB implementation of backend transaction.
+ **
+ ** Stores a mongoClient and clientSession.
+ **
+ ** Also keeps track of if the specific mongo backend being used supports transactions,
+ ** as, it appears that single-node instances do not, and they throw errors if
+ ** you try to do transaction operations in them...  This is configured by the
+ ** corresponding field in the backend metaData.
+ *******************************************************************************/
+public class MongoDBTransaction extends QBackendTransaction
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBTransaction.class);
+
+   private boolean       transactionsSupported;
+   private MongoClient   mongoClient;
+   private ClientSession clientSession;
+
+   private Instant openedAt                  = Instant.now();
+   private Integer logSlowTransactionSeconds = null;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public MongoDBTransaction(MongoDBBackendMetaData backend, MongoClient mongoClient)
+   {
+      this.transactionsSupported = backend.getTransactionsSupported();
+      ClientSession clientSession = mongoClient.startSession();
+
+      if(transactionsSupported)
+      {
+         clientSession.startTransaction();
+      }
+
+      String propertyName = "qqq.mongodb.logSlowTransactionSeconds";
+      try
+      {
+         logSlowTransactionSeconds = Integer.parseInt(System.getProperty(propertyName, "10"));
+      }
+      catch(Exception e)
+      {
+         LOG.debug("Error reading property [" + propertyName + "] value as integer", e);
+      }
+
+      this.mongoClient = mongoClient;
+      this.clientSession = clientSession;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void commit() throws QException
+   {
+      try
+      {
+         Instant commitAt = Instant.now();
+
+         Duration duration = Duration.between(openedAt, commitAt);
+         if(logSlowTransactionSeconds != null && duration.compareTo(Duration.ofSeconds(logSlowTransactionSeconds)) > 0)
+         {
+            LOG.info("Committing long-running transaction", logPair("durationSeconds", duration.getSeconds()));
+         }
+         else
+         {
+            LOG.debug("Committing transaction");
+         }
+
+         if(transactionsSupported)
+         {
+            this.clientSession.commitTransaction();
+            LOG.debug("Commit complete");
+         }
+         else
+         {
+            LOG.debug("Request to commit, but transactions not supported in this mongodb backend");
+         }
+      }
+      catch(Exception e)
+      {
+         LOG.error("Error committing transaction", e);
+         throw new QException("Error committing transaction: " + e.getMessage(), e);
+      }
+      finally
+      {
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // reset this - as after one commit, the transaction is essentially re-opened for any future statements that run on it //
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         openedAt = Instant.now();
+         if(transactionsSupported)
+         {
+            this.clientSession.startTransaction();
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void rollback() throws QException
+   {
+      try
+      {
+         if(transactionsSupported)
+         {
+            LOG.info("Rolling back transaction");
+            this.clientSession.abortTransaction();
+            LOG.info("Rollback complete");
+         }
+         else
+         {
+            LOG.debug("Request to rollback, but transactions not supported in this mongodb backend");
+         }
+      }
+      catch(Exception e)
+      {
+         LOG.error("Error rolling back transaction", e);
+         throw new QException("Error rolling back transaction: " + e.getMessage(), e);
+      }
+      finally
+      {
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // reset this - as after one commit, the transaction is essentially re-opened for any future statements that run on it //
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         openedAt = Instant.now();
+         if(transactionsSupported)
+         {
+            this.clientSession.startTransaction();
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void close()
+   {
+      try
+      {
+         this.clientSession.close();
+         this.mongoClient.close();
+      }
+      catch(Exception e)
+      {
+         LOG.error("Error closing connection - possible mongo connection leak", e);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for mongoClient
+    **
+    *******************************************************************************/
+   public MongoClient getMongoClient()
+   {
+      return mongoClient;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for clientSession
+    **
+    *******************************************************************************/
+   public ClientSession getClientSession()
+   {
+      return clientSession;
+   }
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBUpdateAction.java

@@ -0,0 +1,169 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.interfaces.UpdateInterface;
+import com.kingsrook.qqq.backend.core.actions.tables.helpers.UpdateActionRecordSplitHelper;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.ListingHash;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import com.mongodb.client.model.Filters;
+import com.mongodb.client.model.Updates;
+import com.mongodb.client.result.UpdateResult;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+import org.bson.types.ObjectId;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class MongoDBUpdateAction extends AbstractMongoDBAction implements UpdateInterface
+{
+   private static final QLogger LOG = QLogger.getLogger(MongoDBUpdateAction.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public UpdateOutput execute(UpdateInput updateInput) throws QException
+   {
+      MongoClientContainer   mongoClientContainer = null;
+      QTableMetaData         table                = updateInput.getTable();
+      String                 backendTableName     = getBackendTableName(table);
+      MongoDBBackendMetaData backend              = (MongoDBBackendMetaData) updateInput.getBackend();
+
+      UpdateActionRecordSplitHelper updateActionRecordSplitHelper = new UpdateActionRecordSplitHelper();
+      updateActionRecordSplitHelper.init(updateInput);
+
+      UpdateOutput rs = new UpdateOutput();
+      rs.setRecords(updateActionRecordSplitHelper.getOutputRecords());
+
+      if(!updateActionRecordSplitHelper.getHaveAnyWithoutErrors())
+      {
+         LOG.info("Exiting early - all records have some error.");
+         return (rs);
+      }
+
+      try
+      {
+         mongoClientContainer = openClient(backend, updateInput.getTransaction());
+         MongoDatabase             database   = mongoClientContainer.getMongoClient().getDatabase(backend.getDatabaseName());
+         MongoCollection<Document> collection = database.getCollection(backendTableName);
+
+         /////////////////////////////////////////////////////////////////////////////////////////////
+         // process each distinct list of fields being updated (e.g., each different SQL statement) //
+         /////////////////////////////////////////////////////////////////////////////////////////////
+         ListingHash<List<String>, QRecord> recordsByFieldBeingUpdated = updateActionRecordSplitHelper.getRecordsByFieldBeingUpdated();
+         for(Map.Entry<List<String>, List<QRecord>> entry : recordsByFieldBeingUpdated.entrySet())
+         {
+            updateRecordsWithMatchingListOfFields(updateInput, mongoClientContainer, collection, table, entry.getValue(), entry.getKey());
+         }
+      }
+      catch(Exception e)
+      {
+         throw new QException("Error executing update: " + e.getMessage(), e);
+      }
+      finally
+      {
+         if(mongoClientContainer != null)
+         {
+            mongoClientContainer.closeIfNeeded();
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private void updateRecordsWithMatchingListOfFields(UpdateInput updateInput, MongoClientContainer mongoClientContainer, MongoCollection<Document> collection, QTableMetaData table, List<QRecord> recordList, List<String> fieldsBeingUpdated)
+   {
+      boolean allAreTheSame = UpdateActionRecordSplitHelper.areAllValuesBeingUpdatedTheSame(updateInput, recordList, fieldsBeingUpdated);
+      if(allAreTheSame)
+      {
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // if all records w/ this set of fields have the same values, we can do 1 big updateMany on the whole list //
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         updateRecordsWithMatchingValuesAndFields(mongoClientContainer, collection, table, recordList, fieldsBeingUpdated);
+      }
+      else
+      {
+         /////////////////////////////////////////////////////////////////////////
+         // else, if not all are being updated the same, then update one-by-one //
+         /////////////////////////////////////////////////////////////////////////
+         for(QRecord record : recordList)
+         {
+            updateRecordsWithMatchingValuesAndFields(mongoClientContainer, collection, table, List.of(record), fieldsBeingUpdated);
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private void updateRecordsWithMatchingValuesAndFields(MongoClientContainer mongoClientContainer, MongoCollection<Document> collection, QTableMetaData table, List<QRecord> recordList, List<String> fieldsBeingUpdated)
+   {
+      Long       queryStartTime = System.currentTimeMillis();
+      List<Bson> queryToLog     = new ArrayList<>();
+
+      QRecord        firstRecord = recordList.get(0);
+      List<ObjectId> ids         = recordList.stream().map(r -> new ObjectId(r.getValueString("id"))).toList();
+      Bson           filter      = Filters.in("_id", ids);
+      queryToLog.add(filter);
+
+      List<Bson> updates = new ArrayList<>();
+      for(String fieldName : fieldsBeingUpdated)
+      {
+         QFieldMetaData field            = table.getField(fieldName);
+         String         fieldBackendName = getFieldBackendName(field);
+         Bson set = Updates.set(fieldBackendName, firstRecord.getValue(fieldName));
+         updates.add(set);
+         queryToLog.add(set);
+      }
+      Bson changes = Updates.combine(updates);
+
+      UpdateResult updateResult = collection.updateMany(mongoClientContainer.getMongoSession(), filter, changes);
+      // todo - anything with the output??
+
+      logQuery(getBackendTableName(table), "update", queryToLog, queryStartTime);
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/actions/TimeoutCanceller.java

@@ -1,6 +1,6 @@
 /*
  * QQQ - Low-code Application Framework for Engineers.
- * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
  * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
  * contact@kingsrook.com
  * https://github.com/Kingsrook/
@@ -19,26 +19,50 @@
  * along with this program.  If not, see <https://www.gnu.org/licenses/>.
  */
 
-package com.kingsrook.qqq.backend.core.actions.interfaces;
+package com.kingsrook.qqq.backend.module.mongodb.actions;
 
 
-import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
-import com.kingsrook.qqq.backend.core.exceptions.QException;
-import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
+import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
 
 
 /*******************************************************************************
+ ** Helper to cancel statements that timeout.
+ *******************************************************************************/
+public class TimeoutCanceller implements Runnable
+{
+   private static final QLogger              LOG = QLogger.getLogger(TimeoutCanceller.class);
+   private final        MongoClientContainer mongoClientContainer;
+
+
+
+   /*******************************************************************************
+    ** Constructor
     **
     *******************************************************************************/
-public interface QActionInterface
-{
+   public TimeoutCanceller(MongoClientContainer mongoClientContainer)
+   {
+      this.mongoClientContainer = mongoClientContainer;
+   }
+
+
 
    /*******************************************************************************
     **
     *******************************************************************************/
-   default QBackendTransaction openTransaction(AbstractTableActionInput input) throws QException
+   @Override
+   public void run()
    {
-      return (new QBackendTransaction());
+      try
+      {
+         mongoClientContainer.closeIfNeeded();
+         LOG.info("Cancelled timed out query");
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error trying to cancel statement after timeout", e);
       }
 
+      throw (new QRuntimeException("Statement timed out and was cancelled."));
+   }
 }

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/model/metadata/MongoDBBackendMetaData.java

@@ -0,0 +1,343 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.model.metadata;
+
+
+import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.module.mongodb.MongoDBBackendModule;
+
+
+/*******************************************************************************
+ ** Meta-data to provide details of a MongoDB backend (e.g., connection params)
+ *******************************************************************************/
+public class MongoDBBackendMetaData extends QBackendMetaData
+{
+   private String  host;
+   private Integer port;
+   private String  databaseName;
+   private String  username;
+   private String  password;
+   private String  authSourceDatabase;
+   private String  urlSuffix;
+
+   private boolean transactionsSupported = true;
+
+
+
+   /*******************************************************************************
+    ** Default Constructor.
+    *******************************************************************************/
+   public MongoDBBackendMetaData()
+   {
+      super();
+      setBackendType(MongoDBBackendModule.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter, override to help fluent flows
+    *******************************************************************************/
+   @Override
+   public MongoDBBackendMetaData withName(String name)
+   {
+      setName(name);
+      return this;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for host
+    **
+    *******************************************************************************/
+   public String getHost()
+   {
+      return host;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for host
+    **
+    *******************************************************************************/
+   public void setHost(String host)
+   {
+      this.host = host;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent Setter for host
+    **
+    *******************************************************************************/
+   public MongoDBBackendMetaData withHost(String host)
+   {
+      this.host = host;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for port
+    **
+    *******************************************************************************/
+   public Integer getPort()
+   {
+      return port;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for port
+    **
+    *******************************************************************************/
+   public void setPort(Integer port)
+   {
+      this.port = port;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent Setter for port
+    **
+    *******************************************************************************/
+   public MongoDBBackendMetaData withPort(Integer port)
+   {
+      this.port = port;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for username
+    **
+    *******************************************************************************/
+   public String getUsername()
+   {
+      return username;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for username
+    **
+    *******************************************************************************/
+   public void setUsername(String username)
+   {
+      this.username = username;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent Setter for username
+    **
+    *******************************************************************************/
+   public MongoDBBackendMetaData withUsername(String username)
+   {
+      this.username = username;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for password
+    **
+    *******************************************************************************/
+   public String getPassword()
+   {
+      return password;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for password
+    **
+    *******************************************************************************/
+   public void setPassword(String password)
+   {
+      this.password = password;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent Setter for password
+    **
+    *******************************************************************************/
+   public MongoDBBackendMetaData withPassword(String password)
+   {
+      this.password = password;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Called by the QInstanceEnricher - to do backend-type-specific enrichments.
+    ** Original use case is:  reading secrets into fields (e.g., passwords).
+    *******************************************************************************/
+   @Override
+   public void enrich()
+   {
+      super.enrich();
+      QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter();
+      username = interpreter.interpret(username);
+      password = interpreter.interpret(password);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for urlSuffix
+    *******************************************************************************/
+   public String getUrlSuffix()
+   {
+      return (this.urlSuffix);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for urlSuffix
+    *******************************************************************************/
+   public void setUrlSuffix(String urlSuffix)
+   {
+      this.urlSuffix = urlSuffix;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for urlSuffix
+    *******************************************************************************/
+   public MongoDBBackendMetaData withUrlSuffix(String urlSuffix)
+   {
+      this.urlSuffix = urlSuffix;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for databaseName
+    *******************************************************************************/
+   public String getDatabaseName()
+   {
+      return (this.databaseName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for databaseName
+    *******************************************************************************/
+   public void setDatabaseName(String databaseName)
+   {
+      this.databaseName = databaseName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for databaseName
+    *******************************************************************************/
+   public MongoDBBackendMetaData withDatabaseName(String databaseName)
+   {
+      this.databaseName = databaseName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for transactionsSupported
+    *******************************************************************************/
+   public boolean getTransactionsSupported()
+   {
+      return (this.transactionsSupported);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for transactionsSupported
+    *******************************************************************************/
+   public void setTransactionsSupported(boolean transactionsSupported)
+   {
+      this.transactionsSupported = transactionsSupported;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for transactionsSupported
+    *******************************************************************************/
+   public MongoDBBackendMetaData withTransactionsSupported(boolean transactionsSupported)
+   {
+      this.transactionsSupported = transactionsSupported;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for authSourceDatabase
+    *******************************************************************************/
+   public String getAuthSourceDatabase()
+   {
+      return (this.authSourceDatabase);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for authSourceDatabase
+    *******************************************************************************/
+   public void setAuthSourceDatabase(String authSourceDatabase)
+   {
+      this.authSourceDatabase = authSourceDatabase;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for authSourceDatabase
+    *******************************************************************************/
+   public MongoDBBackendMetaData withAuthSourceDatabase(String authSourceDatabase)
+   {
+      this.authSourceDatabase = authSourceDatabase;
+      return (this);
+   }
+
+}

qqq-backend-module-mongodb/src/main/java/com/kingsrook/qqq/backend/module/mongodb/model/metadata/MongoDBTableBackendDetails.java

@@ -0,0 +1,81 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.model.metadata;
+
+
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableBackendDetails;
+import com.kingsrook.qqq.backend.module.mongodb.MongoDBBackendModule;
+
+
+/*******************************************************************************
+ ** Extension of QTableBackendDetails, with details specific to a MongoDB table.
+ *******************************************************************************/
+public class MongoDBTableBackendDetails extends QTableBackendDetails
+{
+   private String tableName;
+
+
+
+   /*******************************************************************************
+    ** Default Constructor.
+    *******************************************************************************/
+   public MongoDBTableBackendDetails()
+   {
+      super();
+      setBackendType(MongoDBBackendModule.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    **
+    *******************************************************************************/
+   public String getTableName()
+   {
+      return tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    **
+    *******************************************************************************/
+   public void setTableName(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent Setter for tableName
+    **
+    *******************************************************************************/
+   public MongoDBTableBackendDetails withTableName(String tableName)
+   {
+      this.tableName = tableName;
+      return (this);
+   }
+
+}

qqq-backend-module-mongodb/src/test/java/com/kingsrook/qqq/backend/module/mongodb/BaseTest.java

@@ -0,0 +1,157 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb;
+
+
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.module.mongodb.actions.AbstractMongoDBAction;
+import com.kingsrook.qqq.backend.module.mongodb.actions.MongoClientContainer;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.mongodb.client.MongoClient;
+import com.mongodb.client.MongoDatabase;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.utility.DockerImageName;
+
+
+/*******************************************************************************
+ ** Base for all tests in this module
+ *******************************************************************************/
+public class BaseTest
+{
+   private static final QLogger LOG = QLogger.getLogger(BaseTest.class);
+
+   private static GenericContainer<?> mongoDBContainer;
+
+   private static final String MONGO_IMAGE = "mongo:4.2.0-bionic";
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeAll
+   static void beforeAll()
+   {
+      System.setProperty("qqq.mongodb.logQueries", "true");
+
+      mongoDBContainer = new GenericContainer<>(DockerImageName.parse(MONGO_IMAGE))
+         .withEnv("MONGO_INITDB_ROOT_USERNAME", TestUtils.MONGO_USERNAME)
+         .withEnv("MONGO_INITDB_ROOT_PASSWORD", TestUtils.MONGO_PASSWORD)
+         .withEnv("MONGO_INITDB_DATABASE", TestUtils.MONGO_DATABASE)
+         .withExposedPorts(TestUtils.MONGO_PORT);
+
+      mongoDBContainer.start();
+   }
+
+
+
+   /*******************************************************************************
+    ** init the QContext with the instance from TestUtils and a new session
+    *******************************************************************************/
+   @BeforeEach
+   void baseBeforeEach()
+   {
+      QContext.init(TestUtils.defineInstance(), new QSession());
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // host could(?) be different, and mapped port will be, so set them in backend meta-data based on our running container //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      MongoDBBackendMetaData backend = (MongoDBBackendMetaData) QContext.getQInstance().getBackend(TestUtils.DEFAULT_BACKEND_NAME);
+      backend.setHost(mongoDBContainer.getHost());
+      backend.setPort(mongoDBContainer.getMappedPort(TestUtils.MONGO_PORT));
+   }
+
+
+
+   /*******************************************************************************
+    ** clear the QContext
+    *******************************************************************************/
+   @AfterEach
+   void baseAfterEach()
+   {
+      clearDatabase();
+
+      QContext.clear();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected static void clearDatabase()
+   {
+      ///////////////////////////////////////
+      // clear test database between tests //
+      ///////////////////////////////////////
+      MongoClient   mongoClient = getMongoClient();
+      MongoDatabase database    = mongoClient.getDatabase(TestUtils.MONGO_DATABASE);
+      database.drop();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected static MongoClient getMongoClient()
+   {
+      MongoDBBackendMetaData backend              = (MongoDBBackendMetaData) QContext.getQInstance().getBackend(TestUtils.DEFAULT_BACKEND_NAME);
+      MongoClientContainer   mongoClientContainer = new AbstractMongoDBAction().openClient(backend, null);
+      MongoClient            mongoClient          = mongoClientContainer.getMongoClient();
+      return mongoClient;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterAll
+   static void afterAll()
+   {
+      // this.mongoDbReplicaSet.close();
+      mongoDBContainer.close();
+   }
+
+
+
+   /*******************************************************************************
+    ** if needed, re-initialize the QInstance in context.
+    *******************************************************************************/
+   protected static void reInitInstanceInContext(QInstance qInstance)
+   {
+      if(qInstance.equals(QContext.getQInstance()))
+      {
+         LOG.warn("Unexpected condition - the same qInstance that is already in the QContext was passed into reInit.  You probably want a new QInstance object instance.");
+      }
+      QContext.init(qInstance, new QSession());
+   }
+
+}

qqq-backend-module-mongodb/src/test/java/com/kingsrook/qqq/backend/module/mongodb/TestUtils.java

@@ -0,0 +1,353 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PVSValueFormatAndFields;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
+import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType;
+import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.ExposedJoin;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBBackendMetaData;
+import com.kingsrook.qqq.backend.module.mongodb.model.metadata.MongoDBTableBackendDetails;
+
+
+/*******************************************************************************
+ ** Test Utils class for this module
+ **
+ ** Note - tons of copying from RDMBS... wouldn't it be nice to share??
+ *******************************************************************************/
+public class TestUtils
+{
+   public static final String DEFAULT_BACKEND_NAME = "default";
+
+   public static final String TABLE_NAME_PERSON = "personTable";
+
+   public static final String TABLE_NAME_STORE               = "store";
+   public static final String TABLE_NAME_ORDER               = "order";
+   public static final String TABLE_NAME_ORDER_INSTRUCTIONS  = "orderInstructions";
+   public static final String TABLE_NAME_ITEM                = "item";
+   public static final String TABLE_NAME_ORDER_LINE          = "orderLine";
+   public static final String TABLE_NAME_LINE_ITEM_EXTRINSIC = "orderLineExtrinsic";
+   public static final String TABLE_NAME_WAREHOUSE           = "warehouse";
+   public static final String TABLE_NAME_WAREHOUSE_STORE_INT = "warehouseStoreInt";
+
+   public static final String SECURITY_KEY_STORE_ALL_ACCESS = "storeAllAccess";
+
+   public static final String  MONGO_USERNAME = "mongoUser";
+   public static final String  MONGO_PASSWORD = "password";
+   public static final Integer MONGO_PORT     = 27017;
+   public static final String  MONGO_DATABASE = "testDatabase";
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QInstance defineInstance()
+   {
+      QInstance qInstance = new QInstance();
+      qInstance.addBackend(defineBackend());
+      qInstance.addTable(defineTablePerson());
+      qInstance.addPossibleValueSource(definePvsPerson());
+      addOmsTablesAndJoins(qInstance);
+      qInstance.setAuthentication(defineAuthentication());
+      return (qInstance);
+   }
+
+
+
+   /*******************************************************************************
+    ** Define the authentication used in standard tests - using 'mock' type.
+    **
+    *******************************************************************************/
+   public static QAuthenticationMetaData defineAuthentication()
+   {
+      return new QAuthenticationMetaData()
+         .withName("mock")
+         .withType(QAuthenticationType.MOCK);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static MongoDBBackendMetaData defineBackend()
+   {
+      return (new MongoDBBackendMetaData()
+         .withName(DEFAULT_BACKEND_NAME)
+         .withHost("localhost")
+         .withPort(TestUtils.MONGO_PORT)
+         .withUsername(TestUtils.MONGO_USERNAME)
+         .withPassword(TestUtils.MONGO_PASSWORD)
+         .withAuthSourceDatabase("admin")
+         .withDatabaseName(TestUtils.MONGO_DATABASE)
+         .withTransactionsSupported(false));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QTableMetaData defineTablePerson()
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME_PERSON)
+         .withLabel("Person")
+         .withRecordLabelFormat("%s %s")
+         .withRecordLabelFields("firstName", "lastName")
+         .withBackendName(DEFAULT_BACKEND_NAME)
+         .withPrimaryKeyField("id")
+         .withField(new QFieldMetaData("id", QFieldType.STRING).withBackendName("_id"))
+         .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("metaData.createDate"))
+         .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBackendName("metaData.modifyDate"))
+         .withField(new QFieldMetaData("seqNo", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("firstName", QFieldType.STRING))
+         .withField(new QFieldMetaData("lastName", QFieldType.STRING))
+         .withField(new QFieldMetaData("birthDate", QFieldType.DATE))
+         .withField(new QFieldMetaData("email", QFieldType.STRING))
+         .withField(new QFieldMetaData("isEmployed", QFieldType.BOOLEAN))
+         .withField(new QFieldMetaData("annualSalary", QFieldType.DECIMAL))
+         .withField(new QFieldMetaData("daysWorked", QFieldType.INTEGER))
+         .withField(new QFieldMetaData("homeTown", QFieldType.STRING))
+         .withBackendDetails(new MongoDBTableBackendDetails()
+            .withTableName(TABLE_NAME_PERSON));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static QPossibleValueSource definePvsPerson()
+   {
+      return (new QPossibleValueSource()
+         .withName(TABLE_NAME_PERSON)
+         .withType(QPossibleValueSourceType.TABLE)
+         .withTableName(TABLE_NAME_PERSON)
+         .withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY)
+      );
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void addOmsTablesAndJoins(QInstance qInstance)
+   {
+      qInstance.addTable(defineBaseTable(TABLE_NAME_STORE, "store")
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("name")
+         .withRecordSecurityLock(new RecordSecurityLock().withSecurityKeyType(TABLE_NAME_STORE).withFieldName("key"))
+         .withField(new QFieldMetaData("name", QFieldType.STRING))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_ORDER, "order")
+         .withRecordSecurityLock(new RecordSecurityLock().withSecurityKeyType(TABLE_NAME_STORE).withFieldName("storeKey"))
+         .withAssociation(new Association().withName("orderLine").withAssociatedTableName(TABLE_NAME_ORDER_LINE).withJoinName("orderJoinOrderLine"))
+         .withExposedJoin(new ExposedJoin().withJoinTable(TABLE_NAME_ITEM).withJoinPath(List.of("orderJoinOrderLine", "orderLineJoinItem")))
+         .withField(new QFieldMetaData("storeKey", QFieldType.INTEGER).withPossibleValueSourceName(TABLE_NAME_STORE))
+         .withField(new QFieldMetaData("billToPersonId", QFieldType.STRING).withPossibleValueSourceName(TABLE_NAME_PERSON))
+         .withField(new QFieldMetaData("shipToPersonId", QFieldType.STRING).withPossibleValueSourceName(TABLE_NAME_PERSON))
+         .withField(new QFieldMetaData("currentOrderInstructionsId", QFieldType.STRING).withPossibleValueSourceName(TABLE_NAME_PERSON))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_ORDER_INSTRUCTIONS, "order_instructions")
+         .withRecordSecurityLock(new RecordSecurityLock()
+            .withSecurityKeyType(TABLE_NAME_STORE)
+            .withFieldName("order.storeKey")
+            .withJoinNameChain(List.of("orderInstructionsJoinOrder")))
+         .withField(new QFieldMetaData("orderId", QFieldType.STRING))
+         .withField(new QFieldMetaData("instructions", QFieldType.STRING))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_ITEM, "item")
+         .withRecordSecurityLock(new RecordSecurityLock().withSecurityKeyType(TABLE_NAME_STORE).withFieldName("storeKey"))
+         .withExposedJoin(new ExposedJoin().withJoinTable(TABLE_NAME_ORDER).withJoinPath(List.of("orderLineJoinItem", "orderJoinOrderLine")))
+         .withField(new QFieldMetaData("sku", QFieldType.STRING))
+         .withField(new QFieldMetaData("description", QFieldType.STRING))
+         .withField(new QFieldMetaData("storeKey", QFieldType.INTEGER).withPossibleValueSourceName(TABLE_NAME_STORE))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_ORDER_LINE, "order_line")
+         .withRecordSecurityLock(new RecordSecurityLock()
+            .withSecurityKeyType(TABLE_NAME_STORE)
+            .withFieldName("order.storeKey")
+            .withJoinNameChain(List.of("orderJoinOrderLine")))
+         .withAssociation(new Association().withName("extrinsics").withAssociatedTableName(TABLE_NAME_LINE_ITEM_EXTRINSIC).withJoinName("orderLineJoinLineItemExtrinsic"))
+         .withField(new QFieldMetaData("orderId", QFieldType.STRING))
+         .withField(new QFieldMetaData("sku", QFieldType.STRING))
+         .withField(new QFieldMetaData("storeKey", QFieldType.INTEGER).withPossibleValueSourceName(TABLE_NAME_STORE))
+         .withField(new QFieldMetaData("quantity", QFieldType.INTEGER))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_LINE_ITEM_EXTRINSIC, "line_item_extrinsic")
+         .withRecordSecurityLock(new RecordSecurityLock()
+            .withSecurityKeyType(TABLE_NAME_STORE)
+            .withFieldName("order.storeKey")
+            .withJoinNameChain(List.of("orderJoinOrderLine", "orderLineJoinLineItemExtrinsic")))
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
+         .withField(new QFieldMetaData("orderLineId", QFieldType.STRING))
+         .withField(new QFieldMetaData("key", QFieldType.STRING))
+         .withField(new QFieldMetaData("value", QFieldType.STRING))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_WAREHOUSE_STORE_INT, "warehouse_store_int")
+         .withField(new QFieldMetaData("warehouseId", QFieldType.STRING))
+         .withField(new QFieldMetaData("storeKey", QFieldType.INTEGER))
+      );
+
+      qInstance.addTable(defineBaseTable(TABLE_NAME_WAREHOUSE, "warehouse")
+         .withRecordSecurityLock(new RecordSecurityLock()
+            .withSecurityKeyType(TABLE_NAME_STORE)
+            .withFieldName(TABLE_NAME_WAREHOUSE_STORE_INT + ".storeKey")
+            .withJoinNameChain(List.of(QJoinMetaData.makeInferredJoinName(TestUtils.TABLE_NAME_WAREHOUSE, TestUtils.TABLE_NAME_WAREHOUSE_STORE_INT)))
+         )
+         .withField(new QFieldMetaData("name", QFieldType.STRING))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withType(JoinType.ONE_TO_MANY)
+         .withLeftTable(TestUtils.TABLE_NAME_WAREHOUSE)
+         .withRightTable(TestUtils.TABLE_NAME_WAREHOUSE_STORE_INT)
+         .withInferredName()
+         .withJoinOn(new JoinOn("id", "warehouseId"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderJoinStore")
+         .withLeftTable(TABLE_NAME_ORDER)
+         .withRightTable(TABLE_NAME_STORE)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("storeKey", "key"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderJoinBillToPerson")
+         .withLeftTable(TABLE_NAME_ORDER)
+         .withRightTable(TABLE_NAME_PERSON)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("billToPersonId", "id"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderJoinShipToPerson")
+         .withLeftTable(TABLE_NAME_ORDER)
+         .withRightTable(TABLE_NAME_PERSON)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("shipToPersonId", "id"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("itemJoinStore")
+         .withLeftTable(TABLE_NAME_ITEM)
+         .withRightTable(TABLE_NAME_STORE)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("storeKey", "key"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderJoinOrderLine")
+         .withLeftTable(TABLE_NAME_ORDER)
+         .withRightTable(TABLE_NAME_ORDER_LINE)
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn("id", "orderId"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderLineJoinItem")
+         .withLeftTable(TABLE_NAME_ORDER_LINE)
+         .withRightTable(TABLE_NAME_ITEM)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("sku", "sku"))
+         .withJoinOn(new JoinOn("storeKey", "storeKey"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderLineJoinLineItemExtrinsic")
+         .withLeftTable(TABLE_NAME_ORDER_LINE)
+         .withRightTable(TABLE_NAME_LINE_ITEM_EXTRINSIC)
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn("id", "orderLineId"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderJoinCurrentOrderInstructions")
+         .withLeftTable(TABLE_NAME_ORDER)
+         .withRightTable(TABLE_NAME_ORDER_INSTRUCTIONS)
+         .withType(JoinType.ONE_TO_ONE)
+         .withJoinOn(new JoinOn("currentOrderInstructionsId", "id"))
+      );
+
+      qInstance.addJoin(new QJoinMetaData()
+         .withName("orderInstructionsJoinOrder")
+         .withLeftTable(TABLE_NAME_ORDER_INSTRUCTIONS)
+         .withRightTable(TABLE_NAME_ORDER)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("orderId", "id"))
+      );
+
+      qInstance.addPossibleValueSource(new QPossibleValueSource()
+         .withName("store")
+         .withType(QPossibleValueSourceType.TABLE)
+         .withTableName(TABLE_NAME_STORE)
+         .withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY)
+      );
+
+      qInstance.addSecurityKeyType(new QSecurityKeyType()
+         .withName(TABLE_NAME_STORE)
+         .withAllAccessKeyName(SECURITY_KEY_STORE_ALL_ACCESS)
+         .withPossibleValueSourceName(TABLE_NAME_STORE));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static QTableMetaData defineBaseTable(String tableName, String backendTableName)
+   {
+      return new QTableMetaData()
+         .withName(tableName)
+         .withBackendName(DEFAULT_BACKEND_NAME)
+         .withBackendDetails(new MongoDBTableBackendDetails().withTableName(backendTableName))
+         .withPrimaryKeyField("id")
+         .withField(new QFieldMetaData("id", QFieldType.STRING))
+         .withField(new QFieldMetaData("key", QFieldType.INTEGER));
+   }
+
+}

qqq-backend-module-mongodb/src/test/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBAggregateActionTest.java

@@ -0,0 +1,93 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.tables.AggregateAction;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.Aggregate;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.GroupBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.QFilterOrderByAggregate;
+import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.QFilterOrderByGroupBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.module.mongodb.BaseTest;
+import com.kingsrook.qqq.backend.module.mongodb.TestUtils;
+import org.junit.jupiter.api.Test;
+
+
+/*******************************************************************************
+ ** Unit test for MongoDBQueryAction 
+ *******************************************************************************/
+class MongoDBAggregateActionTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      InsertInput insertInput = new InsertInput();
+      insertInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+      insertInput.setRecords(List.of(
+         new QRecord().withValue("firstName", "Darin").withValue("lastName", "Kelkhoff").withValue("isEmployed", true).withValue("annualSalary", 1),
+         new QRecord().withValue("firstName", "Linda").withValue("lastName", "Kelkhoff").withValue("isEmployed", true).withValue("annualSalary", 5),
+         new QRecord().withValue("firstName", "Tim").withValue("lastName", "Chamberlain").withValue("isEmployed", true).withValue("annualSalary", 3),
+         new QRecord().withValue("firstName", "James").withValue("lastName", "Maes").withValue("isEmployed", true).withValue("annualSalary", 5),
+         new QRecord().withValue("firstName", "J.D.").withValue("lastName", "Maes").withValue("isEmployed", false).withValue("annualSalary", 0)
+      ));
+      new InsertAction().execute(insertInput);
+
+      {
+         AggregateInput aggregateInput = new AggregateInput();
+         aggregateInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+         aggregateInput.setFilter(new QQueryFilter()
+            .withOrderBy(new QFilterOrderByAggregate(new Aggregate("annualSalary", AggregateOperator.MAX)).withIsAscending(false))
+            .withOrderBy(new QFilterOrderByGroupBy(new GroupBy(QFieldType.STRING, "lastName")))
+         );
+         aggregateInput.withAggregate(new Aggregate("id", AggregateOperator.COUNT));
+         aggregateInput.withAggregate(new Aggregate("annualSalary", AggregateOperator.SUM));
+         aggregateInput.withAggregate(new Aggregate("annualSalary", AggregateOperator.MAX));
+         aggregateInput.withGroupBy(new GroupBy(QFieldType.STRING, "lastName"));
+         aggregateInput.withGroupBy(new GroupBy(QFieldType.BOOLEAN, "isEmployed"));
+         AggregateOutput aggregateOutput = new AggregateAction().execute(aggregateInput);
+         // todo - actual assertions
+      }
+      {
+         AggregateInput aggregateInput = new AggregateInput();
+         aggregateInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+         aggregateInput.withAggregate(new Aggregate("id", AggregateOperator.COUNT));
+         aggregateInput.withAggregate(new Aggregate("annualSalary", AggregateOperator.AVG));
+         AggregateOutput aggregateOutput = new AggregateAction().execute(aggregateInput);
+         // todo - actual assertions
+      }
+   }
+
+}

qqq-backend-module-mongodb/src/test/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBCountActionTest.java

@@ -0,0 +1,80 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.module.mongodb.BaseTest;
+import com.kingsrook.qqq.backend.module.mongodb.TestUtils;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import org.bson.Document;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for MongoDBQueryAction 
+ *******************************************************************************/
+class MongoDBCountActionTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      ////////////////////////////////////////
+      // directly insert some mongo records //
+      ////////////////////////////////////////
+      MongoDatabase             database   = getMongoClient().getDatabase(TestUtils.MONGO_DATABASE);
+      MongoCollection<Document> collection = database.getCollection(TestUtils.TABLE_NAME_PERSON);
+      collection.insertMany(List.of(
+         Document.parse("""
+            {"firstName": "Darin", "lastName": "Kelkhoff"}"""),
+         Document.parse("""
+            {"firstName": "Tylers", "lastName": "Sample"}"""),
+         Document.parse("""
+            {"firstName": "Tylers", "lastName": "Simple"}"""),
+         Document.parse("""
+            {"firstName": "Thom", "lastName": "Chutterloin"}""")
+      ));
+
+      CountInput countInput = new CountInput();
+      countInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+      assertEquals(4, new CountAction().execute(countInput).getCount());
+
+      countInput.setFilter(new QQueryFilter(new QFilterCriteria("firstName", QCriteriaOperator.EQUALS, "Tylers")));
+      assertEquals(2, new CountAction().execute(countInput).getCount());
+
+      countInput.setFilter(new QQueryFilter(new QFilterCriteria("firstName", QCriteriaOperator.EQUALS, "assdf")));
+      assertEquals(0, new CountAction().execute(countInput).getCount());
+   }
+
+}

qqq-backend-module-mongodb/src/test/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBDeleteActionTest.java

@@ -0,0 +1,102 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.module.mongodb.BaseTest;
+import com.kingsrook.qqq.backend.module.mongodb.TestUtils;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import org.bson.Document;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for MongoDBQueryAction 
+ *******************************************************************************/
+class MongoDBDeleteActionTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      ////////////////////////////////////////
+      // directly insert some mongo records //
+      ////////////////////////////////////////
+      MongoDatabase             database   = getMongoClient().getDatabase(TestUtils.MONGO_DATABASE);
+      MongoCollection<Document> collection = database.getCollection(TestUtils.TABLE_NAME_PERSON);
+      collection.insertMany(List.of(
+         Document.parse("""
+            {"firstName": "Darin", "lastName": "Kelkhoff"}"""),
+         Document.parse("""
+            {"firstName": "Tylers", "lastName": "Sample"}"""),
+         Document.parse("""
+            {"firstName": "Tylers", "lastName": "Simple"}"""),
+         Document.parse("""
+            {"firstName": "Thom", "lastName": "Chutterloin"}""")
+      ));
+      assertEquals(4, collection.countDocuments());
+
+      //////////////////////////////////////////
+      // do a delete by id (look it up first) //
+      //////////////////////////////////////////
+      {
+         QueryInput queryInput = new QueryInput();
+         queryInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+         queryInput.setFilter(new QQueryFilter(new QFilterCriteria("firstName", QCriteriaOperator.EQUALS, "Darin")));
+         QueryOutput queryOutput = new QueryAction().execute(queryInput);
+         String      id0         = queryOutput.getRecords().get(0).getValueString("id");
+
+         DeleteInput deleteInput = new DeleteInput();
+         deleteInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+         deleteInput.setPrimaryKeys(List.of(id0));
+         assertEquals(1, new DeleteAction().execute(deleteInput).getDeletedRecordCount());
+      }
+      assertEquals(3, collection.countDocuments());
+
+      ///////////////////////////
+      // do a delete by filter //
+      ///////////////////////////
+      {
+         DeleteInput deleteInput = new DeleteInput();
+         deleteInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+         deleteInput.setQueryFilter(new QQueryFilter(new QFilterCriteria("firstName", QCriteriaOperator.EQUALS, "Tylers")));
+         assertEquals(2, new DeleteAction().execute(deleteInput).getDeletedRecordCount());
+      }
+      assertEquals(1, collection.countDocuments());
+   }
+
+}

qqq-backend-module-mongodb/src/test/java/com/kingsrook/qqq/backend/module/mongodb/actions/MongoDBInsertActionTest.java

@@ -0,0 +1,101 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.module.mongodb.actions;
+
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.module.mongodb.BaseTest;
+import com.kingsrook.qqq.backend.module.mongodb.TestUtils;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoDatabase;
+import org.bson.Document;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+
+/*******************************************************************************
+ ** Unit test for MongoDBQueryAction 
+ *******************************************************************************/
+class MongoDBInsertActionTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      InsertInput insertInput = new InsertInput();
+      insertInput.setTableName(TestUtils.TABLE_NAME_PERSON);
+      insertInput.setRecords(List.of(
+         new QRecord().withValue("firstName", "Darin")
+            .withValue("unmappedField", 1701)
+            .withValue("unmappedList", new ArrayList<>(List.of("A", "B", "C")))
+            .withValue("unmappedObject", new HashMap<>(Map.of("A", 1, "C", true))),
+         new QRecord().withValue("firstName", "Tim"),
+         new QRecord().withValue("firstName", "Tyler")
+      ));
+      InsertOutput insertOutput = new InsertAction().execute(insertInput);
+
+      /////////////////////////////////////////
+      // make sure id got put on all records //
+      /////////////////////////////////////////
+      for(QRecord record : insertOutput.getRecords())
+      {
+         assertNotNull(record.getValueString("id"));
+      }
+
+      ///////////////////////////////////////////////////
+      // directly query mongo for the inserted records //
+      ///////////////////////////////////////////////////
+      MongoDatabase             database   = getMongoClient().getDatabase(TestUtils.MONGO_DATABASE);
+      MongoCollection<Document> collection = database.getCollection(TestUtils.TABLE_NAME_PERSON);
+      assertEquals(3, collection.countDocuments());
+      for(Document document : collection.find())
+      {
+         /////////////////////////////////////////////////////////////
+         // make sure values got set - including some nested values //
+         /////////////////////////////////////////////////////////////
+         assertNotNull(document.get("firstName"));
+         assertNotNull(document.get("metaData"));
+         assertThat(document.get("metaData")).isInstanceOf(Document.class);
+         assertNotNull(((Document) document.get("metaData")).get("createDate"));
+      }
+
+      Document document = collection.find(new Document("firstName", "Darin")).first();
+      assertNotNull(document);
+      assertEquals(1701, document.get("unmappedField"));
+      assertEquals(List.of("A", "B", "C"), document.get("unmappedList"));
+      assertEquals(Map.of("A", 1, "C", true), document.get("unmappedObject"));
+   }
+
+}