CE-1955 Switch fieldMetaData to use a type from in here for FieldAdornment, to include some better docs, but also to exclude new FILE_UPLOAD adornment type enum value

- Switch wide format to identify associations via comma-number-indexes...
- Add suggested mappings
- use header name instead of column index for mappings
- add counts of children process summary lines
- excel value/type handling

docs/metaData/Processes.adoc

@@ -38,6 +38,13 @@ See {link-permissionRules} for details.
 *** 1) by a single call to `.withStepList(List<QStepMetaData>)`, which internally adds each step into the `steps` map.
 *** 2) by multiple calls to `.addStep(QStepMetaData)`, which adds a step to both the `stepList` and `steps` map.
 ** If a process also needs optional steps (for a <<_custom_process_flow>>), they should be added by a call to `.addOptionalStep(QStepMetaData)`, which only places them in the `steps` map.
+* `stepFlow` - *enum, default LINEAR* - specifies the the flow-control logic between steps.  Possible values are:
+** `LINEAR` - steps are executed in-order, through the `stepList`.
+A backend step _can_ customize the `nextStepName` or re-order the `stepList`, if needed.
+In a frontend step, a user may be given the option to go _back_ to a previous step as well.
+** `STATE_MACHINE` - steps are executed as a Fine State Machine, starting with the first step in `stepList`,
+but then proceeding based on the `nextStepName` specified by the previous step.
+Thus allowing much more flexible flows.
 * `schedule` - *<<QScheduleMetaData>>* - set up the process to run automatically on the specified schedule.
 See below for details.
 * `minInputRecords` - *Integer* - #not used...#
@@ -67,6 +74,11 @@ For processes with a user-interface, they must define one or more "screens" in t
 * `formFields` - *List of String* - list of field names used by the screen as form-inputs.
 * `viewFields` - *List of String* - list of field names used by the screen as visible outputs.
 * `recordListFields` - *List of String* - list of field names used by the screen in a record listing.
+* `format` - *Optional String* - directive for a frontend to use specialized formatting for the display of the process.
+** Consult frontend documentation for supported values and their meanings.
+* `backStepName` - *Optional String* - For processes using `LINEAR` flow, if this value is given,
+then the frontend should offer a control that the user can take (e.g., a button) to move back to an
+earlier step in the process.
 
 ==== QFrontendComponentMetaData
 
@@ -90,10 +102,13 @@ Expects a process value named `html`.
 Expects process values named `downloadFileName` and `serverFilePath`.
 ** `GOOGLE_DRIVE_SELECT_FOLDER` - Special form that presents a UI from Google Drive, where the user can select a folder (e.g., as a target for uploading files in a subsequent backend step).
 ** `BULK_EDIT_FORM` - For use by the standard QQQ Bulk Edit process.
+** `BULK_LOAD_FILE_MAPPING_FORM`, `BULK_LOAD_VALUE_MAPPING_FORM`, or `BULK_LOAD_PROFILE_FORM` - For use by the standard QQQ Bulk Load process.
 ** `VALIDATION_REVIEW_SCREEN` - For use by the QQQ Streamed ETL With Frontend process family of processes.
 Displays a component prompting the user to run full validation or to skip it, or, if full validation has been ran, then showing the results of that validation.
 ** `PROCESS_SUMMARY_RESULTS` - For use by the QQQ Streamed ETL With Frontend process family of processes.
 Displays the summary results of running the process.
+** `WIDGET` - Render a QQQ Widget.
+Requires that `widgetName` be given as a value for the component.
 ** `RECORD_LIST` - _Deprecated.
 Showed a grid with a list of records as populated by the process._
 * `values` - *Map of String → Serializable* - Key=value pairs, with different expectations based on the component's `type`.
@@ -116,6 +131,27 @@ It can be used, however, for example, to cause a `defaultValue` to be applied to
 It can also be used to cause the process to throw an error, if a field is marked as `isRequired`, but a value is not present.
 ** `recordListMetaData` - *RecordListMetaData object* - _Not used at this time._
 
+==== QStateMachineStep
+
+Processes that use `flow = STATE_MACHINE` should use process steps of type `QStateMachineStep`.
+
+A common pattern seen in state-machine processes, is that they will present a frontend-step to a user,
+then always run a given backend-step in response to that screen which the user submitted.
+Inside that backend-step, custom application logic will determine the next state to go to,
+which is typically another frontend-step (which would then submit data to its corresponding backend-step,
+and continue the FSM).
+
+To help facilitate this pattern, factory methods exist on `QStateMachineStep`,
+for constructing the commonly-expected types of state-machine steps:
+
+* `frontendThenBackend(name, frontendStep, backendStep)` - for the frontend-then-backend pattern described above.
+* `backendOnly(name, backendStep)` - for a state that only has a backend step.
+This might be useful as a “reset” step, to run before restarting a state-loop.
+* `frontendOnly(name, frontendStep)` - for a state that only has a frontend step,
+which would always be followed by another state, which must be specified as the `defaultNextStepName`
+on the `QStateMachineStep`.
+
+
 ==== BasepullConfiguration
 
 A "Basepull" process is a common pattern where an application needs to perform some action on all new (or updated) records from a particular data source.
@@ -218,12 +254,10 @@ But for some cases, doing page-level transactions can reduce long-transactions a
 * `withSchedule(QScheduleMetaData schedule)` - Add a <<QScheduleMetaData>> to the process.
 
 [#_custom_process_flow]
-==== Custom Process Flow
-As referenced in the definition of the <<_QProcessMetaData_Properties,QProcessMetaData Properties>>, by default, a process
-will execute each of its steps in-order, as defined in the `stepList` property.
-However, a Backend Step can customize this flow #todo - write more clearly here...
-
-There are generally 2 method to call (in a `BackendStep`) to do a dynamic flow:
+==== How to customize a Linear process flow
+As referenced in the definition of the <<_QProcessMetaData_Properties,QProcessMetaData Properties>>, by default,
+(with `flow = LINEAR`) a process will execute each of its steps in-order, as defined in the `stepList` property.
+However, a Backend Step can customize this flow as follows:
 
 * `RunBackendStepOutput.setOverrideLastStepName(String stepName)`
 ** QQQ's `RunProcessAction` keeps track of which step it "last" ran, e.g., to tell it which one to run next.
@@ -239,7 +273,7 @@ does need to be found in the new `stepNameList` - otherwise, the framework will
 for figuring out where to go next.
 
 [source,java]
-.Example of a defining process that can use a flexible flow:
+.Example of a defining process that can use a customized linear flow:
 ----
 // for a case like this, it would be recommended to define all step names in constants:
 public final static String STEP_START = "start";
@@ -324,4 +358,21 @@ public static class StartStep implements BackendStep
 }
 ----
 
+[#_process_back]
+==== How to allow a process to go back
+
+The simplest option to allow a process to present a "Back" button to users,
+thus allowing them to move backward through a process
+(e.g., from a review screen back to an earlier input screen), is to set the property `backStepName`
+on a `QFrontendStepMetaData`.
+
+If the step that is executed after the user hits "Back" is a backend step, then within that
+step, `runBackendStepInput.getIsStepBack()` will return `true` (but ONLY within that first step after
+the user hits "Back").  It may be necessary within individual processes to be aware that the user
+has chosen to go back, to reset certain values in the process's state.
+
+Alternatively, if a frontend step's "Back" behavior needs to be dynamic (e.g., sometimes not available,
+or sometimes targeting different steps in the process), then in a backend step that runs before the
+frontend step, a call to `runBackendStepOutput.getProcessState().setBackStepName()` can be made,
+to customize the value which would otherwise come from the `QFrontendStepMetaData`.
 

qqq-backend-core/pom.xml

@@ -102,6 +102,11 @@
          <artifactId>fastexcel</artifactId>
          <version>0.12.15</version>
       </dependency>
+      <dependency>
+         <groupId>org.dhatim</groupId>
+         <artifactId>fastexcel-reader</artifactId>
+         <version>0.18.4</version>
+      </dependency>
       <dependency>
          <groupId>org.apache.poi</groupId>
          <artifactId>poi</artifactId>
@@ -112,6 +117,14 @@
          <artifactId>poi-ooxml</artifactId>
          <version>5.2.5</version>
       </dependency>
+
+      <!-- adding to help FastExcel -->
+      <dependency>
+         <groupId>commons-io</groupId>
+         <artifactId>commons-io</artifactId>
+         <version>2.16.0</version>
+      </dependency>
+
       <dependency>
          <groupId>com.auth0</groupId>
          <artifactId>auth0</artifactId>

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/processes/RunProcessAction.java

@@ -122,6 +122,12 @@ public class RunProcessAction
       UUIDAndTypeStateKey stateKey     = new UUIDAndTypeStateKey(UUID.fromString(runProcessInput.getProcessUUID()), StateType.PROCESS_STATUS);
       ProcessState        processState = primeProcessState(runProcessInput, stateKey, process);
 
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // these should always be clear when we're starting a run - so make sure they haven't leaked from previous //
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      processState.clearNextStepName();
+      processState.clearBackStepName();
+
       /////////////////////////////////////////////////////////
       // if process is 'basepull' style, keep track of 'now' //
       /////////////////////////////////////////////////////////
@@ -188,14 +194,35 @@ public class RunProcessAction
    private void runLinearStepLoop(QProcessMetaData process, ProcessState processState, UUIDAndTypeStateKey stateKey, RunProcessInput runProcessInput, RunProcessOutput runProcessOutput) throws Exception
    {
       String lastStepName = runProcessInput.getStartAfterStep();
+      String startAtStep  = runProcessInput.getStartAtStep();
 
       while(true)
       {
          ///////////////////////////////////////////////////////////////////////////////////////////////////////
          // always refresh the step list - as any step that runs can modify it (in the process state).        //
          // this is why we don't do a loop over the step list - as we'd get ConcurrentModificationExceptions. //
+         // deal with if we were told, from the input, to start After a step, or start At a step.             //
          ///////////////////////////////////////////////////////////////////////////////////////////////////////
-         List<QStepMetaData> stepList = getAvailableStepList(processState, process, lastStepName);
+         List<QStepMetaData> stepList;
+         if(startAtStep == null)
+         {
+            stepList = getAvailableStepList(processState, process, lastStepName, false);
+         }
+         else
+         {
+            stepList = getAvailableStepList(processState, process, startAtStep, true);
+
+            ///////////////////////////////////////////////////////////////////////////////////
+            // clear this field - so after we run a step, we'll then loop in last-step mode. //
+            ///////////////////////////////////////////////////////////////////////////////////
+            startAtStep = null;
+
+            ///////////////////////////////////////////////////////////////////////////////////
+            // if we're going to run a backend step now, let it see that this is a step-back //
+            ///////////////////////////////////////////////////////////////////////////////////
+            processState.setIsStepBack(true);
+         }
+
          if(stepList.isEmpty())
          {
             break;
@@ -232,7 +259,18 @@ public class RunProcessAction
             //////////////////////////////////////////////////
             throw (new QException("Unsure how to run a step of type: " + step.getClass().getName()));
          }
+
+         ////////////////////////////////////////////////////////////////////////////////////////
+         // only let this value be set for the original back step - don't let it stick around. //
+         // if a process wants to keep track of this itself, it can, but in a different slot.  //
+         ////////////////////////////////////////////////////////////////////////////////////////
+         processState.setIsStepBack(false);
       }
+
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // in case we broke from the loop above (e.g., by going directly into a frontend step), once again make sure to lower this flag. //
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      processState.setIsStepBack(false);
    }
 
 
@@ -264,6 +302,12 @@ public class RunProcessAction
             processFrontendStepFieldDefaultValues(processState, step);
             processFrontendComponents(processState, step);
             processState.setNextStepName(step.getName());
+
+            if(StringUtils.hasContent(step.getBackStepName()) && processState.getBackStepName().isEmpty())
+            {
+               processState.setBackStepName(step.getBackStepName());
+            }
+
             return LoopTodo.BREAK;
          }
          case SKIP ->
@@ -317,6 +361,7 @@ public class RunProcessAction
          // else run the given lastStepName //
          /////////////////////////////////////
          processState.clearNextStepName();
+         processState.clearBackStepName();
          step = process.getStep(lastStepName);
          if(step == null)
          {
@@ -398,6 +443,7 @@ public class RunProcessAction
                // its sub-steps, or, to fall out of the loop and end the process.                                  //
                //////////////////////////////////////////////////////////////////////////////////////////////////////
                processState.clearNextStepName();
+               processState.clearBackStepName();
                runStateMachineStep(nextStepName.get(), process, processState, stateKey, runProcessInput, runProcessOutput, stackDepth + 1);
                return;
             }
@@ -621,8 +667,10 @@ public class RunProcessAction
 
    /*******************************************************************************
     ** Get the list of steps which are eligible to run.
+    **
+    ** lastStep will be included in the list, or not, based on includeLastStep.
     *******************************************************************************/
-   private List<QStepMetaData> getAvailableStepList(ProcessState processState, QProcessMetaData process, String lastStep) throws QException
+   static List<QStepMetaData> getAvailableStepList(ProcessState processState, QProcessMetaData process, String lastStep, boolean includeLastStep) throws QException
    {
       if(lastStep == null)
       {
@@ -649,6 +697,10 @@ public class RunProcessAction
             if(stepName.equals(lastStep))
             {
                foundLastStep = true;
+               if(includeLastStep)
+               {
+                  validStepNames.add(stepName);
+               }
             }
          }
          return (stepNamesToSteps(process, validStepNames));
@@ -660,7 +712,7 @@ public class RunProcessAction
    /*******************************************************************************
     **
     *******************************************************************************/
-   private List<QStepMetaData> stepNamesToSteps(QProcessMetaData process, List<String> stepNames) throws QException
+   private static List<QStepMetaData> stepNamesToSteps(QProcessMetaData process, List<String> stepNames) throws QException
    {
       List<QStepMetaData> result = new ArrayList<>();
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/DeleteAction.java

@@ -320,7 +320,7 @@ public class DeleteAction
       QTableMetaData table               = deleteInput.getTable();
       List<QRecord>  primaryKeysNotFound = validateRecordsExistAndCanBeAccessed(deleteInput, oldRecordList.get());
 
-      ValidateRecordSecurityLockHelper.validateSecurityFields(table, oldRecordList.get(), ValidateRecordSecurityLockHelper.Action.DELETE);
+      ValidateRecordSecurityLockHelper.validateSecurityFields(table, oldRecordList.get(), ValidateRecordSecurityLockHelper.Action.DELETE, deleteInput.getTransaction());
 
       ///////////////////////////////////////////////////////////////////////////
       // after all validations, run the pre-delete customizer, if there is one //

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/InsertAction.java

@@ -258,7 +258,7 @@ public class InsertAction extends AbstractQActionFunction<InsertInput, InsertOut
       }
 
       runPreInsertCustomizerIfItIsTime(insertInput, isPreview, preInsertCustomizer, AbstractPreInsertCustomizer.WhenToRun.BEFORE_SECURITY_CHECKS);
-      ValidateRecordSecurityLockHelper.validateSecurityFields(insertInput.getTable(), insertInput.getRecords(), ValidateRecordSecurityLockHelper.Action.INSERT);
+      ValidateRecordSecurityLockHelper.validateSecurityFields(insertInput.getTable(), insertInput.getRecords(), ValidateRecordSecurityLockHelper.Action.INSERT, insertInput.getTransaction());
 
       runPreInsertCustomizerIfItIsTime(insertInput, isPreview, preInsertCustomizer, AbstractPreInsertCustomizer.WhenToRun.AFTER_ALL_VALIDATIONS);
    }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/UpdateAction.java

@@ -261,7 +261,7 @@ public class UpdateAction
       }
       else
       {
-         ValidateRecordSecurityLockHelper.validateSecurityFields(table, updateInput.getRecords(), ValidateRecordSecurityLockHelper.Action.UPDATE);
+         ValidateRecordSecurityLockHelper.validateSecurityFields(table, updateInput.getRecords(), ValidateRecordSecurityLockHelper.Action.UPDATE, updateInput.getTransaction());
       }
 
       if(updateInput.getInputSource().shouldValidateRequiredFields())
@@ -374,7 +374,7 @@ public class UpdateAction
             }
          }
 
-         ValidateRecordSecurityLockHelper.validateSecurityFields(table, updateInput.getRecords(), ValidateRecordSecurityLockHelper.Action.UPDATE);
+         ValidateRecordSecurityLockHelper.validateSecurityFields(table, updateInput.getRecords(), ValidateRecordSecurityLockHelper.Action.UPDATE, updateInput.getTransaction());
 
          for(QRecord record : page)
          {

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/helpers/ValidateRecordSecurityLockHelper.java

@@ -28,6 +28,7 @@ import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -83,7 +84,7 @@ public class ValidateRecordSecurityLockHelper
    /*******************************************************************************
     **
     *******************************************************************************/
-   public static void validateSecurityFields(QTableMetaData table, List<QRecord> records, Action action) throws QException
+   public static void validateSecurityFields(QTableMetaData table, List<QRecord> records, Action action, QBackendTransaction transaction) throws QException
    {
       MultiRecordSecurityLock locksToCheck = getRecordSecurityLocks(table, action);
       if(locksToCheck == null || CollectionUtils.nullSafeIsEmpty(locksToCheck.getLocks()))
@@ -101,7 +102,7 @@ public class ValidateRecordSecurityLockHelper
       // actually check lock values //
       ////////////////////////////////
       Map<Serializable, RecordWithErrors> errorRecords = new HashMap<>();
-      evaluateRecordLocks(table, records, action, locksToCheck, errorRecords, new ArrayList<>(), madeUpPrimaryKeys);
+      evaluateRecordLocks(table, records, action, locksToCheck, errorRecords, new ArrayList<>(), madeUpPrimaryKeys, transaction);
 
       /////////////////////////////////
       // propagate errors to records //
@@ -141,7 +142,7 @@ public class ValidateRecordSecurityLockHelper
     ** BUT - WRITE locks - in their case, we read the record no matter what, and in
     ** here we need to verify we have a key that allows us to WRITE the record.
     *******************************************************************************/
-   private static void evaluateRecordLocks(QTableMetaData table, List<QRecord> records, Action action, RecordSecurityLock recordSecurityLock, Map<Serializable, RecordWithErrors> errorRecords, List<Integer> treePosition, Map<Serializable, QRecord> madeUpPrimaryKeys) throws QException
+   private static void evaluateRecordLocks(QTableMetaData table, List<QRecord> records, Action action, RecordSecurityLock recordSecurityLock, Map<Serializable, RecordWithErrors> errorRecords, List<Integer> treePosition, Map<Serializable, QRecord> madeUpPrimaryKeys, QBackendTransaction transaction) throws QException
    {
       if(recordSecurityLock instanceof MultiRecordSecurityLock multiRecordSecurityLock)
       {
@@ -152,7 +153,7 @@ public class ValidateRecordSecurityLockHelper
          for(RecordSecurityLock childLock : CollectionUtils.nonNullList(multiRecordSecurityLock.getLocks()))
          {
             treePosition.add(i);
-            evaluateRecordLocks(table, records, action, childLock, errorRecords, treePosition, madeUpPrimaryKeys);
+            evaluateRecordLocks(table, records, action, childLock, errorRecords, treePosition, madeUpPrimaryKeys, transaction);
             treePosition.remove(treePosition.size() - 1);
             i++;
          }
@@ -225,6 +226,7 @@ public class ValidateRecordSecurityLockHelper
             // query will be like (fkey1=? and fkey2=?) OR (fkey1=? and fkey2=?) OR (fkey1=? and fkey2=?) //
             ////////////////////////////////////////////////////////////////////////////////////////////////
             QueryInput queryInput = new QueryInput();
+            queryInput.setTransaction(transaction);
             queryInput.setTableName(leftMostJoin.getLeftTable());
             QQueryFilter filter = new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR);
             queryInput.setFilter(filter);

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/values/SearchPossibleValueSourceAction.java

@@ -26,8 +26,12 @@ import java.io.Serializable;
 import java.time.Instant;
 import java.time.LocalDate;
 import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
 import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
@@ -50,7 +54,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.backend.core.utils.ValueUtils;
-import org.apache.commons.lang.NotImplementedException;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
 /*******************************************************************************
@@ -61,6 +65,9 @@ public class SearchPossibleValueSourceAction
 {
    private static final QLogger LOG = QLogger.getLogger(SearchPossibleValueSourceAction.class);
 
+   private static final Set<String> warnedAboutUnexpectedValueField                = Collections.synchronizedSet(new HashSet<>());
+   private static final Set<String> warnedAboutUnexpectedNoOfFieldsToSearchByLabel = Collections.synchronizedSet(new HashSet<>());
+
    private QPossibleValueTranslator possibleValueTranslator;
 
 
@@ -110,6 +117,7 @@ public class SearchPossibleValueSourceAction
       List<Serializable>              matchingIds = new ArrayList<>();
 
       List<?> inputIdsAsCorrectType = convertInputIdsToEnumIdType(possibleValueSource, input.getIdList());
+      Set<String> labels = null;
 
       for(QPossibleValue<?> possibleValue : possibleValueSource.getEnumValues())
       {
@@ -122,12 +130,24 @@ public class SearchPossibleValueSourceAction
                match = true;
             }
          }
+         else if(input.getLabelList() != null)
+         {
+            if(labels == null)
+            {
+               labels = input.getLabelList().stream().filter(Objects::nonNull).map(l -> l.toLowerCase()).collect(Collectors.toSet());
+            }
+
+            if(labels.contains(possibleValue.getLabel().toLowerCase()))
+            {
+               match = true;
+            }
+         }
          else
          {
             if(StringUtils.hasContent(input.getSearchTerm()))
             {
                match = (Objects.equals(ValueUtils.getValueAsString(possibleValue.getId()).toLowerCase(), input.getSearchTerm().toLowerCase())
-                  || possibleValue.getLabel().toLowerCase().startsWith(input.getSearchTerm().toLowerCase()));
+                        || possibleValue.getLabel().toLowerCase().startsWith(input.getSearchTerm().toLowerCase()));
             }
             else
             {
@@ -168,21 +188,37 @@ public class SearchPossibleValueSourceAction
 
       Object anIdFromTheEnum = possibleValueSource.getEnumValues().get(0).getId();
 
-      if(anIdFromTheEnum instanceof Integer)
+      for(Serializable inputId : inputIdList)
       {
-         inputIdList.forEach(id -> rs.add(ValueUtils.getValueAsInteger(id)));
-      }
-      else if(anIdFromTheEnum instanceof String)
-      {
-         inputIdList.forEach(id -> rs.add(ValueUtils.getValueAsString(id)));
-      }
-      else if(anIdFromTheEnum instanceof Boolean)
-      {
-         inputIdList.forEach(id -> rs.add(ValueUtils.getValueAsBoolean(id)));
-      }
-      else
-      {
-         LOG.warn("Unexpected type [" + anIdFromTheEnum.getClass().getSimpleName() + "] for ids in enum: " + possibleValueSource.getName());
+         Object properlyTypedId = null;
+         try
+         {
+            if(anIdFromTheEnum instanceof Integer)
+            {
+               properlyTypedId = ValueUtils.getValueAsInteger(inputId);
+            }
+            else if(anIdFromTheEnum instanceof String)
+            {
+               properlyTypedId = ValueUtils.getValueAsString(inputId);
+            }
+            else if(anIdFromTheEnum instanceof Boolean)
+            {
+               properlyTypedId = ValueUtils.getValueAsBoolean(inputId);
+            }
+            else
+            {
+               LOG.warn("Unexpected type [" + anIdFromTheEnum.getClass().getSimpleName() + "] for ids in enum: " + possibleValueSource.getName());
+            }
+         }
+         catch(Exception e)
+         {
+            LOG.debug("Error converting possible value id to expected id type", e, logPair("value", inputId));
+         }
+
+         if (properlyTypedId != null)
+         {
+            rs.add(properlyTypedId);
+         }
       }
 
       return (rs);
@@ -209,6 +245,53 @@ public class SearchPossibleValueSourceAction
       {
          queryFilter.addCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.IN, input.getIdList()));
       }
+      else if(input.getLabelList() != null)
+      {
+         List<String> fieldNames = new ArrayList<>();
+
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // the 'value fields' will either be 'id' or 'label' (which means, use the fields from the tableMetaData's label fields) //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         for(String valueField : possibleValueSource.getValueFields())
+         {
+            if("id".equals(valueField))
+            {
+               fieldNames.add(table.getPrimaryKeyField());
+            }
+            else if("label".equals(valueField))
+            {
+               if(table.getRecordLabelFields() != null)
+               {
+                  fieldNames.addAll(table.getRecordLabelFields());
+               }
+            }
+            else
+            {
+               String message = "Unexpected valueField defined in possibleValueSource when searching possibleValueSource by label (required: 'id' or 'label')";
+               if(!warnedAboutUnexpectedValueField.contains(possibleValueSource.getName()))
+               {
+                  LOG.warn(message, logPair("valueField", valueField), logPair("possibleValueSource", possibleValueSource.getName()));
+                  warnedAboutUnexpectedValueField.add(possibleValueSource.getName());
+               }
+               output.setWarning(message);
+            }
+         }
+
+         if(fieldNames.size() == 1)
+         {
+            queryFilter.addCriteria(new QFilterCriteria(fieldNames.get(0), QCriteriaOperator.IN, input.getLabelList()));
+         }
+         else
+         {
+            String message = "Unexpected number of fields found for searching possibleValueSource by label (required: 1, found: " + fieldNames.size() + ")";
+            if(!warnedAboutUnexpectedNoOfFieldsToSearchByLabel.contains(possibleValueSource.getName()))
+            {
+               LOG.warn(message);
+               warnedAboutUnexpectedNoOfFieldsToSearchByLabel.add(possibleValueSource.getName());
+            }
+            output.setWarning(message);
+         }
+      }
       else
       {
          String searchTerm = input.getSearchTerm();
@@ -269,8 +352,8 @@ public class SearchPossibleValueSourceAction
          queryFilter = input.getDefaultQueryFilter();
       }
 
-      // todo - skip & limit as params
-      queryFilter.setLimit(250);
+      queryFilter.setLimit(input.getLimit());
+      queryFilter.setSkip(input.getSkip());
 
       queryFilter.setOrderBys(possibleValueSource.getOrderByFields());
 
@@ -288,7 +371,7 @@ public class SearchPossibleValueSourceAction
          fieldName = table.getPrimaryKeyField();
       }
 
-      List<Serializable> ids = queryOutput.getRecords().stream().map(r -> r.getValue(fieldName)).toList();
+      List<Serializable>      ids             = queryOutput.getRecords().stream().map(r -> r.getValue(fieldName)).toList();
       List<QPossibleValue<?>> qPossibleValues = possibleValueTranslator.buildTranslatedPossibleValueList(possibleValueSource, ids);
       output.setResults(qPossibleValues);
 
@@ -301,7 +384,7 @@ public class SearchPossibleValueSourceAction
     **
     *******************************************************************************/
    @SuppressWarnings({ "rawtypes", "unchecked" })
-   private SearchPossibleValueSourceOutput searchPossibleValueCustom(SearchPossibleValueSourceInput input, QPossibleValueSource possibleValueSource)
+   private SearchPossibleValueSourceOutput searchPossibleValueCustom(SearchPossibleValueSourceInput input, QPossibleValueSource possibleValueSource) throws QException
    {
       try
       {
@@ -314,11 +397,10 @@ public class SearchPossibleValueSourceAction
       }
       catch(Exception e)
       {
-         // LOG.warn("Error sending [" + value + "] for field [" + field + "] through custom code for PVS [" + field.getPossibleValueSourceName() + "]", e);
+         String message = "Error sending searching custom possible value source [" + input.getPossibleValueSourceName() + "]";
+         LOG.warn(message, e);
+         throw (new QException(message));
       }
-
-      throw new NotImplementedException("Not impleemnted");
-      // return (null);
    }
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceEnricher.java

@@ -42,6 +42,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType.FileUploadAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
@@ -54,6 +55,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPer
 import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
@@ -75,6 +77,11 @@ import com.kingsrook.qqq.backend.core.processes.implementations.bulk.edit.BulkEd
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.edit.BulkEditTransformStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertExtractStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertLoadStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertPrepareFileMappingStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertPrepareFileUploadStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertPrepareValueMappingStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertReceiveFileMappingStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertReceiveValueMappingStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.BulkInsertTransformStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
@@ -833,7 +840,7 @@ public class QInstanceEnricher
    /*******************************************************************************
     **
     *******************************************************************************/
-   private void defineTableBulkInsert(QInstance qInstance, QTableMetaData table, String processName)
+   public void defineTableBulkInsert(QInstance qInstance, QTableMetaData table, String processName)
    {
       Map<String, Serializable> values = new HashMap<>();
       values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, table.getName());
@@ -845,6 +852,7 @@ public class QInstanceEnricher
             values
          )
          .withName(processName)
+         .withIcon(new QIcon().withName("library_add"))
          .withLabel(table.getLabel() + " Bulk Insert")
          .withTableName(table.getName())
          .withIsHidden(true)
@@ -875,18 +883,74 @@ public class QInstanceEnricher
          .map(QFieldMetaData::getLabel)
          .collect(Collectors.joining(", "));
 
+      QBackendStepMetaData prepareFileUploadStep = new QBackendStepMetaData()
+         .withName("prepareFileUpload")
+         .withCode(new QCodeReference(BulkInsertPrepareFileUploadStep.class));
+
       QFrontendStepMetaData uploadScreen = new QFrontendStepMetaData()
          .withName("upload")
          .withLabel("Upload File")
-         .withFormField(new QFieldMetaData("theFile", QFieldType.BLOB).withLabel(table.getLabel() + " File").withIsRequired(true))
-         .withComponent(new QFrontendComponentMetaData()
-            .withType(QComponentType.HELP_TEXT)
-            .withValue("previewText", "file upload instructions")
-            .withValue("text", "Upload a CSV file with the following columns:\n" + fieldsForHelpText))
+         .withFormField(new QFieldMetaData("theFile", QFieldType.BLOB)
+            .withFieldAdornment(FileUploadAdornment.newFieldAdornment()
+               .withValue(FileUploadAdornment.formatDragAndDrop())
+               .withValue(FileUploadAdornment.widthFull()))
+            .withLabel(table.getLabel() + " File")
+            .withIsRequired(true))
+         .withComponent(new QFrontendComponentMetaData().withType(QComponentType.HTML))
          .withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM));
 
-      process.addStep(0, uploadScreen);
-      process.getFrontendStep("review").setRecordListFields(editableFields);
+      QBackendStepMetaData prepareFileMappingStep = new QBackendStepMetaData()
+         .withName("prepareFileMapping")
+         .withCode(new QCodeReference(BulkInsertPrepareFileMappingStep.class));
+
+      QFrontendStepMetaData fileMappingScreen = new QFrontendStepMetaData()
+         .withName("fileMapping")
+         .withLabel("File Mapping")
+         .withBackStepName("upload")
+         .withComponent(new QFrontendComponentMetaData().withType(QComponentType.BULK_LOAD_FILE_MAPPING_FORM));
+
+      QBackendStepMetaData receiveFileMappingStep = new QBackendStepMetaData()
+         .withName("receiveFileMapping")
+         .withCode(new QCodeReference(BulkInsertReceiveFileMappingStep.class));
+
+      QBackendStepMetaData prepareValueMappingStep = new QBackendStepMetaData()
+         .withName("prepareValueMapping")
+         .withCode(new QCodeReference(BulkInsertPrepareValueMappingStep.class));
+
+      QFrontendStepMetaData valueMappingScreen = new QFrontendStepMetaData()
+         .withName("valueMapping")
+         .withLabel("Value Mapping")
+         .withBackStepName("prepareFileMapping")
+         .withComponent(new QFrontendComponentMetaData().withType(QComponentType.BULK_LOAD_VALUE_MAPPING_FORM));
+
+      QBackendStepMetaData receiveValueMappingStep = new QBackendStepMetaData()
+         .withName("receiveValueMapping")
+         .withCode(new QCodeReference(BulkInsertReceiveValueMappingStep.class));
+
+      int i = 0;
+      process.addStep(i++, prepareFileUploadStep);
+      process.addStep(i++, uploadScreen);
+
+      process.addStep(i++, prepareFileMappingStep);
+      process.addStep(i++, fileMappingScreen);
+      process.addStep(i++, receiveFileMappingStep);
+
+      process.addStep(i++, prepareValueMappingStep);
+      process.addStep(i++, valueMappingScreen);
+      process.addStep(i++, receiveValueMappingStep);
+
+      process.getFrontendStep(StreamedETLWithFrontendProcess.STEP_NAME_REVIEW).setRecordListFields(editableFields);
+
+      //////////////////////////////////////////////////////////////////////////////////////////
+      // put the bulk-load profile form (e.g., for saving it) on the review & result screens) //
+      //////////////////////////////////////////////////////////////////////////////////////////
+      process.getFrontendStep(StreamedETLWithFrontendProcess.STEP_NAME_REVIEW)
+         .withBackStepName("prepareFileMapping")
+         .getComponents().add(0, new QFrontendComponentMetaData().withType(QComponentType.BULK_LOAD_PROFILE_FORM));
+
+      process.getFrontendStep(StreamedETLWithFrontendProcess.STEP_NAME_RESULT)
+         .getComponents().add(0, new QFrontendComponentMetaData().withType(QComponentType.BULK_LOAD_PROFILE_FORM));
+
       qInstance.addProcess(process);
    }
 

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/ProcessState.java

@@ -40,6 +40,8 @@ public class ProcessState implements Serializable
    private Map<String, Serializable> values       = new HashMap<>();
    private List<String>              stepList     = new ArrayList<>();
    private Optional<String>          nextStepName = Optional.empty();
+   private Optional<String>          backStepName = Optional.empty();
+   private boolean                   isStepBack   = false;
 
    private ProcessMetaDataAdjustment processMetaDataAdjustment = null;
 
@@ -122,6 +124,39 @@ public class ProcessState implements Serializable
 
 
 
+   /*******************************************************************************
+    ** Getter for backStepName
+    **
+    *******************************************************************************/
+   public Optional<String> getBackStepName()
+   {
+      return backStepName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for backStepName
+    **
+    *******************************************************************************/
+   public void setBackStepName(String backStepName)
+   {
+      this.backStepName = Optional.of(backStepName);
+   }
+
+
+
+   /*******************************************************************************
+    ** clear out the value of backStepName (set the Optional to empty)
+    **
+    *******************************************************************************/
+   public void clearBackStepName()
+   {
+      this.backStepName = Optional.empty();
+   }
+
+
+
    /*******************************************************************************
     ** Getter for stepList
     **
@@ -176,4 +211,35 @@ public class ProcessState implements Serializable
    }
 
 
+
+   /*******************************************************************************
+    ** Getter for isStepBack
+    *******************************************************************************/
+   public boolean getIsStepBack()
+   {
+      return (this.isStepBack);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for isStepBack
+    *******************************************************************************/
+   public void setIsStepBack(boolean isStepBack)
+   {
+      this.isStepBack = isStepBack;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for isStepBack
+    *******************************************************************************/
+   public ProcessState withIsStepBack(boolean isStepBack)
+   {
+      this.isStepBack = isStepBack;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/ProcessSummaryLine.java

@@ -53,6 +53,7 @@ public class ProcessSummaryLine implements ProcessSummaryLineInterface
    //////////////////////////////////////////////////////////////////////////
    private ArrayList<Serializable> primaryKeys;
 
+   private ArrayList<String> bulletsOfText;
 
 
    /*******************************************************************************
@@ -497,4 +498,35 @@ public class ProcessSummaryLine implements ProcessSummaryLineInterface
       return (this);
    }
 
+
+   /*******************************************************************************
+    ** Getter for bulletsOfText
+    *******************************************************************************/
+   public ArrayList<String> getBulletsOfText()
+   {
+      return (this.bulletsOfText);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for bulletsOfText
+    *******************************************************************************/
+   public void setBulletsOfText(ArrayList<String> bulletsOfText)
+   {
+      this.bulletsOfText = bulletsOfText;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for bulletsOfText
+    *******************************************************************************/
+   public ProcessSummaryLine withBulletsOfText(ArrayList<String> bulletsOfText)
+   {
+      this.bulletsOfText = bulletsOfText;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/RunBackendStepInput.java

@@ -419,6 +419,17 @@ public class RunBackendStepInput extends AbstractActionInput
 
 
 
+   /*******************************************************************************
+    ** Accessor for processState's isStepBack attribute
+    **
+    *******************************************************************************/
+   public boolean getIsStepBack()
+   {
+      return processState.getIsStepBack();
+   }
+
+
+
    /*******************************************************************************
     ** Accessor for processState - protected, because we generally want to access
     ** its members through wrapper methods, we think

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/processes/RunProcessInput.java

@@ -49,6 +49,7 @@ public class RunProcessInput extends AbstractActionInput
    private ProcessState         processState;
    private FrontendStepBehavior frontendStepBehavior = FrontendStepBehavior.BREAK;
    private String               startAfterStep;
+   private String               startAtStep;
    private String               processUUID;
    private AsyncJobCallback     asyncJobCallback;
 
@@ -451,4 +452,35 @@ public class RunProcessInput extends AbstractActionInput
    {
       return asyncJobCallback;
    }
+
+   /*******************************************************************************
+    ** Getter for startAtStep
+    *******************************************************************************/
+   public String getStartAtStep()
+   {
+      return (this.startAtStep);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for startAtStep
+    *******************************************************************************/
+   public void setStartAtStep(String startAtStep)
+   {
+      this.startAtStep = startAtStep;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for startAtStep
+    *******************************************************************************/
+   public RunProcessInput withStartAtStep(String startAtStep)
+   {
+      this.startAtStep = startAtStep;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/storage/StorageInput.java

@@ -22,13 +22,14 @@
 package com.kingsrook.qqq.backend.core.model.actions.tables.storage;
 
 
+import java.io.Serializable;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
 
 
 /*******************************************************************************
  ** Input for Storage actions.
  *******************************************************************************/
-public class StorageInput extends AbstractTableActionInput
+public class StorageInput extends AbstractTableActionInput implements Serializable
 {
    private String reference;
    private String contentType;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/values/SearchPossibleValueSourceInput.java

@@ -38,9 +38,10 @@ public class SearchPossibleValueSourceInput extends AbstractActionInput implemen
    private QQueryFilter       defaultQueryFilter;
    private String             searchTerm;
    private List<Serializable> idList;
+   private List<String>       labelList;
 
    private Integer skip  = 0;
-   private Integer limit = 100;
+   private Integer limit = 250;
 
 
 
@@ -281,4 +282,35 @@ public class SearchPossibleValueSourceInput extends AbstractActionInput implemen
       this.limit = limit;
       return (this);
    }
+
+
+   /*******************************************************************************
+    ** Getter for labelList
+    *******************************************************************************/
+   public List<String> getLabelList()
+   {
+      return (this.labelList);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for labelList
+    *******************************************************************************/
+   public void setLabelList(List<String> labelList)
+   {
+      this.labelList = labelList;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for labelList
+    *******************************************************************************/
+   public SearchPossibleValueSourceInput withLabelList(List<String> labelList)
+   {
+      this.labelList = labelList;
+      return (this);
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/values/SearchPossibleValueSourceOutput.java

@@ -35,6 +35,7 @@ public class SearchPossibleValueSourceOutput extends AbstractActionOutput
 {
    private List<QPossibleValue<?>> results = new ArrayList<>();
 
+   private String warning;
 
 
    /*******************************************************************************
@@ -88,4 +89,35 @@ public class SearchPossibleValueSourceOutput extends AbstractActionOutput
       return (this);
    }
 
+
+   /*******************************************************************************
+    ** Getter for warning
+    *******************************************************************************/
+   public String getWarning()
+   {
+      return (this.warning);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for warning
+    *******************************************************************************/
+   public void setWarning(String warning)
+   {
+      this.warning = warning;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for warning
+    *******************************************************************************/
+   public SearchPossibleValueSourceOutput withWarning(String warning)
+   {
+      this.warning = warning;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/AdornmentType.java

@@ -41,6 +41,7 @@ public enum AdornmentType
    RENDER_HTML,
    REVEAL,
    FILE_DOWNLOAD,
+   FILE_UPLOAD,
    ERROR;
    //////////////////////////////////////////////////////////////////////////
    // keep these values in sync with AdornmentType.ts in qqq-frontend-core //
@@ -164,4 +165,65 @@ public enum AdornmentType
       }
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static class FileUploadAdornment
+   {
+      public static String FORMAT = "format";
+      public static String WIDTH  = "width";
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public static FieldAdornment newFieldAdornment()
+      {
+         return (new FieldAdornment(AdornmentType.FILE_UPLOAD));
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public static Pair<String, String> formatDragAndDrop()
+      {
+         return (Pair.of(FORMAT, "dragAndDrop"));
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public static Pair<String, String> formatButton()
+      {
+         return (Pair.of(FORMAT, "button"));
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public static Pair<String, String> widthFull()
+      {
+         return (Pair.of(WIDTH, "full"));
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public static Pair<String, String> widthHalf()
+      {
+         return (Pair.of(WIDTH, "half"));
+      }
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/FieldAdornment.java

@@ -177,7 +177,7 @@ public class FieldAdornment
     ** Fluent setter for values
     **
     *******************************************************************************/
-   public FieldAdornment withValue(Pair<String, Serializable> value)
+   public FieldAdornment withValue(Pair<String, ? extends Serializable> value)
    {
       return (withValue(value.getA(), value.getB()));
    }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/fields/QFieldType.java

@@ -27,6 +27,7 @@ import java.time.Instant;
 import java.time.LocalDate;
 import java.time.LocalTime;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
 
 /*******************************************************************************
@@ -100,6 +101,16 @@ public enum QFieldType
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public String getMixedCaseLabel()
+   {
+      return StringUtils.allCapsToMixedCase(name());
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/frontend/QFrontendFieldMetaData.java

@@ -43,7 +43,7 @@ import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
  *
  *******************************************************************************/
 @JsonInclude(Include.NON_NULL)
-public class QFrontendFieldMetaData
+public class QFrontendFieldMetaData implements Serializable
 {
    private String       name;
    private String       label;

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QComponentType.java

@@ -29,6 +29,9 @@ public enum QComponentType
 {
    HELP_TEXT,
    BULK_EDIT_FORM,
+   BULK_LOAD_FILE_MAPPING_FORM,
+   BULK_LOAD_VALUE_MAPPING_FORM,
+   BULK_LOAD_PROFILE_FORM,
    VALIDATION_REVIEW_SCREEN,
    EDIT_FORM,
    VIEW_FORM,

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/processes/QFrontendStepMetaData.java

@@ -48,6 +48,7 @@ public class QFrontendStepMetaData extends QStepMetaData
    private Map<String, QFieldMetaData>      formFieldMap;
 
    private String format;
+   private String backStepName;
 
    private List<QHelpContent> helpContents;
 
@@ -436,4 +437,35 @@ public class QFrontendStepMetaData extends QStepMetaData
    }
 
 
+
+   /*******************************************************************************
+    ** Getter for backStepName
+    *******************************************************************************/
+   public String getBackStepName()
+   {
+      return (this.backStepName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for backStepName
+    *******************************************************************************/
+   public void setBackStepName(String backStepName)
+   {
+      this.backStepName = backStepName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for backStepName
+    *******************************************************************************/
+   public QFrontendStepMetaData withBackStepName(String backStepName)
+   {
+      this.backStepName = backStepName;
+      return (this);
+   }
+
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/savedbulkloadprofiles/SavedBulkLoadProfile.java

@@ -0,0 +1,285 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.TablesPossibleValueSourceMetaDataProvider;
+
+
+/*******************************************************************************
+ ** Entity bean for the savedBulkLoadProfile table
+ *******************************************************************************/
+public class SavedBulkLoadProfile extends QRecordEntity
+{
+   public static final String TABLE_NAME = "savedBulkLoadProfile";
+
+   @QField(isEditable = false)
+   private Integer id;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+   @QField(isRequired = true, maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.TRUNCATE_ELLIPSIS, label = "Profile Name")
+   private String label;
+
+   @QField(possibleValueSourceName = TablesPossibleValueSourceMetaDataProvider.NAME, maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.ERROR, label = "Table", isRequired = true)
+   private String tableName;
+
+   @QField(maxLength = 250, valueTooLongBehavior = ValueTooLongBehavior.ERROR, dynamicDefaultValueBehavior = DynamicDefaultValueBehavior.USER_ID, label = "Owner")
+   private String userId;
+
+   @QField(label = "Mapping JSON")
+   private String mappingJson;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public SavedBulkLoadProfile()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public SavedBulkLoadProfile(QRecord qRecord) throws QException
+   {
+      populateFromQRecord(qRecord);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    **
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    **
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    **
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    **
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    **
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    **
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for label
+    **
+    *******************************************************************************/
+   public String getLabel()
+   {
+      return label;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for label
+    **
+    *******************************************************************************/
+   public void setLabel(String label)
+   {
+      this.label = label;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for label
+    **
+    *******************************************************************************/
+   public SavedBulkLoadProfile withLabel(String label)
+   {
+      this.label = label;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    **
+    *******************************************************************************/
+   public String getTableName()
+   {
+      return tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    **
+    *******************************************************************************/
+   public void setTableName(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableName
+    **
+    *******************************************************************************/
+   public SavedBulkLoadProfile withTableName(String tableName)
+   {
+      this.tableName = tableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for userId
+    **
+    *******************************************************************************/
+   public String getUserId()
+   {
+      return userId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for userId
+    **
+    *******************************************************************************/
+   public void setUserId(String userId)
+   {
+      this.userId = userId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for userId
+    **
+    *******************************************************************************/
+   public SavedBulkLoadProfile withUserId(String userId)
+   {
+      this.userId = userId;
+      return (this);
+   }
+
+
+
+
+   /*******************************************************************************
+    ** Getter for mappingJson
+    *******************************************************************************/
+   public String getMappingJson()
+   {
+      return (this.mappingJson);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for mappingJson
+    *******************************************************************************/
+   public void setMappingJson(String mappingJson)
+   {
+      this.mappingJson = mappingJson;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for mappingJson
+    *******************************************************************************/
+   public SavedBulkLoadProfile withMappingJson(String mappingJson)
+   {
+      this.mappingJson = mappingJson;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/savedbulkloadprofiles/SavedBulkLoadProfileMetaDataProvider.java

@@ -0,0 +1,169 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles;
+
+
+import java.util.List;
+import java.util.function.Consumer;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.sharing.ShareScopePossibleValueMetaDataProducer;
+import com.kingsrook.qqq.backend.core.model.metadata.sharing.ShareableAudienceType;
+import com.kingsrook.qqq.backend.core.model.metadata.sharing.ShareableTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
+import com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles.DeleteSavedBulkLoadProfileProcess;
+import com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles.QuerySavedBulkLoadProfileProcess;
+import com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles.StoreSavedBulkLoadProfileProcess;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class SavedBulkLoadProfileMetaDataProvider
+{
+   public static final String SHARED_SAVED_BULK_LOAD_PROFILE_JOIN_SAVED_BULK_LOAD_PROFILE = "sharedSavedBulkLoadProfileJoinSavedBulkLoadProfile";
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public void defineAll(QInstance instance, String recordTablesBackendName, Consumer<QTableMetaData> backendDetailEnricher) throws QException
+   {
+      instance.addTable(defineSavedBulkLoadProfileTable(recordTablesBackendName, backendDetailEnricher));
+      instance.addPossibleValueSource(QPossibleValueSource.newForTable(SavedBulkLoadProfile.TABLE_NAME));
+
+      /////////////////////////////////////
+      // todo - param to enable sharing? //
+      /////////////////////////////////////
+      instance.addTable(defineSharedSavedBulkLoadProfileTable(recordTablesBackendName, backendDetailEnricher));
+      instance.addJoin(defineSharedSavedBulkLoadProfileJoinSavedBulkLoadProfile());
+      if(instance.getPossibleValueSource(ShareScopePossibleValueMetaDataProducer.NAME) == null)
+      {
+         instance.addPossibleValueSource(new ShareScopePossibleValueMetaDataProducer().produce(new QInstance()));
+      }
+
+      ////////////////////////////////////
+      // processes for working with 'em //
+      ////////////////////////////////////
+      instance.add(StoreSavedBulkLoadProfileProcess.getProcessMetaData());
+      instance.add(QuerySavedBulkLoadProfileProcess.getProcessMetaData());
+      instance.add(DeleteSavedBulkLoadProfileProcess.getProcessMetaData());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private QJoinMetaData defineSharedSavedBulkLoadProfileJoinSavedBulkLoadProfile()
+   {
+      return (new QJoinMetaData()
+         .withName(SHARED_SAVED_BULK_LOAD_PROFILE_JOIN_SAVED_BULK_LOAD_PROFILE)
+         .withLeftTable(SharedSavedBulkLoadProfile.TABLE_NAME)
+         .withRightTable(SavedBulkLoadProfile.TABLE_NAME)
+         .withType(JoinType.MANY_TO_ONE)
+         .withJoinOn(new JoinOn("savedBulkLoadProfileId", "id")));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QTableMetaData defineSavedBulkLoadProfileTable(String backendName, Consumer<QTableMetaData> backendDetailEnricher) throws QException
+   {
+      QTableMetaData table = new QTableMetaData()
+         .withName(SavedBulkLoadProfile.TABLE_NAME)
+         .withLabel("Bulk Load Profile")
+         .withIcon(new QIcon().withName("drive_folder_upload"))
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("label")
+         .withBackendName(backendName)
+         .withPrimaryKeyField("id")
+         .withFieldsFromEntity(SavedBulkLoadProfile.class)
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
+         .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1, List.of("id", "label", "tableName")))
+         .withSection(new QFieldSection("data", new QIcon().withName("text_snippet"), Tier.T2, List.of("mappingJson")).withIsHidden(true))
+         .withSection(new QFieldSection("hidden", new QIcon().withName("text_snippet"), Tier.T2, List.of("userId")).withIsHidden(true))
+         .withSection(new QFieldSection("dates", new QIcon().withName("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+
+      // todo - want one of these?
+      // table.getField("queryFilterJson").withBehavior(SavedReportJsonFieldDisplayValueFormatter.getInstance());
+
+      table.withShareableTableMetaData(new ShareableTableMetaData()
+         .withSharedRecordTableName(SharedSavedBulkLoadProfile.TABLE_NAME)
+         .withAssetIdFieldName("savedBulkLoadProfileId")
+         .withScopeFieldName("scope")
+         .withThisTableOwnerIdFieldName("userId")
+         .withAudienceType(new ShareableAudienceType().withName("user").withFieldName("userId")));
+
+      if(backendDetailEnricher != null)
+      {
+         backendDetailEnricher.accept(table);
+      }
+
+      return (table);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public QTableMetaData defineSharedSavedBulkLoadProfileTable(String backendName, Consumer<QTableMetaData> backendDetailEnricher) throws QException
+   {
+      QTableMetaData table = new QTableMetaData()
+         .withName(SharedSavedBulkLoadProfile.TABLE_NAME)
+         .withLabel("Shared Bulk Load Profile")
+         .withIcon(new QIcon().withName("share"))
+         .withRecordLabelFormat("%s")
+         .withRecordLabelFields("savedBulkLoadProfileId")
+         .withBackendName(backendName)
+         .withUniqueKey(new UniqueKey("savedBulkLoadProfileId", "userId"))
+         .withPrimaryKeyField("id")
+         .withFieldsFromEntity(SharedSavedBulkLoadProfile.class)
+         // todo - security key
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
+         .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1, List.of("id", "savedBulkLoadProfileId", "userId")))
+         .withSection(new QFieldSection("data", new QIcon().withName("text_snippet"), Tier.T2, List.of("scope")))
+         .withSection(new QFieldSection("dates", new QIcon().withName("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+
+      if(backendDetailEnricher != null)
+      {
+         backendDetailEnricher.accept(table);
+      }
+
+      return (table);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/savedbulkloadprofiles/SharedSavedBulkLoadProfile.java

@@ -0,0 +1,268 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+import com.kingsrook.qqq.backend.core.model.metadata.sharing.ShareScopePossibleValueMetaDataProducer;
+
+
+/*******************************************************************************
+ ** Entity bean for the shared saved bulk load profile table
+ *******************************************************************************/
+public class SharedSavedBulkLoadProfile extends QRecordEntity
+{
+   public static final String TABLE_NAME = "sharedSavedBulkLoadProfile";
+
+   @QField(isEditable = false)
+   private Integer id;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+   @QField(possibleValueSourceName = SavedBulkLoadProfile.TABLE_NAME, label = "Bulk Load Profile")
+   private Integer savedBulkLoadProfileId;
+
+   @QField(label = "User")
+   private String userId;
+
+   @QField(possibleValueSourceName = ShareScopePossibleValueMetaDataProducer.NAME)
+   private String scope;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile(QRecord qRecord) throws QException
+   {
+      populateFromQRecord(qRecord);
+   }
+
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (this.createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return (this.modifyDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modifyDate
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile withModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+      return (this);
+   }
+
+
+
+
+   /*******************************************************************************
+    ** Getter for userId
+    *******************************************************************************/
+   public String getUserId()
+   {
+      return (this.userId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for userId
+    *******************************************************************************/
+   public void setUserId(String userId)
+   {
+      this.userId = userId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for userId
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile withUserId(String userId)
+   {
+      this.userId = userId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for scope
+    *******************************************************************************/
+   public String getScope()
+   {
+      return (this.scope);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for scope
+    *******************************************************************************/
+   public void setScope(String scope)
+   {
+      this.scope = scope;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for scope
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile withScope(String scope)
+   {
+      this.scope = scope;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for savedBulkLoadProfileId
+    *******************************************************************************/
+   public Integer getSavedBulkLoadProfileId()
+   {
+      return (this.savedBulkLoadProfileId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for savedBulkLoadProfileId
+    *******************************************************************************/
+   public void setSavedBulkLoadProfileId(Integer savedBulkLoadProfileId)
+   {
+      this.savedBulkLoadProfileId = savedBulkLoadProfileId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for savedBulkLoadProfileId
+    *******************************************************************************/
+   public SharedSavedBulkLoadProfile withSavedBulkLoadProfileId(Integer savedBulkLoadProfileId)
+   {
+      this.savedBulkLoadProfileId = savedBulkLoadProfileId;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/session/QSession.java

@@ -48,7 +48,7 @@ public class QSession implements Serializable, Cloneable
    private QUser  user;
    private String uuid;
 
-   private Set<String>                     permissions;
+   private Set<String> permissions;
 
    private Map<String, List<Serializable>> securityKeyValues;
    private Map<String, Serializable>       backendVariants;
@@ -360,12 +360,38 @@ public class QSession implements Serializable, Cloneable
          return (false);
       }
 
-      List<Serializable> values      = securityKeyValues.get(keyName);
-      Serializable       valueAsType = ValueUtils.getValueAsFieldType(fieldType, value);
+      List<Serializable> values = securityKeyValues.get(keyName);
+
+      Serializable valueAsType;
+      try
+      {
+         valueAsType = ValueUtils.getValueAsFieldType(fieldType, value);
+      }
+      catch(Exception e)
+      {
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // an exception in getValueAsFieldType would indicate, e.g., a non-number string trying to come back as integer. //
+         // so - assume that any such mismatch means the value isn't in the session.                                      //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         return (false);
+      }
+
       for(Serializable keyValue : values)
       {
-         Serializable keyValueAsType = ValueUtils.getValueAsFieldType(fieldType, keyValue);
-         if(keyValueAsType.equals(valueAsType))
+         Serializable keyValueAsType = null;
+         try
+         {
+            keyValueAsType = ValueUtils.getValueAsFieldType(fieldType, keyValue);
+         }
+         catch(Exception e)
+         {
+            ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // an exception in getValueAsFieldType would indicate, e.g., a non-number string trying to come back as integer. //
+            // so - assume that any such mismatch means this key isn't a match.
+            ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         }
+
+         if(valueAsType.equals(keyValueAsType))
          {
             return (true);
          }
@@ -561,6 +587,7 @@ public class QSession implements Serializable, Cloneable
    }
 
 
+
    /*******************************************************************************
     ** Getter for valuesForFrontend
     *******************************************************************************/
@@ -591,6 +618,7 @@ public class QSession implements Serializable, Cloneable
    }
 
 
+
    /*******************************************************************************
     ** Fluent setter for a single valuesForFrontend
     *******************************************************************************/
@@ -604,5 +632,4 @@ public class QSession implements Serializable, Cloneable
       return (this);
    }
 
-
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/memory/MemoryRecordStore.java

@@ -193,7 +193,7 @@ public class MemoryRecordStore
          if(recordMatches)
          {
             qRecord.setErrors(new ArrayList<>());
-            ValidateRecordSecurityLockHelper.validateSecurityFields(input.getTable(), List.of(qRecord), ValidateRecordSecurityLockHelper.Action.SELECT);
+            ValidateRecordSecurityLockHelper.validateSecurityFields(input.getTable(), List.of(qRecord), ValidateRecordSecurityLockHelper.Action.SELECT, null);
             if(CollectionUtils.nullSafeHasContents(qRecord.getErrors()))
             {
                //////////////////////////////////////////////////////////////////////////////////////////////////////

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertExtractStep.java

@@ -22,84 +22,106 @@
 package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
 
 
+import java.io.InputStream;
 import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Optional;
-import com.kingsrook.qqq.backend.core.adapters.CsvToQRecordAdapter;
-import com.kingsrook.qqq.backend.core.context.QContext;
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
-import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
-import com.kingsrook.qqq.backend.core.model.actions.processes.QUploadedFile;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
-import com.kingsrook.qqq.backend.core.model.actions.shared.mapping.QKeyBasedFieldMapping;
-import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
-import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.RowsToRecordInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep;
-import com.kingsrook.qqq.backend.core.state.AbstractStateKey;
-import com.kingsrook.qqq.backend.core.state.TempFileStateProvider;
 
 
 /*******************************************************************************
  ** Extract step for generic table bulk-insert ETL process
+ **
+ ** This step does a little bit of transforming, actually - taking rows from
+ ** an uploaded file, and potentially merging them (for child-table use-cases)
+ ** and applying the "Mapping" - to put fully built records into the pipe for the
+ ** Transform step.
  *******************************************************************************/
 public class BulkInsertExtractStep extends AbstractExtractStep
 {
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
    @Override
    public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
-      AbstractStateKey        stateKey             = (AbstractStateKey) runBackendStepInput.getValue(QUploadedFile.DEFAULT_UPLOADED_FILE_FIELD_NAME);
-      Optional<QUploadedFile> optionalUploadedFile = TempFileStateProvider.getInstance().get(QUploadedFile.class, stateKey);
-      if(optionalUploadedFile.isEmpty())
+      int rowsAdded     = 0;
+      int originalLimit = Objects.requireNonNullElse(getLimit(), Integer.MAX_VALUE);
+
+      StorageInput          storageInput      = BulkInsertStepUtils.getStorageInputForTheFile(runBackendStepInput);
+      BulkInsertMapping     bulkInsertMapping = (BulkInsertMapping) runBackendStepOutput.getValue("bulkInsertMapping");
+      RowsToRecordInterface rowsToRecord      = bulkInsertMapping.getLayout().newRowsToRecordInterface();
+
+      try
+         (
+            ///////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // open a stream to read from our file, and a FileToRows object, that knows how to read from that stream //
+            ///////////////////////////////////////////////////////////////////////////////////////////////////////////
+            InputStream inputStream = new StorageAction().getInputStream(storageInput);
+            FileToRowsInterface fileToRowsInterface = FileToRowsInterface.forFile(storageInput.getReference(), inputStream);
+         )
       {
-         throw (new QException("Could not find uploaded file"));
-      }
+         ///////////////////////////////////////////////////////////
+         // read the header row (if this file & mapping uses one) //
+         ///////////////////////////////////////////////////////////
+         BulkLoadFileRow headerRow = bulkInsertMapping.getHasHeaderRow() ? fileToRowsInterface.next() : null;
 
-      byte[] bytes    = optionalUploadedFile.get().getBytes();
-      String fileName = optionalUploadedFile.get().getFilename();
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // while there are more rows in the file - and we're under the limit - get more records form the file //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////
+         while(fileToRowsInterface.hasNext() && rowsAdded < originalLimit)
+         {
+            int remainingLimit = originalLimit - rowsAdded;
 
-      /////////////////////////////////////////////////////
-      // let the user specify field labels instead names //
-      /////////////////////////////////////////////////////
-      QTableMetaData        table     = runBackendStepInput.getTable();
-      String                tableName = runBackendStepInput.getTableName();
-      QKeyBasedFieldMapping mapping   = new QKeyBasedFieldMapping();
-      for(Map.Entry<String, QFieldMetaData> entry : table.getFields().entrySet())
-      {
-         mapping.addMapping(entry.getKey(), entry.getValue().getLabel());
-      }
+            ////////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // put a page-size limit on the rows-to-record class, so it won't be tempted to do whole file all at once //
+            ////////////////////////////////////////////////////////////////////////////////////////////////////////////
+            int           pageLimit = Math.min(remainingLimit, getMaxPageSize());
+            List<QRecord> page      = rowsToRecord.nextPage(fileToRowsInterface, headerRow, bulkInsertMapping, pageLimit);
 
-      //////////////////////////////////////////////////////////////////////////
-      // get the non-editable fields - they'll be blanked out in a customizer //
-      //////////////////////////////////////////////////////////////////////////
-      List<QFieldMetaData> nonEditableFields = table.getFields().values().stream()
-         .filter(f -> !f.getIsEditable())
-         .toList();
-
-      if(fileName.toLowerCase(Locale.ROOT).endsWith(".csv"))
-      {
-         new CsvToQRecordAdapter().buildRecordsFromCsv(new CsvToQRecordAdapter.InputWrapper()
-            .withRecordPipe(getRecordPipe())
-            .withLimit(getLimit())
-            .withCsv(new String(bytes))
-            .withDoCorrectValueTypes(true)
-            .withTable(QContext.getQInstance().getTable(tableName))
-            .withMapping(mapping)
-            .withRecordCustomizer((record) ->
+            if(page.size() > remainingLimit)
             {
-               ////////////////////////////////////////////
-               // remove values from non-editable fields //
-               ////////////////////////////////////////////
-               for(QFieldMetaData nonEditableField : nonEditableFields)
-               {
-                  record.setValue(nonEditableField.getName(), null);
-               }
-            }));
+               /////////////////////////////////////////////////////////////
+               // in case we got back more than we asked for, sub-list it //
+               /////////////////////////////////////////////////////////////
+               page = page.subList(0, remainingLimit);
+            }
+
+            /////////////////////////////////////////////
+            // send this page of records into the pipe //
+            /////////////////////////////////////////////
+            getRecordPipe().addRecords(page);
+            rowsAdded += page.size();
+         }
       }
-      else
+      catch(QException qe)
       {
-         throw (new QUserFacingException("Unsupported file type."));
+         throw qe;
       }
+      catch(Exception e)
+      {
+         throw new QException("Unhandled error in bulk insert extract step", e);
+      }
+
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private int getMaxPageSize()
+   {
+      return (1000);
    }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertPrepareFileMappingStep.java

@@ -0,0 +1,175 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.io.File;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.BulkLoadMappingSuggester;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.BulkLoadTableStructureBuilder;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadTableStructure;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkInsertPrepareFileMappingStep implements BackendStep
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      buildFileDetailsForMappingStep(runBackendStepInput, runBackendStepOutput);
+
+      String                 tableName      = runBackendStepInput.getValueString("tableName");
+      BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(tableName);
+      runBackendStepOutput.addValue("tableStructure", tableStructure);
+
+      boolean needSuggestedMapping = true;
+      if(runBackendStepOutput.getProcessState().getIsStepBack())
+      {
+         needSuggestedMapping = false;
+
+         StreamedETLWithFrontendProcess.resetValidationFields(runBackendStepInput);
+      }
+
+      if(needSuggestedMapping)
+      {
+         @SuppressWarnings("unchecked")
+         List<String> headerValues = (List<String>) runBackendStepOutput.getValue("headerValues");
+         buildSuggestedMapping(headerValues, tableStructure, runBackendStepOutput);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void buildSuggestedMapping(List<String> headerValues, BulkLoadTableStructure tableStructure, RunBackendStepOutput runBackendStepOutput)
+   {
+      BulkLoadMappingSuggester bulkLoadMappingSuggester = new BulkLoadMappingSuggester();
+      BulkLoadProfile          bulkLoadProfile          = bulkLoadMappingSuggester.suggestBulkLoadMappingProfile(tableStructure, headerValues);
+      runBackendStepOutput.addValue("bulkLoadProfile", bulkLoadProfile);
+      runBackendStepOutput.addValue("suggestedBulkLoadProfile", bulkLoadProfile);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void buildFileDetailsForMappingStep(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      StorageInput storageInput = BulkInsertStepUtils.getStorageInputForTheFile(runBackendStepInput);
+      File         file         = new File(storageInput.getReference());
+      runBackendStepOutput.addValue("fileBaseName", file.getName());
+
+      try
+         (
+            ///////////////////////////////////////////////////////////////////////////////////////////////////////////
+            // open a stream to read from our file, and a FileToRows object, that knows how to read from that stream //
+            ///////////////////////////////////////////////////////////////////////////////////////////////////////////
+            InputStream inputStream = new StorageAction().getInputStream(storageInput);
+            FileToRowsInterface fileToRowsInterface = FileToRowsInterface.forFile(storageInput.getReference(), inputStream);
+         )
+      {
+         /////////////////////////////////////////////////
+         // read the 1st row, and assume it is a header //
+         /////////////////////////////////////////////////
+         BulkLoadFileRow   headerRow     = fileToRowsInterface.next();
+         ArrayList<String> headerValues  = new ArrayList<>();
+         ArrayList<String> headerLetters = new ArrayList<>();
+         for(int i = 0; i < headerRow.size(); i++)
+         {
+            headerValues.add(ValueUtils.getValueAsString(headerRow.getValue(i)));
+            headerLetters.add(toHeaderLetter(i));
+         }
+         runBackendStepOutput.addValue("headerValues", headerValues);
+         runBackendStepOutput.addValue("headerLetters", headerLetters);
+
+         ///////////////////////////////////////////////////////////////////////////////////////////
+         // while there are more rows in the file - and we're under preview-rows limit, read rows //
+         ///////////////////////////////////////////////////////////////////////////////////////////
+         int                          previewRows      = 0;
+         int                          previewRowsLimit = 5;
+         ArrayList<ArrayList<String>> bodyValues       = new ArrayList<>();
+         for(int i = 0; i < headerRow.size(); i++)
+         {
+            bodyValues.add(new ArrayList<>());
+         }
+
+         while(fileToRowsInterface.hasNext() && previewRows < previewRowsLimit)
+         {
+            BulkLoadFileRow bodyRow = fileToRowsInterface.next();
+            previewRows++;
+
+            for(int i = 0; i < headerRow.size(); i++)
+            {
+               bodyValues.get(i).add(ValueUtils.getValueAsString(bodyRow.getValueElseNull(i)));
+            }
+         }
+         runBackendStepOutput.addValue("bodyValuesPreview", bodyValues);
+
+      }
+      catch(Exception e)
+      {
+         throw (new QException("Error reading bulk load file", e));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static String toHeaderLetter(int i)
+   {
+      StringBuilder rs = new StringBuilder();
+
+      do
+      {
+         rs.insert(0, (char) ('A' + (i % 26)));
+         i = (i / 26) - 1;
+      }
+      while(i >= 0);
+
+      return (rs.toString());
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertPrepareFileUploadStep.java

@@ -0,0 +1,292 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.Collections;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.BulkLoadTableStructureBuilder;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadTableStructure;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ ** step before the upload screen, to prepare dynamic help-text for user.
+ *******************************************************************************/
+public class BulkInsertPrepareFileUploadStep implements BackendStep
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      String                 tableName      = runBackendStepInput.getValueString("tableName");
+      QTableMetaData         table          = QContext.getQInstance().getTable(tableName);
+      BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(tableName);
+      runBackendStepOutput.addValue("tableStructure", tableStructure);
+
+      List<QFieldMetaData> requiredFields   = new ArrayList<>();
+      List<QFieldMetaData> additionalFields = new ArrayList<>();
+      for(QFieldMetaData field : tableStructure.getFields())
+      {
+         if(field.getIsRequired())
+         {
+            requiredFields.add(field);
+         }
+         else
+         {
+            additionalFields.add(field);
+         }
+      }
+
+      StringBuilder html;
+      String        childTableLabels = "";
+
+      StringBuilder tallCSV = new StringBuilder();
+      StringBuilder wideCSV = new StringBuilder();
+      StringBuilder flatCSV = new StringBuilder();
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // potentially this could be a parameter - for now, hard-code false, but keep the code around that did this //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      boolean listFieldsInHelpText = false;
+
+      if(!CollectionUtils.nullSafeHasContents(tableStructure.getAssociations()))
+      {
+         html = new StringBuilder("""
+            <p>Upload either a CSV or Excel (.xlsx) file, with one row for each record you want to
+            insert in the ${tableLabel} table.</p><br />
+            
+            <p>Your file can contain any number of columns.  You will be prompted to map fields from
+            the ${tableLabel} table to columns from your file or default values for all records that
+            you are loading on the next screen.  It is optional (though encouraged) whether you include
+            a header row in your file.  For Excel files, only the first sheet in the workbook will be used.</p><br />
+            """);
+
+         if(listFieldsInHelpText)
+         {
+            appendTableRequiredAndAdditionalFields(html, requiredFields, additionalFields);
+            html.append("""
+               Template: <a href="data:text/csv;base64,${flatCSV}" download="${tableLabel}.csv">${tableLabel}.csv</a>""");
+         }
+         else
+         {
+            html.append("""
+               <p>You can download a template file to see the full list of available fields:
+               <a href="data:text/csv;base64,${flatCSV}" download="${tableLabel}.csv">${tableLabel}.csv</a>
+               </p>
+               """);
+         }
+      }
+      else
+      {
+         childTableLabels = StringUtils.joinWithCommasAndAnd(tableStructure.getAssociations().stream().map(a -> a.getLabel()).toList()) + " table" + StringUtils.plural(table.getAssociations());
+
+         html = new StringBuilder("""
+            <p>Upload either a CSV or Excel (.xlsx) file.  Your file can be in one of three layouts:<p>
+            ${openUL}
+            <li><b>Flat</b>: Each row in the file will create one record in the ${tableLabel} table.</li>
+            <li><b>Wide</b>: Each row in the file will create one record in the ${tableLabel} table,
+            and optionally one or more records in the ${childTableLabels}, by supplying additional columns
+            for each sub-record that you want to create.</li>
+            <li><b>Tall</b>: Rows with matching values in the fields being used for the ${tableLabel}
+            table will be used to create one ${tableLabel} record.  One or more records will also be built
+            in the ${childTableLabels} by providing unique values in each row for the sub-records.</li>
+            </ul><br />
+            
+            <p>Your file can contain any number of columns.  You will be prompted to map fields from
+            the ${tableLabel} table to columns from your file or default values for all records that
+            you are loading on the next screen.  It is optional (though encouraged) whether you include
+            a header row in your file.  For Excel files, only the first sheet in the workbook will be used.</p><br />
+            """);
+
+         if(listFieldsInHelpText)
+         {
+            appendTableRequiredAndAdditionalFields(html, requiredFields, additionalFields);
+         }
+
+         addCsvFields(tallCSV, requiredFields, additionalFields);
+         addCsvFields(wideCSV, requiredFields, additionalFields);
+
+         for(BulkLoadTableStructure association : tableStructure.getAssociations())
+         {
+            if(listFieldsInHelpText)
+            {
+               html.append("""
+                  <p>You can also add values for these ${childLabel} fields:</p>
+                  """.replace("${childLabel}", association.getLabel()));
+               appendFieldsAsUlToHtml(html, association.getFields());
+            }
+
+            addCsvFields(tallCSV, association.getFields(), Collections.emptyList(), association.getLabel() + ": ", "");
+            addCsvFields(wideCSV, association.getFields(), Collections.emptyList(), association.getLabel() + ": ", " - 1");
+            addCsvFields(wideCSV, association.getFields(), Collections.emptyList(), association.getLabel() + ": ", " - 2");
+         }
+
+         finishCSV(tallCSV);
+         finishCSV(wideCSV);
+
+         if(listFieldsInHelpText)
+         {
+            html.append("""
+               Templates: <a href="data:text/csv;base64,${flatCSV}" download="${tableLabel} - Flat.csv">${tableLabel} - Flat.csv</a>
+               | <a href="data:text/csv;base64,${tallCSV}" download="${tableLabel} - Tall.csv">${tableLabel} - Tall.csv</a>
+               | <a href="data:text/csv;base64,${wideCSV}" download="${tableLabel} - Wide.csv">${tableLabel} - Wide.csv</a>
+               """);
+         }
+         else
+         {
+            html.append("""
+               <p>You can download a template file to see the full list of available fields:
+               <a href="data:text/csv;base64,${flatCSV}" download="${tableLabel} - Flat.csv">${tableLabel} - Flat.csv</a>
+               | <a href="data:text/csv;base64,${tallCSV}" download="${tableLabel} - Tall.csv">${tableLabel} - Tall.csv</a>
+               | <a href="data:text/csv;base64,${wideCSV}" download="${tableLabel} - Wide.csv">${tableLabel} - Wide.csv</a>
+               </p>
+               """);
+         }
+      }
+
+      html.insert(0, """
+         <details style="margin-top: 1rem; border: 1px solid gray; padding: 0.5rem; border-radius: 0.5rem; font-size: 0.875rem;">
+            <summary style="cursor: pointer;">File Upload Instructions</summary>
+            <div style="padding-top: 0.5rem;"></div>
+         """);
+      html.append("</details>");
+
+      addCsvFields(flatCSV, requiredFields, additionalFields);
+      finishCSV(flatCSV);
+
+      String htmlString = html.toString()
+         .replace("${tableLabel}", table.getLabel())
+         .replace("${childTableLabels}", childTableLabels)
+         .replace("${flatCSV}", Base64.getEncoder().encodeToString(flatCSV.toString().getBytes(StandardCharsets.UTF_8)))
+         .replace("${tallCSV}", Base64.getEncoder().encodeToString(tallCSV.toString().getBytes(StandardCharsets.UTF_8)))
+         .replace("${wideCSV}", Base64.getEncoder().encodeToString(wideCSV.toString().getBytes(StandardCharsets.UTF_8)))
+         .replace("${openUL}", "<ul style='margin-left: 2rem;'>");
+
+      runBackendStepOutput.addValue("upload.html", htmlString);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void finishCSV(StringBuilder flatCSV)
+   {
+      flatCSV.deleteCharAt(flatCSV.length() - 1);
+      flatCSV.append("\n");
+      flatCSV.append(flatCSV.toString().replaceAll("[^,]", ""));
+      flatCSV.append("\n");
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void addCsvFields(StringBuilder csv, List<QFieldMetaData> requiredFields, List<QFieldMetaData> additionalFields)
+   {
+      addCsvFields(csv, requiredFields, additionalFields, "", "");
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void addCsvFields(StringBuilder csv, List<QFieldMetaData> requiredFields, List<QFieldMetaData> additionalFields, String fieldLabelPrefix, String fieldLabelSuffix)
+   {
+      for(QFieldMetaData field : requiredFields)
+      {
+         csv.append(fieldLabelPrefix).append(field.getLabel()).append(fieldLabelSuffix).append(",");
+      }
+
+      for(QFieldMetaData field : additionalFields)
+      {
+         csv.append(fieldLabelPrefix).append(field.getLabel()).append(fieldLabelSuffix).append(",");
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void appendTableRequiredAndAdditionalFields(StringBuilder html, List<QFieldMetaData> requiredFields, List<QFieldMetaData> additionalFields)
+   {
+      if(!requiredFields.isEmpty())
+      {
+         html.append("""
+            <p> You will be required to supply values (either in a column in the file, or by
+            choosing a default value on the next screen) for the following ${tableLabel} fields:</p>
+            """);
+         appendFieldsAsUlToHtml(html, requiredFields);
+      }
+
+      if(!additionalFields.isEmpty())
+      {
+         if(requiredFields.isEmpty())
+         {
+            html.append("""
+               <p>You can supply values (either in a column in the file, or by choosing a
+               default value on the next screen) for the following ${tableLabel} fields:</p>
+               """);
+         }
+         else
+         {
+            html.append("<p>You can also add values for these fields:</p>");
+         }
+
+         appendFieldsAsUlToHtml(html, additionalFields);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void appendFieldsAsUlToHtml(StringBuilder html, List<QFieldMetaData> additionalFields)
+   {
+      html.append("${openUL}");
+      for(QFieldMetaData field : additionalFields)
+      {
+         html.append("<li>").append(field.getLabel()).append("</li>");
+      }
+      html.append("</ul><br />");
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertPrepareValueMappingStep.java

@@ -0,0 +1,256 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.io.InputStream;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.google.gson.reflect.TypeToken;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
+import com.kingsrook.qqq.backend.core.actions.values.SearchPossibleValueSourceAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceOutput;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkInsertPrepareValueMappingStep implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(BulkInsertPrepareValueMappingStep.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      try
+      {
+         if(runBackendStepOutput.getProcessState().getIsStepBack())
+         {
+            StreamedETLWithFrontendProcess.resetValidationFields(runBackendStepInput);
+         }
+
+         /////////////////////////////////////////////////////////////
+         // prep the frontend for what field we're going to map now //
+         /////////////////////////////////////////////////////////////
+         List<String> fieldNamesToDoValueMapping = (List<String>) runBackendStepInput.getValue("fieldNamesToDoValueMapping");
+         Integer      valueMappingFieldIndex     = runBackendStepInput.getValueInteger("valueMappingFieldIndex");
+         if(valueMappingFieldIndex == null)
+         {
+            valueMappingFieldIndex = 0;
+         }
+         else
+         {
+            valueMappingFieldIndex++;
+         }
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////
+         // if there are no more fields (values) to map, then proceed to the standard streamed-ETL preview //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////
+         if(valueMappingFieldIndex >= fieldNamesToDoValueMapping.size())
+         {
+            BulkInsertStepUtils.setNextStepStreamedETLPreview(runBackendStepOutput);
+            return;
+         }
+
+         runBackendStepInput.addValue("valueMappingFieldIndex", valueMappingFieldIndex);
+
+         String        fullFieldName = fieldNamesToDoValueMapping.get(valueMappingFieldIndex);
+         TableAndField tableAndField = getTableAndField(runBackendStepInput.getValueString("tableName"), fullFieldName);
+
+         runBackendStepInput.addValue("valueMappingField", new QFrontendFieldMetaData(tableAndField.field()));
+         runBackendStepInput.addValue("valueMappingFullFieldName", fullFieldName);
+         runBackendStepInput.addValue("valueMappingFieldTableName", tableAndField.table().getName());
+
+         ////////////////////////////////////////////////////
+         // get all the values from the file in this field //
+         // todo - should do all mapping fields at once?   //
+         ////////////////////////////////////////////////////
+         ArrayList<Serializable> fileValues = getValuesForField(tableAndField.table(), tableAndField.field(), fullFieldName, runBackendStepInput);
+         runBackendStepOutput.addValue("fileValues", fileValues);
+
+         ///////////////////////////////////////////////
+         // clear these in case not getting set below //
+         ///////////////////////////////////////////////
+         runBackendStepOutput.addValue("valueMapping", new HashMap<>());
+         runBackendStepOutput.addValue("mappedValueLabels", new HashMap<>());
+
+         BulkInsertMapping             bulkInsertMapping = (BulkInsertMapping) runBackendStepInput.getValue("bulkInsertMapping");
+         HashMap<String, Serializable> valueMapping      = null;
+         if(bulkInsertMapping.getFieldNameToValueMapping() != null && bulkInsertMapping.getFieldNameToValueMapping().containsKey(fullFieldName))
+         {
+            valueMapping = CollectionUtils.useOrWrap(bulkInsertMapping.getFieldNameToValueMapping().get(fullFieldName), new TypeToken<>() {});
+            runBackendStepOutput.addValue("valueMapping", valueMapping);
+
+            if(StringUtils.hasContent(tableAndField.field().getPossibleValueSourceName()))
+            {
+               HashMap<Serializable, String> possibleValueLabels = loadPossibleValues(tableAndField.field(), valueMapping);
+               runBackendStepOutput.addValue("mappedValueLabels", possibleValueLabels);
+            }
+         }
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error in bulk insert prepare value mapping", e);
+         throw new QException("Unhandled error in bulk insert prepare value mapping step", e);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static TableAndField getTableAndField(String tableName, String fullFieldName) throws QException
+   {
+      List<String> parts         = new ArrayList<>(List.of(fullFieldName.split("\\.")));
+      String       fieldBaseName = parts.remove(parts.size() - 1);
+
+      QTableMetaData table = QContext.getQInstance().getTable(tableName);
+      for(String associationName : parts)
+      {
+         Optional<Association> association = table.getAssociations().stream().filter(a -> a.getName().equals(associationName)).findFirst();
+         if(association.isPresent())
+         {
+            table = QContext.getQInstance().getTable(association.get().getAssociatedTableName());
+         }
+         else
+         {
+            throw new QException("Missing association [" + associationName + "] on table [" + table.getName() + "]");
+         }
+      }
+
+      TableAndField result = new TableAndField(table, table.getField(fieldBaseName));
+      return result;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public record TableAndField(QTableMetaData table, QFieldMetaData field) {}
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private HashMap<Serializable, String> loadPossibleValues(QFieldMetaData field, Map<String, Serializable> valueMapping) throws QException
+   {
+      SearchPossibleValueSourceInput input = new SearchPossibleValueSourceInput();
+      input.setPossibleValueSourceName(field.getPossibleValueSourceName());
+      input.setIdList(new ArrayList<>(new HashSet<>(valueMapping.values()))); // go through a set to strip dupes
+      SearchPossibleValueSourceOutput output = new SearchPossibleValueSourceAction().execute(input);
+
+      HashMap<Serializable, String> rs = new HashMap<>();
+      for(QPossibleValue<?> result : output.getResults())
+      {
+         Serializable id = (Serializable) result.getId();
+         rs.put(id, result.getLabel());
+      }
+      return rs;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private ArrayList<Serializable> getValuesForField(QTableMetaData table, QFieldMetaData field, String fullFieldName, RunBackendStepInput runBackendStepInput) throws QException
+   {
+      StorageInput      storageInput      = BulkInsertStepUtils.getStorageInputForTheFile(runBackendStepInput);
+      BulkInsertMapping bulkInsertMapping = (BulkInsertMapping) runBackendStepInput.getValue("bulkInsertMapping");
+
+      String associationNameChain = null;
+      if(fullFieldName.contains("."))
+      {
+         associationNameChain = fullFieldName.substring(0, fullFieldName.lastIndexOf('.'));
+      }
+
+      try
+         (
+            InputStream inputStream = new StorageAction().getInputStream(storageInput);
+            FileToRowsInterface fileToRowsInterface = FileToRowsInterface.forFile(storageInput.getReference(), inputStream);
+         )
+      {
+         Set<String>          values       = new LinkedHashSet<>();
+         BulkLoadFileRow      headerRow    = bulkInsertMapping.getHasHeaderRow() ? fileToRowsInterface.next() : null;
+         Map<String, Integer> fieldIndexes = bulkInsertMapping.getFieldIndexes(table, associationNameChain, headerRow);
+         int                  index        = fieldIndexes.get(field.getName());
+
+         while(fileToRowsInterface.hasNext())
+         {
+            BulkLoadFileRow row   = fileToRowsInterface.next();
+            Serializable    value = row.getValueElseNull(index);
+            if(value != null)
+            {
+               values.add(ValueUtils.getValueAsString(value));
+            }
+
+            if(values.size() > 100)
+            {
+               throw (new QUserFacingException("Too many unique values were found for mapping for field: " + field.getName()));
+            }
+         }
+
+         return (new ArrayList<>(values));
+      }
+      catch(Exception e)
+      {
+         throw (new QException("Error getting values from file", e));
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertReceiveFileMappingStep.java

@@ -0,0 +1,210 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.io.InputStream;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import org.apache.commons.lang3.BooleanUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkInsertReceiveFileMappingStep implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(BulkInsertReceiveFileMappingStep.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      try
+      {
+         BulkInsertStepUtils.handleSavedBulkLoadProfileIdValue(runBackendStepInput, runBackendStepOutput);
+
+         ///////////////////////////////////////////////////////////////////
+         // read process values - construct a bulkLoadProfile out of them //
+         ///////////////////////////////////////////////////////////////////
+         BulkLoadProfile bulkLoadProfile = BulkInsertStepUtils.getBulkLoadProfile(runBackendStepInput);
+
+         /////////////////////////////////////////////////////////////////////////
+         // put the list of bulk load profile into the process state - it's the //
+         // thing that the frontend will be looking at as the saved profile     //
+         /////////////////////////////////////////////////////////////////////////
+         runBackendStepOutput.addValue("bulkLoadProfile", bulkLoadProfile);
+
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // now build the mapping object that the backend wants - based on the bulkLoadProfile from the frontend //
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////
+         BulkInsertMapping bulkInsertMapping = new BulkInsertMapping();
+         bulkInsertMapping.setTableName(runBackendStepInput.getTableName());
+         bulkInsertMapping.setHasHeaderRow(bulkLoadProfile.getHasHeaderRow());
+         bulkInsertMapping.setLayout(BulkInsertMapping.Layout.valueOf(bulkLoadProfile.getLayout()));
+
+         //////////////////////////////////////////////////////////////////////////////////////////////
+         // handle field to name or index mappings (depending on if there's a header row being used) //
+         //////////////////////////////////////////////////////////////////////////////////////////////
+         if(BooleanUtils.isTrue(bulkInsertMapping.getHasHeaderRow()))
+         {
+            StorageInput storageInput = BulkInsertStepUtils.getStorageInputForTheFile(runBackendStepInput);
+            try
+               (
+                  InputStream inputStream = new StorageAction().getInputStream(storageInput);
+                  FileToRowsInterface fileToRowsInterface = FileToRowsInterface.forFile(storageInput.getReference(), inputStream);
+               )
+            {
+               Map<String, String> fieldNameToHeaderNameMap = new HashMap<>();
+               bulkInsertMapping.setFieldNameToHeaderNameMap(fieldNameToHeaderNameMap);
+
+               BulkLoadFileRow headerRow = fileToRowsInterface.next();
+               for(BulkLoadProfileField bulkLoadProfileField : bulkLoadProfile.getFieldList())
+               {
+                  if(bulkLoadProfileField.getHeaderName() != null)
+                  {
+                     String headerName = bulkLoadProfileField.getHeaderName();
+                     fieldNameToHeaderNameMap.put(bulkLoadProfileField.getFieldName(), headerName);
+                  }
+                  else if(bulkLoadProfileField.getColumnIndex() != null)
+                  {
+                     String headerName = ValueUtils.getValueAsString(headerRow.getValueElseNull(bulkLoadProfileField.getColumnIndex()));
+                     fieldNameToHeaderNameMap.put(bulkLoadProfileField.getFieldName(), headerName);
+                  }
+               }
+            }
+         }
+         else
+         {
+            Map<String, Integer> fieldNameToIndexMap = new HashMap<>();
+            bulkInsertMapping.setFieldNameToIndexMap(fieldNameToIndexMap);
+            for(BulkLoadProfileField bulkLoadProfileField : bulkLoadProfile.getFieldList())
+            {
+               if(bulkLoadProfileField.getColumnIndex() != null)
+               {
+                  fieldNameToIndexMap.put(bulkLoadProfileField.getFieldName(), bulkLoadProfileField.getColumnIndex());
+               }
+            }
+         }
+
+         /////////////////////////////////////
+         // do fields w/ default values now //
+         /////////////////////////////////////
+         HashMap<String, Serializable> fieldNameToDefaultValueMap = new HashMap<>();
+         bulkInsertMapping.setFieldNameToDefaultValueMap(fieldNameToDefaultValueMap);
+         for(BulkLoadProfileField bulkLoadProfileField : bulkLoadProfile.getFieldList())
+         {
+            if(bulkLoadProfileField.getDefaultValue() != null)
+            {
+               fieldNameToDefaultValueMap.put(bulkLoadProfileField.getFieldName(), bulkLoadProfileField.getDefaultValue());
+            }
+         }
+
+         /////////////////////////////////////////////////////////////////////////////////////////////
+         // frontend at this point will have sent just told us which field names need value mapping //
+         // store those - and let them drive the value-mapping screens that we'll go through next   //
+         // todo - uh, what if those come from profile, dummy!?
+         /////////////////////////////////////////////////////////////////////////////////////////////
+         ArrayList<String> fieldNamesToDoValueMapping = new ArrayList<>();
+         for(BulkLoadProfileField bulkLoadProfileField : bulkLoadProfile.getFieldList())
+         {
+            if(BooleanUtils.isTrue(bulkLoadProfileField.getDoValueMapping()))
+            {
+               fieldNamesToDoValueMapping.add(bulkLoadProfileField.getFieldName());
+
+               if(CollectionUtils.nullSafeHasContents(bulkLoadProfileField.getValueMappings()))
+               {
+                  bulkInsertMapping.getFieldNameToValueMapping().put(bulkLoadProfileField.getFieldName(), bulkLoadProfileField.getValueMappings());
+               }
+            }
+         }
+         runBackendStepOutput.addValue("fieldNamesToDoValueMapping", new ArrayList<>(fieldNamesToDoValueMapping));
+
+         ///////////////////////////////////////////////////////////////////////////////////////
+         // figure out what associations are being mapped, by looking at the full field names //
+         ///////////////////////////////////////////////////////////////////////////////////////
+         Set<String> associationNameSet = new HashSet<>();
+         for(BulkLoadProfileField bulkLoadProfileField : bulkLoadProfile.getFieldList())
+         {
+            if(bulkLoadProfileField.getFieldName().contains("."))
+            {
+               //////////////////////////////////////////////////////////////////////////////////////////////////////////
+               // handle parent.child.grandchild.fieldName,index.index.index if we do sub-indexes for grandchildren... //
+               //////////////////////////////////////////////////////////////////////////////////////////////////////////
+               String fieldNameBeforeIndex = bulkLoadProfileField.getFieldName().split(",")[0];
+               associationNameSet.add(fieldNameBeforeIndex.substring(0, fieldNameBeforeIndex.lastIndexOf('.')));
+            }
+         }
+         bulkInsertMapping.setMappedAssociations(new ArrayList<>(associationNameSet));
+
+         /////////////////////////////////////////////////////////////////////////////////////////////////////
+         // at this point we're done populating the bulkInsertMapping object.  put it in the process state. //
+         /////////////////////////////////////////////////////////////////////////////////////////////////////
+         runBackendStepOutput.addValue("bulkInsertMapping", bulkInsertMapping);
+
+         if(CollectionUtils.nullSafeHasContents(fieldNamesToDoValueMapping))
+         {
+            //////////////////////////////////////////////////////////////////////////////////
+            // just go to the prepareValueMapping backend step - it'll figure out the rest. //
+            // it's also where the value-mapping loop of steps points.                      //
+            // and, this will actually be the default (e.g., the step after this one).      //
+            //////////////////////////////////////////////////////////////////////////////////
+            runBackendStepInput.addValue("valueMappingFieldIndex", -1);
+         }
+         else
+         {
+            //////////////////////////////////////////////////////////////////////////////////
+            // else - if no values to map - continue with the standard streamed-ETL preview //
+            //////////////////////////////////////////////////////////////////////////////////
+            BulkInsertStepUtils.setNextStepStreamedETLPreview(runBackendStepOutput);
+         }
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error in bulk insert receive mapping", e);
+         throw new QException("Unhandled error in bulk insert receive mapping step", e);
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertReceiveValueMappingStep.java

@@ -0,0 +1,105 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkInsertReceiveValueMappingStep implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(BulkInsertReceiveValueMappingStep.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      try
+      {
+         BulkInsertStepUtils.handleSavedBulkLoadProfileIdValue(runBackendStepInput, runBackendStepOutput);
+
+         List<String> fieldNamesToDoValueMapping = (List<String>) runBackendStepInput.getValue("fieldNamesToDoValueMapping");
+         Integer      valueMappingFieldIndex     = runBackendStepInput.getValueInteger("valueMappingFieldIndex");
+
+         String fieldName = fieldNamesToDoValueMapping.get(valueMappingFieldIndex);
+
+         ///////////////////////////////////////////////////////////////////
+         // read process values - construct a bulkLoadProfile out of them //
+         ///////////////////////////////////////////////////////////////////
+         BulkLoadProfile bulkLoadProfile = BulkInsertStepUtils.getBulkLoadProfile(runBackendStepInput);
+
+         /////////////////////////////////////////////////////////////////////////
+         // put the list of bulk load profile into the process state - it's the //
+         // thing that the frontend will be looking at as the saved profile     //
+         /////////////////////////////////////////////////////////////////////////
+         runBackendStepOutput.addValue("bulkLoadProfile", bulkLoadProfile);
+
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // get the bulkInsertMapping object from the process, creating a fieldNameToValueMapping map within it if needed //
+         ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         BulkInsertMapping                      bulkInsertMapping       = (BulkInsertMapping) runBackendStepOutput.getValue("bulkInsertMapping");
+         Map<String, Map<String, Serializable>> fieldNameToValueMapping = bulkInsertMapping.getFieldNameToValueMapping();
+         if(fieldNameToValueMapping == null)
+         {
+            fieldNameToValueMapping = new HashMap<>();
+            bulkInsertMapping.setFieldNameToValueMapping(fieldNameToValueMapping);
+         }
+         runBackendStepOutput.addValue("bulkInsertMapping", bulkInsertMapping);
+
+         ////////////////////////////////////////////////
+         // put the mapped values into the mapping map //
+         ////////////////////////////////////////////////
+         Map<String, Serializable> mappedValues = JsonUtils.toObject(runBackendStepInput.getValueString("mappedValuesJSON"), new TypeReference<>() {});
+         fieldNameToValueMapping.put(fieldName, mappedValues);
+
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // always return to the prepare-mapping step - as it will determine if it's time to break the loop or not. //
+         /////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         BulkInsertStepUtils.setNextStepPrepareValueMapping(runBackendStepOutput);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error in bulk insert receive mapping", e);
+         throw new QException("Unhandled error in bulk insert receive mapping step", e);
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertStepUtils.java

@@ -0,0 +1,156 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import org.apache.commons.lang3.BooleanUtils;
+import org.json.JSONArray;
+import org.json.JSONObject;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkInsertStepUtils
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static StorageInput getStorageInputForTheFile(RunBackendStepInput input) throws QException
+   {
+      @SuppressWarnings("unchecked")
+      ArrayList<StorageInput> storageInputs = (ArrayList<StorageInput>) input.getValue("theFile");
+      if(storageInputs == null)
+      {
+         throw (new QException("StorageInputs for theFile were not found in process state"));
+      }
+
+      if(storageInputs.isEmpty())
+      {
+         throw (new QException("StorageInputs for theFile was an empty list"));
+      }
+
+      StorageInput storageInput = storageInputs.get(0);
+      return (storageInput);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static void setNextStepStreamedETLPreview(RunBackendStepOutput runBackendStepOutput)
+   {
+      runBackendStepOutput.setOverrideLastStepName("receiveValueMapping");
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static void setNextStepPrepareValueMapping(RunBackendStepOutput runBackendStepOutput)
+   {
+      runBackendStepOutput.setOverrideLastStepName("receiveFileMapping");
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static BulkLoadProfile getBulkLoadProfile(RunBackendStepInput runBackendStepInput)
+   {
+      String version = runBackendStepInput.getValueString("version");
+      if("v1".equals(version))
+      {
+         String  layout       = runBackendStepInput.getValueString("layout");
+         Boolean hasHeaderRow = runBackendStepInput.getValueBoolean("hasHeaderRow");
+
+         ArrayList<BulkLoadProfileField> fieldList = new ArrayList<>();
+
+         JSONArray array = new JSONArray(runBackendStepInput.getValueString("fieldListJSON"));
+         for(int i = 0; i < array.length(); i++)
+         {
+            JSONObject           jsonObject           = array.getJSONObject(i);
+            BulkLoadProfileField bulkLoadProfileField = new BulkLoadProfileField();
+            fieldList.add(bulkLoadProfileField);
+            bulkLoadProfileField.setFieldName(jsonObject.optString("fieldName"));
+            bulkLoadProfileField.setHeaderName(jsonObject.has("headerName") ? jsonObject.getString("headerName") : null);
+            bulkLoadProfileField.setColumnIndex(jsonObject.has("columnIndex") ? jsonObject.getInt("columnIndex") : null);
+            bulkLoadProfileField.setDefaultValue((Serializable) jsonObject.opt("defaultValue"));
+            bulkLoadProfileField.setDoValueMapping(jsonObject.optBoolean("doValueMapping"));
+
+            if(BooleanUtils.isTrue(bulkLoadProfileField.getDoValueMapping()) && jsonObject.has("valueMappings"))
+            {
+               bulkLoadProfileField.setValueMappings(new HashMap<>());
+               JSONObject valueMappingsJsonObject = jsonObject.getJSONObject("valueMappings");
+               for(String fileValue : valueMappingsJsonObject.keySet())
+               {
+                  bulkLoadProfileField.getValueMappings().put(fileValue, ValueUtils.getValueAsString(valueMappingsJsonObject.get(fileValue)));
+               }
+            }
+         }
+
+         BulkLoadProfile bulkLoadProfile = new BulkLoadProfile()
+            .withVersion(version)
+            .withFieldList(fieldList)
+            .withHasHeaderRow(hasHeaderRow)
+            .withLayout(layout);
+
+         return (bulkLoadProfile);
+      }
+      else
+      {
+         throw (new IllegalArgumentException("Unexpected version for bulk load profile: " + version));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static void handleSavedBulkLoadProfileIdValue(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      Integer savedBulkLoadProfileId = runBackendStepInput.getValueInteger("savedBulkLoadProfileId");
+      if(savedBulkLoadProfileId != null)
+      {
+         QRecord savedBulkLoadProfileRecord = GetAction.execute(SavedBulkLoadProfile.TABLE_NAME, savedBulkLoadProfileId);
+         runBackendStepOutput.addValue("savedBulkLoadProfileRecord", savedBulkLoadProfileRecord);
+      }
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertTransformStep.java

@@ -29,6 +29,7 @@ import java.util.HashSet;
 import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
 import com.kingsrook.qqq.backend.core.actions.customizers.AbstractPreInsertCustomizer;
@@ -47,15 +48,25 @@ import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutp
 import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
 import com.kingsrook.qqq.backend.core.model.actions.tables.QInputSource;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
+import com.kingsrook.qqq.backend.core.model.statusmessages.QErrorMessage;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.AbstractBulkLoadRollableValueError;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.BulkLoadRecordUtils;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.LoadViaInsertStep;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
 import com.kingsrook.qqq.backend.core.processes.implementations.general.ProcessSummaryWarningsAndErrorsRollup;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ListingHash;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
 
 
 /*******************************************************************************
@@ -65,9 +76,14 @@ public class BulkInsertTransformStep extends AbstractTransformStep
 {
    private ProcessSummaryLine okSummary = new ProcessSummaryLine(Status.OK);
 
-   private ProcessSummaryWarningsAndErrorsRollup processSummaryWarningsAndErrorsRollup = ProcessSummaryWarningsAndErrorsRollup.build("inserted");
+   private ProcessSummaryWarningsAndErrorsRollup processSummaryWarningsAndErrorsRollup = ProcessSummaryWarningsAndErrorsRollup.build("inserted")
+      .withDoReplaceSingletonCountLinesWithSuffixOnly(false);
 
-   private Map<UniqueKey, ProcessSummaryLineWithUKSampleValues> ukErrorSummaries = new HashMap<>();
+   private ListingHash<String, RowValue> errorToExampleRowValueMap = new ListingHash<>();
+   private ListingHash<String, String>   errorToExampleRowsMap     = new ListingHash<>();
+
+   private Map<UniqueKey, ProcessSummaryLineWithUKSampleValues> ukErrorSummaries              = new HashMap<>();
+   private Map<String, ProcessSummaryLine>                      associationsToInsertSummaries = new HashMap<>();
 
    private QTableMetaData table;
 
@@ -75,6 +91,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
 
    private int rowsProcessed = 0;
 
+   private static final int EXAMPLE_ROW_LIMIT = 10;
 
 
    /*******************************************************************************
@@ -111,6 +128,49 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       // since we're doing a unique key check in this class, we can tell the loadViaInsert step that it (rather, the InsertAction) doesn't need to re-do one. //
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       runBackendStepOutput.addValue(LoadViaInsertStep.FIELD_SKIP_UNIQUE_KEY_CHECK, true);
+
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // make sure that if a saved profile was selected on a review screen, that the result screen knows about it. //
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      BulkInsertStepUtils.handleSavedBulkLoadProfileIdValue(runBackendStepInput, runBackendStepOutput);
+
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // set up the validationReview widget to render preview records using the table layout, and including the associations //
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      runBackendStepOutput.addValue("formatPreviewRecordUsingTableLayout", table.getName());
+
+      BulkInsertMapping bulkInsertMapping = (BulkInsertMapping) runBackendStepOutput.getValue("bulkInsertMapping");
+      if(bulkInsertMapping != null)
+      {
+         ArrayList<String> previewRecordAssociatedTableNames  = new ArrayList<>();
+         ArrayList<String> previewRecordAssociatedWidgetNames = new ArrayList<>();
+         ArrayList<String> previewRecordAssociationNames      = new ArrayList<>();
+
+         for(String mappedAssociation : bulkInsertMapping.getMappedAssociations())
+         {
+            Optional<Association> association = table.getAssociations().stream().filter(a -> a.getName().equals(mappedAssociation)).findFirst();
+            if(association.isPresent())
+            {
+               for(QFieldSection section : table.getSections())
+               {
+                  QWidgetMetaDataInterface widget = QContext.getQInstance().getWidget(section.getWidgetName());
+                  if(widget != null && WidgetType.CHILD_RECORD_LIST.getType().equals(widget.getType()))
+                  {
+                     Serializable widgetJoinName = widget.getDefaultValues().get("joinName");
+                     if(Objects.equals(widgetJoinName, association.get().getJoinName()))
+                     {
+                        previewRecordAssociatedTableNames.add(association.get().getAssociatedTableName());
+                        previewRecordAssociatedWidgetNames.add(widget.getName());
+                        previewRecordAssociationNames.add(association.get().getName());
+                     }
+                  }
+               }
+            }
+         }
+         runBackendStepOutput.addValue("previewRecordAssociatedTableNames", previewRecordAssociatedTableNames);
+         runBackendStepOutput.addValue("previewRecordAssociatedWidgetNames", previewRecordAssociatedWidgetNames);
+         runBackendStepOutput.addValue("previewRecordAssociationNames", previewRecordAssociationNames);
+      }
    }
 
 
@@ -121,8 +181,55 @@ public class BulkInsertTransformStep extends AbstractTransformStep
    @Override
    public void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
    {
-      int            rowsInThisPage = runBackendStepInput.getRecords().size();
-      QTableMetaData table          = QContext.getQInstance().getTable(runBackendStepInput.getTableName());
+      int            recordsInThisPage = runBackendStepInput.getRecords().size();
+      QTableMetaData table             = QContext.getQInstance().getTable(runBackendStepInput.getTableName());
+
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // split the records into 2 lists:  those w/ errors (e.g., from the bulk-load mapping), and those that are okay //
+      //////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      List<QRecord> recordsWithoutAnyErrors = new ArrayList<>();
+      List<QRecord> recordsWithSomeErrors   = new ArrayList<>();
+      for(QRecord record : runBackendStepInput.getRecords())
+      {
+         if(CollectionUtils.nullSafeHasContents(record.getErrors()))
+         {
+            recordsWithSomeErrors.add(record);
+         }
+         else
+         {
+            recordsWithoutAnyErrors.add(record);
+         }
+      }
+
+      //////////////////////////////////////////////////////////////////
+      // propagate errors that came into this step out to the summary //
+      //////////////////////////////////////////////////////////////////
+      if(!recordsWithSomeErrors.isEmpty())
+      {
+         for(QRecord record : recordsWithSomeErrors)
+         {
+            for(QErrorMessage error : record.getErrors())
+            {
+               if(error instanceof AbstractBulkLoadRollableValueError rollableValueError)
+               {
+                  processSummaryWarningsAndErrorsRollup.addError(rollableValueError.getMessageToUseAsProcessSummaryRollupKey(), null);
+                  addToErrorToExampleRowValueMap(rollableValueError, record);
+               }
+               else
+               {
+                  processSummaryWarningsAndErrorsRollup.addError(error.getMessage(), null);
+               }
+            }
+         }
+      }
+
+      if(recordsWithoutAnyErrors.isEmpty())
+      {
+         /////////////////////////////////////////////////////////////////////////////////
+         // skip the rest of this method if there aren't any records w/o errors in them //
+         /////////////////////////////////////////////////////////////////////////////////
+         this.rowsProcessed += recordsInThisPage;
+      }
 
       /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       // set up an insert-input, which will be used as input to the pre-customizer as well as for additional validations //
@@ -130,7 +237,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       InsertInput insertInput = new InsertInput();
       insertInput.setInputSource(QInputSource.USER);
       insertInput.setTableName(runBackendStepInput.getTableName());
-      insertInput.setRecords(runBackendStepInput.getRecords());
+      insertInput.setRecords(recordsWithoutAnyErrors);
       insertInput.setSkipUniqueKeyCheck(true);
 
       //////////////////////////////////////////////////////////////////////
@@ -145,7 +252,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
          AbstractPreInsertCustomizer.WhenToRun whenToRun = preInsertCustomizer.get().whenToRunPreInsert(insertInput, true);
          if(WhenToRun.BEFORE_ALL_VALIDATIONS.equals(whenToRun) || WhenToRun.BEFORE_UNIQUE_KEY_CHECKS.equals(whenToRun))
          {
-            List<QRecord> recordsAfterCustomizer = preInsertCustomizer.get().preInsert(insertInput, runBackendStepInput.getRecords(), true);
+            List<QRecord> recordsAfterCustomizer = preInsertCustomizer.get().preInsert(insertInput, recordsWithoutAnyErrors, true);
             runBackendStepInput.setRecords(recordsAfterCustomizer);
 
             ///////////////////////////////////////////////////////////////////////////////////////
@@ -159,13 +266,14 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       List<UniqueKey>                         uniqueKeys   = CollectionUtils.nonNullList(table.getUniqueKeys());
       for(UniqueKey uniqueKey : uniqueKeys)
       {
-         existingKeys.put(uniqueKey, UniqueKeyHelper.getExistingKeys(null, table, runBackendStepInput.getRecords(), uniqueKey).keySet());
+         existingKeys.put(uniqueKey, UniqueKeyHelper.getExistingKeys(null, table, recordsWithoutAnyErrors, uniqueKey).keySet());
          ukErrorSummaries.computeIfAbsent(uniqueKey, x -> new ProcessSummaryLineWithUKSampleValues(Status.ERROR));
       }
 
       /////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       // on the validate step, we haven't read the full file, so we don't know how many rows there are - thus        //
       // record count is null, and the ValidateStep won't be setting status counters - so - do it here in that case. //
+      // todo - move this up (before the early return?)                                                              //
       /////////////////////////////////////////////////////////////////////////////////////////////////////////////////
       if(runBackendStepInput.getStepName().equals(StreamedETLWithFrontendProcess.STEP_NAME_VALIDATE))
       {
@@ -187,7 +295,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       // Note, we want to do our own UK checking here, even though InsertAction also tries to do it, because InsertAction //
       // will only be getting the records in pages, but in here, we'll track UK's across pages!!                          //
       //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-      List<QRecord> recordsWithoutUkErrors = getRecordsWithoutUniqueKeyErrors(runBackendStepInput, existingKeys, uniqueKeys, table);
+      List<QRecord> recordsWithoutUkErrors = getRecordsWithoutUniqueKeyErrors(recordsWithoutAnyErrors, existingKeys, uniqueKeys, table);
 
       /////////////////////////////////////////////////////////////////////////////////
       // run all validation from the insert action - in Preview mode (boolean param) //
@@ -205,8 +313,11 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       {
          if(CollectionUtils.nullSafeHasContents(record.getErrors()))
          {
-            String message = record.getErrors().get(0).getMessage();
-            processSummaryWarningsAndErrorsRollup.addError(message, null);
+            for(QErrorMessage error : record.getErrors())
+            {
+               processSummaryWarningsAndErrorsRollup.addError(error.getMessage(), null);
+               addToErrorToExampleRowMap(error.getMessage(), record);
+            }
          }
          else if(CollectionUtils.nullSafeHasContents(record.getWarnings()))
          {
@@ -218,12 +329,49 @@ public class BulkInsertTransformStep extends AbstractTransformStep
          {
             okSummary.incrementCountAndAddPrimaryKey(null);
             outputRecords.add(record);
+
+            for(Map.Entry<String, List<QRecord>> entry : CollectionUtils.nonNullMap(record.getAssociatedRecords()).entrySet())
+            {
+               String             associationName         = entry.getKey();
+               ProcessSummaryLine associationToInsertLine = associationsToInsertSummaries.computeIfAbsent(associationName, x -> new ProcessSummaryLine(Status.OK));
+               associationToInsertLine.incrementCount(CollectionUtils.nonNullList(entry.getValue()).size());
+            }
          }
       }
 
       runBackendStepOutput.setRecords(outputRecords);
+      this.rowsProcessed += recordsInThisPage;
+   }
 
-      this.rowsProcessed += rowsInThisPage;
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void addToErrorToExampleRowValueMap(AbstractBulkLoadRollableValueError bulkLoadRollableValueError, QRecord record)
+   {
+      String         message   = bulkLoadRollableValueError.getMessageToUseAsProcessSummaryRollupKey();
+      List<RowValue> rowValues = errorToExampleRowValueMap.computeIfAbsent(message, k -> new ArrayList<>());
+
+      if(rowValues.size() < EXAMPLE_ROW_LIMIT)
+      {
+         rowValues.add(new RowValue(bulkLoadRollableValueError, record));
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void addToErrorToExampleRowMap(String message, QRecord record)
+   {
+      List<String> rowNos = errorToExampleRowsMap.computeIfAbsent(message, k -> new ArrayList<>());
+
+      if(rowNos.size() < EXAMPLE_ROW_LIMIT)
+      {
+         rowNos.add(BulkLoadRecordUtils.getRowNosString(record));
+      }
    }
 
 
@@ -231,7 +379,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
    /*******************************************************************************
     **
     *******************************************************************************/
-   private List<QRecord> getRecordsWithoutUniqueKeyErrors(RunBackendStepInput runBackendStepInput, Map<UniqueKey, Set<List<Serializable>>> existingKeys, List<UniqueKey> uniqueKeys, QTableMetaData table)
+   private List<QRecord> getRecordsWithoutUniqueKeyErrors(List<QRecord> records, Map<UniqueKey, Set<List<Serializable>>> existingKeys, List<UniqueKey> uniqueKeys, QTableMetaData table)
    {
       ////////////////////////////////////////////////////
       // if there are no UK's, proceed with all records //
@@ -239,7 +387,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       List<QRecord> recordsWithoutUkErrors = new ArrayList<>();
       if(existingKeys.isEmpty())
       {
-         recordsWithoutUkErrors.addAll(runBackendStepInput.getRecords());
+         recordsWithoutUkErrors.addAll(records);
       }
       else
       {
@@ -255,7 +403,7 @@ public class BulkInsertTransformStep extends AbstractTransformStep
          // else, get each records keys and see if it already exists or not       //
          // also, build a set of keys we've seen (within this page (or overall?)) //
          ///////////////////////////////////////////////////////////////////////////
-         for(QRecord record : runBackendStepInput.getRecords())
+         for(QRecord record : records)
          {
             if(CollectionUtils.nullSafeHasContents(record.getErrors()))
             {
@@ -326,6 +474,24 @@ public class BulkInsertTransformStep extends AbstractTransformStep
       okSummary.pickMessage(isForResultScreen);
       okSummary.addSelfToListIfAnyCount(rs);
 
+      for(Map.Entry<String, ProcessSummaryLine> entry : associationsToInsertSummaries.entrySet())
+      {
+         Optional<Association> association = table.getAssociations().stream().filter(a -> a.getName().equals(entry.getKey())).findFirst();
+         if(association.isPresent())
+         {
+            QTableMetaData associationTable = QContext.getQInstance().getTable(association.get().getAssociatedTableName());
+            String         associationLabel = associationTable.getLabel();
+
+            ProcessSummaryLine line = entry.getValue();
+            line.setSingularFutureMessage(associationLabel + " record will be inserted.");
+            line.setPluralFutureMessage(associationLabel + " records will be inserted.");
+            line.setSingularPastMessage(associationLabel + " record was inserted.");
+            line.setPluralPastMessage(associationLabel + " records were inserted.");
+            line.pickMessage(isForResultScreen);
+            line.addSelfToListIfAnyCount(rs);
+         }
+      }
+
       for(Map.Entry<UniqueKey, ProcessSummaryLineWithUKSampleValues> entry : ukErrorSummaries.entrySet())
       {
          UniqueKey                            uniqueKey      = entry.getKey();
@@ -333,8 +499,8 @@ public class BulkInsertTransformStep extends AbstractTransformStep
 
          ukErrorSummary
             .withMessageSuffix(" inserted, because of duplicate values in a unique key on the fields (" + uniqueKey.getDescription(table) + "), with values"
-               + (ukErrorSummary.areThereMoreSampleValues ? " such as: " : ": ")
-               + StringUtils.joinWithCommasAndAnd(new ArrayList<>(ukErrorSummary.sampleValues)))
+                               + (ukErrorSummary.areThereMoreSampleValues ? " such as: " : ": ")
+                               + StringUtils.joinWithCommasAndAnd(new ArrayList<>(ukErrorSummary.sampleValues)))
 
             .withSingularFutureMessage(" record will not be")
             .withPluralFutureMessage(" records will not be")
@@ -344,9 +510,61 @@ public class BulkInsertTransformStep extends AbstractTransformStep
          ukErrorSummary.addSelfToListIfAnyCount(rs);
       }
 
+      ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // for process summary lines that exist in the error-to-example-row-value map, add those example values to the lines. //
+      ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      for(Map.Entry<String, ProcessSummaryLine> entry : processSummaryWarningsAndErrorsRollup.getErrorSummaries().entrySet())
+      {
+         String message = entry.getKey();
+         if(errorToExampleRowValueMap.containsKey(message))
+         {
+            ProcessSummaryLine line = entry.getValue();
+            List<RowValue> rowValues = errorToExampleRowValueMap.get(message);
+            String exampleOrFull = rowValues.size() < line.getCount() ? "Example " : "";
+            line.setMessageSuffix(line.getMessageSuffix() + ".  " + exampleOrFull + "Values:");
+            line.setBulletsOfText(new ArrayList<>(rowValues.stream().map(String::valueOf).toList()));
+         }
+         else if(errorToExampleRowsMap.containsKey(message))
+         {
+            ProcessSummaryLine line = entry.getValue();
+            List<String> rowDescriptions = errorToExampleRowsMap.get(message);
+            String exampleOrFull = rowDescriptions.size() < line.getCount() ? "Example " : "";
+            line.setMessageSuffix(line.getMessageSuffix() + ".  " + exampleOrFull + "Records:");
+            line.setBulletsOfText(new ArrayList<>(rowDescriptions.stream().map(String::valueOf).toList()));
+         }
+      }
+
       processSummaryWarningsAndErrorsRollup.addToList(rs);
 
       return (rs);
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private record RowValue(String row, String value)
+   {
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public RowValue(AbstractBulkLoadRollableValueError bulkLoadRollableValueError, QRecord record)
+      {
+         this(BulkLoadRecordUtils.getRowNosString(record), ValueUtils.getValueAsString(bulkLoadRollableValueError.getValue()));
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public String toString()
+      {
+         return row + " [" + value + "]";
+      }
+   }
+
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/AbstractIteratorBasedFileToRows.java

@@ -0,0 +1,139 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.util.Iterator;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public abstract class AbstractIteratorBasedFileToRows<E> implements FileToRowsInterface
+{
+   private Iterator<E> iterator;
+
+   private boolean         useLast = false;
+   private BulkLoadFileRow last;
+
+   int rowNo = 0;
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public boolean hasNext()
+   {
+      if(iterator == null)
+      {
+         throw new IllegalStateException("Object was not init'ed");
+      }
+
+      if(useLast)
+      {
+         return true;
+      }
+
+      return iterator.hasNext();
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public BulkLoadFileRow next()
+   {
+      rowNo++;
+      if(iterator == null)
+      {
+         throw new IllegalStateException("Object was not init'ed");
+      }
+
+      if(useLast)
+      {
+         useLast = false;
+         return (this.last);
+      }
+
+      E e = iterator.next();
+
+      BulkLoadFileRow row = makeRow(e);
+
+      this.last = row;
+      return (this.last);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public abstract BulkLoadFileRow makeRow(E e);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void unNext()
+   {
+      rowNo--;
+      useLast = true;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for iterator
+    *******************************************************************************/
+   public Iterator<E> getIterator()
+   {
+      return (this.iterator);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for iterator
+    *******************************************************************************/
+   public void setIterator(Iterator<E> iterator)
+   {
+      this.iterator = iterator;
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for rowNo
+    **
+    *******************************************************************************/
+   @Override
+   public int getRowNo()
+   {
+      return rowNo;
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/CsvFileToRows.java

@@ -0,0 +1,112 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class CsvFileToRows extends AbstractIteratorBasedFileToRows<CSVRecord> implements FileToRowsInterface
+{
+   private CSVParser csvParser;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static CsvFileToRows forString(String csv) throws QException
+   {
+      CsvFileToRows csvFileToRows = new CsvFileToRows();
+
+      ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(csv.getBytes());
+      csvFileToRows.init(byteArrayInputStream);
+
+      return (csvFileToRows);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void init(InputStream inputStream) throws QException
+   {
+      try
+      {
+         csvParser = new CSVParser(new InputStreamReader(inputStream), CSVFormat.DEFAULT
+            .withIgnoreSurroundingSpaces()
+         );
+         setIterator(csvParser.iterator());
+      }
+      catch(IOException e)
+      {
+         throw new QException("Error opening CSV Parser", e);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public BulkLoadFileRow makeRow(CSVRecord csvRecord)
+   {
+      Serializable[] values = new Serializable[csvRecord.size()];
+      int            i      = 0;
+      for(String s : csvRecord)
+      {
+         values[i++] = s;
+      }
+
+      return (new BulkLoadFileRow(values, getRowNo()));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void close() throws Exception
+   {
+      if(csvParser != null)
+      {
+         csvParser.close();
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/FileToRowsInterface.java

@@ -0,0 +1,80 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Locale;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public interface FileToRowsInterface extends AutoCloseable, Iterator<BulkLoadFileRow>
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   static FileToRowsInterface forFile(String fileName, InputStream inputStream) throws QException
+   {
+      FileToRowsInterface rs;
+      if(fileName.toLowerCase(Locale.ROOT).endsWith(".csv"))
+      {
+         rs = new CsvFileToRows();
+      }
+      else if(fileName.toLowerCase(Locale.ROOT).endsWith(".xlsx"))
+      {
+         rs = new XlsxFileToRows();
+      }
+      else
+      {
+         throw (new QUserFacingException("Unrecognized file extension - expecting .csv or .xlsx"));
+      }
+
+      rs.init(inputStream);
+      return rs;
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   void init(InputStream inputStream) throws QException;
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   int getRowNo();
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   void unNext();
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/XlsxFileToRows.java

@@ -0,0 +1,141 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.MathContext;
+import java.time.LocalDateTime;
+import java.util.Optional;
+import java.util.stream.Stream;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import org.dhatim.fastexcel.reader.Cell;
+import org.dhatim.fastexcel.reader.ReadableWorkbook;
+import org.dhatim.fastexcel.reader.Sheet;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class XlsxFileToRows extends AbstractIteratorBasedFileToRows<org.dhatim.fastexcel.reader.Row> implements FileToRowsInterface
+{
+   private ReadableWorkbook                        workbook;
+   private Stream<org.dhatim.fastexcel.reader.Row> rows;
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void init(InputStream inputStream) throws QException
+   {
+      try
+      {
+         workbook = new ReadableWorkbook(inputStream);
+         Sheet sheet = workbook.getFirstSheet();
+
+         rows = sheet.openStream();
+         setIterator(rows.iterator());
+      }
+      catch(IOException e)
+      {
+         throw new QException("Error opening XLSX Parser", e);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public BulkLoadFileRow makeRow(org.dhatim.fastexcel.reader.Row readerRow)
+   {
+      Serializable[] values = new Serializable[readerRow.getCellCount()];
+
+      for(int i = 0; i < readerRow.getCellCount(); i++)
+      {
+         Cell cell = readerRow.getCell(i);
+         if(cell.getType() != null)
+         {
+            values[i] = switch(cell.getType())
+            {
+               case NUMBER ->
+               {
+                  ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  // ... with fastexcel reader, we don't get styles... so, we just know type = number, for dates and ints & decimals... //
+                  ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+                  Optional<LocalDateTime> dateTime = readerRow.getCellAsDate(i);
+                  if(dateTime.isPresent() && dateTime.get().getYear() > 1915 && dateTime.get().getYear() < 2100)
+                  {
+                     yield dateTime.get();
+                  }
+
+                  Optional<BigDecimal> optionalBigDecimal = readerRow.getCellAsNumber(i);
+                  if(optionalBigDecimal.isPresent())
+                  {
+                     BigDecimal bigDecimal = optionalBigDecimal.get();
+                     if(bigDecimal.subtract(bigDecimal.round(new MathContext(0))).compareTo(BigDecimal.ZERO) == 0)
+                     {
+                        yield bigDecimal.intValue();
+                     }
+
+                     yield bigDecimal;
+                  }
+
+                  yield (null);
+               }
+               case BOOLEAN -> readerRow.getCellAsBoolean(i).orElse(null);
+               case STRING, FORMULA -> cell.getText();
+               case EMPTY, ERROR -> null;
+            };
+         }
+      }
+
+      return new BulkLoadFileRow(values, getRowNo());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void close() throws Exception
+   {
+      if(workbook != null)
+      {
+         workbook.close();
+      }
+
+      if(rows != null)
+      {
+         rows.close();
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/AbstractBulkLoadRollableValueError.java

@@ -0,0 +1,55 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public abstract class AbstractBulkLoadRollableValueError extends BadInputStatusMessage
+{
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public AbstractBulkLoadRollableValueError(String message)
+   {
+      super(message);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public abstract String getMessageToUseAsProcessSummaryRollupKey();
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public abstract Serializable getValue();
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadMappingSuggester.java

@@ -0,0 +1,232 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadTableStructure;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.collections.ListBuilder;
+
+
+/*******************************************************************************
+ ** Given a bulk-upload, create a suggested mapping
+ *******************************************************************************/
+public class BulkLoadMappingSuggester
+{
+   private Map<String, Integer> massagedHeadersWithoutNumbersToIndexMap;
+   private Map<String, Integer> massagedHeadersWithNumbersToIndexMap;
+
+   private String layout = "FLAT";
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public BulkLoadProfile suggestBulkLoadMappingProfile(BulkLoadTableStructure tableStructure, List<String> headerRow)
+   {
+      massagedHeadersWithoutNumbersToIndexMap = new LinkedHashMap<>();
+      for(int i = 0; i < headerRow.size(); i++)
+      {
+         String headerValue = massageHeader(headerRow.get(i), true);
+
+         if(!massagedHeadersWithoutNumbersToIndexMap.containsKey(headerValue))
+         {
+            massagedHeadersWithoutNumbersToIndexMap.put(headerValue, i);
+         }
+      }
+
+      massagedHeadersWithNumbersToIndexMap = new LinkedHashMap<>();
+      for(int i = 0; i < headerRow.size(); i++)
+      {
+         String headerValue = massageHeader(headerRow.get(i), false);
+
+         if(!massagedHeadersWithNumbersToIndexMap.containsKey(headerValue))
+         {
+            massagedHeadersWithNumbersToIndexMap.put(headerValue, i);
+         }
+      }
+
+      ArrayList<BulkLoadProfileField> fieldList = new ArrayList<>();
+      processTable(tableStructure, fieldList, headerRow);
+
+      /////////////////////////////////////////////////
+      // sort the fields to match the column indexes //
+      /////////////////////////////////////////////////
+      fieldList.sort(Comparator.comparing(blpf -> blpf.getColumnIndex()));
+
+      BulkLoadProfile bulkLoadProfile = new BulkLoadProfile()
+         .withVersion("v1")
+         .withLayout(layout)
+         .withHasHeaderRow(true)
+         .withFieldList(fieldList);
+
+      return (bulkLoadProfile);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void processTable(BulkLoadTableStructure tableStructure, ArrayList<BulkLoadProfileField> fieldList, List<String> headerRow)
+   {
+      Map<String, Integer> rs = new HashMap<>();
+      for(QFieldMetaData field : tableStructure.getFields())
+      {
+         String fieldName           = massageHeader(field.getName(), false);
+         String fieldLabel          = massageHeader(field.getLabel(), false);
+         String tablePlusFieldLabel = massageHeader(QContext.getQInstance().getTable(tableStructure.getTableName()).getLabel() + ": " + field.getLabel(), false);
+         String fullFieldName       = (StringUtils.hasContent(tableStructure.getAssociationPath()) ? (tableStructure.getAssociationPath() + ".") : "") + field.getName();
+
+         ////////////////////////////////////////////////////////////////////////////////////
+         // consider, if this is a many-table, if there are many matches, for wide mode... //
+         ////////////////////////////////////////////////////////////////////////////////////
+         if(tableStructure.getIsMany())
+         {
+            List<Integer> matchingIndexes = new ArrayList<>();
+
+            for(Map.Entry<String, Integer> entry : massagedHeadersWithNumbersToIndexMap.entrySet())
+            {
+               String header = entry.getKey();
+               if(header.matches(fieldName + "\\d*$") || header.matches(fieldLabel + "\\d*$"))
+               {
+                  matchingIndexes.add(entry.getValue());
+               }
+            }
+
+            if(CollectionUtils.nullSafeHasContents(matchingIndexes))
+            {
+               ///////////////////////////////////////////////////////////////////////////////////////////////////////
+               // if we found more than 1 match - consider this a likely wide file, and build fields as wide-fields //
+               // else, if only 1, allow us to go down into the TALL block below                                    //
+               // note - should we do a merger at the end, in case we found some wide, some tall?                   //
+               ///////////////////////////////////////////////////////////////////////////////////////////////////////
+               if(matchingIndexes.size() > 1)
+               {
+                  layout = "WIDE";
+
+                  int i = 0;
+                  for(Integer index : matchingIndexes)
+                  {
+                     fieldList.add(new BulkLoadProfileField()
+                        .withFieldName(fullFieldName + "," + i)
+                        .withHeaderName(headerRow.get(index))
+                        .withColumnIndex(index)
+                     );
+
+                     i++;
+                  }
+
+                  continue;
+               }
+            }
+         }
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // else - look for matches, first w/ headers with numbers, then headers w/o numbers checking labels and names //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         Integer index = null;
+
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // for each of these potential identities of the field:                                                                   //
+         // 1) its label, massaged                                                                                                 //
+         // 2) its name, massaged                                                                                                  //
+         // 3) its label, massaged, with numbers stripped away                                                                     //
+         // 4) its name, massaged, with numbers stripped away                                                                      //
+         // check if that identity is in the massagedHeadersWithNumbersToIndexMap, or the massagedHeadersWithoutNumbersToIndexMap. //
+         // this is currently successful in the both versions of the address 1 / address 2 <=> address / address 2 use-case        //
+         // that is, BulkLoadMappingSuggesterTest.testChallengingAddress1And2                                                      //
+         ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         for(String fieldIdentity : ListBuilder.of(fieldLabel, fieldName, tablePlusFieldLabel, massageHeader(fieldLabel, true), massageHeader(fieldName, true)))
+         {
+            if(massagedHeadersWithNumbersToIndexMap.containsKey(fieldIdentity))
+            {
+               index = massagedHeadersWithNumbersToIndexMap.get(fieldIdentity);
+            }
+            else if(massagedHeadersWithoutNumbersToIndexMap.containsKey(fieldIdentity))
+            {
+               index = massagedHeadersWithoutNumbersToIndexMap.get(fieldIdentity);
+            }
+
+            if(index != null)
+            {
+               break;
+            }
+         }
+
+         if(index != null)
+         {
+            fieldList.add(new BulkLoadProfileField()
+               .withFieldName(fullFieldName)
+               .withHeaderName(headerRow.get(index))
+               .withColumnIndex(index)
+            );
+
+            if(tableStructure.getIsMany() && layout.equals("FLAT"))
+            {
+               //////////////////////////////////////////////////////////////////////////////////////////
+               // the first time we find an is-many child, if we were still marked as flat, go to tall //
+               //////////////////////////////////////////////////////////////////////////////////////////
+               layout = "TALL";
+            }
+         }
+      }
+
+      ////////////////////////////////////////////
+      // recursively process child associations //
+      ////////////////////////////////////////////
+      for(BulkLoadTableStructure associationTableStructure : CollectionUtils.nonNullList(tableStructure.getAssociations()))
+      {
+         processTable(associationTableStructure, fieldList, headerRow);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private String massageHeader(String header, boolean stripNumbers)
+   {
+      if(header == null)
+      {
+         return (null);
+      }
+
+      String massagedWithNumbers = header.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]", "");
+      return stripNumbers ? massagedWithNumbers.replaceAll("[0-9]", "") : massagedWithNumbers;
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadPossibleValueError.java

@@ -0,0 +1,72 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+
+
+/*******************************************************************************
+ ** Specialized error for records, for bulk-load use-cases, where we want to
+ ** report back info to the user about the field & value.
+ *******************************************************************************/
+public class BulkLoadPossibleValueError extends AbstractBulkLoadRollableValueError
+{
+   private final String       fieldLabel;
+   private final Serializable value;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public BulkLoadPossibleValueError(String fieldName, Serializable value, String fieldLabel)
+   {
+      super("Value [" + value + "] for field [" + fieldLabel + "] is not a valid option");
+      this.value = value;
+      this.fieldLabel = fieldLabel;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getMessageToUseAsProcessSummaryRollupKey()
+   {
+      return ("Unrecognized value for field [" + fieldLabel + "]");
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for value
+    **
+    *******************************************************************************/
+   @Override
+   public Serializable getValue()
+   {
+      return value;
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadRecordUtils.java

@@ -0,0 +1,112 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+
+
+/*******************************************************************************
+ ** Utility methods for working with records in a bulk-load.
+ **
+ ** Originally added for working with backendDetails around the source rows.
+ *******************************************************************************/
+public class BulkLoadRecordUtils
+{
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public static QRecord addBackendDetailsAboutFileRows(QRecord record, BulkLoadFileRow fileRow)
+   {
+      return (addBackendDetailsAboutFileRows(record, new ArrayList<>(List.of(fileRow))));
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public static QRecord addBackendDetailsAboutFileRows(QRecord record, ArrayList<BulkLoadFileRow> fileRows)
+   {
+      if(CollectionUtils.nullSafeHasContents(fileRows))
+      {
+         Integer firstRowNo = fileRows.get(0).getRowNo();
+         Integer lastRowNo  = fileRows.get(fileRows.size() - 1).getRowNo();
+
+         if(Objects.equals(firstRowNo, lastRowNo))
+         {
+            record.addBackendDetail("rowNos", "Row " + firstRowNo);
+         }
+         else
+         {
+            record.addBackendDetail("rowNos", "Rows " + firstRowNo + "-" + lastRowNo);
+         }
+      }
+      else
+      {
+         record.addBackendDetail("rowNos", "Rows ?");
+      }
+
+      record.addBackendDetail("fileRows", fileRows);
+      return (record);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static String getRowNosString(QRecord record)
+   {
+      return (record.getBackendDetailString("rowNos"));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @SuppressWarnings("unchecked")
+   public static ArrayList<BulkLoadFileRow> getFileRows(QRecord record)
+   {
+      return (ArrayList<BulkLoadFileRow>) record.getBackendDetail("fileRows");
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static List<Integer> getFileRowNos(QRecord record)
+   {
+      return (getFileRows(record).stream().map(row -> row.getRowNo()).toList());
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadTableStructureBuilder.java

@@ -0,0 +1,132 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadTableStructure;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ObjectUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ ** utility to build BulkLoadTableStructure objects for a QQQ Table.
+ *******************************************************************************/
+public class BulkLoadTableStructureBuilder
+{
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static BulkLoadTableStructure buildTableStructure(String tableName)
+   {
+      return (buildTableStructure(tableName, null, null));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static BulkLoadTableStructure buildTableStructure(String tableName, Association association, String parentAssociationPath)
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(tableName);
+
+      BulkLoadTableStructure tableStructure = new BulkLoadTableStructure();
+      tableStructure.setTableName(tableName);
+      tableStructure.setLabel(table.getLabel());
+
+      Set<String> associationJoinFieldNamesToExclude = new HashSet<>();
+
+      if(association == null)
+      {
+         tableStructure.setIsMain(true);
+         tableStructure.setIsMany(false);
+         tableStructure.setAssociationPath(null);
+      }
+      else
+      {
+         tableStructure.setIsMain(false);
+
+         QJoinMetaData join = QContext.getQInstance().getJoin(association.getJoinName());
+         if(join.getType().equals(JoinType.ONE_TO_MANY) || join.getType().equals(JoinType.MANY_TO_ONE))
+         {
+            tableStructure.setIsMany(true);
+         }
+
+         for(JoinOn joinOn : join.getJoinOns())
+         {
+            ////////////////////////////////////////////////////////////////////////////////////////////////
+            // don't allow the user to map the "join field" from a child up to its parent                 //
+            // (e.g., you can't map lineItem.orderId -- that'll happen automatically via the association) //
+            ////////////////////////////////////////////////////////////////////////////////////////////////
+            if(join.getLeftTable().equals(tableName))
+            {
+               associationJoinFieldNamesToExclude.add(joinOn.getLeftField());
+            }
+            else if(join.getRightTable().equals(tableName))
+            {
+               associationJoinFieldNamesToExclude.add(joinOn.getRightField());
+            }
+         }
+
+         if(!StringUtils.hasContent(parentAssociationPath))
+         {
+            tableStructure.setAssociationPath(association.getName());
+         }
+         else
+         {
+            tableStructure.setAssociationPath(parentAssociationPath + "." + association.getName());
+         }
+      }
+
+      ArrayList<QFieldMetaData> fields = new ArrayList<>();
+      tableStructure.setFields(fields);
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         if(field.getIsEditable() && !associationJoinFieldNamesToExclude.contains(field.getName()))
+         {
+            fields.add(field);
+         }
+      }
+
+      fields.sort(Comparator.comparing(f -> ObjectUtils.requireNonNullElse(f.getLabel(), f.getName(), "")));
+
+      for(Association childAssociation : CollectionUtils.nonNullList(table.getAssociations()))
+      {
+         BulkLoadTableStructure associatedStructure = buildTableStructure(childAssociation.getAssociatedTableName(), childAssociation, parentAssociationPath);
+         tableStructure.addAssociation(associatedStructure);
+      }
+
+      return (tableStructure);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadValueMapper.java

@@ -0,0 +1,238 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.actions.values.SearchPossibleValueSourceAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.ListingHash;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkLoadValueMapper
+{
+   private static final QLogger LOG = QLogger.getLogger(BulkLoadValueMapper.class);
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static void valueMapping(List<QRecord> records, BulkInsertMapping mapping, QTableMetaData table) throws QException
+   {
+      valueMapping(records, mapping, table, null);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void valueMapping(List<QRecord> records, BulkInsertMapping mapping, QTableMetaData table, String associationNameChain) throws QException
+   {
+      if(CollectionUtils.nullSafeIsEmpty(records))
+      {
+         return;
+      }
+
+      String associationNamePrefixForFields = StringUtils.hasContent(associationNameChain) ? associationNameChain + "." : "";
+      String tableLabelPrefix               = StringUtils.hasContent(associationNameChain) ? table.getLabel() + ": " : "";
+
+      Map<String, ListingHash<String, QRecord>> possibleValueToRecordMap = new HashMap<>();
+
+      Map<String, Map<String, Serializable>> mappingForTable = mapping.getFieldNameToValueMappingForTable(associationNameChain);
+      for(QRecord record : records)
+      {
+         for(Map.Entry<String, Serializable> valueEntry : record.getValues().entrySet())
+         {
+            QFieldMetaData field = table.getField(valueEntry.getKey());
+            Serializable   value = valueEntry.getValue();
+
+            ///////////////////
+            // value mappin' //
+            ///////////////////
+            if(mappingForTable.containsKey(field.getName()) && value != null)
+            {
+               Serializable mappedValue = mappingForTable.get(field.getName()).get(ValueUtils.getValueAsString(value));
+               if(mappedValue != null)
+               {
+                  value = mappedValue;
+               }
+            }
+
+            /////////////////////
+            // type convertin' //
+            /////////////////////
+            if(value != null && !"".equals(value))
+            {
+               if(StringUtils.hasContent(field.getPossibleValueSourceName()))
+               {
+                  ListingHash<String, QRecord> fieldPossibleValueToRecordMap = possibleValueToRecordMap.computeIfAbsent(field.getName(), k -> new ListingHash<>());
+                  fieldPossibleValueToRecordMap.add(ValueUtils.getValueAsString(value), record);
+               }
+               else
+               {
+                  QFieldType type = field.getType();
+                  try
+                  {
+                     value = ValueUtils.getValueAsFieldType(type, value);
+                  }
+                  catch(Exception e)
+                  {
+                     record.addError(new BulkLoadValueTypeError(associationNamePrefixForFields + field.getName(), value, type, tableLabelPrefix + field.getLabel()));
+                  }
+               }
+            }
+
+            record.setValue(field.getName(), value);
+         }
+
+         //////////////////////////////////////
+         // recursively process associations //
+         //////////////////////////////////////
+         for(Map.Entry<String, List<QRecord>> entry : record.getAssociatedRecords().entrySet())
+         {
+            String                associationName = entry.getKey();
+            Optional<Association> association     = table.getAssociations().stream().filter(a -> a.getName().equals(associationName)).findFirst();
+            if(association.isPresent())
+            {
+               QTableMetaData associatedTable = QContext.getQInstance().getTable(association.get().getAssociatedTableName());
+               valueMapping(entry.getValue(), mapping, associatedTable, StringUtils.hasContent(associationNameChain) ? associationNameChain + "." + associationName : associationName);
+            }
+            else
+            {
+               throw new QException("Missing association [" + associationName + "] on table [" + table.getName() + "]");
+            }
+         }
+      }
+
+      //////////////////////////////////////////
+      // look up and validate possible values //
+      //////////////////////////////////////////
+      for(Map.Entry<String, ListingHash<String, QRecord>> entry : possibleValueToRecordMap.entrySet())
+      {
+         String                       fieldName                     = entry.getKey();
+         QFieldMetaData               field                         = table.getField(fieldName);
+         ListingHash<String, QRecord> fieldPossibleValueToRecordMap = possibleValueToRecordMap.get(fieldName);
+
+         handlePossibleValues(field, fieldPossibleValueToRecordMap, associationNamePrefixForFields, tableLabelPrefix);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static void handlePossibleValues(QFieldMetaData field, ListingHash<String, QRecord> fieldPossibleValueToRecordMap, String associationNamePrefixForFields, String tableLabelPrefix) throws QException
+   {
+      Set<String>                    values         = fieldPossibleValueToRecordMap.keySet();
+      Map<String, QPossibleValue<?>> valuesFound    = new HashMap<>();
+      Set<String>                    valuesNotFound = new HashSet<>(values);
+
+      ////////////////////////////////////////////////////////
+      // do a search, trying to use all given values as ids //
+      ////////////////////////////////////////////////////////
+      SearchPossibleValueSourceInput searchPossibleValueSourceInput = new SearchPossibleValueSourceInput();
+      searchPossibleValueSourceInput.setPossibleValueSourceName(field.getPossibleValueSourceName());
+      ArrayList<Serializable> idList = new ArrayList<>(values);
+      searchPossibleValueSourceInput.setIdList(idList);
+      searchPossibleValueSourceInput.setLimit(values.size());
+      LOG.debug("Searching possible value source by ids during bulk load mapping", logPair("pvsName", field.getPossibleValueSourceName()), logPair("noOfIds", idList.size()), logPair("firstId", () -> idList.get(0)));
+      SearchPossibleValueSourceOutput searchPossibleValueSourceOutput = new SearchPossibleValueSourceAction().execute(searchPossibleValueSourceInput);
+
+      ////////////////////////////////////////////////////////////////////////////////////////////////////
+      // for each possible value found, remove it from the set of ones not-found, and store it as a hit //
+      ////////////////////////////////////////////////////////////////////////////////////////////////////
+      for(QPossibleValue<?> possibleValue : searchPossibleValueSourceOutput.getResults())
+      {
+         String valueAsString = ValueUtils.getValueAsString(possibleValue.getId());
+         valuesFound.put(valueAsString, possibleValue);
+         valuesNotFound.remove(valueAsString);
+      }
+
+      ///////////////////////////////////////////////////////////////////////////
+      // if there are any that weren't found, try to look them up now by label //
+      ///////////////////////////////////////////////////////////////////////////
+      if(!valuesNotFound.isEmpty())
+      {
+         searchPossibleValueSourceInput = new SearchPossibleValueSourceInput();
+         searchPossibleValueSourceInput.setPossibleValueSourceName(field.getPossibleValueSourceName());
+         ArrayList<String> labelList = new ArrayList<>(valuesNotFound);
+         searchPossibleValueSourceInput.setLabelList(labelList);
+         searchPossibleValueSourceInput.setLimit(valuesNotFound.size());
+
+         LOG.debug("Searching possible value source by labels during bulk load mapping", logPair("pvsName", field.getPossibleValueSourceName()), logPair("noOfLabels", labelList.size()), logPair("firstLabel", () -> labelList.get(0)));
+         searchPossibleValueSourceOutput = new SearchPossibleValueSourceAction().execute(searchPossibleValueSourceInput);
+         for(QPossibleValue<?> possibleValue : searchPossibleValueSourceOutput.getResults())
+         {
+            valuesFound.put(possibleValue.getLabel(), possibleValue);
+            valuesNotFound.remove(possibleValue.getLabel());
+         }
+      }
+
+      ////////////////////////////////////////////////////////////////////////////////
+      // for each record now, either set a usable value (e.g., a PV.id) or an error //
+      ////////////////////////////////////////////////////////////////////////////////
+      for(Map.Entry<String, List<QRecord>> entry : fieldPossibleValueToRecordMap.entrySet())
+      {
+         String value = entry.getKey();
+         for(QRecord record : entry.getValue())
+         {
+            if(valuesFound.containsKey(value))
+            {
+               record.setValue(field.getName(), valuesFound.get(value).getId());
+            }
+            else
+            {
+               record.addError(new BulkLoadPossibleValueError(associationNamePrefixForFields + field.getName(), value, tableLabelPrefix + field.getLabel()));
+            }
+         }
+      }
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadValueTypeError.java

@@ -0,0 +1,75 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+
+
+/*******************************************************************************
+ ** Specialized error for records, for bulk-load use-cases, where we want to
+ ** report back info to the user about the field & value.
+ *******************************************************************************/
+public class BulkLoadValueTypeError extends AbstractBulkLoadRollableValueError
+{
+   private final String       fieldLabel;
+   private final Serializable value;
+   private final QFieldType   type;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public BulkLoadValueTypeError(String fieldName, Serializable value, QFieldType type, String fieldLabel)
+   {
+      super("Value [" + value + "] for field [" + fieldLabel + "] could not be converted to type [" + type + "]");
+      this.value = value;
+      this.type = type;
+      this.fieldLabel = fieldLabel;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String getMessageToUseAsProcessSummaryRollupKey()
+   {
+      return ("Cannot convert value for field [" + fieldLabel + "] to type [" + type.getMixedCaseLabel() + "]");
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for value
+    **
+    *******************************************************************************/
+   @Override
+   public Serializable getValue()
+   {
+      return value;
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/FlatRowsToRecord.java

@@ -0,0 +1,77 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class FlatRowsToRecord implements RowsToRecordInterface
+{
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public List<QRecord> nextPage(FileToRowsInterface fileToRowsInterface, BulkLoadFileRow headerRow, BulkInsertMapping mapping, Integer limit) throws QException
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(mapping.getTableName());
+      if(table == null)
+      {
+         throw (new QException("Table [" + mapping.getTableName() + "] was not found in the Instance"));
+      }
+
+      List<QRecord> rs = new ArrayList<>();
+
+      Map<String, Integer> fieldIndexes = mapping.getFieldIndexes(table, null, headerRow);
+
+      while(fileToRowsInterface.hasNext() && rs.size() < limit)
+      {
+         BulkLoadFileRow row    = fileToRowsInterface.next();
+         QRecord         record = new QRecord();
+         BulkLoadRecordUtils.addBackendDetailsAboutFileRows(record, row);
+
+         for(QFieldMetaData field : table.getFields().values())
+         {
+            setValueOrDefault(record, field, null, mapping, row, fieldIndexes.get(field.getName()));
+         }
+
+         rs.add(record);
+      }
+
+      BulkLoadValueMapper.valueMapping(rs, mapping, table);
+
+      return (rs);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/RowsToRecordInterface.java

@@ -0,0 +1,103 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public interface RowsToRecordInterface
+{
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   List<QRecord> nextPage(FileToRowsInterface fileToRowsInterface, BulkLoadFileRow headerRow, BulkInsertMapping mapping, Integer limit) throws QException;
+
+
+   /***************************************************************************
+    ** returns true if value from row was used, else false.
+    ***************************************************************************/
+   default boolean setValueOrDefault(QRecord record, QFieldMetaData field, String associationNameChain, BulkInsertMapping mapping, BulkLoadFileRow row, Integer columnIndex)
+   {
+      return setValueOrDefault(record, field, associationNameChain, mapping, row, columnIndex, null);
+   }
+
+   /***************************************************************************
+    ** returns true if value from row was used, else false.
+    ***************************************************************************/
+   default boolean setValueOrDefault(QRecord record, QFieldMetaData field, String associationNameChain, BulkInsertMapping mapping, BulkLoadFileRow row, Integer columnIndex, List<Integer> wideAssociationIndexes)
+   {
+      boolean valueFromRowWasUsed = false;
+
+      /////////////////////////////////////////////////////////////////////////////////////////////////
+      // build full field-name -- possibly associations, then field name, then possibly index-suffix //
+      /////////////////////////////////////////////////////////////////////////////////////////////////
+      String fieldName = field.getName();
+      String fieldNameWithAssociationPrefix = StringUtils.hasContent(associationNameChain) ? associationNameChain + "." + fieldName : fieldName;
+
+      String wideAssociationSuffix = "";
+      if(CollectionUtils.nullSafeHasContents(wideAssociationIndexes))
+      {
+         wideAssociationSuffix = "," + StringUtils.join(".", wideAssociationIndexes);
+      }
+
+      String fullFieldName = fieldNameWithAssociationPrefix + wideAssociationSuffix;
+
+      //////////////////////////////////////////////
+      // ok - look in the row - then the defaults //
+      //////////////////////////////////////////////
+      Serializable value = null;
+      if(columnIndex != null && row != null)
+      {
+         value = row.getValueElseNull(columnIndex);
+         if(value != null && !"".equals(value))
+         {
+            valueFromRowWasUsed = true;
+         }
+      }
+      else if(mapping.getFieldNameToDefaultValueMap().containsKey(fullFieldName))
+      {
+         value = mapping.getFieldNameToDefaultValueMap().get(fullFieldName);
+      }
+
+      if(value != null && !"".equals(value))
+      {
+         record.setValue(fieldName, value);
+      }
+
+      return (valueFromRowWasUsed);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/TallRowsToRecord.java

@@ -0,0 +1,341 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import com.google.gson.reflect.TypeToken;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.Pair;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class TallRowsToRecord implements RowsToRecordInterface
+{
+   private Memoization<Pair<String, String>, Boolean> shouldProcesssAssociationMemoization = new Memoization<>();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public List<QRecord> nextPage(FileToRowsInterface fileToRowsInterface, BulkLoadFileRow headerRow, BulkInsertMapping mapping, Integer limit) throws QException
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(mapping.getTableName());
+      if(table == null)
+      {
+         throw (new QException("Table [" + mapping.getTableName() + "] was not found in the Instance"));
+      }
+
+      List<QRecord> rs = new ArrayList<>();
+
+      ArrayList<BulkLoadFileRow> rowsForCurrentRecord = new ArrayList<>();
+      List<Serializable>         recordGroupByValues  = null;
+
+      String associationNameChain = "";
+
+      while(fileToRowsInterface.hasNext() && rs.size() < limit)
+      {
+         BulkLoadFileRow row = fileToRowsInterface.next();
+
+         List<Integer> groupByIndexes = mapping.getTallLayoutGroupByIndexMap().get(table.getName());
+         if(CollectionUtils.nullSafeIsEmpty(groupByIndexes))
+         {
+            groupByIndexes = groupByAllIndexesFromTable(mapping, table, headerRow, null);
+         }
+
+         ////////////////////////
+         // this is suspect... //
+         ////////////////////////
+         List<Serializable> rowGroupByValues = getGroupByValues(row, groupByIndexes);
+         if(rowGroupByValues == null)
+         {
+            continue;
+         }
+
+         if(rowsForCurrentRecord.isEmpty())
+         {
+            ///////////////////////////////////
+            // this is first - so it's a yes //
+            ///////////////////////////////////
+            recordGroupByValues = rowGroupByValues;
+            rowsForCurrentRecord.add(row);
+         }
+         else if(Objects.equals(recordGroupByValues, rowGroupByValues))
+         {
+            /////////////////////////////
+            // a match - so keep going //
+            /////////////////////////////
+            rowsForCurrentRecord.add(row);
+         }
+         else
+         {
+            //////////////////////////////////////////////////////////////
+            // not first, and not a match, so we can finish this record //
+            //////////////////////////////////////////////////////////////
+            QRecord record = makeRecordFromRows(table, associationNameChain, mapping, headerRow, rowsForCurrentRecord);
+            rs.add(record);
+
+            //////////////////////////////////////////////////////////////////////////////////////////////////////
+            // we need to push this row back onto the fileToRows object, so it'll be handled in the next record //
+            //////////////////////////////////////////////////////////////////////////////////////////////////////
+            fileToRowsInterface.unNext();
+
+            ////////////////////////////////////////
+            // reset these record-specific values //
+            ////////////////////////////////////////
+            rowsForCurrentRecord = new ArrayList<>();
+            recordGroupByValues = null;
+         }
+      }
+
+      /////////////////////////////////////////////////////////////////////////////////////////////////
+      // i wrote this condition in here: && rs.size() < limit                                        //
+      // but IJ is saying it's always true... I can't quite see it, but, trusting static analysis... //
+      /////////////////////////////////////////////////////////////////////////////////////////////////
+      if(!rowsForCurrentRecord.isEmpty())
+      {
+         QRecord record = makeRecordFromRows(table, associationNameChain, mapping, headerRow, rowsForCurrentRecord);
+         rs.add(record);
+      }
+
+      BulkLoadValueMapper.valueMapping(rs, mapping, table);
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<Integer> groupByAllIndexesFromTable(BulkInsertMapping mapping, QTableMetaData table, BulkLoadFileRow headerRow, String name) throws QException
+   {
+      Map<String, Integer> fieldIndexes = mapping.getFieldIndexes(table, name, headerRow);
+      return new ArrayList<>(fieldIndexes.values());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private QRecord makeRecordFromRows(QTableMetaData table, String associationNameChain, BulkInsertMapping mapping, BulkLoadFileRow headerRow, List<BulkLoadFileRow> rows) throws QException
+   {
+      QRecord record = new QRecord();
+      BulkLoadRecordUtils.addBackendDetailsAboutFileRows(record, CollectionUtils.useOrWrap(rows, new TypeToken<ArrayList<BulkLoadFileRow>>() {}));
+
+      Map<String, Integer> fieldIndexes = mapping.getFieldIndexes(table, associationNameChain, headerRow);
+
+      //////////////////////////////////////////////////////
+      // get all rows for the main table from the 0th row //
+      //////////////////////////////////////////////////////
+      BulkLoadFileRow row = rows.get(0);
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         setValueOrDefault(record, field, associationNameChain, mapping, row, fieldIndexes.get(field.getName()));
+      }
+
+      /////////////////////////////
+      // associations (children) //
+      /////////////////////////////
+      for(String associationName : CollectionUtils.nonNullList(mapping.getMappedAssociations()))
+      {
+         boolean processAssociation = shouldProcessAssociation(associationNameChain, associationName);
+
+         if(processAssociation)
+         {
+            String associationNameMinusChain = StringUtils.hasContent(associationNameChain)
+               ? associationName.substring(associationNameChain.length() + 1)
+               : associationName;
+
+            Optional<Association> association = table.getAssociations().stream().filter(a -> a.getName().equals(associationNameMinusChain)).findFirst();
+            if(association.isEmpty())
+            {
+               throw (new QException("Couldn't find association: " + associationNameMinusChain + " under table: " + table.getName()));
+            }
+
+            QTableMetaData associatedTable = QContext.getQInstance().getTable(association.get().getAssociatedTableName());
+
+            List<QRecord> associatedRecords = processAssociation(associationNameMinusChain, associationNameChain, associatedTable, mapping, headerRow, rows);
+            record.withAssociatedRecords(associationNameMinusChain, associatedRecords);
+         }
+      }
+
+      return record;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   boolean shouldProcessAssociation(String associationNameChain, String associationName)
+   {
+      return shouldProcesssAssociationMemoization.getResult(Pair.of(associationNameChain, associationName), p ->
+      {
+         List<String> chainParts = new ArrayList<>();
+         List<String> nameParts  = new ArrayList<>();
+
+         if(StringUtils.hasContent(associationNameChain))
+         {
+            chainParts.addAll(Arrays.asList(associationNameChain.split("\\.")));
+         }
+
+         if(StringUtils.hasContent(associationName))
+         {
+            nameParts.addAll(Arrays.asList(associationName.split("\\.")));
+         }
+
+         if(!nameParts.isEmpty())
+         {
+            nameParts.remove(nameParts.size() - 1);
+         }
+
+         return (chainParts.equals(nameParts));
+      }).orElse(false);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<QRecord> processAssociation(String associationName, String associationNameChain, QTableMetaData associatedTable, BulkInsertMapping mapping, BulkLoadFileRow headerRow, List<BulkLoadFileRow> rows) throws QException
+   {
+      List<QRecord> rs = new ArrayList<>();
+
+      QTableMetaData table                                 = QContext.getQInstance().getTable(associatedTable.getName());
+      String         associationNameChainForRecursiveCalls = "".equals(associationNameChain) ? associationName : associationNameChain + "." + associationName;
+
+      List<BulkLoadFileRow> rowsForCurrentRecord = new ArrayList<>();
+      List<Serializable>    recordGroupByValues  = null;
+      for(BulkLoadFileRow row : rows)
+      {
+         List<Integer> groupByIndexes = mapping.getTallLayoutGroupByIndexMap().get(associationNameChainForRecursiveCalls);
+         if(CollectionUtils.nullSafeIsEmpty(groupByIndexes))
+         {
+            groupByIndexes = groupByAllIndexesFromTable(mapping, table, headerRow, associationNameChainForRecursiveCalls);
+            // throw (new QException("Missing group-by-index(es) for association: " + associationNameChainForRecursiveCalls));
+         }
+
+         List<Serializable> rowGroupByValues = getGroupByValues(row, groupByIndexes);
+         if(rowGroupByValues == null)
+         {
+            continue;
+         }
+
+         if(rowsForCurrentRecord.isEmpty())
+         {
+            ///////////////////////////////////
+            // this is first - so it's a yes //
+            ///////////////////////////////////
+            recordGroupByValues = rowGroupByValues;
+            rowsForCurrentRecord.add(row);
+         }
+         else if(Objects.equals(recordGroupByValues, rowGroupByValues))
+         {
+            /////////////////////////////
+            // a match - so keep going //
+            /////////////////////////////
+            rowsForCurrentRecord.add(row);
+         }
+         else
+         {
+            //////////////////////////////////////////////////////////////
+            // not first, and not a match, so we can finish this record //
+            //////////////////////////////////////////////////////////////
+            rs.add(makeRecordFromRows(table, associationNameChainForRecursiveCalls, mapping, headerRow, rowsForCurrentRecord));
+
+            ////////////////////////////////////////
+            // reset these record-specific values //
+            ////////////////////////////////////////
+            rowsForCurrentRecord = new ArrayList<>();
+
+            //////////////////////////////////////////////////
+            // use the current row to start the next record //
+            //////////////////////////////////////////////////
+            rowsForCurrentRecord.add(row);
+            recordGroupByValues = rowGroupByValues;
+         }
+      }
+
+      ///////////
+      // final //
+      ///////////
+      if(!rowsForCurrentRecord.isEmpty())
+      {
+         rs.add(makeRecordFromRows(table, associationNameChainForRecursiveCalls, mapping, headerRow, rowsForCurrentRecord));
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<Serializable> getGroupByValues(BulkLoadFileRow row, List<Integer> indexes)
+   {
+      List<Serializable> rowGroupByValues     = new ArrayList<>();
+      boolean            haveAnyGroupByValues = false;
+      for(Integer index : indexes)
+      {
+         Serializable value = row.getValueElseNull(index);
+         rowGroupByValues.add(value);
+
+         if(value != null && !"".equals(value))
+         {
+            haveAnyGroupByValues = true;
+         }
+      }
+
+      if(!haveAnyGroupByValues)
+      {
+         return (null);
+      }
+
+      return (rowGroupByValues);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping.java

@@ -0,0 +1,198 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.Pair;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+
+
+/*******************************************************************************
+ ** use a flatter mapping object, where field names look like:
+ ** associationChain.fieldName,index.subIndex
+ *******************************************************************************/
+public class WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping implements RowsToRecordInterface
+{
+   private Memoization<Pair<String, String>, Boolean> shouldProcesssAssociationMemoization = new Memoization<>();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public List<QRecord> nextPage(FileToRowsInterface fileToRowsInterface, BulkLoadFileRow headerRow, BulkInsertMapping mapping, Integer limit) throws QException
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(mapping.getTableName());
+      if(table == null)
+      {
+         throw (new QException("Table [" + mapping.getTableName() + "] was not found in the Instance"));
+      }
+
+      List<QRecord> rs = new ArrayList<>();
+
+      Map<String, Integer> fieldIndexes = mapping.getFieldIndexes(table, null, headerRow);
+
+      while(fileToRowsInterface.hasNext() && rs.size() < limit)
+      {
+         BulkLoadFileRow row    = fileToRowsInterface.next();
+         QRecord         record = makeRecordFromRow(mapping, table, "", row, fieldIndexes, headerRow, new ArrayList<>());
+         rs.add(record);
+      }
+
+      BulkLoadValueMapper.valueMapping(rs, mapping, table);
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    ** may return null, if there were no values in the row for this (sub-wide) record.
+    ***************************************************************************/
+   private QRecord makeRecordFromRow(BulkInsertMapping mapping, QTableMetaData table, String associationNameChain, BulkLoadFileRow row, Map<String, Integer> fieldIndexes, BulkLoadFileRow headerRow, List<Integer> wideAssociationIndexes) throws QException
+   {
+      //////////////////////////////////////////////////////
+      // start by building the record with its own fields //
+      //////////////////////////////////////////////////////
+      QRecord record = new QRecord();
+      BulkLoadRecordUtils.addBackendDetailsAboutFileRows(record, row);
+
+      boolean hadAnyValuesInRow = false;
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         hadAnyValuesInRow = setValueOrDefault(record, field, associationNameChain, mapping, row, fieldIndexes.get(field.getName()), wideAssociationIndexes) || hadAnyValuesInRow;
+      }
+
+      if(!hadAnyValuesInRow)
+      {
+         return (null);
+      }
+
+      /////////////////////////////
+      // associations (children) //
+      /////////////////////////////
+      for(String associationName : CollectionUtils.nonNullList(mapping.getMappedAssociations()))
+      {
+         boolean processAssociation = shouldProcessAssociation(associationNameChain, associationName);
+
+         if(processAssociation)
+         {
+            String associationNameMinusChain = StringUtils.hasContent(associationNameChain)
+               ? associationName.substring(associationNameChain.length() + 1)
+               : associationName;
+
+            Optional<Association> association = table.getAssociations().stream().filter(a -> a.getName().equals(associationNameMinusChain)).findFirst();
+            if(association.isEmpty())
+            {
+               throw (new QException("Couldn't find association: " + associationNameMinusChain + " under table: " + table.getName()));
+            }
+
+            QTableMetaData associatedTable = QContext.getQInstance().getTable(association.get().getAssociatedTableName());
+
+            List<QRecord> associatedRecords = processAssociation(associationNameMinusChain, associationNameChain, associatedTable, mapping, row, headerRow);
+            record.withAssociatedRecords(associationNameMinusChain, associatedRecords);
+         }
+      }
+
+      return record;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<QRecord> processAssociation(String associationName, String associationNameChain, QTableMetaData associatedTable, BulkInsertMapping mapping, BulkLoadFileRow row, BulkLoadFileRow headerRow) throws QException
+   {
+      List<QRecord> rs = new ArrayList<>();
+
+      String associationNameChainForRecursiveCalls = "".equals(associationNameChain) ? associationName : associationNameChain + "." + associationName;
+
+      for(int i = 0; true; i++)
+      {
+         // todo - doesn't support grand-children
+         List<Integer>        wideAssociationIndexes = List.of(i);
+         Map<String, Integer> fieldIndexes           = mapping.getFieldIndexes(associatedTable, associationNameChainForRecursiveCalls, headerRow, wideAssociationIndexes);
+         if(fieldIndexes.isEmpty())
+         {
+            break;
+         }
+
+         QRecord record = makeRecordFromRow(mapping, associatedTable, associationNameChainForRecursiveCalls, row, fieldIndexes, headerRow, wideAssociationIndexes);
+         if(record != null)
+         {
+            rs.add(record);
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   boolean shouldProcessAssociation(String associationNameChain, String associationName)
+   {
+      return shouldProcesssAssociationMemoization.getResult(Pair.of(associationNameChain, associationName), p ->
+      {
+         List<String> chainParts = new ArrayList<>();
+         List<String> nameParts  = new ArrayList<>();
+
+         if(StringUtils.hasContent(associationNameChain))
+         {
+            chainParts.addAll(Arrays.asList(associationNameChain.split("\\.")));
+         }
+
+         if(StringUtils.hasContent(associationName))
+         {
+            nameParts.addAll(Arrays.asList(associationName.split("\\.")));
+         }
+
+         if(!nameParts.isEmpty())
+         {
+            nameParts.remove(nameParts.size() - 1);
+         }
+
+         return (chainParts.equals(nameParts));
+      }).orElse(false);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/WideRowsToRecordWithSpreadMapping.java

@@ -0,0 +1,268 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.FileToRowsInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.Pair;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class WideRowsToRecordWithSpreadMapping implements RowsToRecordInterface
+{
+   private Memoization<Pair<String, String>, Boolean> shouldProcesssAssociationMemoization = new Memoization<>();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public List<QRecord> nextPage(FileToRowsInterface fileToRowsInterface, BulkLoadFileRow headerRow, BulkInsertMapping mapping, Integer limit) throws QException
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(mapping.getTableName());
+      if(table == null)
+      {
+         throw (new QException("Table [" + mapping.getTableName() + "] was not found in the Instance"));
+      }
+
+      List<QRecord> rs = new ArrayList<>();
+
+      Map<String, Integer> fieldIndexes = mapping.getFieldIndexes(table, null, headerRow);
+
+      while(fileToRowsInterface.hasNext() && rs.size() < limit)
+      {
+         BulkLoadFileRow row    = fileToRowsInterface.next();
+         QRecord         record = new QRecord();
+
+         for(QFieldMetaData field : table.getFields().values())
+         {
+            setValueOrDefault(record, field, null, mapping, row, fieldIndexes.get(field.getName()));
+         }
+
+         processAssociations("", headerRow, mapping, table, row, record, 0, headerRow.size());
+
+         rs.add(record);
+      }
+
+      BulkLoadValueMapper.valueMapping(rs, mapping, table);
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void processAssociations(String associationNameChain, BulkLoadFileRow headerRow, BulkInsertMapping mapping, QTableMetaData table, BulkLoadFileRow row, QRecord record, int startIndex, int endIndex) throws QException
+   {
+      for(String associationName : mapping.getMappedAssociations())
+      {
+         boolean processAssociation = shouldProcessAssociation(associationNameChain, associationName);
+
+         if(processAssociation)
+         {
+            String associationNameMinusChain = StringUtils.hasContent(associationNameChain)
+               ? associationName.substring(associationNameChain.length() + 1)
+               : associationName;
+
+            Optional<Association> association = table.getAssociations().stream().filter(a -> a.getName().equals(associationNameMinusChain)).findFirst();
+            if(association.isEmpty())
+            {
+               throw (new QException("Couldn't find association: " + associationNameMinusChain + " under table: " + table.getName()));
+            }
+
+            QTableMetaData associatedTable = QContext.getQInstance().getTable(association.get().getAssociatedTableName());
+
+            // List<QRecord> associatedRecords = processAssociation(associationName, associationNameChain, associatedTable, mapping, row, headerRow, record);
+            List<QRecord> associatedRecords = processAssociationV2(associationName, associationNameChain, associatedTable, mapping, row, headerRow, record, startIndex, endIndex);
+            record.withAssociatedRecords(associationNameMinusChain, associatedRecords);
+         }
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<QRecord> processAssociationV2(String associationName, String associationNameChain, QTableMetaData table, BulkInsertMapping mapping, BulkLoadFileRow row, BulkLoadFileRow headerRow, QRecord record, int startIndex, int endIndex) throws QException
+   {
+      List<QRecord> rs = new ArrayList<>();
+
+      Map<String, String> fieldNameToHeaderNameMapForThisAssociation = new HashMap<>();
+      for(Map.Entry<String, String> entry : mapping.getFieldNameToHeaderNameMap().entrySet())
+      {
+         if(entry.getKey().startsWith(associationName + "."))
+         {
+            String fieldName = entry.getKey().substring(associationName.length() + 1);
+
+            //////////////////////////////////////////////////////////////////////////
+            // make sure the name here is for this table - not a sub-table under it //
+            //////////////////////////////////////////////////////////////////////////
+            if(!fieldName.contains("."))
+            {
+               fieldNameToHeaderNameMapForThisAssociation.put(fieldName, entry.getValue());
+            }
+         }
+      }
+
+      /////////////////////////////////////////////////////////////////////
+      // loop over the length of the record, building associated records //
+      /////////////////////////////////////////////////////////////////////
+      QRecord     associatedRecord    = new QRecord();
+      Set<String> processedFieldNames = new HashSet<>();
+      boolean     gotAnyValues        = false;
+      int         subStartIndex       = -1;
+
+      for(int i = startIndex; i < endIndex; i++)
+      {
+         String headerValue = ValueUtils.getValueAsString(headerRow.getValue(i));
+
+         for(Map.Entry<String, String> entry : fieldNameToHeaderNameMapForThisAssociation.entrySet())
+         {
+            if(headerValue.equals(entry.getValue()) || headerValue.matches(entry.getValue() + " ?\\d+"))
+            {
+               ///////////////////////////////////////////////
+               // ok - this is a value for this association //
+               ///////////////////////////////////////////////
+               if(subStartIndex == -1)
+               {
+                  subStartIndex = i;
+               }
+
+               String fieldName = entry.getKey();
+               if(processedFieldNames.contains(fieldName))
+               {
+                  /////////////////////////////////////////////////
+                  // this means we're starting a new sub-record! //
+                  /////////////////////////////////////////////////
+                  if(gotAnyValues)
+                  {
+                     addDefaultValuesToAssociatedRecord(processedFieldNames, table, associatedRecord, mapping, associationName);
+                     processAssociations(associationName, headerRow, mapping, table, row, associatedRecord, subStartIndex, i);
+                     rs.add(associatedRecord);
+                  }
+
+                  associatedRecord = new QRecord();
+                  processedFieldNames = new HashSet<>();
+                  gotAnyValues = false;
+                  subStartIndex = i + 1;
+               }
+
+               processedFieldNames.add(fieldName);
+
+               Serializable value = row.getValueElseNull(i);
+               if(value != null && !"".equals(value))
+               {
+                  gotAnyValues = true;
+               }
+
+               setValueOrDefault(associatedRecord, table.getField(fieldName), associationName, mapping, row, i);
+            }
+         }
+      }
+
+      ////////////////////////
+      // handle final value //
+      ////////////////////////
+      if(gotAnyValues)
+      {
+         addDefaultValuesToAssociatedRecord(processedFieldNames, table, associatedRecord, mapping, associationName);
+         processAssociations(associationName, headerRow, mapping, table, row, associatedRecord, subStartIndex, endIndex);
+         rs.add(associatedRecord);
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void addDefaultValuesToAssociatedRecord(Set<String> processedFieldNames, QTableMetaData table, QRecord associatedRecord, BulkInsertMapping mapping, String associationNameChain)
+   {
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         if(!processedFieldNames.contains(field.getName()))
+         {
+            setValueOrDefault(associatedRecord, field, associationNameChain, mapping, null, null);
+         }
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   boolean shouldProcessAssociation(String associationNameChain, String associationName)
+   {
+      return shouldProcesssAssociationMemoization.getResult(Pair.of(associationNameChain, associationName), p ->
+      {
+         List<String> chainParts = new ArrayList<>();
+         List<String> nameParts  = new ArrayList<>();
+
+         if(StringUtils.hasContent(associationNameChain))
+         {
+            chainParts.addAll(Arrays.asList(associationNameChain.split("\\.")));
+         }
+
+         if(StringUtils.hasContent(associationName))
+         {
+            nameParts.addAll(Arrays.asList(associationName.split("\\.")));
+         }
+
+         if(!nameParts.isEmpty())
+         {
+            nameParts.remove(nameParts.size() - 1);
+         }
+
+         return (chainParts.equals(nameParts));
+      }).orElse(false);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/model/BulkInsertMapping.java

@@ -0,0 +1,581 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.FlatRowsToRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.RowsToRecordInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.TallRowsToRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.Pair;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import com.kingsrook.qqq.backend.core.utils.memoization.Memoization;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkInsertMapping implements Serializable
+{
+   private String  tableName;
+   private Boolean hasHeaderRow;
+
+   private Layout layout;
+
+   /////////////////////////////////////////////////////////////////////
+   // keys in here are:                                               //
+   // fieldName (for the main table)                                  //
+   // association.fieldName (for an associated child table)           //
+   // association.association.fieldName (for grandchild associations) //
+   /////////////////////////////////////////////////////////////////////
+   private Map<String, String>                    fieldNameToHeaderNameMap   = new HashMap<>();
+   private Map<String, Integer>                   fieldNameToIndexMap        = new HashMap<>();
+   private Map<String, Serializable>              fieldNameToDefaultValueMap = new HashMap<>();
+   private Map<String, Map<String, Serializable>> fieldNameToValueMapping    = new HashMap<>();
+
+   private Map<String, List<Integer>> tallLayoutGroupByIndexMap = new HashMap<>();
+
+   private List<String> mappedAssociations = new ArrayList<>();
+
+   private Memoization<Pair<String, String>, Boolean> shouldProcessFieldForTable = new Memoization<>();
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public enum Layout implements PossibleValueEnum<String>
+   {
+      FLAT(FlatRowsToRecord::new),
+      TALL(TallRowsToRecord::new),
+      WIDE(WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping::new);
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      private final Supplier<? extends RowsToRecordInterface> supplier;
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      Layout(Supplier<? extends RowsToRecordInterface> supplier)
+      {
+         this.supplier = supplier;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      public RowsToRecordInterface newRowsToRecordInterface()
+      {
+         return (supplier.get());
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public String getPossibleValueId()
+      {
+         return name();
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public String getPossibleValueLabel()
+      {
+         return StringUtils.ucFirst(name().toLowerCase());
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @JsonIgnore
+   public Map<String, Integer> getFieldIndexes(QTableMetaData table, String associationNameChain, BulkLoadFileRow headerRow) throws QException
+   {
+      return getFieldIndexes(table, associationNameChain, headerRow, null);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @JsonIgnore
+   public Map<String, Integer> getFieldIndexes(QTableMetaData table, String associationNameChain, BulkLoadFileRow headerRow, List<Integer> wideAssociationIndexes) throws QException
+   {
+      if(hasHeaderRow && fieldNameToHeaderNameMap != null)
+      {
+         return (getFieldIndexesForHeaderMappedUseCase(table, associationNameChain, headerRow, wideAssociationIndexes));
+      }
+      else if(fieldNameToIndexMap != null)
+      {
+         return (getFieldIndexesForNoHeaderUseCase(table, associationNameChain, wideAssociationIndexes));
+      }
+
+      throw (new QException("Mapping was not properly configured."));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @JsonIgnore
+   private Map<String, Integer> getFieldIndexesForNoHeaderUseCase(QTableMetaData table, String associationNameChain, List<Integer> wideAssociationIndexes)
+   {
+      Map<String, Integer> rs = new HashMap<>();
+
+      String wideAssociationSuffix = "";
+      if(CollectionUtils.nullSafeHasContents(wideAssociationIndexes))
+      {
+         wideAssociationSuffix = "," + StringUtils.join(".", wideAssociationIndexes);
+      }
+
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // loop over fields - finding what header name they are mapped to - then what index that header is at. //
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////
+      String fieldNamePrefix = !StringUtils.hasContent(associationNameChain) ? "" : associationNameChain + ".";
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         Integer index = fieldNameToIndexMap.get(fieldNamePrefix + field.getName() + wideAssociationSuffix);
+         if(index != null)
+         {
+            rs.put(field.getName(), index);
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @JsonIgnore
+   public Map<String, Map<String, Serializable>> getFieldNameToValueMappingForTable(String associatedTableName)
+   {
+      Map<String, Map<String, Serializable>> rs = new HashMap<>();
+
+      for(Map.Entry<String, Map<String, Serializable>> entry : CollectionUtils.nonNullMap(fieldNameToValueMapping).entrySet())
+      {
+         if(shouldProcessFieldForTable(entry.getKey(), associatedTableName))
+         {
+            String key = StringUtils.hasContent(associatedTableName) ? entry.getKey().substring(associatedTableName.length() + 1) : entry.getKey();
+            rs.put(key, entry.getValue());
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   boolean shouldProcessFieldForTable(String fieldNameWithAssociationPrefix, String associationChain)
+   {
+      return shouldProcessFieldForTable.getResult(Pair.of(fieldNameWithAssociationPrefix, associationChain), p ->
+      {
+         List<String> fieldNameParts   = new ArrayList<>();
+         List<String> associationParts = new ArrayList<>();
+
+         if(StringUtils.hasContent(fieldNameWithAssociationPrefix))
+         {
+            fieldNameParts.addAll(Arrays.asList(fieldNameWithAssociationPrefix.split("\\.")));
+         }
+
+         if(StringUtils.hasContent(associationChain))
+         {
+            associationParts.addAll(Arrays.asList(associationChain.split("\\.")));
+         }
+
+         if(!fieldNameParts.isEmpty())
+         {
+            fieldNameParts.remove(fieldNameParts.size() - 1);
+         }
+
+         return (fieldNameParts.equals(associationParts));
+      }).orElse(false);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private Map<String, Integer> getFieldIndexesForHeaderMappedUseCase(QTableMetaData table, String associationNameChain, BulkLoadFileRow headerRow, List<Integer> wideAssociationIndexes)
+   {
+      Map<String, Integer> rs = new HashMap<>();
+
+      ////////////////////////////////////////////////////////
+      // for the current file, map header values to indexes //
+      ////////////////////////////////////////////////////////
+      Map<String, Integer> headerToIndexMap = new HashMap<>();
+      for(int i = 0; i < headerRow.size(); i++)
+      {
+         String headerValue = ValueUtils.getValueAsString(headerRow.getValue(i));
+         headerToIndexMap.put(headerValue, i);
+      }
+
+      String wideAssociationSuffix = "";
+      if(CollectionUtils.nullSafeHasContents(wideAssociationIndexes))
+      {
+         wideAssociationSuffix = "," + StringUtils.join(".", wideAssociationIndexes);
+      }
+
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // loop over fields - finding what header name they are mapped to - then what index that header is at. //
+      /////////////////////////////////////////////////////////////////////////////////////////////////////////
+      String fieldNamePrefix = !StringUtils.hasContent(associationNameChain) ? "" : associationNameChain + ".";
+      for(QFieldMetaData field : table.getFields().values())
+      {
+         String headerName = fieldNameToHeaderNameMap.get(fieldNamePrefix + field.getName() + wideAssociationSuffix);
+         if(headerName != null)
+         {
+            Integer headerIndex = headerToIndexMap.get(headerName);
+            if(headerIndex != null)
+            {
+               rs.put(field.getName(), headerIndex);
+            }
+         }
+      }
+
+      return (rs);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    *******************************************************************************/
+   public String getTableName()
+   {
+      return (this.tableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    *******************************************************************************/
+   public void setTableName(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableName
+    *******************************************************************************/
+   public BulkInsertMapping withTableName(String tableName)
+   {
+      this.tableName = tableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for hasHeaderRow
+    *******************************************************************************/
+   public Boolean getHasHeaderRow()
+   {
+      return (this.hasHeaderRow);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for hasHeaderRow
+    *******************************************************************************/
+   public void setHasHeaderRow(Boolean hasHeaderRow)
+   {
+      this.hasHeaderRow = hasHeaderRow;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for hasHeaderRow
+    *******************************************************************************/
+   public BulkInsertMapping withHasHeaderRow(Boolean hasHeaderRow)
+   {
+      this.hasHeaderRow = hasHeaderRow;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldNameToHeaderNameMap
+    *******************************************************************************/
+   public Map<String, String> getFieldNameToHeaderNameMap()
+   {
+      return (this.fieldNameToHeaderNameMap);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldNameToHeaderNameMap
+    *******************************************************************************/
+   public void setFieldNameToHeaderNameMap(Map<String, String> fieldNameToHeaderNameMap)
+   {
+      this.fieldNameToHeaderNameMap = fieldNameToHeaderNameMap;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldNameToHeaderNameMap
+    *******************************************************************************/
+   public BulkInsertMapping withFieldNameToHeaderNameMap(Map<String, String> fieldNameToHeaderNameMap)
+   {
+      this.fieldNameToHeaderNameMap = fieldNameToHeaderNameMap;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldNameToIndexMap
+    *******************************************************************************/
+   public Map<String, Integer> getFieldNameToIndexMap()
+   {
+      return (this.fieldNameToIndexMap);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldNameToIndexMap
+    *******************************************************************************/
+   public void setFieldNameToIndexMap(Map<String, Integer> fieldNameToIndexMap)
+   {
+      this.fieldNameToIndexMap = fieldNameToIndexMap;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldNameToIndexMap
+    *******************************************************************************/
+   public BulkInsertMapping withFieldNameToIndexMap(Map<String, Integer> fieldNameToIndexMap)
+   {
+      this.fieldNameToIndexMap = fieldNameToIndexMap;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for mappedAssociations
+    *******************************************************************************/
+   public List<String> getMappedAssociations()
+   {
+      return (this.mappedAssociations);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for mappedAssociations
+    *******************************************************************************/
+   public void setMappedAssociations(List<String> mappedAssociations)
+   {
+      this.mappedAssociations = mappedAssociations;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for mappedAssociations
+    *******************************************************************************/
+   public BulkInsertMapping withMappedAssociations(List<String> mappedAssociations)
+   {
+      this.mappedAssociations = mappedAssociations;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldNameToDefaultValueMap
+    *******************************************************************************/
+   public Map<String, Serializable> getFieldNameToDefaultValueMap()
+   {
+      if(this.fieldNameToDefaultValueMap == null)
+      {
+         this.fieldNameToDefaultValueMap = new HashMap<>();
+      }
+
+      return (this.fieldNameToDefaultValueMap);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldNameToDefaultValueMap
+    *******************************************************************************/
+   public void setFieldNameToDefaultValueMap(Map<String, Serializable> fieldNameToDefaultValueMap)
+   {
+      this.fieldNameToDefaultValueMap = fieldNameToDefaultValueMap;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldNameToDefaultValueMap
+    *******************************************************************************/
+   public BulkInsertMapping withFieldNameToDefaultValueMap(Map<String, Serializable> fieldNameToDefaultValueMap)
+   {
+      this.fieldNameToDefaultValueMap = fieldNameToDefaultValueMap;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldNameToValueMapping
+    *******************************************************************************/
+   public Map<String, Map<String, Serializable>> getFieldNameToValueMapping()
+   {
+      return (this.fieldNameToValueMapping);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldNameToValueMapping
+    *******************************************************************************/
+   public void setFieldNameToValueMapping(Map<String, Map<String, Serializable>> fieldNameToValueMapping)
+   {
+      this.fieldNameToValueMapping = fieldNameToValueMapping;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldNameToValueMapping
+    *******************************************************************************/
+   public BulkInsertMapping withFieldNameToValueMapping(Map<String, Map<String, Serializable>> fieldNameToValueMapping)
+   {
+      this.fieldNameToValueMapping = fieldNameToValueMapping;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for layout
+    *******************************************************************************/
+   public Layout getLayout()
+   {
+      return (this.layout);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for layout
+    *******************************************************************************/
+   public void setLayout(Layout layout)
+   {
+      this.layout = layout;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for layout
+    *******************************************************************************/
+   public BulkInsertMapping withLayout(Layout layout)
+   {
+      this.layout = layout;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tallLayoutGroupByIndexMap
+    *******************************************************************************/
+   public Map<String, List<Integer>> getTallLayoutGroupByIndexMap()
+   {
+      return (this.tallLayoutGroupByIndexMap);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tallLayoutGroupByIndexMap
+    *******************************************************************************/
+   public void setTallLayoutGroupByIndexMap(Map<String, List<Integer>> tallLayoutGroupByIndexMap)
+   {
+      this.tallLayoutGroupByIndexMap = tallLayoutGroupByIndexMap;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tallLayoutGroupByIndexMap
+    *******************************************************************************/
+   public BulkInsertMapping withTallLayoutGroupByIndexMap(Map<String, List<Integer>> tallLayoutGroupByIndexMap)
+   {
+      this.tallLayoutGroupByIndexMap = tallLayoutGroupByIndexMap;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/model/BulkLoadFileRow.java

@@ -0,0 +1,201 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model;
+
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+
+/*******************************************************************************
+ ** A row of values, e.g., from a file, for bulk-load
+ *******************************************************************************/
+public class BulkLoadFileRow implements Serializable
+{
+   private int            rowNo;
+   private Serializable[] values;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public BulkLoadFileRow(Serializable[] values, int rowNo)
+   {
+      this.values = values;
+      this.rowNo = rowNo;
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public int size()
+   {
+      if(values == null)
+      {
+         return (0);
+      }
+
+      return (values.length);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public boolean hasIndex(int i)
+   {
+      if(values == null)
+      {
+         return (false);
+      }
+
+      if(i >= values.length || i < 0)
+      {
+         return (false);
+      }
+
+      return (true);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public Serializable getValue(int i)
+   {
+      if(values == null)
+      {
+         throw new IllegalStateException("Row has no values");
+      }
+
+      if(i >= values.length || i < 0)
+      {
+         throw new IllegalArgumentException("Index out of bounds:  Requested index " + i + "; values.length: " + values.length);
+      }
+
+      return (values[i]);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public Serializable getValueElseNull(int i)
+   {
+      if(!hasIndex(i))
+      {
+         return (null);
+      }
+
+      return (values[i]);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public String toString()
+   {
+      if(values == null)
+      {
+         return ("null");
+      }
+
+      return Arrays.stream(values).map(String::valueOf).collect(Collectors.joining(","));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public boolean equals(Object o)
+   {
+      if(this == o)
+      {
+         return true;
+      }
+
+      if(o == null || getClass() != o.getClass())
+      {
+         return false;
+      }
+
+      BulkLoadFileRow that = (BulkLoadFileRow) o;
+      return rowNo == that.rowNo && Objects.deepEquals(values, that.values);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public int hashCode()
+   {
+      return Objects.hash(rowNo, Arrays.hashCode(values));
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for rowNo
+    *******************************************************************************/
+   public int getRowNo()
+   {
+      return (this.rowNo);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for rowNo
+    *******************************************************************************/
+   public void setRowNo(int rowNo)
+   {
+      this.rowNo = rowNo;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for rowNo
+    *******************************************************************************/
+   public BulkLoadFileRow withRowNo(int rowNo)
+   {
+      this.rowNo = rowNo;
+      return (this);
+   }
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/model/BulkLoadProfile.java

@@ -0,0 +1,165 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+
+
+/***************************************************************************
+ * this is the model of a saved bulk load profile - which is what passes back
+ * and forth with the frontend.
+ ****************************************************************************/
+public class BulkLoadProfile implements Serializable
+{
+   private ArrayList<BulkLoadProfileField> fieldList;
+
+   private Boolean hasHeaderRow;
+   private String  layout;
+   private String  version;
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldList
+    *******************************************************************************/
+   public ArrayList<BulkLoadProfileField> getFieldList()
+   {
+      return (this.fieldList);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for hasHeaderRow
+    *******************************************************************************/
+   public Boolean getHasHeaderRow()
+   {
+      return (this.hasHeaderRow);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for hasHeaderRow
+    *******************************************************************************/
+   public void setHasHeaderRow(Boolean hasHeaderRow)
+   {
+      this.hasHeaderRow = hasHeaderRow;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for hasHeaderRow
+    *******************************************************************************/
+   public BulkLoadProfile withHasHeaderRow(Boolean hasHeaderRow)
+   {
+      this.hasHeaderRow = hasHeaderRow;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for layout
+    *******************************************************************************/
+   public String getLayout()
+   {
+      return (this.layout);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for layout
+    *******************************************************************************/
+   public void setLayout(String layout)
+   {
+      this.layout = layout;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for layout
+    *******************************************************************************/
+   public BulkLoadProfile withLayout(String layout)
+   {
+      this.layout = layout;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldList
+    *******************************************************************************/
+   public void setFieldList(ArrayList<BulkLoadProfileField> fieldList)
+   {
+      this.fieldList = fieldList;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldList
+    *******************************************************************************/
+   public BulkLoadProfile withFieldList(ArrayList<BulkLoadProfileField> fieldList)
+   {
+      this.fieldList = fieldList;
+      return (this);
+   }
+
+
+   /*******************************************************************************
+    ** Getter for version
+    *******************************************************************************/
+   public String getVersion()
+   {
+      return (this.version);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for version
+    *******************************************************************************/
+   public void setVersion(String version)
+   {
+      this.version = version;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for version
+    *******************************************************************************/
+   public BulkLoadProfile withVersion(String version)
+   {
+      this.version = version;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/model/BulkLoadProfileField.java

@@ -0,0 +1,227 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model;
+
+
+import java.io.Serializable;
+import java.util.Map;
+
+
+/***************************************************************************
+ **
+ ***************************************************************************/
+public class BulkLoadProfileField
+{
+   private String                    fieldName;
+   private Integer                   columnIndex;
+   private String                    headerName;
+   private Serializable              defaultValue;
+   private Boolean                   doValueMapping;
+   private Map<String, Serializable> valueMappings;
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldName
+    *******************************************************************************/
+   public String getFieldName()
+   {
+      return (this.fieldName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldName
+    *******************************************************************************/
+   public void setFieldName(String fieldName)
+   {
+      this.fieldName = fieldName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldName
+    *******************************************************************************/
+   public BulkLoadProfileField withFieldName(String fieldName)
+   {
+      this.fieldName = fieldName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for columnIndex
+    *******************************************************************************/
+   public Integer getColumnIndex()
+   {
+      return (this.columnIndex);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for columnIndex
+    *******************************************************************************/
+   public void setColumnIndex(Integer columnIndex)
+   {
+      this.columnIndex = columnIndex;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for columnIndex
+    *******************************************************************************/
+   public BulkLoadProfileField withColumnIndex(Integer columnIndex)
+   {
+      this.columnIndex = columnIndex;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for defaultValue
+    *******************************************************************************/
+   public Serializable getDefaultValue()
+   {
+      return (this.defaultValue);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for defaultValue
+    *******************************************************************************/
+   public void setDefaultValue(Serializable defaultValue)
+   {
+      this.defaultValue = defaultValue;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for defaultValue
+    *******************************************************************************/
+   public BulkLoadProfileField withDefaultValue(Serializable defaultValue)
+   {
+      this.defaultValue = defaultValue;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for doValueMapping
+    *******************************************************************************/
+   public Boolean getDoValueMapping()
+   {
+      return (this.doValueMapping);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for doValueMapping
+    *******************************************************************************/
+   public void setDoValueMapping(Boolean doValueMapping)
+   {
+      this.doValueMapping = doValueMapping;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for doValueMapping
+    *******************************************************************************/
+   public BulkLoadProfileField withDoValueMapping(Boolean doValueMapping)
+   {
+      this.doValueMapping = doValueMapping;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for valueMappings
+    *******************************************************************************/
+   public Map<String, Serializable> getValueMappings()
+   {
+      return (this.valueMappings);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for valueMappings
+    *******************************************************************************/
+   public void setValueMappings(Map<String, Serializable> valueMappings)
+   {
+      this.valueMappings = valueMappings;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for valueMappings
+    *******************************************************************************/
+   public BulkLoadProfileField withValueMappings(Map<String, Serializable> valueMappings)
+   {
+      this.valueMappings = valueMappings;
+      return (this);
+   }
+
+
+   /*******************************************************************************
+    ** Getter for headerName
+    *******************************************************************************/
+   public String getHeaderName()
+   {
+      return (this.headerName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for headerName
+    *******************************************************************************/
+   public void setHeaderName(String headerName)
+   {
+      this.headerName = headerName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for headerName
+    *******************************************************************************/
+   public BulkLoadProfileField withHeaderName(String headerName)
+   {
+      this.headerName = headerName;
+      return (this);
+   }
+
+
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/model/BulkLoadTableStructure.java

@@ -0,0 +1,275 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class BulkLoadTableStructure implements Serializable
+{
+   private boolean isMain;
+   private boolean isMany;
+
+   private String tableName;
+   private String label;
+   private String associationPath; // null/empty for main table, then associationName for a child, associationName.associationName for a grandchild
+
+   private ArrayList<QFieldMetaData>         fields; // mmm, not marked as serializable (at this time) - is okay?
+   private ArrayList<BulkLoadTableStructure> associations;
+
+
+
+   /*******************************************************************************
+    ** Getter for isMain
+    *******************************************************************************/
+   public boolean getIsMain()
+   {
+      return (this.isMain);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for isMain
+    *******************************************************************************/
+   public void setIsMain(boolean isMain)
+   {
+      this.isMain = isMain;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for isMain
+    *******************************************************************************/
+   public BulkLoadTableStructure withIsMain(boolean isMain)
+   {
+      this.isMain = isMain;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for isMany
+    *******************************************************************************/
+   public boolean getIsMany()
+   {
+      return (this.isMany);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for isMany
+    *******************************************************************************/
+   public void setIsMany(boolean isMany)
+   {
+      this.isMany = isMany;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for isMany
+    *******************************************************************************/
+   public BulkLoadTableStructure withIsMany(boolean isMany)
+   {
+      this.isMany = isMany;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    *******************************************************************************/
+   public String getTableName()
+   {
+      return (this.tableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    *******************************************************************************/
+   public void setTableName(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableName
+    *******************************************************************************/
+   public BulkLoadTableStructure withTableName(String tableName)
+   {
+      this.tableName = tableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for label
+    *******************************************************************************/
+   public String getLabel()
+   {
+      return (this.label);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for label
+    *******************************************************************************/
+   public void setLabel(String label)
+   {
+      this.label = label;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for label
+    *******************************************************************************/
+   public BulkLoadTableStructure withLabel(String label)
+   {
+      this.label = label;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for fields
+    *******************************************************************************/
+   public ArrayList<QFieldMetaData> getFields()
+   {
+      return (this.fields);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fields
+    *******************************************************************************/
+   public void setFields(ArrayList<QFieldMetaData> fields)
+   {
+      this.fields = fields;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fields
+    *******************************************************************************/
+   public BulkLoadTableStructure withFields(ArrayList<QFieldMetaData> fields)
+   {
+      this.fields = fields;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for associationPath
+    *******************************************************************************/
+   public String getAssociationPath()
+   {
+      return (this.associationPath);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for associationPath
+    *******************************************************************************/
+   public void setAssociationPath(String associationPath)
+   {
+      this.associationPath = associationPath;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for associationPath
+    *******************************************************************************/
+   public BulkLoadTableStructure withAssociationPath(String associationPath)
+   {
+      this.associationPath = associationPath;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for associations
+    *******************************************************************************/
+   public ArrayList<BulkLoadTableStructure> getAssociations()
+   {
+      return (this.associations);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for associations
+    *******************************************************************************/
+   public void setAssociations(ArrayList<BulkLoadTableStructure> associations)
+   {
+      this.associations = associations;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for associations
+    *******************************************************************************/
+   public BulkLoadTableStructure withAssociations(ArrayList<BulkLoadTableStructure> associations)
+   {
+      this.associations = associations;
+      return (this);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public void addAssociation(BulkLoadTableStructure association)
+   {
+      if(this.associations == null)
+      {
+         this.associations = new ArrayList<>();
+      }
+      this.associations.add(association);
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/NoopLoadStep.java

@@ -23,6 +23,7 @@ package com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwit
 
 
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
 
@@ -33,6 +34,7 @@ import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutp
  *******************************************************************************/
 public class NoopLoadStep extends AbstractLoadStep
 {
+   private static final QLogger LOG = QLogger.getLogger(NoopLoadStep.class);
 
 
    /*******************************************************************************
@@ -45,6 +47,7 @@ public class NoopLoadStep extends AbstractLoadStep
       ///////////
       // noop. //
       ///////////
+      LOG.trace("noop");
    }
 
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/StreamedETLWithFrontendProcess.java

@@ -27,6 +27,7 @@ import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
@@ -193,6 +194,17 @@ public class StreamedETLWithFrontendProcess
    }
 
 
+   /***************************************************************************
+    ** useful for a process step to call upon 'back'
+    ***************************************************************************/
+   public static void resetValidationFields(RunBackendStepInput runBackendStepInput)
+   {
+      runBackendStepInput.addValue(FIELD_DO_FULL_VALIDATION, null);
+      runBackendStepInput.addValue(FIELD_VALIDATION_SUMMARY, null);
+      runBackendStepInput.addValue(FIELD_PROCESS_SUMMARY, null);
+   }
+
+
 
    /*******************************************************************************
     **

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/general/ProcessSummaryWarningsAndErrorsRollup.java

@@ -477,4 +477,13 @@ public class ProcessSummaryWarningsAndErrorsRollup
    }
 
 
+
+   /*******************************************************************************
+    ** Getter for errorSummaries
+    **
+    *******************************************************************************/
+   public Map<String, ProcessSummaryLine> getErrorSummaries()
+   {
+      return errorSummaries;
+   }
 }

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/savedbulkloadprofiles/DeleteSavedBulkLoadProfileProcess.java

@@ -0,0 +1,88 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.ActionHelper;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfile;
+
+
+/*******************************************************************************
+ ** Process used by the delete bulkLoadProfile dialog
+ *******************************************************************************/
+public class DeleteSavedBulkLoadProfileProcess implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(DeleteSavedBulkLoadProfileProcess.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QProcessMetaData getProcessMetaData()
+   {
+      return (new QProcessMetaData()
+         .withName("deleteSavedBulkLoadProfile")
+         .withStepList(List.of(
+            new QBackendStepMetaData()
+               .withCode(new QCodeReference(DeleteSavedBulkLoadProfileProcess.class))
+               .withName("delete")
+         )));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      ActionHelper.validateSession(runBackendStepInput);
+
+      try
+      {
+         Integer savedBulkLoadProfileId = runBackendStepInput.getValueInteger("id");
+
+         DeleteInput input = new DeleteInput();
+         input.setTableName(SavedBulkLoadProfile.TABLE_NAME);
+         input.setPrimaryKeys(List.of(savedBulkLoadProfileId));
+         new DeleteAction().execute(input);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error deleting saved bulkLoadProfile", e);
+         throw (e);
+      }
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/savedbulkloadprofiles/QuerySavedBulkLoadProfileProcess.java

@@ -0,0 +1,129 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles;
+
+
+import java.io.Serializable;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.ActionHelper;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfile;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** Process used by the saved bulkLoadProfile dialogs
+ *******************************************************************************/
+public class QuerySavedBulkLoadProfileProcess implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(QuerySavedBulkLoadProfileProcess.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QProcessMetaData getProcessMetaData()
+   {
+      return (new QProcessMetaData()
+         .withName("querySavedBulkLoadProfile")
+         .withStepList(List.of(
+            new QBackendStepMetaData()
+               .withCode(new QCodeReference(QuerySavedBulkLoadProfileProcess.class))
+               .withName("query")
+         )));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      ActionHelper.validateSession(runBackendStepInput);
+      Integer savedBulkLoadProfileId = runBackendStepInput.getValueInteger("id");
+
+      try
+      {
+         if(savedBulkLoadProfileId != null)
+         {
+            GetInput input = new GetInput();
+            input.setTableName(SavedBulkLoadProfile.TABLE_NAME);
+            input.setPrimaryKey(savedBulkLoadProfileId);
+
+            GetOutput output = new GetAction().execute(input);
+            if(output.getRecord() == null)
+            {
+               throw (new QNotFoundException("The requested bulkLoadProfile was not found."));
+            }
+
+            runBackendStepOutput.addRecord(output.getRecord());
+            runBackendStepOutput.addValue("savedBulkLoadProfile", output.getRecord());
+            runBackendStepOutput.addValue("savedBulkLoadProfileList", (Serializable) List.of(output.getRecord()));
+         }
+         else
+         {
+            String tableName = runBackendStepInput.getValueString("tableName");
+
+            QueryInput input = new QueryInput();
+            input.setTableName(SavedBulkLoadProfile.TABLE_NAME);
+            input.setFilter(new QQueryFilter()
+               .withCriteria(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, tableName))
+               .withOrderBy(new QFilterOrderBy("label")));
+
+            QueryOutput output = new QueryAction().execute(input);
+            runBackendStepOutput.setRecords(output.getRecords());
+            runBackendStepOutput.addValue("savedBulkLoadProfileList", (Serializable) output.getRecords());
+         }
+      }
+      catch(QNotFoundException qnfe)
+      {
+         LOG.info("BulkLoadProfile not found", logPair("savedBulkLoadProfileId", savedBulkLoadProfileId));
+         throw (qnfe);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error querying for saved bulkLoadProfiles", e);
+         throw (e);
+      }
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/savedbulkloadprofiles/StoreSavedBulkLoadProfileProcess.java

@@ -0,0 +1,171 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles;
+
+
+import java.io.Serializable;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.actions.ActionHelper;
+import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfile;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+
+
+/*******************************************************************************
+ ** Process used by the saved bulkLoadProfile dialog
+ *******************************************************************************/
+public class StoreSavedBulkLoadProfileProcess implements BackendStep
+{
+   private static final QLogger LOG = QLogger.getLogger(StoreSavedBulkLoadProfileProcess.class);
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QProcessMetaData getProcessMetaData()
+   {
+      return (new QProcessMetaData()
+         .withName("storeSavedBulkLoadProfile")
+         .withStepList(List.of(
+            new QBackendStepMetaData()
+               .withCode(new QCodeReference(StoreSavedBulkLoadProfileProcess.class))
+               .withName("store")
+         )));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
+   {
+      ActionHelper.validateSession(runBackendStepInput);
+
+      try
+      {
+         String userId    = QContext.getQSession().getUser().getIdReference();
+         String tableName = runBackendStepInput.getValueString("tableName");
+         String label     = runBackendStepInput.getValueString("label");
+
+         String mappingJson = processMappingJson(runBackendStepInput.getValueString("mappingJson"));
+
+         QRecord qRecord = new QRecord()
+            .withValue("id", runBackendStepInput.getValueInteger("id"))
+            .withValue("mappingJson", mappingJson)
+            .withValue("label", label)
+            .withValue("tableName", tableName)
+            .withValue("userId", userId);
+
+         List<QRecord> savedBulkLoadProfileList;
+         if(qRecord.getValueInteger("id") == null)
+         {
+            checkForDuplicates(userId, tableName, label, null);
+
+            InsertInput input = new InsertInput();
+            input.setTableName(SavedBulkLoadProfile.TABLE_NAME);
+            input.setRecords(List.of(qRecord));
+
+            InsertOutput output = new InsertAction().execute(input);
+            savedBulkLoadProfileList = output.getRecords();
+         }
+         else
+         {
+            checkForDuplicates(userId, tableName, label, qRecord.getValueInteger("id"));
+
+            UpdateInput input = new UpdateInput();
+            input.setTableName(SavedBulkLoadProfile.TABLE_NAME);
+            input.setRecords(List.of(qRecord));
+
+            UpdateOutput output = new UpdateAction().execute(input);
+            savedBulkLoadProfileList = output.getRecords();
+         }
+
+         runBackendStepOutput.addValue("savedBulkLoadProfileList", (Serializable) savedBulkLoadProfileList);
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Error storing saved bulkLoadProfile", e);
+         throw (e);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private String processMappingJson(String mappingJson)
+   {
+      return mappingJson;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void checkForDuplicates(String userId, String tableName, String label, Integer id) throws QException
+   {
+      QueryInput queryInput = new QueryInput();
+      queryInput.setTableName(SavedBulkLoadProfile.TABLE_NAME);
+      queryInput.setFilter(new QQueryFilter(
+         new QFilterCriteria("userId", QCriteriaOperator.EQUALS, userId),
+         new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, tableName),
+         new QFilterCriteria("label", QCriteriaOperator.EQUALS, label)));
+
+      if(id != null)
+      {
+         queryInput.getFilter().addCriteria(new QFilterCriteria("id", QCriteriaOperator.NOT_EQUALS, id));
+      }
+
+      QueryOutput queryOutput = new QueryAction().execute(queryInput);
+      if(CollectionUtils.nullSafeHasContents(queryOutput.getRecords()))
+      {
+         throw (new QUserFacingException("You already have a saved Bulk Load Profile on this table with this name."));
+      }
+   }
+}

qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/processes/implementations/scripts/StoreScriptRevisionProcessStep.java

@@ -101,7 +101,7 @@ public class StoreScriptRevisionProcessStep implements BackendStep
          ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
          // in case the app added a security field to the scripts table, make sure the user is allowed to edit the script //
          ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
-         ValidateRecordSecurityLockHelper.validateSecurityFields(QContext.getQInstance().getTable(Script.TABLE_NAME), List.of(script), ValidateRecordSecurityLockHelper.Action.UPDATE);
+         ValidateRecordSecurityLockHelper.validateSecurityFields(QContext.getQInstance().getTable(Script.TABLE_NAME), List.of(script), ValidateRecordSecurityLockHelper.Action.UPDATE, transaction);
          if(CollectionUtils.nullSafeHasContents(script.getErrors()))
          {
             throw (new QPermissionDeniedException(script.getErrors().get(0).getMessage()));

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/processes/RunProcessActionTest.java

@@ -23,10 +23,15 @@ package com.kingsrook.qqq.backend.core.actions.processes;
 
 
 import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
 import com.kingsrook.qqq.backend.core.BaseTest;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessState;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceLambda;
@@ -35,6 +40,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaD
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QStateMachineStep;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
+import com.kingsrook.qqq.backend.core.utils.collections.MultiLevelMapHelper;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import static org.assertj.core.api.Assertions.assertThat;
@@ -73,14 +80,14 @@ class RunProcessActionTest extends BaseTest
          /////////////////////////////////////////////////////////////////
          // two-steps - a, points at b; b has no next-step, so it exits //
          /////////////////////////////////////////////////////////////////
-         .addStep(QStateMachineStep.backendOnly("a", new QBackendStepMetaData().withName("aBackend")
+         .withStep(QStateMachineStep.backendOnly("a", new QBackendStepMetaData().withName("aBackend")
             .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
             {
                log.add("in StepA");
                runBackendStepOutput.getProcessState().setNextStepName("b");
             }))))
 
-         .addStep(QStateMachineStep.backendOnly("b", new QBackendStepMetaData().withName("bBackend")
+         .withStep(QStateMachineStep.backendOnly("b", new QBackendStepMetaData().withName("bBackend")
             .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
             {
                log.add("in StepB");
@@ -109,8 +116,8 @@ class RunProcessActionTest extends BaseTest
    {
       QProcessMetaData process = new QProcessMetaData().withName("test")
 
-         .addStep(QStateMachineStep.frontendOnly("a", new QFrontendStepMetaData().withName("aFrontend")).withDefaultNextStepName("b"))
-         .addStep(QStateMachineStep.frontendOnly("b", new QFrontendStepMetaData().withName("bFrontend")))
+         .withStep(QStateMachineStep.frontendOnly("a", new QFrontendStepMetaData().withName("aFrontend")).withDefaultNextStepName("b"))
+         .withStep(QStateMachineStep.frontendOnly("b", new QFrontendStepMetaData().withName("bFrontend")))
 
          .withStepFlow(ProcessStepFlow.STATE_MACHINE);
 
@@ -150,7 +157,7 @@ class RunProcessActionTest extends BaseTest
          // since it never goes to the frontend, it'll stack overflow //
          // (though we'll catch it ourselves before JVM does)         //
          ///////////////////////////////////////////////////////////////
-         .addStep(QStateMachineStep.backendOnly("a", new QBackendStepMetaData().withName("aBackend")
+         .withStep(QStateMachineStep.backendOnly("a", new QBackendStepMetaData().withName("aBackend")
             .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
             {
                log.add("in StepA");
@@ -193,14 +200,14 @@ class RunProcessActionTest extends BaseTest
          // since it never goes to the frontend, it'll stack overflow //
          // (though we'll catch it ourselves before JVM does)         //
          ///////////////////////////////////////////////////////////////
-         .addStep(QStateMachineStep.backendOnly("a", new QBackendStepMetaData().withName("aBackend")
+         .withStep(QStateMachineStep.backendOnly("a", new QBackendStepMetaData().withName("aBackend")
             .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
             {
                log.add("in StepA");
                runBackendStepOutput.getProcessState().setNextStepName("b");
             }))))
 
-         .addStep(QStateMachineStep.backendOnly("b", new QBackendStepMetaData().withName("bBackend")
+         .withStep(QStateMachineStep.backendOnly("b", new QBackendStepMetaData().withName("bBackend")
             .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
             {
                log.add("in StepB");
@@ -238,7 +245,7 @@ class RunProcessActionTest extends BaseTest
    {
       QProcessMetaData process = new QProcessMetaData().withName("test")
 
-         .addStep(QStateMachineStep.frontendThenBackend("a",
+         .withStep(QStateMachineStep.frontendThenBackend("a",
             new QFrontendStepMetaData().withName("aFrontend"),
             new QBackendStepMetaData().withName("aBackend")
                .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
@@ -247,7 +254,7 @@ class RunProcessActionTest extends BaseTest
                   runBackendStepOutput.getProcessState().setNextStepName("b");
                }))))
 
-         .addStep(QStateMachineStep.frontendThenBackend("b",
+         .withStep(QStateMachineStep.frontendThenBackend("b",
             new QFrontendStepMetaData().withName("bFrontend"),
             new QBackendStepMetaData().withName("bBackend")
                .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
@@ -256,7 +263,7 @@ class RunProcessActionTest extends BaseTest
                   runBackendStepOutput.getProcessState().setNextStepName("c");
                }))))
 
-         .addStep(QStateMachineStep.frontendThenBackend("c",
+         .withStep(QStateMachineStep.frontendThenBackend("c",
             new QFrontendStepMetaData().withName("cFrontend"),
             new QBackendStepMetaData().withName("cBackend")
                .withCode(new QCodeReferenceLambda<BackendStep>((runBackendStepInput, runBackendStepOutput) ->
@@ -265,7 +272,7 @@ class RunProcessActionTest extends BaseTest
                   runBackendStepOutput.getProcessState().setNextStepName("d");
                }))))
 
-         .addStep(QStateMachineStep.frontendOnly("d",
+         .withStep(QStateMachineStep.frontendOnly("d",
             new QFrontendStepMetaData().withName("dFrontend")))
 
          .withStepFlow(ProcessStepFlow.STATE_MACHINE);
@@ -321,7 +328,132 @@ class RunProcessActionTest extends BaseTest
       runProcessOutput = new RunProcessAction().execute(input);
       assertEquals(List.of("in StepA", "in StepB", "in StepC"), log);
       assertThat(runProcessOutput.getProcessState().getNextStepName()).isEmpty();
+   }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testGoingBack() throws QException
+   {
+      AtomicInteger        backCount     = new AtomicInteger(0);
+      Map<String, Integer> stepRunCounts = new HashMap<>();
+
+      BackendStep backendStep = (runBackendStepInput, runBackendStepOutput) ->
+      {
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         // shared backend-step lambda, that will do the same thing for both - but using step name to count how many times each is executed. //
+         //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+         MultiLevelMapHelper.getOrPutAndIncrement(stepRunCounts, runBackendStepInput.getStepName());
+         if(runBackendStepInput.getIsStepBack())
+         {
+            backCount.incrementAndGet();
+         }
+      };
+
+      ///////////////////////////////////////////////////////////
+      // normal flow here:  a -> b -> c                        //
+      // but, b can go back to a, as in: a -> b -> a -> b -> c //
+      ///////////////////////////////////////////////////////////
+      QProcessMetaData process = new QProcessMetaData().withName("test")
+         .withStep(new QBackendStepMetaData()
+            .withName("a")
+            .withCode(new QCodeReferenceLambda<>(backendStep)))
+         .withStep(new QFrontendStepMetaData()
+            .withName("b")
+            .withBackStepName("a"))
+         .withStep(new QBackendStepMetaData()
+            .withName("c")
+            .withCode(new QCodeReferenceLambda<>(backendStep)))
+         .withStepFlow(ProcessStepFlow.LINEAR);
+
+      QContext.getQInstance().addProcess(process);
+
+      RunProcessInput input = new RunProcessInput();
+      input.setProcessName("test");
+      input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK);
+
+      ///////////////////////////////////////////////////////////
+      // start the process - we should be sent to b (frontend) //
+      ///////////////////////////////////////////////////////////
+      RunProcessOutput runProcessOutput = new RunProcessAction().execute(input);
+      assertThat(runProcessOutput.getProcessState().getNextStepName())
+         .isPresent().get()
+         .isEqualTo("b");
+
+      assertEquals(0, backCount.get());
+      assertEquals(Map.of("a", 1), stepRunCounts);
+
+      ////////////////////////////////////////////////////////////////
+      // resume after b, but in back-mode - should end up back at b //
+      ////////////////////////////////////////////////////////////////
+      input.setStartAfterStep(null);
+      input.setStartAtStep("a");
+      runProcessOutput = new RunProcessAction().execute(input);
+      assertThat(runProcessOutput.getProcessState().getNextStepName())
+         .isPresent().get()
+         .isEqualTo("b");
+
+      assertEquals(1, backCount.get());
+      assertEquals(Map.of("a", 2), stepRunCounts);
+
+      ////////////////////////////////////////////////////////////////////////////
+      // resume after b, in regular (forward) mode - should wrap up the process //
+      ////////////////////////////////////////////////////////////////////////////
+      input.setStartAfterStep("b");
+      input.setStartAtStep(null);
+      runProcessOutput = new RunProcessAction().execute(input);
+      assertThat(runProcessOutput.getProcessState().getNextStepName())
+         .isEmpty();
+
+      assertEquals(1, backCount.get());
+      assertEquals(Map.of("a", 2, "c", 1), stepRunCounts);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testGetAvailableStepList() throws QException
+   {
+      QProcessMetaData process = new QProcessMetaData()
+         .withStep(new QBackendStepMetaData().withName("A"))
+         .withStep(new QBackendStepMetaData().withName("B"))
+         .withStep(new QBackendStepMetaData().withName("C"))
+         .withStep(new QBackendStepMetaData().withName("D"))
+         .withStep(new QBackendStepMetaData().withName("E"));
+
+      ProcessState processState = new ProcessState();
+      processState.setStepList(process.getStepList().stream().map(s -> s.getName()).toList());
+
+      assertStepListNames(List.of("A", "B", "C", "D", "E"), RunProcessAction.getAvailableStepList(processState, process, null, false));
+      assertStepListNames(List.of("A", "B", "C", "D", "E"), RunProcessAction.getAvailableStepList(processState, process, null, true));
+
+      assertStepListNames(List.of("B", "C", "D", "E"), RunProcessAction.getAvailableStepList(processState, process, "A", false));
+      assertStepListNames(List.of("A", "B", "C", "D", "E"), RunProcessAction.getAvailableStepList(processState, process, "A", true));
+
+      assertStepListNames(List.of("D", "E"), RunProcessAction.getAvailableStepList(processState, process, "C", false));
+      assertStepListNames(List.of("C", "D", "E"), RunProcessAction.getAvailableStepList(processState, process, "C", true));
+
+      assertStepListNames(Collections.emptyList(), RunProcessAction.getAvailableStepList(processState, process, "E", false));
+      assertStepListNames(List.of("E"), RunProcessAction.getAvailableStepList(processState, process, "E", true));
+
+      assertStepListNames(Collections.emptyList(), RunProcessAction.getAvailableStepList(processState, process, "Z", false));
+      assertStepListNames(Collections.emptyList(), RunProcessAction.getAvailableStepList(processState, process, "Z", true));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void assertStepListNames(List<String> expectedNames, List<QStepMetaData> actualSteps)
+   {
+      assertEquals(expectedNames, actualSteps.stream().map(s -> s.getName()).toList());
    }
 
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/reporting/GenerateReportActionTest.java

@@ -90,7 +90,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
  *******************************************************************************/
 public class GenerateReportActionTest extends BaseTest
 {
-   private static final String REPORT_NAME = "personReport1";
+   public static final String REPORT_NAME = "personReport1";
 
 
 
@@ -655,7 +655,7 @@ public class GenerateReportActionTest extends BaseTest
       Iterator<Map<String, String>> iterator = list.iterator();
       Map<String, String>           row      = iterator.next();
       assertEquals(5, list.size());
-      assertThat(row).containsOnlyKeys("Id", "First Name", "Last Name");
+      assertThat(row).containsOnlyKeys("Id", "First Name", "Last Name", "Birth Date");
    }
 
 
@@ -663,7 +663,7 @@ public class GenerateReportActionTest extends BaseTest
    /*******************************************************************************
     **
     *******************************************************************************/
-   private static QReportMetaData defineTableOnlyReport()
+   public static QReportMetaData defineTableOnlyReport()
    {
       QReportMetaData report = new QReportMetaData()
          .withName(REPORT_NAME)
@@ -686,7 +686,9 @@ public class GenerateReportActionTest extends BaseTest
                .withColumns(List.of(
                   new QReportField().withName("id"),
                   new QReportField().withName("firstName"),
-                  new QReportField().withName("lastName")))));
+                  new QReportField().withName("lastName"),
+                  new QReportField().withName("birthDate")
+               ))));
 
       return report;
    }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/actions/values/SearchPossibleValueSourceActionTest.java

@@ -217,6 +217,63 @@ class SearchPossibleValueSourceActionTest extends BaseTest
    }
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSearchPvsAction_tableByLabels() throws QException
+   {
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of("Square", "Circle"), TestUtils.POSSIBLE_VALUE_SOURCE_SHAPE);
+         assertEquals(2, output.getResults().size());
+         assertThat(output.getResults()).anyMatch(pv -> pv.getId().equals(2) && pv.getLabel().equals("Square"));
+         assertThat(output.getResults()).anyMatch(pv -> pv.getId().equals(3) && pv.getLabel().equals("Circle"));
+      }
+
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of(), TestUtils.POSSIBLE_VALUE_SOURCE_SHAPE);
+         assertEquals(0, output.getResults().size());
+      }
+
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of("notFound"), TestUtils.POSSIBLE_VALUE_SOURCE_SHAPE);
+         assertEquals(0, output.getResults().size());
+      }
+   }
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSearchPvsAction_enumByLabel() throws QException
+   {
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of("IL", "MO", "XX"), TestUtils.POSSIBLE_VALUE_SOURCE_STATE);
+         assertEquals(2, output.getResults().size());
+         assertThat(output.getResults()).anyMatch(pv -> pv.getId().equals(1) && pv.getLabel().equals("IL"));
+         assertThat(output.getResults()).anyMatch(pv -> pv.getId().equals(2) && pv.getLabel().equals("MO"));
+      }
+
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of("Il", "mo", "XX"), TestUtils.POSSIBLE_VALUE_SOURCE_STATE);
+         assertEquals(2, output.getResults().size());
+         assertThat(output.getResults()).anyMatch(pv -> pv.getId().equals(1) && pv.getLabel().equals("IL"));
+         assertThat(output.getResults()).anyMatch(pv -> pv.getId().equals(2) && pv.getLabel().equals("MO"));
+      }
+
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of(), TestUtils.POSSIBLE_VALUE_SOURCE_STATE);
+         assertEquals(0, output.getResults().size());
+      }
+
+      {
+         SearchPossibleValueSourceOutput output = getSearchPossibleValueSourceOutputByLabels(List.of("not-found"), TestUtils.POSSIBLE_VALUE_SOURCE_STATE);
+         assertEquals(0, output.getResults().size());
+      }
+   }
+
+
 
    /*******************************************************************************
     **
@@ -414,4 +471,18 @@ class SearchPossibleValueSourceActionTest extends BaseTest
       return output;
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private SearchPossibleValueSourceOutput getSearchPossibleValueSourceOutputByLabels(List<String> labels, String possibleValueSourceName) throws QException
+   {
+      SearchPossibleValueSourceInput input = new SearchPossibleValueSourceInput();
+      input.setLabelList(labels);
+      input.setPossibleValueSourceName(possibleValueSourceName);
+      SearchPossibleValueSourceOutput output = new SearchPossibleValueSourceAction().execute(input);
+      return output;
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/actions/processes/ProcessSummaryAssert.java

@@ -0,0 +1,208 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.actions.processes;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import org.assertj.core.api.AbstractAssert;
+import org.assertj.core.api.Assertions;
+import static org.junit.jupiter.api.Assertions.fail;
+
+
+/*******************************************************************************
+ ** AssertJ assert class for ProcessSummary - that is - a list of ProcessSummaryLineInterface's
+ *******************************************************************************/
+public class ProcessSummaryAssert extends AbstractAssert<ProcessSummaryAssert, List<ProcessSummaryLineInterface>>
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected ProcessSummaryAssert(List<ProcessSummaryLineInterface> actual, Class<?> selfType)
+   {
+      super(actual, selfType);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   public static ProcessSummaryAssert assertThat(RunProcessOutput runProcessOutput)
+   {
+      List<ProcessSummaryLineInterface> processResults = (List<ProcessSummaryLineInterface>) runProcessOutput.getValue("processResults");
+      if(processResults == null)
+      {
+         processResults = (List<ProcessSummaryLineInterface>) runProcessOutput.getValue("validationSummary");
+      }
+
+      return (new ProcessSummaryAssert(processResults, ProcessSummaryAssert.class));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   public static ProcessSummaryAssert assertThat(RunBackendStepOutput runBackendStepOutput)
+   {
+      List<ProcessSummaryLineInterface> processResults = (List<ProcessSummaryLineInterface>) runBackendStepOutput.getValue("processResults");
+      if(processResults == null)
+      {
+         processResults = (List<ProcessSummaryLineInterface>) runBackendStepOutput.getValue("validationSummary");
+      }
+
+      if(processResults == null)
+      {
+         fail("Could not find process results in backend step output.");
+      }
+
+      return (new ProcessSummaryAssert(processResults, ProcessSummaryAssert.class));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static ProcessSummaryAssert assertThat(List<ProcessSummaryLineInterface> actual)
+   {
+      return (new ProcessSummaryAssert(actual, ProcessSummaryAssert.class));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryAssert hasSize(int expectedSize)
+   {
+      Assertions.assertThat(actual).hasSize(expectedSize);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasLineWithMessageMatching(String regExp)
+   {
+      List<String> foundMessages = new ArrayList<>();
+      for(ProcessSummaryLineInterface processSummaryLineInterface : actual)
+      {
+         if(processSummaryLineInterface.getMessage() == null)
+         {
+            processSummaryLineInterface.prepareForFrontend(false);
+         }
+
+         if(processSummaryLineInterface.getMessage() != null && processSummaryLineInterface.getMessage().matches(regExp))
+         {
+            return (new ProcessSummaryLineInterfaceAssert(processSummaryLineInterface, ProcessSummaryLineInterfaceAssert.class));
+         }
+         else
+         {
+            foundMessages.add(processSummaryLineInterface.getMessage());
+         }
+      }
+
+      failWithMessage("Failed to find a ProcessSummaryLine with message matching [" + regExp + "].\nFound messages were:\n" + StringUtils.join("\n", foundMessages));
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasLineWithMessageContaining(String substr)
+   {
+      List<String> foundMessages = new ArrayList<>();
+      for(ProcessSummaryLineInterface processSummaryLineInterface : actual)
+      {
+         if(processSummaryLineInterface.getMessage() == null)
+         {
+            processSummaryLineInterface.prepareForFrontend(false);
+         }
+
+         if(processSummaryLineInterface.getMessage() != null && processSummaryLineInterface.getMessage().contains(substr))
+         {
+            return (new ProcessSummaryLineInterfaceAssert(processSummaryLineInterface, ProcessSummaryLineInterfaceAssert.class));
+         }
+         else
+         {
+            foundMessages.add(processSummaryLineInterface.getMessage());
+         }
+      }
+
+      failWithMessage("Failed to find a ProcessSummaryLine with message containing [" + substr + "].\nFound messages were:\n" + StringUtils.join("\n", foundMessages));
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasLineWithStatus(Status status)
+   {
+      List<String> foundStatuses = new ArrayList<>();
+      for(ProcessSummaryLineInterface processSummaryLineInterface : actual)
+      {
+         if(status.equals(processSummaryLineInterface.getStatus()))
+         {
+            return (new ProcessSummaryLineInterfaceAssert(processSummaryLineInterface, ProcessSummaryLineInterfaceAssert.class));
+         }
+         else
+         {
+            foundStatuses.add(String.valueOf(processSummaryLineInterface.getStatus()));
+         }
+      }
+
+      failWithMessage("Failed to find a ProcessSummaryLine with status [" + status + "].\nFound statuses were:\n" + StringUtils.join("\n", foundStatuses));
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryAssert hasNoLineWithStatus(Status status)
+   {
+      for(ProcessSummaryLineInterface processSummaryLineInterface : actual)
+      {
+         if(status.equals(processSummaryLineInterface.getStatus()))
+         {
+            failWithMessage("Found a ProcessSummaryLine with status [" + status + "], which was not supposed to happen.");
+            return (null);
+         }
+      }
+
+      return (this);
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/actions/processes/ProcessSummaryLineInterfaceAssert.java

@@ -0,0 +1,188 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.model.actions.processes;
+
+
+import org.assertj.core.api.AbstractAssert;
+import org.assertj.core.api.Assertions;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** AssertJ assert class for ProcessSummaryLine.
+ *******************************************************************************/
+public class ProcessSummaryLineInterfaceAssert extends AbstractAssert<ProcessSummaryLineInterfaceAssert, ProcessSummaryLineInterface>
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   protected ProcessSummaryLineInterfaceAssert(ProcessSummaryLineInterface actual, Class<?> selfType)
+   {
+      super(actual, selfType);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static ProcessSummaryLineInterfaceAssert assertThat(ProcessSummaryLineInterface actual)
+   {
+      return (new ProcessSummaryLineInterfaceAssert(actual, ProcessSummaryLineInterfaceAssert.class));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasCount(Integer count)
+   {
+      if(actual instanceof ProcessSummaryLine psl)
+      {
+         assertEquals(count, psl.getCount(), "Expected count in process summary line");
+      }
+      else
+      {
+         failWithMessage("ProcessSummaryLineInterface is not of concrete type ProcessSummaryLine (is: " + actual.getClass().getSimpleName() + ")");
+      }
+
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasStatus(Status status)
+   {
+      assertEquals(status, actual.getStatus(), "Expected status in process summary line");
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasMessageMatching(String regExp)
+   {
+      if(actual.getMessage() == null)
+      {
+         actual.prepareForFrontend(false);
+      }
+
+      Assertions.assertThat(actual.getMessage()).matches(regExp);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasMessageContaining(String substring)
+   {
+      if(actual.getMessage() == null)
+      {
+         actual.prepareForFrontend(false);
+      }
+
+      Assertions.assertThat(actual.getMessage()).contains(substring);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert doesNotHaveMessageMatching(String regExp)
+   {
+      if(actual.getMessage() == null)
+      {
+         actual.prepareForFrontend(false);
+      }
+
+      Assertions.assertThat(actual.getMessage()).doesNotMatch(regExp);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert doesNotHaveMessageContaining(String substring)
+   {
+      if(actual.getMessage() == null)
+      {
+         actual.prepareForFrontend(false);
+      }
+
+      Assertions.assertThat(actual.getMessage()).doesNotContain(substring);
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert hasAnyBulletsOfTextContaining(String substring)
+   {
+      if(actual instanceof ProcessSummaryLine psl)
+      {
+         Assertions.assertThat(psl.getBulletsOfText())
+            .isNotNull()
+            .anyMatch(s -> s.contains(substring));
+      }
+      else
+      {
+         Assertions.fail("Process Summary Line was not the expected type.");
+      }
+
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public ProcessSummaryLineInterfaceAssert doesNotHaveAnyBulletsOfTextContaining(String substring)
+   {
+      if(actual instanceof ProcessSummaryLine psl)
+      {
+         if(psl.getBulletsOfText() != null)
+         {
+            Assertions.assertThat(psl.getBulletsOfText())
+               .noneMatch(s -> s.contains(substring));
+         }
+      }
+
+      return (this);
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/model/session/QSessionTest.java

@@ -76,7 +76,7 @@ class QSessionTest extends BaseTest
    void testMixedValueTypes()
    {
       QSession session = new QSession().withSecurityKeyValues(Map.of(
-         "storeId", List.of("100", "200", 300)
+         "storeId", List.of("100", "200", 300, "four-hundred")
       ));
 
       for(int i : List.of(100, 200, 300))
@@ -86,6 +86,18 @@ class QSessionTest extends BaseTest
          assertTrue(session.hasSecurityKeyValue("storeId", i, QFieldType.STRING), "Should contain: " + i);
          assertTrue(session.hasSecurityKeyValue("storeId", String.valueOf(i), QFieldType.STRING), "Should contain: " + i);
       }
+
+      ////////////////////////////////////////////////////////////////////////////
+      // next two blocks - used to throw exceptions - now, gracefully be false. //
+      ////////////////////////////////////////////////////////////////////////////
+      int i = 400;
+      assertFalse(session.hasSecurityKeyValue("storeId", i, QFieldType.INTEGER), "Should not contain: " + i);
+      assertFalse(session.hasSecurityKeyValue("storeId", String.valueOf(i), QFieldType.INTEGER), "Should not contain: " + i);
+      assertFalse(session.hasSecurityKeyValue("storeId", i, QFieldType.STRING), "Should not contain: " + i);
+      assertFalse(session.hasSecurityKeyValue("storeId", String.valueOf(i), QFieldType.STRING), "Should not contain: " + i);
+
+      assertFalse(session.hasSecurityKeyValue("storeId", "one-hundred", QFieldType.INTEGER), "Should not contain: " + i);
+      assertFalse(session.hasSecurityKeyValue("storeId", "one-hundred", QFieldType.STRING), "Should not contain: " + i);
    }
 
 

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertFullProcessTest.java

@@ -0,0 +1,252 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2022.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.function.Consumer;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
+import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.storage.StorageInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendFieldMetaData;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+
+/*******************************************************************************
+ ** Unit test for full bulk insert process
+ *******************************************************************************/
+class BulkInsertFullProcessTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @BeforeEach
+   @AfterEach
+   void beforeAndAfterEach()
+   {
+      MemoryRecordStore.getInstance().reset();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String getPersonCsvRow1()
+   {
+      return ("""
+         "0","2021-10-26 14:39:37","2021-10-26 14:39:37","John","Doe","1980-01-01","john@doe.com","Missouri",42
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String getPersonCsvRow2()
+   {
+      return ("""
+         "0","2021-10-26 14:39:37","2021-10-26 14:39:37","Jane","Doe","1981-01-01","john@doe.com","Illinois",
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static String getPersonCsvHeaderUsingLabels()
+   {
+      return ("""
+         "Id","Create Date","Modify Date","First Name","Last Name","Birth Date","Email","Home State",noOfShoes
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws Exception
+   {
+      String defaultEmail = "noone@kingsrook.com";
+
+      ///////////////////////////////////////
+      // make sure table is empty to start //
+      ///////////////////////////////////////
+      assertThat(TestUtils.queryTable(TestUtils.TABLE_NAME_PERSON_MEMORY)).isEmpty();
+
+      QInstance qInstance   = QContext.getQInstance();
+      String    processName = "PersonBulkInsertV2";
+      new QInstanceEnricher(qInstance).defineTableBulkInsert(qInstance, qInstance.getTable(TestUtils.TABLE_NAME_PERSON_MEMORY), processName);
+
+      /////////////////////////////////////////////////////////
+      // start the process - expect to go to the upload step //
+      /////////////////////////////////////////////////////////
+      RunProcessInput runProcessInput = new RunProcessInput();
+      runProcessInput.setProcessName(processName);
+      runProcessInput.addValue("tableName", TestUtils.TABLE_NAME_PERSON_MEMORY);
+      RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+      String           processUUID      = runProcessOutput.getProcessUUID();
+      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("upload");
+
+      //////////////////////////////
+      // simulate the file upload //
+      //////////////////////////////
+      String       storageReference = UUID.randomUUID() + ".csv";
+      StorageInput storageInput     = new StorageInput(TestUtils.TABLE_NAME_MEMORY_STORAGE).withReference(storageReference);
+      try(OutputStream outputStream = new StorageAction().createOutputStream(storageInput))
+      {
+         outputStream.write((getPersonCsvHeaderUsingLabels() + getPersonCsvRow1() + getPersonCsvRow2()).getBytes());
+      }
+      catch(IOException e)
+      {
+         throw (e);
+      }
+
+      //////////////////////////
+      // continue post-upload //
+      //////////////////////////
+      runProcessInput.setProcessUUID(processUUID);
+      runProcessInput.setStartAfterStep("upload");
+      runProcessInput.addValue("theFile", new ArrayList<>(List.of(storageInput)));
+      runProcessOutput = new RunProcessAction().execute(runProcessInput);
+      assertEquals(List.of("Id", "Create Date", "Modify Date", "First Name", "Last Name", "Birth Date", "Email", "Home State", "noOfShoes"), runProcessOutput.getValue("headerValues"));
+      assertEquals(List.of("A", "B", "C", "D", "E", "F", "G", "H", "I"), runProcessOutput.getValue("headerLetters"));
+
+      //////////////////////////////////////////////////////
+      // assert about the suggested mapping that was done //
+      //////////////////////////////////////////////////////
+      Serializable bulkLoadProfile = runProcessOutput.getValue("bulkLoadProfile");
+      assertThat(bulkLoadProfile).isInstanceOf(BulkLoadProfile.class);
+      assertThat(((BulkLoadProfile) bulkLoadProfile).getFieldList()).hasSizeGreaterThan(5);
+      assertEquals("firstName", ((BulkLoadProfile) bulkLoadProfile).getFieldList().get(0).getFieldName());
+      assertEquals(3, ((BulkLoadProfile) bulkLoadProfile).getFieldList().get(0).getColumnIndex());
+      assertEquals("lastName", ((BulkLoadProfile) bulkLoadProfile).getFieldList().get(1).getFieldName());
+      assertEquals(4, ((BulkLoadProfile) bulkLoadProfile).getFieldList().get(1).getColumnIndex());
+      assertEquals("birthDate", ((BulkLoadProfile) bulkLoadProfile).getFieldList().get(2).getFieldName());
+      assertEquals(5, ((BulkLoadProfile) bulkLoadProfile).getFieldList().get(2).getColumnIndex());
+
+      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("fileMapping");
+
+      ////////////////////////////////////////////////////////////////////////////////
+      // all subsequent steps will want these data - so set up a lambda to set them //
+      ////////////////////////////////////////////////////////////////////////////////
+      Consumer<RunProcessInput> addProfileToRunProcessInput = (RunProcessInput input) ->
+      {
+         input.addValue("version", "v1");
+         input.addValue("layout", "FLAT");
+         input.addValue("hasHeaderRow", "true");
+         input.addValue("fieldListJSON", JsonUtils.toJson(List.of(
+            new BulkLoadProfileField().withFieldName("firstName").withColumnIndex(3),
+            new BulkLoadProfileField().withFieldName("lastName").withColumnIndex(4),
+            new BulkLoadProfileField().withFieldName("email").withDefaultValue(defaultEmail),
+            new BulkLoadProfileField().withFieldName("homeStateId").withColumnIndex(7).withDoValueMapping(true).withValueMappings(Map.of("Illinois", 1)),
+            new BulkLoadProfileField().withFieldName("noOfShoes").withColumnIndex(8)
+         )));
+      };
+
+      ////////////////////////////////
+      // continue post file-mapping //
+      ////////////////////////////////
+      runProcessInput.setStartAfterStep("fileMapping");
+      addProfileToRunProcessInput.accept(runProcessInput);
+      runProcessOutput = new RunProcessAction().execute(runProcessInput);
+      Serializable valueMappingField = runProcessOutput.getValue("valueMappingField");
+      assertThat(valueMappingField).isInstanceOf(QFrontendFieldMetaData.class);
+      assertEquals("homeStateId", ((QFrontendFieldMetaData) valueMappingField).getName());
+      assertEquals(List.of("Missouri", "Illinois"), runProcessOutput.getValue("fileValues"));
+      assertEquals(List.of("homeStateId"), runProcessOutput.getValue("fieldNamesToDoValueMapping"));
+      assertEquals(Map.of(1, "IL"), runProcessOutput.getValue("mappedValueLabels"));
+      assertEquals(0, runProcessOutput.getValue("valueMappingFieldIndex"));
+      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("valueMapping");
+
+      /////////////////////////////////
+      // continue post value-mapping //
+      /////////////////////////////////
+      runProcessInput.setStartAfterStep("valueMapping");
+      runProcessInput.addValue("mappedValuesJSON", JsonUtils.toJson(Map.of("Illinois", 1, "Missouri", 2)));
+      addProfileToRunProcessInput.accept(runProcessInput);
+      runProcessOutput = new RunProcessAction().execute(runProcessInput);
+      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("review");
+
+      /////////////////////////////////
+      // continue post review screen //
+      /////////////////////////////////
+      runProcessInput.setStartAfterStep("review");
+      addProfileToRunProcessInput.accept(runProcessInput);
+      runProcessOutput = new RunProcessAction().execute(runProcessInput);
+      assertThat(runProcessOutput.getRecords()).hasSize(2);
+      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("result");
+      assertThat(runProcessOutput.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY)).isNotNull().isInstanceOf(List.class);
+      assertThat(runProcessOutput.getException()).isEmpty();
+
+      ////////////////////////////////////
+      // query for the inserted records //
+      ////////////////////////////////////
+      List<QRecord> records = TestUtils.queryTable(TestUtils.TABLE_NAME_PERSON_MEMORY);
+
+      assertEquals("John", records.get(0).getValueString("firstName"));
+      assertEquals("Jane", records.get(1).getValueString("firstName"));
+
+      assertNotNull(records.get(0).getValue("id"));
+      assertNotNull(records.get(1).getValue("id"));
+
+      assertEquals(2, records.get(0).getValueInteger("homeStateId"));
+      assertEquals(1, records.get(1).getValueInteger("homeStateId"));
+
+      assertEquals(defaultEmail, records.get(0).getValueString("email"));
+      assertEquals(defaultEmail, records.get(1).getValueString("email"));
+
+      assertEquals(42, records.get(0).getValueInteger("noOfShoes"));
+      assertNull(records.get(1).getValue("noOfShoes"));
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertPrepareFileMappingStepTest.java

@@ -0,0 +1,69 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for BulkInsertPrepareMappingStep 
+ *******************************************************************************/
+class BulkInsertPrepareFileMappingStepTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("PointlessArithmeticExpression")
+   @Test
+   void testToHeaderLetter()
+   {
+      assertEquals("A", BulkInsertPrepareFileMappingStep.toHeaderLetter(0));
+      assertEquals("B", BulkInsertPrepareFileMappingStep.toHeaderLetter(1));
+      assertEquals("Z", BulkInsertPrepareFileMappingStep.toHeaderLetter(25));
+
+      assertEquals("AA", BulkInsertPrepareFileMappingStep.toHeaderLetter(26 + 0));
+      assertEquals("AB", BulkInsertPrepareFileMappingStep.toHeaderLetter(26 + 1));
+      assertEquals("AZ", BulkInsertPrepareFileMappingStep.toHeaderLetter(26 + 25));
+
+      assertEquals("BA", BulkInsertPrepareFileMappingStep.toHeaderLetter(2 * 26 + 0));
+      assertEquals("BB", BulkInsertPrepareFileMappingStep.toHeaderLetter(2 * 26 + 1));
+      assertEquals("BZ", BulkInsertPrepareFileMappingStep.toHeaderLetter(2 * 26 + 25));
+
+      assertEquals("ZA", BulkInsertPrepareFileMappingStep.toHeaderLetter(26 * 26 + 0));
+      assertEquals("ZB", BulkInsertPrepareFileMappingStep.toHeaderLetter(26 * 26 + 1));
+      assertEquals("ZZ", BulkInsertPrepareFileMappingStep.toHeaderLetter(26 * 26 + 25));
+
+      assertEquals("AAA", BulkInsertPrepareFileMappingStep.toHeaderLetter(27 * 26 + 0));
+      assertEquals("AAB", BulkInsertPrepareFileMappingStep.toHeaderLetter(27 * 26 + 1));
+      assertEquals("AAC", BulkInsertPrepareFileMappingStep.toHeaderLetter(27 * 26 + 2));
+
+      assertEquals("ABA", BulkInsertPrepareFileMappingStep.toHeaderLetter(28 * 26 + 0));
+      assertEquals("ABB", BulkInsertPrepareFileMappingStep.toHeaderLetter(28 * 26 + 1));
+
+      assertEquals("BAA", BulkInsertPrepareFileMappingStep.toHeaderLetter(2 * 26 * 26 + 26 + 0));
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertPrepareValueMappingStepTest.java

@@ -0,0 +1,55 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for BulkInsertPrepareValueMappingStep 
+ *******************************************************************************/
+class BulkInsertPrepareValueMappingStepTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      assertEquals(TestUtils.TABLE_NAME_ORDER, BulkInsertPrepareValueMappingStep.getTableAndField(TestUtils.TABLE_NAME_ORDER, "orderNo").table().getName());
+      assertEquals("orderNo", BulkInsertPrepareValueMappingStep.getTableAndField(TestUtils.TABLE_NAME_ORDER, "orderNo").field().getName());
+
+      assertEquals(TestUtils.TABLE_NAME_LINE_ITEM, BulkInsertPrepareValueMappingStep.getTableAndField(TestUtils.TABLE_NAME_ORDER, "orderLine.sku").table().getName());
+      assertEquals("sku", BulkInsertPrepareValueMappingStep.getTableAndField(TestUtils.TABLE_NAME_ORDER, "orderLine.sku").field().getName());
+
+      assertEquals(TestUtils.TABLE_NAME_LINE_ITEM_EXTRINSIC, BulkInsertPrepareValueMappingStep.getTableAndField(TestUtils.TABLE_NAME_ORDER, "orderLine.extrinsics.key").table().getName());
+      assertEquals("key", BulkInsertPrepareValueMappingStep.getTableAndField(TestUtils.TABLE_NAME_ORDER, "orderLine.extrinsics.key").field().getName());
+
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertTest.java

@@ -1,157 +0,0 @@
-/*
- * QQQ - Low-code Application Framework for Engineers.
- * Copyright (C) 2021-2022.  Kingsrook, LLC
- * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
- * contact@kingsrook.com
- * https://github.com/Kingsrook/
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Affero General Public License as
- * published by the Free Software Foundation, either version 3 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Affero General Public License for more details.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program.  If not, see <https://www.gnu.org/licenses/>.
- */
-
-package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
-
-
-import java.util.List;
-import com.kingsrook.qqq.backend.core.BaseTest;
-import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
-import com.kingsrook.qqq.backend.core.exceptions.QException;
-import com.kingsrook.qqq.backend.core.model.actions.processes.QUploadedFile;
-import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
-import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
-import com.kingsrook.qqq.backend.core.model.data.QRecord;
-import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
-import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
-import com.kingsrook.qqq.backend.core.state.StateType;
-import com.kingsrook.qqq.backend.core.state.TempFileStateProvider;
-import com.kingsrook.qqq.backend.core.state.UUIDAndTypeStateKey;
-import com.kingsrook.qqq.backend.core.utils.TestUtils;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import static org.assertj.core.api.Assertions.assertThat;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-
-/*******************************************************************************
- ** Unit test for full bulk insert process
- *******************************************************************************/
-class BulkInsertTest extends BaseTest
-{
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   @BeforeEach
-   @AfterEach
-   void beforeAndAfterEach()
-   {
-      MemoryRecordStore.getInstance().reset();
-   }
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   public static String getPersonCsvRow1()
-   {
-      return ("""
-         "0","2021-10-26 14:39:37","2021-10-26 14:39:37","John","Doe","1980-01-01","john@doe.com"
-         """);
-   }
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   public static String getPersonCsvRow2()
-   {
-      return ("""
-         "0","2021-10-26 14:39:37","2021-10-26 14:39:37","Jane","Doe","1981-01-01","john@doe.com"
-         """);
-   }
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   public static String getPersonCsvHeaderUsingLabels()
-   {
-      return ("""
-         "Id","Create Date","Modify Date","First Name","Last Name","Birth Date","Email"
-         """);
-   }
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   @Test
-   void test() throws QException
-   {
-      ///////////////////////////////////////
-      // make sure table is empty to start //
-      ///////////////////////////////////////
-      assertThat(TestUtils.queryTable(TestUtils.TABLE_NAME_PERSON_MEMORY)).isEmpty();
-
-      ////////////////////////////////////////////////////////////////
-      // create an uploaded file, similar to how an http server may //
-      ////////////////////////////////////////////////////////////////
-      QUploadedFile qUploadedFile = new QUploadedFile();
-      qUploadedFile.setBytes((getPersonCsvHeaderUsingLabels() + getPersonCsvRow1() + getPersonCsvRow2()).getBytes());
-      qUploadedFile.setFilename("test.csv");
-      UUIDAndTypeStateKey uploadedFileKey = new UUIDAndTypeStateKey(StateType.UPLOADED_FILE);
-      TempFileStateProvider.getInstance().put(uploadedFileKey, qUploadedFile);
-
-      RunProcessInput runProcessInput = new RunProcessInput();
-      runProcessInput.setProcessName(TestUtils.TABLE_NAME_PERSON_MEMORY + ".bulkInsert");
-      RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
-      String           processUUID      = runProcessOutput.getProcessUUID();
-
-      runProcessInput.setProcessUUID(processUUID);
-      runProcessInput.setStartAfterStep("upload");
-      runProcessInput.addValue(QUploadedFile.DEFAULT_UPLOADED_FILE_FIELD_NAME, uploadedFileKey);
-      runProcessOutput = new RunProcessAction().execute(runProcessInput);
-      assertThat(runProcessOutput.getRecords()).hasSize(2);
-      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("review");
-
-      runProcessInput.addValue(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
-      runProcessInput.setStartAfterStep("review");
-      runProcessOutput = new RunProcessAction().execute(runProcessInput);
-      assertThat(runProcessOutput.getRecords()).hasSize(2);
-      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("review");
-      assertThat(runProcessOutput.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY)).isNotNull().isInstanceOf(List.class);
-
-      runProcessInput.setStartAfterStep("review");
-      runProcessOutput = new RunProcessAction().execute(runProcessInput);
-      assertThat(runProcessOutput.getRecords()).hasSize(2);
-      assertThat(runProcessOutput.getProcessState().getNextStepName()).isPresent().get().isEqualTo("result");
-      assertThat(runProcessOutput.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY)).isNotNull().isInstanceOf(List.class);
-      assertThat(runProcessOutput.getException()).isEmpty();
-
-      ////////////////////////////////////
-      // query for the inserted records //
-      ////////////////////////////////////
-      List<QRecord> records = TestUtils.queryTable(TestUtils.TABLE_NAME_PERSON_MEMORY);
-      assertEquals("John", records.get(0).getValueString("firstName"));
-      assertEquals("Jane", records.get(1).getValueString("firstName"));
-      assertNotNull(records.get(0).getValue("id"));
-      assertNotNull(records.get(1).getValue("id"));
-   }
-
-}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/BulkInsertTransformStepTest.java

@@ -22,10 +22,14 @@
 package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert;
 
 
+import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.stream.Collectors;
 import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryAssert;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
@@ -38,8 +42,12 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
 import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.BulkLoadRecordUtils;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping.BulkLoadValueTypeError;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
 import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import com.kingsrook.qqq.backend.core.utils.collections.ListBuilder;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -87,9 +95,9 @@ class BulkInsertTransformStepTest extends BaseTest
       // insert some records that will cause some UK violations //
       ////////////////////////////////////////////////////////////
       TestUtils.insertRecords(table, List.of(
-         newQRecord("uuid-A", "SKU-1", 1),
-         newQRecord("uuid-B", "SKU-2", 1),
-         newQRecord("uuid-C", "SKU-2", 2)
+         newUkTestQRecord("uuid-A", "SKU-1", 1),
+         newUkTestQRecord("uuid-B", "SKU-2", 1),
+         newUkTestQRecord("uuid-C", "SKU-2", 2)
       ));
 
       ///////////////////////////////////////////
@@ -102,13 +110,13 @@ class BulkInsertTransformStepTest extends BaseTest
       input.setTableName(TABLE_NAME);
       input.setStepName(StreamedETLWithFrontendProcess.STEP_NAME_VALIDATE);
       input.setRecords(List.of(
-         newQRecord("uuid-1", "SKU-A", 1), // OK.
-         newQRecord("uuid-1", "SKU-B", 1), // violate uuid UK in this set
-         newQRecord("uuid-2", "SKU-C", 1), // OK.
-         newQRecord("uuid-3", "SKU-C", 2), // OK.
-         newQRecord("uuid-4", "SKU-C", 1), // violate sku/storeId UK in this set
-         newQRecord("uuid-A", "SKU-X", 1), // violate uuid UK from pre-existing records
-         newQRecord("uuid-D", "SKU-2", 1)  // violate sku/storeId UK from pre-existing records
+         newUkTestQRecord("uuid-1", "SKU-A", 1), // OK.
+         newUkTestQRecord("uuid-1", "SKU-B", 1), // violate uuid UK in this set
+         newUkTestQRecord("uuid-2", "SKU-C", 1), // OK.
+         newUkTestQRecord("uuid-3", "SKU-C", 2), // OK.
+         newUkTestQRecord("uuid-4", "SKU-C", 1), // violate sku/storeId UK in this set
+         newUkTestQRecord("uuid-A", "SKU-X", 1), // violate uuid UK from pre-existing records
+         newUkTestQRecord("uuid-D", "SKU-2", 1)  // violate sku/storeId UK from pre-existing records
       ));
       bulkInsertTransformStep.preRun(input, output);
       bulkInsertTransformStep.runOnePage(input, output);
@@ -171,9 +179,9 @@ class BulkInsertTransformStepTest extends BaseTest
       // insert some records that will cause some UK violations //
       ////////////////////////////////////////////////////////////
       TestUtils.insertRecords(table, List.of(
-         newQRecord("uuid-A", "SKU-1", 1),
-         newQRecord("uuid-B", "SKU-2", 1),
-         newQRecord("uuid-C", "SKU-2", 2)
+         newUkTestQRecord("uuid-A", "SKU-1", 1),
+         newUkTestQRecord("uuid-B", "SKU-2", 1),
+         newUkTestQRecord("uuid-C", "SKU-2", 2)
       ));
 
       ///////////////////////////////////////////
@@ -186,20 +194,20 @@ class BulkInsertTransformStepTest extends BaseTest
       input.setTableName(TABLE_NAME);
       input.setStepName(StreamedETLWithFrontendProcess.STEP_NAME_VALIDATE);
       input.setRecords(List.of(
-         newQRecord("uuid-1", "SKU-A", 1), // OK.
-         newQRecord("uuid-1", "SKU-B", 1), // violate uuid UK in this set
-         newQRecord("uuid-2", "SKU-C", 1), // OK.
-         newQRecord("uuid-3", "SKU-C", 2), // OK.
-         newQRecord("uuid-4", "SKU-C", 1), // violate sku/storeId UK in this set
-         newQRecord("uuid-A", "SKU-X", 1), // violate uuid UK from pre-existing records
-         newQRecord("uuid-D", "SKU-2", 1)  // violate sku/storeId UK from pre-existing records
+         newUkTestQRecord("uuid-1", "SKU-A", 1), // OK.
+         newUkTestQRecord("uuid-1", "SKU-B", 1), // violate uuid UK in this set
+         newUkTestQRecord("uuid-2", "SKU-C", 1), // OK.
+         newUkTestQRecord("uuid-3", "SKU-C", 2), // OK.
+         newUkTestQRecord("uuid-4", "SKU-C", 1), // violate sku/storeId UK in this set
+         newUkTestQRecord("uuid-A", "SKU-X", 1), // violate uuid UK from pre-existing records
+         newUkTestQRecord("uuid-D", "SKU-2", 1)  // violate sku/storeId UK from pre-existing records
       ));
       bulkInsertTransformStep.preRun(input, output);
       bulkInsertTransformStep.runOnePage(input, output);
 
-      ///////////////////////////////////////////////////////
-      // assert that all records pass.
-      ///////////////////////////////////////////////////////
+      ///////////////////////////////////
+      // assert that all records pass. //
+      ///////////////////////////////////
       assertEquals(7, output.getRecords().size());
    }
 
@@ -211,8 +219,8 @@ class BulkInsertTransformStepTest extends BaseTest
    private boolean recordEquals(QRecord record, String uuid, String sku, Integer storeId)
    {
       return (record.getValue("uuid").equals(uuid)
-         && record.getValue("sku").equals(sku)
-         && record.getValue("storeId").equals(storeId));
+              && record.getValue("sku").equals(sku)
+              && record.getValue("storeId").equals(storeId));
    }
 
 
@@ -220,7 +228,7 @@ class BulkInsertTransformStepTest extends BaseTest
    /*******************************************************************************
     **
     *******************************************************************************/
-   private QRecord newQRecord(String uuid, String sku, int storeId)
+   private QRecord newUkTestQRecord(String uuid, String sku, int storeId)
    {
       return new QRecord()
          .withValue("uuid", uuid)
@@ -229,4 +237,134 @@ class BulkInsertTransformStepTest extends BaseTest
          .withValue("name", "Some Item");
    }
 
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testValueMappingTypeErrors() throws QException
+   {
+      ///////////////////////////////////////////
+      // setup & run the bulk insert transform //
+      ///////////////////////////////////////////
+      BulkInsertTransformStep bulkInsertTransformStep = new BulkInsertTransformStep();
+      RunBackendStepInput     input                   = new RunBackendStepInput();
+      RunBackendStepOutput    output                  = new RunBackendStepOutput();
+      Serializable[]          emptyValues             = new Serializable[0];
+
+      input.setTableName(TestUtils.TABLE_NAME_ORDER);
+      input.setStepName(StreamedETLWithFrontendProcess.STEP_NAME_VALIDATE);
+      input.setRecords(ListBuilder.of(
+         BulkLoadRecordUtils.addBackendDetailsAboutFileRows(new QRecord(), new BulkLoadFileRow(emptyValues, 1))
+            .withError(new BulkLoadValueTypeError("storeId", "A", QFieldType.INTEGER, "Store"))
+            .withError(new BulkLoadValueTypeError("orderDate", "47", QFieldType.DATE, "Order Date")),
+         BulkLoadRecordUtils.addBackendDetailsAboutFileRows(new QRecord(), new BulkLoadFileRow(emptyValues, 2))
+            .withError(new BulkLoadValueTypeError("storeId", "BCD", QFieldType.INTEGER, "Store"))
+      ));
+
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      // add 102 records with an error in the total field - which is more than the number of examples that should be given //
+      ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
+      for(int i = 0; i < 102; i++)
+      {
+         input.getRecords().add(BulkLoadRecordUtils.addBackendDetailsAboutFileRows(new QRecord(), new BulkLoadFileRow(emptyValues, 3 + i))
+            .withError(new BulkLoadValueTypeError("total", "three-fifty-" + i, QFieldType.DECIMAL, "Total")));
+      }
+
+      bulkInsertTransformStep.preRun(input, output);
+      bulkInsertTransformStep.runOnePage(input, output);
+      ArrayList<ProcessSummaryLineInterface> processSummary = bulkInsertTransformStep.getProcessSummary(output, false);
+
+      ProcessSummaryAssert.assertThat(processSummary)
+         .hasLineWithMessageContaining("Cannot convert value for field [Store] to type [Integer]")
+         .hasMessageContaining("Values:")
+         .doesNotHaveMessageContaining("Example Values:")
+         .hasAnyBulletsOfTextContaining("Row 1 [A]")
+         .hasAnyBulletsOfTextContaining("Row 2 [BCD]")
+         .hasStatus(Status.ERROR)
+         .hasCount(2);
+
+      ProcessSummaryAssert.assertThat(processSummary)
+         .hasLineWithMessageContaining("Cannot convert value for field [Order Date] to type [Date]")
+         .hasMessageContaining("Values:")
+         .doesNotHaveMessageContaining("Example Values:")
+         .hasAnyBulletsOfTextContaining("Row 1 [47]")
+         .hasStatus(Status.ERROR)
+         .hasCount(1);
+
+      ProcessSummaryAssert.assertThat(processSummary)
+         .hasLineWithMessageContaining("Cannot convert value for field [Total] to type [Decimal]")
+         .hasMessageContaining("Example Values:")
+         .hasAnyBulletsOfTextContaining("Row 3 [three-fifty-0]")
+         .hasAnyBulletsOfTextContaining("Row 4 [three-fifty-1]")
+         .hasAnyBulletsOfTextContaining("Row 5 [three-fifty-2]")
+         .doesNotHaveAnyBulletsOfTextContaining("three-fifty-101")
+         .hasStatus(Status.ERROR)
+         .hasCount(102);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testRollupOfValidationErrors() throws QException
+   {
+      QContext.getQSession().withSecurityKeyValue(TestUtils.SECURITY_KEY_TYPE_STORE, 1);
+
+      ///////////////////////////////////////////
+      // setup & run the bulk insert transform //
+      ///////////////////////////////////////////
+      BulkInsertTransformStep bulkInsertTransformStep = new BulkInsertTransformStep();
+      RunBackendStepInput     input                   = new RunBackendStepInput();
+      RunBackendStepOutput    output                  = new RunBackendStepOutput();
+      Serializable[]          emptyValues             = new Serializable[0];
+
+      String tooLong = ".".repeat(201);
+
+      input.setTableName(TestUtils.TABLE_NAME_ORDER);
+      input.setStepName(StreamedETLWithFrontendProcess.STEP_NAME_VALIDATE);
+      input.setRecords(ListBuilder.of(
+         BulkLoadRecordUtils.addBackendDetailsAboutFileRows(new QRecord().withValue("shipToName", tooLong), new BulkLoadFileRow(emptyValues, 1)),
+         BulkLoadRecordUtils.addBackendDetailsAboutFileRows(new QRecord().withValue("shipToName", "OK").withValue("storeId", 1), new BulkLoadFileRow(emptyValues, 2))
+      ));
+
+      /////////////////////////////////////////////////////////////////////
+      // add 102 records with no security key - which should be an error //
+      /////////////////////////////////////////////////////////////////////
+      for(int i = 0; i < 102; i++)
+      {
+         input.getRecords().add(BulkLoadRecordUtils.addBackendDetailsAboutFileRows(new QRecord(), new BulkLoadFileRow(emptyValues, 3 + i)));
+      }
+
+      bulkInsertTransformStep.preRun(input, output);
+      bulkInsertTransformStep.runOnePage(input, output);
+      ArrayList<ProcessSummaryLineInterface> processSummary = bulkInsertTransformStep.getProcessSummary(output, false);
+
+      ProcessSummaryAssert.assertThat(processSummary)
+         .hasLineWithMessageContaining("value for Ship To Name is too long")
+         .hasMessageContaining("Records:")
+         .doesNotHaveMessageContaining("Example Records:")
+         .hasAnyBulletsOfTextContaining("Row 1")
+         .hasStatus(Status.ERROR)
+         .hasCount(1);
+
+      ProcessSummaryAssert.assertThat(processSummary)
+         .hasLineWithMessageContaining("without a value in the field: Store Id")
+         .hasMessageContaining("Example Records:")
+         .hasAnyBulletsOfTextContaining("Row 1")
+         .hasAnyBulletsOfTextContaining("Row 3")
+         .hasAnyBulletsOfTextContaining("Row 4")
+         .doesNotHaveAnyBulletsOfTextContaining("Row 101")
+         .hasStatus(Status.ERROR)
+         .hasCount(103); // the 102, plus row 1.
+
+      ProcessSummaryAssert.assertThat(processSummary)
+         .hasLineWithMessageContaining("Order record will be inserted")
+         .hasStatus(Status.OK)
+         .hasCount(1);
+   }
+
 }

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/CsvFileToRowsTest.java

@@ -0,0 +1,60 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.io.ByteArrayInputStream;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+
+/*******************************************************************************
+ ** Unit test for CsvFileToRows 
+ *******************************************************************************/
+class CsvFileToRowsTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      byte[] csvBytes = """
+         one,two,three
+         1,2,3,4
+         """.getBytes();
+      FileToRowsInterface fileToRowsInterface = FileToRowsInterface.forFile("someFile.csv", new ByteArrayInputStream(csvBytes));
+
+      BulkLoadFileRow headerRow = fileToRowsInterface.next();
+      BulkLoadFileRow bodyRow   = fileToRowsInterface.next();
+
+      assertEquals(new BulkLoadFileRow(new String[] { "one", "two", "three" }, 1), headerRow);
+      assertEquals(new BulkLoadFileRow(new String[] { "1", "2", "3", "4" }, 2), bodyRow);
+      assertFalse(fileToRowsInterface.hasNext());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/TestFileToRows.java

@@ -0,0 +1,86 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.io.InputStream;
+import java.io.Serializable;
+import java.util.List;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+
+
+/***************************************************************************
+ **
+ ***************************************************************************/
+public class TestFileToRows extends AbstractIteratorBasedFileToRows<Serializable[]> implements FileToRowsInterface
+{
+   private final List<Serializable[]> rows;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public TestFileToRows(List<Serializable[]> rows)
+   {
+      this.rows = rows;
+      setIterator(this.rows.iterator());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void init(InputStream inputStream) throws QException
+   {
+      ///////////
+      // noop! //
+      ///////////
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public void close() throws Exception
+   {
+      ///////////
+      // noop! //
+      ///////////
+   }
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @Override
+   public BulkLoadFileRow makeRow(Serializable[] values)
+   {
+      return (new BulkLoadFileRow(values, getRowNo()));
+   }
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/filehandling/XlsxFileToRowsTest.java

@@ -0,0 +1,110 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling;
+
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.Month;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.reporting.GenerateReportAction;
+import com.kingsrook.qqq.backend.core.actions.reporting.GenerateReportActionTest;
+import com.kingsrook.qqq.backend.core.actions.reporting.excel.fastexcel.ExcelFastexcelExportStreamer;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination;
+import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
+import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import org.junit.jupiter.api.Test;
+import static com.kingsrook.qqq.backend.core.actions.reporting.GenerateReportActionTest.REPORT_NAME;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+
+/*******************************************************************************
+ ** Unit test for XlsxFileToRows 
+ *******************************************************************************/
+class XlsxFileToRowsTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException, IOException
+   {
+      byte[] byteArray = writeExcelBytes();
+
+      FileToRowsInterface fileToRowsInterface = FileToRowsInterface.forFile("someFile.xlsx", new ByteArrayInputStream(byteArray));
+
+      BulkLoadFileRow headerRow = fileToRowsInterface.next();
+      BulkLoadFileRow bodyRow   = fileToRowsInterface.next();
+
+      assertEquals(new BulkLoadFileRow(new String[] {"Id", "First Name", "Last Name", "Birth Date"}, 1), headerRow);
+      assertEquals(new BulkLoadFileRow(new Serializable[] {1, "Darin", "Jonson", LocalDateTime.of(1980, Month.JANUARY, 31, 0, 0)}, 2), bodyRow);
+
+      ///////////////////////////////////////////////////////////////////////////////////////
+      // make sure there's at least a limit (less than 20) to how many more rows there are //
+      ///////////////////////////////////////////////////////////////////////////////////////
+      int otherRowCount = 0;
+      while(fileToRowsInterface.hasNext() && otherRowCount < 20)
+      {
+         fileToRowsInterface.next();
+         otherRowCount++;
+      }
+      assertFalse(fileToRowsInterface.hasNext());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static byte[] writeExcelBytes() throws QException, IOException
+   {
+      ReportFormat          format = ReportFormat.XLSX;
+      ByteArrayOutputStream baos   = new ByteArrayOutputStream();
+
+      QInstance qInstance = QContext.getQInstance();
+      qInstance.addReport(GenerateReportActionTest.defineTableOnlyReport());
+      GenerateReportActionTest.insertPersonRecords(qInstance);
+
+      ReportInput reportInput = new ReportInput();
+      reportInput.setReportName(REPORT_NAME);
+      reportInput.setReportDestination(new ReportDestination().withReportFormat(format).withReportOutputStream(baos));
+      reportInput.setInputValues(Map.of("startDate", LocalDate.of(1970, Month.MAY, 15), "endDate", LocalDate.now()));
+      reportInput.setOverrideExportStreamerSupplier(ExcelFastexcelExportStreamer::new);
+      new GenerateReportAction().execute(reportInput);
+
+      byte[] byteArray = baos.toByteArray();
+      // FileUtils.writeByteArrayToFile(new File("/tmp/xlsx.xlsx"), byteArray);
+      return byteArray;
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadMappingSuggesterTest.java

@@ -0,0 +1,209 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfileField;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadTableStructure;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+
+/*******************************************************************************
+ ** Unit test for BulkLoadMappingSuggester 
+ *******************************************************************************/
+class BulkLoadMappingSuggesterTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSimpleFlat()
+   {
+      BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_PERSON_MEMORY);
+      List<String>           headerRow      = List.of("Id", "First Name", "lastname", "email", "homestate");
+
+      BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+      assertEquals("v1", bulkLoadProfile.getVersion());
+      assertEquals("FLAT", bulkLoadProfile.getLayout());
+      assertNull(getFieldByName(bulkLoadProfile, "id"));
+      assertEquals(1, getFieldByName(bulkLoadProfile, "firstName").getColumnIndex());
+      assertEquals(2, getFieldByName(bulkLoadProfile, "lastName").getColumnIndex());
+      assertEquals(3, getFieldByName(bulkLoadProfile, "email").getColumnIndex());
+      assertEquals(4, getFieldByName(bulkLoadProfile, "homeStateId").getColumnIndex());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSimpleTall()
+   {
+      BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_ORDER);
+      List<String>           headerRow      = List.of("orderNo", "shipto name", "sku", "quantity");
+
+      BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+      assertEquals("v1", bulkLoadProfile.getVersion());
+      assertEquals("TALL", bulkLoadProfile.getLayout());
+      assertEquals(0, getFieldByName(bulkLoadProfile, "orderNo").getColumnIndex());
+      assertEquals(1, getFieldByName(bulkLoadProfile, "shipToName").getColumnIndex());
+      assertEquals(2, getFieldByName(bulkLoadProfile, "orderLine.sku").getColumnIndex());
+      assertEquals(3, getFieldByName(bulkLoadProfile, "orderLine.quantity").getColumnIndex());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testTallWithTableNamesOnAssociations()
+   {
+      BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_ORDER);
+      List<String>           headerRow      = List.of("Order No", "Ship To Name", "Order Line: SKU", "Order Line: Quantity");
+
+      BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+      assertEquals("v1", bulkLoadProfile.getVersion());
+      assertEquals("TALL", bulkLoadProfile.getLayout());
+      assertEquals(0, getFieldByName(bulkLoadProfile, "orderNo").getColumnIndex());
+      assertEquals(1, getFieldByName(bulkLoadProfile, "shipToName").getColumnIndex());
+      assertEquals(2, getFieldByName(bulkLoadProfile, "orderLine.sku").getColumnIndex());
+      assertEquals(3, getFieldByName(bulkLoadProfile, "orderLine.quantity").getColumnIndex());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testChallengingAddress1And2()
+   {
+      try
+      {
+         {
+            QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_ORDER);
+            table.addField(new QFieldMetaData("address1", QFieldType.STRING));
+            table.addField(new QFieldMetaData("address2", QFieldType.STRING));
+
+            BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_ORDER);
+            List<String>           headerRow      = List.of("orderNo", "ship to name", "address 1", "address 2");
+
+            BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+            assertEquals(0, getFieldByName(bulkLoadProfile, "orderNo").getColumnIndex());
+            assertEquals(1, getFieldByName(bulkLoadProfile, "shipToName").getColumnIndex());
+            assertEquals(2, getFieldByName(bulkLoadProfile, "address1").getColumnIndex());
+            assertEquals(3, getFieldByName(bulkLoadProfile, "address2").getColumnIndex());
+            reInitInstanceInContext(TestUtils.defineInstance());
+         }
+
+         {
+            QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_ORDER);
+            table.addField(new QFieldMetaData("address", QFieldType.STRING));
+            table.addField(new QFieldMetaData("address2", QFieldType.STRING));
+
+            BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_ORDER);
+            List<String>           headerRow      = List.of("orderNo", "ship to name", "address 1", "address 2");
+
+            BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+            assertEquals(0, getFieldByName(bulkLoadProfile, "orderNo").getColumnIndex());
+            assertEquals(1, getFieldByName(bulkLoadProfile, "shipToName").getColumnIndex());
+            assertEquals(2, getFieldByName(bulkLoadProfile, "address").getColumnIndex());
+            assertEquals(3, getFieldByName(bulkLoadProfile, "address2").getColumnIndex());
+            reInitInstanceInContext(TestUtils.defineInstance());
+         }
+
+         {
+            QTableMetaData table = QContext.getQInstance().getTable(TestUtils.TABLE_NAME_ORDER);
+            table.addField(new QFieldMetaData("address1", QFieldType.STRING));
+            table.addField(new QFieldMetaData("address2", QFieldType.STRING));
+
+            BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_ORDER);
+            List<String>           headerRow      = List.of("orderNo", "ship to name", "address", "address 2");
+
+            BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+            assertEquals(0, getFieldByName(bulkLoadProfile, "orderNo").getColumnIndex());
+            assertEquals(1, getFieldByName(bulkLoadProfile, "shipToName").getColumnIndex());
+            assertEquals(2, getFieldByName(bulkLoadProfile, "address1").getColumnIndex());
+            assertEquals(3, getFieldByName(bulkLoadProfile, "address2").getColumnIndex());
+            reInitInstanceInContext(TestUtils.defineInstance());
+         }
+      }
+      finally
+      {
+         reInitInstanceInContext(TestUtils.defineInstance());
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSimpleWide()
+   {
+      BulkLoadTableStructure tableStructure = BulkLoadTableStructureBuilder.buildTableStructure(TestUtils.TABLE_NAME_ORDER);
+      List<String>           headerRow      = List.of("orderNo", "ship to name", "sku", "quantity1", "sku 2", "quantity 2");
+
+      BulkLoadProfile bulkLoadProfile = new BulkLoadMappingSuggester().suggestBulkLoadMappingProfile(tableStructure, headerRow);
+      assertEquals("v1", bulkLoadProfile.getVersion());
+      assertEquals("WIDE", bulkLoadProfile.getLayout());
+      assertEquals(0, getFieldByName(bulkLoadProfile, "orderNo").getColumnIndex());
+      assertEquals(1, getFieldByName(bulkLoadProfile, "shipToName").getColumnIndex());
+      assertEquals(2, getFieldByName(bulkLoadProfile, "orderLine.sku,0").getColumnIndex());
+      assertEquals(3, getFieldByName(bulkLoadProfile, "orderLine.quantity,0").getColumnIndex());
+      assertEquals(4, getFieldByName(bulkLoadProfile, "orderLine.sku,1").getColumnIndex());
+      assertEquals(5, getFieldByName(bulkLoadProfile, "orderLine.quantity,1").getColumnIndex());
+
+      /////////////////////////////////////////////////////////////////
+      // assert that the order of fields matches the file's ordering //
+      /////////////////////////////////////////////////////////////////
+      assertEquals(List.of("orderNo", "shipToName", "orderLine.sku,0", "orderLine.quantity,0", "orderLine.sku,1", "orderLine.quantity,1"),
+         bulkLoadProfile.getFieldList().stream().map(f -> f.getFieldName()).toList());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private BulkLoadProfileField getFieldByName(BulkLoadProfile bulkLoadProfile, String fieldName)
+   {
+      return (bulkLoadProfile.getFieldList().stream()
+         .filter(f -> f.getFieldName().equals(fieldName))
+         .findFirst().orElse(null));
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/BulkLoadValueMapperTest.java

@@ -0,0 +1,138 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.json.JSONArray;
+import org.json.JSONObject;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Unit test for ValueMapper 
+ *******************************************************************************/
+class BulkLoadValueMapperTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      BulkInsertMapping mapping = new BulkInsertMapping().withFieldNameToValueMapping(Map.of(
+         "storeId", Map.of("QQQMart", 1, "Q'R'Us", 2),
+         "shipToName", Map.of("HoJu", "Homer", "Bart", "Bartholomew"),
+         "orderLine.sku", Map.of("ABC", "Alphabet"),
+         "orderLine.extrinsics.value", Map.of("foo", "bar", "bar", "baz"),
+         "extrinsics.key", Map.of("1", "one", "2", "two")
+      ));
+
+      QRecord inputRecord = new QRecord()
+         .withValue("storeId", "QQQMart")
+         .withValue("shipToName", "HoJu")
+         .withAssociatedRecord("orderLine", new QRecord()
+            .withValue("sku", "ABC")
+            .withAssociatedRecord("extrinsics", new QRecord()
+               .withValue("key", "myKey")
+               .withValue("value", "foo")
+            )
+            .withAssociatedRecord("extrinsics", new QRecord()
+               .withValue("key", "yourKey")
+               .withValue("value", "bar")
+            )
+         )
+         .withAssociatedRecord("extrinsics", new QRecord()
+            .withValue("key", 1)
+            .withValue("value", "foo")
+         );
+      JSONObject beforeJson = recordToJson(inputRecord);
+
+      QRecord expectedRecord = new QRecord()
+         .withValue("storeId", 1)
+         .withValue("shipToName", "Homer")
+         .withAssociatedRecord("orderLine", new QRecord()
+            .withValue("sku", "Alphabet")
+            .withAssociatedRecord("extrinsics", new QRecord()
+               .withValue("key", "myKey")
+               .withValue("value", "bar")
+            )
+            .withAssociatedRecord("extrinsics", new QRecord()
+               .withValue("key", "yourKey")
+               .withValue("value", "baz")
+            )
+         )
+         .withAssociatedRecord("extrinsics", new QRecord()
+            .withValue("key", "one")
+            .withValue("value", "foo")
+         );
+      JSONObject expectedJson = recordToJson(expectedRecord);
+
+      BulkLoadValueMapper.valueMapping(List.of(inputRecord), mapping, QContext.getQInstance().getTable(TestUtils.TABLE_NAME_ORDER));
+      JSONObject actualJson = recordToJson(inputRecord);
+
+      System.out.println("Before");
+      System.out.println(beforeJson.toString(3));
+      System.out.println("Actual");
+      System.out.println(actualJson.toString(3));
+      System.out.println("Expected");
+      System.out.println(expectedJson.toString(3));
+
+      assertThat(actualJson).usingRecursiveComparison().isEqualTo(expectedJson);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static JSONObject recordToJson(QRecord record)
+   {
+      JSONObject jsonObject = new JSONObject();
+      for(Map.Entry<String, Serializable> valueEntry : CollectionUtils.nonNullMap(record.getValues()).entrySet())
+      {
+         jsonObject.put(valueEntry.getKey(), valueEntry.getValue());
+      }
+      for(Map.Entry<String, List<QRecord>> associationEntry : CollectionUtils.nonNullMap(record.getAssociatedRecords()).entrySet())
+      {
+         JSONArray jsonArray = new JSONArray();
+         for(QRecord associationRecord : CollectionUtils.nonNullList(associationEntry.getValue()))
+         {
+            jsonArray.put(recordToJson(associationRecord));
+         }
+         jsonObject.put(associationEntry.getKey(), jsonArray);
+      }
+      return (jsonObject);
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/FlatRowsToRecordTest.java

@@ -0,0 +1,261 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.TestFileToRows;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import com.kingsrook.qqq.backend.core.utils.collections.ListBuilder;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for FlatRowsToRecord 
+ *******************************************************************************/
+class FlatRowsToRecordTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testFieldNameToHeaderNameMapping() throws QException
+   {
+      TestFileToRows fileToRows = new TestFileToRows(List.of(
+         new Serializable[] { "id", "firstName", "Last Name", "Ignore", "cost" },
+         new Serializable[] { 1, "Homer", "Simpson", true, "three fifty" },
+         new Serializable[] { 2, "Marge", "Simpson", false, "" },
+         new Serializable[] { 3, "Bart", "Simpson", "A", "99.95" },
+         new Serializable[] { 4, "Ned", "Flanders", 3.1, "one$" }
+      ));
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      FlatRowsToRecord rowsToRecord = new FlatRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "firstName", "firstName",
+            "lastName", "Last Name",
+            "cost", "cost"
+         ))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "noOfShoes", 2
+         ))
+         .withFieldNameToValueMapping(Map.of("cost", Map.of("three fifty", new BigDecimal("3.50"), "one$", new BigDecimal("1.00"))))
+         .withTableName(TestUtils.TABLE_NAME_PERSON)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, 1);
+      assertEquals(1, records.size());
+      assertEquals(List.of("Homer"), getValues(records, "firstName"));
+      assertEquals(List.of("Simpson"), getValues(records, "lastName"));
+      assertEquals(List.of(2), getValues(records, "noOfShoes"));
+      assertEquals(List.of(new BigDecimal("3.50")), getValues(records, "cost"));
+      assertEquals(4, records.get(0).getValues().size()); // make sure no additional values were set
+      assertEquals(1, ((List<?>) records.get(0).getBackendDetail("fileRows")).size());
+      assertEquals("Row 2", records.get(0).getBackendDetail("rowNos"));
+
+      records = rowsToRecord.nextPage(fileToRows, header, mapping, 2);
+      assertEquals(2, records.size());
+      assertEquals(List.of("Marge", "Bart"), getValues(records, "firstName"));
+      assertEquals(List.of(2, 2), getValues(records, "noOfShoes"));
+      assertEquals(ListBuilder.of(null, new BigDecimal("99.95")), getValues(records, "cost"));
+      assertEquals(1, ((List<?>) records.get(0).getBackendDetail("fileRows")).size());
+      assertEquals("Row 3", records.get(0).getBackendDetail("rowNos"));
+      assertEquals("Row 4", records.get(1).getBackendDetail("rowNos"));
+
+      records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(1, records.size());
+      assertEquals(List.of("Ned"), getValues(records, "firstName"));
+      assertEquals(List.of(2), getValues(records, "noOfShoes"));
+      assertEquals(ListBuilder.of(new BigDecimal("1.00")), getValues(records, "cost"));
+      assertEquals("Row 5", records.get(0).getBackendDetail("rowNos"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testFieldNameToColumnIndexMapping() throws QException
+   {
+      TestFileToRows fileToRows = new TestFileToRows(List.of(
+         //                   0, 1,       2,         3,    4
+         new Serializable[] { 1, "Homer", "Simpson", true, "three fifty" },
+         new Serializable[] { 2, "Marge", "Simpson", false, "" },
+         new Serializable[] { 3, "Bart", "Simpson", "A", "99.95" },
+         new Serializable[] { 4, "Ned", "Flanders", 3.1, "one$" }
+      ));
+
+      FlatRowsToRecord rowsToRecord = new FlatRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToIndexMap(Map.of(
+            "firstName", 1,
+            "lastName", 2,
+            "cost", 4
+         ))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "noOfShoes", 2
+         ))
+         .withFieldNameToValueMapping(Map.of("cost", Map.of("three fifty", new BigDecimal("3.50"), "one$", new BigDecimal("1.00"))))
+         .withTableName(TestUtils.TABLE_NAME_PERSON)
+         .withHasHeaderRow(false);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, null, mapping, 1);
+      assertEquals(1, records.size());
+      assertEquals(List.of("Homer"), getValues(records, "firstName"));
+      assertEquals(List.of("Simpson"), getValues(records, "lastName"));
+      assertEquals(List.of(2), getValues(records, "noOfShoes"));
+      assertEquals(List.of(new BigDecimal("3.50")), getValues(records, "cost"));
+      assertEquals(4, records.get(0).getValues().size()); // make sure no additional values were set
+      assertEquals(1, ((List<?>) records.get(0).getBackendDetail("fileRows")).size());
+      assertEquals("Row 1", records.get(0).getBackendDetail("rowNos"));
+
+      records = rowsToRecord.nextPage(fileToRows, null, mapping, 2);
+      assertEquals(2, records.size());
+      assertEquals(List.of("Marge", "Bart"), getValues(records, "firstName"));
+      assertEquals(List.of(2, 2), getValues(records, "noOfShoes"));
+      assertEquals(ListBuilder.of(null, new BigDecimal("99.95")), getValues(records, "cost"));
+      assertEquals(1, ((List<?>) records.get(0).getBackendDetail("fileRows")).size());
+      assertEquals("Row 2", records.get(0).getBackendDetail("rowNos"));
+      assertEquals("Row 3", records.get(1).getBackendDetail("rowNos"));
+
+      records = rowsToRecord.nextPage(fileToRows, null, mapping, Integer.MAX_VALUE);
+      assertEquals(1, records.size());
+      assertEquals(List.of("Ned"), getValues(records, "firstName"));
+      assertEquals(List.of(2), getValues(records, "noOfShoes"));
+      assertEquals(ListBuilder.of(new BigDecimal("1.00")), getValues(records, "cost"));
+      assertEquals("Row 4", records.get(0).getBackendDetail("rowNos"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testFieldNameToIndexMapping() throws QException
+   {
+      TestFileToRows fileToRows = new TestFileToRows(List.of(
+         new Serializable[] { 1, "Homer", "Simpson", true },
+         new Serializable[] { 2, "Marge", "Simpson", false },
+         new Serializable[] { 3, "Bart", "Simpson", "A" },
+         new Serializable[] { 4, "Ned", "Flanders", 3.1 }
+      ));
+
+      BulkLoadFileRow header = null;
+
+      FlatRowsToRecord rowsToRecord = new FlatRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToIndexMap(Map.of(
+            "firstName", 1,
+            "lastName", 2
+         ))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "noOfShoes", 2
+         ))
+         .withTableName(TestUtils.TABLE_NAME_PERSON)
+         .withHasHeaderRow(false);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, 1);
+      assertEquals(List.of("Homer"), getValues(records, "firstName"));
+      assertEquals(List.of("Simpson"), getValues(records, "lastName"));
+      assertEquals(List.of(2), getValues(records, "noOfShoes"));
+      assertEquals(3, records.get(0).getValues().size()); // make sure no additional values were set
+
+      records = rowsToRecord.nextPage(fileToRows, header, mapping, 2);
+      assertEquals(List.of("Marge", "Bart"), getValues(records, "firstName"));
+      assertEquals(List.of(2, 2), getValues(records, "noOfShoes"));
+
+      records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(List.of("Ned"), getValues(records, "firstName"));
+      assertEquals(List.of(2), getValues(records, "noOfShoes"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testPossibleValueMappings() throws QException
+   {
+      TestFileToRows fileToRows = new TestFileToRows(List.of(
+         new Serializable[] { "id", "firstName", "Last Name", "Home State" },
+         new Serializable[] { 1, "Homer", "Simpson", 1 },
+         new Serializable[] { 2, "Marge", "Simpson", "MO" },
+         new Serializable[] { 3, "Bart", "Simpson", null },
+         new Serializable[] { 4, "Ned", "Flanders", "Not a state" },
+         new Serializable[] { 5, "Mr.", "Burns", 5 }
+      ));
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      FlatRowsToRecord rowsToRecord = new FlatRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "firstName", "firstName",
+            "lastName", "Last Name",
+            "homeStateId", "Home State"
+         ))
+         .withTableName(TestUtils.TABLE_NAME_PERSON)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(5, records.size());
+      assertEquals(List.of("Homer", "Marge", "Bart", "Ned", "Mr."), getValues(records, "firstName"));
+      assertEquals(ListBuilder.of(1, 2, null, "Not a state", 5), getValues(records, "homeStateId"));
+
+      assertThat(records.get(0).getErrors()).isNullOrEmpty();
+      assertThat(records.get(1).getErrors()).isNullOrEmpty();
+      assertThat(records.get(2).getErrors()).isNullOrEmpty();
+      assertThat(records.get(3).getErrors()).hasSize(1).element(0).matches(e -> e.getMessage().contains("not a valid option"));
+      assertThat(records.get(4).getErrors()).hasSize(1).element(0).matches(e -> e.getMessage().contains("not a valid option"));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<Serializable> getValues(List<QRecord> records, String fieldName)
+   {
+      return (records.stream().map(r -> r.getValue(fieldName)).toList());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/TallRowsToRecordTest.java

@@ -0,0 +1,544 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.CsvFileToRows;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+/*******************************************************************************
+ ** Unit test for TallRowsToRecord 
+ *******************************************************************************/
+class TallRowsToRecordTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderAndLines() throws QException
+   {
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         orderNo, Ship To, lastName, SKU,       Quantity
+         1,       Homer,   Simpson,  DONUT,     12
+         ,        Homer,   Simpson,  BEER,      500
+         ,        Homer,   Simpson,  COUCH,     1
+         2,       Ned,     Flanders, BIBLE,     7
+         ,        Ned,     Flanders, LAWNMOWER, 1
+         """);
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity"
+         ))
+         .withTallLayoutGroupByIndexMap(Map.of(
+            TestUtils.TABLE_NAME_ORDER, List.of(1, 2),
+            "orderLine", List.of(3)
+         ))
+         .withMappedAssociations(List.of("orderLine"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(3, order.getAssociatedRecords().get("orderLine").size());
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals("Rows 2-4", order.getBackendDetail("rowNos"));
+      assertEquals(3, ((List<?>) order.getBackendDetail("fileRows")).size());
+      assertEquals("Row 2", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+      assertEquals("Row 3", order.getAssociatedRecords().get("orderLine").get(1).getBackendDetail("rowNos"));
+      assertEquals("Row 4", order.getAssociatedRecords().get("orderLine").get(2).getBackendDetail("rowNos"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(2, order.getAssociatedRecords().get("orderLine").size());
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals("Rows 5-6", order.getBackendDetail("rowNos"));
+      assertEquals(2, ((List<?>) order.getBackendDetail("fileRows")).size());
+   }
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderAndLinesWithoutHeader() throws QException
+   {
+      // 0, 1,     2,        3,         4
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         1, Homer, Simpson,  DONUT,     12
+         ,  Homer, Simpson,  BEER,      500
+         ,  Homer, Simpson,  COUCH,     1
+         2, Ned,   Flanders, BIBLE,     7
+         ,  Ned,   Flanders, LAWNMOWER, 1
+         """);
+
+      BulkLoadFileRow header = null;
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToIndexMap(Map.of(
+            "orderNo", 0,
+            "shipToName", 1,
+            "orderLine.sku", 3,
+            "orderLine.quantity", 4
+         ))
+         .withTallLayoutGroupByIndexMap(Map.of(
+            TestUtils.TABLE_NAME_ORDER, List.of(1, 2),
+            "orderLine", List.of(3)
+         ))
+         .withMappedAssociations(List.of("orderLine"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(false);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(3, order.getAssociatedRecords().get("orderLine").size());
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals("Rows 1-3", order.getBackendDetail("rowNos"));
+      assertEquals(3, ((List<?>) order.getBackendDetail("fileRows")).size());
+      assertEquals("Row 1", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+      assertEquals("Row 2", order.getAssociatedRecords().get("orderLine").get(1).getBackendDetail("rowNos"));
+      assertEquals("Row 3", order.getAssociatedRecords().get("orderLine").get(2).getBackendDetail("rowNos"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(2, order.getAssociatedRecords().get("orderLine").size());
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals("Rows 4-5", order.getBackendDetail("rowNos"));
+      assertEquals(2, ((List<?>) order.getBackendDetail("fileRows")).size());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesAndOrderExtrinsic() throws QException
+   {
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         orderNo, Ship To, lastName, SKU,       Quantity, Extrinsic Key, Extrinsic Value
+         1,       Homer,   Simpson,  DONUT,     12,       Store Name,    QQQ Mart
+         1,       ,        ,         BEER,      500,      Coupon Code,   10QOff
+         1,       ,        ,         COUCH,     1
+         2,       Ned,     Flanders, BIBLE,     7
+         2,       ,        ,         LAWNMOWER, 1
+         """);
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity",
+            "extrinsics.key", "Extrinsic Key",
+            "extrinsics.value", "Extrinsic Value"
+         ))
+         .withTallLayoutGroupByIndexMap(Map.of(
+            TestUtils.TABLE_NAME_ORDER, List.of(0),
+            "orderLine", List.of(3),
+            "extrinsics", List.of(5)
+         ))
+         .withMappedAssociations(List.of("orderLine", "extrinsics"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(3, order.getAssociatedRecords().get("orderLine").size());
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(2, order.getAssociatedRecords().get("extrinsics").size());
+      assertEquals(List.of("Store Name", "Coupon Code"), getValues(order.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("QQQ Mart", "10QOff"), getValues(order.getAssociatedRecords().get("extrinsics"), "value"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(2, order.getAssociatedRecords().get("orderLine").size());
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertThat(order.getAssociatedRecords().get("extrinsics")).isNullOrEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesWithLineExtrinsicsAndOrderExtrinsic() throws QException
+   {
+      Integer defaultStoreId              = 101;
+      String  defaultLineNo               = "102";
+      String  defaultOrderLineExtraSource = "file";
+
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         orderNo, Ship To, lastName, SKU,       Quantity, Extrinsic Key, Extrinsic Value, Line Extrinsic Key, Line Extrinsic Value
+         1,       Homer,   Simpson,  DONUT,     12,       Store Name,    QQQ Mart,        Flavor,             Chocolate
+         1,       ,        ,         DONUT,     ,         Coupon Code,   10QOff,          Size,               Large
+         1,       ,        ,         BEER,      500,      ,              ,                Flavor,             Hops
+         1,       ,        ,         COUCH,     1
+         2,       Ned,     Flanders, BIBLE,     7,        ,              ,                Flavor,             King James
+         2,       ,        ,         LAWNMOWER, 1
+         """);
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity",
+            "extrinsics.key", "Extrinsic Key",
+            "extrinsics.value", "Extrinsic Value",
+            "orderLine.extrinsics.key", "Line Extrinsic Key",
+            "orderLine.extrinsics.value", "Line Extrinsic Value"
+         ))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "storeId", defaultStoreId,
+            "orderLine.lineNumber", defaultLineNo,
+            "orderLine.extrinsics.source", defaultOrderLineExtraSource
+         ))
+         .withFieldNameToValueMapping(Map.of("orderLine.sku", Map.of("DONUT", "D'OH-NUT")))
+         .withTallLayoutGroupByIndexMap(Map.of(
+            TestUtils.TABLE_NAME_ORDER, List.of(0),
+            "orderLine", List.of(3),
+            "extrinsics", List.of(5),
+            "orderLine.extrinsics", List.of(7)
+         ))
+         .withMappedAssociations(List.of("orderLine", "extrinsics", "orderLine.extrinsics"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals(List.of("D'OH-NUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of(defaultLineNo, defaultLineNo, defaultLineNo), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertEquals(List.of("Store Name", "Coupon Code"), getValues(order.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("QQQ Mart", "10QOff"), getValues(order.getAssociatedRecords().get("extrinsics"), "value"));
+
+      QRecord lineItem = order.getAssociatedRecords().get("orderLine").get(0);
+      assertEquals(List.of("Flavor", "Size"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Chocolate", "Large"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+      assertEquals(List.of(defaultOrderLineExtraSource, defaultOrderLineExtraSource), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "source"));
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(1);
+      assertEquals(List.of("Flavor"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Hops"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of(defaultLineNo, defaultLineNo), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertThat(order.getAssociatedRecords().get("extrinsics")).isNullOrEmpty();
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(0);
+      assertEquals(List.of("Flavor"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("King James"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+      assertEquals(List.of(defaultOrderLineExtraSource), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "source"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testAutomaticGroupByAllIndexes() throws QException
+   {
+      Integer defaultStoreId              = 101;
+      String  defaultLineNo               = "102";
+      String  defaultOrderLineExtraSource = "file";
+
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         orderNo, Ship To, lastName, SKU,       Quantity, Extrinsic Key, Extrinsic Value, Line Extrinsic Key, Line Extrinsic Value
+         1,       Homer,   Simpson,  DONUT,     12,       Store Name,    QQQ Mart,        Flavor,             Chocolate
+         1,       Homer,   Simpson,  DONUT,     12,       Coupon Code,   10QOff,          Size,               Large
+         1,       Homer,   Simpson,  BEER,      500,      ,              ,                Flavor,             Hops
+         1,       Homer,   Simpson,  COUCH,     1
+         2,       Ned,     Flanders, BIBLE,     7,        ,              ,                Flavor,             King James
+         2,       Ned,     Flanders, LAWNMOWER, 1
+         """);
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity",
+            "extrinsics.key", "Extrinsic Key",
+            "extrinsics.value", "Extrinsic Value",
+            "orderLine.extrinsics.key", "Line Extrinsic Key",
+            "orderLine.extrinsics.value", "Line Extrinsic Value"
+         ))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "storeId", defaultStoreId,
+            "orderLine.lineNumber", defaultLineNo,
+            "orderLine.extrinsics.source", defaultOrderLineExtraSource
+         ))
+         .withFieldNameToValueMapping(Map.of("orderLine.sku", Map.of("DONUT", "D'OH-NUT")))
+         .withMappedAssociations(List.of("orderLine", "extrinsics", "orderLine.extrinsics"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals(List.of("D'OH-NUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of(defaultLineNo, defaultLineNo, defaultLineNo), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertEquals(List.of("Store Name", "Coupon Code"), getValues(order.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("QQQ Mart", "10QOff"), getValues(order.getAssociatedRecords().get("extrinsics"), "value"));
+
+      QRecord lineItem = order.getAssociatedRecords().get("orderLine").get(0);
+      assertEquals(List.of("Flavor", "Size"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Chocolate", "Large"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+      assertEquals(List.of(defaultOrderLineExtraSource, defaultOrderLineExtraSource), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "source"));
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(1);
+      assertEquals(List.of("Flavor"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Hops"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of(defaultLineNo, defaultLineNo), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertThat(order.getAssociatedRecords().get("extrinsics")).isNullOrEmpty();
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(0);
+      assertEquals(List.of("Flavor"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("King James"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+      assertEquals(List.of(defaultOrderLineExtraSource), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "source"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testSingleLine() throws QException
+   {
+      Integer defaultStoreId = 101;
+
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         orderNo, Ship To, lastName
+         1,       Homer,   Simpson
+         """);
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To"
+         ))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "storeId", defaultStoreId
+         ))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(1, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals("Row 2", order.getBackendDetail("rowNos"));
+      assertEquals(1, ((List<?>) order.getBackendDetail("fileRows")).size());
+   }
+
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testPagination() throws QException
+   {
+      CsvFileToRows fileToRows = CsvFileToRows.forString("""
+         orderNo, Ship To, lastName, SKU,       Quantity
+         1,       Homer,   Simpson,  DONUT,     12
+         2,       Ned,     Flanders, BIBLE,     7
+         2,       Ned,     Flanders, LAWNMOWER, 1
+         3,       Bart,    Simpson,  SKATEBOARD,1
+         3,       Bart,    Simpson,  SLINGSHOT, 1
+         """);
+
+      BulkLoadFileRow header = fileToRows.next();
+
+      TallRowsToRecord rowsToRecord = new TallRowsToRecord();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity"
+         ))
+         .withTallLayoutGroupByIndexMap(Map.of(
+            TestUtils.TABLE_NAME_ORDER, List.of(1, 2),
+            "orderLine", List.of(3)
+         ))
+         .withMappedAssociations(List.of("orderLine"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.TALL)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, 2);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(List.of("DONUT"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals("Row 2", order.getBackendDetail("rowNos"));
+      assertEquals(1, ((List<?>) order.getBackendDetail("fileRows")).size());
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals("Rows 3-4", order.getBackendDetail("rowNos"));
+      assertEquals(2, ((List<?>) order.getBackendDetail("fileRows")).size());
+
+      records = rowsToRecord.nextPage(fileToRows, header, mapping, 2);
+      assertEquals(1, records.size());
+      order = records.get(0);
+      assertEquals(3, order.getValueInteger("orderNo"));
+      assertEquals("Bart", order.getValueString("shipToName"));
+      assertEquals(List.of("SKATEBOARD", "SLINGSHOT"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals("Rows 5-6", order.getBackendDetail("rowNos"));
+      assertEquals(2, ((List<?>) order.getBackendDetail("fileRows")).size());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testShouldProcessAssociation()
+   {
+      TallRowsToRecord tallRowsToRecord = new TallRowsToRecord();
+      assertTrue(tallRowsToRecord.shouldProcessAssociation(null, "foo"));
+      assertTrue(tallRowsToRecord.shouldProcessAssociation("", "foo"));
+      assertTrue(tallRowsToRecord.shouldProcessAssociation("foo", "foo.bar"));
+      assertTrue(tallRowsToRecord.shouldProcessAssociation("foo.bar", "foo.bar.baz"));
+
+      assertFalse(tallRowsToRecord.shouldProcessAssociation(null, "foo.bar"));
+      assertFalse(tallRowsToRecord.shouldProcessAssociation("", "foo.bar"));
+      assertFalse(tallRowsToRecord.shouldProcessAssociation("fiz", "foo.bar"));
+      assertFalse(tallRowsToRecord.shouldProcessAssociation("fiz.biz", "foo.bar"));
+      assertFalse(tallRowsToRecord.shouldProcessAssociation("foo", "foo.bar.baz"));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<Serializable> getValues(List<QRecord> records, String fieldName)
+   {
+      return (records.stream().map(r -> r.getValue(fieldName)).toList());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMappingTest.java

@@ -0,0 +1,235 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.CsvFileToRows;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping 
+ *******************************************************************************/
+class WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMappingTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderAndLinesWithoutDupes() throws QException
+   {
+      String csv = """
+         orderNo, Ship To, lastName, SKU 1,     Quantity 1, SKU 2,     Quantity 2, SKU 3, Quantity 3
+         1,       Homer,   Simpson,  DONUT,     12,         BEER,      500,        COUCH,  1
+         2,       Ned,     Flanders, BIBLE,     7,          LAWNMOWER, 1
+         """;
+
+      CsvFileToRows   fileToRows = CsvFileToRows.forString(csv);
+      BulkLoadFileRow header     = fileToRows.next();
+
+      WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping rowsToRecord = new WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku,0", "SKU 1",
+            "orderLine.quantity,0", "Quantity 1",
+            "orderLine.sku,1", "SKU 2",
+            "orderLine.quantity,1", "Quantity 2",
+            "orderLine.sku,2", "SKU 3",
+            "orderLine.quantity,2", "Quantity 3"
+         ))
+         .withMappedAssociations(List.of("orderLine"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.WIDE)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(1, ((List<?>) order.getBackendDetail("fileRows")).size());
+      assertEquals("Row 2", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(1, ((List<?>) order.getBackendDetail("fileRows")).size());
+      assertEquals("Row 3", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+   }
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderAndLinesWithoutHeader() throws QException
+   {
+      // 0, 1,     2,        3,     4,  5,         6,   7,     8
+      String csv = """
+         1, Homer, Simpson,  DONUT, 12, BEER,      500, COUCH, 1
+         2, Ned,   Flanders, BIBLE, 7,  LAWNMOWER, 1
+         """;
+
+      CsvFileToRows   fileToRows = CsvFileToRows.forString(csv);
+      BulkLoadFileRow header     = null;
+
+      WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping rowsToRecord = new WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToIndexMap(Map.of(
+            "orderNo", 0,
+            "shipToName", 1,
+            "orderLine.sku,0", 3,
+            "orderLine.quantity,0", 4,
+            "orderLine.sku,1", 5,
+            "orderLine.quantity,1", 6,
+            "orderLine.sku,2", 7,
+            "orderLine.quantity,2", 8
+         ))
+         .withMappedAssociations(List.of("orderLine"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.WIDE)
+         .withHasHeaderRow(false);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(1, ((List<?>) order.getBackendDetail("fileRows")).size());
+      assertEquals("Row 1", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+      assertEquals("Row 1", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+      assertEquals("Row 1", order.getAssociatedRecords().get("orderLine").get(1).getBackendDetail("rowNos"));
+      assertEquals("Row 1", order.getAssociatedRecords().get("orderLine").get(2).getBackendDetail("rowNos"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(1, ((List<?>) order.getBackendDetail("fileRows")).size());
+      assertEquals("Row 2", order.getAssociatedRecords().get("orderLine").get(0).getBackendDetail("rowNos"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesAndOrderExtrinsicWithoutDupes() throws QException
+   {
+      String csv = """
+         orderNo, Ship To, lastName, SKU 1, Quantity 1, SKU 2,     Quantity 2, SKU 3, Quantity 3, Extrinsic Key 1, Extrinsic Value 1, Extrinsic Key 2, Extrinsic Value 2
+         1,       Homer,   Simpson,  DONUT, 12,         BEER,      500,        COUCH, 1,          Store Name,      QQQ Mart,          Coupon Code,     10QOff
+         2,       Ned,     Flanders, BIBLE, 7,          LAWNMOWER, 1
+         """;
+
+      CsvFileToRows   fileToRows = CsvFileToRows.forString(csv);
+      BulkLoadFileRow header     = fileToRows.next();
+
+      WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping rowsToRecord = new WideRowsToRecordWithExplicitFieldNameSuffixIndexBasedMapping();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(MapBuilder.of(() -> new HashMap<String, String>())
+            .with("orderNo", "orderNo")
+            .with("shipToName", "Ship To")
+
+            .with("orderLine.sku,0", "SKU 1")
+            .with("orderLine.quantity,0", "Quantity 1")
+            .with("orderLine.sku,1", "SKU 2")
+            .with("orderLine.quantity,1", "Quantity 2")
+            .with("orderLine.sku,2", "SKU 3")
+            .with("orderLine.quantity,2", "Quantity 3")
+
+            .with("extrinsics.key,0", "Extrinsic Key 1")
+            .with("extrinsics.value,0", "Extrinsic Value 1")
+            .with("extrinsics.key,1", "Extrinsic Key 2")
+            .with("extrinsics.value,1", "Extrinsic Value 2")
+            .build()
+         )
+         .withFieldNameToDefaultValueMap(Map.of(
+            "orderLine.lineNumber,0", "1",
+            "orderLine.lineNumber,1", "2",
+            "orderLine.lineNumber,2", "3"
+         ))
+         .withMappedAssociations(List.of("orderLine", "extrinsics"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.WIDE)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of("1", "2", "3"), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertEquals(List.of("Store Name", "Coupon Code"), getValues(order.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("QQQ Mart", "10QOff"), getValues(order.getAssociatedRecords().get("extrinsics"), "value"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of("1", "2"), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertThat(order.getAssociatedRecords().get("extrinsics")).isNullOrEmpty();
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<Serializable> getValues(List<QRecord> records, String fieldName)
+   {
+      return (records.stream().map(r -> r.getValue(fieldName)).toList());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/bulk/insert/mapping/WideRowsToRecordWithSpreadMappingTest.java

@@ -0,0 +1,306 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.mapping;
+
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.filehandling.CsvFileToRows;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkInsertMapping;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadFileRow;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+/*******************************************************************************
+ ** Unit test for WideRowsToRecord
+ *******************************************************************************/
+class WideRowsToRecordWithSpreadMappingTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderAndLinesWithoutDupes() throws QException
+   {
+      testOrderAndLines("""
+         orderNo, Ship To, lastName, SKU 1,     Quantity 1, SKU 2,     Quantity 2, SKU 3, Quantity 3
+         1,       Homer,   Simpson,  DONUT,     12,         BEER,      500,        COUCH,  1
+         2,       Ned,     Flanders, BIBLE,     7,          LAWNMOWER, 1
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderAndLinesWithDupes() throws QException
+   {
+      testOrderAndLines("""
+         orderNo, Ship To, lastName, SKU,   Quantity, SKU,       Quantity, SKU,   Quantity
+         1,       Homer,   Simpson,  DONUT, 12,       BEER,      500,      COUCH, 1
+         2,       Ned,     Flanders, BIBLE, 7,        LAWNMOWER, 1
+         """);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void testOrderAndLines(String csv) throws QException
+   {
+      CsvFileToRows   fileToRows = CsvFileToRows.forString(csv);
+      BulkLoadFileRow header     = fileToRows.next();
+
+      WideRowsToRecordWithSpreadMapping rowsToRecord = new WideRowsToRecordWithSpreadMapping();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity"
+         ))
+         .withMappedAssociations(List.of("orderLine"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.WIDE)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesAndOrderExtrinsicWithoutDupes() throws QException
+   {
+      testOrderLinesAndOrderExtrinsic("""
+         orderNo, Ship To, lastName, SKU 1, Quantity 1, SKU 2,     Quantity 2, SKU 3, Quantity 3, Extrinsic Key 1, Extrinsic Value 1, Extrinsic Key 2, Extrinsic Value 2
+         1,       Homer,   Simpson,  DONUT, 12,         BEER,      500,        COUCH, 1,          Store Name,      QQQ Mart,          Coupon Code,     10QOff
+         2,       Ned,     Flanders, BIBLE, 7,          LAWNMOWER, 1
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesAndOrderExtrinsicWithDupes() throws QException
+   {
+      testOrderLinesAndOrderExtrinsic("""
+         orderNo, Ship To, lastName, SKU,   Quantity, SKU,       Quantity, SKU,   Quantity, Extrinsic Key, Extrinsic Value, Extrinsic Key, Extrinsic Value
+         1,       Homer,   Simpson,  DONUT, 12,       BEER,      500,      COUCH, 1,        Store Name,    QQQ Mart,        Coupon Code,   10QOff
+         2,       Ned,     Flanders, BIBLE, 7,        LAWNMOWER, 1
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private void testOrderLinesAndOrderExtrinsic(String csv) throws QException
+   {
+      CsvFileToRows   fileToRows = CsvFileToRows.forString(csv);
+      BulkLoadFileRow header     = fileToRows.next();
+
+      WideRowsToRecordWithSpreadMapping rowsToRecord = new WideRowsToRecordWithSpreadMapping();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity",
+            "extrinsics.key", "Extrinsic Key",
+            "extrinsics.value", "Extrinsic Value"
+         ))
+         .withMappedAssociations(List.of("orderLine", "extrinsics"))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.WIDE)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of("Store Name", "Coupon Code"), getValues(order.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("QQQ Mart", "10QOff"), getValues(order.getAssociatedRecords().get("extrinsics"), "value"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertThat(order.getAssociatedRecords().get("extrinsics")).isNullOrEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesWithLineExtrinsicsAndOrderExtrinsicWithoutDupes() throws QException
+   {
+      testOrderLinesWithLineExtrinsicsAndOrderExtrinsic("""
+         orderNo, Ship To, lastName,  Extrinsic Key 1, Extrinsic Value 1, Extrinsic Key 2, Extrinsic Value 2, SKU 1, Quantity 1, Line Extrinsic Key 1, Line Extrinsic Value 1, Line Extrinsic Key 2, Line Extrinsic Value 2, SKU 2,     Quantity 2, Line Extrinsic Key 1, Line Extrinsic Value 1, SKU 3, Quantity 3, Line Extrinsic Key 1, Line Extrinsic Value 1, Line Extrinsic Key 2
+         1,       Homer,   Simpson,   Store Name,      QQQ Mart,          Coupon Code,     10QOff,            DONUT, 12,         Flavor,               Chocolate,              Size,                 Large,                  BEER,      500,        Flavor,               Hops,                   COUCH, 1,          Color,                Brown,                  foo,
+         2,       Ned,     Flanders,  ,                ,                  ,                ,                  BIBLE, 7,          Flavor,               King James,             Size,                 X-Large,                LAWNMOWER, 1
+         """);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testOrderLinesWithLineExtrinsicsAndOrderExtrinsicWithDupes() throws QException
+   {
+      testOrderLinesWithLineExtrinsicsAndOrderExtrinsic("""
+         orderNo, Ship To, lastName,  Extrinsic Key, Extrinsic Value, Extrinsic Key, Extrinsic Value, SKU,   Quantity, Line Extrinsic Key, Line Extrinsic Value, Line Extrinsic Key, Line Extrinsic Value, SKU,       Quantity, Line Extrinsic Key, Line Extrinsic Value, SKU,   Quantity, Line Extrinsic Key, Line Extrinsic Value, Line Extrinsic Key
+         1,       Homer,   Simpson,   Store Name,    QQQ Mart,        Coupon Code,   10QOff,          DONUT, 12,       Flavor,             Chocolate,            Size,               Large,                BEER,      500,      Flavor,             Hops,                 COUCH, 1,        Color,              Brown,                foo
+         2,       Ned,     Flanders,  ,              ,                ,              ,                BIBLE, 7,        Flavor,             King James,           Size,               X-Large,              LAWNMOWER, 1
+         """);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private void testOrderLinesWithLineExtrinsicsAndOrderExtrinsic(String csv) throws QException
+   {
+      Integer defaultStoreId        = 42;
+      String  defaultLineNo         = "47";
+      String  defaultLineExtraValue = "bar";
+
+      CsvFileToRows   fileToRows = CsvFileToRows.forString(csv);
+      BulkLoadFileRow header     = fileToRows.next();
+
+      WideRowsToRecordWithSpreadMapping rowsToRecord = new WideRowsToRecordWithSpreadMapping();
+
+      BulkInsertMapping mapping = new BulkInsertMapping()
+         .withFieldNameToHeaderNameMap(Map.of(
+            "orderNo", "orderNo",
+            "shipToName", "Ship To",
+            "orderLine.sku", "SKU",
+            "orderLine.quantity", "Quantity",
+            "extrinsics.key", "Extrinsic Key",
+            "extrinsics.value", "Extrinsic Value",
+            "orderLine.extrinsics.key", "Line Extrinsic Key",
+            "orderLine.extrinsics.value", "Line Extrinsic Value"
+         ))
+         .withMappedAssociations(List.of("orderLine", "extrinsics", "orderLine.extrinsics"))
+         .withFieldNameToValueMapping(Map.of("orderLine.extrinsics.value", Map.of("Large", "L", "X-Large", "XL")))
+         .withFieldNameToDefaultValueMap(Map.of(
+            "storeId", defaultStoreId,
+            "orderLine.lineNumber", defaultLineNo,
+            "orderLine.extrinsics.value", defaultLineExtraValue
+         ))
+         .withTableName(TestUtils.TABLE_NAME_ORDER)
+         .withLayout(BulkInsertMapping.Layout.WIDE)
+         .withHasHeaderRow(true);
+
+      List<QRecord> records = rowsToRecord.nextPage(fileToRows, header, mapping, Integer.MAX_VALUE);
+      assertEquals(2, records.size());
+
+      QRecord order = records.get(0);
+      assertEquals(1, order.getValueInteger("orderNo"));
+      assertEquals("Homer", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals(List.of("DONUT", "BEER", "COUCH"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(12, 500, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of(defaultLineNo, defaultLineNo, defaultLineNo), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertEquals(List.of("Store Name", "Coupon Code"), getValues(order.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("QQQ Mart", "10QOff"), getValues(order.getAssociatedRecords().get("extrinsics"), "value"));
+
+      QRecord lineItem = order.getAssociatedRecords().get("orderLine").get(0);
+      assertEquals(List.of("Flavor", "Size"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Chocolate", "L"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(1);
+      assertEquals(List.of("Flavor"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Hops"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(2);
+      assertEquals(List.of("Color", "foo"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("Brown", defaultLineExtraValue), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+
+      order = records.get(1);
+      assertEquals(2, order.getValueInteger("orderNo"));
+      assertEquals("Ned", order.getValueString("shipToName"));
+      assertEquals(defaultStoreId, order.getValue("storeId"));
+      assertEquals(List.of("BIBLE", "LAWNMOWER"), getValues(order.getAssociatedRecords().get("orderLine"), "sku"));
+      assertEquals(List.of(7, 1), getValues(order.getAssociatedRecords().get("orderLine"), "quantity"));
+      assertEquals(List.of(defaultLineNo, defaultLineNo), getValues(order.getAssociatedRecords().get("orderLine"), "lineNumber"));
+      assertThat(order.getAssociatedRecords().get("extrinsics")).isNullOrEmpty();
+
+      lineItem = order.getAssociatedRecords().get("orderLine").get(0);
+      assertEquals(List.of("Flavor", "Size"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "key"));
+      assertEquals(List.of("King James", "XL"), getValues(lineItem.getAssociatedRecords().get("extrinsics"), "value"));
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private List<Serializable> getValues(List<QRecord> records, String fieldName)
+   {
+      return (records.stream().map(r -> r.getValue(fieldName)).toList());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/etl/streamedwithfrontend/NoopLoadStepTest.java

@@ -0,0 +1,50 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend;
+
+
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import org.junit.jupiter.api.Test;
+
+
+/*******************************************************************************
+ ** Unit test for NoopLoadStep 
+ *******************************************************************************/
+class NoopLoadStepTest extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      //////////////////////////////////////
+      // sorry, just here for coverage... //
+      //////////////////////////////////////
+      new NoopLoadStep().runOnePage(new RunBackendStepInput(), new RunBackendStepOutput());
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/processes/implementations/savedbulkloadprofiles/SavedBulkLoadProfileProcessTests.java

@@ -0,0 +1,186 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.backend.core.processes.implementations.savedbulkloadprofiles;
+
+
+import java.util.List;
+import com.kingsrook.qqq.backend.core.BaseTest;
+import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider;
+import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.TestUtils;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+
+/*******************************************************************************
+ ** Unit tests for all saved-bulk-load-profile processes
+ *******************************************************************************/
+class SavedBulkLoadProfileProcessTests extends BaseTest
+{
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void test() throws QException
+   {
+      QInstance qInstance = QContext.getQInstance();
+      new SavedBulkLoadProfileMetaDataProvider().defineAll(qInstance, TestUtils.MEMORY_BACKEND_NAME, null);
+      String tableName = TestUtils.TABLE_NAME_PERSON_MEMORY;
+
+      {
+         ////////////////////////////////////////////
+         // query - should be no profiles to start //
+         ////////////////////////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(QuerySavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("tableName", tableName);
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+         assertEquals(0, ((List<?>) runProcessOutput.getValues().get("savedBulkLoadProfileList")).size());
+      }
+
+      Integer savedBulkLoadProfileId;
+      {
+         /////////////////////////
+         // store a new profile //
+         /////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(StoreSavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("label", "My Profile");
+         runProcessInput.addValue("tableName", tableName);
+         runProcessInput.addValue("mappingJson", JsonUtils.toJson(new BulkLoadProfile()));
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+         List<QRecord>    savedBulkLoadProfileList    = (List<QRecord>) runProcessOutput.getValues().get("savedBulkLoadProfileList");
+         assertEquals(1, savedBulkLoadProfileList.size());
+         savedBulkLoadProfileId = savedBulkLoadProfileList.get(0).getValueInteger("id");
+         assertNotNull(savedBulkLoadProfileId);
+
+         //////////////////////////////////////////////////////////////////
+         // try to store it again - should throw a "duplicate" exception //
+         //////////////////////////////////////////////////////////////////
+         assertThatThrownBy(() -> new RunProcessAction().execute(runProcessInput))
+            .isInstanceOf(QUserFacingException.class)
+            .hasMessageContaining("already have a saved Bulk Load Profile");
+      }
+
+      {
+         //////////////////////////////////////
+         // query - should find our profiles //
+         //////////////////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(QuerySavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("tableName", tableName);
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+         List<QRecord>    savedBulkLoadProfileList    = (List<QRecord>) runProcessOutput.getValues().get("savedBulkLoadProfileList");
+         assertEquals(1, savedBulkLoadProfileList.size());
+         assertEquals(1, savedBulkLoadProfileList.get(0).getValueInteger("id"));
+         assertEquals("My Profile", savedBulkLoadProfileList.get(0).getValueString("label"));
+      }
+
+      {
+         ////////////////////////
+         // update our Profile //
+         ////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(StoreSavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("id", savedBulkLoadProfileId);
+         runProcessInput.addValue("label", "My Updated Profile");
+         runProcessInput.addValue("tableName", tableName);
+         runProcessInput.addValue("mappingJson", JsonUtils.toJson(new BulkLoadProfile()));
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+         List<QRecord>    savedBulkLoadProfileList    = (List<QRecord>) runProcessOutput.getValues().get("savedBulkLoadProfileList");
+         assertEquals(1, savedBulkLoadProfileList.size());
+         assertEquals(1, savedBulkLoadProfileList.get(0).getValueInteger("id"));
+         assertEquals("My Updated Profile", savedBulkLoadProfileList.get(0).getValueString("label"));
+      }
+
+      Integer anotherSavedProfileId;
+      {
+         /////////////////////////////////////////////////////////////////////////////////////////////
+         // store a second one w/ different name (will be used below in update-dupe-check use-case) //
+         /////////////////////////////////////////////////////////////////////////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(StoreSavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("label", "My Second Profile");
+         runProcessInput.addValue("tableName", tableName);
+         runProcessInput.addValue("mappingJson", JsonUtils.toJson(new BulkLoadProfile()));
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+         List<QRecord>    savedBulkLoadProfileList    = (List<QRecord>) runProcessOutput.getValues().get("savedBulkLoadProfileList");
+         anotherSavedProfileId = savedBulkLoadProfileList.get(0).getValueInteger("id");
+      }
+
+      {
+         /////////////////////////////////////////////////
+         // try to rename the second to match the first //
+         /////////////////////////////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(StoreSavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("id", anotherSavedProfileId);
+         runProcessInput.addValue("label", "My Updated Profile");
+         runProcessInput.addValue("tableName", tableName);
+         runProcessInput.addValue("mappingJson", JsonUtils.toJson(new BulkLoadProfile()));
+
+         //////////////////////////////////////////
+         // should throw a "duplicate" exception //
+         //////////////////////////////////////////
+         assertThatThrownBy(() -> new RunProcessAction().execute(runProcessInput))
+            .isInstanceOf(QUserFacingException.class)
+            .hasMessageContaining("already have a saved Bulk Load Profile");
+      }
+
+      {
+         /////////////////////////
+         // delete our profiles //
+         /////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(DeleteSavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("id", savedBulkLoadProfileId);
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+
+         runProcessInput.addValue("id", anotherSavedProfileId);
+         runProcessOutput = new RunProcessAction().execute(runProcessInput);
+      }
+
+      {
+         /////////////////////////////////////////
+         // query - should be no profiles again //
+         /////////////////////////////////////////
+         RunProcessInput runProcessInput = new RunProcessInput();
+         runProcessInput.setProcessName(QuerySavedBulkLoadProfileProcess.getProcessMetaData().getName());
+         runProcessInput.addValue("tableName", tableName);
+         RunProcessOutput runProcessOutput = new RunProcessAction().execute(runProcessInput);
+         assertEquals(0, ((List<?>) runProcessOutput.getValues().get("savedBulkLoadProfileList")).size());
+      }
+   }
+
+}

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/TestUtils.java

@@ -70,6 +70,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
@@ -142,6 +143,7 @@ public class TestUtils
    public static final String APP_NAME_MISCELLANEOUS = "miscellaneous";
 
    public static final String TABLE_NAME_TWO_KEYS            = "twoKeys";
+   public static final String TABLE_NAME_MEMORY_STORAGE      = "memoryStorage";
    public static final String TABLE_NAME_PERSON              = "person";
    public static final String TABLE_NAME_SHAPE               = "shape";
    public static final String TABLE_NAME_SHAPE_CACHE         = "shapeCache";
@@ -204,6 +206,7 @@ public class TestUtils
 
       qInstance.addTable(defineTablePerson());
       qInstance.addTable(defineTableTwoKeys());
+      qInstance.addTable(defineTableMemoryStorage());
       qInstance.addTable(definePersonFileTable());
       qInstance.addTable(definePersonMemoryTable());
       qInstance.addTable(definePersonMemoryCacheTable());
@@ -594,6 +597,22 @@ public class TestUtils
 
 
 
+   /*******************************************************************************
+    ** Define a table in the memory store that can be used for the StorageAction
+    *******************************************************************************/
+   public static QTableMetaData defineTableMemoryStorage()
+   {
+      return new QTableMetaData()
+         .withName(TABLE_NAME_MEMORY_STORAGE)
+         .withLabel("Memory Storage")
+         .withBackendName(MEMORY_BACKEND_NAME)
+         .withPrimaryKeyField("reference")
+         .withField(new QFieldMetaData("reference", QFieldType.STRING).withIsEditable(false))
+         .withField(new QFieldMetaData("contents", QFieldType.BLOB));
+   }
+
+
+
    /*******************************************************************************
     ** Define the 'person' table used in standard tests.
     *******************************************************************************/
@@ -644,6 +663,7 @@ public class TestUtils
          .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
          .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
          .withField(new QFieldMetaData("orderNo", QFieldType.STRING))
+         .withField(new QFieldMetaData("shipToName", QFieldType.STRING).withMaxLength(200).withBehavior(ValueTooLongBehavior.ERROR))
          .withField(new QFieldMetaData("orderDate", QFieldType.DATE))
          .withField(new QFieldMetaData("storeId", QFieldType.INTEGER))
          .withField(new QFieldMetaData("total", QFieldType.DECIMAL).withDisplayFormat(DisplayFormat.CURRENCY).withFieldSecurityLock(new FieldSecurityLock()
@@ -700,7 +720,8 @@ public class TestUtils
          .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
          .withField(new QFieldMetaData("lineItemId", QFieldType.INTEGER))
          .withField(new QFieldMetaData("key", QFieldType.STRING))
-         .withField(new QFieldMetaData("value", QFieldType.STRING));
+         .withField(new QFieldMetaData("value", QFieldType.STRING))
+         .withField(new QFieldMetaData("source", QFieldType.STRING)); // doesn't really make sense, but useful to have an extra field here in some bulk-load tests
    }
 
 

qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/utils/aggregates/AggregatesTest.java

@@ -91,6 +91,57 @@ class AggregatesTest extends BaseTest
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testLong()
+   {
+      LongAggregates aggregates = new LongAggregates();
+
+      assertEquals(0, aggregates.getCount());
+      assertNull(aggregates.getMin());
+      assertNull(aggregates.getMax());
+      assertNull(aggregates.getSum());
+      assertNull(aggregates.getAverage());
+
+      aggregates.add(5L);
+      assertEquals(1, aggregates.getCount());
+      assertEquals(5, aggregates.getMin());
+      assertEquals(5, aggregates.getMax());
+      assertEquals(5, aggregates.getSum());
+      assertThat(aggregates.getAverage()).isCloseTo(new BigDecimal("5"), Offset.offset(BigDecimal.ZERO));
+
+      aggregates.add(10L);
+      assertEquals(2, aggregates.getCount());
+      assertEquals(5, aggregates.getMin());
+      assertEquals(10, aggregates.getMax());
+      assertEquals(15, aggregates.getSum());
+      assertThat(aggregates.getAverage()).isCloseTo(new BigDecimal("7.5"), Offset.offset(BigDecimal.ZERO));
+
+      aggregates.add(15L);
+      assertEquals(3, aggregates.getCount());
+      assertEquals(5, aggregates.getMin());
+      assertEquals(15, aggregates.getMax());
+      assertEquals(30, aggregates.getSum());
+      assertThat(aggregates.getAverage()).isCloseTo(new BigDecimal("10"), Offset.offset(BigDecimal.ZERO));
+
+      aggregates.add(null);
+      assertEquals(3, aggregates.getCount());
+      assertEquals(5, aggregates.getMin());
+      assertEquals(15, aggregates.getMax());
+      assertEquals(30, aggregates.getSum());
+      assertThat(aggregates.getAverage()).isCloseTo(new BigDecimal("10"), Offset.offset(BigDecimal.ZERO));
+
+      assertEquals(new BigDecimal("750"), aggregates.getProduct());
+      assertEquals(new BigDecimal("25.0000"), aggregates.getVariance());
+      assertEquals(new BigDecimal("5.0000"), aggregates.getStandardDeviation());
+      assertThat(aggregates.getVarP()).isCloseTo(new BigDecimal("16.6667"), Offset.offset(new BigDecimal(".0001")));
+      assertThat(aggregates.getStdDevP()).isCloseTo(new BigDecimal("4.0824"), Offset.offset(new BigDecimal(".0001")));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/backend/javalin/QJavalinProcessHandler.java

@@ -26,6 +26,7 @@ import java.io.File;
 import java.io.FileInputStream;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.OutputStream;
 import java.io.PipedOutputStream;
 import java.io.Serializable;
 import java.time.LocalDate;
@@ -50,24 +51,22 @@ import com.kingsrook.qqq.backend.core.actions.processes.CancelProcessAction;
 import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallback;
 import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
 import com.kingsrook.qqq.backend.core.actions.reporting.GenerateReportAction;
-import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.StorageAction;
 import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
 import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
 import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessState;
-import com.kingsrook.qqq.backend.core.model.actions.processes.QUploadedFile;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
 import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
-import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
@@ -79,9 +78,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMeta
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.reporting.QReportMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
-import com.kingsrook.qqq.backend.core.state.StateType;
-import com.kingsrook.qqq.backend.core.state.TempFileStateProvider;
-import com.kingsrook.qqq.backend.core.state.UUIDAndTypeStateKey;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.ExceptionUtils;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
@@ -93,6 +89,7 @@ import io.javalin.apibuilder.EndpointGroup;
 import io.javalin.http.Context;
 import io.javalin.http.UploadedFile;
 import org.apache.commons.lang.NotImplementedException;
+import org.apache.commons.lang3.BooleanUtils;
 import org.eclipse.jetty.http.HttpStatus;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 import static io.javalin.apibuilder.ApiBuilder.get;
@@ -325,7 +322,7 @@ public class QJavalinProcessHandler
     *******************************************************************************/
    public static void processInit(Context context)
    {
-      doProcessInitOrStep(context, null, null, RunProcessInput.FrontendStepBehavior.BREAK);
+      doProcessInitOrStep(context, null, null, null, RunProcessInput.FrontendStepBehavior.BREAK);
    }
 
 
@@ -339,7 +336,7 @@ public class QJavalinProcessHandler
     *******************************************************************************/
    public static void processRun(Context context)
    {
-      doProcessInitOrStep(context, null, null, RunProcessInput.FrontendStepBehavior.SKIP);
+      doProcessInitOrStep(context, null, null, null, RunProcessInput.FrontendStepBehavior.SKIP);
    }
 
 
@@ -347,7 +344,7 @@ public class QJavalinProcessHandler
    /*******************************************************************************
     **
     *******************************************************************************/
-   private static void doProcessInitOrStep(Context context, String processUUID, String startAfterStep, RunProcessInput.FrontendStepBehavior frontendStepBehavior)
+   private static void doProcessInitOrStep(Context context, String processUUID, String startAfterStep, String startAtStep, RunProcessInput.FrontendStepBehavior frontendStepBehavior)
    {
       Map<String, Object> resultForCaller    = new HashMap<>();
       Exception           returningException = null;
@@ -361,8 +358,23 @@ public class QJavalinProcessHandler
          resultForCaller.put("processUUID", processUUID);
 
          String processName = context.pathParam("processName");
-         LOG.info(startAfterStep == null ? "Initiating process [" + processName + "] [" + processUUID + "]"
-            : "Resuming process [" + processName + "] [" + processUUID + "] after step [" + startAfterStep + "]");
+
+         if(startAfterStep == null && startAtStep == null)
+         {
+            LOG.info("Initiating process [" + processName + "] [" + processUUID + "]");
+         }
+         else if(startAfterStep != null)
+         {
+            LOG.info("Resuming process [" + processName + "] [" + processUUID + "] after step [" + startAfterStep + "]");
+         }
+         else if(startAtStep != null)
+         {
+            LOG.info("Resuming process [" + processName + "] [" + processUUID + "] at step [" + startAtStep + "]");
+         }
+         else
+         {
+            LOG.warn("A logical impossibility was reached, regarding the nullity of startAfterStep and startAtStep, at least given how this code was originally written.");
+         }
 
          RunProcessInput runProcessInput = new RunProcessInput();
          QJavalinImplementation.setupSession(context, runProcessInput);
@@ -371,11 +383,13 @@ public class QJavalinProcessHandler
          runProcessInput.setFrontendStepBehavior(frontendStepBehavior);
          runProcessInput.setProcessUUID(processUUID);
          runProcessInput.setStartAfterStep(startAfterStep);
+         runProcessInput.setStartAtStep(startAtStep);
          populateRunProcessRequestWithValuesFromContext(context, runProcessInput);
 
          String reportName = ValueUtils.getValueAsString(runProcessInput.getValue("reportName"));
          QJavalinAccessLogger.logStart(startAfterStep == null ? "processInit" : "processStep", logPair("processName", processName), logPair("processUUID", processUUID),
             StringUtils.hasContent(startAfterStep) ? logPair("startAfterStep", startAfterStep) : null,
+            StringUtils.hasContent(startAtStep) ? logPair("startAtStep", startAfterStep) : null,
             StringUtils.hasContent(reportName) ? logPair("reportName", reportName) : null);
 
          //////////////////////////////////////////////////////////////////////////////////////////////////
@@ -464,6 +478,7 @@ public class QJavalinProcessHandler
       }
       resultForCaller.put("values", runProcessOutput.getValues());
       runProcessOutput.getProcessState().getNextStepName().ifPresent(nextStep -> resultForCaller.put("nextStep", nextStep));
+      runProcessOutput.getProcessState().getBackStepName().ifPresent(backStep -> resultForCaller.put("backStep", backStep));
 
       /////////////////////////////////////////////////////////////////////////////////////////////////////////////
       // todo - delete after all frontends look for processMetaDataAdjustment instead of updatedFrontendStepList //
@@ -510,7 +525,7 @@ public class QJavalinProcessHandler
     ** todo - make query params have a "field-" type of prefix??
     **
     *******************************************************************************/
-   private static void populateRunProcessRequestWithValuesFromContext(Context context, RunProcessInput runProcessInput) throws IOException
+   private static void populateRunProcessRequestWithValuesFromContext(Context context, RunProcessInput runProcessInput) throws IOException, QException
    {
       //////////////////////////
       // process query string //
@@ -541,20 +556,42 @@ public class QJavalinProcessHandler
       ////////////////////////////
       // process uploaded files //
       ////////////////////////////
-      for(UploadedFile uploadedFile : context.uploadedFiles())
+      for(Map.Entry<String, List<UploadedFile>> entry : context.uploadedFileMap().entrySet())
       {
-         try(InputStream content = uploadedFile.content())
+         String                  name          = entry.getKey();
+         List<UploadedFile>      uploadedFiles = entry.getValue();
+         ArrayList<StorageInput> storageInputs = new ArrayList<>();
+         runProcessInput.addValue(name, storageInputs);
+
+         String storageTableName = QJavalinImplementation.javalinMetaData.getUploadedFileArchiveTableName();
+         if(!StringUtils.hasContent(storageTableName))
          {
-            QUploadedFile qUploadedFile = new QUploadedFile();
-            qUploadedFile.setBytes(content.readAllBytes());
-            qUploadedFile.setFilename(uploadedFile.filename());
+            throw (new QException("UploadFileArchiveTableName was not specified in javalinMetaData.  Cannot accept file uploads."));
+         }
 
-            UUIDAndTypeStateKey key = new UUIDAndTypeStateKey(StateType.UPLOADED_FILE);
-            TempFileStateProvider.getInstance().put(key, qUploadedFile);
-            LOG.info("Stored uploaded file in TempFileStateProvider under key: " + key);
-            runProcessInput.addValue(QUploadedFile.DEFAULT_UPLOADED_FILE_FIELD_NAME, key);
+         for(UploadedFile uploadedFile : uploadedFiles)
+         {
+            String reference = QValueFormatter.formatDate(LocalDate.now())
+                               + File.separator + runProcessInput.getProcessName()
+                               + File.separator + UUID.randomUUID()
+                               + File.separator + uploadedFile.filename();
 
-            archiveUploadedFile(runProcessInput, qUploadedFile);
+            StorageInput storageInput = new StorageInput(storageTableName).withReference(reference);
+            storageInputs.add(storageInput);
+
+            try
+               (
+                  InputStream content = uploadedFile.content();
+                  OutputStream outputStream = new StorageAction().createOutputStream(storageInput);
+               )
+            {
+               content.transferTo(outputStream);
+               LOG.info("Streamed uploaded file", logPair("storageTable", storageTableName), logPair("reference", reference), logPair("processName", runProcessInput.getProcessName()), logPair("uploadFileName", uploadedFile.filename()));
+            }
+            catch(QException e)
+            {
+               throw (new QException("Error creating output stream in table [" + storageTableName + "] for storage action", e));
+            }
          }
       }
 
@@ -585,27 +622,6 @@ public class QJavalinProcessHandler
 
 
 
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   private static void archiveUploadedFile(RunProcessInput runProcessInput, QUploadedFile qUploadedFile)
-   {
-      String fileName = QValueFormatter.formatDate(LocalDate.now())
-         + File.separator + runProcessInput.getProcessName()
-         + File.separator + qUploadedFile.getFilename();
-
-      InsertInput insertInput = new InsertInput();
-      insertInput.setTableName(QJavalinImplementation.javalinMetaData.getUploadedFileArchiveTableName());
-      insertInput.setRecords(List.of(new QRecord()
-         .withValue("fileName", fileName)
-         .withValue("contents", qUploadedFile.getBytes())
-      ));
-
-      new InsertAction().executeAsync(insertInput);
-   }
-
-
-
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -663,8 +679,19 @@ public class QJavalinProcessHandler
    public static void processStep(Context context)
    {
       String processUUID = context.pathParam("processUUID");
-      String lastStep    = context.pathParam("step");
-      doProcessInitOrStep(context, processUUID, lastStep, RunProcessInput.FrontendStepBehavior.BREAK);
+
+      String startAfterStep = null;
+      String startAtStep = null;
+      if(BooleanUtils.isTrue(ValueUtils.getValueAsBoolean(context.queryParam("isStepBack"))))
+      {
+         startAtStep = context.pathParam("step");
+      }
+      else
+      {
+         startAfterStep = context.pathParam("step");
+      }
+
+      doProcessInitOrStep(context, processUUID, startAfterStep, startAtStep, RunProcessInput.FrontendStepBehavior.BREAK);
    }
 
 

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/executors/io/ProcessInitOrStepInput.java

@@ -45,6 +45,7 @@ public class ProcessInitOrStepInput extends AbstractMiddlewareInput
    /////////////////////////////////////
    private String processUUID;
    private String startAfterStep;
+   // todo - add (in next version?) startAtStep (for back)
 
    private RunProcessInput.FrontendStepBehavior frontendStepBehavior = RunProcessInput.FrontendStepBehavior.BREAK;
 

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/executors/io/ProcessInitOrStepOrStatusOutputInterface.java

@@ -58,6 +58,8 @@ public interface ProcessInitOrStepOrStatusOutputInterface extends AbstractMiddle
     *******************************************************************************/
    void setNextStep(String nextStep);
 
+   // todo - add (in next version?) backStep
+
    /*******************************************************************************
     ** Setter for values
     *******************************************************************************/

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/schemabuilder/SchemaBuilder.java

@@ -38,6 +38,7 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIDescription;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIEnumSubSet;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIExclude;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIHasAdditionalProperties;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIIncludeProperties;
@@ -123,7 +124,25 @@ public class SchemaBuilder
       if(c.isEnum())
       {
          schema.withType(Type.STRING);
-         schema.withEnumValues(Arrays.stream(c.getEnumConstants()).map(e -> String.valueOf(e)).collect(Collectors.toList()));
+
+         if(element.isAnnotationPresent(OpenAPIEnumSubSet.class))
+         {
+            try
+            {
+               OpenAPIEnumSubSet enumSubSetAnnotation = element.getAnnotation(OpenAPIEnumSubSet.class);
+               Class<? extends OpenAPIEnumSubSet.EnumSubSet<?>> enumSubSetClass = enumSubSetAnnotation.value();
+               OpenAPIEnumSubSet.EnumSubSet<?> enumSubSetContainer = enumSubSetClass.getConstructor().newInstance();
+               schema.withEnumValues(enumSubSetContainer.getSubSet().stream().map(e -> String.valueOf(e)).collect(Collectors.toList()));
+            }
+            catch(Exception e)
+            {
+               throw new QRuntimeException("Error processing OpenAPIEnumSubSet on element: " + element, e);
+            }
+         }
+         else
+         {
+            schema.withEnumValues(Arrays.stream(c.getEnumConstants()).map(e -> String.valueOf(e)).collect(Collectors.toList()));
+         }
       }
       else if(c.equals(String.class))
       {

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/specs/v1/responses/components/FieldAdornment.java

@@ -0,0 +1,115 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2024.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+package com.kingsrook.qqq.middleware.javalin.specs.v1.responses.components;
+
+
+import java.io.Serializable;
+import java.util.EnumSet;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.ToSchema;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIDescription;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIEnumSubSet;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIExclude;
+
+
+/*******************************************************************************
+ **
+ *******************************************************************************/
+public class FieldAdornment implements ToSchema
+{
+   @OpenAPIExclude()
+   private com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment wrapped;
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public FieldAdornment(com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment wrapped)
+   {
+      this.wrapped = wrapped;
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor
+    **
+    *******************************************************************************/
+   public FieldAdornment()
+   {
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static class FieldAdornmentSubSet implements OpenAPIEnumSubSet.EnumSubSet<AdornmentType>
+   {
+      private static EnumSet<AdornmentType> subSet = null;
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public EnumSet<AdornmentType> getSubSet()
+      {
+         if(subSet == null)
+         {
+            EnumSet<AdornmentType> subSet = EnumSet.allOf(AdornmentType.class);
+            subSet.remove(AdornmentType.FILE_UPLOAD); // todo - remove for next version!
+            FieldAdornmentSubSet.subSet = subSet;
+         }
+
+         return (subSet);
+      }
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @OpenAPIDescription("Type of this adornment")
+   @OpenAPIEnumSubSet(FieldAdornmentSubSet.class)
+   public AdornmentType getType()
+   {
+      return (this.wrapped == null || this.wrapped.getType() == null ? null : this.wrapped.getType());
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   @OpenAPIDescription("Values associated with this adornment.  Keys and the meanings of their values will differ by type.")
+   public Map<String, Serializable> getValues()
+   {
+      return (this.wrapped.getValues());
+   }
+
+}

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/specs/v1/responses/components/FieldMetaData.java

@@ -23,7 +23,6 @@ package com.kingsrook.qqq.middleware.javalin.specs.v1.responses.components;
 
 
 import java.util.List;
-import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.ToSchema;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIDescription;
@@ -61,6 +60,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -82,6 +82,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -92,6 +93,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -102,6 +104,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -112,6 +115,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -122,6 +126,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -143,6 +148,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -153,6 +159,7 @@ public class FieldMetaData implements ToSchema
    }
 
 
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -166,6 +173,8 @@ public class FieldMetaData implements ToSchema
 
    // todo - inline PVS
 
+
+
    /***************************************************************************
     **
     ***************************************************************************/
@@ -177,14 +186,17 @@ public class FieldMetaData implements ToSchema
 
    // todo behaviors?
 
+
+
+
    /***************************************************************************
     **
     ***************************************************************************/
    @OpenAPIDescription("Special UI dressings to add to the field.")
-   @OpenAPIListItems(value = FieldAdornment.class) // todo!
+   @OpenAPIListItems(value = FieldAdornment.class, useRef = true)
    public List<FieldAdornment> getAdornments()
    {
-      return (this.wrapped.getAdornments());
+      return (this.wrapped.getAdornments() == null ? null : this.wrapped.getAdornments().stream().map(a -> new FieldAdornment(a)).toList());
    }
 
    // todo help content

qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/middleware/javalin/specs/v1/responses/components/FrontendComponent.java

@@ -23,11 +23,13 @@ package com.kingsrook.qqq.middleware.javalin.specs.v1.responses.components;
 
 
 import java.io.Serializable;
+import java.util.EnumSet;
 import java.util.Map;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.ToSchema;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIDescription;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIEnumSubSet;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIExclude;
 import com.kingsrook.qqq.middleware.javalin.schemabuilder.annotations.OpenAPIMapKnownEntries;
 
@@ -63,10 +65,39 @@ public class FrontendComponent implements ToSchema
 
 
 
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public static class QComponentTypeSubSet implements OpenAPIEnumSubSet.EnumSubSet<QComponentType>
+   {
+      private static EnumSet<QComponentType> subSet = null;
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public EnumSet<QComponentType> getSubSet()
+      {
+         if(subSet == null)
+         {
+            EnumSet<QComponentType> subSet = EnumSet.allOf(QComponentType.class);
+            subSet.remove(QComponentType.BULK_LOAD_FILE_MAPPING_FORM);
+            subSet.remove(QComponentType.BULK_LOAD_VALUE_MAPPING_FORM);
+            subSet.remove(QComponentType.BULK_LOAD_PROFILE_FORM);
+            QComponentTypeSubSet.subSet = subSet;
+         }
+
+         return(subSet);
+      }
+   }
+
+
+
    /***************************************************************************
     **
     ***************************************************************************/
    @OpenAPIDescription("The type of this component.  e.g., what kind of UI element(s) should be presented to the user.")
+   @OpenAPIEnumSubSet(QComponentTypeSubSet.class)
    public QComponentType getType()
    {
       return (this.wrapped.getType());

qqq-middleware-javalin/src/main/resources/openapi/v1/openapi.yaml

@@ -130,26 +130,31 @@ components:
           description: "Description of the error"
           type: "string"
       type: "object"
+    FieldAdornment:
+      properties:
+        type:
+          description: "Type of this adornment"
+          enum:
+          - "LINK"
+          - "CHIP"
+          - "SIZE"
+          - "CODE_EDITOR"
+          - "RENDER_HTML"
+          - "REVEAL"
+          - "FILE_DOWNLOAD"
+          - "ERROR"
+          type: "string"
+        values:
+          description: "Values associated with this adornment.  Keys and the meanings\
+            \ of their values will differ by type."
+          type: "object"
+      type: "object"
     FieldMetaData:
       properties:
         adornments:
           description: "Special UI dressings to add to the field."
           items:
-            properties:
-              type:
-                enum:
-                - "LINK"
-                - "CHIP"
-                - "SIZE"
-                - "CODE_EDITOR"
-                - "RENDER_HTML"
-                - "REVEAL"
-                - "FILE_DOWNLOAD"
-                - "ERROR"
-                type: "string"
-              values:
-                type: "object"
-            type: "object"
+            $ref: "#/components/schemas/FieldAdornment"
           type: "array"
         defaultValue:
           description: "Default value to use in this field."
@@ -973,21 +978,7 @@ components:
                   adornments:
                     description: "Special UI dressings to add to the field."
                     items:
-                      properties:
-                        type:
-                          enum:
-                          - "LINK"
-                          - "CHIP"
-                          - "SIZE"
-                          - "CODE_EDITOR"
-                          - "RENDER_HTML"
-                          - "REVEAL"
-                          - "FILE_DOWNLOAD"
-                          - "ERROR"
-                          type: "string"
-                        values:
-                          type: "object"
-                      type: "object"
+                      $ref: "#/components/schemas/FieldAdornment"
                     type: "array"
                   defaultValue:
                     description: "Default value to use in this field."
@@ -1483,6 +1474,8 @@ paths:
                     processes:
                       person.bulkInsert:
                         hasPermission: true
+                        icon:
+                          name: "library_add"
                         isHidden: true
                         label: "Person Bulk Insert"
                         name: "person.bulkInsert"

qqq-sample-project/pom.xml

@@ -74,7 +74,6 @@
          <groupId>com.h2database</groupId>
          <artifactId>h2</artifactId>
          <version>2.2.220</version>
-         <scope>test</scope>
       </dependency>
 
       <!-- 3rd party deps specifically for this module -->

qqq-sample-project/src/main/java/com/kingsrook/sampleapp/SampleJavalinServer.java

@@ -25,6 +25,7 @@ package com.kingsrook.sampleapp;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.javalin.QJavalinImplementation;
+import com.kingsrook.qqq.backend.javalin.QJavalinMetaData;
 import com.kingsrook.sampleapp.metadata.SampleMetaDataProvider;
 import io.javalin.Javalin;
 
@@ -63,7 +64,12 @@ public class SampleJavalinServer
       {
          qInstance = SampleMetaDataProvider.defineInstance();
 
+         SampleMetaDataProvider.primeTestDatabase("prime-test-database.sql");
+
          QJavalinImplementation qJavalinImplementation = new QJavalinImplementation(qInstance);
+         qJavalinImplementation.setJavalinMetaData(new QJavalinMetaData()
+            .withUploadedFileArchiveTableName(SampleMetaDataProvider.UPLOAD_FILE_ARCHIVE_TABLE_NAME));
+
          javalinService = Javalin.create(config ->
          {
             config.router.apiBuilder(qJavalinImplementation.getRoutes());

qqq-sample-project/src/main/java/com/kingsrook/sampleapp/metadata/SampleMetaDataProvider.java

@@ -22,7 +22,10 @@
 package com.kingsrook.sampleapp.metadata;
 
 
+import java.io.InputStream;
 import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.sql.Connection;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -37,7 +40,10 @@ import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInpu
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
 import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerHelper;
 import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
 import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.branding.QBrandingMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
@@ -48,8 +54,13 @@ import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QuickSightChartMe
 import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
@@ -58,6 +69,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMet
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionOutputMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QRecordListMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
@@ -70,9 +82,13 @@ import com.kingsrook.qqq.backend.module.filesystem.base.model.metadata.Cardinali
 import com.kingsrook.qqq.backend.module.filesystem.base.model.metadata.RecordFormat;
 import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemBackendMetaData;
 import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemTableBackendDetails;
+import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3TableBackendDetails;
+import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager;
+import com.kingsrook.qqq.backend.module.rdbms.jdbc.QueryManager;
 import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
 import com.kingsrook.sampleapp.dashboard.widgets.PersonsByCreateDateBarChart;
 import com.kingsrook.sampleapp.processes.clonepeople.ClonePeopleTransformStep;
+import org.apache.commons.io.IOUtils;
 
 
 /*******************************************************************************
@@ -80,7 +96,7 @@ import com.kingsrook.sampleapp.processes.clonepeople.ClonePeopleTransformStep;
  *******************************************************************************/
 public class SampleMetaDataProvider
 {
-   public static boolean USE_MYSQL = true;
+   public static boolean USE_MYSQL = false;
 
    public static final String RDBMS_BACKEND_NAME      = "rdbms";
    public static final String FILESYSTEM_BACKEND_NAME = "filesystem";
@@ -97,6 +113,7 @@ public class SampleMetaDataProvider
    public static final String PROCESS_NAME_SLEEP_INTERACTIVE = "sleepInteractive";
 
    public static final String TABLE_NAME_PERSON  = "person";
+   public static final String TABLE_NAME_PET     = "pet";
    public static final String TABLE_NAME_CARRIER = "carrier";
    public static final String TABLE_NAME_CITY    = "city";
 
@@ -106,6 +123,9 @@ public class SampleMetaDataProvider
    public static final String SCREEN_0 = "screen0";
    public static final String SCREEN_1 = "screen1";
 
+   public static final String BACKEND_NAME_UPLOAD_ARCHIVE    = "uploadArchive";
+   public static final String UPLOAD_FILE_ARCHIVE_TABLE_NAME = "uploadFileArchive";
+
 
 
    /*******************************************************************************
@@ -120,6 +140,10 @@ public class SampleMetaDataProvider
       qInstance.addBackend(defineFilesystemBackend());
       qInstance.addTable(defineTableCarrier());
       qInstance.addTable(defineTablePerson());
+      qInstance.addPossibleValueSource(QPossibleValueSource.newForTable(TABLE_NAME_PERSON));
+      qInstance.addPossibleValueSource(QPossibleValueSource.newForEnum(PetSpecies.NAME, PetSpecies.values()));
+      qInstance.addTable(defineTablePet());
+      qInstance.addJoin(defineTablePersonJoinPet());
       qInstance.addTable(defineTableCityFile());
       qInstance.addProcess(defineProcessGreetPeople());
       qInstance.addProcess(defineProcessGreetPeopleInteractive());
@@ -128,6 +152,9 @@ public class SampleMetaDataProvider
       qInstance.addProcess(defineProcessScreenThenSleep());
       qInstance.addProcess(defineProcessSimpleThrow());
 
+      qInstance.add(defineUploadArchiveBackend());
+      qInstance.add(defineTableUploadFileArchive());
+
       MetaDataProducerHelper.processAllMetaDataProducersInPackage(qInstance, SampleMetaDataProvider.class.getPackageName());
 
       defineWidgets(qInstance);
@@ -139,6 +166,26 @@ public class SampleMetaDataProvider
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static void primeTestDatabase(String sqlFileName) throws Exception
+   {
+      try(Connection connection = ConnectionManager.getConnection(SampleMetaDataProvider.defineRdbmsBackend()))
+      {
+         InputStream  primeTestDatabaseSqlStream = SampleMetaDataProvider.class.getResourceAsStream("/" + sqlFileName);
+         List<String> lines                      = IOUtils.readLines(primeTestDatabaseSqlStream, StandardCharsets.UTF_8);
+         lines = lines.stream().filter(line -> !line.startsWith("-- ")).toList();
+         String joinedSQL = String.join("\n", lines);
+         for(String sql : joinedSQL.split(";"))
+         {
+            QueryManager.executeUpdate(connection, sql);
+         }
+      }
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -192,6 +239,7 @@ public class SampleMetaDataProvider
          .withIcon(new QIcon().withName("emoji_people"))
          .withChild(qInstance.getProcess(PROCESS_NAME_GREET).withIcon(new QIcon().withName("emoji_people")))
          .withChild(qInstance.getTable(TABLE_NAME_PERSON).withIcon(new QIcon().withName("person")))
+         .withChild(qInstance.getTable(TABLE_NAME_PET).withIcon(new QIcon().withName("pets")))
          .withChild(qInstance.getTable(TABLE_NAME_CITY).withIcon(new QIcon().withName("location_city")))
          .withChild(qInstance.getProcess(PROCESS_NAME_GREET_INTERACTIVE).withIcon(new QIcon().withName("waving_hand")))
          .withWidgets(List.of(PersonsByCreateDateBarChart.class.getSimpleName(), QuickSightChartRenderer.class.getSimpleName()))
@@ -328,7 +376,7 @@ public class SampleMetaDataProvider
          .withField(new QFieldMetaData("firstName", QFieldType.STRING).withBackendName("first_name").withIsRequired(true))
          .withField(new QFieldMetaData("lastName", QFieldType.STRING).withBackendName("last_name").withIsRequired(true))
          .withField(new QFieldMetaData("birthDate", QFieldType.DATE).withBackendName("birth_date"))
-         .withField(new QFieldMetaData("email", QFieldType.STRING))
+         .withField(new QFieldMetaData("email", QFieldType.STRING).withIsRequired(true))
          .withField(new QFieldMetaData("isEmployed", QFieldType.BOOLEAN).withBackendName("is_employed"))
          .withField(new QFieldMetaData("annualSalary", QFieldType.DECIMAL).withBackendName("annual_salary").withDisplayFormat(DisplayFormat.CURRENCY))
          .withField(new QFieldMetaData("daysWorked", QFieldType.INTEGER).withBackendName("days_worked").withDisplayFormat(DisplayFormat.COMMAS))
@@ -340,11 +388,62 @@ public class SampleMetaDataProvider
 
       QInstanceEnricher.setInferredFieldBackendNames(qTableMetaData);
 
+      qTableMetaData.withAssociation(new Association()
+         .withAssociatedTableName(TABLE_NAME_PET)
+         .withName("pets")
+         .withJoinName(QJoinMetaData.makeInferredJoinName(TABLE_NAME_PERSON, TABLE_NAME_PET)));
+
       return (qTableMetaData);
    }
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QTableMetaData defineTablePet()
+   {
+      QTableMetaData qTableMetaData = new QTableMetaData()
+         .withName(TABLE_NAME_PET)
+         .withLabel("Pet")
+         .withBackendName(RDBMS_BACKEND_NAME)
+         .withPrimaryKeyField("id")
+         .withRecordLabelFormat("%s %s")
+         .withRecordLabelFields("name")
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
+         .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("create_date").withIsEditable(false))
+         .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBackendName("modify_date").withIsEditable(false))
+         .withField(new QFieldMetaData("name", QFieldType.STRING).withBackendName("name").withIsRequired(true))
+         .withField(new QFieldMetaData("personId", QFieldType.INTEGER).withBackendName("person_id").withIsRequired(true).withPossibleValueSourceName(TABLE_NAME_PERSON))
+         .withField(new QFieldMetaData("speciesId", QFieldType.INTEGER).withBackendName("species_id").withIsRequired(true).withPossibleValueSourceName(PetSpecies.NAME))
+         .withField(new QFieldMetaData("birthDate", QFieldType.DATE).withBackendName("birth_date"))
+
+         .withSection(new QFieldSection("identity", "Identity", new QIcon("badge"), Tier.T1, List.of("id", "name")))
+         .withSection(new QFieldSection("basicInfo", "Basic Info", new QIcon("dataset"), Tier.T2, List.of("personId", "speciesId", "birthDate")))
+         .withSection(new QFieldSection("dates", "Dates", new QIcon("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+
+      QInstanceEnricher.setInferredFieldBackendNames(qTableMetaData);
+
+      return (qTableMetaData);
+   }
+
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   private static QJoinMetaData defineTablePersonJoinPet()
+   {
+      return new QJoinMetaData()
+         .withLeftTable(TABLE_NAME_PERSON)
+         .withRightTable(TABLE_NAME_PET)
+         .withInferredName()
+         .withType(JoinType.ONE_TO_MANY)
+         .withJoinOn(new JoinOn("id", "personId"));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/
@@ -524,6 +623,40 @@ public class SampleMetaDataProvider
 
 
 
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public static QBackendMetaData defineUploadArchiveBackend()
+   {
+      return new FilesystemBackendMetaData()
+         .withName(BACKEND_NAME_UPLOAD_ARCHIVE)
+         .withBasePath("/tmp/" + BACKEND_NAME_UPLOAD_ARCHIVE);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static QTableMetaData defineTableUploadFileArchive()
+   {
+      return (new QTableMetaData()
+         .withName(UPLOAD_FILE_ARCHIVE_TABLE_NAME)
+         .withBackendName(BACKEND_NAME_UPLOAD_ARCHIVE)
+         .withPrimaryKeyField("fileName")
+         // .withSupplementalMetaData(new ApiTableMetaDataContainer()) // empty container means no apis for this table
+         .withField(new QFieldMetaData("fileName", QFieldType.STRING))
+         .withField(new QFieldMetaData("contents", QFieldType.BLOB))
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
+         .withBackendDetails(new S3TableBackendDetails()
+            .withCardinality(Cardinality.ONE)
+            .withFileNameFieldName("fileName")
+            .withContentsFieldName("contents")
+            .withBasePath("upload-file-archive")));
+   }
+
+
+
    /*******************************************************************************
     ** Testing backend step - just sleeps however long you ask it to (or, throws if
     ** you don't provide a number of seconds to sleep).
@@ -625,4 +758,53 @@ public class SampleMetaDataProvider
       }
    }
 
+
+
+   /***************************************************************************
+    **
+    ***************************************************************************/
+   public enum PetSpecies implements PossibleValueEnum<Integer>
+   {
+      DOG(1, "Dog"),
+      CAT(1, "Cat");
+
+      private final Integer id;
+      private final String  label;
+
+      public static final String NAME = "petSpecies";
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      PetSpecies(int id, String label)
+      {
+         this.id = id;
+         this.label = label;
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public Integer getPossibleValueId()
+      {
+         return (id);
+      }
+
+
+
+      /***************************************************************************
+       **
+       ***************************************************************************/
+      @Override
+      public String getPossibleValueLabel()
+      {
+         return (label);
+      }
+   }
+
 }

qqq-sample-project/src/main/resources/prime-test-database.sql

@@ -42,6 +42,27 @@ INSERT INTO person (id, first_name, last_name, birth_date, email, is_employed, a
 INSERT INTO person (id, first_name, last_name, birth_date, email, is_employed, annual_salary, days_worked) VALUES (4, 'Tyler', 'Samples', NULL, 'tsamples@mmltholdings.com', 1, 950000, 75);
 INSERT INTO person (id, first_name, last_name, birth_date, email, is_employed, annual_salary, days_worked) VALUES (5, 'Garret', 'Richardson', '1981-01-01', 'grichardson@mmltholdings.com', 0, 1500000, 1);
 
+DROP TABLE IF EXISTS pet;
+CREATE TABLE pet
+(
+   id INT AUTO_INCREMENT primary key ,
+   create_date TIMESTAMP DEFAULT now(),
+   modify_date TIMESTAMP DEFAULT now(),
+
+   name VARCHAR(80) NOT NULL,
+   species_id INTEGER NOT NULL,
+   person_id INTEGER NOT NULL,
+   birth_date DATE
+);
+
+INSERT INTO pet (id, name, species_id, person_id) VALUES (1, 'Charlie', 1, 1);
+INSERT INTO pet (id, name, species_id, person_id) VALUES (2, 'Coco', 1, 1);
+INSERT INTO pet (id, name, species_id, person_id) VALUES (3, 'Louie', 1, 1);
+INSERT INTO pet (id, name, species_id, person_id) VALUES (4, 'Barkley', 1, 1);
+INSERT INTO pet (id, name, species_id, person_id) VALUES (5, 'Toby', 1, 2);
+INSERT INTO pet (id, name, species_id, person_id) VALUES (6, 'Mae', 2, 3);
+
+
 DROP TABLE IF EXISTS carrier;
 CREATE TABLE carrier
 (