Compare commits

...

1129 Commits

Author SHA1 Message Date
8e8d3b5d2b downgraded some loggly infos to debugs to stop filling up 2024-02-06 14:55:38 -06:00
4ca9c9dcaf Merge branch 'feature/create-and-modify-date-as-field-behaviors' into dev 2024-02-05 16:00:08 -06:00
c6a58ac68f added tests to StringUtils.safeAppend() 2024-02-01 16:00:45 -06:00
bb69dddb81 Merge pull request #64 from Kingsrook/feature/CE-779-order-level-ship-date
Feature/ce 779 order level ship date
2024-02-01 12:07:29 -06:00
052cb9fab9 Merge branch 'dev' into feature/CE-779-order-level-ship-date 2024-02-01 11:28:40 -06:00
2681d66b32 CE-779: updated all addHeader calls to setHeader to avoid duplicate entries 2024-02-01 11:13:30 -06:00
612370fc13 Fix the check for primary key of integer (to work for null primary keys, and to be inside the try-catch); tests on that 2024-01-31 16:16:14 -06:00
74d66d0fa5 Add Log for missing value in sourceKeyField 2024-01-31 11:14:50 -06:00
08e14ac346 CE-781 - remove unintended changes from last commit 2024-01-23 19:15:42 -06:00
0dd26b8f31 CE-781 - eliminate instance validation errors when using FilesystemImporterStep FIELD_IMPORT_SECURITY_VALUE_SUPPLIERs 2024-01-23 19:12:41 -06:00
7e0b7ddbf9 Merge pull request #61 from Kingsrook/feature/CE-781-fixes
Feature/ce 781 fixes
2024-01-23 15:18:43 -06:00
fb69c60e10 CE-781 - Add option to set secuirty key values in importFile & importRecord records dynamically through a QCodeReference to a Function 2024-01-23 14:44:10 -06:00
e7e93a6ab2 CE-781 - Rework api for adding automations (don't clobber if adding more than 1) 2024-01-23 14:41:43 -06:00
1baade0449 Change insert & update actions to set default values for createDate & modifyDate based on FieldBehaviors instead of based on field names (though field names are used in Enricher to add those beavhiors); Some refactoring of FieldBehaviors. 2024-01-18 11:50:40 -06:00
0eb8356759 Remove unused import 2024-01-17 20:35:05 -06:00
6098e64934 Add warn instead of silent noop in setInputFieldDefaultValue, if field not found 2024-01-17 20:34:19 -06:00
dfb1e637a3 Updated afterEach method to package-private to quiet warning 2024-01-17 20:33:33 -06:00
ce28ce2e02 Add TableSyncProcess; remove previously built htmls & pdfs 2024-01-17 20:32:56 -06:00
7109cffa27 Merge branch 'wip/asciidoc' into dev 2024-01-17 20:29:17 -06:00
6f4d44df10 Merge pull request #59 from Kingsrook/feature/CE-781
Feature/ce 781
2024-01-17 20:27:44 -06:00
84eaa09932 CE-781 Feedback from code review 2024-01-17 19:39:00 -06:00
911978c74b CE-781 Feedback from code review 2024-01-17 19:09:19 -06:00
00a5b72bf3 added string util method for appending strings 2024-01-16 14:32:40 -06:00
a845ead466 Add option to exclude an enum possible value's fields from docs 2024-01-16 10:58:20 -06:00
3ebc567299 CE-781 Fix archivePath as field on table; set maxRows 100 on child-widget; always archive files; allow security name/value; significant tests on importer step 2024-01-16 10:33:59 -06:00
494f0242ac CE-781 Remove coverage ratios = 0 - as we might be good in here now!! 2024-01-15 20:29:14 -06:00
e6e7e3f9a7 Add overloaded constructor to SyncProcessConfig (defaults doInserts & doUpdates to true) 2024-01-15 20:22:41 -06:00
6dc7a8dde9 CE-781 Add qqq-backend-module-mongodb 2024-01-15 20:22:03 -06:00
dccbed87a7 CE-781 Add javadoc 2024-01-15 20:21:48 -06:00
7b141abcec CE-781 Add logQuery, queryStats, actionTimeouts to MongoDB; fix many query operators while adding test coverage 2024-01-15 20:21:16 -06:00
252c92913c CE-781 Do not assume recordIds are integers 2024-01-12 12:31:38 -06:00
f0150a3543 Remove debug system output from previous 2024-01-12 09:40:09 -06:00
16f0a8c3a7 Add src/main/resources to build as well 2024-01-12 09:06:26 -06:00
e5c35e90a6 Add some system outs to debug test fail 2024-01-12 08:57:40 -06:00
4286001b4d Turn on including src java files in jars 2024-01-12 08:40:46 -06:00
c1ce933d6c CE-781 Temp disable coverage ratios 2024-01-12 08:18:41 -06:00
f78d9a11b2 CE-781 javadoc syntax fix 2024-01-11 10:42:58 -06:00
d0233e839b CE-781 initial set of tets for mongodb 2024-01-11 10:28:59 -06:00
e4d7797bbe CE-781 Adding fake tests to ensure class-coverage on records... 2024-01-11 08:59:54 -06:00
f66f2d622a CE-781 Adding tests for new classes 2024-01-11 08:43:55 -06:00
624a723b54 CE-781 Initial checkin of filesystem importer meta-data template and process 2024-01-11 07:46:30 -06:00
b64883f34f CE-781 rename beforeEach and afterEach (to help avoid overwriting in sub-classes) 2024-01-10 19:59:40 -06:00
4b1bdebe44 CE-781 Move backend type (name) up to public static final constant 2024-01-10 19:59:11 -06:00
8d668d12ec CE-781 Get fields using getFields().containsKey/get, rather than getField(String) - to avoid it throwing, to cut down on exceptions (and warn if we get a real exception, vs., we'll expect non-fields sometimes, so be okay with that) 2024-01-10 19:58:26 -06:00
c27a2a986a CE-781 Add cases for LinkedHashMap and HashMap in deepCopySimpleMap 2024-01-10 19:57:20 -06:00
fed8cbbb45 CE-781 Make tie-break for sorting do backends earlier than everything else 2024-01-10 19:52:22 -06:00
5147a022fa CE-781 add sortOrder attribute to apps, for sorting them... 2024-01-10 19:51:44 -06:00
615ff6fce5 CE-781 more fluent methods in process meta data builders 2024-01-10 19:51:07 -06:00
f5c4c12388 CE-781 Initial build of mongodb backend module 2024-01-08 20:00:57 -06:00
68911190fa CE-781 Updates for compatibility with corresponding changes, refactoring, in backend-core 2024-01-08 16:40:56 -06:00
56a2949911 CE-781 Remove check for empty record list (has been moved up to core InsertAction) 2024-01-08 14:18:32 -06:00
96013878bc CE-781 Update some of the getValueAs methods to take Object instead of Serializable 2024-01-08 14:17:55 -06:00
f879575b32 CE-781 Gracefully ignore request to add null uniqueKey or recordSecurityLock 2024-01-08 14:16:59 -06:00
bab3c7b374 CE-781 Initial checkin 2024-01-08 14:15:34 -06:00
1c69784897 CE-781 Add method add(TopLevelMetaDataInterface) 2024-01-08 14:15:05 -06:00
8822c1bb99 CE-781 Add overload constructor that takes Collection of values 2024-01-08 14:14:47 -06:00
a5420bff4c CE-781 Add concept of sharded automations - schedule multiple instances of job, filter implicitly by shard value 2024-01-08 14:13:44 -06:00
06259041f8 CE-781 Update to work without a table specified (just getting field names from the json keys) 2024-01-08 12:39:43 -06:00
a00d4f3cbd CE-781 Refactoring this code out of RDBMS update, to be shared with MongoDB update 2024-01-08 12:37:38 -06:00
8473e11444 CE-781 Refactoring of backend actions - moving openTransaction out of insert-action only (up to backendModule); re-using the exit-early-if-0 and set-default-create-and-modify-date logics; 2024-01-08 12:37:22 -06:00
bc3f462d13 CE-781 log (once) & noop for tables w/o integer primary key, as that is required for auditing... 2024-01-08 12:31:37 -06:00
56a2099515 CE-781 Add option to treat CSV headers as field names (rather than working with a table's fields) 2024-01-08 12:30:43 -06:00
93dcee9f61 Add QRecord as a handled type inside deepCopySimpleMap (e.g., so copy constructor won't need to warn about it and do slow serialization-based cloning). 2024-01-04 18:11:05 -06:00
92b052fe59 CE-773 avoid s3 list requests that start with / if backend & table have no basePaths 2023-12-29 19:12:34 -06:00
3f431b39b9 Merge pull request #58 from Kingsrook/feature/CE-773-cartonization-playground
CE-773 Fixing globs for local filesystem by using Files.walkFileTree.…
2023-12-29 10:42:41 -06:00
688e221f9a CE-773 Fixing globs for local filesystem by using Files.walkFileTree. Refactored to share filter matching between s3 & local fs. 2023-12-29 08:20:38 -06:00
959f8c8041 Merge pull request #55 from Kingsrook/feature/CE-773-cartonization-playground
Feature/ce 773 cartonization playground
2023-12-28 18:45:30 -06:00
6e1ea5c8f1 CE-773 fix tables created in here, per new validationing! 2023-12-28 16:46:51 -06:00
872dec3177 CE-773 change fileNameFieldName and contentsFieldName to default as null - add validation to tableBackendDetails, specifically implemented in filesystem module 2023-12-28 16:38:40 -06:00
78d9ec87a2 Merge pull request #57 from Kingsrook/feature/basepull-subtract-seconds
Add option to move timestamps, e.g., to make overlapping windows
2023-12-28 16:24:17 -06:00
01c78534ef Add test for previous commit (Add option to move timestamps, e.g., to make overlapping windows) 2023-12-28 16:20:38 -06:00
cfab10c8e8 Add option to move timestamps, e.g., to make overlapping windows 2023-12-28 15:54:44 -06:00
2da6878e70 Make sure to always return an empty list rather than a null 2023-12-28 10:33:23 -06:00
345d8022c1 CE-773 Feedback from code review 2023-12-28 10:33:23 -06:00
aff4b43296 Merged dev into feature/CE-773-cartonization-playground 2023-12-27 20:20:08 -06:00
b805e7645b CE-773 Update for compat. with previous commit, but also, fix all generics and move inputStream into try-with-resources 2023-12-27 16:11:19 -06:00
940080bc86 CE-773 update to support listing/filtering filesystem tables with Cardinality.ONE (single-record per-file) 2023-12-27 16:10:45 -06:00
6c9506d18b Merge pull request #54 from Kingsrook/feature/meta-data-producer-interface-and-is-enabled
Add isEnabled method to meta-data producers; Put interface on top of …
2023-12-22 19:03:45 -06:00
8fc2b548ee Add isEnabled method to meta-data producers; Put interface on top of MetaDataProducer, for times when someone wants that; update MetaDataProducerHelper to work w/ the interface. 2023-12-21 15:28:34 -06:00
a8c30b1bed Merge pull request #53 from Kingsrook/feature/CE-773-cartonization-playground
CE-773 Pass script revision id through
2023-12-21 11:59:36 -06:00
fd18568785 Only apply mysql result set optimization per a system property, default to false. 2023-12-20 14:18:34 -06:00
db2e5fb7fc CE-775 Add some sleep to help timeout test 2023-12-19 15:32:15 -06:00
dceb0ee142 CE-775 add timeouts to outbound http calls 2023-12-19 15:12:52 -06:00
1d022200c5 CE-773 Pass script revision id through 2023-12-18 12:43:55 -06:00
d1bfc834d6 Merge pull request #52 from Kingsrook/feature/gc-perf-debug
Feature/gc perf debug
2023-12-18 11:48:52 -06:00
5f586d30c7 Switch to do mysql optimizations if connection is com.mysql class 2023-12-18 08:45:20 -06:00
4703d3bb24 Fixed last commit (meant to use backend.vendor, not name, compare to aurora) 2023-12-16 10:27:25 -06:00
d624a42dac Checkpoint on qqq docs 2023-12-15 18:41:30 -06:00
2b90d7e4b3 Update to use mysql optimizations for statements on aurora too... 2023-12-15 18:36:17 -06:00
9144754e74 Merge pull request #51 from Kingsrook/feature/CE-752-add-information-to-order
Feature/ce 752 add information to order
2023-12-14 13:13:54 -06:00
fb80c92f73 Merge pull request #47 from Kingsrook/dependabot/maven/qqq-backend-core/org.json-json-20231013
Bump org.json:json from 20230618 to 20231013 in /qqq-backend-core
2023-12-08 16:39:57 -06:00
000a01eb88 Add getNoDifferencesNoUpdateLine() 2023-12-07 12:21:25 -06:00
376438bdc5 CE-752 Add help content concept to QQQ (table fields, sections, and process fields at this time) 2023-12-07 08:48:32 -06:00
b14d8401fa Initial checkin 2023-12-04 16:02:42 -06:00
41009a5c84 Update ValueUtils.getValueAsInteger and ValueUtils.getValueAsString, as well as RDBMS's bindParamObject to all be tolerant of an input PossibleValueEnum (which people sometimes accidentally pass in), by extracting its id 2023-12-04 16:01:50 -06:00
6f5c2c16bb change sorting to put apps after everything else, since that's often what one would want, so you don't always have to set sortOrder yourself 2023-12-01 10:58:27 -06:00
c3d35bf110 Add byte[] and ArrayList to more efficient version of deep copy / copy constructor 2023-11-30 20:18:30 -06:00
19d7559dbf update instance enricher to make children of app-sections become children of apps 2023-11-30 20:17:50 -06:00
0c6d8d23c2 Update javadoc (add 2nd reference to review screen) 2023-11-30 20:09:33 -06:00
94fcc36c64 Add byte[] as a type that we can getAsString 2023-11-30 20:09:10 -06:00
191bcdf0dd not sure what i was doing here, now checking record value correctly 2023-11-30 15:45:01 -06:00
4c1298d531 fixes for unit test where create date was being specified before insert action 2023-11-30 14:37:17 -06:00
081be690d5 enhanced memory backend somewhat 2023-11-30 14:27:15 -06:00
ff9a2c261c Fix NPE from previous commit 2023-11-30 13:39:50 -06:00
3e01491546 Fix to remove skip & limit from filter before doing count 2023-11-30 13:36:31 -06:00
50c9a3eef2 Merge pull request #45 from Kingsrook/feature/garbage-collector
Feature/garbage collector
2023-11-24 09:24:24 -06:00
2b263204b8 Merge pull request #50 from Kingsrook/fix/circleci-localstack
Try turning s3 tests w/ localstack back on (w/ new orb)
2023-11-24 09:01:34 -06:00
9e66bc0ab9 Try turning s3 tests w/ localstack back on (w/ new orb) 2023-11-24 08:30:46 -06:00
1f9921b918 Add TopLevelMetaDataInterface 2023-11-22 08:57:42 -06:00
fde84cc077 Update font to add SF Pro Display first 2023-11-21 08:19:47 -06:00
a3597a878c Add attributes to widget data classes 2023-11-21 08:19:30 -06:00
de9f15e760 Change dropdown options to be mutable map. 2023-11-21 08:18:52 -06:00
0ee0e2d4a4 Merge branch 'wip/asciidoc' into dev 2023-11-21 07:51:57 -06:00
17b2a3e0a1 CE-740 1 decimial on percents 2023-11-16 19:00:57 -06:00
7fae7054d4 Merge pull request #49 from Kingsrook/feature/misc-updates-20231115
Feature/misc updates 20231115
2023-11-15 12:02:21 -06:00
ad72ea4c23 Merge pull request #48 from Kingsrook/bugfix/thread-pools
Update to use static ThreadPoolExecutors
2023-11-15 12:00:24 -06:00
74464809c4 Fix method updateRecordsWithDisplayValuesAndPossibleValues - was acting like it used DESTINATION_TABLE, but then changing to input.getTable 2023-11-15 08:53:25 -06:00
6aa4867bba Add getStringFromPropertyOrEnvironment 2023-11-15 08:52:32 -06:00
1b58cdeb3c Move packagesToKeep to a system-property/env-var 2023-11-15 08:49:04 -06:00
d2fd0d13b5 Add method getProcessMetaData 2023-11-15 08:39:21 -06:00
c2bdcb9465 Update to use static ThreadPoolExecutors 2023-11-15 08:35:30 -06:00
caf72b605f Bump org.json:json from 20230618 to 20231013 in /qqq-backend-core
Bumps [org.json:json](https://github.com/douglascrockford/JSON-java) from 20230618 to 20231013.
- [Release notes](https://github.com/douglascrockford/JSON-java/releases)
- [Changelog](https://github.com/stleary/JSON-java/blob/master/docs/RELEASES.md)
- [Commits](https://github.com/douglascrockford/JSON-java/commits)

---
updated-dependencies:
- dependency-name: org.json:json
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-08 15:14:20 +00:00
d3de05165b Merge pull request #46 from Kingsrook/feature/CE-604-complete-shipment-sla-updates-and-local-tnt-rules
Feature/ce 604 complete shipment sla updates and local tnt rules
2023-11-08 09:13:53 -06:00
55725a6cca Update to avoid NPE if used without a callback (e.g., scheduled process) 2023-11-08 07:54:55 -06:00
724e9c024e CE-604 add backgroundColors list 2023-11-03 19:25:29 -05:00
784cfbbaf4 CE-604 Add overloads that take ZoneId so caller can work with a custom zone id instead of session zone id 2023-11-02 16:16:19 -05:00
c58d8fd7de CE-604 Let a customizer set a record label 2023-11-02 14:32:53 -05:00
9e3054381a CE-604 Add java.util.Date to the list of types we can just copy in deepCopySimpleMap (not sure where they're coming from, but we sure have 'em) 2023-11-02 12:30:50 -05:00
a2b56a8871 Fix order of args to a log call (exception was being treated as a {} fill-in, not the actual exception) 2023-11-01 12:12:08 -05:00
a7d5741d1c CE-604 expose fetching MemoizedResult (so nulls can be cached and differentiated from not-founds) 2023-11-01 12:11:36 -05:00
8b6eb63253 CE-604 Rewrite copy constructor to try to not use SerializationUtils, which was seen as a runtime dominator during profiling bulk loads 2023-10-31 08:20:48 -05:00
a16643cf88 CE-604 update most calls to logger to take supplier rather than calling makeJsonString (to avoid the cost of building the string when it won't be logged) 2023-10-31 08:20:04 -05:00
41cebbc7de Merged dev into feature/CE-604-complete-shipment-sla-updates-and-local-tnt-rules 2023-10-27 16:10:29 -05:00
ee66eae2a0 CE-604 add method newForEnum 2023-10-27 14:33:53 -05:00
2fcb0106d2 CE-604 Add properties: fixedStickyLastRow, fixedHeight 2023-10-24 11:38:06 -05:00
ff9655aceb CE-604 Reverting jackson upgrade to 2.15.2 - was causing some runtime errors - needs re-evaluated! 2023-10-24 11:37:33 -05:00
754d4594a9 CE-604 Update to put filter into process value with name expected elsewhere (lowercase "Json") 2023-10-24 11:37:08 -05:00
c2c9a3acce CE-604 ADd overload to getSqlExpression that takes zoneId 2023-10-24 11:36:49 -05:00
a19a789449 CE-604 Update test per change in ProcessSummaryWarningsAndErrorsRollup (also, fix bug in it (thanks test) where warnings were changing to errors during the doReplaceSingletonCountLinesWithSuffixOnly thing) 2023-10-20 12:33:17 -05:00
a5b18c9020 CE-604 Keep all jackson versions in sync (from earlier commit that just updated jackson-dataformat-yaml) 2023-10-20 12:29:23 -05:00
06863e97f6 CE-604 Update to recognize Yes as true. 2023-10-20 12:27:24 -05:00
a68b739147 CE-604 Add map of icons, tooltips, icon colors 2023-10-20 12:21:49 -05:00
036d26cb5b Update deps (json, jackson-dataformat-yaml) that intellij flagged as having vulnerabilities 2023-10-20 12:21:49 -05:00
b46dbf3ec4 CE-604 Add LayoutType field/enum 2023-10-20 12:21:49 -05:00
1aae13913e CE-604 Add ChartSubheaderData concept 2023-10-20 12:21:49 -05:00
a0d217ed44 CE-604 Bulk load improvements - type handling (via csv adapter) and better errors 2023-10-20 12:21:49 -05:00
e2859aeb89 Add Memoization class and supports 2023-10-16 15:10:30 -05:00
e633ea8ed1 Support searching for a localdate or instant. 2023-10-16 12:17:17 -05:00
b1c287a4e2 Moved getValueAsString call from last commit into earlier check for "%s" 2023-10-16 08:59:36 -05:00
caba27448d Bypass calling String.formatted for most common case: "%s" (seen to be a hotspot in huge exports) 2023-10-16 08:56:01 -05:00
d28426562a Update to actually use defaultValues when inserting records - nulls become the default. 2023-10-16 08:41:57 -05:00
af852b0612 Added 'scrubbing' of values being used in query filters. removed unused param from scrubValue method 2023-10-16 08:31:25 -05:00
017addc188 Initial checkin 2023-10-16 08:28:08 -05:00
8102dbc8b2 Add remove method to StateProviderInterface 2023-10-16 08:28:00 -05:00
14d0d18045 Possibly run pre-insert-customizer in here, before checking UK values (in case customizer adjusts such fields). Add 'sample values' to UK error messaging. 2023-10-16 08:22:41 -05:00
3cfdf99b43 Let pre-insert customizers specify when-to-run during pre-insert validation. e.g., if they need to adjust values before UK lookups are done. 2023-10-16 08:16:18 -05:00
118433178d Add support for instant fields as well as AbstractFilterExpressions 2023-10-16 08:15:14 -05:00
7339ad90cc Standard QQQ garbage collector process 2023-10-16 08:12:39 -05:00
61a6c0e8b0 Merge pull request #44 from Kingsrook/hotfix/export-crashes
Hotfix/export crashes
2023-10-13 10:37:01 -05:00
d905efb1c4 Update sanitize method to only call replaceAll if needed - save a ton of memory allocations (gigs) & runtime (~10%) in big exports. 2023-10-12 19:09:52 -05:00
0c2b078af9 Update to *actually* clear the internal cache when it's seen to be big. make the size a param, so test can set it. also add unused method to use transactions as a way to do connection pooling, to avoid so many new connections... 2023-10-12 19:09:12 -05:00
7ef8f9f181 Updating to 0.20.0 2023-10-11 14:13:48 -05:00
4d6c64df0c Merge tag 'version-0.19.0' into dev
Tag release
2023-10-11 14:13:44 -05:00
586f22560e Merge branch 'release/0.19.0' 2023-10-11 14:12:31 -05:00
287b0a2bc7 Update for next development version 2023-10-11 14:10:42 -05:00
c48b18ba0d Update versions for release 2023-10-11 14:10:37 -05:00
d979c985f6 Add sort (by summary) of processes - for stability (and just to be a little nicer) 2023-10-03 08:31:40 -05:00
6c2c9b83ed Micro optimization in hot-spot - setValueIfTableHasField - use fields.containsKey, rather than getField, which throws, and is expensive when so frequent 2023-09-29 17:08:24 -05:00
d4df533f5d Make charsetForEntityi a method, rather than hard-code UTF-8. 2023-09-29 17:07:45 -05:00
e89093f339 Add 'unspecifiedError' for higher-level exceptions; add primaryKeys in summary lines 2023-09-29 17:07:24 -05:00
dd57a327dd Temp disable coverage checks while localstack is failing... 2023-09-28 15:36:42 -05:00
107086094a Temp disable while localstack is failing... 2023-09-28 15:31:18 -05:00
4f896dde97 Temp disable while localstack is failing... 2023-09-28 15:20:27 -05:00
df397ee68c Temp disable localstack/startup, as it's failing... 2023-09-28 15:15:49 -05:00
c452305a99 Merge branch 'feature/outbound-api-use-utf-8' into feature/CE-680-push-tracking-data-to-ship-station 2023-09-28 14:53:58 -05:00
fe8af54ee5 Add getMessage to ProcessSummaryLineInterface 2023-09-28 14:53:02 -05:00
ac37e3492b Fix NPE if no unique keys 2023-09-28 14:52:51 -05:00
7160b87048 Add method willTheBasePullQueryBeUsed 2023-09-27 19:50:30 -05:00
a30d8cb490 simplify getProcessSummary by using StandardProcessSummaryLineProducer; don't add records to okTo{insert,update} summaries if populateRecordToStore returns null 2023-09-27 19:46:27 -05:00
582d375597 Add constructors 2023-09-27 16:21:06 -05:00
687c5fce41 Add method addAuditForExecuteStep 2023-09-27 16:20:57 -05:00
71302eefdf Comment out a LOG.debug 2023-09-26 10:54:01 -05:00
eefbdd212f Merge pull request #42 from Kingsrook/feature/CE-609-infrastructure-remove-permissions-from-header
Feature/ce 609 infrastructure remove permissions from header
2023-09-25 16:01:46 -05:00
9e9d2926c6 Nicer user-facting exceptions for throwUnsupportedCriteriaOperator and throwUnsupportedCriteriaField 2023-09-25 14:54:25 -05:00
994ab15652 Remove unused fields 2023-09-25 14:09:55 -05:00
164087beb0 Merge pull request #41 from Kingsrook/integration/20230921
automationStatus → OK fixes; script + audit updates;
2023-09-25 13:24:44 -05:00
070dec1266 Merge branch 'feature/script-audit-and-audit-change-cleanup' into integration/20230921 2023-09-21 15:04:00 -05:00
27c9694433 Merge branch 'feature/automation-status-fixes' into integration/20230921 2023-09-21 15:03:33 -05:00
6524f19ff7 Update to use UTF-8 for entities we post or put (default in underlying library is ISO-8859-1... 2023-09-21 14:56:54 -05:00
a95e9d06a2 Add shortRepo name for Infoplus-Scripts... 2023-09-21 14:55:28 -05:00
b9b32d4b7d Add option to (poorly) format SQL for logs 2023-09-21 14:54:54 -05:00
1c99ea2c6f Build audits when running Record Scripts; add script name to audit context; clean up some bogus 'changed x to x' messages. 2023-09-21 14:42:01 -05:00
f19cd26892 Fix canWeSkipPendingAndGoToOkay to only ever return true if its input status is a Pending status. 2023-09-21 13:45:04 -05:00
253e93c356 Merge pull request #40 from Kingsrook/dev
refreshign CE-609 with dev
2023-09-14 14:37:47 -05:00
3e8afde744 Merge pull request #39 from Kingsrook/feature/instance-and-script-deployment-mode
Add deploymentMode as a field in QInstance; pass it into scripts (e.g…
2023-09-14 14:13:43 -05:00
ce823ad22f Add deploymentMode as a field in QInstance; pass it into scripts (e.g., in executeCodeAction) 2023-09-14 12:16:58 -05:00
93e1c01939 Fixed getIntegerFromPropertyOrEnvironment, when it gets a value from env (was parsing the prop value instead); added tests on getIntegerFromPropertyOrEnvironment 2023-09-08 10:58:38 -05:00
c37eead6be Add a reasonable order-by to all table-based PossibleValueSources defined in QQQ; fix using order-by in SearchPossibleValueSourceAction 2023-09-08 10:53:31 -05:00
d9458ced34 Add LOG.warns any time we rollback a transaction from top-level StreamedETL process code. 2023-09-07 12:08:43 -05:00
01a19180b9 Fix some cases of joins in exports w/ multiple possible paths 2023-08-18 16:25:00 -05:00
406069768b Updating shortRepo values 2023-08-17 15:48:48 -05:00
83055e1784 Merge branch 'dev' into feature/CE-609-infrastructure-remove-permissions-from-header 2023-08-17 11:46:43 -05:00
2e0d1dbb1c Updating to 0.19.0 2023-08-17 10:20:18 -05:00
a899db4b3e Merge tag 'version-0.18.0' into dev
Tag release
2023-08-17 10:20:14 -05:00
1a52e3354e Merge branch 'release/0.18.0' 2023-08-17 10:18:46 -05:00
c912fe7cc2 Update for next development version 2023-08-17 10:16:56 -05:00
0aa0f0a085 Update versions for release 2023-08-17 10:16:51 -05:00
4b9d7b135b Merge branch 'integration/sprint-31' into dev 2023-08-17 09:59:58 -05:00
7082f7c2b1 Merge pull request #38 from Kingsrook/feature/CE-567-script-writer-dev-setup-sdlc-ci-cd-setup
CE-567 Add concept of security lock Scope - e.g., READ-WRITE (blockin…
2023-08-15 19:41:31 -05:00
d28249e5ce Merge pull request #37 from Kingsrook/feature/CE-567-script-writer-dev-setup-sdlc-ci-cd-setup
CE-567 Add concept of security lock Scope - e.g., READ-WRITE (blockin…
2023-08-15 19:40:38 -05:00
7da34d70da CE-609 Remove tests for now-removed /api/oauth/token paths 2023-08-15 18:48:57 -05:00
0d0ab6c2e5 CE-567 Add concept of security lock Scope - e.g., READ-WRITE (blocking all access to a record), or just WRITE - which means anyone can read, but you must have the key to WRITE. 2023-08-15 16:55:36 -05:00
2577bbeb37 Restore QJavalinImplementation to original state after testHotSwap 2023-08-15 11:38:46 -05:00
db0b434e52 CE-609 - Support for staged rollout: Check sessionUUID before any other value; add logging. 2023-08-15 11:27:51 -05:00
d4e18d8f55 CE-608: updated check for jsonObject when processing API GET request to consider the object being jsonObject.isNull(), added ability to use CUSTOM authorization in an API util override 2023-08-14 19:37:00 -05:00
f2e674ded4 Merge pull request #36 from Kingsrook/feature/CE-607-mvp-of-transportation-plan-record
Feature/ce 607 mvp of transportation plan record
2023-08-09 12:27:46 -05:00
366639c882 CE-609 Increase javalin test coverage (manageSessions and hotSwap) 2023-08-09 10:31:59 -05:00
dbaad85ec7 CE-609 Restore usage of sessionId cookie/auth-key (used by a test on table-based auth) 2023-08-09 09:55:59 -05:00
8479ef4b90 Initial WIP Checkpoint of auth0 userSessions 2023-08-09 09:47:41 -05:00
1da85ce0a2 CE-607 Go go tests 2023-08-08 16:51:47 -05:00
5dfa10912e CE-607 Slight tweaks to exposed join field validation 2023-08-08 16:45:30 -05:00
05f2341099 CE-607 Instance validation for section-fields from join tables 2023-08-08 16:21:28 -05:00
3406929e75 process query joins in Get 2023-08-08 13:18:27 -05:00
c548952281 Fixing a case in query joins, where a joinMetaData was given, but it needed flipped. 2023-08-08 13:18:13 -05:00
d811ed725d Support api queryCriteria and orderBy for removed fields; more/better use of api names for tables & fields in openApi spec; pass qInstance through supplemental validation chain; 2023-08-08 13:17:11 -05:00
4cb00670ed CE-607 Switch a tryElse to a tryAndRequireNonNullElse, to avoid NPE 2023-08-04 19:39:28 -05:00
4cbd808a55 CE-607 add query joins to GetInput 2023-08-04 19:39:06 -05:00
fc17ef6106 Avoid an NPE if initial version not set 2023-08-04 16:50:56 -05:00
b01023e541 Turn down some noisy logs 2023-08-04 16:50:41 -05:00
a4df67f9f9 Attempt to fix building proper x.y.z versions by respecting tag version-x.y.z as one that shouldn't edit the pom 2023-08-04 16:49:55 -05:00
b4a63e6e1b Updating to 0.18.0 2023-08-03 12:28:00 -05:00
0d78555a05 Merge tag 'version-0.17.0' into dev
Tag release
2023-08-03 12:27:56 -05:00
6a1db1c533 Merge branch 'release/0.17.0' 2023-08-03 12:25:49 -05:00
fabde303ab Update for next development version 2023-08-03 12:21:47 -05:00
6d173d5485 Update versions for release 2023-08-03 12:21:01 -05:00
79ac48b7f9 Merge branch 'integration/sprint-30' into dev 2023-08-03 11:59:20 -05:00
f7c8513845 CE-564 - Adding support for override warehouse and carrier service. 2023-08-03 11:43:06 -05:00
be30422c18 CE-564 - Adding support for override warehouse and carrier service. 2023-08-03 11:33:45 -05:00
53c005051e CE-537 - Updating to support API Delete 2023-08-03 11:16:24 -05:00
3879d5412c Merge pull request #35 from Kingsrook/feature/CE-548-script-writer-dev-setup-intelli-j-ide-local-ide-unit-testing
Feature/ce 548 script writer dev setup intelli j ide local ide unit testing
2023-08-01 18:49:23 -05:00
d596346c44 Merge pull request #34 from Kingsrook/dev
dev into sprint-30
2023-08-01 18:46:48 -05:00
ac88def08c CE-548 Update to handle process that aren't tied to a (single) table, but still take ids as input (e.g,. runScript) 2023-08-01 18:44:03 -05:00
29bb7252e8 CE-548 Add option to not includeUUIDs in logs 2023-08-01 09:16:09 -05:00
f0bd6b4b80 CE-548 Add override of addApiUtilityToContext 2023-08-01 09:15:45 -05:00
726075f041 CE-548 Add System.out script execution logger 2023-08-01 09:15:18 -05:00
67a1afdc1a CE-548 add some support for a single file's contents being submitted under input key "contents" (e.g., when used via API). 2023-08-01 09:11:59 -05:00
c832028961 Implement CHILD_POINTS_AT_PARENT use-case 2023-08-01 08:57:24 -05:00
774309e846 Add percents to ColumnStats 2023-07-27 08:37:05 -05:00
a19a516fc0 Merge pull request #33 from Kingsrook/feature/CE-551-change-logic-for-fed-ex
Feature/ce 551 change logic for fed ex
2023-07-26 08:42:47 -05:00
e153d3a7b4 Merge pull request #32 from Kingsrook/dev
dev into sprint-30
2023-07-26 08:42:11 -05:00
34a1755e44 CE-551 Add defaultValue to frontend field meta data 2023-07-25 13:06:42 -05:00
b4a2ba9582 Make implement TopLevelMetaDataInterface 2023-07-25 08:25:54 -05:00
9bb6600a9d Move default sort order to constant; add comment 'small runs first' 2023-07-25 08:25:38 -05:00
4f081e7c79 Split up PVS definition methods (in case an instance needs some (for scripts), but not all (not doing api log)); add some non-null checks around version lists 2023-07-25 08:25:17 -05:00
a0a43d48f5 Initial checkin (went with query timeout, but was missed) 2023-07-25 08:24:21 -05:00
7c4e06abcc Merge pull request #31 from Kingsrook/feature/query-timeout-and-cancel
Feature/query timeout and cancel
2023-07-25 08:14:09 -05:00
39d714fbb1 Updating to 0.17.0 2023-07-24 15:21:50 -05:00
6975069049 Merge tag 'version-0.16.0' into dev
Tag release
2023-07-24 15:21:46 -05:00
f05759ae83 Merge branch 'release/0.16.0' 2023-07-24 15:19:57 -05:00
81e4d5d36d Update for next development version 2023-07-24 15:17:13 -05:00
bff8a0f78a Update versions for release 2023-07-24 15:16:37 -05:00
87fbbc797a Merge pull request #26 from Kingsrook/dependabot/maven/qqq-middleware-picocli/com.h2database-h2-2.2.220
Bump h2 from 2.1.210 to 2.2.220 in /qqq-middleware-picocli
2023-07-24 14:45:53 -05:00
742caba8d2 Merge pull request #27 from Kingsrook/dependabot/maven/qqq-sample-project/com.h2database-h2-2.2.220
Bump h2 from 2.1.212 to 2.2.220 in /qqq-sample-project
2023-07-24 14:45:20 -05:00
f6f5a07d3a Merge pull request #28 from Kingsrook/dependabot/maven/qqq-backend-module-rdbms/com.h2database-h2-2.2.220
Bump h2 from 2.1.214 to 2.2.220 in /qqq-backend-module-rdbms
2023-07-24 14:44:47 -05:00
bc50c1e22e Merge pull request #29 from Kingsrook/dependabot/maven/qqq-middleware-javalin/com.h2database-h2-2.2.220
Bump h2 from 2.1.214 to 2.2.220 in /qqq-middleware-javalin
2023-07-24 14:44:13 -05:00
71672d46ee Initial checkin 2023-07-20 20:11:46 -05:00
75c84cd0ff Added constants referenced in last commit 2023-07-20 20:10:31 -05:00
0ff98ce7ea Add internal timeouts to RDBMS query, count, and aggregate, with timeoutSeconds field on their inputs; also add cancel method on those 3 actions, implemented down in RDBMS as well (e.g., to cancel inresponse to http request being abandoned) 2023-07-20 20:10:03 -05:00
c53f5e935d Merge pull request #30 from Kingsrook/integration/sprint-29
Integration/sprint 29
2023-07-20 09:57:34 -05:00
f5f2cc5007 CE-508 - Updated to support setCredentialsInHeader 2023-07-20 09:32:52 -05:00
d62a4c6daf CE-536 Add getRecordByUniqueKey 2023-07-18 16:28:32 -05:00
367fa4f657 Merge branch 'feature/datetime-query-expressions' into integration/sprint-29 2023-07-17 16:26:48 -05:00
5a5c9a0072 Revert: CE-536 If records are supplied to the process input, then use them instead of running a query. 2023-07-17 12:16:45 -05:00
2db1adc9ab CE-536 If records are supplied to the process input, then use them instead of running a query. 2023-07-17 11:34:01 -05:00
3d2708da23 CE-535 All more points of overridability, and make keys in existing record map a pair of {fieldName,value} 2023-07-14 14:08:27 -05:00
6ec838c48b Merge branch 'feature/CE-535-make-ip-a-wms-that-ct-live-oms-can-control' into integration/sprint-29 2023-07-14 11:32:42 -05:00
ebb7e7ab45 Try to add hints about unrecognized field names (if they're in other api versions) 2023-07-13 17:10:42 -05:00
8c3648920d Don't audit values for masked field types 2023-07-13 17:09:41 -05:00
6d6510c223 Add swapMultiLevelMapKeys 2023-07-13 17:09:18 -05:00
2422d09c31 Stop doing criteria expressions as their own thing, and instead put them in the values list 2023-07-13 09:17:34 -05:00
c04ab42bd9 CE-534: updates to support direct carrier tracker 2023-07-12 21:09:18 -05:00
c003d448d6 updates from last sprint's story 2023-07-12 21:07:20 -05:00
de8d668ea2 CE-535 new replace action, with test 2023-07-11 08:40:25 -05:00
953d97c554 CE-535 add auditContext 2023-07-11 08:31:28 -05:00
086787a5ca CE-535 Cleanup in DeleteAction - add omitDmlAudit & auditContext; be more sure not to delete associations if errors 2023-07-11 08:31:15 -05:00
e6816174c3 Test for APIRecordUtils.jsonQueryStyleQRecordToJSONObject 2023-07-10 11:07:06 -05:00
ed60ad2a96 Add doCopySourcePrimaryKeyToCache option - to, copy primary keys from source to cache; apply this in qqq-table cache 2023-07-10 09:49:11 -05:00
a943628e84 Update to flush buffered pipes - fixes issue where static data supplier records may not appear 2023-07-10 09:47:50 -05:00
5cfcb420d0 CE-535 Initial checkin 2023-07-10 09:46:13 -05:00
593c9f25f9 Bump h2 from 2.1.214 to 2.2.220 in /qqq-middleware-javalin
Bumps [h2](https://github.com/h2database/h2database) from 2.1.214 to 2.2.220.
- [Release notes](https://github.com/h2database/h2database/releases)
- [Commits](https://github.com/h2database/h2database/compare/version-2.1.214...version-2.2.220)

---
updated-dependencies:
- dependency-name: com.h2database:h2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-07 21:59:44 +00:00
ca560c933d Bump h2 from 2.1.214 to 2.2.220 in /qqq-backend-module-rdbms
Bumps [h2](https://github.com/h2database/h2database) from 2.1.214 to 2.2.220.
- [Release notes](https://github.com/h2database/h2database/releases)
- [Commits](https://github.com/h2database/h2database/compare/version-2.1.214...version-2.2.220)

---
updated-dependencies:
- dependency-name: com.h2database:h2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-07 21:59:09 +00:00
51dd0b6b29 Bump h2 from 2.1.212 to 2.2.220 in /qqq-sample-project
Bumps [h2](https://github.com/h2database/h2database) from 2.1.212 to 2.2.220.
- [Release notes](https://github.com/h2database/h2database/releases)
- [Commits](https://github.com/h2database/h2database/compare/version-2.1.212...version-2.2.220)

---
updated-dependencies:
- dependency-name: com.h2database:h2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-07 21:57:48 +00:00
b924fdcffa Bump h2 from 2.1.210 to 2.2.220 in /qqq-middleware-picocli
Bumps [h2](https://github.com/h2database/h2database) from 2.1.210 to 2.2.220.
- [Release notes](https://github.com/h2database/h2database/releases)
- [Commits](https://github.com/h2database/h2database/compare/version-2.1.210...version-2.2.220)

---
updated-dependencies:
- dependency-name: com.h2database:h2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-07 21:56:58 +00:00
5b84df1752 Setting loadScriptTestDetails as not-protected 2023-07-07 09:33:05 -05:00
9af1fed422 Initial backend work to support datetime query expressions from frontend 2023-07-06 18:53:10 -05:00
4299199947 CTLE-507 Fix api field sorting (make sure they have a label too) 2023-07-06 16:13:15 -05:00
6f578eb2f0 CTLE-507 Update to sort fields AFTER adding removed ones 2023-07-06 15:57:38 -05:00
be5b8f0869 CTLE-436: added missing null check 2023-07-06 13:42:02 -05:00
c27723e956 CTLE-436: added variant id to basepull key 2023-07-06 12:06:19 -05:00
cbff44f6a3 Merge branch 'dev' into integration/sprint-28 2023-07-05 08:54:32 -05:00
9167f356e9 Do not manage associations on records w/ errors 2023-07-05 08:53:17 -05:00
dccefb0a40 Merge pull request #25 from Kingsrook/feature/CTLE-503-optimization-weather-api-data
Feature/ctle 503 optimization weather api data
2023-07-03 15:41:59 -05:00
e24f15e2d8 Add commit from merge 2023-07-03 15:41:41 -05:00
d0a0f93933 Merge remote-tracking branch 'origin/integration/sprint-28' into feature/CTLE-503-optimization-weather-api-data
# Conflicts:
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/QueryAction.java
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/metadata/frontend/QFrontendTableMetaData.java
#	qqq-backend-module-api/src/main/java/com/kingsrook/qqq/backend/module/api/actions/BaseAPIActionUtil.java
#	qqq-middleware-api/src/main/java/com/kingsrook/qqq/api/actions/QRecordApiAdapter.java
2023-07-03 15:38:55 -05:00
76d226b5e8 Update query action cache helper to respect if a table says it can't do QUERY - to do GET instead 2023-07-03 15:02:26 -05:00
2bf611b24f Change apiJsonObjectToQRecord's includePrimaryKey param to be includeNonEditableFields instead 2023-07-03 14:05:33 -05:00
56b9738f15 Add convertJavaObject to QCodeExecutor 2023-07-03 14:05:07 -05:00
40d073bf54 Update to put the generated id on newly inserted cached records 2023-07-03 11:09:01 -05:00
c0297dea91 Merge branch 'feature/CTLE-436-move-to-integration-per-client' into integration/sprint-28 2023-07-03 09:57:26 -05:00
9e4743e8d8 Upgrade javalin to 5.6.1 2023-06-30 20:07:02 -05:00
976f173c93 Wrap cache table writes in try-catch - don't let a cache-updating action break a read 2023-06-30 20:06:47 -05:00
3f74594c69 Add handling of javascript Dates in convertObjectToJava 2023-06-30 20:04:49 -05:00
da08382055 Add getMaxResponseMessageLengthForLog 2023-06-30 20:00:08 -05:00
22a9e4b06b Change type to come from abstract getType method, rather than member field in base class (force sub-class to deal with it); Add ability to incldue supplemental table meta data in frontend table meta data requests 2023-06-30 14:10:23 -05:00
c086874e64 Refactor caching out of GetAction - namely, to support initial use-cases in QueryAction. 2023-06-30 12:36:15 -05:00
75ae848afd Let jsonObjectToRecord return null as a way to mean record wasn't found (therefore, don't add it to queryOutput) 2023-06-30 12:32:37 -05:00
53b74fb61d CTLE-436: attempt to add more test coverage 2023-06-29 16:03:54 -05:00
3ae938ac6e Support api-key in query-string for backend-api; 2023-06-29 11:15:24 -05:00
905ac6d72a fixed style issue 2023-06-29 10:10:00 -05:00
a6af75ebdc CTLE-436: syntax error 2023-06-29 09:38:03 -05:00
2039c727b5 CTLE-436: added variant endpoint, refactored variant code a bit 2023-06-28 19:45:37 -05:00
c38b8ac595 Fix test test and propagate exceptions more 2023-06-28 13:40:25 -05:00
3187706967 Implement RecordScriptTestInterface.execute, to fix record-script testing from UI 2023-06-28 12:39:29 -05:00
ffdb392b9f Fix handling heavy fields form joins 2023-06-28 12:39:03 -05:00
f79940d4c3 Update to clear internal caches between tests 2023-06-28 12:38:49 -05:00
be14afc11c Update to clear internal caches between tests 2023-06-28 11:17:38 -05:00
360bf56481 Add association api-meta data (so they can be versioned or excluded); add api field custom value mapper 2023-06-28 11:06:15 -05:00
b4507ba431 Remove unused withInstance method 2023-06-27 14:53:46 -05:00
57675528b5 Set query stat first result time immediately after loop (as well as inside rs loop) in case no results found (is this why we have the slow fed-ex cache use-cases?) 2023-06-27 14:53:35 -05:00
3fae35a2bf More fluent interface on core table actions & inputs 2023-06-27 14:53:01 -05:00
688d104635 Pass logs through; cleanup & comment 2023-06-27 12:33:17 -05:00
d533e59a84 Add exposed joins to QueryStat 2023-06-27 12:25:54 -05:00
184ef8db47 Mark as serializable 2023-06-27 12:25:21 -05:00
a62a1f10cd Make useOrWrap null input give null output 2023-06-27 12:25:10 -05:00
a056c4618c Fixed test (was query for contents, where they are no longer stored) 2023-06-27 08:53:20 -05:00
ed22ab5917 More test coverage on new script processes 2023-06-27 08:45:46 -05:00
598e26d9a1 Updating to support multi-file scripts 2023-06-27 08:14:11 -05:00
b53d1823df Switch to store all script contents in scriptRevisionFile sub-table; make test interface for all scripts work the same 2023-06-26 20:18:57 -05:00
6bc543fff7 Merge branch 'feature/query-stats' into feature/CTLE-507-custom-ct-live-packing-slips 2023-06-26 12:09:52 -05:00
e28f8b8317 Add null check around context 2023-06-23 16:42:47 -05:00
9d1266036c Adding scriptType fileModes, with multi-files under a script Revision. 2023-06-23 16:38:00 -05:00
b93032aae4 Merge remote-tracking branch 'origin/feature/query-stats' into feature/query-stats 2023-06-22 19:14:11 -05:00
300af89687 change 'fine grained' to have a dash 2023-06-22 19:13:55 -05:00
059dffb620 Merge pull request #23 from Kingsrook/dev
refresh querystats from dev
2023-06-22 19:07:15 -05:00
1822dd8189 add query stats to count, aggregate actions; add system/env prop checks; ready for initial dev deployment 2023-06-22 11:07:24 -05:00
b75fd29a57 Updating to 0.16.0 2023-06-22 10:41:50 -05:00
5ff91739d4 Merge tag 'version-0.15.0' into dev
Tag release
2023-06-22 10:41:46 -05:00
ada95431c1 Merge branch 'release/0.15.0' 2023-06-22 10:40:30 -05:00
c24b7cd84b Update for next development version 2023-06-22 10:38:42 -05:00
bf80eb1845 Update versions for release 2023-06-22 10:38:40 -05:00
db0c490a32 Merge pull request #20 from Kingsrook/feature/only-log-warnings-and-errors-once
initial version of attempting to downgrade logs if a warning or error…
2023-06-22 10:33:44 -05:00
a73bd40c6f dowgraded log to a debug 2023-06-22 10:32:56 -05:00
7eea0c08bb update to get rid of a few warnings 2023-06-22 10:19:08 -05:00
18c11d3869 Fix NPEs 2023-06-21 16:25:15 -05:00
a367ec717c Add of factory method 2023-06-21 16:18:13 -05:00
65e8f7a71f For runProcess, send the input object through processBodyToJsonString (e.g., for js to java object conversion0 2023-06-21 16:17:53 -05:00
900484c01c More consistent behavior of field labels & descriptions everywhere 2023-06-21 16:17:32 -05:00
6bfe0cd3ea Give error about null recordSecurityLock (vs null list of locks) 2023-06-21 16:17:01 -05:00
12d1de7135 Add labelMappings to instanceEnricher 2023-06-21 16:16:38 -05:00
e5efe8a64c Add message, current, and total to get-job-status 202(ACCEPTED) response spec 2023-06-20 11:43:30 -05:00
9d3cd50c7b Add 'tag' field to ApiProcessMetaData; use that when generating spec (for non-table processes for now) 2023-06-20 11:36:34 -05:00
5ae18c31f9 Merge pull request #22 from Kingsrook/feature/CTLE-509-support-sending-custom-data-fields-to-deposco
Feature/ctle 509 support sending custom data fields to deposco
2023-06-20 10:37:37 -05:00
ad11bf64db Merge branch 'integration/sprint-27' into feature/CTLE-509-support-sending-custom-data-fields-to-deposco 2023-06-20 10:37:20 -05:00
3a69957b43 Merge pull request #21 from Kingsrook/feature/CTLE-448-processes-via-api
Feature/ctle 448 processes via api
2023-06-20 10:36:35 -05:00
7af5ad2655 Fix to support null-filter id on table-triggers 2023-06-20 10:22:21 -05:00
57569e4c84 Escape identifiers in column names 2023-06-20 09:07:31 -05:00
3791c069c7 Add convertObjectToJava to code executors - for converting language objects to java objects 2023-06-20 09:07:31 -05:00
0f799339d6 Set user in new sessions 2023-06-19 12:33:01 -05:00
3e113a12b3 Initial checkin 2023-06-19 12:30:10 -05:00
79304adcb0 Move saved filter processes to qqq 2023-06-19 12:19:11 -05:00
2c192a3fd9 Add SavedFiltersMetaDataProvider (as we've introduced a dependency between it and ScriptsMetaDataProvider (through tableTriggers) 2023-06-19 12:14:26 -05:00
3772cf725f Make table-triggers respect saved filters 2023-06-19 12:03:50 -05:00
1cf83fb441 Add factory method: newForTable 2023-06-16 16:45:10 -05:00
4efb818bfe Add override executeWithStringDetails 2023-06-16 16:44:53 -05:00
f30b2a9ef8 Change times to 60 seconds 2023-06-16 16:44:23 -05:00
2efc732530 Fixing for CI 2023-06-16 09:55:06 -05:00
1ed51e0a35 Disabling these tests - poorly written, and no longer viable as a concept 2023-06-16 09:36:15 -05:00
54bf5bed8f Add some coverage for query stats; remove old classes (re-added in merge?) 2023-06-16 09:29:43 -05:00
12eb3be3cb Mark all fields as @QField 2023-06-16 08:44:17 -05:00
4105f034aa Merge remote-tracking branch 'origin/feature/query-stats' into feature/query-stats
# Conflicts:
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/interfaces/QueryInterface.java
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/QueryAction.java
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/data/QRecordEntity.java
2023-06-16 08:40:04 -05:00
a38d57c7af Update to work with new version of entities; actually working 2023-06-16 08:38:39 -05:00
14fa7fdb74 Update to only treat field as QField if @QField annotation is present 2023-06-16 08:38:03 -05:00
c07c007bc2 Add capability: QUERY_STATS; rework capabilities to be smarter w/ enable, then disable 2023-06-16 08:37:23 -05:00
9fe5067374 Initial checkin 2023-06-16 08:37:23 -05:00
599aff3487 Initial checkin 2023-06-16 08:36:04 -05:00
59b7e0529c Add getActionIdentity 2023-06-16 08:35:03 -05:00
d9a98c5987 Checkpoint - query stats (plus recordEntities with associations) 2023-06-15 10:19:31 -05:00
b58f93e627 Revert a few changes, to help with stability of generated api specs 2023-06-15 08:19:04 -05:00
1c1a0f99e8 changes to make script processes in api better 2023-06-14 16:42:04 -05:00
d273d091df Test fixes 2023-06-14 15:50:38 -05:00
d0194d9580 Missing copyright 2023-06-14 13:32:50 -05:00
54a0d6720f Missing copyright 2023-06-14 13:26:57 -05:00
19ee5bcb23 Checkpoint, WIP on processes in api - mostly done 2023-06-14 11:53:43 -05:00
eee7354e77 Checkpoint, WIP on processes in api 2023-06-12 18:19:48 -05:00
6b590324be initial version of attempting to downgrade logs if a warning or error has already been logged from the stack of throwables 2023-06-12 16:04:46 -05:00
a340299c67 Initial implementation of api processes 2023-06-12 10:58:30 -05:00
6a01754479 Renaming MiddlewareMetaData to SupplementalMetaData 2023-06-08 18:24:56 -05:00
4ccc726f2e Fix binding of long values 2023-06-08 14:38:27 -05:00
eb151f0610 Allow full JDBC URL to be set in RDBMS meta-data, used directly (w/ a known vendor) 2023-06-08 14:37:59 -05:00
a18d2afee5 Updating to 0.15.0 2023-06-08 14:27:29 -05:00
0007909792 Merge tag 'version-0.14.0' into dev
Tag release
2023-06-08 14:27:25 -05:00
ceed7081ca Merge branch 'release/0.14.0' 2023-06-08 14:25:38 -05:00
78ba2b591c Update for next development version 2023-06-08 14:22:52 -05:00
a4aeafdf91 Update versions for release 2023-06-08 14:22:14 -05:00
f47ae7f1fa Merge branch 'integration/sprint-26' into dev 2023-06-08 12:35:35 -05:00
fcf452836b Enrich api field name to label if missing 2023-06-06 19:27:33 -05:00
3b398942e7 Pass original records in as oldRecords for UpdateAction.performValidations 2023-06-06 19:25:05 -05:00
b52a014154 Don't try to manageAssociations for a record that had errors 2023-06-06 19:24:40 -05:00
e97ca5b5c5 Fix usage of subfilters in automation actions 2023-06-06 13:31:25 -05:00
a117c6ff3f Merge branch 'feature/CTLE-435-review-and-complete-parcel-sla' into integration/sprint-26 2023-06-06 11:15:11 -05:00
c08856a92c Merge branch 'feature/CTLE-433-cart-rover-now-extensiv-integration' into integration/sprint-26 2023-06-06 11:14:58 -05:00
c5b14cd22c Give explicit error if table doesn't have a primary key 2023-06-06 11:06:57 -05:00
8de9288c05 Fix merge conflict 2023-06-06 09:59:44 -05:00
9b5d1e1208 Merge branch 'feature/CTLE-153-default-ct-live-packing-slips-to-deposco' into integration/sprint-26
# Conflicts:
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidator.java
#	qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/instances/QInstanceValidatorTest.java
#	qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/backend/javalin/QJavalinImplementation.java
#	qqq-middleware-javalin/src/test/java/com/kingsrook/qqq/backend/javalin/QJavalinImplementationTest.java
#	qqq-middleware-javalin/src/test/java/com/kingsrook/qqq/backend/javalin/TestUtils.java
2023-06-06 09:58:12 -05:00
f5047e5f50 Merge branch 'dev' into feature/CTLE-434-oms-update-business-logic 2023-06-06 09:06:34 -05:00
12dd3617e5 Add a few more known http status for outbound status codes 2023-06-05 16:50:41 -05:00
ba544adb76 Read filter & fields from formParam if not found as queryParam, in exports 2023-06-05 16:48:35 -05:00
c91a678905 More control options around javalin access logging 2023-06-05 14:49:05 -05:00
f1ebff28eb updated shouldBeRetryableServerErrorException in base api action utils to only retry on 500 errors if query or get actions 2023-06-05 11:19:56 -05:00
e4ae717f7f fix missing import 2023-06-05 08:30:04 -05:00
fdbc751048 more support for blobs
- fetch heavy flag for process extract
- ignore blobs in bulk edit, load
- set download urls in child-list render and javalin via shared method in QValueFormatter

also, make ETL load step aware of transform step, and more flexible process summary between them
2023-06-05 08:17:45 -05:00
c67d042e26 Add null check 2023-06-02 11:34:18 -05:00
a4bd3b56f6 Fix comments, cleanup from code review 2023-06-02 09:41:07 -05:00
5330f3de90 Fix merge conflict in update method 2023-06-02 09:04:16 -05:00
eca176a7f0 Merge branch 'dev' into feature/CTLE-434-oms-update-business-logic
# Conflicts:
#	qqq-backend-core/src/test/java/com/kingsrook/qqq/backend/core/modules/backend/implementations/memory/MemoryBackendModuleTest.java
#	qqq-middleware-javalin/src/main/java/com/kingsrook/qqq/backend/javalin/QJavalinImplementation.java
2023-06-02 09:02:05 -05:00
0b525f8775 Checkpoint - query stats (plus recordEntities with associations) 2023-06-02 08:58:24 -05:00
499d59ade2 Add overload of loadTableToMap that takes Consumer<QueryInput> (e.g., to request heavy fields) 2023-06-01 16:32:34 -05:00
b67813b7ad Update test to handle blob correctly 2023-06-01 16:31:54 -05:00
7634246a03 Handling of BLOBs 2023-06-01 16:13:02 -05:00
1db6dfb2ad avoid calling auditDetail insert if no records 2023-06-01 15:03:21 -05:00
f5516c8122 Make column stats treat "" and null the same (as a non-value or blank) 2023-06-01 15:03:21 -05:00
6455171cee removed unused subclass 2023-05-30 14:57:34 -05:00
ca1cc853fc fixed access on subclass method in test 2023-05-30 14:55:56 -05:00
c78d598035 Add an eventCartridge (handler) for methods that throw; add template identifier (a name) to input 2023-05-30 14:05:00 -05:00
a5387ff9db added retryable exception and method that can be overridden in base classes to allow retrying in different circumstances, added SC_GATEWAY_TIMEOUT as a logger.info 2023-05-30 14:04:30 -05:00
794fb5e87a Expand javalin tests 2023-05-30 13:48:26 -05:00
343f3fe01a Update to support both JSON and multipart form bodies for create and update. 2023-05-30 11:20:49 -05:00
364e9f420b Update to support both JSON and multipart form bodies for create and update. 2023-05-30 11:13:20 -05:00
a75530b466 Updates for more heavy-field handling 2023-05-30 10:13:04 -05:00
489f12996d updated to allow 'trying again' when server side 500 error occur in makeRequest() 2023-05-25 11:34:50 -05:00
6b1e3aa572 Merge branch 'dev' into feature/CTLE-434-oms-update-business-logic
# Conflicts:
#	qqq-middleware-api/src/test/java/com/kingsrook/qqq/api/javalin/QJavalinApiHandlerTest.java
2023-05-25 11:08:31 -05:00
8235bb2bb7 Merge branch 'dev' into feature/CTLE-153-default-ct-live-packing-slips-to-deposco 2023-05-25 10:57:51 -05:00
5185758855 add check for "snapshot-BRANCH_SLUG" arg to update-all-qqq-deps.sh -- if given, then we look for branchy versions, else we use main versions [skip ci] 2023-05-25 10:55:10 -05:00
515e04ecfe Update api json mapping to include null & empty values 2023-05-25 10:22:04 -05:00
76b102b811 updated to still log info on api gateway error, but still throw exception 2023-05-24 21:28:52 -05:00
36efc4c2d9 More BLOB; FILE_DOWNLOAD adornment; javalin field download endpoint 2023-05-24 17:07:19 -05:00
6ca06bf49d CTLE-435: removed unused input param from html renderer methods 2023-05-24 11:05:59 -05:00
614aead348 add more support for byte[]/BLOB field type; checkpoint 2023-05-23 14:07:36 -05:00
de74455de7 more changes to help reduce warnings that should probably be infos in loggly 2023-05-23 10:11:54 -05:00
7491e5f819 api association fixes; mostly about propagating ids/fkeys, and having fields (in maps) as expected field types (cherry pick 74d003ed) 2023-05-23 09:00:41 -05:00
74d003ed3c api association fixes; mostly about propagating ids/fkeys, and having fields (in maps) as expected field types 2023-05-22 16:05:34 -05:00
4b6b60f331 Add concept of inputSource on insert/update/delete actions. 2023-05-19 16:34:47 -05:00
e10af188ef Add concept of inputSource on insert/update/delete actions. 2023-05-19 16:34:26 -05:00
486a942fdc Merge remote-tracking branch 'origin/dev' into feature/CTLE-434-oms-update-business-logic 2023-05-19 14:57:51 -05:00
9dc6f4ccf8 Add DEFAULT_PREVIEW_MESSAGE_PREFIX 2023-05-19 08:36:19 -05:00
1161e65c03 Add showReloadButton, showExportButton 2023-05-19 08:35:31 -05:00
fd550bad85 Add method icon(name, color) 2023-05-19 08:35:09 -05:00
5bd1fd4a7f Add environmentBannerText and environmentBannerColor 2023-05-18 14:17:59 -05:00
46afb46910 Add log info for committing of slow transactions 2023-05-18 09:49:57 -05:00
7e1a7c7fd7 CTLE-433: updates to support extensiv integration 2023-05-17 20:57:03 -05:00
274567aefa Merge branch 'dev' into feature/CTLE-434-oms-update-business-logic 2023-05-16 19:59:15 -05:00
c0237e9d09 Merge pull request #18 from Kingsrook/hotfix/fix-cache-bugs
hotfix: attempt to remove some caching bugs when updating old cache r…
2023-05-16 13:59:55 -05:00
3f9370e9b5 hotfix: updated to null out the getoutput's record if it was not returned by the original source 2023-05-16 12:02:47 -05:00
01d3937889 hotfix: attempt to remove some caching bugs when updating old cache records and we got uncachable errors 2023-05-16 11:38:24 -05:00
caaf76c9a5 Add placeholder lines for uniqueKey, recordSecurityLock, and auditRules 2023-05-16 08:32:46 -05:00
0e60811c43 Fixed previous fix (was adding wrong queryJoin in exposedJoins loop else block) 2023-05-16 08:32:02 -05:00
4eb28cd1b7 Fix for table being added to query twice, if it's added for security, and then for being in a where clause. 2023-05-15 15:08:56 -05:00
8470da40f1 Checkpoitn 2023-05-15 12:14:27 -05:00
dd63e8d4e2 Initial checkin 2023-05-12 16:48:27 -05:00
298e73e144 Don't try to manage associations of empty lists 2023-05-12 16:40:50 -05:00
4e5fd62808 Merge branch 'dev' into feature/CTLE-434-oms-update-business-logic 2023-05-12 14:57:51 -05:00
14fc7b0ba8 Add criteria operator NOT_EQUALS_OR_IS_NULL 2023-05-12 14:50:27 -05:00
676783fdf5 Add toQRecordOnlyChangedFields 2023-05-12 12:21:59 -05:00
3e7684bb8d Don't audit for records that failed their DML 2023-05-12 12:21:34 -05:00
815bd8b0ce Updates to work with branch-specific maven deployments in/with circleci 2023-05-11 10:15:40 -05:00
7a5124ae06 Reset the input list of primary keys - callers may expect that! 2023-05-10 17:59:25 -05:00
e9328c6653 Make list of primary keys explicitly serializble, so when they change to field's type later, it doesn't blow up (should we fix that on the other side? (yes)) 2023-05-10 17:50:09 -05:00
1121dc14d4 Fixed processing output records (those without errors aren't present in map) 2023-05-10 17:40:07 -05:00
0b996ef008 CTLE-433: attempt to get over 80% coverage in javalin 2023-05-10 17:31:20 -05:00
d6acb0c1ef you see, checkstyle was disabled in intellij, for reasons 2023-05-10 17:10:04 -05:00
fb99c615fe Added javadoc explaining how doQuery fetches many pages up to limit 2023-05-10 17:05:12 -05:00
eef5936282 bulk insert & delete w/ pre-validation and warnings and errors and such 2023-05-10 17:04:57 -05:00
e66b649699 CTLE-433: fixed more tests due to new data 2023-05-10 16:35:00 -05:00
ef8db2786d CTLE-433: added boolean type to deserializer 2023-05-10 16:23:48 -05:00
e06a5ab4b3 CTLE-433: checkpoint commit of backend variants, updated process utils to no longer take in input object since now comes from qContext, put instruction coverage back to 80% 2023-05-10 15:50:03 -05:00
b9ad0e7e21 Warnings & errors on update 2023-05-10 10:10:14 -05:00
33555701a4 Updates to allow validations on bulk-edit, with warnings and errors coming back on review & result screens. 2023-05-10 10:09:36 -05:00
88e24a08fc Updates to work with branch-specific maven deployments in/with circleci 2023-05-09 13:00:49 -05:00
2c7919abce Update to look at tag name too 2023-05-09 12:27:03 -05:00
21251b8d9b attempt to dploy branches to unique verion name 2023-05-09 10:43:23 -05:00
b412a424ca Fix delete test for error handling from customizers 2023-05-09 10:24:29 -05:00
7af164e002 More error handling from customizers 2023-05-09 10:09:29 -05:00
b2c7062709 Convert QRecord errors and warnings to new QStatusMessage type hierarchy. 2023-05-09 08:49:46 -05:00
cedc1edfac Add queryJoins to access log if-slow 2023-05-08 15:27:57 -05:00
c75d19d72a Add possible value 422... would be nice to get these all from a lib... 2023-05-08 15:27:40 -05:00
647c5968d3 Updated expected type on post-read customizer 2023-05-08 15:24:49 -05:00
db770c7e03 Fixed pre-delete warning check 2023-05-08 15:14:03 -05:00
265847e01a Completed first round implementation of {pre,post}{insert,delete} actions 2023-05-08 15:06:31 -05:00
6aef4d92e8 Merge branch 'integration/sprint-26' into feature/CTLE-434-oms-update-business-logic 2023-05-08 11:29:21 -05:00
1e1a33c250 fixed billing dashboard links and made them permissed 2023-05-08 11:25:08 -05:00
0fa418496a Merge branch 'dev' of github.com:Kingsrook/qqq into dev 2023-05-08 11:22:15 -05:00
d39698740c Removing TableCustomizer (mostly redundant with TableCustomizers and confusing for no real gain). Initial pass at update, delete customizers 2023-05-05 17:00:47 -05:00
036b7dc115 Refactor to get rid of Usage parameter in QCodeReference 2023-05-05 16:59:52 -05:00
cc765c66d6 Initial checkin - WIP 2023-05-05 16:57:54 -05:00
6acf0bf93a Increasing size of api log fields (mediumtext, 16MB) 2023-05-05 15:39:42 -05:00
16d54aa81f Merge branch 'integration/sprint-26' into dev 2023-05-05 13:12:04 -05:00
a1b96c6cee updated to hide hidden fields in frontentmetadata, updated error responses 2023-05-05 13:05:35 -05:00
fff7f5ad8e Turn on CORS headers 2023-05-05 08:43:41 -05:00
854c8bf1ba Change to use HttpEntityEnclosingRequestBase instad of Post 2023-05-04 15:43:41 -05:00
a0e45b983f Hide other-versions dropdown while page is loading 2023-05-04 15:42:31 -05:00
05a0e13a44 Add pipes between builds; remove 'time' 2023-05-04 15:39:57 -05:00
8d55ee2706 Rename scriptApi to qqqScriptUtils, putting that into context; 2023-05-04 15:38:49 -05:00
b9d90cdddb fixed failing test because of class cast exception 2023-05-04 15:03:50 -05:00
0ce989b75c CTLE-421: minor bug fixes from demo 2023-05-04 13:55:31 -05:00
405e6a6e2c Merge pull request #17 from Kingsrook/feature/api-home-page
Feature/api home page
2023-05-04 13:53:59 -05:00
74976061d4 Insert test script so it can be found 2023-05-04 12:10:55 -05:00
eed4cc270f Flow table name down 2023-05-04 12:05:48 -05:00
9c5106d7a8 Fix to skip, not blow up, on unrecognized field names 2023-05-03 16:47:03 -05:00
4aa1aed632 Fixes for expoesd joins 2023-05-03 16:36:46 -05:00
c799645658 CTLE-421: updated to check expiration and to get auth0 access token the correct way 2023-05-03 09:29:20 -05:00
5ed02be23f Update to use static times instead of now() in examples, so there aren't always diffs in specs 2023-05-03 08:26:22 -05:00
0159459db2 more flexible (app-defined) security schemes; 2023-05-02 19:58:19 -05:00
b0dbede6fe Add count, totalRows to child record lists 2023-05-02 15:52:20 -05:00
80ae3e1e0c Merge branch 'feature/CTLE-421-migrate-to-use-api-keys' into integration/sprint-25 2023-05-02 15:16:05 -05:00
d300ec162d CTLE-421: checkpoint commit 2023-05-02 15:15:49 -05:00
7625e593d2 Fix merge conlifct w/ bulk insert; add warnings to single-insert; add tests re: insert warnings 2023-05-02 15:08:06 -05:00
d237f4c1ad Make api log methods public 2023-05-02 14:12:05 -05:00
29f26e2ada temporarily lowering coverage limits 2023-05-02 11:42:03 -05:00
845b03bbca Merge branch 'feature/CTLE-421-migrate-to-use-api-keys' into integration/sprint-25 2023-05-02 11:32:27 -05:00
2ddf1fd5c7 CTLE-421: fixed bug caught by tests 2023-05-02 10:34:48 -05:00
83e628d2d6 CTLE-421: added reveal adornment, warnings on child record insert failures 2023-05-02 10:12:55 -05:00
b1ea33b2a2 Move limit & skip from query input to filter 2023-05-02 07:57:50 -05:00
f290cdeb6d Merge branch 'feature/CTLE-207-query-joins' into integration/sprint-25
# Conflicts:
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/model/actions/tables/query/QueryInput.java
#	qqq-middleware-api/src/main/java/com/kingsrook/qqq/api/javalin/QJavalinApiHandler.java
2023-05-02 07:52:15 -05:00
476924b030 Merge branch 'feature/CTLE-422-api-for-scripts' into integration/sprint-25
# Conflicts:
#	qqq-backend-core/src/main/java/com/kingsrook/qqq/backend/core/actions/tables/QueryAction.java
2023-05-01 20:13:22 -05:00
e857e87b18 Merge remote-tracking branch 'origin/feature/CTLE-421-migrate-to-use-api-keys' into integration/sprint-25 2023-05-01 20:00:16 -05:00
1a8ab34fe3 make a new collection if given a null one as input. 2023-05-01 14:17:25 -05:00
5070502bde Search for join meta data through exposed joins. 2023-05-01 14:17:08 -05:00
1e053d67ce Update to not NPE is record is null (rather, return null) 2023-05-01 08:32:02 -05:00
15acaec523 More api name & version with scripts (eg, running test scripts) 2023-04-30 19:56:41 -05:00
9ce45934a8 Try to make happier when scripts & apis don't live together 2023-04-28 16:00:06 -05:00
d35f150202 Scripts and apis and all kinds of stuff 2023-04-28 15:46:36 -05:00
0b7c2db452 Introduce RecordPipeBufferedWrapper, to be used in QueryAction when includingAssociations and writing to a pipe 2023-04-28 14:22:29 -05:00
2832566dbd CTLE-421: fixes for failing tests, removed no longer necessary loop 2023-04-28 13:59:08 -05:00
b0d0de5d49 CTLE-421: implemented fieldLevel is hidden, updated to mask password fields 2023-04-28 13:21:08 -05:00
b7e39d6953 Update to allow includeAssociations when querying into a record pipe. This meant propagating a lot of exceptions... 2023-04-28 12:14:12 -05:00
6f99111c52 Move api name & version into ScriptRevision; make the records that go into record-scripts be api versions of records. 2023-04-28 12:11:56 -05:00
4135607a4c Base class for run-script inputs 2023-04-28 12:07:45 -05:00
acfcc422f9 script to open jacoco report 2023-04-28 10:05:25 -05:00
a40f9afd38 Move insert person methods to TestUtils 2023-04-27 20:20:17 -05:00
1314ee98b8 add more store-jacoco calls 2023-04-27 20:19:59 -05:00
6b6dd546fd Initial checkin 2023-04-27 20:19:44 -05:00
37fa78417f Initial checkin 2023-04-27 20:19:31 -05:00
4003323b88 Working version of ApiScriptUtils. Moved actual api imlpementation out of javalin class, into implemnetation class. 2023-04-27 18:55:40 -05:00
5de42b9390 Merge pull request #16 from Kingsrook/feature/todo-more-test-coverage
Feature/todo more test coverage
2023-04-27 15:21:29 -05:00
0c5e3a8002 Many tests. small refactors to support. 2023-04-27 12:50:54 -05:00
d12bf3decc Make it an option in the module's interface, whether or not to query for all records being updated or deleted first (makes more sense for an api backend to NOT do this). 2023-04-27 12:50:46 -05:00
f3509ae1bf Fixed import 2023-04-27 11:12:38 -05:00
dab6a75340 Add dropdown to change apis (and an un-used, but POC method to list apis on a page...) 2023-04-27 11:05:02 -05:00
be09412755 CTLE-421: checkpoint commit of api key story so that CTL test coverage can be worked on instead 2023-04-27 10:31:13 -05:00
e0e4519708 Downgrade some logs 2023-04-26 10:21:22 -05:00
8094c29ec7 handle ExposedJoins in exports 2023-04-26 10:19:12 -05:00
f7f001d430 Improve tense of okSummary 2023-04-26 10:18:55 -05:00
04a8fa94f9 Move skip & limit out of QueryInput, into QQueryFilter... 2023-04-26 10:18:44 -05:00
b4328040aa CTLE-419: updated SyncProcessConfig record to take in performInserts/performUpdates as params 2023-04-25 09:34:21 -05:00
3370da109c CTLE-419: added ability to tell table sync step to either not do inserts or updates, another instant parse format, typo 2023-04-24 18:53:29 -05:00
caf9f102f6 Moved stuff so jacoco reporting happens before failures, i think. Moved untested class reporting into pom, out of circleci 2023-04-24 12:54:42 -05:00
de77f902ac Removed wip code, not meant to be commited 2023-04-24 12:53:39 -05:00
475deee993 Upgrade javalin to 5.4.2 2023-04-24 12:12:54 -05:00
1407f3c63c Add count distinct option to count action 2023-04-24 12:12:41 -05:00
2495989584 add exposed joins to frontend metadata; checkpoing on validation & enrichment of eposed joins 2023-04-24 12:11:46 -05:00
d086284de7 Checkpoint 2023-04-20 16:33:46 -05:00
6ce5845ec8 Checkpoint - good version of getJoinConnections now i think 2023-04-20 16:33:46 -05:00
3bf18e8b51 Initial checkin 2023-04-20 16:33:46 -05:00
88e47ef9ca WIP on getting joins to frontend 2023-04-20 16:33:46 -05:00
912b3885f5 updated nf scripts name 2023-04-20 16:12:22 -05:00
8cc16d44e5 more updates for change to coldtrack 2023-04-20 14:22:26 -05:00
2d81f24887 updated copyrights 2023-04-20 12:27:08 -05:00
3512cde424 Updating to 0.14.0 2023-04-20 10:49:56 -05:00
95dad80d6f Merge tag 'version-0.13.0' into dev
Tag release
2023-04-20 10:49:51 -05:00
72a9f68de7 Merge branch 'release/0.13.0' 2023-04-20 10:48:45 -05:00
b6db572a28 Update for next development version 2023-04-20 10:47:10 -05:00
c9fd7c087a Update versions for release 2023-04-20 10:47:06 -05:00
19d6739723 Merge pull request #15 from Kingsrook/dependabot/maven/qqq-backend-core/org.json-json-20230227
Bump json from 20220320 to 20230227 in /qqq-backend-core
2023-04-20 10:40:58 -05:00
cc881e3168 CTLE-397: fixed overflow issues with manual chips 2023-04-19 15:36:13 -05:00
2f0ef03eec CTLE-397: added pending icon to utils for widget velocity templates 2023-04-19 15:31:20 -05:00
4a65efa5ac CTLE-397: added ability to have footer HTML on widgets 2023-04-19 12:14:15 -05:00
6ebfbc6984 Merge branch 'feature/CTLE-397-oms' of github.com:Kingsrook/qqq into feature/CTLE-397-oms 2023-04-18 22:49:54 -05:00
1f931ddec6 CTLE-397: added more html util methods for chips and process filter links 2023-04-18 22:49:44 -05:00
fc6a72eb09 Replace random ChatGPT method with random StackOverflow method for getting classes from jar 2023-04-18 13:56:08 -05:00
22d0c5c79b (try at least) to set maxLength on string fields 2023-04-18 13:41:26 -05:00
07e575a7d7 Initialize auditInputList as new list, not null 2023-04-18 13:41:10 -05:00
a6e02ea1bf Set url field size XLARGE; set record label field as id 2023-04-18 13:40:53 -05:00
bf5273fa4d Add getAssociationNamesToInclude() 2023-04-18 13:40:32 -05:00
6bc160a213 Merge branch 'feature/CTLE-397-oms' of github.com:Kingsrook/qqq into feature/CTLE-397-oms 2023-04-17 10:45:08 -05:00
a159bc4076 CTLE-397: made some methods static, overloaded ahref filter method 2023-04-17 10:44:55 -05:00
514b70eb88 Add outbound api logs. FTW. 2023-04-14 16:29:07 -05:00
c1c8528dcb Bump json from 20220320 to 20230227 in /qqq-backend-core
Bumps [json](https://github.com/douglascrockford/JSON-java) from 20220320 to 20230227.
- [Release notes](https://github.com/douglascrockford/JSON-java/releases)
- [Changelog](https://github.com/stleary/JSON-java/blob/master/docs/RELEASES.md)
- [Commits](https://github.com/douglascrockford/JSON-java/commits)

---
updated-dependencies:
- dependency-name: org.json:json
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-14 18:12:56 +00:00
d760831431 Initial checkin 2023-04-14 09:52:57 -05:00
726fad0e5e Make unique key helper more helpful 2023-04-13 15:34:21 -05:00
d4181cc157 Make unique key helper more helpful 2023-04-13 15:21:43 -05:00
ca9834204b Split fieldNameToLabel out into a method 2023-04-13 11:50:47 -05:00
1b4c754314 Add with'ers 2023-04-13 11:50:31 -05:00
905c2d1296 Move DoFullValidation from validate step to preview step (so you can set from the top-level thing) 2023-04-12 11:25:41 -05:00
cbf84fd76b Fix spelling of Verson; add withers 2023-04-12 11:20:46 -05:00
106976a060 Add option to omitDmlAudits 2023-04-12 11:20:26 -05:00
8f6e4144d2 Change to use getTableWrapperObjectName in places it was intended (e.g., within JSON objects), instead of getTablePath(meant for URLs) 2023-04-12 11:20:12 -05:00
d009169770 Initial add of MultiLevelMapHelper 2023-04-12 11:19:22 -05:00
5453e2e081 Initial add of MetaDataProducer 2023-04-12 11:18:32 -05:00
51b2a9d924 Remove mocked up openapi spec files 2023-04-06 13:17:17 -05:00
a408e4c0d2 Initial checkin 2023-04-06 12:40:55 -05:00
8af83f9286 Include audit label as part of audit 2023-04-06 12:39:38 -05:00
b5c5fb7dd8 Add ability to disable operations per-table, etc. check for insert & update (single) errors better; 2023-04-06 12:39:25 -05:00
12a92c6330 Recursively add all association component schemas; add recordCount to bulk api action logs 2023-04-05 10:03:10 -05:00
0268dad395 Checkstyle 2023-04-04 15:50:43 -05:00
74cd0e0a57 add /apis.json and (re)add .../versions.json paths 2023-04-04 15:45:51 -05:00
e779c392bb Support multiple api's within a q instance. For science! 2023-04-04 13:40:32 -05:00
e35761e0a6 configurable servers; paths in swagger from root 2023-04-04 08:22:30 -05:00
cb5f3ba188 Updated to propagate transactions 2023-04-03 12:00:52 -05:00
b3b7c0b381 Fixed ids on test data. no actual change apparently 2023-04-03 10:46:36 -05:00
f4b8f5c782 Possible values for method, statusCode, apiVersion 2023-04-03 10:46:22 -05:00
ffe8da448b Add api logs user, security fields 2023-04-03 10:35:11 -05:00
b021aebabb Fix NPE with null field list 2023-03-31 16:37:35 -05:00
f79bf85c14 misc fixes 2023-03-31 16:30:56 -05:00
6fbfbe9db2 API Logs! 2023-03-31 16:29:30 -05:00
084630918f Add check for records pre-delete action (for security and better errors); 404s and ids in 207s for bulk update & delete; ignore non-editable fields; 2023-03-31 12:15:17 -05:00
21e3cdd0a5 Making log warn/error messages not have unique values, but instead to use logPairs (for easier grouping in loggly) 2023-03-31 08:25:34 -05:00
5babdd11b6 fixing returned records from memory store to be clones 2023-03-30 19:24:13 -05:00
7e368c6ff9 Adding associated records to Get, Query. 2023-03-30 16:56:18 -05:00
3df4513cd1 Add security, required fields, record-exists validation to UpdateAction. refactor InsertAction to help there 2023-03-30 15:24:40 -05:00
adcddff440 Add 429 (Too Many Requests) error, as option 2023-03-30 11:56:29 -05:00
a9e793dfb8 Check for required fields 2023-03-30 11:55:36 -05:00
f6f2cb7070 Fix test (good catch - 200 for GET /api/, and 404 for bad version doc pages) 2023-03-30 11:53:11 -05:00
944a93bd99 small api doc cleanup; large refactor (resources as resources, not java strings) 2023-03-30 11:31:04 -05:00
7e6a09fc21 Handle null table labels (probably only happens in test, but 🤷) 2023-03-30 08:30:59 -05:00
f15f63b021 oh checkstyle :) 2023-03-29 19:39:11 -05:00
037d67dd38 Add lock NullValueBehavior ALLOW_WRITE_ONLY... fixes a case where audit couldn't be inserted 2023-03-29 18:11:02 -05:00
b0d0c0ce6c much more flesh in openapi, doc page 2023-03-29 18:09:45 -05:00
4a87856601 look for user (client) name in another place 2023-03-29 18:08:36 -05:00
07aa9cfa27 Turning off some capabilities 2023-03-29 18:08:14 -05:00
ef6ccc61c3 Checking record security locks that are more than 1 join away. 2023-03-29 09:55:11 -05:00
e62d2332ac CTLE-307: updated to sort api fields when building spec 2023-03-28 16:53:08 -05:00
0eff8d7d03 Adding requiredField, security validation to insert action 2023-03-28 10:23:53 -05:00
a64a2801c0 CTLE-307: added handling for translating 'too big' auth0 access_tokens into a smaller uuid when authorizing 2023-03-27 21:24:30 -05:00
a43660a05a Manage associations in UpdateAction 2023-03-27 15:07:23 -05:00
df259b5f82 basic validation on Associations 2023-03-27 15:06:58 -05:00
7034671070 Revert -T4 2023-03-27 14:59:42 -05:00
6629015fbf Pin localstack to versiosn 1.4, which seems to NOT be borken. 2023-03-27 14:42:52 -05:00
e9c66b48f2 temp turn of -T4 on mvn verify 2023-03-27 14:07:18 -05:00
ba805a4c92 Initial support for associated records (implemented insert, delete).
Include "api" on audit.
2023-03-27 09:52:39 -05:00
17d4c81cc3 Add LIKE criteria operator; more api endpoints to understand versions, get swagger json; more field name mapping 2023-03-24 10:20:26 -05:00
74cf24a00e Bulk update & delete; errors if more than jsut the expected json 2023-03-23 12:44:40 -05:00
1d2acc7364 Checkstyle!! 2023-03-23 09:21:45 -05:00
11977624bf Add table override name, isExcluded;
Remove more hidden, excluded, etc tables;
Update to comply with rules of openapi spec;
2023-03-23 09:14:56 -05:00
c369430261 implemented replacedBy on removed field; apiFieldName; exclude 2023-03-22 18:38:14 -05:00
94d32fa854 pass tests; redo removed api fields 2023-03-22 15:21:15 -05:00
4da3cc9206 Checkpoint; bulkInsert working; some api instance validation 2023-03-22 15:12:12 -05:00
8ea571ccf7 Remove unused ApiPathNotFoundException 2023-03-22 09:39:48 -05:00
924a6ba31f Checkpoint; working insert, update, delete 2023-03-22 09:33:04 -05:00
3f7f2b58e2 Checkpoint - writing somewhat valid versions of all single-record actions 2023-03-21 17:00:47 -05:00
fd167a7c64 Reverted to copy array, fixes test 2023-03-21 15:17:11 -05:00
f311c7af88 Build out all query operators 2023-03-21 15:02:07 -05:00
af425bd567 exclude model classes from coverge checkstyle-idea.xml 2023-03-21 13:28:53 -05:00
c6fa22524c Coverage on query 2023-03-21 11:50:54 -05:00
303cd4aec0 Passing tests? 2023-03-21 11:13:12 -05:00
90a7745246 cehckpoint - adding security to openapi spec 2023-03-21 11:00:10 -05:00
4a29898405 Working tests 2023-03-21 08:54:16 -05:00
4485898d0e Checkstyle 2023-03-21 08:06:04 -05:00
8924343490 Checkpoint, semi-working query endpoint 2023-03-21 07:59:15 -05:00
f13ee0d1ca Refactoring to work with new API middleware routes 2023-03-20 14:35:31 -05:00
d0e8bd9db2 Initial checkin of qqq-middleware-api 2023-03-20 14:34:39 -05:00
d6a9c8f0e0 Add instance, table, and field-level middleware meta-data 2023-03-20 14:33:27 -05:00
8cfa2736da Add jackson-dataformat-yaml and Initial checkin 2023-03-20 14:33:06 -05:00
9825893f9b Initial checkin 2023-03-20 10:58:46 -05:00
e04d50674b updated to return exception, now that it's in output, not thrown 2023-03-16 16:08:07 -05:00
c9e86896e5 Merge pull request #13 from Kingsrook/feature/column-stats
Feature/column stats
2023-03-16 12:07:31 -05:00
b16eaca394 Let caller specify type to use for an aggregate expression 2023-03-16 11:38:26 -05:00
939dcc308c Fix test; add comment 2023-03-16 09:06:09 -05:00
bf44f97630 Renamed tableStats to columnStats 2023-03-16 07:53:58 -05:00
9391284479 Adjust field labels 2023-03-16 07:53:58 -05:00
ad2eff0e73 More stats & aggregates 2023-03-16 07:53:58 -05:00
bbde64b02d Add table permission check; add display & possible values; 2023-03-16 07:53:58 -05:00
001ec3a34a Add overload that works on list of fields rather than table 2023-03-16 07:53:58 -05:00
dd28c95fc0 Use sessino from context, not input 2023-03-16 07:53:58 -05:00
da17145f66 WIP version of table/column stats process & supporting aggregate changes 2023-03-16 07:53:58 -05:00
03e9f27866 Remove Auth0PermissionsHelper (move to CTL CLI, where it is used) 2023-03-15 18:07:14 -05:00
248db43c8f different exception propagation. 2023-03-15 17:55:53 -05:00
9cbb899788 Add getUsers, getRoleName, getUserName, and getPermissionsForUser 2023-03-15 17:03:09 -05:00
d569541b77 Add some with'ers 2023-03-15 17:03:09 -05:00
21500b642f Add script maxBatchSize, to influence pipe capacity to avoid pipe-full-too-long errors; add link to script logs after running process; add logs to script view screen 2023-03-15 17:03:09 -05:00
fb6cef66ef 2 bugs: clone the filter, and ignore empty-string in condition 2023-03-15 17:03:09 -05:00
4d7c7f48be Add maxRows field (todo - show in UI if you didn't fetch all?) 2023-03-15 17:03:09 -05:00
0e01372200 Make pipeLoop minRecords a parameter; add input to getOverrideRecordPipeCapacity 2023-03-15 17:03:09 -05:00
b6e089a364 Initial checkin 2023-03-15 17:03:09 -05:00
61286cf013 added test to confirm cache use case exclusions support OR boolean operation 2023-03-15 16:21:44 -05:00
7150886e39 Merge branch 'feature/handle-api-request-failures' into dev 2023-03-15 12:04:22 -05:00
7ca9ecbcec Fix to clone a possibleValueSource filter before calling interpret values... added warning to the javadoc on that method - how to make better? 2023-03-15 11:49:40 -05:00
1429d1000c fixed typo, updated validate response to take a list of QRecord objects 2023-03-15 10:50:23 -05:00
54d3e4a6c8 Better/more timezone support 2023-03-14 16:32:34 -05:00
b395ee6778 Initial checkin 2023-03-14 16:32:34 -05:00
ffc574e83f another attempt at fixing binding instants 2023-03-13 20:39:23 -05:00
e0f5c3ff49 added ability to use filters to stop certain records from being cached, fixed insert bug due to binding instants to its default timestamp 2023-03-13 20:02:37 -05:00
ec05f7ab7e Switch to bind Instants as strings instead of timestamps - seems to fix some timezone issues. 2023-03-13 10:41:49 -05:00
33f4c1235a Fixed NPE for null instant 2023-03-13 08:37:35 -05:00
ad8ed4c574 Fix division by zero (check bigDecimal.compareto, not equals 0) 2023-03-09 18:19:02 -06:00
bfe138c018 Fix checkstyle 2023-03-09 16:44:03 -06:00
054c34918d add testScriptProcess 2023-03-09 16:39:14 -06:00
7956c8f455 Added new files missed in last commit 2023-03-09 11:41:03 -06:00
3a172b3fb4 Make postRun methods take a subclass of backendInput/Output, that make it clear they don't have the full record list 2023-03-09 11:36:21 -06:00
e53a982d12 Merge branch 'CTLE-346-add-client-warehouse-int' into dev 2023-03-09 10:01:57 -06:00
02c51ae9ab fixed tostring 2023-03-09 09:43:14 -06:00
eae164c686 Add toStrings 2023-03-09 09:39:15 -06:00
10afa1a80e update to not just assume object is a JSONArray, but to check it to try to avoid some type errors 2023-03-08 16:34:31 -06:00
2164c2115d CTLE-346: fixed select count clause syntax error 2023-03-08 15:56:54 -06:00
00baf64587 Merge pull request #12 from Kingsrook/CTLE-346-add-client-warehouse-int
added ability to log sql to system out, added handling for when joins…
2023-03-08 14:17:37 -06:00
5368903723 CTLE-346: updates from feedback 2023-03-08 14:17:01 -06:00
81a8f100b9 Merge pull request #11 from Kingsrook/feature/record-scripts
Feature/record scripts
2023-03-08 12:39:27 -06:00
7e87950ef5 Fixing (?) placement of when's for circleci 2023-03-08 12:31:56 -06:00
d7abab2fd1 added ability to log sql to system out, added handling for when joins happen and the key field is on the many side 2023-03-08 12:31:36 -06:00
55fa105797 Add when: always to all test reporting steps (store jacocos and find un-tested) 2023-03-08 11:23:48 -06:00
259329e9aa Try to fix find un-tested classes w/ pipefail 2023-03-08 11:13:15 -06:00
46baceee31 More robust test (not based on exact number of tables) 2023-03-08 10:55:08 -06:00
8131dcc644 Updated messages thrown by some non-findable joins 2023-03-08 10:48:09 -06:00
f4ea645055 Fix find un-tested classes; remove slack 2023-03-08 10:24:06 -06:00
f454e0aefa add pvs filters (via post) to table endpoint; more test coverage, plus maybe report on untested classes in ci 2023-03-08 10:18:42 -06:00
11a16590ef Possible value source filtering 2023-03-08 08:39:09 -06:00
5ca3c088a6 Fixed test 2023-03-07 16:59:33 -06:00
1212b47926 Getting test coverage above bar 2023-03-07 16:54:47 -06:00
9526c0b59f Updating to pass tests 2023-03-07 15:51:11 -06:00
1ebe43fe6f Support for run scripts process on table query screen 2023-03-07 15:18:39 -06:00
2162a6832b Removed zombie code 2023-03-07 15:18:17 -06:00
d0fb8c33e9 Update to a workable MVP version of running scripts as table automations 2023-03-07 15:18:08 -06:00
eafa82eb85 initial checkin 2023-03-07 15:14:15 -06:00
47d2291d96 Better handling of joins (flip the join-on if needed) 2023-03-07 10:24:51 -06:00
68686c0e17 Checkpoint 2023-03-07 10:24:51 -06:00
22644b6a36 Initial WIP on record scripts 2023-03-07 10:24:51 -06:00
c640561f53 Initial checkin 2023-03-07 10:24:51 -06:00
c091440848 Add overload of TableSyncProcess.Builder.withExtractStepClass 2023-03-07 10:15:41 -06:00
0a2d1d66da Don't reset qqq-frontend-material-dashboard to SNAPSHOT 2023-03-06 09:51:15 -06:00
9eaa04abbf Initial checkin 2023-03-03 11:24:30 -06:00
60056caa2b Merge branch 'feature/sprint-21' into dev 2023-03-02 15:17:27 -06:00
0c9d6ba912 Attempt at more correct timezone logic in getInstant 2023-03-02 14:54:45 -06:00
17e9a91c86 Add queue-sizes and ad-hock widget value sources; fix some npe's in widget calculation 2023-03-02 14:51:09 -06:00
f0fd0d3fe6 removed the whole omit reload thing which only half worked 2023-03-01 20:08:38 -06:00
d41e92d213 Add new pluralFormat method... 2023-02-27 11:01:29 -06:00
4f3c03de1a Let StreamedETLWithFrontendProcesses have different transaction levels 2023-02-27 10:27:11 -06:00
ea731bac5c Fix test for step without name (enricher now handles) 2023-02-24 17:04:40 -06:00
8a8f0d6e6f avoid run-away PVS caching - clear if over 50,000 entries. 2023-02-24 16:23:08 -06:00
1baf7d8f86 Give good error message if running in a QQQ instance without script tables 2023-02-24 16:18:05 -06:00
4db174b66d Make debug-logging SQL controlled by system property 2023-02-24 16:15:56 -06:00
5074ed1867 infer backend process step name from code name, if present 2023-02-24 15:45:03 -06:00
9be0eb9e76 push & pop action (to get process name in audit) 2023-02-24 15:44:10 -06:00
94a970b8e8 Add warningIcon 2023-02-24 12:42:24 -06:00
80eee299d7 Update to call updateStatusOnlyUpwards 2023-02-23 18:53:01 -06:00
b5aa8e8152 Adding QJavalin Process Handler Test for possibleValues fields in process. 2023-02-23 14:00:19 -06:00
ea795ed701 Missed things re: custom pvs 2023-02-23 13:19:13 -06:00
8440536d35 SPRINT-21: added min height 2023-02-23 11:10:34 -06:00
7ea1750800 make processes able to render a no-code widget output! also search on custom PVS's 2023-02-22 17:50:06 -06:00
4cf8e37e7e Add good getFirstNonNull method 2023-02-22 11:22:14 -06:00
1bdce12b8d Renamed several references from nf to ct 2023-02-21 22:30:49 -06:00
f28af62c5e Update insert action to do pre-step - e.g., to prime amazon s3 client 2023-02-21 22:30:49 -06:00
63be3f01a7 Update to not audit automation status changes 2023-02-21 22:19:20 -06:00
f3cf327384 avoid null pointer on empty record list 2023-02-21 22:18:34 -06:00
d5cb752132 Remove auditInput from the process values (so it doesn't get serialized to frontend) 2023-02-21 14:45:24 -06:00
8833563d26 Update to fetch full records for audit purposes, if security key is needed 2023-02-20 09:58:36 -06:00
5a6a0e2ac5 Add method to let some lower-level actions try to generically update counts, but not to go lower than original counts were. 2023-02-20 09:44:08 -06:00
395f18f513 Fix some process frontend serialization issues 2023-02-20 09:43:26 -06:00
67244d6c6e Initial version of abstract merge duplicates process 2023-02-20 09:43:04 -06:00
05a7f9d847 Audit cleanups (process names for automations; no audit if no fields changed; 2023-02-17 10:25:14 -06:00
049c60c6e3 Remove cp to src/main/ui for nf-one 2023-02-17 10:11:28 -06:00
9659e5b9ea updated snapshot version, fixed parsing of pom.xml when determining version number 2023-02-16 13:15:03 -06:00
0e4fef561e Updating to <revision>0.13.0</revision> 2023-02-16 12:53:21 -06:00
ac074b8492 Merge tag 'version-0.12.0' into dev
Tag release
2023-02-16 12:53:17 -06:00
078536a020 Merge branch 'release/0.12.0' 2023-02-16 12:52:20 -06:00
97b22774f1 Update for next development version 2023-02-16 12:51:07 -06:00
194833bf07 Update versions for release 2023-02-16 12:51:05 -06:00
8924657fc1 automatic audits 2023-02-15 16:16:05 -06:00
3071c63857 turning off evaluateDateTimeParamValues 2023-02-14 11:16:25 -06:00
c07237b7d2 Bump 2023-02-14 10:38:30 -06:00
f01a1ac7a1 2nd iteration on no-code dashboards. add conditional filters, timeframes, more utils, calcualtions 2023-02-14 09:00:50 -06:00
7e07fd04a1 SPRINT-20: made pagination options avaialble for table widgets, updated 'primary color' to come from branding metadata 2023-02-13 13:20:31 -06:00
ff6c2b7fa6 First version of no-code dashboard widgets 2023-02-13 10:43:39 -06:00
d9a17ac99b SPRINT-20: fixed more broken tests due to gettablepath sig change 2023-02-08 22:52:11 -06:00
81f9f4e49a SPRINT-20: updated getTablePath to no longer require input param, added some permission checks to widget links, added utils to get zoned starts of day, year, month 2023-02-08 22:46:49 -06:00
927e7f725a initial checkin 2023-02-08 18:12:47 -06:00
07e6c7019d Add filterExpressions as a concept 2023-02-08 18:12:47 -06:00
eae01bb8c4 update to not make a possible-value field be a record-link if it has a chip too. 2023-02-08 18:12:47 -06:00
2d09a521cd updated to do bulk audits better, along with audit details 2023-02-08 18:12:47 -06:00
92f6f7da04 Add auditInputs to process step outputs, and execution of them in streamed ETL Execute 2023-02-08 18:12:47 -06:00
c863027629 Add okToDelete and error lines 2023-02-08 18:12:47 -06:00
b0cca3f1d7 Add ability to disable one-off lookups 2023-02-08 18:12:47 -06:00
e3c4a3d91d add more date/time format methods 2023-02-08 18:12:47 -06:00
0dcc5ef8c5 Add more overloads (message, throwable, ...logPairs) 2023-02-08 18:12:47 -06:00
f1515e2ba4 Update to accept queryJoins in count and query actions 2023-02-08 18:12:47 -06:00
c620917606 Add print stack trace and exception message 2023-02-08 16:58:21 -06:00
700802f329 ((fixed) bad) Test for HttpDeleteWithBody 2023-02-08 15:08:10 -06:00
f6220482cd (bad) Test for HttpDeleteWithBody 2023-02-08 14:56:46 -06:00
1e1ecbccee Run a little code 2023-02-08 14:36:28 -06:00
1863c31907 Initial checkin 2023-02-08 14:33:00 -06:00
95040d06b8 Initial checkin 2023-02-08 14:29:31 -06:00
dd78c7e51f Add appName and companyUrl 2023-02-03 19:23:20 -06:00
84e2a7e718 SPRINT-20: added setting userId security key value upon qInstance instantiation 2023-02-01 21:55:53 -06:00
668bf5e622 Fix the case where the same source record (identified by sourceKey) is passed in the input more than once (fixes some cases of duplicates) 2023-01-31 13:37:16 -06:00
27b48b62f9 Move check forˆ 404 for unknown report name 2023-01-31 13:36:03 -06:00
583d716f94 Add tense-independent singular/plural methods 2023-01-31 13:34:22 -06:00
142bd70212 Revert "Turning up logging"
This reverts commit e75b645639.
2023-01-30 16:53:51 -06:00
13a7281d3a Switch from using the frustratingly ummutable Collections.emptyList to new ArrayList in JoinsContext for delete-by-filter 2023-01-30 16:51:35 -06:00
f61a3a13be fixed snapshot version 2023-01-30 13:59:24 -06:00
3e56bb4ecd Merge branch 'dev' of github.com:Kingsrook/qqq into dev 2023-01-30 13:40:23 -06:00
789a1c8e63 Updating to <revision>0.12.0</revision> 2023-01-30 13:39:10 -06:00
81d8b66c0c Merge tag 'version-0.11.0' into dev
Tag release
2023-01-30 13:39:05 -06:00
a7faf52817 Merge branch 'release/0.11.0' 2023-01-30 13:37:55 -06:00
45c703a6df Update for next development version 2023-01-30 13:31:20 -06:00
5a9a0754a6 Update versions for release 2023-01-30 13:31:17 -06:00
4790f55243 Fixing scheduled process context; better thread names; add serverInfo endpoint 2023-01-26 21:55:24 -06:00
f0450ef621 PRDONE-170 - Adding support for passing in custom ScriptUtls for scripts. 2023-01-26 12:45:07 -06:00
ed5839aa0a Add /run endpoint for running processes w/o any frontend 2023-01-26 11:23:41 -06:00
56f05c74fc Add secrets manager; update org.json 2023-01-26 11:15:47 -06:00
4f8f8bad9a Initial checkin 2023-01-26 11:15:02 -06:00
4a1853faa5 SPRINT-19: added more chart data and widget meta data 2023-01-25 19:13:35 -06:00
c972f9cecc Update to allow _qStepTimeoutMillis to come from formBody 2023-01-25 10:09:45 -06:00
efa796bb39 Add property/env to add processTag to all logs 2023-01-24 15:56:50 -06:00
a05e74a7d0 SPRINT-19: added stacked bar chart, more widget metadata 2023-01-24 15:52:15 -06:00
41fcb09c70 Update to reset memory record store before and after each 2023-01-24 14:22:09 -06:00
6b12390f6c make scheduler look at env var in addition to system property for deciding whether to start or not. 2023-01-24 14:19:55 -06:00
9889320fa6 add call to validate api responses to doInsert (POST) 2023-01-24 14:08:45 -06:00
c00120a1fc add overloads that take 'String message, LogPair... logPairs' 2023-01-24 13:52:44 -06:00
029f436071 Update to cache script and revision ids (could be better); new AccumulatingBuildScriptLogAndScriptLogLineExecutionLogger, to just do 2 inserts across multiple script runs; add child-log-lines to script log table 2023-01-24 11:57:50 -06:00
f1ff53059f Handle null case in convertInputIdsToEnumIdType 2023-01-24 11:08:44 -06:00
18a8656ba2 Fix enum looks by ids of wrong type 2023-01-24 11:04:34 -06:00
e75b645639 Turning up logging 2023-01-20 15:14:10 -06:00
804efbd608 SPRINT-19: fixed xbar images 2023-01-20 13:13:29 -06:00
fe30d2654b update to implicity add a queryJoin, if a filter field calls out a table name that matches a join in the instance 2023-01-20 10:05:44 -06:00
97b803a86a add audit joins, move names to constants; fixes 2023-01-20 10:03:52 -06:00
2daa42aeb1 Merge branch 'feature/sprint-19' of github.com:Kingsrook/qqq into feature/sprint-19 2023-01-19 16:32:58 -06:00
4344fa472a Initial checkin 2023-01-19 16:31:18 -06:00
6bb810c1f4 Good exception for bad table name 2023-01-19 16:30:59 -06:00
d65af53090 Add some withers 2023-01-19 16:30:44 -06:00
ac6a7ba15a fix setValueIfTableHasField - should catch if field isn't found (helps tables w/o createDate work) 2023-01-19 16:12:30 -06:00
1c150e207a add nonNullArray and mergeLists 2023-01-19 16:12:02 -06:00
f9408716ac update to propagate default values into state before frontend steps 2023-01-19 16:11:45 -06:00
396f02265b explicit errors if context isn't setup 2023-01-19 16:11:19 -06:00
a1674792c6 Merge branch 'dev' into feature/sprint-19 2023-01-18 17:36:46 -06:00
22565b3ecd Initial checkin 2023-01-18 14:19:12 -06:00
d2e7b794f4 moving QLogger package 2023-01-18 12:11:40 -06:00
7000da409a moving standard lambdas to common package 2023-01-18 12:11:40 -06:00
3f0e09e32a Adding javalin access logs 2023-01-18 12:11:40 -06:00
70d9d259c1 Adding heavy field concept 2023-01-18 12:11:35 -06:00
4dc9c52ee0 Adding toLogPair method 2023-01-18 11:56:10 -06:00
964405d210 Initial checkin 2023-01-18 11:55:17 -06:00
369ba3c8d7 switch syslog to a json format (via patternLayout) 2023-01-18 11:46:12 -06:00
24d5406ee3 add filterStackTrace; move unsafeSupplier 2023-01-18 11:40:39 -06:00
984012f3a3 heavy fields, qvalueFormatter static 2023-01-18 11:39:47 -06:00
e8264d915f hotfix: updated to move pie chart label to be specified in the metacritic, added 'add' to pom resolver 2023-01-18 11:33:10 -06:00
46adfa8e24 WIP - logger migrations; initial work for data bag view widget 2023-01-17 10:44:45 -06:00
178078282c Switch to use QLogger everywhere 2023-01-17 10:44:45 -06:00
d03e947889 Merge branch 'dev' into feature/sprint-19 2023-01-17 10:29:16 -06:00
43de1cf749 Add post to download endpoint 2023-01-16 09:03:13 -06:00
d3fa1df56f Implementation of QContext everywhere, instead of passing QInstance and QSession in all ActionInputs 2023-01-15 19:41:23 -06:00
69a6104393 updated snapshot version, improved end of sprint script 2023-01-13 15:59:01 -06:00
e831c75e1e Merge branch 'dev' of github.com:Kingsrook/qqq into dev 2023-01-13 15:56:56 -06:00
eba9b0b4af Updating to <revision>0.11.0</revision> 2023-01-13 15:39:36 -06:00
9f82f35bcf Merge tag 'version-0.10.0' into dev
Tag release
2023-01-13 15:39:32 -06:00
ff07490f94 Merge branch 'release/0.10.0' 2023-01-13 15:38:34 -06:00
680696491f Update for next development version 2023-01-13 15:36:06 -06:00
a35e30059c Update versions for release 2023-01-13 15:36:05 -06:00
a37d22b0d0 Update export to work off post; add check for Authorization as a form param 2023-01-13 14:13:21 -06:00
9e02476ee7 Merge branch 'dev' 2023-01-13 11:02:59 -06:00
2b0974f4a5 Remove single-parent concept on app-children; more working version of recordLock from join 2023-01-13 09:42:31 -06:00
9a58c7683b SPRINT-18: updated module list to include slack and remove lambda 2023-01-12 21:07:55 -06:00
a556bf9764 SPRINT-18: added slack middleware module 2023-01-12 20:16:06 -06:00
41877a7055 Update to not setup a join context for non-table data sources 2023-01-12 08:38:09 -06:00
c07a77d4a6 Updated to disable view-all link if tablePath can't be found (e.g., because user can't view table) 2023-01-11 16:44:42 -06:00
23e9abeb74 implementation of record security locks, and permissions 2023-01-11 13:08:59 -06:00
e4d37e3db9 Add variant to pre-load by in-list, to cache misses as null 2023-01-05 14:33:32 -06:00
4b31a8b4bb Merge branch 'refs/heads/0.7.0-12-hotfix' into feature/sprint-18 2023-01-05 14:26:20 -06:00
d7e7315dc8 Merge branch 'feature/auth0-basic-auth-reuse' into feature/sprint-18 2023-01-05 11:36:39 -06:00
de05e4ae58 SPRINT-18: fixed unit test on auth0 reuse 2023-01-04 22:20:38 -06:00
150582964b Change setupSession to be public 2023-01-04 12:05:19 -06:00
2874b98b66 Add re-use of tokens from basicAuth 2023-01-04 10:07:44 -06:00
7fae3e2329 Add table-based authentication module; update javalin to support Authentication: Basic header; Move authentication classes 2022-12-28 17:00:08 -06:00
428f48602b Better testing on join reports, possible value translations; renamed left & right in QueryJoin (now joinTable, baseTable) 2022-12-22 13:46:32 -06:00
799b695e14 Checkpoint on report and export changes, possible value translating 2022-12-21 11:37:16 -06:00
19d88910b5 Updating table sync api 2022-12-20 10:56:59 -06:00
2ad4b22f55 Add withSchedule method 2022-12-19 15:02:20 -06:00
040dae55d5 Add withSchedule; fix return on withBasepullConfiguration; add overload withTransformStepClass 2022-12-19 14:59:58 -06:00
dd9253fde4 Updated comment about field behaviorfs 2022-12-19 14:56:37 -06:00
8a4d5bfb34 Update to not NPE if data source doesn't have a filter 2022-12-19 14:56:25 -06:00
e1c53b9d48 Updated interface in sync processes; more status updates in ETL processes; Basepull only update timestamp if ran as basepull; javalin report endpoint; 2022-12-19 12:04:01 -06:00
a6656af040 Updated for older api on QQueryFilter (noarg constructor) 2022-12-15 16:18:14 -06:00
0e68bf1e72 Hotfix - sqs batch mode, and rdbms delete-by-filter 2022-12-15 16:13:56 -06:00
1b672afcd0 Fixed rdbms delete test 2022-12-15 16:04:46 -06:00
5005c38c18 Fixes for performance of sqs (batch mode), plus bug in deleteAction by query-filter (with test that proves it) 2022-12-15 15:54:58 -06:00
1cdf437551 SPRINT-18: upped current snapshot version 2022-12-15 15:39:33 -06:00
04cddb8d5d Updating to <revision>0.9.0</revision> 2022-12-15 15:34:43 -06:00
0d6bd013f5 Merge branch 'release/0.9.0' 2022-12-15 15:33:49 -06:00
7b6284c06f Update for next development version 2022-12-15 15:24:00 -06:00
edcc27485e Update versions for release 2022-12-15 15:23:59 -06:00
280373ddc5 SPRINT-17: added code coverage test class to meet minimum requirements 2022-12-15 10:36:43 -06:00
25815ebc25 SPRINT-17: updated some widgets to look less broken when data is 'not available now', checkpoint commit on 'real dashboards' 2022-12-15 10:20:48 -06:00
59cbf83860 Add field increaseIsGood 2022-12-14 14:53:03 -06:00
6186b17e92 Add environmentValues to qInstance 2022-12-14 14:50:45 -06:00
30003b729c Add overload of getForeignRecordMap that takes additional (base) filter for query 2022-12-09 20:25:59 -06:00
293b3e4207 Add qruntime exception; let transform step set pipe capacity 2022-12-09 16:44:53 -06:00
470321dcf6 add equals and hashcode 2022-12-09 16:06:26 -06:00
48cfdeffa1 Change post-query customizer to be class (that can do list), not function 2022-12-09 12:05:39 -06:00
14c7fbe370 Moving dropdowns to work for all widgets 2022-12-09 09:53:13 -06:00
8454f94020 SPRINT-17: updated dropdowns to be required, added divider 2022-12-08 15:06:40 -06:00
697261f91b Add aHrefViewRecord 2022-12-08 10:37:11 -06:00
61d493a4f5 Add FieldValueList widget type, more html-helper methods 2022-12-08 09:18:52 -06:00
c21c89e85f SPRINT-17: test was not using groupby to fetch values 2022-12-07 16:43:27 -06:00
a9a3e3b19e SPRINT-17: fixed dumb checkstyle violation 2022-12-07 16:31:04 -06:00
5cf9e7b60d SPRINT-17: fixed failing test and removed pvs name member from parent meta data 2022-12-07 15:47:15 -06:00
9b34ee7fe7 SPRINT-17: updates to parent widget dropdown data, updated group bys to be objects allowing group by with custom formats 2022-12-07 15:31:48 -06:00
241741e2e5 Fix default for canAddChildRecord to be false; add more link-generating functions (to child-modals on view screens, etc) 2022-12-06 15:57:32 -06:00
a769d8942c Adding unique key check to insert action; adding post-insert customizer 2022-12-05 15:46:17 -06:00
c22fc89cbb SPRINT-17: changed some variable names 2022-12-05 12:18:23 -06:00
060da69afb Adding table-cacheOf concept; ability to add a child record from child-list widget 2022-12-05 10:26:53 -06:00
3691ad87e5 Merge remote-tracking branch 'origin/0.7.0-12-hotfix' into dev 2022-12-01 16:59:19 -06:00
583ac70563 Merge tag 'version-0.8.0' into dev
Tag release
2022-12-01 16:56:13 -06:00
2afd8864a8 Merge branch 'release/0.8.0' 2022-12-01 16:53:46 -06:00
28e0bdadbf Update for next development version 2022-12-01 16:49:34 -06:00
c856a7b0b3 updated versions for release 2022-12-01 16:33:37 -06:00
3a721d8df5 cd to correct directory after environment setup 2022-12-01 15:56:52 -06:00
ab08a99477 updated to move to proper directory before beginning work 2022-12-01 15:44:34 -06:00
e170dab1db SPRINT-16: added ERROR adornment type 2022-11-30 16:52:20 -06:00
40afb629e7 Add overrideBatchSize to table automation details 2022-11-30 11:30:18 -06:00
ce2cccfa2a SPRINT-16: added more tests 2022-11-30 11:29:48 -06:00
6813617a21 SPRINT-16: added new widget types, moved some things to a different package, etc. 2022-11-29 14:34:59 -06:00
2df9576e20 Initial checkin 2022-11-28 15:19:14 -06:00
792383d21a Add ConvertHtmlToPdfAction 2022-11-28 11:52:48 -06:00
b2d76e8206 Much implementation of joins for RDBMS 2022-11-23 16:37:54 -06:00
6685e61500 Merge branch 'dev' into feature/sprint-16 2022-11-22 13:01:50 -06:00
aef8f5cd59 Adding maxLength to fields, along with initial version of FieldBehviors and ValueBehaviorApplier, including ValueTooLongBehavior 2022-11-22 12:59:57 -06:00
26dd323f34 SPRINT-16: upped current snapshot version 2022-11-22 12:24:06 -06:00
209ada8065 Add more util methods; add AbstractHTMLWidgetRenderer 2022-11-21 14:46:14 -06:00
105b2c92c9 Add aggregateAction; Add renderTemplateAction 2022-11-18 16:51:54 -06:00
1d1461deea Adding filesystem writing - used by javalin to store uploaded files; done async, via new base class for actions 2022-11-17 19:59:29 -06:00
6b2860e303 Initial checkin 2022-11-17 15:59:13 -06:00
0233edc7b2 Updating to <revision>0.7.0</revision> 2022-11-17 12:13:02 -06:00
f2dbe5bf12 Merge branch 'release/0.7.0' 2022-11-17 12:12:12 -06:00
31c1a8ecb6 Update for next development version 2022-11-17 12:07:27 -06:00
373ff20e9d Update versions for release 2022-11-17 12:07:26 -06:00
3305fb76d5 Fixed to insert & update correct lists 2022-11-16 14:42:22 -06:00
d2d32e5ab1 SPRINT-15: reverted log change 2022-11-16 10:22:16 -06:00
2920774266 SPRINT-15: reverted log change 2022-11-16 10:18:57 -06:00
03bce59f2a SPRINT-15: reverted log change 2022-11-16 10:14:14 -06:00
955a30a660 SPRINT-15: attempt to improve json logging 2022-11-16 08:55:02 -06:00
d2bfe9f610 SPRINT-15: attempt to improve json logging 2022-11-15 21:44:28 -06:00
798dc6e6e5 Merge branch 'feature/sprint-15' of github.com:Kingsrook/qqq into feature/sprint-15 2022-11-15 21:15:23 -06:00
126d51c8f9 SPRINT-15: attempt to improve json logging 2022-11-15 21:14:06 -06:00
7d72487a75 Initial checkin 2022-11-15 15:41:46 -06:00
848a09cef5 SPRINT-15: attempt to fix loggly datetimes 2022-11-15 12:17:32 -06:00
322efa2102 Initial checkin 2022-11-15 10:47:49 -06:00
20607bbf9b SPRINT-15: added initial version of the QLogger class 2022-11-14 17:37:47 -06:00
430e1bc9c7 Add table name to record pipe loop 2022-11-14 14:00:55 -06:00
2a7e76b0f9 Add joins and ChildRecordList widget 2022-11-14 14:00:55 -06:00
f0a464ce9e SPRINT-15: trying to fix circle ci 2022-11-14 12:30:02 -06:00
e305c64572 SPRINT-15: trying to fix circle ci 2022-11-14 12:26:56 -06:00
514f105c86 SPRINT-15: ¯\_(ツ)_/¯ 2022-11-14 12:06:57 -06:00
87dc7fd96c SPRINT-15: adding reinstalling of ca-certificates to try to fix circle ci failures 2022-11-14 11:38:20 -06:00
d54e0c71a8 SPRINT-15: moved response validation into its own method which can be overriden in subclasses 2022-11-14 10:47:02 -06:00
a4ad8ac08a SPRINT-15: added default constructor 2022-11-13 14:35:46 -06:00
b2aaffeb6d SPRINT-15: updates to logging 2022-11-11 21:48:50 -06:00
11aa55cf40 SPRINT-15: refactor of api actions, moving logics into utils class, unified all calls to apis, clean ups, etc. 2022-11-11 21:39:05 -06:00
a2da8c4127 Add withValues(Pair...) and iconAndColorValues 2022-11-11 16:20:50 -06:00
8b31cee890 Add TableSyncProcess 2022-11-11 14:34:13 -06:00
a5ec33b51b Script Tests and further enhancements 2022-11-10 14:22:43 -06:00
6d08afa4c2 SPRINT-15: added json util helper method 2022-11-10 13:21:11 -06:00
6496986aab bump 2022-11-09 11:05:08 -06:00
f9c14eb08c update to use same try-with-resources for CloseableHttpClient and CloseableHttpResponse 2022-11-09 11:02:25 -06:00
230dde2e52 Refactoring javascript executor scripts.main error handling 2022-11-09 10:36:37 -06:00
e701ae0ea3 add try-catch around script.main business 2022-11-09 10:03:14 -06:00
955294ae18 Update javascript executor to work w/ compiled ts scripts that export a main function; add output to javalin storeRecordAssociatedScript 2022-11-09 09:49:58 -06:00
2975b90505 Allowing load step to set pipe sizes to avoid 'Giving up adding record to pipe' in easypost tracker creation; Make status never have current > total; 2022-11-09 08:44:16 -06:00
1e09931218 SPRINT-15: updated apiupdateAction to handle 200/207 better, added better exception handling 2022-11-08 16:33:06 -06:00
8932ef891e ADd method objectToMap 2022-11-08 09:28:37 -06:00
1a287fe35a Log and thread name adjustments - trying to make loggly more useful 2022-11-08 09:08:15 -06:00
236eff523e SPRINT-15: added methods to get lists and maps of record entities 2022-11-04 14:28:15 -05:00
e815bd37ab Merge branch 'dev' into feature/sprint-15 2022-11-04 10:02:33 -05:00
669b6d3cb7 Adding status object in standard loadVia steps and updating it in api insert; add user timezone header to session 2022-11-04 09:59:07 -05:00
f99430d2bc sprint-15:updated current snapshot version 2022-11-03 15:09:28 -05:00
ebf9556a5d Merge branch 'feature/sprint-14' into feature/sprint-15 2022-11-03 14:36:32 -05:00
a3f9df09a9 Updating to 0.7.0 2022-11-03 14:31:42 -05:00
2105e3dfc5 sprint-14: changed update to do a post not put 2022-11-03 14:29:40 -05:00
99561ecf81 Merge tag 'version-0.6.0' into dev
Tag release
2022-11-03 11:57:15 -05:00
30ecd2d331 Merge branch 'release/0.6.0' 2022-11-03 11:56:25 -05:00
bd85526261 Update for next development version 2022-11-03 11:45:59 -05:00
11648a5759 Update versions for release 2022-11-03 11:45:58 -05:00
e433b78e5b Merge pull request #10 from Kingsrook/feature/sprint-14
Feature/sprint 14
2022-11-03 11:43:31 -05:00
ce9316453b PRDONE-136 - Adding support for basic auth login via auth0 2022-11-03 11:36:18 -05:00
ab0b38dd82 Add RunReportForRecordProcess; 1st version of AbstractProcessMetaDataBuilder 2022-11-03 10:29:44 -05:00
3c04841f73 sprint-14: initial checkin of api update action 2022-11-02 11:04:05 -05:00
683b3c658d Cleanup from code review 2022-11-01 16:21:30 -05:00
165583cd98 Initial version of scripts, javascript 2022-10-31 15:48:27 -05:00
0ada444fd4 sprint-14: moved 'final' before 'static' 2022-10-28 11:12:57 -05:00
662fefea19 sprint-14: put json data into backend details 2022-10-28 11:09:35 -05:00
1cdb4b37e9 sprint-14: fixed test which expected an instant but now receives a string 2022-10-27 13:22:14 -05:00
622183e276 sprint-14: moved getting of timestamps into their own methods that can be overridden by subclasses 2022-10-27 13:17:38 -05:00
80af7a3710 Add RunProcessAction.BASEPULL_READY_TO_UPDATE_TIMESTAMP_FIELD 2022-10-27 11:17:08 -05:00
e6db303bc2 Fix last build 2022-10-27 11:12:27 -05:00
da116349ff Update to only update basepull timestamp after execution; make sure preRun runs before count 2022-10-27 11:02:47 -05:00
5e7b6f40df Add path to icon; other cleanup 2022-10-26 18:09:07 -05:00
82810c2b66 Add ExtractViaBasepullQueryStep; add pagination & piping to api query 2022-10-26 12:31:09 -05:00
77927fd318 sprint-14: minor updates to allow flexibility when extending loadviainsertorupdatestep 2022-10-26 11:56:17 -05:00
033dbeb76c Update to run an executeStep pre-action 2022-10-25 13:31:48 -05:00
8ffc1c1a63 udpated api json parsing (lenient mode); add escaping table names in rdbms 2022-10-25 10:47:06 -05:00
dae803f709 Initial checkin of QueryStringBuilder 2022-10-25 09:55:28 -05:00
128c379f10 sprint-14: initial checkin of basepull capability on processes 2022-10-24 17:06:11 -05:00
44537e182d Add dedicated method for api count in baseApiActionUtil; improve null handling in json to record 2022-10-24 15:34:37 -05:00
22b2e01cca Add withSectionOfFields 2022-10-24 08:40:04 -05:00
f5f6446069 Add GET action, and usage in API 2022-10-21 14:40:38 -05:00
204d67dd21 Merge branch 'main' into dev
# Conflicts:
#	qqq-backend-core/pom.xml
2022-10-21 13:42:30 -05:00
234ec4823b Merge pull request #8 from Kingsrook/feature/sprint-11
Feature/sprint 11
2022-10-21 12:21:40 -05:00
fa2ab18e30 Add unique keys, and checking of them in bulk load; add some more validation (sqs and unique keys) 2022-10-21 11:23:30 -05:00
20c42deae5 Fix where automation status got set to OK instead of running; switch to do an automation polling thread per-table/status 2022-10-20 10:48:58 -05:00
8b3b300eb1 Add custom values to api backend meta data 2022-10-19 18:02:35 -05:00
18a3f72c4a updated api backend to support count and query 2022-10-19 10:43:39 -05:00
bf3835bd4c Set to 0 covered ratio 2022-10-19 10:35:19 -05:00
84ccd92a6e Add rate-limit records to output 2022-10-19 10:33:01 -05:00
0c37630754 Disablign in CI 2022-10-19 10:32:51 -05:00
98e846d1f1 add try-catch around value casting 2022-10-19 09:27:41 -05:00
b12de62295 Fix bulk process PVS rendering 2022-10-19 09:09:44 -05:00
bb975e7c6a Merge pull request #9 from Kingsrook/dependabot/maven/qqq-backend-core/com.fasterxml.jackson.core-jackson-databind-2.13.4.1
Bump jackson-databind from 2.13.2.1 to 2.13.4.1 in /qqq-backend-core
2022-10-19 08:40:31 -05:00
8423a8db2e Bump jackson-databind from 2.13.2.1 to 2.13.4.1 in /qqq-backend-core
Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.1 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-19 06:42:19 +00:00
2d3d1091fd Add basic rate limit handling for POST 2022-10-18 11:12:25 -05:00
6345846eba Adding queues and queue providers; Adding schedules and ScheduleManager 2022-10-18 09:00:21 -05:00
888239b265 Refactored lambda to do standard qqq actions (once implemented) 2022-10-14 16:55:59 -05:00
e3cc2e63f6 Initial checkin of lambda module 2022-10-14 11:45:15 -05:00
dbfc2fe2d8 Initial checkin of lambda module 2022-10-14 11:07:43 -05:00
37fbfd1c7c Update to accept count filter as POST 2022-10-14 10:19:21 -05:00
117bb621ff Update to accept query & filter as POST 2022-10-14 10:11:53 -05:00
7c339b4e81 Add qqq-backend-module-api 2022-10-12 18:24:17 -05:00
471954e8b9 Initial checkin of API module 2022-10-12 18:00:08 -05:00
b91273a53a Add LoadViaInsertOrUpdateStep; make PVS field labels not have Id suffix; add populateFromQRecord 2022-10-11 16:37:33 -05:00
aa64f1b7f3 Update to only call consumer after the loop when it's smart to do so 2022-10-11 11:30:28 -05:00
6ec6b173b9 Updated to not try to prime PVS cache for non-table PVS's 2022-10-11 08:41:45 -05:00
c43a8998ec Updating to support possible value searching 2022-10-10 17:01:15 -05:00
262038bc87 Adding booleanOperator and subFilters to QQueryFilter 2022-10-10 09:13:16 -05:00
28060e95e3 Fixed NPE with null value from possible values 2022-10-07 12:24:27 -05:00
d0c26719e1 SPRINT-12: added test for Qbackend module interface 2022-10-07 10:19:23 -05:00
1548d7f35b Initial checkin 2022-10-07 10:14:35 -05:00
e53d559b29 SPRINT-12: added process step class validation, added noop transform step 2022-10-06 19:44:27 -05:00
73df50add1 Handle booleans better 2022-10-04 11:33:04 -05:00
c5a3534d43 Fix query bugs w/ in-list, empty; format booleans as Yes/No 2022-10-04 09:56:33 -05:00
e3903c0ab9 SPRINT-12: fixed broken test due to types on widgets now 2022-10-03 14:15:46 -05:00
ba4fddb7e5 SPRINT-12: fixed broken test due to types on widgets now 2022-10-03 14:07:10 -05:00
bb85d362fb SPRINT-12: added widget meta data 2022-10-03 14:02:55 -05:00
7415732c6c Update to translate possible values when building other possible values... 2022-10-03 13:31:13 -05:00
21cd07b2df Update scrubValues method to make instants out of DateTimes - fixes update actions in javalin apps 2022-10-03 10:40:45 -05:00
3f84271a36 Feedback from code reviews 2022-10-03 09:09:06 -05:00
17cace070c Change process summary line to interface; add record-link summary-line 2022-09-30 13:32:18 -05:00
456364de2a Add static data provider capability to reports 2022-09-29 16:55:08 -05:00
2d2eae5c06 Add rollback of transaction in last test 2022-09-29 14:39:43 -05:00
86ebe6ee4e Add transaction to transform step and query action (and rdbms query) 2022-09-29 14:34:51 -05:00
8a1110abf0 SPRINT-12: added stepper widget, added linkability to table widget 2022-09-29 12:04:24 -05:00
720200b6cf Update to clone query filters 2022-09-28 14:49:46 -05:00
70ded4c887 Add conversion of instant to localDate 2022-09-28 08:22:51 -05:00
89c9c72772 make all the method names not be the same 2022-09-27 19:05:10 -05:00
4d16bc0fc7 Auto-adorn PVS tables with link-to-record; change query to fetch instants, not LocalDateTimes 2022-09-27 18:59:58 -05:00
3587cc0676 Updated the update action, to set a null value for fields that came in the request as empty string 2022-09-27 14:14:46 -05:00
833b1f9643 SPRINT-12: added 'multi statitistics' widget 2022-09-27 14:14:36 -05:00
c002522cb8 Update to use column labels, when specified, in reports 2022-09-26 11:27:50 -05:00
3402a20b04 SPRINT-12: added rawHTML widget type 2022-09-26 10:18:25 -05:00
ac82928dd7 Do possible-values before display values, so a rendered possible-value can be part of a record label 2022-09-26 08:37:05 -05:00
d73e546c7b Add field adornments (links, chips, widths) 2022-09-23 17:01:46 -05:00
3ac6b34963 SPRINT-12: more tests fixed 2022-09-23 17:00:52 -05:00
742cf7fa3b SPRINT-12: fixed compile error due to merging 2022-09-23 16:26:13 -05:00
ce48933cbd SPRINT-12: fixed style issue and broken tests 2022-09-23 16:21:38 -05:00
5efd2da636 SPRINT-12: updated to define widgets at table level, refactoring of some of the widget stuff to match other "Action"s 2022-09-23 15:55:27 -05:00
3d07d215a9 Adding format date & time methods 2022-09-23 14:17:08 -05:00
9397934769 QQQ-42 adding data-sources to reports, customizer points 2022-09-23 09:55:34 -05:00
f83d2b3fc8 Merge branch 'QQQ-41-v-2-of-app-home-pages-dashboards-etc' into feature/sprint-11 2022-09-20 15:20:05 -05:00
d8c6bba6b4 QQQ-41: checkpoint commit for moving to next sprint 2022-09-20 15:18:49 -05:00
d5a319c458 Merge branch 'feature/QQQ-42-reports' into feature/sprint-11 2022-09-20 12:49:37 -05:00
1f546d8c7d QQQ-42 checkpoint of qqq reports 2022-09-20 12:46:32 -05:00
197dec3105 Merge remote-tracking branch 'origin/QQQ-41-v-2-of-app-home-pages-dashboards-etc' into feature/QQQ-42-reports 2022-09-19 13:52:54 -05:00
525389e62e QQQ-42 checkpoint of qqq reports 2022-09-19 13:52:43 -05:00
80ff7a26e0 QQQ-41: fixed failing test 2022-09-16 11:01:26 -05:00
0b10717116 Merge branch 'QQQ-41-v-2-of-app-home-pages-dashboards-etc' into feature/sprint-11 2022-09-16 10:51:24 -05:00
8891b3e7ea QQQ-41: added app sections, new widgets, ability to have no tables under app, etc. 2022-09-16 10:19:06 -05:00
01afdaacfd More live templates 2022-09-14 14:03:29 -05:00
112c62d035 Initial checkin 2022-09-14 13:59:34 -05:00
1e1c07958b Merge branch 'feature/QQQ-42-reports' into feature/sprint-11 2022-09-14 13:12:13 -05:00
b05c5749b4 QQQ-42 initial implementation of qqq reports (pivots, WIP) 2022-09-14 13:05:31 -05:00
247e419e07 Updated for new assumptions re: fieldSection labels 2022-09-09 15:19:57 -05:00
2f787036d0 Update to set QFieldSection label from name; try to avoid backend-module-not-defined errors 2022-09-09 15:13:28 -05:00
a1f5e90106 Initial import from qqq-dev-tools standalone repo 2022-09-09 15:00:20 -05:00
bee8a7a2d9 Merge tag 'version-0.5.0' into dev
Tag release
2022-09-08 11:18:10 -05:00
d34555cfb2 Merge branch 'release/0.5.0' 2022-09-08 11:17:25 -05:00
9a95fe36e6 Update for next development version 2022-09-08 11:12:56 -05:00
79607a8cac Update versions for release 2022-09-08 11:12:55 -05:00
278910e4ee Merge pull request #7 from Kingsrook/feature/sprint-10
Feature/sprint 10
2022-09-08 11:09:43 -05:00
0d0a7aec1c Initial checkin 2022-09-08 10:58:21 -05:00
b01758879c QQQ-37 Redo bulk processes in streamed-etl mode 2022-09-07 16:59:42 -05:00
1c75df3a09 added initial version of branding as metadata 2022-09-06 19:00:12 -05:00
25c9376ce4 QQQ-37 update streamed-etl steps to not have to use different record-list 2022-09-06 15:15:11 -05:00
31e6bf4d49 PRDONE-94: updates from code review feedback added .env test 2022-09-06 11:41:25 -05:00
12925127b2 Feedback from code reviews 2022-09-06 09:29:24 -05:00
9a8b49f1a7 Feedback from code reviews 2022-09-05 09:47:43 -05:00
4af7757fdd Merge commit '3a69ce7' into feature/sprint-10 2022-09-01 15:59:50 -05:00
91ba6f4b4e Merge branch 'feature/sprint-10' of github.com:Kingsrook/qqq into feature/sprint-10 2022-09-01 15:59:19 -05:00
3a69ce7d2f QQQ-40 getting closer to production-ready on automations 2022-09-01 15:57:01 -05:00
64e801747f QQQ-40 getting closer to production-ready on automations 2022-09-01 15:53:35 -05:00
6b4417d3e8 Merge branch 'feature/QQQ-38-app-home-widgets' into feature/sprint-10 2022-08-31 15:59:32 -05:00
e157809b35 Merge branch 'feature/QQQ-40-record-automations' into feature/sprint-10 2022-08-31 15:12:54 -05:00
a08ec0ae6f QQQ-40 Initial working POC 2022-08-31 15:01:45 -05:00
f08ffe691f PRDONE-94: updated to use interpreter for getting environment credentials, updated interpreter to load Dotenv files as environment overrides 2022-08-31 12:05:35 -05:00
69b9ed5b19 QQQ-37 adding pre & post run to ETL transform & load; minor QOL 2022-08-30 13:44:34 -05:00
4bf1fe8638 PRDONE-94: updated to look for dotenv properties and fall back to environment vars 2022-08-30 13:33:13 -05:00
dcea96579c PRDONE-94: updated to set dotenv variables as system properties 2022-08-30 12:28:40 -05:00
4c2ebf8a94 PRDONE-94: updated to ignore missing .env file 2022-08-30 12:17:37 -05:00
4316b47916 fixed incorrect import order 2022-08-30 12:13:29 -05:00
48b8d295e3 initial checkin of quicksight dashboard widget POC, updated to remove hard coded credentials 2022-08-30 11:46:46 -05:00
6142b8e703 Update tests to run w/ h2 instead of mysql 2022-08-29 14:29:01 -05:00
bc95899a50 Increase coverage 2022-08-29 14:20:00 -05:00
9106b82560 Fixing tests 2022-08-29 14:12:09 -05:00
39f065e23e Fixing tests 2022-08-29 13:52:54 -05:00
cb22f86793 Checkpoint - working versions of streamed with frontend processes, with validation 2022-08-29 13:33:35 -05:00
d32538bf45 Merge branch 'dev' into feature/QQQ-38-app-home-widgets 2022-08-25 10:14:26 -05:00
5f8f063b99 Merge branch 'dev' into feature/QQQ-37-streamed-processes 2022-08-25 10:09:23 -05:00
c3bba7cf8c Merge tag 'version-0.4.0' into dev
Tag release
2022-08-25 10:08:05 -05:00
103d229c82 Update for next development version 2022-08-25 10:01:52 -05:00
7355a9c8aa QQQ-37 Updating test coverage 2022-08-24 10:18:59 -05:00
c2972cd4df QQQ-37 checkpoint 2022-08-23 16:54:36 -05:00
834b4136de Merge branch 'feature/sprint-9-support-updates' into feature/QQQ-37-streamed-processes 2022-08-23 11:17:45 -05:00
459a533f60 upated to work in CI (e.g., w/o database) 2022-08-22 10:53:51 -05:00
937304e7f1 QQQ-38 Initial build of app home page widgets 2022-08-22 10:48:55 -05:00
21d66cc7fc QQQ-37 added test coverage for StreamedETLWithFrontendProcess 2022-08-22 10:08:27 -05:00
a86f42f373 QQQ-37 initial buidout of StreamedETLWithFrontendProcess 2022-08-22 08:36:09 -05:00
1036 changed files with 177361 additions and 4882 deletions

23
.circleci/adjust-pom-version.sh Executable file
View File

@ -0,0 +1,23 @@
#!/bin/bash
if [ -z "$CIRCLE_BRANCH" ] && [ -z "$CIRCLE_TAG" ]; then
echo "Error: env vars CIRCLE_BRANCH and CIRCLE_TAG were not set."
exit 1;
fi
if [ "$CIRCLE_BRANCH" == "dev" ] || [ "$CIRCLE_BRANCH" == "staging" ] || [ "$CIRCLE_BRANCH" == "main" ] || [ \! -z $(echo "$CIRCLE_TAG" | grep "^version-") ]; then
echo "On a primary branch or tag [${CIRCLE_BRANCH}${CIRCLE_TAG}] - will not edit the pom version.";
exit 0;
fi
if [ -n "$CIRCLE_BRANCH" ]; then
SLUG=$(echo $CIRCLE_BRANCH | sed 's/[^a-zA-Z0-9]/-/g')
else
SLUG=$(echo $CIRCLE_TAG | sed 's/^snapshot-//g')
fi
POM=$(dirname $0)/../pom.xml
echo "Updating $POM <revision> to: $SLUG-SNAPSHOT"
sed -i "s/<revision>.*/<revision>$SLUG-SNAPSHOT<\/revision>/" $POM
git diff $POM

View File

@ -1,7 +1,6 @@
version: 2.1
orbs:
slack: circleci/slack@4.10.1
localstack: localstack/platform@1.0
commands:
@ -12,18 +11,26 @@ commands:
steps:
- store_artifacts:
path: << parameters.module >>/target/site/jacoco/index.html
when: always
- store_artifacts:
path: << parameters.module >>/target/site/jacoco/jacoco-resources
when: always
install_java17:
steps:
- run:
name: Install Java 17
command: |
sudo add-apt-repository -y ppa:openjdk-r/ppa
sudo apt-get update
sudo apt install -y openjdk-17-jdk
sudo rm /etc/alternatives/java
sudo ln -s /usr/lib/jvm/java-17-openjdk-amd64/bin/java /etc/alternatives/java
- run:
## used by jacoco uncovered class reporting in pom.xml
name: Install html2text
command: |
sudo apt-get update
sudo apt-get install -y html2text
mvn_verify:
steps:
@ -45,10 +52,18 @@ commands:
module: qqq-backend-module-filesystem
- store_jacoco_site:
module: qqq-backend-module-rdbms
- store_jacoco_site:
module: qqq-backend-module-api
- store_jacoco_site:
module: qqq-middleware-api
- store_jacoco_site:
module: qqq-middleware-javalin
- store_jacoco_site:
module: qqq-middleware-picocli
- store_jacoco_site:
module: qqq-middleware-slack
- store_jacoco_site:
module: qqq-language-support-javascript
- store_jacoco_site:
module: qqq-sample-project
- run:
@ -67,6 +82,10 @@ commands:
mvn_jar_deploy:
steps:
- checkout
- run:
name: Adjust pom version
command: |
.circleci/adjust-pom-version.sh
- restore_cache:
keys:
- v1-dependencies-{{ checksum "pom.xml" }}
@ -83,27 +102,23 @@ jobs:
mvn_test:
executor: localstack/default
steps:
- localstack/startup
## - localstack/startup
- install_java17
- mvn_verify
- slack/notify:
event: fail
mvn_deploy:
executor: localstack/default
steps:
- localstack/startup
## - localstack/startup
- install_java17
- mvn_verify
- mvn_jar_deploy
- slack/notify:
event: always
workflows:
test_only:
jobs:
- mvn_test:
context: [ qqq-maven-registry-credentials, kingsrook-slack, build-qqq-sample-app ]
context: [ qqq-maven-registry-credentials, build-qqq-sample-app ]
filters:
branches:
ignore: /dev/
@ -113,7 +128,7 @@ workflows:
deploy:
jobs:
- mvn_deploy:
context: [ qqq-maven-registry-credentials, kingsrook-slack, build-qqq-sample-app ]
context: [ qqq-maven-registry-credentials, build-qqq-sample-app ]
filters:
branches:
only: /dev/

3
.gitignore vendored
View File

@ -2,6 +2,7 @@ target/
*.iml
.env
.idea
#############################################
## Original contents from github template: ##
@ -29,6 +30,8 @@ target/
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*
.DS_Store
*.swp
.flattened-pom.xml
dependency-reduced-pom.xml

View File

@ -10,7 +10,7 @@ The bundle contains all of the sub-jars. It is named:
```qqq-${version}.jar```
You can also use fine grained jars:
You can also use fine-grained jars:
- `qqq-backend-core`: The core module. Useful if you're developing other modules.
- `qqq-backend-module-rdbms`: Backend module for working with Relational Databases.
- `qqq-backend-module-filesystem`: Backend module for working with Filesystems (including AWS S3).

View File

@ -179,9 +179,7 @@
<property name="ignoreFinal" value="false"/>
<property name="allowedAbbreviationLength" value="1"/>
</module>
-->
<module name="OverloadMethodsDeclarationOrder"/>
<!--
<module name="VariableDeclarationUsageDistance"/>
<module name="CustomImportOrder">
<property name="sortImportsInGroupAlphabetically" value="true"/>
@ -261,5 +259,10 @@
<module name="MissingOverride"/>
</module>
<module name="Header">
<property name="headerFile" value="checkstyle/license.txt"/>
<property name="fileExtensions" value="java"/>
<property name="ignoreLines" value="3"/>
</module>
<module name="SuppressWarningsFilter"/>
</module>

20
checkstyle/license.txt Normal file
View File

@ -0,0 +1,20 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/

176
docs/Introduction.adoc Normal file
View File

@ -0,0 +1,176 @@
= Introduction
include::variables.adoc[]
QQQ is a Low-code Application Framework for Engineers.
Its purpose is to provide the basic structural elements of an application - things that every application needs - so that the engineers building an application on top of QQQ don't need to worry about those pieces, and can instead focus on the unique needs of the application that they are building.
== What makes QQQ special?
The scope of what QQQ provides is far-reaching, and most likely goes beyond what you may initially be thinking.
That is to say, QQQ includes code all the way from the backend of an application, through its middleware layer, and including its frontend.
For example, a common set of modules deployed in a QQQ application will provide:
* Backend RDBMS/Database connectivity and access.
* Frontend web UI (e.g., a React application)
* A java web server acting as middleware between the frontend web UI and the backend
That is to say - as an engineer deploying a QQQ application - you do not need to write a single line of code that is concerned with any of those things.
* You do not need to write code to connect to your database.
* You do not write any web UI code.
* You do not write any middleware code to tie together the frontend and backend.
Instead, QQQ includes *all* of these pieces.
QQQ knows how to connect to databases (and actually, several other kinds of backend systems - but ignore that for now).
Plus it knows how to do most of what an application needs to do with a database (single-record lookups, complex queries, joins, aggregates, bulk inserts, updates, deletes).
QQQ also knows how to present the data from a database - in table views, or single-record views, or exports, or reports or widgets.
And it knows how to present a powerful ad-hoc query interface to users, and how to show screens where users can create, update, and delete records.
It also provides the connective tissue (middleware) between those backend layers where data is stored, and frontend layers were users interact with data.
== What makes your application special?
I've said a lot about what does QQQ knows - but let's dig a little deeper.
What does QQQ know, and what does it not know?
Well - what it doesn't know is, it doesn't know the special or unique aspects of the application that you are building.
So, what makes your application unique?
Is your application unique because it needs to have screens where users can search for records in a table?
No!
QQQ assumes (as does the author of this document) that all applications (at least of the nature that QQQ supports) need what we call Query Screens.
So QQQ gives you a Query Screen for all of your tables - with zero code from you.
Is your application unique because you want users to be able to view, create, edit, and delete records from tables?
No!
QQQ assumes that all applications need these basic https://en.wikipedia.org/wiki/Create,_read,_update_and_delete[CRUD] capabilities.
So QQQ provides all of these UI screens - for view, create, edit, delete - along with the supporting middleware and backend code - all the way down to the SQL that selects & manages the data.
You get it all for free - zero code.
Is your application unique because you have a `fiz_bar` table, with 47 columns, and a `whoz_zat` table, with 42 columns of its own, that joins to `fiz_bar` on the `zizzy_ziz_id` field?
Yes!
OK - we found some of it - what makes your application unique is the data that you're working with.
Your tables - their fields - the connection info for your database.
QQQ doesn't know those details.
So - that's your first job as a QQQ application engineer - to describe your data to QQQ.
For the example above - you need to tell QQQ that you have a `fiz_bar` table, and that you have a `whoz_zat` table - and you need to tell QQQ what the fields or columns in thsoe tables are.
You can even tell QQQ how to join those tables (on that `zizzy_ziz_id` field).
And then that's it.
Once QQQ has this {link-table} meta data, it can then provide its Query screens, and full CRUD screens to your tables, in your database, with your fields.
And at the risk of repeating myself - you can do this (get a full Query & CRUD application) with zero lines of actual procedural code.
You only need to supply meta-data (which, at the time of this writing, is done in Java, but it's just creating objects - and in the future could be done in YAML files, for example).
== Beyond Basics
Going beyond the basic wiring as described above, QQQ also provides some of the more advanced elements needed in a modern data-driven web application, including:
* Authentication & Authorization
* ad-hoc Query engine for access to data tables
* Full CRUD (Create, Read, Update, Delete) capabilities
* Multistep custom workflows ("Processes" in QQQ parlance)
* Scheduled jobs
* Enterprise Service Bus
So what do we mean by all of this?
We said that basically every application needs, for example, Authentication & Authorization.
Login screens.
User & Role tables.
Permissions.
So, when it's time for you to build a new application for your _Big Tall Floor Lamp_ manufacturing business, do you need to start by writing a login screen?
And a Permissions scheme?
And throwing HTTP 401 errors?
And managing user-role relationships?
And then having a bug in the check permission logic on the _Light Bulb Inventory Edit_ screen, so Jim is always keying in bad quantities, even though he isn't supposed to have permission there?
No!
All of the (really important, even though application developers hate doing it) aspects of security - you don't need to write ANY code for dealing with that.
Just tell QQQ what Authentication provider you want to use (e.g., https://auth0.com/[Auth0]), and - to paraphrase the old https://www.youtube.com/watch?v=YHzM4avGrKI[iMac ad] - there's no step 2.
QQQ just does it.
''''
QQQ can provide this type of application using a variety and/or combination of backend data storage types.
And, whichever type of backend is used, QQQ gives a common interface (both user-facing and programmer-facing).
Backend types include:
* Relational Databases (RDBMS)
* File Systems
* Web APIs
* NoSQL/Document Databases (_Future_)
In addition, out-of-the-box, QQQ also goes beyond these basics, delivering:
* Bulk versions of all CRUD operations.
* Automatically generated JSON APIs.
* Auditing of data changes.
* End-user (e.g., non-engineer) customization via dynamic scripting capabilities
#todo say much more#
== QQQ Architecture
Like a house!
== Developing a QQQ Application
In developing an application with QQQ, engineers will generally have to define two types of code:
. *Meta Data* - This is the code that you use to tell QQQ the shape of your application - your unique Tables, Processes, Apps, Reports, Widgets, etc.
In general, this code is 100% declarative in nature (as opposed to procedural or functional).
That is to say - it has no logic.
It is just a definition of what exists - but it has no instructions, algorithms, or business logic.
* _In a future version of QQQ, we anticipate being able to define meta-data in a format such as YAML or JSON, or to even load it from a relational or document database.
This speaks to the fact that this "code" is not executable code - but rather is a simple declaration of (meta) data._
* A key function of QQQ then is to drive all of its layers of functionality - frontend UIs, middleware, and core backend actions (e.g., ORM operations) - based on this meta-data.
** For example:
... You define the meta-data for a table in your application - including its fields and their data types, as well as what backend system the table exists within.
... Then, the QQQ Frontend Material Dashboard UI's Query Screen loads that table's meta-data, and uses it to control the screen that is presented. Including:
**** The data grid shown on the screen will have columns for each field in the table.
**** The Filter button in the Query Screen will present a menu listing all fields from the table for the user to build ad-hoc queries against the table.
The data-types specified for the fields (in the meta-data) dictate what operators QQQ allows the user to use against fields (e.g., Strings offer "contains" vs Numbers offer "greater than").
**** Values for records from the table will be formatted for presentation based on the meta-data (such as a numeric field being shown with commas if it represents a quantity, or formatted as currency).
...
[start=2]
. *Meta Data* - declarative code - java object instances (potentially which could be read from `.yaml` files or other data sources in a future version of QQQ), which tell QQQ about the backend systems, tables, processes, reports, widgets, etc, that make up the application.
For example:
* Details about the database you are using, and how to connect to it.
* A database table's name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).
* The description of web API - its URL and authentication mechanism.
* A table/path within a web API, and the fields returned in the JSON at that endpoint.
* The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.
* Details about a chart that summarizes data from a table for presentation as a dashboard widget.
// the section below is kinda dumb. like, it says you have to write application code, but
// then it just talks about how your app code gets for-free the same shit that QQQ does.
// it should instead say more about what your custom app code is or does.
// 2. *Application code* - to customize beyond what the framework does out-of-the box, and to provide application-specific business-logic.
// QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model.
// For example:
// * The same record-security model that is enforced for ad-hoc user queries through the frontend is applied to custom application code.
// ** So if your table has a security key defined, which says that users can only see Order records that are associated with the user's assigned Store, then QQQ's order Query Screen will enforce that rule.
// ** And at the same time - any custom processes ran by a user will have the same security applied to any queries that they run against the Order table.
// ** And any custom dashboard widgets - will only include data that the user is allowed to see.
// * Record audits are performed in custom code the same as they are in framework-driven actions.
// ** So if a custom process edits a record, details of the changed fields show up in the record's audit, the same as if the record was edited using the standard QQQ edit action.
// * Changed records are sent through the ESB automatically regardless of whether they are updated by custom application code or standard framework code.
// ** Meaning record automations are triggered regardless of how a record is created or edited - without you, as an application engineering, needing to send records through the bus.
// * The multi-threaded, paged producer/consumer pattern used in standard framework actions is how all custom application actions are also invoked.
// ** For example, the standard QQQ Bulk Edit action uses the same streamed-ETL process that custom application processes can use.
// Meaning your custom processes can take full advantage of the same complex frontend, middleware, and backend structural pieces, and you can just focus on your unique busines logic needs.
2. *Application code* - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic.
QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model.
For example:
== Lifecycle?
* define meta data
** enrichment
** validation
* start application
* for dev - hotSwap

View File

@ -0,0 +1,192 @@
== QueryAction
include::../variables.adoc[]
The `*QueryAction*` is the basic action that is used to get records from a {link-table}, generally according to a <<QQueryFilter,Filter>>.
In SQL/RDBMS terms, it is analogous to a `SELECT` statement, where 0 or more records may be found and returned.
=== Examples
==== Basic Form
[source,java]
----
QueryInput input = new QueryInput();
input.setTableName("orders");
input.setFilter(new QQueryFilter()
.withCriteria(new QFilterCriteria("total", GREATER_THAN, new BigDecimal("3.50")))
.withOrderBy(new QFilterOrderBy("orderDate", false)));
QueryOutput output = new QueryAction.execute(input);
List<QRecord> records = output.getRecords();
----
=== Details
`QueryAction`, in general, can be called in two different modes:
. The most common use-case case, and default, fetches all records synchronously, does any post-processing (as requested in the <<QueryInput>>), and returns all records as a list in the <<QueryOutput>>).
. The alternative use-case is meant for larger operations, where one wouldn't want all records matching a query in-memory.
For this scenario, a `RecordPipe` object can be passed in to the <<QueryInput>>.
This causes `QueryAction` to run its post-processing action on records as they are placed into the pipe, and to potentially block (per the pipe's settings).
This method of usage needs to be done on a separate thread from another thread which would be consuming records from the pipe.
QQQ's `AsyncRecordPipeLoop` class provides an implementation of doing such a dual-threaded job.
If the {link-table} has a `POST_QUERY_CUSTOMIZER` defined, then after records are fetched from the backend, that code is executed on the records before they leave the `QueryAction` (either through its `QueryOutput` or `RecordPipe`).
=== QueryInput
* `table` - *String, Required* - Name of the table being queried against.
* `filter` - *<<QQueryFilter>> object* - Specification for what records should be returned, based on *<<QFilterCriteria>>* objects, and how they should be sorted, based on *<<QFilterOrderBy>>* objects.
If a `filter` is not given, then all rows in the table will be returned by the query.
* `skip` - *Integer* - Optional number of records to be skipped at the beginning of the result set.
e.g., for implementing pagination.
* `limit` - *Integer* - Optional maximum number of records to be returned by the query.
* `transaction` - *QBackendTransaction object* - Optional transaction object.
** Behavior for this object is backend-dependant.
In an RDBMS backend, this object is generally needed if you want your query to see data that may have been modified within the same transaction.
* `recordPipe` - *RecordPipe object* - Optional pipe object that records are placed into, for asynchronous processing.
** If a *recordPipe* is used, then records cannot be retrieved from the *QueryOutput*.
Rather, such records must be read from the pipe's `consumeAvailableRecords()` method.
** A *recordPipe* should only be used when a *QueryAction* is running in a separate Thread from the record's consumer.
* `shouldTranslatePossibleValues` - *boolean, default: false* - Controls whether any fields in the table with a *possibleValueSource* assigned to them should have those possible values looked up
(e.g., to provide text translations in the generated records' `displayValues` map).
** For example, if running a query to present results to a user, this would generally need to be *true*.
But if running a query to provide data as part of a process, then this can generally be left as *false*.
* `shouldGenerateDisplayValues` - *boolean, default: false* - Controls whether field level *displayFormats* should be used to populate the generated records' `displayValues` map.
** For example, if running a query to present results to a user, this would generally need to be *true*.
But if running a query to provide data as part of a process, then this can generally be left as *false*.
* `shouldFetchHeavyFields` - *boolean, default: true* - Controls whether or not fields marked as `isHeavy` should be fetched & returned or not.
* `shouldOmitHiddenFields` - *boolean, default: true* - Controls whether or not fields marked as `isHidden` should be included in the result or not.
* `shouldMaskPassword` - *boolean, default: true* - Controls whether or not fields with `type` = `PASSWORD` should be masked, or if their actual values should be returned.
* `queryJoins` - *List of <<QueryJoin>> objects* - Optional list of tables to be joined with the main table being queried.
See QueryJoin below for further details.
==== QQueryFilter
A key component of *<<QueryInput>>*, a *QQueryFilter* defines both what records should be included in a query's results (e.g., an SQL `WHERE`), as well as how those results should be sorted (SQL `ORDER BY`).
* `criteria` - *List of <<QFilterCriteria>>* - Individual conditions or clauses to filter records.
They are combined using the *booleanOperator* specified in the *QQueryFilter*. See below for further details.
* `orderBys` - *List of <<QFilterOrderBy>>* - List of fields (and directions) to control the sorting of query results.
In general, multiple *orderBys* can be given (depending on backend implementations).
* `booleanOperator` - *Enum of AND, OR, default: AND* - Specifies the logical joining operator used among individual criteria.
* `subFilters` - *List of QQueryFilter* - To build arbitrarily complex queries, with nested boolean logic, 0 or more *subFilters* may be provided.
** Each *subFilter* can include its own additional *subFilters*.
** Each *subFilter* can specify a different *booleanOperator*.
** For example, consider the following *QQueryFilter*, that uses two *subFilters*, and a mix of *booleanOperators*
[source,java]
----
queryInput.setFilter(new QQueryFilter()
.withBooleanOperator(OR)
.withSubFilters(List.of(
new QQueryFilter().withBooleanOperator(AND)
.withCriteria(new QFilterCriteria("firstName", EQUALS, "James"))
.withCriteria(new QFilterCriteria("lastName", EQUALS, "Maes")),
new QQueryFilter().withBooleanOperator(AND)
.withCriteria(new QFilterCriteria("firstName", EQUALS, "Darin"))
.withCriteria(new QFilterCriteria("lastName", EQUALS, "Kelkhoff"))
)));
// which would generate the following WHERE clause in an RDBMS backend:
// WHERE (first_name='James' AND last_name='Maes') OR (first_name='Darin' AND last_name='Kelkhoff')
----
===== QFilterCriteria
* `fieldName` - *String, required* - Reference to a field on the table being queried.
** Or, in the case of a query with *queryJoins*, a qualified name of a field from a join-table (where the qualifier would be the joined table's name or alias, followed by a dot)
*** For example: `orderLine.sku` or `orderBillToCustomer.firstName`
* `operator` - *Enum of QCriteriaOperator, required* - Comparison operation to be applied to the field specified as *fieldName* and the *values* or *otherFieldName*.
** e.g., `EQUALS`, `NOT_IN`, `GREATER_THAN`, `BETWEEN`, `IS_BLANK`, etc.
* `values` - *List of values, conditional* - Provides the value(s) that the field is compared against.
The number of values (0, 1, 2, or more) required are based on the *operator* being used.
If an *otherFieldName* is given, and the *operator* expects 1 value, then *values* is ignored, and *otherFieldName* is used.
* `otherFieldName` - *String, conditional* - Specifies that the *fieldName* should be compared against another field in the records, rather than the values in the *values* property.
Only used for *operators* that expect 1 value (e.g., `EQUALS` or `LESS_THAN_OR_EQUALS` - not `IS_NOT_BLANK` or `IN`).
[source,java]
.QFilterCriteria definition examples:
----
// in-line, via constructors that take (List<Serializable> values) or (Serializable... values) as 3rd arg
new QFilterCriteria("id", IN, 1, 2, 3)
new QFilterCriteria("name", IS_BLANK)
new QFilterCriteria("orderNo", IN, orderNoList)
new QFilterCriteria("state", EQUALS, "MO")
// long-form, with fluent setters
new QFilterCriteria()
.withFieldName("quantity")
.withOpeartor(QCriteriaOperator.GREATER_THAN)
.withValues(List.of(47));
// to use otherFieldName, long-form must be used
new QFilterCriteria()
.withFieldName("firstName")
.withOpeartor(QCriteriaOperator.EQUALS)
.withOtherFieldName("lastName");
// using otherFieldName to build a criterion that looks at two fields from two different join tables
new QFilterCriteria()
.withFieldName("billToCustomer.lastName")
.withOpeartor(QCriteriaOperator.NOT_EQUALS)
.withOtherFieldName("shipToCustomer.lastName");
----
===== QFilterOrderBy
* `fieldName` - *String, required* - Reference to a field on the table being queried.
** Or, in the case of a query with *queryJoins*, a qualified name of a field from a join-table (where the qualifier would be the joined table's name or alias, followed by a dot)
* `isAscending` - *boolean, default: true* - Specify if the sort is ascending or descending.
[source,java]
.QFilterOrderBy definition examples:
----
// short-form, via constructors
new QFilterOrderBy("id") // isAscending defaults to true.
new QFilterOrderBy("name", false)
// long-form, with fluent setters
new QFilterOrderBy()
.withFieldName("birthDate")
.withIsAscending(false);
----
==== QueryJoin
* `joinTable` - *String, required* - Name of the table that is being joined in to the existing query.
** Will be inferred from *joinMetaData*, if *joinTable* is not set when *joinMetaData* gets set.
* `baseTableOrAlias` - *String, required* - Name of a table (or an alias) already defined in the query, to which the *joinTable* will be joined.
** Will be inferred from *joinMetaData*, if *baseTableOrAlias* is not set when *joinMetaData* gets set (which will only use the leftTableName from the joinMetaData - never an alias).
* `joinMetaData` - *QJoinMetaData object* - Optional specification of a {link-join} in the current QInstance.
If not set, will be looked up at runtime based on *baseTableOrAlias* and *joinTable*.
** If set before *baseTableOrAlias* and *joinTable*, then they will be set based on the *leftTable* and *rightTable* in this object.
* `alias` - *String* - Optional (unless multiple instances of the same table are being joined together, when it becomes required).
Behavior based on SQL `FROM` clause aliases.
If given, must be used as the part before the dot in field name specifications throughout the rest of the query input.
* `select` - *boolean, default: false* - Specify whether fields from the *rightTable* should be selected by the query.
If *true*, then the `QRecord` objects returned by this query will have values with corresponding to the (table-or-alias `.` field-name) form.
* `type` - *Enum of INNER, LEFT, RIGHT, FULL, default: INNER* - specifies the SQL-style type of join being performed.
[source,java]
.QueryJoin definition examples:
----
// selecting from an "orderLine" table - then join to its corresponding "order" table
queryInput.withTableName("orderLine");
queryInput.withQueryJoin(new QueryJoin("order").withSelect(true));
...
queryOutput.getRecords().get(0).getValueBigDecimal("order.grandTotal");
// given an "order" table with 2 foreign keys to a customer table (billToCustomerId and shipToCustomerId)
// Note, we must supply the JoinMetaData to the QueryJoin, to drive what fields to join on in each case.
queryInput.withTableName("order");
queryInput.withQueryJoins(List.of(
new QueryJoin(instance.getJoin("orderJoinShipToCustomer")
.withAlias("shipToCustomer")
.withSelect(true)),
new QueryJoin(instance.getJoin("orderJoinBillToCustomer")
.withAlias("billToCustomer")
.withSelect(true))));
...
record.getValueString("billToCustomer.firstName")
+ " placed an order for "
+ record.getValueString("shipToCustomer.firstName")
----
=== QueryOutput
* `records` - *List of QRecord* - List of 0 or more records that match the query filter.
** _Note: If a *recordPipe* was supplied to the QueryInput, then calling `queryOutput.getRecords()` will result in an `IllegalStateException` being thrown - as the records were placed into the pipe as they were fetched, and cannot all be accessed as a single list._

View File

@ -0,0 +1,33 @@
== RenderTemplateAction
include::../variables.adoc[]
The `*RenderTemplateAction*` performs the job of taking a template - that is, a string of code, in a templating language, such as https://velocity.apache.org/engine/1.7/user-guide.html[Velocity], and merging it with a set of data (known as a context), to produce some using-facing output, such as a String of HTML.
=== Examples
==== Canonical Form
[source,java]
----
RenderTemplateInput input = new RenderTemplateInput(qInstance);
input.setSession(session);
input.setCode("Hello, ${name}");
input.setTemplateType(TemplateType.VELOCITY);
input.setContext(Map.of("name", "Darin"));
RenderTemplateOutput output = new RenderTemplateAction.execute(input);
String result = output.getResult();
assertEquals("Hello, Darin", result);
----
==== Convenient Form
[source,java]
----
String result = RenderTemplateAction.renderVelocity(input, Map.of("name", "Darin"), "Hello, ${name}");
assertEquals("Hello, Darin", result);
----
=== RenderTemplateInput
* `code` - *String, Required* - String of template code to be rendered, in the templating language specified by the `type` parameter.
* `type` - *Enum of VELOCITY, Required* - Specifies the language of the template code.
* `context` - *Map of String → Object* - Data to be made available to the template during rendering.
=== RenderTemplateOutput
* `result` - *String* - Result of rendering the input template and context.

File diff suppressed because it is too large Load Diff

6
docs/docinfo.html Normal file
View File

@ -0,0 +1,6 @@
<style>
.sect2 + .sect2
{
border-top: 1px solid #e7e7e9;
}
</style>

View File

@ -0,0 +1,406 @@
== TableSyncProcess
include::../variables.adoc[]
The `TableSyncProcess` is designed to help solve the common use-case where you have a set of records in one table,
you want to apply a transformation to them, and then you want them to result in records in a different table.
=== Sample Scenario
For example, you may be receiving an import file feed from a partner - say, a CSV file of order records
- that you need to synchronize into your local database's orders table.
=== High Level Steps
At a high-level, the steps of this task are:
1. Get the records from the partner's feed.
2. Decide Insert/Update - e.g., if you already have any of these orders in your database table, in which case, they need updated, versus ones
you don't have, which need inserted.
3. Map fields from the partner's order record to your own fields.
4. Store the necessary records in your database (insert or update).
=== What you need to tell QQQ
`TableSyncProcess`, as defined by QQQ, knows how to do everything for a job like this, other than the part that's unique to your business.
Those unique parts that you need to tell QQQ are:
* What are the source & destination tables?
* What are the criteria to identify source records to be processed?
* What are the key fields are that "link" together records from the source & destination tables?
** For the example above, imagine you have a "partnerOrderNo" field in your database's order table - then you'd need
to tell QQQ what field in the partner's table provides the value for that field in your table.
* How do fields / values from the source table map to fields & values in the destination table?
=== Detailed Steps
To get more specific, in a QQQ `TableSyncProcess`, here's what happens for each of the high-level steps given above:
* Getting records from the partner's feed (done by QQQ):
** Records from the source will be fetched via an Extract step, which runs a query to find the records that need processing.
** Depending on your use-case, you may use an `ExtractViaQueryStep` (maybe with a Table Automation) or `ExtractViaBasepullQueryStep`
(e.g., if you are polling a remote data source).
* Deciding Insert/Update (done by QQQ):
** Given a set of records from the source table (e.g., output of the Extract step mentioned above), get values from the "key" field in that table.
** Do a lookup in the destination table, where its corresponding "key" field has the values extracted from the source records.
** For each source record, if its "key" was found in the destination table, then plan an update to that existing corresponding
destination record; else, plan to insert a new record in the destination table.
* Mapping values (done by your custom Application code):
** Specifically, mapping is done in a subclass of `AbstractTableSyncTransformStep`, in the `populateRecordToStore` method.
Of particular interest are these two parameters to that method:
*** `QRecord destinationRecord` - as determined by the insert/update logic above, this will either be a
new empty record (e.g., for inserting), or a fully populated record from the destination table (for updating).
*** `QRecord sourceRecord` - this is the record being processed, from the source table.
** This method is responsible for setting values in the `destinationRecord`, and returning that record
(unless it has decided that for some reason the record should _not_ be stored, in which case it may return `null`).
* Storing the records (done by QQQ):
** This is typically done with the `LoadViaInsertOrUpdateStep`, though it is customizable if additional work is needed (e.g., via a
subclass of `LoadViaInsertOrUpdateStep`, or a more custom subclass of `AbstractLoadStep`).
=== Bare-bones Example
For this example, let's assume we're setting up a partner-order-feed as described above, with the following details:
* We have records from a partner in a table named `"partnerOrderImport"` (let's assume the records may have been created
using the QQQ `FilesystemImporter` process).
These records have the following fields:
** `orderNo, date, city, state, postal, whseNo`
* We need to synchronize those records with table in our database named `"order"`, with the following fields corresponding to those from the partner:
** `partnerOrderNo, orderDate, shipToCity, shipToState, shipToZipCode, warehouseId`
* The same conceptual order may appear in the `"partnerOrderImport"` multiple times, e.g., if they update some data on the order and re-transmit it to us.
Meaning, we need to update our `"order"` records when we receive a new version an existing order.
To use `*TableSyncProcess*` for solving this use-case, we'll need to create 2 things:
1. A `QProcessMetaData` object, which we can create using the builder object provided by class `TableSyncProcess`.
Note that this type of process is specialization of the standard QQQ `StreamedETLWithFrontendProcess`, as described elsewhere in this documentation.
2. A subclass of `AbstractTableSyncTransformStep`, where we implement our mapping logic.
Again, note that `AbstractTableSyncTransformStep` is a subclass of `AbstractTransformStep`, as used by `StreamedETLWithFrontendProcess`.
And to be good programmers, we'll actually create a 3rd thing:
[start=3]
. A unit test for our Transform step.
Here are examples of these pieces of code:
[source,java]
.Example of building process using the TableSyncProcess builder:
----
// the false argument below tells the build we are not a basepull-style process
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)
// give our process a unique name within our QInstance
.withName("partnerOrderToLocalOrderProcess")
// tell the process to what class to use for transforming records from source to destination
.withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)
.getProcessMetaData();
----
[source,java]
.Example implementation of an AbstractTableSyncTransformStep
----
public class PartnerOrderToOrderTransformStep extends AbstractTableSyncTransformStep
{
@Override
protected SyncProcessConfig getSyncProcessConfig()
{
return (new SyncProcessConfig(
"partnerOrderImport", // source tableName
"orderNo", // source table key fieldName
"order", // destination tableName
"partnerOrderNo" // destination table foreign key fieldName
));
}
@Override
public QRecord populateRecordToStore(RunBackendStepInput runBackendStepInput, QRecord destinationRecord, QRecord sourceRecord) throws QException
{
// map simple values from source table to destination table
destinationRecord.setValue("orderDate", sourceRecord.get("date"));
destinationRecord.setValue("shipToAddressCity", sourceRecord.get("city"));
destinationRecord.setValue("shipToAddressState", sourceRecord.get("state"));
destinationRecord.setValue("shipToAddressZipCode", sourceRecord.get("postal"));
return (destinationRecord);
}
}
----
[source,java]
.Example Unit Test for a transform step
----
@Test
void testTransformStep()
{
// insert 1 test order, that will be updated by the transform step
Integer existingId = new InsertAction().execute(new InsertInput("order").withRecords(List.of(
new QRecord().withValue("partnerOrderNo", 101).withValue("shipToState", "IL")
))).getRecords().get(0).getValueInteger("id");
// set up input for the step - a list of 2 of the partner's orders
RunBackendStepInput input = new RunBackendStepInput();
input.setRecords(List.of(
new QRecord().withValue("orderNo", 101).withValue("state", "NY"), // will update the order above
new QRecord().withValue("orderNo", 102).withValue("state", "CA") // will insert a new order
));
RunBackendStepOutput output = new RunBackendStepOutput();
// run the code under test - our transform step
new PartnerOrderToOrderTransformStep().run(input, output);
// Note that by just running the transform step, no records have been stored.
// We can assert against the output of this step.
assertEquals(existingId, output.getRecords().get(0).getValue("id"));
assertEquals(101, output.getRecords().get(0).getValue("partnerOrderNo"));
assertEquals("NY", output.getRecords().get(0).getValue("shipToState"));
assertNull(output.getRecords().get(1).getValue("id"));
assertEquals(102, output.getRecords().get(1).getValue("partnerOrderNo"));
assertEquals("CA", output.getRecords().get(1).getValue("shipToState"));
}
@Test
void testFullProcess()
{
// todo! :)
}
----
=== Pseudocode process flow
Now that we've seen the bare-bones example, let's see an even more detailed breakdown of how a full `TableSyncProcess` works,
by looking at its 3 "ETL" steps in pseudocode:
==== ExtractStep (Producer Thread)
* Queries source table for records
** Often based on Table Automations (e.g., for all newly inserted records) or Basepull pattern (polling for new/updated records).
==== TransformStep (Consumer Thread)
* Receives pages of records from `ExtractStep` in the `run` method.
* Makes `sourceKeyList` by getting `sourceTableKeyField` values from the records.
* Calls `initializeRecordLookupHelper(runBackendStepInput, sourceRecordList)`
** Calls `getLookupsToPreLoad` to control which lookups are performed.
* Calls `getExistingRecordsByForeignKey(runBackendStepInput, destinationTableForeignKeyField, destinationTableName, sourceKeyList);`
** Calls `getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList)` as part of querying the `destinationTable`
** Returns the output of `buildExistingRecordsMap(destinationTableForeignKeyField, queryOutput.getRecords())`
* foreach input record (from `sourceTable`):
** Calls `getExistingRecord(existingRecordsByForeignKey, destinationForeignKeyField, sourceKeyValue)`
** if an existing record was returned (and if the syncConfig says `performUpdates`), this record is set as `recordToStore`
** else if no existing record was returned (and if the syncConfig says `performInserts`), a new record is set as `recordToStore`
** else continue the foreach.
** call `populateRecordToStore(runBackendStepInput, recordToStore, sourceRecord)`
** if a record is returned it is added to the process step output (to be stored in the LoadStep)
==== LoadStep (Consumer Thread)
* Receives records from the output of the `TransformStep`.
* Inserts and/or Updates `destinationTable`, with records returned by `populateRecordToStore`
=== Additional Process Configuration Examples
The following examples show how to use additional settings in the `TableSyncProcess` builder.
==== UI
While a `TableSyncProcess` will often run via a schedule and/or automation, we may also want to allow users
to manually run it in a UI.
[source,java]
.Making our process available for a UI
----
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(true)
.withName("partnerOrderToLocalOrderProcess")
.withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)
// attach our process to its source table, to show up in UI
.withTableName("partnerOrderImport")
// add some fields to display on the review screen, in UI
.withReviewStepRecordFields(List.of(
new QFieldMetaData("clientId", QFieldType.STRING).withLabel("Client"),
new QFieldMetaData("warehouseId", QFieldType.STRING).withLabel("Warehouse"),
new QFieldMetaData("partnerOrderNo", QFieldType.STRING)))
.getProcessMetaData();
----
==== Basepull
The previous example would work as a Table Automation (e.g., where the list of records identified in the
Extract step were determined by the Automation system).
However, a second common pattern is to use `Basepull` (e.g., if polling for updated records from a partner API endpoint).
[source,java]
.Configuring our process as a Basepull
----
// the true argument below tells the build we ARE a basepull-style process
// this changes the default extract-step.
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)
.withName("partnerOrderToLocalOrderProcess")
.withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)
// See Basepull documentation for details
.withBasepullConfiguration(new BasepullConfiguration())
// schedule our process to run automatically every minute
.withSchedule(new QScheduleMetaData().withRepeatSeconds(60))
.getProcessMetaData();
----
=== Additional Options in the Transform Step
==== Specifying to not perform Inserts or not perform Updates
We may have a scenario where we want our sync process to never update records if the key is already found in the destination table.
We can configure this with an additional optional parameter to the `SyncProcessConfig` constructor:
[source,java]
.Specifying to not do updates in a TableSyncProcess
----
@Override
protected SyncProcessConfig getSyncProcessConfig()
{
return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
true, // performInserts
false // performUpdates
));
}
----
Similarly, we may want to disallow inserts from a particular sync process.
The `performInserts` argument to the `SyncProcessConfig` constructor lets us do that:
[source,java]
.Specifying to not do inserts in a TableSyncProcess
----
@Override
protected SyncProcessConfig getSyncProcessConfig()
{
return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
false, // performInserts
true // performUpdates
));
}
----
==== Customizing the query for existing records
In some cases, a specific Table Sync process may need to refine the query filter that is used
to lookup existing records in the destination table (e.g. for determining insert vs. update).
For example, in our orders-from-a-partner scenario, if we have more than 1 partner sending us orders,
where there could be overlapping orderNo values among them - we may have an additional field in our
orders table to identify which partner an order came from.
So then when we're looking up orders by `partnerOrderNo`, we would need to also include the `partnerId` field
in our query, so that we only update orders from the specific partner that we're dealing with.
To do this (to customize the existing record query filter), we need can just override the method `getExistingRecordQueryFilter`.
Generally we would start by calling the `super` version of the method, and then add to it additional criteria.
[source,java]
.Customizing the query filter used to look for existing records
----
/*******************************************************************************
** Define the query filter to find existing records. e.g., for determining
** insert vs. update. Subclasses may override this to customize the behavior,
** e.g., in case an additional field is needed in the query.
*******************************************************************************/
protected QQueryFilter getExistingRecordQueryFilter(RunBackendStepInput runBackendStepInput, List<Serializable> sourceKeyList)
{
QQueryFilter filter = super.getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList);
filter.addCriteria(new QFilterCriteria("partnerId", EQUALS, PARTNER_ID));
return (filter);
}
----
==== More efficient additional record lookups
It is a common use-case to need to map various ids from a partner's system to ids in your own system.
For the orders example, we might need to know what warehouse the order is shipping from.
The customer may send their identifier for the warehouse, and we may need to map those identifiers to our own warehouse ids.
The QQQ-provided class `RecordLookupHelper` exists to help with performing lookups like this,
and in particular, it can be used to execute one query to fetch a full table, storing records
by a key field, then returning those records without performing additional queries.
`AbstractTableSyncTransformStep` has a protected `recordLookupHelper` member.
If we override the method `getLookupsToPreLoad()`, then this object is
populated by calling its `preloadRecords` method with each specified pair of tableNames and fieldNames.
[source,java]
.Specifying tables to pre-load using a RecordLookupHelper
----
/*******************************************************************************
** Specify a list of tableName/keyColumnName pairs to run through
** the preloadRecords method of the recordLookupHelper.
*******************************************************************************/
@Override
protected List<Pair<String, String>> getLookupsToPreLoad()
{
return (List.of(
Pair.of("warehouse", "partnerWarehouseNo")
));
}
----
If we have preloaded some lookups, we can then use them in our `populateRecordToStore` method as follows:
[source,java]
.Using the recordLookupHelper in populateRecordToStore
----
// lookup warehouse with partnerWarehouseNo=whseNo from partner, and use our id in destination record
String partnerWarehouseNo = sourceRecord.getValue("whseNo");
Integer warehouseId = recordLookupHelper.getRecordId("warehouse", "partnerWarehouseNo", whseNo, Integer.class);
destinationRecord.setValue("warehouseId", warehouseId);
----
==== Additional override points
There are more methods which can be overridden in your `AbstractTableSyncTransformStep` subclass,
to provide further customizations of behaviors, specifically in the area of dealing with existing
records (e.g., the insert/update use-case).
[source,java]
.Additional AbstractTableSyncTransformStep overrides
----
/*******************************************************************************
** Run the existingRecordQueryFilter - to look in the destinationTable for
** any records that may need an update (rather than an insert).
**
** Generally returns a Map, keyed by a Pair of the destinationTableForeignKeyField
** and the value in that field. But, for more complex use-cases, one can override
** the buildExistingRecordsMap method, to make different keys (e.g., if there are
** two possible destinationTableForeignKeyFields).
*******************************************************************************/
protected Map<Pair<String, Serializable>, QRecord> getExistingRecordsByForeignKey
(
RunBackendStepInput runBackendStepInput,
String destinationTableForeignKeyField,
String destinationTableName,
List<Serializable> sourceKeyList
) throws QException;
/*******************************************************************************
** Overridable point where you can, for example, keys in the existingRecordsMap
** with different fieldNames from the destinationTable.
**
** Note, if you're overriding this method, you'll likely also want & need to
** override getExistingRecord.
*******************************************************************************/
protected Map<Pair<String, Serializable>, QRecord> buildExistingRecordsMap
(
String destinationTableForeignKeyField,
List<QRecord> existingRecordList
);
/*******************************************************************************
** Given the map of existingRecordsByForeignKey (as built by
** getExistingRecordsByForeignKey which calls buildExistingRecordsMap),
** get one record from that map, for a given key-value from a source record.
**
** The destinationForeignKeyField is given as advice if needed (e.g., to see its type)
*******************************************************************************/
protected QRecord getExistingRecord
(
Map<Pair<String, Serializable>, QRecord> existingRecordsByForeignKey,
QFieldMetaData destinationForeignKeyField,
Serializable sourceKeyValue
);
----

63
docs/index.adoc Normal file
View File

@ -0,0 +1,63 @@
= QQQ
:doctype: article
:toc: left
:toclevels: 2
:source-highlighter: coderay
include::Introduction.adoc[leveloffset=+1]
== Meta Data
// Organizational units
include::metaData/QInstance.adoc[leveloffset=+1]
include::metaData/Backends.adoc[leveloffset=+1]
include::metaData/Apps.adoc[leveloffset=+1]
// Primary meta-data types
include::metaData/Tables.adoc[leveloffset=+1]
include::metaData/Processes.adoc[leveloffset=+1]
include::metaData/Widgets.adoc[leveloffset=+1]
// Helper meta-data types
include::metaData/Fields.adoc[leveloffset=+1]
include::metaData/PossibleValueSources.adoc[leveloffset=+1]
include::metaData/Joins.adoc[leveloffset=+1]
include::metaData/SecurtiyKeyTypes.adoc[leveloffset=+1]
include::metaData/Reports.adoc[leveloffset=+1]
include::metaData/Icons.adoc[leveloffset=+1]
include::metaData/PermissionRules.adoc[leveloffset=+1]
== Custom Application Code
include::misc/QContext.adoc[leveloffset=+1]
include::misc/QRecords.adoc[leveloffset=+1]
include::misc/QRecordEntities.adoc[leveloffset=+1]
include::misc/ProcessBackendSteps.adoc[leveloffset=+1]
=== Table Customizers
#todo#
== QQQ Core Actions
include::actions/QueryAction.adoc[leveloffset=+1]
include::actions/GetAction.adoc[leveloffset=+1]
=== CountAction
#todo#
=== AggregateAction
#todo#
include::actions/InsertAction.adoc[leveloffset=+1]
=== UpdateAction
#todo#
=== DeleteAction
#todo#
=== AuditAction
#todo#
== QQQ Default Implementations
include::implementations/TableSync.adoc[leveloffset=+1]
// later... include::actions/RenderTemplateAction.adoc[leveloffset=+1]

92
docs/metaData/Apps.adoc Normal file
View File

@ -0,0 +1,92 @@
[#Apps]
== Apps
include::../variables.adoc[]
QQQ User Interfaces (e.g., Material Dashboard) generally organize their contents via *Apps*.
Apps are a lightweight construct in QQQ - basically just containers for other objects.
Specifically, Apps can contain:
* {link-widgets}
* {link-tables}
* {link-process}
* {link-reports}
* Other {link-apps} - to create a multi-tiered navigational hierarchy.
=== QAppMetaData
Apps are defined in a QQQ Instance in `*QAppMetaData*` objects. Apps must consist of either 1 or more {link-widgets}, or 1 or more *Sections*, which are expected to contain 1 or more {link-tables}, {link-processes}, or {link-reports}.
*QAppMetaData Properties:*
* `name` - *String, Required* - Unique name for the app within the QQQ Instance.
* `label` - *String* - User-facing label for the app, presented in User Interfaces.
Inferred from `name` if not set.
* `permissionRules` - *QPermissionRules* - Permissions to apply to the app. See {link-permissionRules} for details.
* `children` - *List of QAppChildMetaData* - Objects contained within the app. These can be {link-tables}, {link-processes}, {link-reports} or other {link-apps}.
** See the example below for some common patterns for how these child-meta data objects are added to an App.
* `parentAppName` - *String* - For an app which is a child of another app, the parent app's name is referenced in this field.
** Note that this is generally automatically set when the child is added to its parent, in the `addChild` method.
* `icon` - *QIcon* - An icon to display in a UI for the app. See {link-icons}.
* `widgets` - *List of String* - A list of names of {link-widgets} to include in the app.
* `sections` - *List of <<QAppSection>>* - A list of `QAppSection` objects, to create organizational subdivisions within the app.
** As shown in the example below, the method `withSectionOfChildren` can be used to fluently add a new `QAppSection`, along with its child objects, to both an app and a section all at once.
==== QAppSection
A `QAppSection` is an organizational subsection of a {link-app}.
* `name` - *String, Required* - Unique name for the section within its app.
* `label` - *String* - User-facing label for the section, presented in User Interfaces.
Inferred from `name` if not set.
* `icon` - *QIcon* - An icon to display in a UI for the section. See {link-icons}.
* `tables` - *List of String* - A list of names of {link-tables} to include in the section.
* `processes` - *List of String* - A list of names of {link-processes} to include in the section.
* `reports` - *List of String* - A list of names of {link-reports} to include in the section.
*Examples*
[source,java]
----
/*******************************************************************************
** Full example of constructing a QAppMetaData object.
*******************************************************************************/
public class ExampleAppMetaDataProducer extends MetaDataProducer<QAppMetaData>
{
/*******************************************************************************
** Produce the QAppMetaData
*******************************************************************************/
@Override
public QAppMetaData produce(QInstance qInstance) throws QException
{
return (new QAppMetaData()
.withName("sample")
.withLabel("My Sample App")
.withIcon(new QIcon().withName("thumb_up"))
.withWidgets(List.of(
UserWelcomeWidget.NAME,
SystemHealthLineChartWidget.NAME))
.withSectionOfChildren(new QAppSection().withName("peoplePlacesAndThings"),
qInstance.getTable(People.TABLE_NAME),
qInstance.getTable(Places.TABLE_NAME),
qInstance.getTable(Things.TABLE_NAME),
qInstance.getProcess(AssociatePeopleWithPlacesProcess.NAME))
.withSectionOfChildren(new QAppSection().withName("math").withLabel("Mathematics"),
qInstance.getProcess(ComputePiProcess.NAME),
qInstance.getReport(PrimeNumbersReport.NAME),
qInstance.getReport(PolygonReport.NAME)));
}
/*******************************************************************************
** Since this meta-data producer expects to find other meta-data objects in the
** QInstance, give it a sortOrder higher than the default (which we'll expect
** the other objects used).
*******************************************************************************/
@Override
public int getSortOrder()
{
return (Integer.MAX_VALUE);
}
}
----

150
docs/metaData/Backends.adoc Normal file
View File

@ -0,0 +1,150 @@
[#Backends]
== Backends
include::../variables.adoc[]
A key component of QQQ is its ability to connect to various backend data stores, while providing the same interfaces to those backends - both User Interfaces, and Programming Interfaces.
For example, out-of-the-box, QQQ can connect to:
* <<RDBMSBackendMetaData,RDBMS>> (Relational Database Management Systems, such as MySQL)
* File Systems (<<S3BackendMetaData,Amazon S3>> or <<FilesystemBackendMetaData,local disk>>)
* <<APIBackendMetaData,JSON Web APIs>> (_using a custom mapping class per-API backend_).
* In-Memory data stores
All {link-tables} in a QQQ instance must belong to a backend. As such, any instance using tables (which would be almost all instances) must define 1 or more backends.
=== QBackendMetaData
Backends are defined in a QQQ Instance in a `*QBackendMetaData*` object.
These objects will have some common attributes, but many different attributes based on the type of backend being used.
*QBackendMetaData Properties:*
* `name` - *String, Required* - Unique name for the backend within the QQQ Instance.
* `backendType` - *String, Required* - Identifier for the backend type being defined.
** This attribute is typically set in the constructor of a `QBackendMetaData` subclass, and as such does not need to be set when defining a backend meta data object.
* `enabledCapabilities` and `disabledCapability` - *Sets*, containing *Capability* enum values.
Basic rules that apply to all tables in the backend, describing what actions (such as Delete, or Count) are supported in the backend.
** By default, QQQ assumes that a backend supports _most_ capabilities, with one exception being `QUERY_STATS`.
** #TODO# fully explain rules here
* `usesVariants` - *Boolean, default false* - Control whether or not the backend supports the concept of "Variants".
** Supporting variants means that tables within the backend can exist in alternative "variants" of the backend.
For example, this might mean a sharded or multi-tenant database backend (perhaps a different server or database name per-client).
Or this might mean using more than one set of credentials for connecting to an API backend - each of those credential sets would be a "variant".
** A backend that uses variants requires additional properties to be set. #TODO complete variant documentation#
In a QQQ application, one will typically not create an instance of `QBackendMetaData` directly, but instead will create an instance of one of its subclasses, specific to the type of backend being used.
The currently available list of such classes are:
==== RDBMSBackendMetaData
The meta data required for working with tables in an RDBMS (relational database) backend are defined in an instance of the `*RDBMSBackendMetaData*` class.
*RDBMSBackendMetaData Properties:*
* `vendor` - *String, Required* - Database vendor. Currently supported values are: `aurora`, `mysql`, `h2`.
* `jdbcUrl` - *String, Optional* - Full JDBC URL for connecting to the database.
** If this property is provided, then following properties (which are the components of a JDBC URL) are ignored.
In other words, you can either provide the `jdbcUrl`, or the individual components that make it up.
* `hostName` - *String, Conditionally Required* - Hostname or ip address of the RDBMS server.
* `port` - *Integer, Conditionally Required* - Port used to connect to the RDBMS server.
* `databaseName` - *String, Conditionally Required* - Name of the database being connected to, within the RDBMS server.
* `username` - *String, Conditionally Required* - Username for authenticating in the database server.
* `password` - *String, Conditionally Required* - Password for authenticating in the database server.
*Examples*
[source,java]
----
/*******************************************************************************
** Full example of constructing an RDBMSBackendMetaData
*******************************************************************************/
public class ExampleDatabaseBackendMetaDataProducer extends MetaDataProducer<QBackendMetaData>
{
public static final String NAME = "rdbmsBackend";
/*******************************************************************************
** Produce the QBackendMetaData
*******************************************************************************/
@Override
public QBackendMetaData produce(QInstance qInstance)
{
///////////////////////////////////////////////////////////////////////
// read settings from either a .env file or the system's environment //
///////////////////////////////////////////////////////////////////////
QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter();
String vendor = interpreter.interpret("${env.RDBMS_VENDOR}");
String hostname = interpreter.interpret("${env.RDBMS_HOSTNAME}");
String port = interpreter.interpret("${env.RDBMS_PORT}");
String databaseName = interpreter.interpret("${env.RDBMS_DATABASE_NAME}");
String username = interpreter.interpret("${env.RDBMS_USERNAME}");
String password = interpreter.interpret("${env.RDBMS_PASSWORD}");
return (new RDBMSBackendMetaData()
.withName(NAME)
.withVendor(vendor)
.withHostName(hostname)
.withPort(ValueUtils.getValueAsInteger(port))
.withDatabaseName(databaseName)
.withUsername(username)
.withPassword(password)
.withCapability(Capability.QUERY_STATS));
}
}
----
==== S3BackendMetaData
The meta data required for working with tables in an Amazon S3 backend are defined in an instance of the `*S3BackendMetaData*` class.
*S3BackendMetaData Properties:*
* `bucketName` - *String, Required* - Bucket name to connect to inside AWS S3.
* `accessKey` - *String, Required* - Access key for connecting to S3 inside AWS S3.
* `secretKey` - *String, Required* - Secret key for connecting to S3 inside AWS S3.
* `region` - *String, Required* - AWS region containing the Bucket in S3.
* `basePath` - *String, Required* - Base path to the files within the S3 bucket.
==== FilesystemBackendMetaData
The meta data required for working with tables in a (local) filesystem backend are defined in an instance of the `*FilesystemBackendMetaData*` class.
*FilesystemBackendMetaData Properties:*
* `basePath` - *String, Required* - Base path to the backend's files.
==== APIBackendMetaData
The meta data required for working with tables in a web API are defined in an instance of the `*APIBackendMetaData*` class.
QQQ provides a minimal, reasonable default implementation for working with web APIs, making assumptions such as using `POST` to insert records, and `GET` with a primary key in the URL to get a single record.
However, in our experience, almost all APIs are implemented differently enough, that a layer of custom code is required.
For example, query/search endpoints are almost always unique in how they take their search parameters, and how they wrap their output.
To deal with this, a QQQ API Backend can define a custom class in the `actionUtil` property of the backend meta-data, as a subclass of `BaseAPIActionUtil`, which ideally can override methods at the level where unique functionality is needed.
For example, an application need not define the full method for executing a Query against an API backend (which would need to make the HTTP request (actually multiple, to deal with pagination)).
Instead, one can just override the `buildQueryStringForGet` method, where the unique details of making the request are defined, and maybe the `jsonObjectToRecord` method, where records are mapped from the API's response to a QRecord.
#todo - full reference and examples for `BaseAPIActionUtil`#
*APIBackendMetaData Properties:*
* `baseUrl` - *String, Required* - Base URL for connecting to the API.
* `contentType` - *String, Required* - value of `Content-type` header included in all requests to the API.
* `actionUtil` - *QCodeReference, Required* - Reference to a class that extends `BaseAPIActionUtil`, where custom code for working with this API backend is defined.
* `customValues` - *Map of String → Serializable* - Application-defined additional name=value pairs that can
* `authorizationType` - *Enum, Required* - Specifies how authentication is provided for the API.
The value here, dictates which other authentication-related properties are required.
Possible values are:
** `API_KEY_HEADER` - Uses the `apiKey` property in an HTTP header named `API-Key`.
_In the future, when needed, QQQ will add a property, tentatively named `apiKeyHeaderName`, to allow customization of this header name._
** `API_TOKEN` - Uses the `apiKey` property in an `Authroization` header, as: `"Token " + apiKey`
** `BASIC_AUTH_API_KEY` - Uses the `apiKey` property, Base64-encoded, in an `Authroization` header, as `"Basic " + base64(apiKey)`
** `BASIC_AUTH_USERNAME_PASSWORD` - Combines the `username` and `password` properties, Base64-encoded, in an `Authroization` header, as `"Basic " + base64(username + ":" + password)`
** `OAUTH2` - Implements an OAuth2 client credentials token grant flow, using the properties `clientId` and `clientSecret`.
By default, the id & secret are sent as query-string parameters to the API's `oauth/token` endpoint.
Alternatively, if the meta-data has a `customValue` of `setCredentialsInHeader=true`, then the id & secret are posted in an `Authorization` header (base-64 encoded, and concatenated with `":"`).
** `API_KEY_QUERY_PARAM` - Uses the `apiKey` property as a query-string parameter, with its name taken from the `apiKeyQueryParamName` property.
** `CUSTOM` - Has a no-op implementation at the QQQ layer.
Assumes that an override of `protected void handleCustomAuthorization(HttpRequestBase request)` be implemented in the backend's `actionUtil` class.
This would be
* `apiKey` - *String, Conditional* - See `authorizationType` above for details.
* `clientId` - *String, Conditional* - See `authorizationType` above for details.
* `clientSecret` - *String, Conditional* - See `authorizationType` above for details.
* `username` - *String, Conditional* - See `authorizationType` above for details.
* `password` - *String, Conditional* - See `authorizationType` above for details.
* `apiKeyQueryParamName` - *String, Conditional* - See `authorizationType` above for details.

25
docs/metaData/Fields.adoc Normal file
View File

@ -0,0 +1,25 @@
[#Fields]
== Fields
include::../variables.adoc[]
QQQ Fields define
=== QFieldMetaData
*QFieldMetaData Properties:*
* `name` - *String, Required* - Unique name for the field within its container (table, process, etc).
* `label` - *String* - User-facing label for the field, presented in User Interfaces.
* `type` - *enum of QFieldType, Required* - Data type for values in the field.
* `backendName` - *String* - Name of the field within its backend.
** For example, in an RDBMS-backed table, a field's `name` may be written in camel case, but its `backendName` written with underscores.
* `isRequired` - *boolean, default false* - Indicator that a value is required in this field.
* `isEditable` - *boolean, default true* - Indicator that users may edit values in this field.
* `displayFormat` - *String, default `%s`* - Java Format Specifier string, used to format values in the field for display in user interfaces.
Used to set values in the `displayValues` map within a `QRecord`.
** Recommended values for `displayFormat` come from the `DisplayFormat` interface, such as `DisplayFormat.CURRENCY`, `DisplayFormat.COMMAS`, or `DisplayFormat.DECIMAL2_COMMAS`.
* `defaultValue` - Value to use for the field if no other value is given. Type is based on the field's `type`.
* `possibleValueSourceName` - *String* - Reference to a {link-pvs} to be used for this field.
Values in this field should correspond to ids from the referenced Possible Value Source.
* `maxLength` - *Integer* - Maximum length (number of characters) allowed for values in this field.
Only applicable for fields with `type=STRING`.
* `

17
docs/metaData/Joins.adoc Normal file
View File

@ -0,0 +1,17 @@
[#Joins]
== Joins
include::../variables.adoc[]
#TODO#
=== QJoinMetaData
Joins are defined in a QQQ Instance in a `*QJoinMetaData*` object.
#TODO#
*QJoinMetaData Properties:*
* `name` - *String, Required* - Unique name for the join within the QQQ Instance. #todo infererences or conventions?#
#TODO#

View File

@ -0,0 +1,17 @@
[#PossibleValueSources]
== Possible Value Sources
include::../variables.adoc[]
#TODO#
=== QPossibleValueSource
A Possible Value Source is defined in a QQQ Instance in a `*QPossibleValueSource*` object.
#TODO#
*QPossibleValueSource Properties:*
* `name` - *String, Required* - Unique name for the possible value source within the QQQ Instance.
#TODO#

View File

@ -0,0 +1,216 @@
[#Processes]
== Processes
include::../variables.adoc[]
Besides {link-tables}, the other most common type of object in a QQQ Instance is the Process.
Processes are "custom" actions (e.g., defined by the application developers, rather than QQQ) that users and/or the system can execute against records.
Processes generally are made up of two types of sub-objects:
* *Screens* - i.e., User Interfaces (e.g., for gathering input and/or showing output).
* *BackendSteps* - Java classes (of type `BackendStep`) that execute the logic of the process.
=== QProcessMetaData
Processes are defined in a QQQ Instance in a `*QProcessMetaData*` object.
In addition to directly building a `QProcessMetaData` object setting its properties directly, there are a few common process patterns that provide *Builder* objects for ease-of-use.
See StreamedETLWithFrontendProcess below for a common example
*QProcessMetaData Properties:*
* `name` - *String, Required* - Unique name for the process within the QQQ Instance.
* `label` - *String* - User-facing label for the process, presented in User Interfaces.
Inferred from `name` if not set.
* `icon` - *QIcon* - Icon associated with this process in certain user interfaces.
See {link-icons}.
* `tableName` - *String* - Name of a {link-table} that the process is associated with in User Interfaces (e.g., Action menu).
* `isHidden` - *Boolean, default false* - Option to hide the process from all User Interfaces.
* `basepullConfiguration` - *<<BasepullConfiguration>>* - config for the common "Basepull" pattern, of identifying records with a timestamp greater than the last time the process was ran.
See below for details.
* `permissionRules` - *QPermissionRules object* - define the permission/access rules for the process.
See {link-permissionRules} for details.
* `steps` and `stepList` - *Map of String → <<QStepMetaData>>* and *List of QStepMetaData* - Defines the <<QFrontendStepMetaData,screens>> and <<QBackendStepMetaData,backend code>> that makes up the process.
** `stepList` is the list of steps in the order that they will by default be executed.
** `steps` is a map, including all steps from `stepList`, but which may also include steps which can used by the process if its backend steps make the decision to do so, at run-time.
** A process's steps are normally defined in one of two was:
*** 1) by a single call to `.withStepList(List<QStepMetaData>)`, which internally adds each step into the `steps` map.
*** 2) by multiple calls to `.addStep(QStepMetaData)`, which adds a step to both the `stepList` and `steps` map.
** If a process also needs optional steps, they should be added by a call to `.addOptionalStep(QStepMetaData)`, which only places them in the `steps` map.
* `schedule` - *<<QScheduleMetaData>>* - set up the process to run automatically on the specified schedule.
See below for details.
* `minInputRecords` - *Integer* - #not used...#
* `maxInputRecords` - *Integer* - #not used...#
#todo: supplementalMetaData (API)#
==== QStepMetaData
This is the base class for the two types of steps in a process - <<QFrontendStepMetaData,screens>> and <<QBackendStepMetaData,backend code>>.
There are some shared attributes of both of them, defined here.
*QStepMetaData Properties:*
* `name` - *String, Required* - Unique name for the step within the process.
* `label` - *String* - User-facing label for the step, presented in User Interfaces.
Inferred from `name` if not set.
* `stepType` - *String* - _Deprecated._
==== QFrontendStepMetaData
For processes with a user-interface, they must define one or more "screens" in the form of `QFrontendStepMetaData` objects.
*QFrontendStepMetaData Properties:*
* `components` - *List of <<QFrontendComponentMetaData>>* - a list of components to be rendered on the screen.
* `formFields` - *List of String* - list of field names used by the screen as form-inputs.
* `viewFields` - *List of String* - list of field names used by the screen as visible outputs.
* `recordListFields` - *List of String* - list of field names used by the screen in a record listing.
==== QFrontendComponentMetaData
A screen in a process may consist of multiple "components" - e.g., help text, and a form, and a list of records.
Each of these components are defined in a `QFrontendComponentMetaData`.
*QFrontendComponentMetaData Properties:*
* `type` - *enum, Required* - The type of component to display.
Each component type works with different keys in the `values` map.
Possible values for `type` are:
** `EDIT_FORM` - Displays a list of fields for editing, similar to a record edit screen.
Requires that `formFields` be populated in the step.
** `VIEW_FORM` - Displays a list of fields for viewing (not editing), similar to a record view screen.
Requires that `viewFields` be populated in the step.
** `HELP_TEXT` - Block of help text to be display on the screen.
Requires an entry in the component's `values` map named `"text"`.
** `HTML` - Block of custom HTML, generated by the process backend.
Expects a process value named `html`.
** `DOWNLOAD_FORM` - Presentation of a link to download a file generated by the process.
Expects process values named `downloadFileName` and `serverFilePath`.
** `GOOGLE_DRIVE_SELECT_FOLDER` - Special form that presents a UI from Google Drive, where the user can select a folder (e.g., as a target for uploading files in a subsequent backend step).
** `BULK_EDIT_FORM` - For use by the standard QQQ Bulk Edit process.
** `VALIDATION_REVIEW_SCREEN` - For use by the QQQ Streamed ETL With Frontend process family of processes.
Displays a component prompting the user to run full validation or to skip it, or, if full validation has been ran, then showing the results of that validation.
** `PROCESS_SUMMARY_RESULTS` - For use by the QQQ Streamed ETL With Frontend process family of processes.
Displays the summary results of running the process.
** `RECORD_LIST` - _Deprecated.
Showed a grid with a list of records as populated by the process._
* `values` - *Map of String → Serializable* - Key=value pairs, with different expectations based on the component's `type`.
See above for details.
==== QBackendStepMetaData
Process Backend Steps are where custom (at this time, Java, but in the future, potentially, from any supported language) code is executed, to provide the logic of the process.
QQQ comes with several common backend steps, such as for extracting lists of records, storing lists of records, etc.
*QBackendStepMetaData Properties:*
* `code` - *QCodeReference, Required* - Reference to the code to be executed for the step.
The referenced code must implement the `BackendStep` interface.
* `inputMetaData` - *QFunctionInputMetaData* - Definition of the data that the backend step expects and/or requires.
Sub-properties are:
** `fieldList` - *List of {link-fields}* - Optional list of fields used by the process step.
In general, a process does not _have to_ specify the fields that its steps use.
It can be used, however, for example, to cause a `defaultValue` to be applied to a field if it isn't specified in the process's input.
It can also be used to cause the process to throw an error, if a field is marked as `isRequired`, but a value is not present.
** `recordListMetaData` - *RecordListMetaData object* - _Not used at this time._
==== BasepullConfiguration
A "Basepull" process is a common pattern where an application needs to perform some action on all new (or updated) records from a particular data source.
To implement this pattern, one needs to store the timestamp of when the action runs, then query the source for records where a date-time field is after that timestamp.
QQQ helps facilitate this pattern by automatically retrieving and updating that timestamp field, and by building a default query filter based on that timestamp.
This is done by adding a `BasepullConfiguration` object to a process's meta-data.
*BasepullConfiguration Properties:*
* `tableName` - *String, Required* - Name of a {link-table} in the QQQ Instance where the basepull timestamps are stored.
* `keyField` - *String, Required* - Name of a {link-field} in the basepull table that stores a unique identifier for the process.
* `keyValue` - *String* - Optional value to be stored in the `keyField` of the basepull table as the unique identifier for the process.
If not set, then the process's `name` is used.
* `lastRunTimeFieldName` - *String, Required* - Name of a {link-field} in the basepull table that stores the last-run time for the process.
* `hoursBackForInitialTimestamp` - *Integer* - Optional number of hours to go back in time (from `now`) for the first time that the process is executed (i.e., if there is no timestamp stored in the basepull table).
* `timestampField` - *String, Required* - Name of a {link-field} in the table being queried against the last-run timestamp.
==== QScheduleMetaData
QQQ can automatically run processes using an internal scheduler, if they are configured with a `QScheduleMetaData` object.
*QScheduleMetaData Properties*
* `repeatSeconds` - *Integer, Conditional* - How often the process should be executed, in seconds.
* `repeatMillis` - *Integer, Conditional* - How often the process should be executed, in milliseconds.
Mutually exclusive with `repeatSeconds`.
* `initialDelaySeconds` - *Integer, Conditional* - How long between when the scheduler starts and the process should first run, in seconds.
* `initialDelayMillis` - *Integer, Conditional* - How long between when the scheduler starts and the process should first run, in milliseconds.
Mutually exclusive with `initialDelaySeconds`.
* `variantRunStrategory` - *enum, Conditional* - For processes than run against {link-tables} that use a {link-backend} with Variants enabled, this property controls if the various instances of the process should run in `PARALLEL` or in `SERIAL`.
* `variantBackend` - *enum, Conditional* - For processes than run against {link-tables} that use a {link-backend} with Variants enabled, this property specifies the name of the {link-backend}.
==== StreamedETLWithFrontendProcess
A common pattern for QQQ processes to exhibit is called the "Streamed ETL With Frontend" process pattern.
This pattern is to do an "Extract, Transform, Load" job on a potentially large set of records.
The records are Streamed through the process's steps, meaning, QQQ runs multiple threads - a producer, which is selecting records, and a consumer, which is processing records.
As such, in general, an unlimited number of records can be processed by a process, without worrying about exhausting server resources (e.g., OutOfMemory).
These processes also have a standard user-interface for displaying a summary of what the process will do (and has done), with a small number of records as a preview.
The goal of the summary is to give the user the big-picture of what the process will do (e.g., X records will be inserted; Y records will be updated), along with a small view of some details on the records that will be stored (e.g., on record A field B will be set to C).
This type of process uses 3 backend steps, and 2 frontend steps, as follows:
* *preview* (backend) - does just a little work (limited # of rows), to give the user a preview of what the final result will be - e.g., some data to seed the review screen.
* *review* (frontend) - a review screen, which after the preview step does not have a full process summary, but can generally tell the user how many records are input to the process, and can show a preview of a small number of the records.
* *validate* (backend) - optionally (per input on review screen), does like the preview step, but does it for all records from the extract step.
* *review* (frontend) - a second view of the review screen, if the validate step was executed.
Now that the full validation was performed, a full process summary can be shown, along with a some preview records.
* *execute* (backend) - processes all the rows - does all the work - stores data in the backend.
* *result* (frontend) - a result screen, showing a "past-tense" version of the process summary.
These backend steps are defined within QQQ, meaning they themselves do not execute any application-defined custom code.
Instead, these steps use the following secondary <<QBackendStepMetaData,backend steps>>:
* *Extract* - Fetch the rows to be processed.
Used by preview (but only for a limited number of rows), validate (without limit), and execute (without limit).
* *Transform* - Do whatever transformation is needed to the rows.
Done on preview, validate, and execute.
Always works with a page of records at a time.
Since it is called on the preview & validate steps, it should *NOT* ever store any data (unless it does a specific check to confirm that it is being used on an *execute* step).
* *Load* - Store the records into the backend, as appropriate.
Only called by the execute step.
Always works with a page of records at a time.
The meta-data for a `StreamedETLWithFrontendProcess` uses several input fields on its steps.
As such, it can be somewhat clumsy and error-prone to fully define a `StreamedETLWithFrontendProcess`.
To improve this programmer-interface, an inner `Builder` class exists within `StreamedETLWithFrontendProcess` (generated by a call to `StreamedETLWithFrontendProcess.processMetaDataBuilder()`).
*StreamedETLWithFrontendProcess.Builder methods:*
* `withName(String name) - Set the name for the process.
* `withLabel(String label) - Set the label for the process.
* `withIcon(QIcon icon)` - Set an {link-icon} to be display with the process in the UI.
* `withExtractStepClass(Class<? extends AbstractExtractStep>)` - Define the Extract step for the process.
If no special extraction logic is needed, `ExtractViaQuery.class` is often a reasonable default.
In other cases, `ExtractViaQuery` can be a reasonable class to extend for a custom extract step.
* `withTransformStepClass(Class<? extends AbstractTransformStep>)` - Define the Transform step for the process.
If no transformation logic is needed, `NoopTransformStep.class` can be used (though this is not very common).
* `withLoadStepClass(Class<? extends AbstractLoadStep>)` - Define the Load step for the process.
Several standard implementations exist, such as: `LoadViaInsertStep.class`, `LoadViaUpdateStep.class`, and `LoadViaDeleteStep.class`.
* `withTableName(String tableName)` - Specify the name of the {link-table} that the process should be associated with in the UI.
* `withSourceTable(String sourceTable)` - Specify the name of the {link-table} to be used as the source of records for the process.
* `withDestinationTable(String destinationTable)` - Specify the name of the {link-table} to be used as the destination for records from the process.
* `withSupportsFullValidation(Boolean supportsFullValidation)` - By default, all StreamedETLWithFrontendProcesses do allow the user to choose to run the full validation step.
However, in case cases it may not make sense to do so - so this method can be used to turn off that option.
* `withDoFullValidation(Boolean doFullValidation)` - By default, all StreamedETLWithFrontendProcesses will prompt the user if they want to run the full validation step or not.
However, in case cases you may want to enforce that the validation step always be executed.
Calling this method will remove the option from the user, and always run a full validation.
* `withTransactionLevelAutoCommit()`, `withTransactionLevelPage()`, and `withTransactionLevelProcess()` - Change the transaction-level used by the process.
By default, these processes are ran with a single transaction for all pages of their execute step.
But for some cases, doing page-level transactions can reduce long-transactions and locking within the system.
* `withPreviewMessage(String previewMessage)` - Sets the message shown on the validation review screen(s) above the preview records.
* `withReviewStepRecordFields(List<QFieldMetaData> fieldList)` -
* `withFields(List<QFieldMetaData> fieldList)` - Adds additional input fields to the preview step of the process.
* `withBasepullConfiguration(BasepullConfiguration basepullConfiguration)` - Add a <<BasepullConfiguration>> to the process.
* `withSchedule(QScheduleMetaData schedule)` - Add a <<QScheduleMetaData>> to the process.

174
docs/metaData/Reports.adoc Normal file
View File

@ -0,0 +1,174 @@
[#Reports]
== Reports
include::../variables.adoc[]
QQQ can generate reports based on {link-tables} defined within a QQQ Instance.
Users can run reports, providing input values.
Alternatively, application code can run reports as needed, supplying input values.
=== QReportMetaData
Reports are defined in a QQQ Instance with a `*QReportMetaData*` object.
Reports are defined in terms of their sources of data (`QReportDataSource`), and their view(s) of that data (`QReportView`).
*QReportMetaData Properties:*
* `name` - *String, Required* - Unique name for the report within the QQQ Instance.
* `label` - *String* - User-facing label for the report, presented in User Interfaces.
Inferred from `name` if not set.
* `processName` - *String* - Name of a {link-process} used to run the report in a User Interface.
* `inputFields` - *List of {link-field}* - Optional list of fields used as input to the report.
** The values in these fields can be used via the syntax `${input.NAME}`, where `NAME` is the `name` attribute of the `inputField`.
** For example:
[source,java]
----
// given this inputField:
new QFieldMetaData("storeId", QFieldType.INTEGER)
// its run-time value can be accessed, e.g., in a query filter under a data source:
new QFilterCriteria("storeId", QCriteriaOperator.EQUALS, List.of("${input.storeId}"))
// or in a report view's title or field formulas:
.withTitleFields(List.of("${input.storeId}"))
new QReportField().withName("storeId").withFormula("${input.storeId}")
----
* `dataSources` - *List of QReportDataSource, Required* - Definitions of the sources of data for the report.
At least one is required.
==== QReportDataSource
Data sources for QQQ Reports can either reference {link-tables} within the QQQ Instance, or they can provide custom code in the form of a `CodeReference` to a `Supplier`, for use cases such as a static data tab in an Excel report.
*QReportDataSource Properties:*
* `name` - *String, Required* - Unique name for the data source within its containing Report.
* `sourceTable` - *String, Conditional* - Reference to a {link-table} in the QQQ Instance, which the data source queries data from.
* `queryFilter` - *QQueryFilter* - If a `sourceTable` is defined, then the filter specified here is used to filter and sort the records queried from that table when generating the report.
* `staticDataSupplier` - *QCodeReference, Conditional* - Reference to custom code which can be used to supply the data for the data source, as an alternative to querying a `sourceTable`.
** Must be a `JAVA` code type
** Must be a `REPORT_STATIC_DATA_SUPPLIER` code usage.
** The referenced class must implement the interface: `Supplier<List<List<Serializable>>>`.
==== QReportView
Report Views control how the source data for a report is organized and presented to the user in the output report file.
If a DataSource describes the rows for a report (e.g., what table provides what records), then a View may be thought of as describing the columns in the report.
A single report can have multiple views, specifically, for the use-case where an Excel file is being generated, in which case each View creates a tab or sheet within the `xlsx` file.
*QReportView Properties:*
* `name` - *String, Required* - Unique name for the view within its containing Report.
* `label` - *String* - Used as a sheet (tab) label in Excel formatted reports.
* `type` - *enum of TABLE, SUMMARY, PIVOT. Required* - Defines the type of view being defined.
** *TABLE* views are a simple listing of the records from the data source.
** *SUMMARY* views are essentially pre-computed Pivot Tables.
That is to say, the aggregation done by a Pivot Table in a spreadsheet file is done by QQQ while generating the report.
In this way, a non-spreadsheet report (e.g., PDF or CSV) can have summarized data, as though it were a Pivot Table in a live spreadsheet.
** *PIVOT* views produce actual Pivot Tables, and are only supported in Excel files _(and are not supported at the time of this writing)_.
* `dataSourceName` - *String, Required* - Reference to a DataSource within the report, that is used to provide the rows for the view.
* `varianceDataSourceName` - *String* - Optional reference to a second DataSource within the report, that is used in `*SUMMARY*` type views for computing variances.
** For example, given a Data Source with a filter that selects all sales records for a given year, a Variance Data Source may have a filter that selects the previous year, for doing comparissons.
* `pivotFields` - *List of String, Conditional* - For *SUMMARY* or *PIVOT* type views, specify the field(s) used as pivot rows.
** For example, in a summary view of orders, you may "pivot" on the *customerId* field, to produce one row per-customer, with aggregate data for that customer.
* `titleFormat` - *String* - Java Format String, used with `titleFields` (if given), to produce a title row, e.g., first row in the view (before any rows from the data source).
* `titleFields` - *List of String, Conditional* - Used with `titleFormat`, to provide values for any format specifiers in the format string.
Syntax to reference a field (e.g., from a report input field) is: `${input.NAME}`, where `NAME` is the `name` attribute of the inputField.
** Example of using `titleFormat` and `titleFields`:
[source,java]
----
// given these inputFields:
new QFieldMetaData("startDate", QFieldType.DATE)
new QFieldMetaData("endDate", QFieldType.DATE)
// a view can have a title row like this:
.withTitleFormat("Weekly Sales Report - %s - %s")
.withTitleFields(List.of("${input.startDate}", "${input.endDate}"))
----
* `includeHeaderRow` - *boolean, default true* - Indication that first row of the view should be the column labels.
** If true, then header row is put in the view.
** If false, then no header row is put in the view.
* `includeTotalRow` - *boolean, default false* - Indication that a totals row should be added to the view.
All numeric columns are summed to produce values in the totals row.
** If true, then totals row is put in the view.
** If false, then no totals row is put in the view.
* `includePivotSubTotals` - *boolean, default false* - For a *SUMMARY* or *PIVOT* type view, if there are more than 1 *pivotFields* being used, this field is an indication that each higher-level pivot should include sub-totals.
** #TODO - provide example#
* `columns` - *List of QReportField, required* - Definition of the columns to appear in the view. See section on QReportField for details.
* `orderByFields` - *List of QFilterOrderBy, optional* - For a *SUMMARY* or *PIVOT* type view, how to sort the rows.
* `recordTransformStep` - *QCodeReference, subclass of `AbstractTransformStep`* - Custom code reference that can be used to transform records after they are queried from the data source, and before they are placed into the view.
Can be used to transform or customize values, or to look up additional values to add to the report.
** #TODO - provide example#
* `viewCustomizer` - *QCodeReference, implementation of interface `Function<QReportView, QReportView>`* - Custom code reference that can be used to customize the report view, at runtime.
Can be used, for example, to dynamically define the report's *columns*.
** #TODO - provide example#
===== QReportField
Report Fields define the fields (AKA columns) of data that appear in a report view.
These fields can either be direct references to fields from the report's data sources, or values computed using formula defined in the QReportField.
*QReportField Properties:*
* `name` - *String, required* - Unique identifier for the field within its ReportView.
In general, will be a reference to a field from the ReportView's DataSource *unless a *formula* is given (for *SUMMARY* type views), the field is marked as *isVirtual*, or the field is marked as *showPossibleValueLabel*).
* `label` - *String* - Optional text label to identify the field, for example, in a header row.
If not given, may be derived from field, where possible.
* `type` - *QFieldType*
* `formula` - *String, conditional* - Required for *SUMMARY* type views.
Defines the formula to be used for computing the value in this field.
** For example:
[source,java]
----
.withName("reportEndDate").withFormula("${input.endDate}")
.withName("count").withFormula("${pivot.count.id}")
.withName("percentOfTotal").withFormula("=DIVIDE(${pivot.count.id},${total.count.id})")
.withName("sumCost").withFormula("${pivot.sum.cost}")
.withName("sumCharge").withFormula("${pivot.sum.charge}")
.withName("profit").withFormula("=MINUS(${pivot.sum.charge},${pivot.sum.cost})")
.withName("totalCost").withFormula("=DIVIDE_SCALE(${pivot.sum.cost},${pivot.count.id},2)")
.withName("revenuePer").withFormula("=DIVIDE_SCALE(${pivot.sum.charge},${pivot.count.id},2)")
.withName("marginPer").withFormula("=MINUS(DIVIDE_SCALE(${pivot.sum.charge},${pivot.count.id},2),DIVIDE_SCALE(${pivot.sum.cost},${pivot.count.id},2))")
.withName("thisWeekMargin").withFormula("=SCALE(DIVIDE(${thisRow.profit},${pivot.sum.charge}),2)")
.withName("previousWeekProfit").withFormula("=MINUS(${variancePivot.sum.charge},${variancePivot.sum.cost})")
.withName("previousWeekMargin").withFormula("=SCALE(DIVIDE(${thisRow.previousWeekProfit},${variancePivot.sum.charge}),2)")
.withName("marginThisVsPrevious").withFormula("=SCALE(MINUS(${thisRow.margin},${thisRow.marginPrevious}),3)")
.withName("exception").withFormula("""
=IF(LT(${thisRow.margin},0),Negative Margin,IF(LT(${thisRow.marginThisVsPrevious},0),Margin Decreased,""))""")
----
* `displayFormat` *String*
* `isVirtual` *Boolean, default false* - (needs reviewed - may only be required for report views using a data source with a *staticDataSupplier*)
* `showPossibleValueLabel` *Boolean, default false* - To show a translated value for a Possible Value field (e.g., a name or other value meaningful to a user, instead of a foreign key).
* `sourceFieldName` *String* - Used for the scenario where a possibleValue field is included in a report both as the foreign key (raw, id value), and the translated "label" value.
In that case, the field marked with *showPossibleValueLabel* = true should be given a different name, and should use *sourceFieldName* to indicate the field that has the id value.
** For example:
[source,java]
----
// this field would have the "raw" warehouseId values
// e.g., integers - foreign keys to a warehouse table. Generally useful for machines to know.
new QReportField("warehouseId")
.withLabel("Warehouse Id"),
// this field would have the translated values from the warehouse PossibleValueSource
// for example, maybe the name field from the warehouse table. A string, useful for humans to read.
new QReportField("warehouseName")
.withSourceFieldName("warehouseId")
.withShowPossibleValueLabel(true)
.withLabel("Warehouse Name"),
----

View File

@ -0,0 +1,17 @@
[#SecurityKeyTypes]
== Security Key Types
include::../variables.adoc[]
#TODO#
=== QSecurityKeyType
A Security Key Type is defined in a QQQ Instance in a `*QSecurityKeyType*` object.
#TODO#
*QSecurityKeyType Properties:*
* `name` - *String, Required* - Unique name for the security key type within the QQQ Instance.
#TODO#

246
docs/metaData/Tables.adoc Normal file
View File

@ -0,0 +1,246 @@
[#Tables]
== Tables
include::../variables.adoc[]
One of the most common types of object in a QQQ Instance is the Table.
In the most common use-case, a QQQ Table may be the in-app representation of a Database table.
That is, it is a collection of records (or rows) of data, each of which has a set of fields (or columns).
QQQ also allows other types of data sources ({link-backends}) to be used as tables, such as File systems, API's, etc.
All of these backend types present the same interfaces (both user-interfaces, and application programming interfaces), regardless of their backend type.
=== QTableMetaData
Tables are defined in a QQQ Instance in `*QTableMetaData*` objects.
All tables must reference a {link-backend}, a list of fields that define the shape of records in the table, and additional data to describe how to work with the table within its backend.
*QTableMetaData Properties:*
* `name` - *String, Required* - Unique name for the table within the QQQ Instance.
* `label` - *String* - User-facing label for the table, presented in User Interfaces.
Inferred from `name` if not set.
* `backendName` - *String, Required* - Name of a {link-backend} in which this table's data is managed.
* `fields` - *Map of String → {link-field}, Required* - The columns of data that make up all records in this table.
* `primaryKeyField` - *String, Conditional* - Name of a {link-field} that serves as the primary key (unique identifier) for records in this table.
** Whether a primary key field is required or not depends on the backend type that the table belongs to.
* `uniqueKeys` - *List of UniqueKey* - Definition of additional unique keys or constraints (from an RDBMS point of view) from the table.
e.g., sets of columns which must have unique values for each record in the table.
The properties of the `UniqueKey` object are:
** `fieldNames` - *List of String, Required* - List of field names from this table.
** `label` - *String* - Optional label to be shown to users with error messages (e.g., for violation of this unique key).
* `backendDetails` - *QTableBackendDetails or subclass* - Additional data to configure the table within its {link-backend}.
** For example, for an RDBMS-type backend, the name of the table within the database.
** vs. a FileSystem backend, this may be the sub-path where files for the table are stored.
** #todo - details on these#
* `automationDetails` - *<<QTableAutomationDetails>>* - Configuration of automated jobs that run against records in the table, e.g., upon insert or update.
* `customizers` - *Map of String → QCodeReference* - References to custom code that are injected into standard table actions, that allow applications to customize certain parts of how the table works.
** Allowed values for keys in this map come from the `role` property of the `TableCustomizers` enum.
** Based on the key in this map, the `QCodeReference` used as the value must be of the appropriate java type, as specified in the `expectedType` property of the `TableCustomizers` enum value corresponding to the key.
** Example:
[source,java]
----
// in defining a QTableMetaData, a customizer can be added as:
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(MyPreInsCustomizer.class))
// where MyPreInsCustomizer would be defined as:
public class MyPreInsCustomizer extends AbstractPreInsertCustomizer
----
* `isHidden` - *Boolean, default false* - Option to hide the table from all User Interfaces.
* `parentAppName` - *String* - Name of a {link-app} that this table exists within.
** This field generally does not need to be set on the table when it is defined, but rather, is set when the table gets placed within an app.
* `icon` - *QIcon* - Icon associated with this table in certain user interfaces. See {link-icons}.
* `recordLabelFormat` - *String* - Java Format String, used with `recordLabelFields` to produce a label shown for records from the table.
* `recordLabelFields` - *List of String, Conditional* - Used with `recordLabelFormat` to provide values for any format specifiers in the format string.
These strings must be field names within the table.
** Example of using `recordLabelFormat` and `recordLabelFields`:
[source,java]
----
// given these fields in the table:
new QFieldMetaData("name", QFieldType.STRING)
new QFieldMetaData("birthDate", QFieldType.DATE)
// We can produce a record label such as "Darin Kelkhoff (1980-05-31)" via:
.withRecordLabelFormat("%s (%s)")
.withRecordLabelFields(List.of("name", "birthDate"))
----
* `sections` - *List of <<QFieldSection>>* - Mechanism to organize fields within user interfaces, into logical sections.
If any sections are present in the table meta data, then all fields in the table must be listed in exactly 1 section.
If no sections are defined, then instance enrichment will define default sections.
* `associatedScripts` - *List of <<AssociatedScript>>* - Definition of user-defined scripts that can be associated with records within the table.
* `enabledCapabilities` and `disabledCapabilities` - *Set of Capability enum values* - Overrides from the backend level, for capabilities that this table does or does not possess.
* `associations` - *List of <<Association>>* - tables whose records can be managed along with records from this table. See below for details.
* `recordSecurityLocks` - *List of <<RecordSecurityLock>>* - locks that apply to records in the table - e.g., to control what users can or cannot access records in the table.
See RecordSecurityLock below for details.
* `permissionRules` - *QPermissionRules object* - define the permission/access rules for the table.
See {link-permissionRules} for details.
* `auditRules` - *<<QAuditRules>> object* - define the audit rules for the table.
See QAuditRules below for details.
* `cacheOf` - *<<CacheOf>> object* - specify that this table serves as a "cache of" another table.
See CacheOf object below for details.
* `exposedJoins` - *List of <<ExposedJoin>> objects* - optional list of joined tables that are to be exposed in User Interfaces.
See ExposedJoin object below for details.
#todo: supplementalMetaData (API)#
==== QFieldSection
When users view records from a QQQ Table in a UI, fields are organized on the screen based on the `QFieldSection` objects in the table's meta-data.
*QFieldSection Properties:*
* `name` - *String, Required* - unique identifier for the section within its table.
* `label` - *String* - User-facing label for the section, presented in User Interfaces.
Inferred from `name` if not set.
* `tier` - *enum* - importance of the fields in section for the table.
Different tiers may be presented differently in UI's.
Only a single `T1` section is allowed per-table. Possible values are: `T1`, `T2`, and `T3`.
* `icon` - *QIcon* - Icon associated with this section in certain user interfaces. See {link-icons}.
* `isHidden` - *Boolean, default false* - Option to hide the table from all User Interfaces.
* `gridColumns` - *Integer* - Option to specify how many columns in a grid layout the section should use.
For the Material-Dashboard frontend, this is a grid of 12.
* `fieldNames` - *List of String, Conditional* - List of names of {link-fields} from this table to be included in this section.
* `widgetName` - *String, Conditional* - Name of a {link-widget} to be displayed in this section.
** Note that exactly one of `fieldNames` or `widgetName` must be used.
==== QTableAutomationDetails
Records in QQQ can have application-defined custom actions automatically asynchronously executed against them after they are inserted or updated.
The configuration to enable this functionality is assigned to a table in a `QTableAutomationDetails` object.
*QTableAutomationDetails Properties:*
* `statusTracking` - *AutomationStatusTracking object, Required* - define how QQQ should keep track, per record, its status (e.g., pending-insert-automations, running-update-automations, etc).
Properties of `AutomationStatusTracking` object are:
** `type` - *enum, Required* - what type of tracking is used for the table.
Possible values are:
*** `FIELD_IN_TABLE` - specifies that the table has a field which stores an `AutomationStatus` id.
*** _Additional types may be defined in the future, such as ONE_TO_ONE_TABLE or SHARED_TABLE._
** `fieldName` - *String, Conditional* - for `type=FIELD_IN_TABLE`, this property specifies the name of the {link-field} in the table that stores the `AutomationStatus` id.
* `providerName` - *String, Required* - name of an Automation Provider within the QQQ Instance, which is responsible for running the automations on this table.
* `overrideBatchSize` - *Integer* - optional control over how many records from the table are processed in a single batch/page.
For tables with "slow" actions (e.g., one that may need to make an API call per-record), using a smaller batch size (say, 50) may be required to avoid timeout errors.
* `actions` - *List of TableAutomationAction* - list of the actions to perform on new and updated records in the table.
Properties are:
** `name` - *String, Required* - unique identifier for the action within its table.
** `triggerEvent` - *enum, Required* - indicate which event type (`POST_INSERT`, `POST_UPDATE`, or `PRE_DELETE` (which is not yet implemented)) the action applies to.
** `priority` - *Integer, default 500* - mechanism to control the order in which actions on a table are executed, if there are more than one.
Actions with a smaller value for `priority` are executed first. Ties are broken in an undefined manner.
** `filter` - *QQueryFilter* - optional filter that gets applied to records when they match the `triggerEvent`, to control which records have the action ran against them.
** `includeRecordAssociations` - *Boolean, default false* - for tables that have associations, control whether or not a record's associated records are loaded when records are fetched and passed into the action's custom code.
** `values` - *Map of String → Serializable* - optional application-defined map of name=value pairs that can be passed into the action's custom code.
** `processName` - *String, Conditional* - name of a {link-processes} in the QQQ Instance which is executed as the custom-code of the action.
** `codeReference` - *QCodeReference, Conditional* - reference to a class that extends `RecordAutomationHandler`, to be executed as the custom-code of the action.
*** Note, exactly one of `processName` or `codeReference` must be provided.
==== Association
An `Association` is a way to define a relationship between tables, that facilitates, for example, a parent record having a list of its child records included in it when it is returned from a Query.
Similarly, associated records can automatically be inserted/updated/deleted if they are included in a parent record when it is stored.
*Association Properties:*
* `name` - *String, Required* - unique name for the association within this table.
Used as the key in the `associatedRecords` map within `QRecord` objects for this table.
* `associatedTableName` - *String, Required* - name of a {link-table}, which is the associated table.
* `joinName` - *String, Required* - name of a {link-join} in the instance, which defines how the tables are joined.
==== RecordSecurityLock
A `RecordSecurityLock` is the mechanism through which users can be allowed or denied access to read and/or write records, based on values in the record, and values in the user's session.
Record security locks must correspond to a {link-securityKeyType}.
For example:
* An instance may have a security key type called `clientId`.
* Users may have 1 or more `clientId` values in their Session, or, they may have an "All Clients" key in their session (e.g., for internal/admin users).
* For some tables, it may be required to limit visibility to records based on a user's `clientId` key.
To do this. a *RecordSecurityLock* would be applied to the table, specifying the `clientId` field corresponds to the `clientId` security key.
* With these settings in place, QQQ will prevent users from viewing records from this table that do not have a matching key, and will similarly prevent users from writing records with an invalid key value.
** For example, in an RDBMS backend, all `SELECT` statements generated against such a table will have an implicit filter, such as `AND client_id = ?` based on the user's security key values.
*RecordSecurityLock Properties:*
* `securityKeyType` - *String, Required* - name of a {link-securityKeyType} in the Instance.
* `fieldName` - *String, Required* - name of a {link-field} in this table (or a joined table, if `joinNameChain` is set), where the value for the lock is stored.
* `joinNameChain` - *List of String* - if the lock value is not stored in this table, but rather comes from a joined table, then this property defines the path of joins from this table to the table with the lock field.
* `nullValueBehavior` - *enum, default: DENY* - control how records with a `null` value in the lock field should behave.
Possible values are:
** `DENY` - deny all users access to a record with a `null` value in the lock field (unless the user has an all-access key - see {link-securityKeyType})
** `ALLOW` - allow all users access to a record with a `null` value in the lock field.
** `ALLOW_WRITE_ONLY` - allow all users to write records with `null` in the lock field, but deny reads on records with `null` in the lock field (also excepted by all-access keys).
* `lockScope` - *enum, default: READ_AND_WRITE* - control what types of operations the lock applies to.
Possible values are:
** `READ_AND_WRITE` - control both reading and writing records based on the user having an appropriate security key.
** `WRITE` - allow all users to read the record, but limit writes to users with an appropriate security key.
==== QAuditRules
The audit rules on a table define the level of detail that is automatically stored in the audit table (if any) for DML actions (Insert, Update, Delete).
*QAuditRules Properties:*
* `auditLevel` - *enum, Required* - level of details that are audited.
Possible values are:
** `NONE` - no automatic audits are stored for the table.
** `RECORD` - only record-level audits are stored for the table (e.g., a message such as "record was edited", but without field-level details)
** `FIELD` - full field-level audits are stored (e.g., including all old & new values as audit details).
==== CacheOf
One QQQ Table can be defined as a "cache of" another QQQ Table by assigning a `CacheOf` object to the table which will function as the cache.
_Note, at this time, only limited use-cases are supported._
*CacheOf Properties:*
* `sourceTable` - *String, Required* - name of the other QQQ Table that is the source of data in this cache.
* `expirationSeconds` - *Integer* - optional number of seconds that a cached record is allowed to exist before it is considered expired, and must be re-fetched from the source table.
* `cachedDateFieldName` - *String, Conditional* - used with `expirationSeconds` to define the field in this table that is used for storing the timestamp for when the record was cached.
* `useCases` - *List of CacheUseCase* - what caching use-cases are to be implemented.
Properties of *CacheUseCase* are:
* `type` - *Enum, Required* - the type of use-case. Possible values are:
** `PRIMARY_KEY_TO_PRIMARY_KEY` - the primary key in the cache table equals the primary key in the source table.
** `UNIQUE_KEY_TO_PRIMARY_KEY` - a unique key in the cache table equals the primary key in the source table.
** `UNIQUE_KEY_TO_UNIQUE_KEY` - a unique key in the cache table equals a unique key in the source table.
* `cacheSourceMisses` - *Boolean, default false* - whether or not, if a "miss" happens in the source, if that fact gets cached.
* `cacheUniqueKey` - *UniqueKey, conditional* - define the fields in the cache table that define the unique key being used as the cache key.
* `sourceUniqueKey` - *UniqueKey, conditional* - define the fields in the source table that define the unique key being used as the cache key.
* `doCopySourcePrimaryKeyToCache` - *Boolean, default false* - specify whether or not the value of the primary key in the source table should be copied into records built in the cache table.
* `excludeRecordsMatching` - *List of QQueryFilter* - optional filter to be applied to records before they are cached.
If a record matches the filter, then it will not be cached.
==== ExposedJoin
Query screens in QQQ applications can potentially allow users to both display fields from joined tables, and filter by fields from joined tables, for any {link-join} explicitly defined as an *Exposed Join*.
_The reasoning why not all joins are implicitly exposed is that in many applications, the full join-graph can sometimes be overwhelming, surprisingly broad, and not necessarily practically useful.
This could be subject to change in the future, e.g., given a UI that allowed users to more explicitly add additional join tables..._
*ExposedJoin Properties:*
* `label` - *String, Required* - how the joined table should be presented in the UI.
* `joinTable` - *String, Required* - name of the QQQ Table that is joined to this table, and is being exposed as a join in the UI.
* `joinPath` - *List of String, Required* - names of 1 or more QQQ Joins that describe how to get from this table to the join table.
==== AssociatedScript
A QQQ Table can have end-user defined Script records associated with individual records in the table by use of the `associatedScripts` property of the table's meta-data.
The "types" of these scripts (e.g., how they are used in an application) are wholly application-designed & managed.
QQQ provides the mechanism for UI's to present and manage such scripts (e.g., the *Developer Mode* screen in the Material Dashboard), as well as an interface to load & execute such scripts `RunAssociatedScriptAction`).
*AssociatedScript Properties:*
* `fieldName` - *String, Required* - name of a {link-field} in the table which stores the id of the associated script record.
* `scriptTypeId` - *Serializable (typically Integer), Required* - primary key value from the `"scriptType"` table in the instance, to designate the type of the Script.
* `scriptTester` - *QCodeReference* - reference to a class which implements `TestScriptActionInterface`, that can be used by UI's for running an associated script to test it.

553
docs/metaData/Tables.html Normal file
View File

@ -0,0 +1,553 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.18">
<title>QQQ Tables</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
/*! Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */
/* Uncomment the following line when using as a custom stylesheet */
/* @import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"; */
html{font-family:sans-serif;-webkit-text-size-adjust:100%}
a{background:none}
a:focus{outline:thin dotted}
a:active,a:hover{outline:0}
h1{font-size:2em;margin:.67em 0}
b,strong{font-weight:bold}
abbr{font-size:.9em}
abbr[title]{cursor:help;border-bottom:1px dotted #dddddf;text-decoration:none}
dfn{font-style:italic}
hr{height:0}
mark{background:#ff0;color:#000}
code,kbd,pre,samp{font-family:monospace;font-size:1em}
pre{white-space:pre-wrap}
q{quotes:"\201C" "\201D" "\2018" "\2019"}
small{font-size:80%}
sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
sup{top:-.5em}
sub{bottom:-.25em}
img{border:0}
svg:not(:root){overflow:hidden}
figure{margin:0}
audio,video{display:inline-block}
audio:not([controls]){display:none;height:0}
fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
legend{border:0;padding:0}
button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
button,input{line-height:normal}
button,select{text-transform:none}
button,html input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer}
button[disabled],html input[disabled]{cursor:default}
input[type=checkbox],input[type=radio]{padding:0}
button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
textarea{overflow:auto;vertical-align:top}
table{border-collapse:collapse;border-spacing:0}
*,::before,::after{box-sizing:border-box}
html,body{font-size:100%}
body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;line-height:1;position:relative;cursor:auto;-moz-tab-size:4;-o-tab-size:4;tab-size:4;word-wrap:anywhere;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased}
a:hover{cursor:pointer}
img,object,embed{max-width:100%;height:auto}
object,embed{height:100%}
img{-ms-interpolation-mode:bicubic}
.left{float:left!important}
.right{float:right!important}
.text-left{text-align:left!important}
.text-right{text-align:right!important}
.text-center{text-align:center!important}
.text-justify{text-align:justify!important}
.hide{display:none}
img,object,svg{display:inline-block;vertical-align:middle}
textarea{height:auto;min-height:50px}
select{width:100%}
.subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0}
a{color:#2156a5;text-decoration:underline;line-height:inherit}
a:hover,a:focus{color:#1d4b8f}
a img{border:0}
p{line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
p aside{font-size:.875em;line-height:1.35;font-style:italic}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
h1{font-size:2.125em}
h2{font-size:1.6875em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
h4,h5{font-size:1.125em}
h6{font-size:1em}
hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em}
em,i{font-style:italic;line-height:inherit}
strong,b{font-weight:bold;line-height:inherit}
small{font-size:60%;line-height:inherit}
code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
ul,ol,dl{line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
ul,ol{margin-left:1.5em}
ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0}
ul.circle{list-style-type:circle}
ul.disc{list-style-type:disc}
ul.square{list-style-type:square}
ul.circle ul:not([class]),ul.disc ul:not([class]),ul.square ul:not([class]){list-style:inherit}
ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
dl dt{margin-bottom:.3125em;font-weight:bold}
dl dd{margin-bottom:1.25em}
blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
@media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
h1{font-size:2.75em}
h2{font-size:2.3125em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
h4{font-size:1.4375em}}
table{background:#fff;margin-bottom:1.25em;border:1px solid #dedede;word-wrap:normal}
table thead,table tfoot{background:#f7f8f7}
table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
table tr.even,table tr.alt{background:#f8f8f7}
table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{line-height:1.6}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
.center{margin-left:auto;margin-right:auto}
.stretch{width:100%}
.clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table}
.clearfix::after,.float-group::after{clear:both}
:not(pre).nobreak{word-wrap:normal}
:not(pre).nowrap{white-space:nowrap}
:not(pre).pre-wrap{white-space:pre-wrap}
:not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed}
pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed}
pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit}
pre>code{display:block}
pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal}
em em{font-style:normal}
strong strong{font-weight:400}
.keyseq{color:rgba(51,51,51,.8)}
kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;border-radius:3px;box-shadow:0 1px 0 rgba(0,0,0,.2),inset 0 0 0 .1em #fff;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
.keyseq kbd:first-child{margin-left:0}
.keyseq kbd:last-child{margin-right:0}
.menuseq,.menuref{color:#000}
.menuseq b:not(.caret),.menuref{font-weight:inherit}
.menuseq{word-spacing:-.02em}
.menuseq b.caret{font-size:1.25em;line-height:.8}
.menuseq i.caret{font-weight:bold;text-align:center;width:.45em}
b.button::before,b.button::after{position:relative;top:-1px;font-weight:400}
b.button::before{content:"[";padding:0 3px 0 2px}
b.button::after{content:"]";padding:0 2px 0 3px}
p a>code:hover{color:rgba(0,0,0,.9)}
#header,#content,#footnotes,#footer{width:100%;margin:0 auto;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
#header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table}
#header::after,#content::after,#footnotes::after,#footer::after{clear:both}
#content{margin-top:1.25em}
#content::before{content:none}
#header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
#header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf}
#header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px}
#header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:flex;flex-flow:row wrap}
#header .details span:first-child{margin-left:-.125em}
#header .details span.email a{color:rgba(0,0,0,.85)}
#header .details br{display:none}
#header .details br+span::before{content:"\00a0\2013\00a0"}
#header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
#header .details br+span#revremark::before{content:"\00a0|\00a0"}
#header #revnumber{text-transform:capitalize}
#header #revnumber::after{content:"\00a0"}
#content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
#toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em}
#toc>ul{margin-left:.125em}
#toc ul.sectlevel0>li>a{font-style:italic}
#toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
#toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
#toc li{line-height:1.3334;margin-top:.3334em}
#toc a{text-decoration:none}
#toc a:active{text-decoration:underline}
#toctitle{color:#7a2518;font-size:1.2em}
@media screen and (min-width:768px){#toctitle{font-size:1.375em}
body.toc2{padding-left:15em;padding-right:0}
#toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
#toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
#toc.toc2>ul{font-size:.9em;margin-bottom:0}
#toc.toc2 ul ul{margin-left:0;padding-left:1em}
#toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
body.toc2.toc-right{padding-left:0;padding-right:15em}
body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}}
@media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
#toc.toc2{width:20em}
#toc.toc2 #toctitle{font-size:1.375em}
#toc.toc2>ul{font-size:.95em}
#toc.toc2 ul ul{padding-left:1.25em}
body.toc2.toc-right{padding-left:0;padding-right:20em}}
#content #toc{border:1px solid #e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;border-radius:4px}
#content #toc>:first-child{margin-top:0}
#content #toc>:last-child{margin-bottom:0}
#footer{max-width:none;background:rgba(0,0,0,.8);padding:1.25em}
#footer-text{color:hsla(0,0%,100%,.8);line-height:1.44}
#content{margin-bottom:.625em}
.sect1{padding-bottom:.625em}
@media screen and (min-width:768px){#content{margin-bottom:1.25em}
.sect1{padding-bottom:1.25em}}
.sect1:last-child{padding-bottom:0}
.sect1+.sect1{border-top:1px solid #e7e7e9}
#content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
#content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
#content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
#content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
#content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
details{margin-left:1.25rem}
details>summary{cursor:pointer;display:block;position:relative;line-height:1.6;margin-bottom:.625rem;outline:none;-webkit-tap-highlight-color:transparent}
details>summary::-webkit-details-marker{display:none}
details>summary::before{content:"";border:solid transparent;border-left:solid;border-width:.3em 0 .3em .5em;position:absolute;top:.5em;left:-1.25rem;transform:translateX(15%)}
details[open]>summary::before{border:solid transparent;border-top:solid;border-width:.5em .3em 0;transform:translateY(15%)}
details>summary::after{content:"";width:1.25rem;height:1em;position:absolute;top:.3em;left:-1.25rem}
.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
table.tableblock.fit-content>caption.title{white-space:nowrap;width:0}
.paragraph.lead>p,#preamble>.sectionbody>[class=paragraph]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)}
.admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
.admonitionblock>table td.icon{text-align:center;width:80px}
.admonitionblock>table td.icon img{max-width:none}
.admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
.admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6);word-wrap:anywhere}
.admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
.exampleblock>.content{border:1px solid #e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;border-radius:4px}
.exampleblock>.content>:first-child{margin-top:0}
.exampleblock>.content>:last-child{margin-bottom:0}
.sidebarblock{border:1px solid #dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;border-radius:4px}
.sidebarblock>:first-child{margin-top:0}
.sidebarblock>:last-child{margin-bottom:0}
.sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
.literalblock pre,.listingblock>.content>pre{border-radius:4px;overflow-x:auto;padding:1em;font-size:.8125em}
@media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}}
@media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}}
.literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class=highlight],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8}
.literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)}
.listingblock>.content{position:relative}
.listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5}
.listingblock:hover code[data-lang]::before{display:block}
.listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5}
.listingblock.terminal pre .command:not([data-prompt])::before{content:"$"}
.listingblock pre.highlightjs{padding:0}
.listingblock pre.highlightjs>code{padding:1em;border-radius:4px}
.listingblock pre.prettyprint{border-width:0}
.prettyprint{background:#f7f7f8}
pre.prettyprint .linenums{line-height:1.45;margin-left:2em}
pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0}
pre.prettyprint li code[data-lang]::before{opacity:1}
pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none}
table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none}
table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal}
table.linenotable td.code{padding-left:.75em}
table.linenotable td.linenos,pre.pygments .linenos{border-right:1px solid;opacity:.35;padding-right:.5em;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}
pre.pygments span.linenos{display:inline-block;margin-right:.75em}
.quoteblock{margin:0 1em 1.25em 1.5em;display:table}
.quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em}
.quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
.quoteblock blockquote{margin:0;padding:0;border:0}
.quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
.quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
.quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right}
.verseblock{margin:0 1em 1.25em}
.verseblock pre{font-family:"Open Sans","DejaVu Sans",sans-serif;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
.verseblock pre strong{font-weight:400}
.verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
.quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
.quoteblock .attribution br,.verseblock .attribution br{display:none}
.quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
.quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none}
.quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0}
.quoteblock.abstract{margin:0 1em 1.25em;display:block}
.quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center}
.quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf}
.quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0}
.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem}
.quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;font-size:.85rem;text-align:left;margin-right:0}
p.tableblock:last-child{margin-bottom:0}
td.tableblock>.content{margin-bottom:1.25em;word-wrap:anywhere}
td.tableblock>.content>:last-child{margin-bottom:-1.25em}
table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
table.grid-all>*>tr>*{border-width:1px}
table.grid-cols>*>tr>*{border-width:0 1px}
table.grid-rows>*>tr>*{border-width:1px 0}
table.frame-all{border-width:1px}
table.frame-ends{border-width:1px 0}
table.frame-sides{border-width:0 1px}
table.frame-none>colgroup+*>:first-child>*,table.frame-sides>colgroup+*>:first-child>*{border-top-width:0}
table.frame-none>:last-child>:last-child>*,table.frame-sides>:last-child>:last-child>*{border-bottom-width:0}
table.frame-none>*>tr>:first-child,table.frame-ends>*>tr>:first-child{border-left-width:0}
table.frame-none>*>tr>:last-child,table.frame-ends>*>tr>:last-child{border-right-width:0}
table.stripes-all>*>tr,table.stripes-odd>*>tr:nth-of-type(odd),table.stripes-even>*>tr:nth-of-type(even),table.stripes-hover>*>tr:hover{background:#f8f8f7}
th.halign-left,td.halign-left{text-align:left}
th.halign-right,td.halign-right{text-align:right}
th.halign-center,td.halign-center{text-align:center}
th.valign-top,td.valign-top{vertical-align:top}
th.valign-bottom,td.valign-bottom{vertical-align:bottom}
th.valign-middle,td.valign-middle{vertical-align:middle}
table thead th,table tfoot th{font-weight:bold}
tbody tr th{background:#f7f8f7}
tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
p.tableblock>code:only-child{background:none;padding:0}
p.tableblock{font-size:1em}
ol{margin-left:1.75em}
ul li ol{margin-left:1.5em}
dl dd{margin-left:1.125em}
dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none}
ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em}
ul.unstyled,ol.unstyled{margin-left:0}
li>p:empty:only-child::before{content:"";display:inline-block}
ul.checklist>li>p:first-child{margin-left:-1em}
ul.checklist>li>p:first-child>.fa-square-o:first-child,ul.checklist>li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em}
ul.checklist>li>p:first-child>input[type=checkbox]:first-child{margin-right:.25em}
ul.inline{display:flex;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em}
ul.inline>li{margin-left:1.25em}
.unstyled dl dt{font-weight:400;font-style:normal}
ol.arabic{list-style-type:decimal}
ol.decimal{list-style-type:decimal-leading-zero}
ol.loweralpha{list-style-type:lower-alpha}
ol.upperalpha{list-style-type:upper-alpha}
ol.lowerroman{list-style-type:lower-roman}
ol.upperroman{list-style-type:upper-roman}
ol.lowergreek{list-style-type:lower-greek}
.hdlist>table,.colist>table{border:0;background:none}
.hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
td.hdlist1{font-weight:bold;padding-bottom:1.25em}
td.hdlist2{word-wrap:anywhere}
.literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
.colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top}
.colist td:not([class]):first-child img{max-width:none}
.colist td:not([class]):last-child{padding:.25em 0}
.thumb,.th{line-height:0;display:inline-block;border:4px solid #fff;box-shadow:0 0 0 1px #ddd}
.imageblock.left{margin:.25em .625em 1.25em 0}
.imageblock.right{margin:.25em 0 1.25em .625em}
.imageblock>.title{margin-bottom:0}
.imageblock.thumb,.imageblock.th{border-width:6px}
.imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
.image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
.image.left{margin-right:.625em}
.image.right{margin-left:.625em}
a.image{text-decoration:none;display:inline-block}
a.image object{pointer-events:none}
sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
sup.footnote a,sup.footnoteref a{text-decoration:none}
sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline}
#footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
#footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0}
#footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em}
#footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em}
#footnotes .footnote:last-of-type{margin-bottom:0}
#content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
div.unbreakable{page-break-inside:avoid}
.big{font-size:larger}
.small{font-size:smaller}
.underline{text-decoration:underline}
.overline{text-decoration:overline}
.line-through{text-decoration:line-through}
.aqua{color:#00bfbf}
.aqua-background{background:#00fafa}
.black{color:#000}
.black-background{background:#000}
.blue{color:#0000bf}
.blue-background{background:#0000fa}
.fuchsia{color:#bf00bf}
.fuchsia-background{background:#fa00fa}
.gray{color:#606060}
.gray-background{background:#7d7d7d}
.green{color:#006000}
.green-background{background:#007d00}
.lime{color:#00bf00}
.lime-background{background:#00fa00}
.maroon{color:#600000}
.maroon-background{background:#7d0000}
.navy{color:#000060}
.navy-background{background:#00007d}
.olive{color:#606000}
.olive-background{background:#7d7d00}
.purple{color:#600060}
.purple-background{background:#7d007d}
.red{color:#bf0000}
.red-background{background:#fa0000}
.silver{color:#909090}
.silver-background{background:#bcbcbc}
.teal{color:#006060}
.teal-background{background:#007d7d}
.white{color:#bfbfbf}
.white-background{background:#fafafa}
.yellow{color:#bfbf00}
.yellow-background{background:#fafa00}
span.icon>.fa{cursor:default}
a span.icon>.fa{cursor:inherit}
.admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
.admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c}
.admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
.admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900}
.admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400}
.admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000}
.conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);border-radius:50%;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
.conum[data-value] *{color:#fff!important}
.conum[data-value]+b{display:none}
.conum[data-value]::after{content:attr(data-value)}
pre .conum[data-value]{position:relative;top:-.125em}
b.conum *{color:inherit!important}
.conum:not([data-value]):empty{display:none}
dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
h1,h2,p,td.content,span.alt,summary{letter-spacing:-.01em}
p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
p,blockquote,dt,td.content,span.alt,summary{font-size:1.0625rem}
p{margin-bottom:1.25rem}
.sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
.exampleblock>.content{background:#fffef7;border-color:#e0e0dc;box-shadow:0 1px 4px #e0e0dc}
.print-only{display:none!important}
@page{margin:1.25cm .75cm}
@media print{*{box-shadow:none!important;text-shadow:none!important}
html{font-size:80%}
a{color:inherit!important;text-decoration:underline!important}
a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
abbr[title]{border-bottom:1px dotted}
abbr[title]::after{content:" (" attr(title) ")"}
pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
thead{display:table-header-group}
svg{max-width:100%}
p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
#header,#content,#footnotes,#footer{max-width:none}
#toc,.sidebarblock,.exampleblock>.content{background:none!important}
#toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important}
body.book #header{text-align:center}
body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em}
body.book #header .details{border:0!important;display:block;padding:0!important}
body.book #header .details span:first-child{margin-left:0!important}
body.book #header .details br{display:block}
body.book #header .details br+span::before{content:none!important}
body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
.listingblock code[data-lang]::before{display:block}
#footer{padding:0 .9375em}
.hide-on-print{display:none!important}
.print-only{display:block!important}
.hide-for-print{display:none!important}
.show-for-print{display:inherit!important}}
@media amzn-kf8,print{#header>h1:first-child{margin-top:1.25rem}
.sect1{padding:0!important}
.sect1+.sect1{border:0}
#footer{background:none}
#footer-text{color:rgba(0,0,0,.6);font-size:.9em}}
@media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}}
</style>
</head>
<body class="article">
<div id="header">
</div>
<div id="content">
<div class="sect1">
<h2 id="_qqq_tables">QQQ Tables</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The core type of object in a QQQ Instance is the Table.
In the most common use-case, a QQQ Table may be the in-app representation of a Database table.
That is, it is a collection of records (or rows) of data, each of which has a set of fields (or columns).</p>
</div>
<div class="paragraph">
<p>QQQ also allows other types of data sources (<a href="Backends{relfilesuffix}">QQQ Backends</a>) to be used as tables, such as File systems, API&#8217;s, Java enums or objects, etc.
All of these backend types present the same interfaces (both user-interfaces, and application programming interfaces), regardless of their backend type.</p>
</div>
<div class="sect2">
<h3 id="_qtablemetadata">QTableMetaData</h3>
<div class="paragraph">
<p>Tables are defined in a QQQ Instance in a <code><strong>QTableMetaData</strong></code> object.
All tables must reference a <a href="Backends{relfilesuffix}">QQQ Backend</a>, a list of fields that define the shape of records in the table, and additional data to describe how to work with the table within its backend.</p>
</div>
<div class="paragraph">
<p><strong>QTableMetaData Properties:</strong></p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>name</code> - <strong>String, Required</strong> - Unique name for the table within the QQQ Instance.</p>
</li>
<li>
<p><code>label</code> - <strong>String</strong> - User-facing label for the table, presented in User Interfaces.
Inferred from <code>name</code> if not set.</p>
</li>
<li>
<p><code>backendName</code> - <strong>String, Required</strong> - Name of a <a href="Backends{relfilesuffix}">QQQ Backend</a> in which this table&#8217;s data is managed.</p>
</li>
<li>
<p><code>fields</code> - <strong>Map of String → <a href="Fields{relfilesuffix}">QQQ Field</a>, Required</strong> - The columns of data that make up all records in this table.</p>
</li>
<li>
<p><code>primaryKeyField</code> - <strong>String, Conditional</strong> - Name of a <a href="Fields{relfilesuffix}">QQQ Field</a> that serves as the primary key (e.g., unique identifier) for records in this table.</p>
</li>
<li>
<p><code>uniqueKeys</code> - <strong>List of UniqueKey</strong> - Definition of additional unique constraints (from an RDBMS point of view) from the table.
e.g., sets of columns which must have unique values for each record in the table.</p>
</li>
<li>
<p><code>backendDetails</code> - <strong>QTableBackendDetails or subclass</strong> - Additional data to configure the table within its <a href="Backends{relfilesuffix}">QQQ Backend</a>.</p>
</li>
<li>
<p><code>automationDetails</code> - <strong>QTableAutomationDetails</strong> - Configuration of automated jobs that run against records in the table, e.g., upon insert or update.</p>
</li>
<li>
<p><code>customizers</code> - <strong>Map of String → QCodeReference</strong> - References to custom code that are injected into standard table actions, that allow applications to customize certain parts of how the table works.</p>
</li>
<li>
<p><code>parentAppName</code> - <strong>String</strong> - Name of a <a href="Apps{relfilesuffix}">QQQ App</a> that this table exists within.</p>
</li>
<li>
<p><code>icon</code> - <strong>QIcon</strong> - Icon associated with this table in certain user interfaces.</p>
</li>
<li>
<p><code>recordLabelFormat</code> - <strong>String</strong> - Java Format String, used with <code>recordLabelFields</code> to produce a label shown for records from the table.</p>
</li>
<li>
<p><code>recordLabelFields</code> - <strong>List of String, Conditional</strong> - Used with <code>recordLabelFormat</code> to provide values for any format specifiers in the format string.
These strings must be field names within the table.</p>
<div class="ulist">
<ul>
<li>
<p>Example of using <code>recordLabelFormat</code> and <code>recordLabelFields</code>:</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><span style="color:#777">// given these fields in the table:</span>
<span style="color:#080;font-weight:bold">new</span> QFieldMetaData(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">name</span><span style="color:#710">&quot;</span></span>, QFieldType.STRING)
<span style="color:#080;font-weight:bold">new</span> QFieldMetaData(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">birthDate</span><span style="color:#710">&quot;</span></span>, QFieldType.DATE)
<span style="color:#777">// We can produce a record label such as &quot;Darin Kelkhoff (1980-05-31)&quot; via:</span>
.withRecordLabelFormat(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">%s (%s)</span><span style="color:#710">&quot;</span></span>)
.withRecordLabelFields(<span style="color:#0a8;font-weight:bold">List</span>.of(<span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">name</span><span style="color:#710">&quot;</span></span>, <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">birthDate</span><span style="color:#710">&quot;</span></span>))</code></pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p><code>sections</code> - <strong>List of QFieldSection</strong> - Mechanism to organize fields within user interfaces, into logical sections.
If any sections are present in the table meta data, then all fields in the table must be listed in exactly 1 section.
If no sections are defined, then instance enrichment will define default sections.</p>
</li>
<li>
<p><code>associatedScripts</code> - <strong>List of AssociatedScript</strong> - Definition of user-defined scripts that can be associated with records within the table.</p>
</li>
<li>
<p><code>enabledCapabilities</code> and <code>disabledCapabilities</code> - <strong>Set of Capability enum values</strong> - Overrides from the backend level, for capabilities that this table does or does not possess.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div id="footer">
<div id="footer-text">
Last updated 2022-11-21 09:02:56 -0600
</div>
</div>
</body>
</html>

View File

@ -0,0 +1,17 @@
[#Widgets]
== Widgets
include::../variables.adoc[]
#TODO#
=== QWidgetMetaData
A Widget is defined in a QQQ Instance in a `*QWidgetMetaData*` object.
#TODO#
*QWidgetMetaData Properties:*
* `name` - *String, Required* - Unique name for the widget within the QQQ Instance.
#TODO#

25
docs/variables.adoc Normal file
View File

@ -0,0 +1,25 @@
:link-app: <<Apps,QQQ App>>
:link-apps: <<Apps,QQQ Apps>>
:link-backend: <<Backends,QQQ Backend>>
:link-backends: <<Backends,QQQ Backends>>
:link-field: <<Fields,QQQ Field>>
:link-fields: <<Fields,QQQ Fields>>
:link-icon: <<Icons,Icon>>
:link-icons: <<Icons,Icons>>
:link-instance: <<QInstance,QInstance>>
:link-join: <<Joins,QQQ Join>>
:link-joins: <<Joins,QQQ Joins>>
:link-permissionRule: <<PermissionRules,Permission Rule>>
:link-permissionRules: <<PermissionRules,Permission Rules>>
:link-possibleValueSource: <<PossibleValueSources,QQQ Possible Value Source>>
:link-possibleValueSources: <<PossibleValueSources,QQQ Possible Value Sources>>
:link-process: <<Processes,QQQ Process>>
:link-processes: <<Processes,QQQ Processes>>
:link-report: <<Reports,QQQ Report>>
:link-reports: <<Reports,QQQ Reports>>
:link-securityKeyType: <<SecurityKeyTypes,Security Key Type>>
:link-securityKeyTypes: <<SecurityKeyTypes,Security Key Types>>
:link-table: <<Tables,QQQ Table>>
:link-tables: <<Tables,QQQ Tables>>
:link-widget: <<Widgets,QQQ Widget>>
:link-widgets: <<Widgets,QQQ Widgets>>

173
pom.xml
View File

@ -30,15 +30,22 @@
<modules>
<module>qqq-backend-core</module>
<module>qqq-backend-module-rdbms</module>
<module>qqq-backend-module-api</module>
<module>qqq-backend-module-filesystem</module>
<module>qqq-backend-module-rdbms</module>
<module>qqq-backend-module-mongodb</module>
<module>qqq-language-support-javascript</module>
<module>qqq-middleware-picocli</module>
<module>qqq-middleware-javalin</module>
<module>qqq-middleware-lambda</module>
<module>qqq-middleware-slack</module>
<module>qqq-middleware-api</module>
<module>qqq-utility-lambdas</module>
<module>qqq-sample-project</module>
</modules>
<properties>
<revision>0.4.0</revision>
<revision>0.20.0-SNAPSHOT</revision>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@ -49,8 +56,20 @@
<coverage.haltOnFailure>true</coverage.haltOnFailure>
<coverage.instructionCoveredRatioMinimum>0.80</coverage.instructionCoveredRatioMinimum>
<coverage.classCoveredRatioMinimum>0.95</coverage.classCoveredRatioMinimum>
<plugin.shade.phase>none</plugin.shade.phase>
</properties>
<profiles>
<profile>
<!-- For qqq-middleware-lambda - to build its shaded jar, its qqq dependencies also need
to build a shaded jar. So to activate that mode, use this profile (-P buildShadedJar)-->
<id>buildShadedJar</id>
<properties>
<plugin.shade.phase>package</plugin.shade.phase>
</properties>
</profile>
</profiles>
<dependencyManagement>
<dependencies>
<dependency>
@ -74,16 +93,32 @@
<version>5.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.23.1</version>
<scope>test</scope>
</dependency>
</dependencies>
</dependencies>
</dependencyManagement>
<build>
<resources>
<resource>
<directory>src/main/java</directory>
<filtering>false</filtering>
</resource>
<resource>
<directory>src/main/resources</directory>
<filtering>false</filtering>
</resource>
</resources>
<plugins>
<!-- plugins specifically for this module -->
<!-- none at this time -->
@ -122,7 +157,8 @@
<id>validate</id>
<phase>validate</phase>
<configuration>
<configLocation>checkstyle.xml</configLocation>
<configLocation>checkstyle/config.xml</configLocation>
<headerLocation>checkstyle/license.txt</headerLocation>
<!-- <suppressionsLocation>checkstyle-suppressions.xml</suppressionsLocation> -->
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
@ -181,6 +217,63 @@
<skipUpdateVersion>true</skipUpdateVersion>
</configuration>
</plugin>
<plugin>
<artifactId>exec-maven-plugin</artifactId>
<groupId>org.codehaus.mojo</groupId>
<version>3.0.0</version>
<executions>
<execution>
<id>test-coverage-summary</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>sh</executable>
<arguments>
<argument>-c</argument>
<argument>
<![CDATA[
if [ ! -e target/site/jacoco/index.html ]; then
echo "No jacoco coverage report here.";
exit;
fi
echo
echo "Jacoco coverage summary report:"
echo " See also target/site/jacoco/index.html"
echo " and https://www.jacoco.org/jacoco/trunk/doc/counters.html"
echo "------------------------------------------------------------"
which xpath > /dev/null 2>&1
if [ "$?" == "0" ]; then
echo "Element\nInstructions Missed\nInstruction Coverage\nBranches Missed\nBranch Coverage\nComplexity Missed\nComplexity Hit\nLines Missed\nLines Hit\nMethods Missed\nMethods Hit\nClasses Missed\nClasses Hit\n" > /tmp/$$.headers
xpath -n -q -e '/html/body/table/tfoot/tr[1]/td/text()' target/site/jacoco/index.html > /tmp/$$.values
paste /tmp/$$.headers /tmp/$$.values | tail +2 | awk -v FS='\t' '{printf("%-20s %s\n",$1,$2)}'
rm /tmp/$$.headers /tmp/$$.values
else
echo "xpath is not installed. Jacoco coverage summary will not be produced here...";
fi
which xpath > /dev/null 2>&1
if [ "$?" == "0" ]; then
echo "Untested classes, per Jacoco:"
echo "-----------------------------"
for i in target/site/jacoco/*/index.html; do
html2text -width 500 -nobs $i | sed '1,/^Total/d;' | grep -v Created | sed 's/ \+/ /g' | sed 's/ [[:digit:]]$//' | grep -v 0$ | cut -d' ' -f1;
done;
echo
else
echo "html2text is not installed. Untested classes from Jacoco will not be printed here...";
fi
]]>
</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
@ -224,59 +317,43 @@
</execution>
<execution>
<id>post-unit-test</id>
<phase>verify</phase>
<!-- <phase>verify</phase> -->
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>exec-maven-plugin</artifactId>
<groupId>org.codehaus.mojo</groupId>
<version>3.0.0</version>
<executions>
<execution>
<id>test-coverage-summary</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>sh</executable>
<arguments>
<argument>-c</argument>
<argument>
<![CDATA[
if [ ! -e target/site/jacoco/index.html ]; then
echo "No jacoco coverage report here.";
exit;
fi
echo
echo "Jacoco coverage summary report:"
echo " See also target/site/jacoco/index.html"
echo " and https://www.jacoco.org/jacoco/trunk/doc/counters.html"
echo "------------------------------------------------------------"
which xpath > /dev/null 2>&1
if [ "$?" == "0" ]; then
echo "Element\nInstructions Missed\nInstruction Coverage\nBranches Missed\nBranch Coverage\nComplexity Missed\nComplexity Hit\nLines Missed\nLines Hit\nMethods Missed\nMethods Hit\nClasses Missed\nClasses Hit\n" > /tmp/$$.headers
xpath -n -q -e '/html/body/table/tfoot/tr[1]/td/text()' target/site/jacoco/index.html > /tmp/$$.values
paste /tmp/$$.headers /tmp/$$.values | tail +2 | awk -v FS='\t' '{printf("%-20s %s\n",$1,$2)}'
rm /tmp/$$.headers /tmp/$$.values
else
echo "xpath is not installed. Jacoco coverage summary will not be produced here..";
fi
]]>
</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<!-- mvn javadoc:aggregate -->
<reporting>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>3.6.2</version>
<reportSets>
<reportSet>
<id>aggregate</id>
<inherited>false</inherited>
<reports>
<report>aggregate</report>
</reports>
</reportSet>
<reportSet>
<id>default</id>
<reports>
<report>javadoc</report>
</reports>
</reportSet>
</reportSets>
</plugin>
</plugins>
</reporting>
<repositories>
<repository>
<id>github-qqq-maven-registry</id>

View File

@ -36,25 +36,55 @@
<!-- noe at this time -->
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.17.259</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<!-- other qqq modules deps -->
<!-- none, this is core. -->
<!-- 3rd party deps specifically for this module -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>quicksight</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apigateway</artifactId>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-secretsmanager</artifactId>
<version>1.12.385</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.13.2.1</version>
<version>2.14.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>2.13.0</version>
<version>2.14.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.14.0</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20210307</version>
<version>20231013</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
@ -74,14 +104,64 @@
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>mvc-auth-commons</artifactId>
<version>1.9.2</version>
<artifactId>auth0</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>4.4.0</version>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>jwks-rsa</artifactId>
<version>0.22.0</version>
</dependency>
<dependency>
<groupId>io.github.cdimascio</groupId>
<artifactId>java-dotenv</artifactId>
<version>5.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.velocity</groupId>
<artifactId>velocity-engine-core</artifactId>
<version>2.3</version>
</dependency>
<!-- the next 2 deps are for html to pdf - per https://www.baeldung.com/java-html-to-pdf -->
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.15.3</version>
</dependency>
<dependency>
<groupId>org.xhtmlrenderer</groupId>
<artifactId>flying-saucer-pdf-openpdf</artifactId>
<version>9.1.22</version>
</dependency>
<!-- the next 3 deps are being added for google drive support -->
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.35.2</version>
</dependency>
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
<version>1.11.0</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-drive</artifactId>
<version>v3-rev20220815-2.0.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sqs</artifactId>
<version>1.12.321</version>
</dependency>
<!-- Common deps for all qqq modules -->
<dependency>
@ -101,12 +181,24 @@
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
@ -121,6 +213,30 @@
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>${plugin.shade.phase}</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

View File

@ -0,0 +1,76 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Future;
import com.kingsrook.qqq.backend.core.context.CapturedContext;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionOutput;
/*******************************************************************************
** Base class for QQQ Actions (both framework and application defined) that
** have a signature like a BiConsumer - taking both Input and Output objects as
** parameters, with void output.
*******************************************************************************/
public abstract class AbstractQActionBiConsumer<I extends AbstractActionInput, O extends AbstractActionOutput>
{
/*******************************************************************************
**
*******************************************************************************/
public abstract void execute(I input, O output) throws QException;
/*******************************************************************************
**
*******************************************************************************/
public Future<Void> executeAsync(I input, O output)
{
CapturedContext capturedContext = QContext.capture();
CompletableFuture<Void> completableFuture = new CompletableFuture<>();
ActionHelper.getExecutorService().submit(() ->
{
try
{
QContext.init(capturedContext);
execute(input, output);
completableFuture.complete(null);
}
catch(QException e)
{
completableFuture.completeExceptionally(e);
}
finally
{
QContext.clear();
}
});
return (completableFuture);
}
}

View File

@ -0,0 +1,76 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Future;
import com.kingsrook.qqq.backend.core.context.CapturedContext;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionOutput;
/*******************************************************************************
** Base class for QQQ Actions (both framework and application defined) that
** have a signature like a Function - taking an Input object as a parameter,
** and returning an Output object.
*******************************************************************************/
public abstract class AbstractQActionFunction<I extends AbstractActionInput, O extends AbstractActionOutput>
{
/*******************************************************************************
**
*******************************************************************************/
public abstract O execute(I input) throws QException;
/*******************************************************************************
**
*******************************************************************************/
public Future<O> executeAsync(I input)
{
CapturedContext capturedContext = QContext.capture();
CompletableFuture<O> completableFuture = new CompletableFuture<>();
ActionHelper.getExecutorService().submit(() ->
{
try
{
QContext.init(capturedContext);
O output = execute(input);
completableFuture.complete(output);
}
catch(QException e)
{
completableFuture.completeExceptionally(e);
}
finally
{
QContext.clear();
}
});
return (completableFuture);
}
}

View File

@ -22,9 +22,19 @@
package com.kingsrook.qqq.backend.core.actions;
import java.io.Serializable;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleInterface;
@ -34,18 +44,68 @@ import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModu
*******************************************************************************/
public class ActionHelper
{
/////////////////////////////////////////////////////////////////////////////
// we would probably use Executors.newCachedThreadPool() - but - it has no //
// maxPoolSize... we think some limit is good, so that at a large number //
// of attempted concurrent jobs we'll have new jobs block, rather than //
// exhausting all server resources and locking up "everything" //
// also, it seems like keeping a handful of core-threads around is very //
// little actual waste, and better than ever wasting time starting a new //
// one, which we know we'll often be doing. //
/////////////////////////////////////////////////////////////////////////////
private static Integer CORE_THREADS = 8;
private static Integer MAX_THREADS = 500;
private static ExecutorService executorService = new ThreadPoolExecutor(CORE_THREADS, MAX_THREADS, 60L, TimeUnit.SECONDS, new SynchronousQueue<>());
/*******************************************************************************
**
*******************************************************************************/
public static void validateSession(AbstractActionInput request) throws QException
{
QInstance qInstance = QContext.getQInstance();
QSession qSession = QContext.getQSession();
if(qInstance == null)
{
throw (new QException("QInstance was not set in QContext."));
}
if(qSession == null)
{
throw (new QException("QSession was not set in QContext."));
}
QAuthenticationModuleDispatcher qAuthenticationModuleDispatcher = new QAuthenticationModuleDispatcher();
QAuthenticationModuleInterface authenticationModule = qAuthenticationModuleDispatcher.getQModule(request.getAuthenticationMetaData());
if(!authenticationModule.isSessionValid(request.getInstance(), request.getSession()))
QAuthenticationModuleInterface authenticationModule = qAuthenticationModuleDispatcher.getQModule(qInstance.getAuthentication());
if(!authenticationModule.isSessionValid(qInstance, qSession))
{
throw new QAuthenticationException("Invalid session in request");
}
}
/*******************************************************************************
** access an executor service for sharing among the executeAsync methods of all
** actions.
*******************************************************************************/
static ExecutorService getExecutorService()
{
return (executorService);
}
/*******************************************************************************
**
*******************************************************************************/
public static void editFirstValue(List<Serializable> values, Function<String, String> editFunction)
{
if(values.size() > 0)
{
values.set(0, editFunction.apply(String.valueOf(values.get(0))));
}
}
}

View File

@ -23,6 +23,9 @@ package com.kingsrook.qqq.backend.core.actions;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
/*******************************************************************************
@ -30,12 +33,26 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
** part of a transaction.
**
** Most obvious use-case would be a JDBC Connection. See subclass in rdbms module.
** Ditto MongoDB.
**
** Note: One would imagine that this class shouldn't ever implement Serializable...
*******************************************************************************/
public class QBackendTransaction
{
/*******************************************************************************
**
*******************************************************************************/
public static QBackendTransaction openFor(AbstractTableActionInput input) throws QException
{
QBackendModuleDispatcher qBackendModuleDispatcher = new QBackendModuleDispatcher();
QBackendModuleInterface qModule = qBackendModuleDispatcher.getQBackendModule(input.getBackend());
QBackendTransaction transaction = qModule.openTransaction(input);
return (transaction);
}
/*******************************************************************************
** Commit the transaction.
*******************************************************************************/

View File

@ -31,7 +31,6 @@ import com.kingsrook.qqq.backend.core.state.UUIDAndTypeStateKey;
** Argument passed to an AsyncJob when it runs, which can be used to communicate
** data back out of the job.
**
** TODO - future - allow cancellation to be indicated here?
*******************************************************************************/
public class AsyncJobCallback
{
@ -51,6 +50,17 @@ public class AsyncJobCallback
/*******************************************************************************
** Setter for asyncJobStatus
**
*******************************************************************************/
public void setAsyncJobStatus(AsyncJobStatus asyncJobStatus)
{
this.asyncJobStatus = asyncJobStatus;
}
/*******************************************************************************
** Update the message
*******************************************************************************/
@ -68,9 +78,7 @@ public class AsyncJobCallback
public void updateStatus(String message, int current, int total)
{
this.asyncJobStatus.setMessage(message);
this.asyncJobStatus.setCurrent(current);
this.asyncJobStatus.setTotal(total);
storeUpdatedStatus();
updateStatus(current, total); // this call will storeUpdatedStatus.
}
@ -80,13 +88,65 @@ public class AsyncJobCallback
*******************************************************************************/
public void updateStatus(int current, int total)
{
this.asyncJobStatus.setCurrent(current);
this.asyncJobStatus.setCurrent(current > total ? total : current);
this.asyncJobStatus.setTotal(total);
storeUpdatedStatus();
}
/*******************************************************************************
** Update the current and total fields, but ONLY if the new values are
** both >= the previous values.
*******************************************************************************/
public void updateStatusOnlyUpwards(int current, int total)
{
boolean currentIsOkay = (this.asyncJobStatus.getCurrent() == null || this.asyncJobStatus.getCurrent() <= current);
boolean totalIsOkay = (this.asyncJobStatus.getTotal() == null || this.asyncJobStatus.getTotal() <= total);
if(currentIsOkay && totalIsOkay)
{
updateStatus(current, total);
}
}
/*******************************************************************************
** Increase the 'current' value in the '1 of 2' sense.
*******************************************************************************/
public void incrementCurrent()
{
incrementCurrent(1);
}
/*******************************************************************************
** Increase the 'current' value in the '1 of 2' sense.
*******************************************************************************/
public void incrementCurrent(int amount)
{
if(asyncJobStatus.getCurrent() != null)
{
if(asyncJobStatus.getTotal() != null && asyncJobStatus.getCurrent() + amount > asyncJobStatus.getTotal())
{
/////////////////////////////////////////////////////
// make sure we don't ever make current > total... //
/////////////////////////////////////////////////////
asyncJobStatus.setCurrent(asyncJobStatus.getTotal());
}
else
{
asyncJobStatus.setCurrent(asyncJobStatus.getCurrent() + amount);
}
storeUpdatedStatus();
}
}
/*******************************************************************************
** Remove the values from the current & total fields
*******************************************************************************/
@ -107,4 +167,17 @@ public class AsyncJobCallback
AsyncJobManager.getStateProvider().put(new UUIDAndTypeStateKey(jobUUID, StateType.ASYNC_JOB_STATUS), asyncJobStatus);
}
/*******************************************************************************
** Check if the asyncJobStatus had a cancellation requested.
**
** TODO - concern about multiple threads writing this object to a non-in-memory
** state provider, and this value getting lost...
*******************************************************************************/
public boolean wasCancelRequested()
{
return (this.asyncJobStatus.getCancelRequested());
}
}

View File

@ -28,15 +28,23 @@ import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import com.kingsrook.qqq.backend.core.context.CapturedContext;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.state.InMemoryStateProvider;
import com.kingsrook.qqq.backend.core.state.StateProviderInterface;
import com.kingsrook.qqq.backend.core.state.StateType;
import com.kingsrook.qqq.backend.core.state.UUIDAndTypeStateKey;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import org.apache.logging.log4j.Level;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
@ -44,7 +52,23 @@ import org.apache.logging.log4j.Logger;
*******************************************************************************/
public class AsyncJobManager
{
private static final Logger LOG = LogManager.getLogger(AsyncJobManager.class);
private static final QLogger LOG = QLogger.getLogger(AsyncJobManager.class);
/////////////////////////////////////////////////////////////////////////////
// we would probably use Executors.newCachedThreadPool() - but - it has no //
// maxPoolSize... we think some limit is good, so that at a large number //
// of attempted concurrent jobs we'll have new jobs block, rather than //
// exhausting all server resources and locking up "everything" //
// also, it seems like keeping a handful of core-threads around is very //
// little actual waste, and better than ever wasting time starting a new //
// one, which we know we'll often be doing. //
/////////////////////////////////////////////////////////////////////////////
private static Integer CORE_THREADS = 8;
private static Integer MAX_THREADS = 500;
private static ExecutorService executorService = new ThreadPoolExecutor(CORE_THREADS, MAX_THREADS, 60L, TimeUnit.SECONDS, new SynchronousQueue<>());
private String forcedJobUUID = null;
@ -65,17 +89,20 @@ public class AsyncJobManager
*******************************************************************************/
public <T extends Serializable> T startJob(String jobName, long timeout, TimeUnit timeUnit, AsyncJob<T> asyncJob) throws JobGoingAsyncException, QException
{
UUIDAndTypeStateKey uuidAndTypeStateKey = new UUIDAndTypeStateKey(UUID.randomUUID(), StateType.ASYNC_JOB_STATUS);
UUID jobUUID = StringUtils.hasContent(forcedJobUUID) ? UUID.fromString(forcedJobUUID) : UUID.randomUUID();
UUIDAndTypeStateKey uuidAndTypeStateKey = new UUIDAndTypeStateKey(jobUUID, StateType.ASYNC_JOB_STATUS);
AsyncJobStatus asyncJobStatus = new AsyncJobStatus();
asyncJobStatus.setState(AsyncJobState.RUNNING);
getStateProvider().put(uuidAndTypeStateKey, asyncJobStatus);
try
{
CapturedContext capturedContext = QContext.capture();
CompletableFuture<T> future = CompletableFuture.supplyAsync(() ->
{
QContext.init(capturedContext);
return (runAsyncJob(jobName, asyncJob, uuidAndTypeStateKey, asyncJobStatus));
});
}, executorService);
if(timeout == 0)
{
@ -91,7 +118,7 @@ public class AsyncJobManager
}
catch(TimeoutException e)
{
LOG.info("Job going async " + uuidAndTypeStateKey.getUuid());
LOG.debug("Job going async " + uuidAndTypeStateKey.getUuid());
throw (new JobGoingAsyncException(uuidAndTypeStateKey.getUuid().toString()));
}
}
@ -135,11 +162,11 @@ public class AsyncJobManager
Thread.currentThread().setName("Job:" + jobName + ":" + uuidAndTypeStateKey.getUuid().toString().substring(0, 8));
try
{
LOG.info("Starting job " + uuidAndTypeStateKey.getUuid());
LOG.debug("Starting job " + uuidAndTypeStateKey.getUuid());
T result = asyncJob.run(new AsyncJobCallback(uuidAndTypeStateKey.getUuid(), asyncJobStatus));
asyncJobStatus.setState(AsyncJobState.COMPLETE);
getStateProvider().put(uuidAndTypeStateKey, asyncJobStatus);
LOG.info("Completed job " + uuidAndTypeStateKey.getUuid());
LOG.debug("Completed job " + uuidAndTypeStateKey.getUuid());
return (result);
}
catch(Exception e)
@ -147,12 +174,17 @@ public class AsyncJobManager
asyncJobStatus.setState(AsyncJobState.ERROR);
asyncJobStatus.setCaughtException(e);
getStateProvider().put(uuidAndTypeStateKey, asyncJobStatus);
LOG.warn("Job " + uuidAndTypeStateKey.getUuid() + " ended with an exception: ", e);
//////////////////////////////////////////////////////
// if user facing, just log an info, warn otherwise //
//////////////////////////////////////////////////////
LOG.log((e instanceof QUserFacingException) ? Level.INFO : Level.WARN, "Job ended with an exception", e, logPair("jobId", uuidAndTypeStateKey.getUuid()));
throw (new CompletionException(e));
}
finally
{
Thread.currentThread().setName(originalThreadName);
QContext.clear();
}
}
@ -183,4 +215,46 @@ public class AsyncJobManager
// return TempFileStateProvider.getInstance();
}
/*******************************************************************************
**
*******************************************************************************/
public void cancelJob(String jobUUID)
{
Optional<AsyncJobStatus> jobStatus = getJobStatus(jobUUID);
jobStatus.ifPresent(asyncJobStatus -> asyncJobStatus.setCancelRequested(true));
}
/*******************************************************************************
** Getter for forcedJobUUID
*******************************************************************************/
public String getForcedJobUUID()
{
return (this.forcedJobUUID);
}
/*******************************************************************************
** Setter for forcedJobUUID
*******************************************************************************/
public void setForcedJobUUID(String forcedJobUUID)
{
this.forcedJobUUID = forcedJobUUID;
}
/*******************************************************************************
** Fluent setter for forcedJobUUID
*******************************************************************************/
public AsyncJobManager withForcedJobUUID(String forcedJobUUID)
{
this.forcedJobUUID = forcedJobUUID;
return (this);
}
}

View File

@ -37,6 +37,8 @@ public class AsyncJobStatus implements Serializable
private Integer total;
private Exception caughtException;
private boolean cancelRequested;
/*******************************************************************************
@ -163,4 +165,26 @@ public class AsyncJobStatus implements Serializable
{
this.caughtException = caughtException;
}
/*******************************************************************************
** Getter for cancelRequested
**
*******************************************************************************/
public boolean getCancelRequested()
{
return cancelRequested;
}
/*******************************************************************************
** Setter for cancelRequested
**
*******************************************************************************/
public void setCancelRequested(boolean cancelRequested)
{
this.cancelRequested = cancelRequested;
}
}

View File

@ -0,0 +1,217 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.async;
import java.io.Serializable;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import com.kingsrook.qqq.backend.core.actions.reporting.BufferedRecordPipe;
import com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.utils.SleepUtils;
import com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeFunction;
import com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeSupplier;
/*******************************************************************************
** Class that knows how to Run an asynchronous job (lambda, supplier) that writes into a
** RecordPipe, with another lambda (consumer) that consumes records from the pipe.
**
** Takes care of the job status monitoring, blocking when the pipe is empty, etc.
*******************************************************************************/
public class AsyncRecordPipeLoop
{
private static final QLogger LOG = QLogger.getLogger(AsyncRecordPipeLoop.class);
private static final int TIMEOUT_AFTER_NO_RECORDS_MS = 10 * 60 * 1000;
private static final int MAX_SLEEP_MS = 1000;
private static final int INIT_SLEEP_MS = 10;
private Integer minRecordsToConsume = 10;
/*******************************************************************************
** Run an async-record-pipe-loop.
**
** @param jobName name for the async job thread
** @param recordLimit optionally, cancel the supplier/job after this number of records.
* e.g., for a preview step.
** @param recordPipe constructed before this call, and used in both of the lambdas
** @param supplier lambda that adds records into the pipe.
* e.g., a query or extract step.
** @param consumer lambda that consumes records from the pipe
* e.g., a transform/load step.
*******************************************************************************/
public int run(String jobName, Integer recordLimit, RecordPipe recordPipe, UnsafeFunction<AsyncJobCallback, ? extends Serializable, QException> supplier, UnsafeSupplier<Integer, QException> consumer) throws QException
{
///////////////////////////////////////////////////
// start the extraction function as an async job //
///////////////////////////////////////////////////
AsyncJobManager asyncJobManager = new AsyncJobManager();
String jobUUID = asyncJobManager.startJob(jobName, supplier::apply);
LOG.debug("Started supplier job [" + jobUUID + "] for record pipe.");
AsyncJobState jobState = AsyncJobState.RUNNING;
AsyncJobStatus asyncJobStatus = null;
int recordCount = 0;
int nextSleepMillis = INIT_SLEEP_MS;
long lastReceivedRecordsAt = System.currentTimeMillis();
long jobStartTime = System.currentTimeMillis();
boolean everCalledConsumer = false;
while(jobState.equals(AsyncJobState.RUNNING))
{
if(recordPipe.countAvailableRecords() < minRecordsToConsume)
{
///////////////////////////////////////////////////////////////
// if the pipe is too empty, sleep to let the producer work. //
// todo - smarter sleep? like get notified vs. sleep? //
///////////////////////////////////////////////////////////////
LOG.trace("Too few records are available in the pipe. Sleeping [" + nextSleepMillis + "] ms to give producer a chance to work");
SleepUtils.sleep(nextSleepMillis, TimeUnit.MILLISECONDS);
nextSleepMillis = Math.min(nextSleepMillis * 2, MAX_SLEEP_MS);
long timeSinceLastReceivedRecord = System.currentTimeMillis() - lastReceivedRecordsAt;
if(timeSinceLastReceivedRecord > TIMEOUT_AFTER_NO_RECORDS_MS)
{
throw (new QException("Job appears to have stopped producing records (last record received " + timeSinceLastReceivedRecord + " ms ago)."));
}
}
else
{
////////////////////////////////////////////////////////////////////////////////////////////////////////
// if the pipe has records, consume them. reset the sleep timer so if we sleep again it'll be short. //
////////////////////////////////////////////////////////////////////////////////////////////////////////
lastReceivedRecordsAt = System.currentTimeMillis();
nextSleepMillis = INIT_SLEEP_MS;
everCalledConsumer = true;
recordCount += consumer.get();
LOG.debug(String.format("Processed %,d records so far", recordCount));
if(recordLimit != null && recordCount >= recordLimit)
{
asyncJobManager.cancelJob(jobUUID);
////////////////////////////////////////////////////////////////////////////////////////////////////
// in case the extract function doesn't recognize the cancellation request, //
// tell the pipe to "terminate" - meaning - flush its queue and just noop when given new records. //
// this should prevent anyone writing to such a pipe from potentially filling & blocking. //
////////////////////////////////////////////////////////////////////////////////////////////////////
recordPipe.terminate();
break;
}
}
//////////////////////////////
// refresh the job's status //
//////////////////////////////
Optional<AsyncJobStatus> optionalAsyncJobStatus = asyncJobManager.getJobStatus(jobUUID);
if(optionalAsyncJobStatus.isEmpty())
{
/////////////////////////////////////////////////
// todo - ... maybe some version of try-again? //
/////////////////////////////////////////////////
throw (new QException("Could not get status of job [" + jobUUID + "]"));
}
asyncJobStatus = optionalAsyncJobStatus.get();
jobState = asyncJobStatus.getState();
}
if(recordPipe instanceof BufferedRecordPipe bufferedRecordPipe)
{
bufferedRecordPipe.finalFlush();
}
LOG.debug("Job [" + jobUUID + "][" + jobName + "] completed with status: " + asyncJobStatus);
///////////////////////////////////
// propagate errors from the job //
///////////////////////////////////
if(asyncJobStatus != null && asyncJobStatus.getState().equals(AsyncJobState.ERROR))
{
throw (new QException("Job failed with an error", asyncJobStatus.getCaughtException()));
}
///////////////////////////////////////////////////////////////////////////////////////////
// send the final records to the consumer //
// note - we'll only make this "final" call to the consumer if: //
// - there are currently records in the pipe //
// - OR we never called the consumer (e.g., there were 0 rows produced by the supplier //
// This prevents cases where a consumer may get pages of records in the loop, but then //
// be called here post-loop w/ 0 records, and may interpret it as a sign that no records //
// were ever supplied. //
///////////////////////////////////////////////////////////////////////////////////////////
if(recordPipe.countAvailableRecords() > 0 || !everCalledConsumer)
{
recordCount += consumer.get();
}
long endTime = System.currentTimeMillis();
if(recordCount > 0)
{
LOG.info(String.format("Processed %,d records", recordCount)
+ String.format(" at end of job [%s] in %,d ms (%.2f records/second).", jobName, (endTime - jobStartTime), 1000d * (recordCount / (.001d + (endTime - jobStartTime)))));
}
return (recordCount);
}
/*******************************************************************************
** Getter for minRecordsToConsume
*******************************************************************************/
public Integer getMinRecordsToConsume()
{
return (this.minRecordsToConsume);
}
/*******************************************************************************
** Setter for minRecordsToConsume
*******************************************************************************/
public void setMinRecordsToConsume(Integer minRecordsToConsume)
{
this.minRecordsToConsume = minRecordsToConsume;
}
/*******************************************************************************
** Fluent setter for minRecordsToConsume
*******************************************************************************/
public AsyncRecordPipeLoop withMinRecordsToConsume(Integer minRecordsToConsume)
{
this.minRecordsToConsume = minRecordsToConsume;
return (this);
}
}

View File

@ -0,0 +1,406 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.audits;
import java.io.Serializable;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import com.kingsrook.qqq.backend.core.actions.AbstractQActionFunction;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.audits.AuditInput;
import com.kingsrook.qqq.backend.core.model.actions.audits.AuditOutput;
import com.kingsrook.qqq.backend.core.model.actions.audits.AuditSingleInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock;
import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLockFilters;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QUser;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.Pair;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** Insert 1 or more audits (and optionally their children, auditDetails)
**
** Takes care of managing the foreign key tables (auditTable, auditUser).
**
** Enforces that security key values are provided, if the table has any. Note that
** might mean a null is given for a particular key, but at least the key must be present.
*******************************************************************************/
public class AuditAction extends AbstractQActionFunction<AuditInput, AuditOutput>
{
private static final QLogger LOG = QLogger.getLogger(AuditAction.class);
private Map<Pair<String, String>, Integer> cachedFetches = new HashMap<>();
/*******************************************************************************
** Execute to insert 1 audit, with no details (child records)
*******************************************************************************/
public static void execute(String tableName, Integer recordId, Map<String, Serializable> securityKeyValues, String message)
{
execute(tableName, recordId, securityKeyValues, message, null);
}
/*******************************************************************************
** Execute to insert 1 audit, with a list of detail child records provided as just string messages
*******************************************************************************/
public static void executeWithStringDetails(String tableName, Integer recordId, Map<String, Serializable> securityKeyValues, String message, List<String> detailMessages)
{
List<QRecord> detailRecords = null;
if(CollectionUtils.nullSafeHasContents(detailMessages))
{
detailRecords = detailMessages.stream().map(m -> new QRecord().withValue("message", m)).toList();
}
execute(tableName, recordId, securityKeyValues, message, detailRecords);
}
/*******************************************************************************
** Execute to insert 1 audit, with a list of detail child records
*******************************************************************************/
public static void execute(String tableName, Integer recordId, Map<String, Serializable> securityKeyValues, String message, List<QRecord> details)
{
new AuditAction().execute(new AuditInput().withAuditSingleInput(new AuditSingleInput()
.withAuditTableName(tableName)
.withRecordId(recordId)
.withSecurityKeyValues(securityKeyValues)
.withMessage(message)
.withDetails(details)
));
}
/*******************************************************************************
** Simple overload that internally figures out primary key and security key values
**
** Be aware - if the record doesn't have its security key values set (say it's a
** partial record as part of an update), then those values won't be in the
** security key map... This should probably be considered a bug.
*******************************************************************************/
public static void appendToInput(AuditInput auditInput, QTableMetaData table, QRecord record, String auditMessage)
{
appendToInput(auditInput, table.getName(), record.getValueInteger(table.getPrimaryKeyField()), getRecordSecurityKeyValues(table, record, Optional.empty()), auditMessage);
}
/*******************************************************************************
** Add 1 auditSingleInput to an AuditInput object - with no details (child records).
*******************************************************************************/
public static void appendToInput(AuditInput auditInput, String tableName, Integer recordId, Map<String, Serializable> securityKeyValues, String message)
{
appendToInput(auditInput, tableName, recordId, securityKeyValues, message, null);
}
/*******************************************************************************
** Add 1 auditSingleInput to an AuditInput object - with a list of details (child records).
*******************************************************************************/
public static AuditInput appendToInput(AuditInput auditInput, String tableName, Integer recordId, Map<String, Serializable> securityKeyValues, String message, List<QRecord> details)
{
if(auditInput == null)
{
auditInput = new AuditInput();
}
return auditInput.withAuditSingleInput(new AuditSingleInput()
.withAuditTableName(tableName)
.withRecordId(recordId)
.withSecurityKeyValues(securityKeyValues)
.withMessage(message)
.withDetails(details)
);
}
/*******************************************************************************
** For a given record, from a given table, build a map of the record's security
** key values.
**
** If, in case, the record has null value(s), and the oldRecord is given (e.g.,
** for the case of an update, where the record may not have all fields set, and
** oldRecord should be known for doing field-diffs), then try to get the value(s)
** from oldRecord.
**
** Currently, will leave values null if they aren't found after that.
**
** An alternative could be to re-fetch the record from its source if needed...
*******************************************************************************/
public static Map<String, Serializable> getRecordSecurityKeyValues(QTableMetaData table, QRecord record, Optional<QRecord> oldRecord)
{
Map<String, Serializable> securityKeyValues = new HashMap<>();
for(RecordSecurityLock recordSecurityLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks())))
{
Serializable keyValue = record == null ? null : record.getValue(recordSecurityLock.getFieldName());
if(keyValue == null && oldRecord.isPresent())
{
LOG.debug("Table with a securityLock, but value not found in field", logPair("table", table.getName()), logPair("field", recordSecurityLock.getFieldName()));
keyValue = oldRecord.get().getValue(recordSecurityLock.getFieldName());
}
if(keyValue == null)
{
LOG.debug("Table with a securityLock, but value not found in field", logPair("table", table.getName()), logPair("field", recordSecurityLock.getFieldName()), logPair("oldRecordIsPresent", oldRecord.isPresent()));
}
securityKeyValues.put(recordSecurityLock.getSecurityKeyType(), keyValue);
}
return securityKeyValues;
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public AuditOutput execute(AuditInput input)
{
AuditOutput auditOutput = new AuditOutput();
if(CollectionUtils.nullSafeHasContents(input.getAuditSingleInputList()))
{
try
{
List<QRecord> auditRecords = new ArrayList<>();
for(AuditSingleInput auditSingleInput : CollectionUtils.nonNullList(input.getAuditSingleInputList()))
{
/////////////////////////////////////////
// validate table is known in instance //
/////////////////////////////////////////
QTableMetaData table = QContext.getQInstance().getTable(auditSingleInput.getAuditTableName());
if(table == null)
{
throw (new QException("Requested audit for an unrecognized table name: " + auditSingleInput.getAuditTableName()));
}
///////////////////////////////////////////////////
// validate security keys on the table are given //
///////////////////////////////////////////////////
for(RecordSecurityLock recordSecurityLock : RecordSecurityLockFilters.filterForReadLocks(CollectionUtils.nonNullList(table.getRecordSecurityLocks())))
{
if(auditSingleInput.getSecurityKeyValues() == null || !auditSingleInput.getSecurityKeyValues().containsKey(recordSecurityLock.getSecurityKeyType()))
{
throw (new QException("Missing securityKeyValue [" + recordSecurityLock.getSecurityKeyType() + "] in audit request for table " + auditSingleInput.getAuditTableName()));
}
}
////////////////////////////////////////////////
// map names to ids and handle default values //
////////////////////////////////////////////////
Integer auditTableId = getIdForName("auditTable", auditSingleInput.getAuditTableName());
Integer auditUserId = getIdForName("auditUser", Objects.requireNonNullElse(auditSingleInput.getAuditUserName(), getSessionUserName()));
Instant timestamp = Objects.requireNonNullElse(auditSingleInput.getTimestamp(), Instant.now());
//////////////////
// build record //
//////////////////
QRecord record = new QRecord()
.withValue("auditTableId", auditTableId)
.withValue("auditUserId", auditUserId)
.withValue("timestamp", timestamp)
.withValue("message", auditSingleInput.getMessage())
.withValue("recordId", auditSingleInput.getRecordId());
if(auditSingleInput.getSecurityKeyValues() != null)
{
for(Map.Entry<String, Serializable> entry : auditSingleInput.getSecurityKeyValues().entrySet())
{
record.setValue(entry.getKey(), entry.getValue());
}
}
auditRecords.add(record);
}
/////////////////////////////
// do a single bulk insert //
/////////////////////////////
InsertInput insertInput = new InsertInput();
insertInput.setTableName("audit");
insertInput.setRecords(auditRecords);
InsertOutput insertOutput = new InsertAction().execute(insertInput);
//////////////////////////////////////////
// now look for children (auditDetails) //
//////////////////////////////////////////
int i = 0;
List<QRecord> auditDetailRecords = new ArrayList<>();
for(AuditSingleInput auditSingleInput : CollectionUtils.nonNullList(input.getAuditSingleInputList()))
{
Integer auditId = insertOutput.getRecords().get(i++).getValueInteger("id");
if(auditId == null)
{
LOG.warn("Missing an id for inserted audit - so won't be able to store its child details...");
continue;
}
for(QRecord detail : CollectionUtils.nonNullList(auditSingleInput.getDetails()))
{
auditDetailRecords.add(detail.withValue("auditId", auditId));
}
}
if(!auditDetailRecords.isEmpty())
{
insertInput = new InsertInput();
insertInput.setTableName("auditDetail");
insertInput.setRecords(auditDetailRecords);
new InsertAction().execute(insertInput);
}
}
catch(Exception e)
{
LOG.error("Error performing an audit", e);
}
}
return (auditOutput);
}
/*******************************************************************************
**
*******************************************************************************/
private static String getSessionUserName()
{
QUser user = QContext.getQSession().getUser();
if(user == null)
{
return ("Unknown");
}
return (user.getFullName());
}
/*******************************************************************************
**
*******************************************************************************/
private Integer getIdForName(String tableName, String nameValue) throws QException
{
Pair<String, String> key = new Pair<>(tableName, nameValue);
if(!cachedFetches.containsKey(key))
{
Integer id = fetchIdFromName(tableName, nameValue);
if(id != null)
{
cachedFetches.put(key, id);
return id;
}
try
{
LOG.debug("Inserting " + tableName + " named " + nameValue);
InsertInput insertInput = new InsertInput();
insertInput.setTableName(tableName);
QRecord record = new QRecord().withValue("name", nameValue);
if(tableName.equals("auditTable"))
{
QTableMetaData table = QContext.getQInstance().getTable(nameValue);
if(table != null)
{
record.setValue("label", table.getLabel());
}
}
insertInput.setRecords(List.of(record));
InsertOutput insertOutput = new InsertAction().execute(insertInput);
id = insertOutput.getRecords().get(0).getValueInteger("id");
if(id != null)
{
cachedFetches.put(key, id);
return id;
}
}
catch(Exception e)
{
////////////////////////////////////////////////////////////////////
// assume this may mean a dupe-key - so - try another fetch below //
////////////////////////////////////////////////////////////////////
LOG.debug("Caught error inserting " + tableName + " named " + nameValue + " - will try to re-fetch", e);
}
id = fetchIdFromName(tableName, nameValue);
if(id != null)
{
cachedFetches.put(key, id);
return id;
}
/////////////
// give up //
/////////////
throw (new QException("Unable to get id for " + tableName + " named " + nameValue));
}
return (cachedFetches.get(key));
}
/*******************************************************************************
**
*******************************************************************************/
private Integer fetchIdFromName(String tableName, String nameValue) throws QException
{
GetInput getInput = new GetInput();
getInput.setTableName(tableName);
getInput.setUniqueKey(Map.of("name", nameValue));
GetOutput getOutput = new GetAction().execute(getInput);
if(getOutput.getRecord() != null)
{
return (getOutput.getRecord().getValueInteger("id"));
}
return (null);
}
}

View File

@ -0,0 +1,576 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.audits;
import java.io.Serializable;
import java.math.BigDecimal;
import java.time.Instant;
import java.time.ZoneId;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import com.kingsrook.qqq.backend.core.actions.AbstractQActionFunction;
import com.kingsrook.qqq.backend.core.actions.values.QPossibleValueTranslator;
import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
import com.kingsrook.qqq.backend.core.model.actions.audits.AuditInput;
import com.kingsrook.qqq.backend.core.model.actions.audits.DMLAuditInput;
import com.kingsrook.qqq.backend.core.model.actions.audits.DMLAuditOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import static com.kingsrook.qqq.backend.core.actions.audits.AuditAction.getRecordSecurityKeyValues;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** Audit for a standard DML (Data Manipulation Language) activity - e.g.,
** insert, edit, or delete.
*******************************************************************************/
public class DMLAuditAction extends AbstractQActionFunction<DMLAuditInput, DMLAuditOutput>
{
private static final QLogger LOG = QLogger.getLogger(DMLAuditAction.class);
public static final String AUDIT_CONTEXT_FIELD_NAME = "auditContext";
private static Set<String> loggedUnauditableTableNames = new HashSet<>();
/*******************************************************************************
**
*******************************************************************************/
@Override
public DMLAuditOutput execute(DMLAuditInput input) throws QException
{
DMLAuditOutput output = new DMLAuditOutput();
AbstractTableActionInput tableActionInput = input.getTableActionInput();
List<QRecord> oldRecordList = input.getOldRecordList();
QTableMetaData table = tableActionInput.getTable();
long start = System.currentTimeMillis();
DMLType dmlType = getDMLType(tableActionInput);
try
{
List<QRecord> recordList = CollectionUtils.nonNullList(input.getRecordList()).stream()
.filter(r -> CollectionUtils.nullSafeIsEmpty(r.getErrors())).toList();
AuditLevel auditLevel = getAuditLevel(tableActionInput);
if(auditLevel == null || auditLevel.equals(AuditLevel.NONE) || CollectionUtils.nullSafeIsEmpty(recordList))
{
/////////////////////////////////////////////
// return with noop for null or level NONE //
/////////////////////////////////////////////
return (output);
}
///////////////////////////////////////////////////////////////////////////////////////////////////////
// currently, the table's primary key must be integer... so, log (once) and return early if not that //
// (or, if no primary key!) //
///////////////////////////////////////////////////////////////////////////////////////////////////////
QFieldMetaData field = table.getFields().get(table.getPrimaryKeyField());
if(field == null || !QFieldType.INTEGER.equals(field.getType()))
{
if(!loggedUnauditableTableNames.contains(table.getName()))
{
LOG.info("Cannot audit table without integer as its primary key", logPair("tableName", table.getName()));
loggedUnauditableTableNames.add(table.getName());
}
return (output);
}
String contextSuffix = getContentSuffix(input);
AuditInput auditInput = new AuditInput();
if(auditLevel.equals(AuditLevel.RECORD) || (auditLevel.equals(AuditLevel.FIELD) && !dmlType.supportsFields))
{
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// make many simple audits (no details) for RECORD level //
// or for FIELD level, but on a DML type that doesn't support field-level details (e.g., DELETE or OTHER) //
////////////////////////////////////////////////////////////////////////////////////////////////////////////
for(QRecord record : recordList)
{
AuditAction.appendToInput(auditInput, table.getName(), record.getValueInteger(table.getPrimaryKeyField()), getRecordSecurityKeyValues(table, record, Optional.empty()), "Record was " + dmlType.pastTenseVerb + contextSuffix);
}
}
else if(auditLevel.equals(AuditLevel.FIELD))
{
Map<Serializable, QRecord> oldRecordMap = buildOldRecordMap(table, oldRecordList);
///////////////////////////////////////////////////////////////////
// do many audits, all with field level details, for FIELD level //
///////////////////////////////////////////////////////////////////
QPossibleValueTranslator qPossibleValueTranslator = new QPossibleValueTranslator(QContext.getQInstance(), QContext.getQSession());
qPossibleValueTranslator.translatePossibleValuesInRecords(table, CollectionUtils.mergeLists(recordList, oldRecordList));
//////////////////////////////////////////
// sort the field names by their labels //
//////////////////////////////////////////
List<String> sortedFieldNames = table.getFields().keySet().stream()
.sorted(Comparator.comparing(fieldName -> table.getFields().get(fieldName).getLabel()))
.toList();
QFieldMetaData primaryKeyField = table.getField(table.getPrimaryKeyField());
//////////////////////////////////////////////
// build single audit input for each record //
//////////////////////////////////////////////
for(QRecord record : recordList)
{
QRecord oldRecord = oldRecordMap.get(ValueUtils.getValueAsFieldType(primaryKeyField.getType(), record.getValue(primaryKeyField.getName())));
List<QRecord> details = new ArrayList<>();
for(String fieldName : sortedFieldNames)
{
makeAuditDetailRecordForField(fieldName, table, dmlType, record, oldRecord)
.ifPresent(details::add);
}
if(details.isEmpty() && DMLType.UPDATE.equals(dmlType))
{
// no, let's just noop.
// details.add(new QRecord().withValue("message", "No fields values were changed."));
}
else
{
AuditAction.appendToInput(auditInput, table.getName(), record.getValueInteger(table.getPrimaryKeyField()), getRecordSecurityKeyValues(table, record, Optional.ofNullable(oldRecord)), "Record was " + dmlType.pastTenseVerb + contextSuffix, details);
}
}
}
// new AuditAction().executeAsync(auditInput); // todo async??? maybe get that from rules???
new AuditAction().execute(auditInput);
long end = System.currentTimeMillis();
LOG.trace("Audit performance", logPair("auditLevel", String.valueOf(auditLevel)), logPair("recordCount", recordList.size()), logPair("millis", (end - start)));
}
catch(Exception e)
{
LOG.error("Error performing DML audit", e, logPair("type", String.valueOf(dmlType)), logPair("table", table.getName()));
}
return (output);
}
/*******************************************************************************
**
*******************************************************************************/
static String getContentSuffix(DMLAuditInput input)
{
StringBuilder contextSuffix = new StringBuilder();
/////////////////////////////////////////////////////////////////////////////
// start with context from the input wrapper //
// note, these contexts get propagated down from Input/Update/Delete Input //
/////////////////////////////////////////////////////////////////////////////
if(StringUtils.hasContent(input.getAuditContext()))
{
contextSuffix.append(" ").append(input.getAuditContext());
}
/////////////////////////////////////////////////////////////////////////////////////
// note process label (and a possible context from the process's state) if present //
/////////////////////////////////////////////////////////////////////////////////////
Optional<AbstractActionInput> actionInput = QContext.getFirstActionInStack();
if(actionInput.isPresent() && actionInput.get() instanceof RunProcessInput runProcessInput)
{
String processAuditContext = ValueUtils.getValueAsString(runProcessInput.getValue(AUDIT_CONTEXT_FIELD_NAME));
if(StringUtils.hasContent(processAuditContext))
{
contextSuffix.append(" ").append(processAuditContext);
}
String processName = runProcessInput.getProcessName();
QProcessMetaData process = QContext.getQInstance().getProcess(processName);
if(process != null)
{
contextSuffix.append(" during process: ").append(process.getLabel());
}
}
///////////////////////////////////////////////////
// use api label & version if present in session //
///////////////////////////////////////////////////
QSession qSession = QContext.getQSession();
String apiVersion = qSession.getValue("apiVersion");
if(apiVersion != null)
{
String apiLabel = qSession.getValue("apiLabel");
if(!StringUtils.hasContent(apiLabel))
{
apiLabel = "API";
}
contextSuffix.append(" via ").append(apiLabel).append(" Version: ").append(apiVersion);
}
return (contextSuffix.toString());
}
/*******************************************************************************
**
*******************************************************************************/
static Optional<QRecord> makeAuditDetailRecordForField(String fieldName, QTableMetaData table, DMLType dmlType, QRecord record, QRecord oldRecord)
{
if(!record.getValues().containsKey(fieldName))
{
////////////////////////////////////////////////////////////////////////////////////////////////
// if the stored record doesn't have this field name, then don't audit anything about it //
// this is to deal with our Patch style updates not looking like every field was cleared out. //
////////////////////////////////////////////////////////////////////////////////////////////////
return (Optional.empty());
}
if(fieldName.equals("modifyDate") || fieldName.equals("createDate") || fieldName.equals("automationStatus"))
{
return (Optional.empty());
}
QFieldMetaData field = table.getField(fieldName);
Serializable value = ValueUtils.getValueAsFieldType(field.getType(), record.getValue(fieldName));
Serializable oldValue = oldRecord == null ? null : ValueUtils.getValueAsFieldType(field.getType(), oldRecord.getValue(fieldName));
QRecord detailRecord = null;
if(oldRecord == null)
{
if(DMLType.INSERT.equals(dmlType) && value == null)
{
return (Optional.empty());
}
if(field.getType().equals(QFieldType.BLOB) || field.getType().needsMasked())
{
detailRecord = new QRecord().withValue("message", "Set " + field.getLabel());
}
else
{
String formattedValue = getFormattedValueForAuditDetail(record, fieldName, field, value);
detailRecord = new QRecord().withValue("message", "Set " + field.getLabel() + " to " + formattedValue);
detailRecord.withValue("newValue", formattedValue);
}
}
else
{
if(areValuesDifferentForAudit(field, value, oldValue))
{
if(field.getType().equals(QFieldType.BLOB) || field.getType().needsMasked())
{
if(oldValue == null)
{
detailRecord = new QRecord().withValue("message", "Set " + field.getLabel());
}
else if(value == null)
{
detailRecord = new QRecord().withValue("message", "Removed " + field.getLabel());
}
else
{
detailRecord = new QRecord().withValue("message", "Changed " + field.getLabel());
}
}
else
{
String formattedValue = getFormattedValueForAuditDetail(record, fieldName, field, value);
String formattedOldValue = getFormattedValueForAuditDetail(oldRecord, fieldName, field, oldValue);
if(oldValue == null)
{
detailRecord = new QRecord().withValue("message", "Set " + field.getLabel() + " to " + formatFormattedValueForDetailMessage(field, formattedValue));
detailRecord.withValue("newValue", formattedValue);
}
else if(value == null)
{
detailRecord = new QRecord().withValue("message", "Removed " + formatFormattedValueForDetailMessage(field, formattedOldValue) + " from " + field.getLabel());
detailRecord.withValue("oldValue", formattedOldValue);
}
else
{
detailRecord = new QRecord().withValue("message", "Changed " + field.getLabel() + " from " + formatFormattedValueForDetailMessage(field, formattedOldValue) + " to " + formatFormattedValueForDetailMessage(field, formattedValue));
detailRecord.withValue("oldValue", formattedOldValue);
detailRecord.withValue("newValue", formattedValue);
}
}
}
}
if(detailRecord != null)
{
////////////////////////////////////////////////////////////////////
// useful if doing dev in here - but overkill for any other time. //
////////////////////////////////////////////////////////////////////
// LOG.debug("Returning with message: " + detailRecord.getValueString("message"));
detailRecord.withValue("fieldName", fieldName);
return (Optional.of(detailRecord));
}
return (Optional.empty());
}
/*******************************************************************************
**
*******************************************************************************/
static boolean areValuesDifferentForAudit(QFieldMetaData field, Serializable value, Serializable oldValue)
{
try
{
///////////////////
// decimal rules //
///////////////////
if(field.getType().equals(QFieldType.DECIMAL))
{
BigDecimal newBD = ValueUtils.getValueAsBigDecimal(value);
BigDecimal oldBD = ValueUtils.getValueAsBigDecimal(oldValue);
if(newBD == null && oldBD == null)
{
return (false);
}
if(newBD == null || oldBD == null)
{
return (true);
}
return (newBD.compareTo(oldBD) != 0);
}
////////////////////
// dateTime rules //
////////////////////
if(field.getType().equals(QFieldType.DATE_TIME))
{
Instant newI = ValueUtils.getValueAsInstant(value);
Instant oldI = ValueUtils.getValueAsInstant(oldValue);
if(newI == null && oldI == null)
{
return (false);
}
if(newI == null || oldI == null)
{
return (true);
}
////////////////////////////////
// just compare to the second //
////////////////////////////////
return (newI.truncatedTo(ChronoUnit.SECONDS).compareTo(oldI.truncatedTo(ChronoUnit.SECONDS)) != 0);
}
//////////////////
// string rules //
//////////////////
if(field.getType().isStringLike())
{
String newString = ValueUtils.getValueAsString(value);
String oldString = ValueUtils.getValueAsString(oldValue);
boolean newIsNullOrEmpty = !StringUtils.hasContent(newString);
boolean oldIsNullOrEmpty = !StringUtils.hasContent(oldString);
if(newIsNullOrEmpty && oldIsNullOrEmpty)
{
return (false);
}
if(newIsNullOrEmpty || oldIsNullOrEmpty)
{
return (true);
}
return (newString.compareTo(oldString) != 0);
}
/////////////////////////////////////
// default just use Objects.equals //
/////////////////////////////////////
return !Objects.equals(oldValue, value);
}
catch(Exception e)
{
LOG.debug("Error checking areValuesDifferentForAudit", e, logPair("fieldName", field.getName()), logPair("value", value), logPair("oldValue", oldValue));
}
////////////////////////////////////
// default to something simple... //
////////////////////////////////////
return !Objects.equals(oldValue, value);
}
/*******************************************************************************
**
*******************************************************************************/
private static String getFormattedValueForAuditDetail(QRecord record, String fieldName, QFieldMetaData field, Serializable value)
{
String formattedValue = null;
if(value != null)
{
if(field.getType().equals(QFieldType.DATE_TIME) && value instanceof Instant instant)
{
formattedValue = QValueFormatter.formatDateTimeWithZone(instant.atZone(ZoneId.of(Objects.requireNonNullElse(QContext.getQInstance().getDefaultTimeZoneId(), "UTC"))));
}
else if(record.getDisplayValue(fieldName) != null)
{
formattedValue = record.getDisplayValue(fieldName);
}
else
{
formattedValue = QValueFormatter.formatValue(field, value);
}
}
return formattedValue;
}
/*******************************************************************************
**
*******************************************************************************/
private static String formatFormattedValueForDetailMessage(QFieldMetaData field, String formattedValue)
{
if(formattedValue == null || "null".equals(formattedValue))
{
formattedValue = "--";
}
else
{
if(QFieldType.STRING.equals(field.getType()) || field.getPossibleValueSourceName() != null)
{
formattedValue = '"' + formattedValue + '"';
}
}
return (formattedValue);
}
/*******************************************************************************
**
*******************************************************************************/
private Map<Serializable, QRecord> buildOldRecordMap(QTableMetaData table, List<QRecord> oldRecordList)
{
Map<Serializable, QRecord> rs = new HashMap<>();
for(QRecord record : CollectionUtils.nonNullList(oldRecordList))
{
rs.put(record.getValue(table.getPrimaryKeyField()), record);
}
return (rs);
}
/*******************************************************************************
**
*******************************************************************************/
private DMLType getDMLType(AbstractTableActionInput tableActionInput)
{
if(tableActionInput instanceof InsertInput)
{
return DMLType.INSERT;
}
else if(tableActionInput instanceof UpdateInput)
{
return DMLType.UPDATE;
}
else if(tableActionInput instanceof DeleteInput)
{
return DMLType.DELETE;
}
else
{
return DMLType.OTHER;
}
}
/*******************************************************************************
**
*******************************************************************************/
public static AuditLevel getAuditLevel(AbstractTableActionInput tableActionInput)
{
QTableMetaData table = tableActionInput.getTable();
if(table.getAuditRules() == null)
{
return (AuditLevel.NONE);
}
return (table.getAuditRules().getAuditLevel());
}
/*******************************************************************************
**
*******************************************************************************/
enum DMLType
{
INSERT("Inserted", true),
UPDATE("Edited", true),
DELETE("Deleted", false),
OTHER("Processed", false);
private final String pastTenseVerb;
private final boolean supportsFields;
/*******************************************************************************
**
*******************************************************************************/
DMLType(String pastTenseVerb, boolean supportsFields)
{
this.pastTenseVerb = pastTenseVerb;
this.supportsFields = supportsFields;
}
}
}

View File

@ -0,0 +1,115 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.automation;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
/*******************************************************************************
** enum of possible values for a record's Automation Status.
*******************************************************************************/
public enum AutomationStatus implements PossibleValueEnum<Integer>
{
PENDING_INSERT_AUTOMATIONS(1, "Pending Insert Automations"),
RUNNING_INSERT_AUTOMATIONS(2, "Running Insert Automations"),
FAILED_INSERT_AUTOMATIONS(3, "Failed Insert Automations"),
PENDING_UPDATE_AUTOMATIONS(4, "Pending Update Automations"),
RUNNING_UPDATE_AUTOMATIONS(5, "Running Update Automations"),
FAILED_UPDATE_AUTOMATIONS(6, "Failed Update Automations"),
OK(7, "OK");
private final Integer id;
private final String label;
/*******************************************************************************
**
*******************************************************************************/
AutomationStatus(int id, String label)
{
this.id = id;
this.label = label;
}
/*******************************************************************************
** Getter for id
**
*******************************************************************************/
public Integer getId()
{
return (id);
}
/*******************************************************************************
** Getter for label
**
*******************************************************************************/
public String getLabel()
{
return (label);
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public Integer getPossibleValueId()
{
return (getId());
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (getLabel());
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("checkstyle:indentation")
public String getInsertOrUpdate()
{
return switch(this)
{
case PENDING_INSERT_AUTOMATIONS, RUNNING_INSERT_AUTOMATIONS, FAILED_INSERT_AUTOMATIONS -> "Insert";
case PENDING_UPDATE_AUTOMATIONS, RUNNING_UPDATE_AUTOMATIONS, FAILED_UPDATE_AUTOMATIONS -> "Update";
case OK -> "";
};
}
}

View File

@ -0,0 +1,40 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.automation;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.automation.RecordAutomationInput;
/*******************************************************************************
** Base class for custom-codes to run as an automation action
*******************************************************************************/
public abstract class RecordAutomationHandler
{
/*******************************************************************************
**
*******************************************************************************/
public abstract void execute(RecordAutomationInput recordAutomationInput) throws QException;
}

View File

@ -0,0 +1,253 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.automation;
import java.util.Collections;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.automation.TableTrigger;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.AutomationStatusTrackingType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.QTableAutomationDetails;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TableAutomationAction;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TriggerEvent;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import org.apache.commons.lang.NotImplementedException;
/*******************************************************************************
** Utility class for updating the automation status data for records
*******************************************************************************/
public class RecordAutomationStatusUpdater
{
private static final QLogger LOG = QLogger.getLogger(RecordAutomationStatusUpdater.class);
/*******************************************************************************
** for a list of records from a table, set their automation status - based on
** how the table is configured.
*******************************************************************************/
public static boolean setAutomationStatusInRecords(QSession session, QTableMetaData table, List<QRecord> records, AutomationStatus automationStatus)
{
if(table == null || table.getAutomationDetails() == null || CollectionUtils.nullSafeIsEmpty(records))
{
return (false);
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// In case an automation is running, and it updates records - don't let those records be marked //
// as PENDING_UPDATE_AUTOMATIONS... this is meant to avoid having a record's automation update //
// itself, and then continue to do so in a loop (infinitely). //
// BUT - shouldn't this be allowed to update OTHER records to be pending updates? It seems like //
// yes - so -that'll probably be a bug to fix at some point in the future todo //
///////////////////////////////////////////////////////////////////////////////////////////////////
if(automationStatus.equals(AutomationStatus.PENDING_UPDATE_AUTOMATIONS))
{
Exception e = new Exception();
for(StackTraceElement stackTraceElement : e.getStackTrace())
{
String className = stackTraceElement.getClassName();
if(className.contains("com.kingsrook.qqq.backend.core.actions.automation") && !className.equals(RecordAutomationStatusUpdater.class.getName()) && !className.endsWith("Test"))
{
LOG.debug("Avoiding re-setting automation status to PENDING_UPDATE while running an automation");
return (false);
}
}
}
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Avoid setting records to PENDING_INSERT or PENDING_UPDATE even if they don't have any insert or update automations or triggers //
// such records should go straight to OK status. //
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
if(canWeSkipPendingAndGoToOkay(table, automationStatus))
{
automationStatus = AutomationStatus.OK;
}
QTableAutomationDetails automationDetails = table.getAutomationDetails();
if(automationDetails.getStatusTracking() != null && AutomationStatusTrackingType.FIELD_IN_TABLE.equals(automationDetails.getStatusTracking().getType()))
{
for(QRecord record : records)
{
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// todo - seems like there's some case here, where if an order was in PENDING_INSERT, but then some other job updated the record, that we'd //
// lose that pending status, which would be a Bad Thing™... //
// problem is - we may not have the full record in here, so we can't necessarily check the record to see what status it's currently in... //
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
record.setValue(automationDetails.getStatusTracking().getFieldName(), automationStatus.getId());
// todo - another field - for the automation timestamp??
}
}
return (true);
}
/*******************************************************************************
** If a table has no automation actions defined for Insert (or Update), and we're
** being asked to set status to PENDING_INSERT (or PENDING_UPDATE), then just
** move the status straight to OK.
*******************************************************************************/
static boolean canWeSkipPendingAndGoToOkay(QTableMetaData table, AutomationStatus automationStatus)
{
List<TableAutomationAction> tableActions = Collections.emptyList();
if(table.getAutomationDetails() != null && table.getAutomationDetails().getActions() != null)
{
tableActions = table.getAutomationDetails().getActions();
}
if(automationStatus.equals(AutomationStatus.PENDING_INSERT_AUTOMATIONS))
{
if(tableActions.stream().anyMatch(a -> TriggerEvent.POST_INSERT.equals(a.getTriggerEvent())))
{
return (false);
}
else if(areThereTableTriggersForTable(table, TriggerEvent.POST_INSERT))
{
return (false);
}
////////////////////////////////////////////////////////////////////////////////////////
// if we're going to pending-insert, and there are no insert automations or triggers, //
// then we may skip pending and go to okay. //
////////////////////////////////////////////////////////////////////////////////////////
return (true);
}
else if(automationStatus.equals(AutomationStatus.PENDING_UPDATE_AUTOMATIONS))
{
if(tableActions.stream().anyMatch(a -> TriggerEvent.POST_UPDATE.equals(a.getTriggerEvent())))
{
return (false);
}
else if(areThereTableTriggersForTable(table, TriggerEvent.POST_UPDATE))
{
return (false);
}
////////////////////////////////////////////////////////////////////////////////////////
// if we're going to pending-update, and there are no insert automations or triggers, //
// then we may skip pending and go to okay. //
////////////////////////////////////////////////////////////////////////////////////////
return (true);
}
else
{
///////////////////////////////////////////////////////////////////////////////////////////////////////
// if we're going to any other automation status - then we may never "skip pending" and go to okay - //
// because we weren't asked to go to pending! //
///////////////////////////////////////////////////////////////////////////////////////////////////////
return (false);
}
}
/*******************************************************************************
**
*******************************************************************************/
private static boolean areThereTableTriggersForTable(QTableMetaData table, TriggerEvent triggerEvent)
{
if(QContext.getQInstance().getTable(TableTrigger.TABLE_NAME) == null)
{
return (false);
}
try
{
///////////////////
// todo - cache? //
///////////////////
CountInput countInput = new CountInput();
countInput.setTableName(TableTrigger.TABLE_NAME);
countInput.setFilter(new QQueryFilter(
new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()),
new QFilterCriteria(triggerEvent.equals(TriggerEvent.POST_INSERT) ? "postInsert" : "postUpdate", QCriteriaOperator.EQUALS, true)
));
CountOutput countOutput = new CountAction().execute(countInput);
return (countOutput.getCount() != null && countOutput.getCount() > 0);
}
catch(Exception e)
{
///////////////////////////////////////////////////////////////////////////////////////////////////////////
// if the count query failed, we're a bit safer to err on the side of "yeah, there might be automations" //
///////////////////////////////////////////////////////////////////////////////////////////////////////////
return (true);
}
}
/*******************************************************************************
** for a list of records, update their automation status and actually Update the
** backend as well.
*******************************************************************************/
public static void setAutomationStatusInRecordsAndUpdate(QInstance instance, QSession session, QTableMetaData table, List<QRecord> records, AutomationStatus automationStatus) throws QException
{
QTableAutomationDetails automationDetails = table.getAutomationDetails();
if(automationDetails != null && AutomationStatusTrackingType.FIELD_IN_TABLE.equals(automationDetails.getStatusTracking().getType()))
{
boolean didSetStatusField = setAutomationStatusInRecords(session, table, records, automationStatus);
if(didSetStatusField)
{
UpdateInput updateInput = new UpdateInput();
updateInput.setTableName(table.getName());
/////////////////////////////////////////////////////////////////////////////////////
// build records with just their pkey & status field for this update, to avoid //
// changing other values (relies on assumption of Patch semantics in UpdateAction) //
/////////////////////////////////////////////////////////////////////////////////////
updateInput.setRecords(records.stream().map(r -> new QRecord()
.withTableName(r.getTableName())
.withValue(table.getPrimaryKeyField(), r.getValue(table.getPrimaryKeyField()))
.withValue(automationDetails.getStatusTracking().getFieldName(), r.getValue(automationDetails.getStatusTracking().getFieldName()))).toList());
updateInput.setAreAllValuesBeingUpdatedTheSame(true);
updateInput.setOmitDmlAudit(true);
new UpdateAction().execute(updateInput);
}
}
else
{
// todo - verify if this is valid as other types are built
throw (new NotImplementedException("Updating record automation status is not implemented for table [" + table + "], tracking type: "
+ (automationDetails == null ? "null" : automationDetails.getStatusTracking().getType())));
}
}
}

View File

@ -0,0 +1,96 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.automation;
import java.io.Serializable;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.scripts.RunAdHocRecordScriptAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.scripts.RunAdHocRecordScriptInput;
import com.kingsrook.qqq.backend.core.model.actions.scripts.RunAdHocRecordScriptOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
import com.kingsrook.qqq.backend.core.model.automation.RecordAutomationInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.AdHocScriptCodeReference;
import com.kingsrook.qqq.backend.core.model.scripts.Script;
import com.kingsrook.qqq.backend.core.model.scripts.ScriptRevision;
import com.kingsrook.qqq.backend.core.model.scripts.ScriptsMetaDataProvider;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
**
*******************************************************************************/
public class RunRecordScriptAutomationHandler extends RecordAutomationHandler
{
private static final QLogger LOG = QLogger.getLogger(RunRecordScriptAutomationHandler.class);
/*******************************************************************************
**
*******************************************************************************/
@Override
public void execute(RecordAutomationInput recordAutomationInput) throws QException
{
String tableName = recordAutomationInput.getTableName();
Map<String, Serializable> values = recordAutomationInput.getAction().getValues();
Integer scriptId = ValueUtils.getValueAsInteger(values.get("scriptId"));
if(scriptId == null)
{
throw (new QException("ScriptId was not provided in values map for record automations on table: " + tableName));
}
QueryInput queryInput = new QueryInput();
queryInput.setTableName(ScriptRevision.TABLE_NAME);
queryInput.setFilter(new QQueryFilter(new QFilterCriteria("scriptId", QCriteriaOperator.EQUALS, scriptId)));
queryInput.withQueryJoin(new QueryJoin(Script.TABLE_NAME).withBaseTableOrAlias(ScriptRevision.TABLE_NAME).withJoinMetaData(QContext.getQInstance().getJoin(ScriptsMetaDataProvider.CURRENT_SCRIPT_REVISION_JOIN_NAME)));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
if(CollectionUtils.nullSafeIsEmpty(queryOutput.getRecords()))
{
throw (new QException("Could not find current revision for scriptId: " + scriptId + " on table " + tableName));
}
QRecord scriptRevision = queryOutput.getRecords().get(0);
LOG.info("Running script against records", logPair("scriptRevisionId", scriptRevision.getValue("id")), logPair("scriptId", scriptRevision.getValue("scriptIdd")));
RunAdHocRecordScriptInput input = new RunAdHocRecordScriptInput();
input.setCodeReference(new AdHocScriptCodeReference().withScriptRevisionRecord(scriptRevision));
input.setTableName(tableName);
input.setRecordList(recordAutomationInput.getRecordList());
RunAdHocRecordScriptOutput output = new RunAdHocRecordScriptOutput();
new RunAdHocRecordScriptAction().run(input, output);
}
}

View File

@ -0,0 +1,621 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.automation.polling;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.function.Supplier;
import java.util.stream.Collectors;
import com.kingsrook.qqq.backend.core.actions.async.AsyncRecordPipeLoop;
import com.kingsrook.qqq.backend.core.actions.automation.AutomationStatus;
import com.kingsrook.qqq.backend.core.actions.automation.RecordAutomationHandler;
import com.kingsrook.qqq.backend.core.actions.automation.RecordAutomationStatusUpdater;
import com.kingsrook.qqq.backend.core.actions.automation.RunRecordScriptAutomationHandler;
import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallback;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.LogPair;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
import com.kingsrook.qqq.backend.core.model.automation.RecordAutomationInput;
import com.kingsrook.qqq.backend.core.model.automation.TableTrigger;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.AutomationStatusTrackingType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.QTableAutomationDetails;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TableAutomationAction;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TriggerEvent;
import com.kingsrook.qqq.backend.core.model.savedfilters.SavedFilter;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
import org.apache.commons.lang.NotImplementedException;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** Runnable for the Polling Automation Provider, that looks for records that
** need automations, and executes them.
**
** An instance of this class should be created for each table/automation-status
** - see the TableActions inner record for that definition, and the static
** getTableActions method that helps someone who wants to start these threads
** figure out which ones are needed.
*******************************************************************************/
public class PollingAutomationPerTableRunner implements Runnable
{
private static final QLogger LOG = QLogger.getLogger(PollingAutomationPerTableRunner.class);
private final TableActionsInterface tableActions;
private String name;
private QInstance instance;
private Supplier<QSession> sessionSupplier;
private static Map<TriggerEvent, AutomationStatus> triggerEventAutomationStatusMap = Map.of(
TriggerEvent.POST_INSERT, AutomationStatus.PENDING_INSERT_AUTOMATIONS,
TriggerEvent.POST_UPDATE, AutomationStatus.PENDING_UPDATE_AUTOMATIONS
);
private static Map<AutomationStatus, TriggerEvent> automationStatusTriggerEventMap = Map.of(
AutomationStatus.PENDING_INSERT_AUTOMATIONS, TriggerEvent.POST_INSERT,
AutomationStatus.PENDING_UPDATE_AUTOMATIONS, TriggerEvent.POST_UPDATE
);
private static Map<AutomationStatus, AutomationStatus> pendingToRunningStatusMap = Map.of(
AutomationStatus.PENDING_INSERT_AUTOMATIONS, AutomationStatus.RUNNING_INSERT_AUTOMATIONS,
AutomationStatus.PENDING_UPDATE_AUTOMATIONS, AutomationStatus.RUNNING_UPDATE_AUTOMATIONS
);
private static Map<AutomationStatus, AutomationStatus> pendingToFailedStatusMap = Map.of(
AutomationStatus.PENDING_INSERT_AUTOMATIONS, AutomationStatus.FAILED_INSERT_AUTOMATIONS,
AutomationStatus.PENDING_UPDATE_AUTOMATIONS, AutomationStatus.FAILED_UPDATE_AUTOMATIONS
);
/*******************************************************************************
** Interface to be used by 2 records in this class - normal TableActions, and
** ShardedTableActions.
*******************************************************************************/
public interface TableActionsInterface
{
/*******************************************************************************
**
*******************************************************************************/
String tableName();
/*******************************************************************************
**
*******************************************************************************/
AutomationStatus status();
}
/*******************************************************************************
** Wrapper for a pair of (tableName, automationStatus)
*******************************************************************************/
public record TableActions(String tableName, AutomationStatus status) implements TableActionsInterface
{
/*******************************************************************************
**
*******************************************************************************/
public void noopToFakeTestCoverage()
{
}
}
/*******************************************************************************
** extended version of TableAction, for sharding use-case - adds the shard
** details.
*******************************************************************************/
public record ShardedTableActions(String tableName, AutomationStatus status, String shardByFieldName, Serializable shardValue, String shardLabel) implements TableActionsInterface
{
/*******************************************************************************
**
*******************************************************************************/
public void noopToFakeTestCoverage()
{
}
}
/*******************************************************************************
** basically just get a list of tables which at least *could* have automations
** run - either meta-data automations, or table-triggers (data/user defined).
*******************************************************************************/
public static List<TableActionsInterface> getTableActions(QInstance instance, String providerName)
{
List<TableActionsInterface> tableActionList = new ArrayList<>();
for(QTableMetaData table : instance.getTables().values())
{
QTableAutomationDetails automationDetails = table.getAutomationDetails();
if(automationDetails != null && providerName.equals(automationDetails.getProviderName()))
{
if(StringUtils.hasContent(automationDetails.getShardByFieldName()))
{
//////////////////////////////////////////////////////////////////////////////////////////////
// for sharded automations, add a tableAction (of the sharded subtype) for each shard-value //
//////////////////////////////////////////////////////////////////////////////////////////////
try
{
QueryInput queryInput = new QueryInput();
queryInput.setTableName(automationDetails.getShardSourceTableName());
QueryOutput queryOutput = new QueryAction().execute(queryInput);
for(QRecord record : queryOutput.getRecords())
{
Serializable shardId = record.getValue(automationDetails.getShardIdFieldName());
String label = record.getValueString(automationDetails.getShardLabelFieldName());
tableActionList.add(new ShardedTableActions(table.getName(), AutomationStatus.PENDING_INSERT_AUTOMATIONS, automationDetails.getShardByFieldName(), shardId, label));
tableActionList.add(new ShardedTableActions(table.getName(), AutomationStatus.PENDING_UPDATE_AUTOMATIONS, automationDetails.getShardByFieldName(), shardId, label));
}
}
catch(Exception e)
{
LOG.error("Error getting sharded table automation actions for a table", e, new LogPair("tableName", table.getName()));
}
}
else
{
///////////////////////////////////////////////////////////////////
// for non-sharded, we just need tabler name & automation status //
///////////////////////////////////////////////////////////////////
tableActionList.add(new TableActions(table.getName(), AutomationStatus.PENDING_INSERT_AUTOMATIONS));
tableActionList.add(new TableActions(table.getName(), AutomationStatus.PENDING_UPDATE_AUTOMATIONS));
}
}
}
return (tableActionList);
}
/*******************************************************************************
**
*******************************************************************************/
public PollingAutomationPerTableRunner(QInstance instance, String providerName, Supplier<QSession> sessionSupplier, TableActionsInterface tableActions)
{
this.instance = instance;
this.sessionSupplier = sessionSupplier;
this.tableActions = tableActions;
this.name = providerName + ">" + tableActions.tableName() + ">" + tableActions.status().getInsertOrUpdate();
if(tableActions instanceof ShardedTableActions shardedTableActions)
{
this.name += ":" + shardedTableActions.shardLabel();
}
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public void run()
{
QContext.init(instance, sessionSupplier.get());
String originalThreadName = Thread.currentThread().getName();
Thread.currentThread().setName(name);
LOG.debug("Running " + this.getClass().getSimpleName() + "[" + name + "]");
try
{
QSession session = sessionSupplier != null ? sessionSupplier.get() : new QSession();
processTableInsertOrUpdate(instance.getTable(tableActions.tableName()), session, tableActions.status());
}
catch(Exception e)
{
LOG.warn("Error running automations", e);
}
finally
{
Thread.currentThread().setName(originalThreadName);
QContext.clear();
}
}
/*******************************************************************************
** Query for and process records that have a PENDING_INSERT or PENDING_UPDATE status on a given table.
*******************************************************************************/
public void processTableInsertOrUpdate(QTableMetaData table, QSession session, AutomationStatus automationStatus) throws QException
{
/////////////////////////////////////////////////////////////////////////
// get the actions to run against this table in this automation status //
/////////////////////////////////////////////////////////////////////////
List<TableAutomationAction> actions = getTableActions(table, automationStatus);
if(CollectionUtils.nullSafeIsEmpty(actions))
{
return;
}
LOG.debug(" Query for records " + automationStatus + " in " + table);
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// run an async-pipe loop - that will query for records in PENDING - put them in a pipe - then apply actions to them //
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
QTableAutomationDetails automationDetails = table.getAutomationDetails();
AsyncRecordPipeLoop asyncRecordPipeLoop = new AsyncRecordPipeLoop();
RecordPipe recordPipe = automationDetails.getOverrideBatchSize() == null
? new RecordPipe() : new RecordPipe(automationDetails.getOverrideBatchSize());
asyncRecordPipeLoop.run("PollingAutomationRunner>Query>" + automationStatus + ">" + table.getName(), null, recordPipe, (status) ->
{
QueryInput queryInput = new QueryInput();
queryInput.setTableName(table.getName());
AutomationStatusTrackingType statusTrackingType = automationDetails.getStatusTracking().getType();
if(AutomationStatusTrackingType.FIELD_IN_TABLE.equals(statusTrackingType))
{
queryInput.setFilter(new QQueryFilter().withCriteria(new QFilterCriteria(automationDetails.getStatusTracking().getFieldName(), QCriteriaOperator.EQUALS, List.of(automationStatus.getId()))));
}
else
{
throw (new NotImplementedException("Automation Status Tracking type [" + statusTrackingType + "] is not yet implemented in here."));
}
if(tableActions instanceof ShardedTableActions shardedTableActions)
{
//////////////////////////////////////////////////////////////
// for sharded actions, add the shardBy field as a criteria //
//////////////////////////////////////////////////////////////
QQueryFilter filter = queryInput.getFilter();
filter.addCriteria(new QFilterCriteria(shardedTableActions.shardByFieldName(), QCriteriaOperator.EQUALS, shardedTableActions.shardValue()));
}
queryInput.setRecordPipe(recordPipe);
return (new QueryAction().execute(queryInput));
}, () ->
{
List<QRecord> records = recordPipe.consumeAvailableRecords();
applyActionsToRecords(session, table, records, actions, automationStatus);
return (records.size());
}
);
}
/*******************************************************************************
** get the actions to run against a table in an automation status. both from
** metaData and tableTriggers/data.
*******************************************************************************/
private List<TableAutomationAction> getTableActions(QTableMetaData table, AutomationStatus automationStatus) throws QException
{
List<TableAutomationAction> rs = new ArrayList<>();
TriggerEvent triggerEvent = automationStatusTriggerEventMap.get(automationStatus);
///////////////////////////////////////////////////////////
// start with any actions defined in the table meta data //
///////////////////////////////////////////////////////////
for(TableAutomationAction action : table.getAutomationDetails().getActions())
{
if(action.getTriggerEvent().equals(triggerEvent))
{
///////////////////////////////////////////////////////////
// for sharded configs, only run if the shard id matches //
///////////////////////////////////////////////////////////
if(tableActions instanceof ShardedTableActions shardedTableActions)
{
if(shardedTableActions.shardValue().equals(action.getShardId()))
{
rs.add(action);
}
}
else
{
////////////////////////////////////////////
// for non-sharded, always add the action //
////////////////////////////////////////////
rs.add(action);
}
}
}
/////////////////////////////////////////////////
// next add any tableTriggers, defined in data //
/////////////////////////////////////////////////
if(QContext.getQInstance().getTable(TableTrigger.TABLE_NAME) != null)
{
QueryInput queryInput = new QueryInput();
queryInput.setTableName(TableTrigger.TABLE_NAME);
queryInput.setFilter(new QQueryFilter(
new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()),
new QFilterCriteria(triggerEvent.equals(TriggerEvent.POST_INSERT) ? "postInsert" : "postUpdate", QCriteriaOperator.EQUALS, true)
));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
for(QRecord record : queryOutput.getRecords())
{
TableTrigger tableTrigger = new TableTrigger(record);
try
{
QQueryFilter filter = null;
Integer filterId = tableTrigger.getFilterId();
if(filterId != null)
{
GetInput getInput = new GetInput();
getInput.setTableName(SavedFilter.TABLE_NAME);
getInput.setPrimaryKey(filterId);
GetOutput getOutput = new GetAction().execute(getInput);
if(getOutput.getRecord() != null)
{
SavedFilter savedFilter = new SavedFilter(getOutput.getRecord());
filter = JsonUtils.toObject(savedFilter.getFilterJson(), QQueryFilter.class);
}
}
rs.add(new TableAutomationAction()
.withName("Script:" + tableTrigger.getScriptId())
.withFilter(filter)
.withTriggerEvent(triggerEvent)
.withPriority(tableTrigger.getPriority())
.withCodeReference(new QCodeReference(RunRecordScriptAutomationHandler.class))
.withValues(MapBuilder.of("scriptId", tableTrigger.getScriptId()))
.withIncludeRecordAssociations(true)
);
}
catch(Exception e)
{
LOG.error("Error setting up table trigger", e, logPair("tableTriggerId", tableTrigger.getId()));
}
}
}
rs.sort(Comparator.comparing(taa -> Objects.requireNonNullElse(taa.getPriority(), Integer.MAX_VALUE)));
return (rs);
}
/*******************************************************************************
** For a set of records that were found to be in a PENDING state - run all the
** table's actions against them - IF they are found to match the action's filter
** (assuming it has one - if it doesn't, then all records match).
*******************************************************************************/
private void applyActionsToRecords(QSession session, QTableMetaData table, List<QRecord> records, List<TableAutomationAction> actions, AutomationStatus automationStatus) throws QException
{
if(CollectionUtils.nullSafeIsEmpty(records))
{
return;
}
///////////////////////////////////////////////////
// mark the records as RUNNING their automations //
///////////////////////////////////////////////////
RecordAutomationStatusUpdater.setAutomationStatusInRecordsAndUpdate(instance, session, table, records, pendingToRunningStatusMap.get(automationStatus));
///////////////////////////////////////////////////////////////////////////////////////////////////////////////
// foreach action - run it against the records (but only if they match the action's filter, if there is one) //
///////////////////////////////////////////////////////////////////////////////////////////////////////////////
boolean anyActionsFailed = false;
for(TableAutomationAction action : actions)
{
boolean hadError = applyActionToRecords(table, records, action);
if(hadError)
{
anyActionsFailed = true;
}
}
////////////////////////////////////////
// update status on all these records //
////////////////////////////////////////
if(anyActionsFailed)
{
RecordAutomationStatusUpdater.setAutomationStatusInRecordsAndUpdate(instance, session, table, records, pendingToFailedStatusMap.get(automationStatus));
}
else
{
RecordAutomationStatusUpdater.setAutomationStatusInRecordsAndUpdate(instance, session, table, records, AutomationStatus.OK);
}
}
/*******************************************************************************
** Run one action over a list of records (if they match the action's filter).
**
** @return hadError - true if an exception was caught; false if all OK.
*******************************************************************************/
protected boolean applyActionToRecords(QTableMetaData table, List<QRecord> records, TableAutomationAction action)
{
try
{
//////////////////////////////////////////////////////////////////////////////////////////////////////////////
// note - this method - will re-query the objects, so we should have confidence that their data is fresh... //
//////////////////////////////////////////////////////////////////////////////////////////////////////////////
List<QRecord> matchingQRecords = getRecordsMatchingActionFilter(table, records, action);
LOG.debug("Of the {} records that were pending automations, {} of them match the filter on the action {}", records.size(), matchingQRecords.size(), action);
if(CollectionUtils.nullSafeHasContents(matchingQRecords))
{
LOG.debug(" Processing " + matchingQRecords.size() + " records in " + table + " for action " + action);
applyActionToMatchingRecords(table, matchingQRecords, action);
}
return (false);
}
catch(Exception e)
{
LOG.warn("Caught exception processing records on " + table + " for action " + action, e);
return (true);
}
}
/*******************************************************************************
** For a given action, and a list of records - return a new list, of the ones
** which match the action's filter (if there is one - if not, then all match).
**
** Note that this WILL re-query the objects (ALWAYS - even if the action has no filter).
** This has the nice side effect of always giving fresh/updated records, despite having
** some cost.
**
** At one point, we considered just applying the filter using java-comparisons,
** but that will almost certainly give potentially different results than a true
** backend - e.g., just consider if the DB is case-sensitive for strings...
*******************************************************************************/
private List<QRecord> getRecordsMatchingActionFilter(QTableMetaData table, List<QRecord> records, TableAutomationAction action) throws QException
{
QueryInput queryInput = new QueryInput();
queryInput.setTableName(table.getName());
///////////////////////////////////////////////////////////////////////////////////////
// set up a filter that is for the primary keys IN the list that we identified above //
///////////////////////////////////////////////////////////////////////////////////////
QQueryFilter filter = new QQueryFilter();
filter.addCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.IN, records.stream().map(r -> r.getValue(table.getPrimaryKeyField())).toList()));
if(action.getFilter() != null)
{
/////////////////////////////////////////////////////////////////////////////////////////////////////
// if the action defines a filter of its own, add that to the filter we'll run now as a sub-filter //
// not entirely clear if this needs to be a clone, but, it feels safe and cheap enough //
/////////////////////////////////////////////////////////////////////////////////////////////////////
filter.addSubFilter(action.getFilter().clone());
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// we also want to set order-bys from the action into our filter (since they only apply at the top-level) //
////////////////////////////////////////////////////////////////////////////////////////////////////////////
if(action.getFilter().getOrderBys() != null)
{
action.getFilter().getOrderBys().forEach(filter::addOrderBy);
}
}
/////////////////////////////////////////////////////////////////////////////////////////////
// always add order-by the primary key, to give more predictable/consistent results //
// todo - in future - if this becomes a source of slowness, make this a config to opt-out? //
/////////////////////////////////////////////////////////////////////////////////////////////
filter.addOrderBy(new QFilterOrderBy().withFieldName(table.getPrimaryKeyField()));
queryInput.setFilter(filter);
queryInput.setIncludeAssociations(action.getIncludeRecordAssociations());
return (new QueryAction().execute(queryInput).getRecords());
}
/*******************************************************************************
** Finally, actually run action code against a list of known matching records.
** todo not commit - move to somewhere genericer
*******************************************************************************/
public static void applyActionToMatchingRecords(QTableMetaData table, List<QRecord> records, TableAutomationAction action) throws Exception
{
if(StringUtils.hasContent(action.getProcessName()))
{
/////////////////////////////////////////////////////////////////////////////////////////
// if the action has a process associated with it - run that process. //
// tell it to SKIP frontend steps. //
// give the process a callback w/ a query filter that has the p-keys of these records. //
/////////////////////////////////////////////////////////////////////////////////////////
RunProcessInput runProcessInput = new RunProcessInput();
runProcessInput.setProcessName(action.getProcessName());
runProcessInput.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
runProcessInput.setCallback(new QProcessCallback()
{
@Override
public QQueryFilter getQueryFilter()
{
List<Serializable> recordIds = records.stream().map(r -> r.getValue(table.getPrimaryKeyField())).collect(Collectors.toList());
return (new QQueryFilter().withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.IN, recordIds)));
}
});
try
{
QContext.pushAction(runProcessInput);
RunProcessAction runProcessAction = new RunProcessAction();
RunProcessOutput runProcessOutput = runProcessAction.execute(runProcessInput);
if(runProcessOutput.getException().isPresent())
{
throw (runProcessOutput.getException().get());
}
}
finally
{
QContext.popAction();
}
}
else if(action.getCodeReference() != null)
{
LOG.debug(" Executing action: [" + action.getName() + "] as code reference: " + action.getCodeReference());
RecordAutomationInput input = new RecordAutomationInput();
input.setTableName(table.getName());
input.setRecordList(records);
input.setAction(action);
RecordAutomationHandler recordAutomationHandler = QCodeLoader.getRecordAutomationHandler(action);
recordAutomationHandler.execute(input);
}
}
/*******************************************************************************
** Getter for name
**
*******************************************************************************/
public String getName()
{
return name;
}
}

View File

@ -0,0 +1,82 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Abstract class that a table can specify an implementation of, to provide
** custom actions after a delete takes place.
**
** General implementation would be, to iterate over the records (ones which didn't
** have a delete error), and look at their values:
** - possibly adding Errors (`addError`) or Warnings (`addWarning`) to the records?
** - possibly throwing an exception - though doing so won't stop the delete, and instead
** will just set a warning on all of the deleted records...
** - doing "whatever else" you may want to do.
** - returning the list of records (can be the input list) that you want to go back
** to the caller - this is how errors and warnings are propagated .
**
** Note that the full deleteInput is available as a field in this class.
**
** A future enhancement here may be to take (as fields in this class) the list of
** records that the delete action marked in error - the user might want to do
** something special with them (idk, try some other way to delete them?)
*******************************************************************************/
public abstract class AbstractPostDeleteCustomizer
{
protected DeleteInput deleteInput;
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records) throws QException;
/*******************************************************************************
** Getter for deleteInput
**
*******************************************************************************/
public DeleteInput getDeleteInput()
{
return deleteInput;
}
/*******************************************************************************
** Setter for deleteInput
**
*******************************************************************************/
public void setDeleteInput(DeleteInput deleteInput)
{
this.deleteInput = deleteInput;
}
}

View File

@ -0,0 +1,77 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Abstract class that a table can specify an implementation of, to provide
** custom actions after an insert takes place.
**
** General implementation would be, to iterate over the records (the outputs of
** the insert action), and look at their values:
** - possibly adding Errors (`addError`) or Warnings (`addWarning`) to the records
** - possibly throwing an exception - though doing so won't stop the update, and instead
** will just set a warning on all of the updated records...
** - doing "whatever else" you may want to do.
** - returning the list of records (can be the input list) that you want to go back to the caller.
**
** Note that the full insertInput is available as a field in this class.
*******************************************************************************/
public abstract class AbstractPostInsertCustomizer
{
protected InsertInput insertInput;
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records) throws QException;
/*******************************************************************************
** Getter for insertInput
**
*******************************************************************************/
public InsertInput getInsertInput()
{
return insertInput;
}
/*******************************************************************************
** Setter for insertInput
**
*******************************************************************************/
public void setInsertInput(InsertInput insertInput)
{
this.insertInput = insertInput;
}
}

View File

@ -0,0 +1,65 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.List;
import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
**
*******************************************************************************/
public abstract class AbstractPostQueryCustomizer
{
protected AbstractTableActionInput input;
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records);
/*******************************************************************************
** Getter for input
**
*******************************************************************************/
public AbstractTableActionInput getInput()
{
return (input);
}
/*******************************************************************************
** Setter for input
**
*******************************************************************************/
public void setInput(AbstractTableActionInput input)
{
this.input = input;
}
}

View File

@ -0,0 +1,130 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Abstract class that a table can specify an implementation of, to provide
** custom actions after an update takes place.
**
** General implementation would be, to iterate over the records (the outputs of
** the update action), and look at their values:
** - possibly adding Errors (`addError`) or Warnings (`addWarning`) to the records?
** - possibly throwing an exception - though doing so won't stop the update, and instead
** will just set a warning on all of the updated records...
** - doing "whatever else" you may want to do.
** - returning the list of records (can be the input list) that you want to go back to the caller.
**
** Note that the full updateInput is available as a field in this class, and the
** "old records" (e.g., with values freshly fetched from the backend) will be
** available (if the backend supports it) - both as a list (`getOldRecordList`)
** and as a memoized (by this class) map of primaryKey to record (`getOldRecordMap`).
*******************************************************************************/
public abstract class AbstractPostUpdateCustomizer
{
protected UpdateInput updateInput;
protected List<QRecord> oldRecordList;
private Map<Serializable, QRecord> oldRecordMap = null;
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records) throws QException;
/*******************************************************************************
** Getter for updateInput
**
*******************************************************************************/
public UpdateInput getUpdateInput()
{
return updateInput;
}
/*******************************************************************************
** Setter for updateInput
**
*******************************************************************************/
public void setUpdateInput(UpdateInput updateInput)
{
this.updateInput = updateInput;
}
/*******************************************************************************
**
*******************************************************************************/
public void setOldRecordList(List<QRecord> oldRecordList)
{
this.oldRecordList = oldRecordList;
}
/*******************************************************************************
**
*******************************************************************************/
public List<QRecord> getOldRecordList()
{
return oldRecordList;
}
/*******************************************************************************
**
*******************************************************************************/
protected Map<Serializable, QRecord> getOldRecordMap()
{
if(oldRecordMap == null)
{
oldRecordMap = new HashMap<>();
if(oldRecordList != null && updateInput != null)
{
for(QRecord qRecord : oldRecordList)
{
oldRecordMap.put(qRecord.getValue(updateInput.getTable().getPrimaryKeyField()), qRecord);
}
}
}
return (oldRecordMap);
}
}

View File

@ -0,0 +1,119 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Abstract class that a table can specify an implementation of, to provide
** custom actions before a delete takes place.
**
** It's important for implementations to be aware of the isPreview field, which
** is set to true when the code is running to give users advice, e.g., on a review
** screen - vs. being false when the action is ACTUALLY happening. So, if you're doing
** things like storing data, you don't want to do that if isPreview is true!!
**
** General implementation would be, to iterate over the records (which the DeleteAction
** would look up based on the inputs to the delete action), and look at their values:
** - possibly adding Errors (`addError`) or Warnings (`addWarning`) to the records
** - possibly throwing an exception - if you really don't want the delete operation to continue.
** - doing "whatever else" you may want to do.
** - returning the list of records (can be the input list) - this is how errors
** and warnings are propagated to the DeleteAction. Note that any records with
** an error will NOT proceed to the backend's delete interface - but those with
** warnings will.
**
** Note that the full deleteInput is available as a field in this class.
**
*******************************************************************************/
public abstract class AbstractPreDeleteCustomizer
{
protected DeleteInput deleteInput;
protected boolean isPreview = false;
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records) throws QException;
/*******************************************************************************
** Getter for deleteInput
**
*******************************************************************************/
public DeleteInput getDeleteInput()
{
return deleteInput;
}
/*******************************************************************************
** Setter for deleteInput
**
*******************************************************************************/
public void setDeleteInput(DeleteInput deleteInput)
{
this.deleteInput = deleteInput;
}
/*******************************************************************************
** Getter for isPreview
*******************************************************************************/
public boolean getIsPreview()
{
return (this.isPreview);
}
/*******************************************************************************
** Setter for isPreview
*******************************************************************************/
public void setIsPreview(boolean isPreview)
{
this.isPreview = isPreview;
}
/*******************************************************************************
** Fluent setter for isPreview
*******************************************************************************/
public AbstractPreDeleteCustomizer withIsPreview(boolean isPreview)
{
this.isPreview = isPreview;
return (this);
}
}

View File

@ -0,0 +1,141 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Abstract class that a table can specify an implementation of, to provide
** custom actions before an insert takes place.
**
** It's important for implementations to be aware of the isPreview field, which
** is set to true when the code is running to give users advice, e.g., on a review
** screen - vs. being false when the action is ACTUALLY happening. So, if you're doing
** things like storing data, you don't want to do that if isPreview is true!!
**
** General implementation would be, to iterate over the records (the inputs to
** the insert action), and look at their values:
** - possibly adding Errors (`addError`) or Warnings (`addWarning`) to the records
** - possibly manipulating values (`setValue`)
** - possibly throwing an exception - if you really don't want the insert operation to continue.
** - doing "whatever else" you may want to do.
** - returning the list of records (can be the input list) that you want to go on to the backend implementation class.
**
** Note that the full insertInput is available as a field in this class.
*******************************************************************************/
public abstract class AbstractPreInsertCustomizer
{
protected InsertInput insertInput;
protected boolean isPreview = false;
/////////////////////////////////////////////////////////////////////////////////
// allow the customizer to specify when it should be executed as part of the //
// insert action. default (per method in this class) is AFTER_ALL_VALIDATIONS //
/////////////////////////////////////////////////////////////////////////////////
public enum WhenToRun
{
BEFORE_ALL_VALIDATIONS,
BEFORE_UNIQUE_KEY_CHECKS,
BEFORE_REQUIRED_FIELD_CHECKS,
BEFORE_SECURITY_CHECKS,
AFTER_ALL_VALIDATIONS
}
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records) throws QException;
/*******************************************************************************
**
*******************************************************************************/
public WhenToRun getWhenToRun()
{
return (WhenToRun.AFTER_ALL_VALIDATIONS);
}
/*******************************************************************************
** Getter for insertInput
**
*******************************************************************************/
public InsertInput getInsertInput()
{
return insertInput;
}
/*******************************************************************************
** Setter for insertInput
**
*******************************************************************************/
public void setInsertInput(InsertInput insertInput)
{
this.insertInput = insertInput;
}
/*******************************************************************************
** Getter for isPreview
*******************************************************************************/
public boolean getIsPreview()
{
return (this.isPreview);
}
/*******************************************************************************
** Setter for isPreview
*******************************************************************************/
public void setIsPreview(boolean isPreview)
{
this.isPreview = isPreview;
}
/*******************************************************************************
** Fluent setter for isPreview
*******************************************************************************/
public AbstractPreInsertCustomizer withIsPreview(boolean isPreview)
{
this.isPreview = isPreview;
return (this);
}
}

View File

@ -0,0 +1,163 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Abstract class that a table can specify an implementation of, to provide
** custom actions before an update takes place.
**
** It's important for implementations to be aware of the isPreview field, which
** is set to true when the code is running to give users advice, e.g., on a review
** screen - vs. being false when the action is ACTUALLY happening. So, if you're doing
** things like storing data, you don't want to do that if isPreview is true!!
**
** General implementation would be, to iterate over the records (the inputs to
** the update action), and look at their values:
** - possibly adding Errors (`addError`) or Warnings (`addWarning`) to the records
** - possibly manipulating values (`setValue`)
** - possibly throwing an exception - if you really don't want the update operation to continue.
** - doing "whatever else" you may want to do.
** - returning the list of records (can be the input list) that you want to go on to the backend implementation class.
**
** Note that the full updateInput is available as a field in this class, and the
** "old records" (e.g., with values freshly fetched from the backend) will be
** available (if the backend supports it) - both as a list (`getOldRecordList`)
** and as a memoized (by this class) map of primaryKey to record (`getOldRecordMap`).
*******************************************************************************/
public abstract class AbstractPreUpdateCustomizer
{
protected UpdateInput updateInput;
protected List<QRecord> oldRecordList;
protected boolean isPreview = false;
private Map<Serializable, QRecord> oldRecordMap = null;
/*******************************************************************************
**
*******************************************************************************/
public abstract List<QRecord> apply(List<QRecord> records) throws QException;
/*******************************************************************************
** Getter for updateInput
**
*******************************************************************************/
public UpdateInput getUpdateInput()
{
return updateInput;
}
/*******************************************************************************
** Setter for updateInput
**
*******************************************************************************/
public void setUpdateInput(UpdateInput updateInput)
{
this.updateInput = updateInput;
}
/*******************************************************************************
**
*******************************************************************************/
public void setOldRecordList(List<QRecord> oldRecordList)
{
this.oldRecordList = oldRecordList;
}
/*******************************************************************************
**
*******************************************************************************/
public List<QRecord> getOldRecordList()
{
return oldRecordList;
}
/*******************************************************************************
**
*******************************************************************************/
protected Map<Serializable, QRecord> getOldRecordMap()
{
if(oldRecordMap == null)
{
oldRecordMap = new HashMap<>();
for(QRecord qRecord : oldRecordList)
{
oldRecordMap.put(qRecord.getValue(updateInput.getTable().getPrimaryKeyField()), qRecord);
}
}
return (oldRecordMap);
}
/*******************************************************************************
** Getter for isPreview
*******************************************************************************/
public boolean getIsPreview()
{
return (this.isPreview);
}
/*******************************************************************************
** Setter for isPreview
*******************************************************************************/
public void setIsPreview(boolean isPreview)
{
this.isPreview = isPreview;
}
/*******************************************************************************
** Fluent setter for isPreview
*******************************************************************************/
public AbstractPreUpdateCustomizer withIsPreview(boolean isPreview)
{
this.isPreview = isPreview;
return (this);
}
}

View File

@ -0,0 +1,232 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QRuntimeException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.statusmessages.QStatusMessage;
import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
/*******************************************************************************
** Standard/re-usable post-insert customizer, for the use case where, when we
** do an insert into table "parent", we want a record automatically inserted into
** table "child". Optionally (based on RelationshipType), there can be a foreign
** key in "parent", pointed at "child". e.g., named: "parent.childId".
**
*******************************************************************************/
public abstract class ChildInserterPostInsertCustomizer extends AbstractPostInsertCustomizer
{
public enum RelationshipType
{
PARENT_POINTS_AT_CHILD,
CHILD_POINTS_AT_PARENT
}
/*******************************************************************************
**
*******************************************************************************/
public abstract QRecord buildChildForRecord(QRecord parentRecord) throws QException;
/*******************************************************************************
**
*******************************************************************************/
public abstract String getChildTableName();
/*******************************************************************************
**
*******************************************************************************/
public String getForeignKeyFieldName()
{
return (null);
}
/*******************************************************************************
**
*******************************************************************************/
public abstract RelationshipType getRelationshipType();
/*******************************************************************************
**
*******************************************************************************/
@Override
public List<QRecord> apply(List<QRecord> records)
{
try
{
List<QRecord> rs = records;
List<QRecord> childrenToInsert = new ArrayList<>();
QTableMetaData table = getInsertInput().getTable();
QTableMetaData childTable = getInsertInput().getInstance().getTable(getChildTableName());
////////////////////////////////////////////////////////////////////////////////
// iterate over the inserted records, building a list child records to insert //
// for ones missing a value in the foreign key field. //
////////////////////////////////////////////////////////////////////////////////
switch(getRelationshipType())
{
case PARENT_POINTS_AT_CHILD ->
{
String foreignKeyFieldName = getForeignKeyFieldName();
try
{
table.getField(foreignKeyFieldName);
}
catch(Exception e)
{
throw new QRuntimeException("For RelationshipType.PARENT_POINTS_AT_CHILD, a valid foreignKeyFieldName in the parent table must be given. "
+ "[" + foreignKeyFieldName + "] is not a valid field name in table [" + table.getName() + "]");
}
for(QRecord record : records)
{
if(record.getValue(foreignKeyFieldName) == null)
{
childrenToInsert.add(buildChildForRecord(record));
}
}
}
case CHILD_POINTS_AT_PARENT ->
{
for(QRecord record : records)
{
childrenToInsert.add(buildChildForRecord(record));
}
}
default -> throw new IllegalStateException("Unexpected value: " + getRelationshipType());
}
///////////////////////////////////////////////////////////////////////////////////
// if there are no children to insert, then just return the original record list //
///////////////////////////////////////////////////////////////////////////////////
if(childrenToInsert.isEmpty())
{
return (records);
}
/////////////////////////
// insert the children //
/////////////////////////
InsertInput insertInput = new InsertInput();
insertInput.setTableName(getChildTableName());
insertInput.setRecords(childrenToInsert);
insertInput.setTransaction(this.insertInput.getTransaction());
InsertOutput insertOutput = new InsertAction().execute(insertInput);
Iterator<QRecord> insertedRecordIterator = insertOutput.getRecords().iterator();
/////////////////////////////////////////////////////////////////////////////////
// check for any errors when inserting the children, if any errors were found, //
// then set a warning in the parent with the details of the problem //
/////////////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////////////////////////////////
// for the PARENT_POINTS_AT_CHILD relationship type:
// iterate over the original list of records again - for any that need a child (e.g., are missing //
// foreign key), set their foreign key to a newly inserted child's key, and add them to be updated. //
//////////////////////////////////////////////////////////////////////////////////////////////////////
switch(getRelationshipType())
{
case PARENT_POINTS_AT_CHILD ->
{
rs = new ArrayList<>();
List<QRecord> recordsToUpdate = new ArrayList<>();
for(QRecord record : records)
{
Serializable primaryKey = record.getValue(table.getPrimaryKeyField());
if(record.getValue(getForeignKeyFieldName()) == null)
{
///////////////////////////////////////////////////////////////////////////////////////////////////
// get the corresponding child record, if it has any errors, set that as a warning in the parent //
///////////////////////////////////////////////////////////////////////////////////////////////////
QRecord childRecord = insertedRecordIterator.next();
if(CollectionUtils.nullSafeHasContents(childRecord.getErrors()))
{
for(QStatusMessage error : childRecord.getErrors())
{
record.addWarning(new QWarningMessage("Error creating child " + childTable.getLabel() + " (" + error.toString() + ")"));
}
rs.add(record);
continue;
}
Serializable foreignKey = childRecord.getValue(childTable.getPrimaryKeyField());
recordsToUpdate.add(new QRecord().withValue(table.getPrimaryKeyField(), primaryKey).withValue(getForeignKeyFieldName(), foreignKey));
record.setValue(getForeignKeyFieldName(), foreignKey);
rs.add(record);
}
else
{
rs.add(record);
}
}
////////////////////////////////////////////////////////////////////////////
// update the originally inserted records to reference their new children //
////////////////////////////////////////////////////////////////////////////
UpdateInput updateInput = new UpdateInput();
updateInput.setTableName(getInsertInput().getTableName());
updateInput.setRecords(recordsToUpdate);
updateInput.setTransaction(this.insertInput.getTransaction());
new UpdateAction().execute(updateInput);
}
case CHILD_POINTS_AT_PARENT ->
{
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// todo - some version of looking at the inserted children to confirm that they were inserted, and updating the parents with warnings if they weren't //
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
}
default -> throw new IllegalStateException("Unexpected value: " + getRelationshipType());
}
return (rs);
}
catch(RuntimeException re)
{
throw (re);
}
catch(Exception e)
{
throw new RuntimeException("Error inserting new child records for new parent records", e);
}
}
}

View File

@ -24,14 +24,17 @@ package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.Optional;
import java.util.function.Function;
import com.kingsrook.qqq.backend.core.actions.automation.RecordAutomationHandler;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.actions.values.QCustomPossibleValueProvider;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeType;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import com.kingsrook.qqq.backend.core.model.metadata.tables.automation.TableAutomationAction;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
@ -39,7 +42,7 @@ import org.apache.logging.log4j.Logger;
*******************************************************************************/
public class QCodeLoader
{
private static final Logger LOG = LogManager.getLogger(QCodeLoader.class);
private static final QLogger LOG = QLogger.getLogger(QCodeLoader.class);
@ -58,6 +61,21 @@ public class QCodeLoader
/*******************************************************************************
**
*******************************************************************************/
public static <T> Optional<T> getTableCustomizer(Class<T> expectedClass, QTableMetaData table, String customizerName)
{
Optional<QCodeReference> codeReference = table.getCustomizer(customizerName);
if(codeReference.isPresent())
{
return (Optional.ofNullable(QCodeLoader.getAdHoc(expectedClass, codeReference.get())));
}
return (Optional.empty());
}
/*******************************************************************************
**
*******************************************************************************/
@ -84,7 +102,7 @@ public class QCodeLoader
}
catch(Exception e)
{
LOG.error("Error initializing customizer: " + codeReference);
LOG.error("Error initializing customizer", logPair("codeReference", codeReference), e);
//////////////////////////////////////////////////////////////////////////////////////////////////////////
// return null here - under the assumption that during normal run-time operations, we'll never hit here //
@ -97,6 +115,120 @@ public class QCodeLoader
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
public static <T extends BackendStep> T getBackendStep(Class<T> expectedType, QCodeReference codeReference)
{
if(codeReference == null)
{
return (null);
}
if(!codeReference.getCodeType().equals(QCodeType.JAVA))
{
///////////////////////////////////////////////////////////////////////////////////////
// todo - 1) support more languages, 2) wrap them w/ java Functions here, 3) profit! //
///////////////////////////////////////////////////////////////////////////////////////
throw (new IllegalArgumentException("Only JAVA BackendSteps are supported at this time."));
}
try
{
Class<?> customizerClass = Class.forName(codeReference.getName());
return ((T) customizerClass.getConstructor().newInstance());
}
catch(Exception e)
{
LOG.error("Error initializing customizer", logPair("codeReference", codeReference), e);
//////////////////////////////////////////////////////////////////////////////////////////////////////////
// return null here - under the assumption that during normal run-time operations, we'll never hit here //
// as we'll want to validate all functions in the instance validator at startup time (and IT will throw //
// if it finds an invalid code reference //
//////////////////////////////////////////////////////////////////////////////////////////////////////////
return (null);
}
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
public static <T> T getAdHoc(Class<T> expectedType, QCodeReference codeReference)
{
if(codeReference == null)
{
return (null);
}
if(!codeReference.getCodeType().equals(QCodeType.JAVA))
{
///////////////////////////////////////////////////////////////////////////////////////
// todo - 1) support more languages, 2) wrap them w/ java Functions here, 3) profit! //
///////////////////////////////////////////////////////////////////////////////////////
throw (new IllegalArgumentException("Only JAVA code references are supported at this time."));
}
try
{
Class<?> customizerClass = Class.forName(codeReference.getName());
return ((T) customizerClass.getConstructor().newInstance());
}
catch(Exception e)
{
LOG.error("Error initializing customizer", logPair("codeReference", codeReference), e);
//////////////////////////////////////////////////////////////////////////////////////////////////////////
// return null here - under the assumption that during normal run-time operations, we'll never hit here //
// as we'll want to validate all functions in the instance validator at startup time (and IT will throw //
// if it finds an invalid code reference //
//////////////////////////////////////////////////////////////////////////////////////////////////////////
return (null);
}
}
/*******************************************************************************
**
*******************************************************************************/
public static RecordAutomationHandler getRecordAutomationHandler(TableAutomationAction action) throws QException
{
try
{
QCodeReference codeReference = action.getCodeReference();
if(!codeReference.getCodeType().equals(QCodeType.JAVA))
{
///////////////////////////////////////////////////////////////////////////////////////
// todo - 1) support more languages, 2) wrap them w/ java Functions here, 3) profit! //
///////////////////////////////////////////////////////////////////////////////////////
throw (new IllegalArgumentException("Only JAVA customizers are supported at this time."));
}
Class<?> codeClass = Class.forName(codeReference.getName());
Object codeObject = codeClass.getConstructor().newInstance();
if(!(codeObject instanceof RecordAutomationHandler recordAutomationHandler))
{
throw (new QException("The supplied code [" + codeClass.getName() + "] is not an instance of RecordAutomationHandler"));
}
return (recordAutomationHandler);
}
catch(QException qe)
{
throw (qe);
}
catch(Exception e)
{
throw (new QException("Error getting record automation handler for action [" + action.getName() + "]", e));
}
}
/*******************************************************************************
**
*******************************************************************************/
@ -121,4 +253,5 @@ public class QCodeLoader
throw (new QException("Error getting custom possible value provider for PVS [" + possibleValueSource.getName() + "]", e));
}
}
}

View File

@ -0,0 +1,149 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** Interface with utility methods that pre insert/update/delete customizers
** may want to use.
*******************************************************************************/
public interface RecordCustomizerUtilityInterface
{
QLogger LOG = QLogger.getLogger(RecordCustomizerUtilityInterface.class);
/*******************************************************************************
** Container for an old value and a new value.
*******************************************************************************/
@SuppressWarnings("checkstyle:MethodName")
record Change(Serializable oldValue, Serializable newValue)
{
}
/*******************************************************************************
**
*******************************************************************************/
default Map<String, Change> getChanges(String tableName, QRecord oldRecord, QRecord newRecord)
{
Map<String, Change> rs = new HashMap<>();
QTableMetaData table = QContext.getQInstance().getTable(tableName);
for(Map.Entry<String, Serializable> entry : newRecord.getValues().entrySet())
{
String fieldName = entry.getKey();
Serializable newValue = entry.getValue();
Serializable oldValue = oldRecord.getValue(fieldName);
try
{
QFieldMetaData field = table.getField(fieldName);
Serializable newTypedValue = ValueUtils.getValueAsFieldType(field.getType(), newValue);
Serializable oldTypedValue = ValueUtils.getValueAsFieldType(field.getType(), oldValue);
if(!Objects.equals(oldTypedValue, newTypedValue))
{
rs.put(fieldName, new Change(oldTypedValue, newTypedValue));
}
}
catch(Exception e)
{
LOG.info("Error getting a value as field's type", e, logPair("fieldName", fieldName), logPair("oldValue", oldValue), logPair("newValue", newValue));
}
}
return (rs);
}
/*******************************************************************************
**
*******************************************************************************/
default void errorIfNoValue(Serializable value, QRecord record, String errorMessage)
{
errorIf(!StringUtils.hasContent(ValueUtils.getValueAsString(value)), record, errorMessage);
}
/*******************************************************************************
**
*******************************************************************************/
default void errorIfEditedValue(QRecord oldRecord, QRecord newRecord, String fieldName, String errorMessage)
{
if(newRecord.getValues().containsKey(fieldName))
{
errorIf(isChangedValue(oldRecord.getValue(fieldName), newRecord.getValue(fieldName)), newRecord, errorMessage);
}
}
/*******************************************************************************
**
*******************************************************************************/
default boolean isChangedValue(Serializable oldValue, Serializable newValue)
{
//////////////////////////////////////////////
// todo - probably ... some type "coercion" //
//////////////////////////////////////////////
return (!Objects.equals(oldValue, newValue));
}
/*******************************************************************************
**
*******************************************************************************/
default void errorIfAnyValue(Serializable value, QRecord record, String errorMessage)
{
if(StringUtils.hasContent(ValueUtils.getValueAsString(value)))
{
record.addError(new BadInputStatusMessage(errorMessage));
}
}
/*******************************************************************************
**
*******************************************************************************/
default void errorIf(boolean condition, QRecord record, String errorMessage)
{
if(condition)
{
record.addError(new BadInputStatusMessage(errorMessage));
}
}
}

View File

@ -1,61 +0,0 @@
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.function.Consumer;
/*******************************************************************************
** Object used by TableCustomizers enum (and similar enums in backend modules)
** to assist with definition and validation of Customizers applied to tables.
*******************************************************************************/
public class TableCustomizer
{
private final String role;
private final Class<?> expectedType;
private final Consumer<Object> validationFunction;
/*******************************************************************************
**
*******************************************************************************/
public TableCustomizer(String role, Class<?> expectedType, Consumer<Object> validationFunction)
{
this.role = role;
this.expectedType = expectedType;
this.validationFunction = validationFunction;
}
/*******************************************************************************
** Getter for role
**
*******************************************************************************/
public String getRole()
{
return role;
}
/*******************************************************************************
** Getter for expectedType
**
*******************************************************************************/
public Class<?> getExpectedType()
{
return expectedType;
}
/*******************************************************************************
** Getter for validationFunction
**
*******************************************************************************/
public Consumer<Object> getValidationFunction()
{
return validationFunction;
}
}

View File

@ -1,44 +1,55 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.customizers;
import java.util.function.Function;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Enum definition of possible table customizers - "roles" for custom code that
** can be applied to tables.
**
** Works with TableCustomizer (singular version of this name) objects, during
** instance validation, to provide validation of the referenced code (and to
** make such validation from sub-backend-modules possible in the future).
**
** The idea of the 3rd argument here is to provide a way that we can enforce
** the type-parameters for the custom code. E.g., if it's a Function - how
** can we check at run-time that the type-params are correct? We couldn't find
** how to do this "reflectively", so we can instead try to run the custom code,
** passing it objects of the type that this customizer expects, and a validation
** error will raise upon ClassCastException... This maybe could improve!
*******************************************************************************/
public enum TableCustomizers
{
POST_QUERY_RECORD(new TableCustomizer("postQueryRecord", Function.class, ((Object x) ->
{
Function<QRecord, QRecord> function = (Function<QRecord, QRecord>) x;
QRecord output = function.apply(new QRecord());
})));
POST_QUERY_RECORD("postQueryRecord", AbstractPostQueryCustomizer.class),
PRE_INSERT_RECORD("preInsertRecord", AbstractPreInsertCustomizer.class),
POST_INSERT_RECORD("postInsertRecord", AbstractPostInsertCustomizer.class),
PRE_UPDATE_RECORD("preUpdateRecord", AbstractPreUpdateCustomizer.class),
POST_UPDATE_RECORD("postUpdateRecord", AbstractPostUpdateCustomizer.class),
PRE_DELETE_RECORD("preDeleteRecord", AbstractPreDeleteCustomizer.class),
POST_DELETE_RECORD("postDeleteRecord", AbstractPostDeleteCustomizer.class);
private final TableCustomizer tableCustomizer;
private final String role;
private final Class<?> expectedType;
/*******************************************************************************
**
*******************************************************************************/
TableCustomizers(TableCustomizer tableCustomizer)
TableCustomizers(String role, Class<?> expectedType)
{
this.tableCustomizer = tableCustomizer;
this.role = role;
this.expectedType = expectedType;
}
@ -51,7 +62,7 @@ public enum TableCustomizers
{
for(TableCustomizers value : values())
{
if(value.tableCustomizer.getRole().equals(name))
if(value.role.equals(name))
{
return (value);
}
@ -62,24 +73,23 @@ public enum TableCustomizers
/*******************************************************************************
** Getter for tableCustomizer
**
*******************************************************************************/
public TableCustomizer getTableCustomizer()
{
return tableCustomizer;
}
/*******************************************************************************
** get the role from the tableCustomizer
**
*******************************************************************************/
public String getRole()
{
return (tableCustomizer.getRole());
return (role);
}
/*******************************************************************************
** Getter for expectedType
**
*******************************************************************************/
public Class<?> getExpectedType()
{
return expectedType;
}
}

View File

@ -0,0 +1,422 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard;
import java.io.Serializable;
import java.net.URLEncoder;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
/*******************************************************************************
** Base class for rendering qqq HTML dashboard widgets
**
*******************************************************************************/
public abstract class AbstractHTMLWidgetRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
public static String openTopLevelBulletList()
{
return ("""
<div style="padding-left: 2rem;">
<ul>""");
}
/*******************************************************************************
**
*******************************************************************************/
public static String closeTopLevelBulletList()
{
return ("""
</ul>
</div>""");
}
/*******************************************************************************
**
*******************************************************************************/
protected String bulletItalics(String text)
{
return ("<li><i>" + text + "</i></li>");
}
/*******************************************************************************
**
*******************************************************************************/
protected String bulletLink(String href, String text)
{
return ("<li><a href=\"" + href + "\">" + text + "</a></li>");
}
/*******************************************************************************
**
*******************************************************************************/
protected String bulletNameLink(String name, String href, String text)
{
return (bulletNameValue(name, "<a href=\"" + href + "\">" + text + "</a>"));
}
/*******************************************************************************
**
*******************************************************************************/
protected String bulletNameValue(String name, String value)
{
return ("<li><b>" + name + "</b> &nbsp; " + Objects.requireNonNullElse(value, "--") + "</li>");
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableBulkLoad(RenderWidgetInput input, String tableName) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
return (tablePath + "/" + tableName + ".bulkInsert");
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableBulkLoadChildren(RenderWidgetInput input, String tableName) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return (null);
}
return ("#/launchProcess=" + tableName + ".bulkInsert");
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableCreate(RenderWidgetInput input, String tableName) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
return (tablePath + "/create");
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableCreateWithDefaultValues(RenderWidgetInput input, String tableName, Map<String, Serializable> defaultValues) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
return (tablePath + "/create?defaultValues=" + URLEncoder.encode(JsonUtils.toJson(defaultValues), Charset.defaultCharset()));
}
/*******************************************************************************
**
*******************************************************************************/
public static String getCountLink(RenderWidgetInput input, String tableName, QQueryFilter filter, int count) throws QException
{
String totalString = QValueFormatter.formatValue(DisplayFormat.COMMAS, count);
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null || filter == null)
{
return (totalString);
}
return ("<a href='" + tablePath + "?filter=" + JsonUtils.toJson(filter) + "'>" + totalString + "</a>");
}
/*******************************************************************************
**
*******************************************************************************/
public static void addTableFilterToListIfPermissed(RenderWidgetInput input, String tableName, List<String> urls, QQueryFilter filter) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return;
}
urls.add(tablePath + "?filter=" + JsonUtils.toJson(filter));
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableFilterUnencoded(RenderWidgetInput input, String tableName, QQueryFilter filter) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return (null);
}
return (tablePath + "?filter=" + JsonUtils.toJson(filter));
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableFilter(String tableName, QQueryFilter filter) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return (null);
}
return (tablePath + "?filter=" + URLEncoder.encode(JsonUtils.toJson(filter), Charset.defaultCharset()));
}
/*******************************************************************************
**
*******************************************************************************/
public static String aHrefTableFilterNoOfRecords(String tableName, QQueryFilter filter, Integer noOfRecords, String singularLabel, String pluralLabel) throws QException
{
return (aHrefTableFilterNoOfRecords(tableName, filter, noOfRecords, singularLabel, pluralLabel, false));
}
/*******************************************************************************
**
*******************************************************************************/
public static String aHrefTableFilterNoOfRecords(String tableName, QQueryFilter filter, Integer noOfRecords, String singularLabel, String pluralLabel, boolean onlyLinkCount) throws QException
{
String plural = StringUtils.plural(noOfRecords, singularLabel, pluralLabel);
String countString = QValueFormatter.formatValue(DisplayFormat.COMMAS, noOfRecords);
String displayText = StringUtils.hasContent(plural) ? (" " + plural) : "";
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return (countString + displayText);
}
String href = linkTableFilter(tableName, filter);
if(onlyLinkCount)
{
return ("<a href=\"" + href + "\">" + countString + "</a>" + displayText);
}
else
{
return ("<a href=\"" + href + "\">" + countString + displayText + "</a>");
}
}
/*******************************************************************************
**
*******************************************************************************/
public static String aHrefViewRecord(String tableName, Serializable id, String linkText) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return (linkText);
}
return ("<a href=\"" + linkRecordView(tableName, id) + "\">" + linkText + "</a>");
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkRecordEdit(AbstractActionInput input, String tableName, Serializable recordId) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
return (tablePath + "/" + recordId + "/edit");
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkRecordView(String tableName, Serializable recordId) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(tableName);
if(tablePath == null)
{
return (null);
}
return (tablePath + "/" + recordId);
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkProcessForFilter(AbstractActionInput input, String processName, QQueryFilter filter) throws QException
{
QProcessMetaData process = QContext.getQInstance().getProcess(processName);
if(process == null)
{
return (null);
}
String tableName = process.getTableName();
if(tableName == null)
{
return (null);
}
String tablePath = QContext.getQInstance().getTablePath(tableName);
return (tablePath + "/" + processName + "?recordsParam=filterJSON&filterJSON=" + URLEncoder.encode(JsonUtils.toJson(filter), StandardCharsets.UTF_8));
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkProcessForRecord(AbstractActionInput input, String processName, Serializable recordId) throws QException
{
QProcessMetaData process = QContext.getQInstance().getProcess(processName);
String tableName = process.getTableName();
String tablePath = QContext.getQInstance().getTablePath(tableName);
return (tablePath + "/" + recordId + "/" + processName);
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableCreateChild(RenderWidgetInput input, String childTableName, Map<String, Serializable> defaultValues) throws QException
{
return (linkTableCreateChild(input, childTableName, defaultValues, defaultValues.keySet()));
}
/*******************************************************************************
**
*******************************************************************************/
public static String aHrefTableCreateChild(RenderWidgetInput input, String childTableName, Map<String, Serializable> defaultValues) throws QException
{
return (aHrefTableCreateChild(input, childTableName, defaultValues, defaultValues.keySet()));
}
/*******************************************************************************
**
*******************************************************************************/
public static String linkTableCreateChild(RenderWidgetInput input, String childTableName, Map<String, Serializable> defaultValues, Set<String> disabledFields) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(childTableName);
if(tablePath == null)
{
return (null);
}
Map<String, Integer> disabledFieldsMap = disabledFields.stream().collect(Collectors.toMap(k -> k, k -> 1));
return ("#/createChild=" + childTableName
+ "/defaultValues=" + URLEncoder.encode(JsonUtils.toJson(defaultValues), StandardCharsets.UTF_8).replaceAll("\\+", "%20")
+ "/disabledFields=" + URLEncoder.encode(JsonUtils.toJson(disabledFieldsMap), StandardCharsets.UTF_8).replaceAll("\\+", "%20"));
}
/*******************************************************************************
**
*******************************************************************************/
public static String getChipElement(String icon, String label, String color) throws QException
{
color = color != null ? color : "info";
color = StringUtils.ucFirst(color);
String html = "<span style='display: flex;'>";
html += "<div style='overflow: hidden; flex: none; display: flex; align-content: flex-start; align-items: center; height: 24px; padding-right: 8px; font-size: 13px; font-weight: 500; border: 1px solid; border-radius: 16px; color: " + color + "'>";
if(icon != null)
{
html += "<span style='font-size: 16px; padding: 5px' class='material-icons-round notranslate MuiIcon-root MuiIcon-fontSizeInherit MuiChip-icon MuiChip-iconSmall MuiChip-iconColor" + color + "'>" + icon + "</span>";
}
html += "<span class='MuiChip-label MuiChip-labelSmall'>" + label + "</span></div></span>";
return (html);
}
/*******************************************************************************
**
*******************************************************************************/
public static String aHrefTableCreateChild(RenderWidgetInput input, String childTableName, Map<String, Serializable> defaultValues, Set<String> disabledFields) throws QException
{
String tablePath = QContext.getQInstance().getTablePath(childTableName);
if(tablePath == null)
{
return (null);
}
return ("<a href=\"" + linkTableCreateChild(input, childTableName, defaultValues, defaultValues.keySet()) + "\">Create new</a>");
}
}

View File

@ -0,0 +1,62 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard;
import java.io.Serializable;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
/*******************************************************************************
** Class for loading widget implementation code and rendering of widgets
**
*******************************************************************************/
public class RenderWidgetAction
{
/*******************************************************************************
**
*******************************************************************************/
public RenderWidgetOutput execute(RenderWidgetInput input) throws QException
{
ActionHelper.validateSession(input);
AbstractWidgetRenderer widgetRenderer = QCodeLoader.getAdHoc(AbstractWidgetRenderer.class, input.getWidgetMetaData().getCodeReference());
///////////////////////////////////////////////////////////////
// move default values from meta data into this render input //
///////////////////////////////////////////////////////////////
for(Map.Entry<String, Serializable> entry : input.getWidgetMetaData().getDefaultValues().entrySet())
{
input.addQueryParam(entry.getKey(), ValueUtils.getValueAsString(entry.getValue()));
}
return (widgetRenderer.render(input));
}
}

View File

@ -0,0 +1,165 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
import com.kingsrook.qqq.backend.core.actions.values.SearchPossibleValueSourceAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceOutput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.QWidgetData;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.WidgetDropdownData;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
/*******************************************************************************
** Base class for rendering qqq dashboard widgets
**
*******************************************************************************/
public abstract class AbstractWidgetRenderer
{
public static final QValueFormatter valueFormatter = new QValueFormatter();
public static final DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(ZoneId.systemDefault());
/*******************************************************************************
**
*******************************************************************************/
public abstract RenderWidgetOutput render(RenderWidgetInput input) throws QException;
/*******************************************************************************
**
*******************************************************************************/
protected boolean setupDropdowns(RenderWidgetInput input, QWidgetMetaData metaData, QWidgetData widgetData) throws QException
{
List<List<Map<String, String>>> pvsData = new ArrayList<>();
List<String> pvsLabels = new ArrayList<>();
List<String> pvsNames = new ArrayList<>();
List<String> missingRequiredSelections = new ArrayList<>();
for(WidgetDropdownData dropdownData : CollectionUtils.nonNullList(metaData.getDropdowns()))
{
String possibleValueSourceName = dropdownData.getPossibleValueSourceName();
QPossibleValueSource possibleValueSource = input.getInstance().getPossibleValueSource(possibleValueSourceName);
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// this looks complicated, but is just look for a label in the dropdown data and if found use it, //
// otherwise look for label in PVS and if found use that, otherwise just use the PVS name //
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
String pvsLabel = dropdownData.getLabel() != null ? dropdownData.getLabel() : (possibleValueSource.getLabel() != null ? possibleValueSource.getLabel() : possibleValueSourceName);
pvsLabels.add(pvsLabel);
pvsNames.add(possibleValueSourceName);
SearchPossibleValueSourceInput pvsInput = new SearchPossibleValueSourceInput();
pvsInput.setPossibleValueSourceName(possibleValueSourceName);
if(dropdownData.getForeignKeyFieldName() != null)
{
////////////////////////////////////////
// look for an id in the query params //
////////////////////////////////////////
Integer id = null;
if(input.getQueryParams() != null && input.getQueryParams().containsKey("id") && StringUtils.hasContent(input.getQueryParams().get("id")))
{
id = Integer.parseInt(input.getQueryParams().get("id"));
}
if(id != null)
{
pvsInput.setDefaultQueryFilter(new QQueryFilter().withCriteria(
new QFilterCriteria(
dropdownData.getForeignKeyFieldName(),
QCriteriaOperator.EQUALS,
id)));
}
}
SearchPossibleValueSourceOutput output = new SearchPossibleValueSourceAction().execute(pvsInput);
List<Map<String, String>> dropdownOptionList = new ArrayList<>();
pvsData.add(dropdownOptionList);
//////////////////////////////////////////
// sort results, dedupe, and add to map //
//////////////////////////////////////////
Set<String> exists = new HashSet<>();
output.getResults().removeIf(pvs -> !exists.add(pvs.getLabel()));
for(QPossibleValue<?> possibleValue : output.getResults())
{
dropdownOptionList.add(MapBuilder.of(
"id", String.valueOf(possibleValue.getId()),
"label", possibleValue.getLabel()
));
}
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// because we know the dropdowns and what the field names will be when something is selected, we can make //
// sure that something has been selected, and if not, display a message that a selection needs made //
////////////////////////////////////////////////////////////////////////////////////////////////////////////
if(dropdownData.getIsRequired())
{
if(!input.getQueryParams().containsKey(possibleValueSourceName) || !StringUtils.hasContent(input.getQueryParams().get(possibleValueSourceName)))
{
missingRequiredSelections.add(pvsLabel);
}
}
}
widgetData.setDropdownNameList(pvsNames);
widgetData.setDropdownLabelList(pvsLabels);
widgetData.setDropdownDataList(pvsData);
////////////////////////////////////////////////////////////////////////////////
// if there are any missing required dropdowns, build up a message to display //
////////////////////////////////////////////////////////////////////////////////
if(missingRequiredSelections.size() > 0)
{
StringBuilder sb = new StringBuilder("Please select a ").append(StringUtils.joinWithCommasAndAnd(missingRequiredSelections));
sb.append(" from the ").append(StringUtils.plural(missingRequiredSelections.size(), "dropdown", "dropdowns")).append(" above.");
widgetData.setDropdownNeedsSelectedText(sb.toString());
return (false);
}
else
{
return (true);
}
}
}

View File

@ -0,0 +1,258 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.io.Serializable;
import java.net.URLEncoder;
import java.nio.charset.Charset;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.ChildRecordListData;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.AbstractWidgetMetaDataBuilder;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import org.apache.commons.lang.BooleanUtils;
/*******************************************************************************
** Generic widget for display a list of child records.
*******************************************************************************/
public class ChildRecordListRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
public static Builder widgetMetaDataBuilder(QJoinMetaData join)
{
return (new Builder(new QWidgetMetaData()
.withName(join.getName())
.withIsCard(true)
.withCodeReference(new QCodeReference(ChildRecordListRenderer.class))
.withType(WidgetType.CHILD_RECORD_LIST.getType())
.withDefaultValue("joinName", join.getName())));
}
/*******************************************************************************
**
*******************************************************************************/
public static class Builder extends AbstractWidgetMetaDataBuilder
{
/*******************************************************************************
** Constructor
**
*******************************************************************************/
public Builder(QWidgetMetaData widgetMetaData)
{
super(widgetMetaData);
}
/*******************************************************************************
**
*******************************************************************************/
public Builder withName(String name)
{
widgetMetaData.setName(name);
return (this);
}
/*******************************************************************************
**
*******************************************************************************/
public Builder withLabel(String label)
{
widgetMetaData.setLabel(label);
return (this);
}
/*******************************************************************************
**
*******************************************************************************/
public Builder withMaxRows(Integer maxRows)
{
widgetMetaData.withDefaultValue("maxRows", maxRows);
return (this);
}
/*******************************************************************************
**
*******************************************************************************/
public Builder withCanAddChildRecord(boolean b)
{
widgetMetaData.withDefaultValue("canAddChildRecord", true);
return (this);
}
/*******************************************************************************
**
*******************************************************************************/
public Builder withDisabledFieldsForNewChildRecords(Set<String> disabledFieldsForNewChildRecords)
{
widgetMetaData.withDefaultValue("disabledFieldsForNewChildRecords", new HashSet<>(disabledFieldsForNewChildRecords));
return (this);
}
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
String widgetLabel = input.getQueryParams().get("widgetLabel");
String joinName = input.getQueryParams().get("joinName");
QJoinMetaData join = input.getInstance().getJoin(joinName);
String id = input.getQueryParams().get("id");
QTableMetaData leftTable = input.getInstance().getTable(join.getLeftTable());
QTableMetaData rightTable = input.getInstance().getTable(join.getRightTable());
Integer maxRows = null;
if(StringUtils.hasContent(input.getQueryParams().get("maxRows")))
{
maxRows = ValueUtils.getValueAsInteger(input.getQueryParams().get("maxRows"));
}
else if(input.getWidgetMetaData().getDefaultValues().containsKey("maxRows"))
{
maxRows = ValueUtils.getValueAsInteger(input.getWidgetMetaData().getDefaultValues().containsKey("maxRows"));
}
////////////////////////////////////////////////////////
// fetch the record that we're getting children for. //
// e.g., the left-side of the join, with the input id //
////////////////////////////////////////////////////////
GetInput getInput = new GetInput();
getInput.setTableName(join.getLeftTable());
getInput.setPrimaryKey(id);
GetOutput getOutput = new GetAction().execute(getInput);
QRecord record = getOutput.getRecord();
if(record == null)
{
throw (new QNotFoundException("Could not find " + (leftTable == null ? "" : leftTable.getLabel()) + " with primary key " + id));
}
////////////////////////////////////////////////////////////////////
// set up the query - for the table on the right side of the join //
////////////////////////////////////////////////////////////////////
QQueryFilter filter = new QQueryFilter();
for(JoinOn joinOn : join.getJoinOns())
{
filter.addCriteria(new QFilterCriteria(joinOn.getRightField(), QCriteriaOperator.EQUALS, List.of(record.getValue(joinOn.getLeftField()))));
}
filter.setOrderBys(join.getOrderBys());
filter.setLimit(maxRows);
QueryInput queryInput = new QueryInput();
queryInput.setTableName(join.getRightTable());
queryInput.setShouldTranslatePossibleValues(true);
queryInput.setShouldGenerateDisplayValues(true);
queryInput.setFilter(filter);
QueryOutput queryOutput = new QueryAction().execute(queryInput);
QValueFormatter.setBlobValuesToDownloadUrls(rightTable, queryOutput.getRecords());
int totalRows = queryOutput.getRecords().size();
if(maxRows != null && (queryOutput.getRecords().size() == maxRows))
{
/////////////////////////////////////////////////////////////////////////////////////
// if the input said to only do some max, and the # of results we got is that max, //
// then do a count query, for displaying 1-n of <count> //
/////////////////////////////////////////////////////////////////////////////////////
CountInput countInput = new CountInput();
countInput.setTableName(join.getRightTable());
countInput.setFilter(filter);
totalRows = new CountAction().execute(countInput).getCount();
}
String tablePath = input.getInstance().getTablePath(rightTable.getName());
String viewAllLink = tablePath == null ? null : (tablePath + "?filter=" + URLEncoder.encode(JsonUtils.toJson(filter), Charset.defaultCharset()));
ChildRecordListData widgetData = new ChildRecordListData(widgetLabel, queryOutput, rightTable, tablePath, viewAllLink, totalRows);
if(BooleanUtils.isTrue(ValueUtils.getValueAsBoolean(input.getQueryParams().get("canAddChildRecord"))))
{
widgetData.setCanAddChildRecord(true);
//////////////////////////////////////////////////////////
// new child records must have values from the join-ons //
//////////////////////////////////////////////////////////
Map<String, Serializable> defaultValuesForNewChildRecords = new HashMap<>();
for(JoinOn joinOn : join.getJoinOns())
{
defaultValuesForNewChildRecords.put(joinOn.getRightField(), record.getValue(joinOn.getLeftField()));
}
widgetData.setDefaultValuesForNewChildRecords(defaultValuesForNewChildRecords);
Map<String, Serializable> widgetValues = input.getWidgetMetaData().getDefaultValues();
if(widgetValues.containsKey("disabledFieldsForNewChildRecords"))
{
widgetData.setDisabledFieldsForNewChildRecords((Set<String>) widgetValues.get("disabledFieldsForNewChildRecords"));
}
}
return (new RenderWidgetOutput(widgetData));
}
}

View File

@ -0,0 +1,300 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.time.DayOfWeek;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.time.temporal.ChronoField;
import java.time.temporal.ChronoUnit;
import java.time.temporal.IsoFields;
import java.time.temporal.TemporalAdjusters;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
/*******************************************************************************
** Enum to define various "levels" of group-by for on dashboards that want to
** group records by, e.g., year, or month, or week, or day, or hour.
*******************************************************************************/
public enum DateTimeGroupBy
{
/////////////////////////////////////////////////////////////////////////////////////////////////////////////
// note - double %'s on the time format strings here, because this is a java-format string, which will get //
// its '%s' replaced with a column name, and so then those %'s for the date_format need escaped as %%. //
// See https://www.w3schools.com/sql/func_mysql_date_format.asp for DATE_FORMAT args //
/////////////////////////////////////////////////////////////////////////////////////////////////////////////
YEAR("%%Y", MillisPer.YEAR, 1, ChronoUnit.YEARS, DateTimeFormatter.ofPattern("yyyy"), DateTimeFormatter.ofPattern("yyyy")),
MONTH("%%Y-%%m", 2 * MillisPer.MONTH, 1, ChronoUnit.MONTHS, DateTimeFormatter.ofPattern("yyyy-MM"), DateTimeFormatter.ofPattern("MMM'.' yyyy")),
WEEK("%%XW%%V", 35 * MillisPer.DAY, 7, ChronoUnit.DAYS, DateTimeFormatter.ofPattern("YYYY'W'ww"), DateTimeFormatter.ofPattern("YYYY'W'w")),
DAY("%%Y-%%m-%%d", 36 * MillisPer.HOUR, 1, ChronoUnit.DAYS, DateTimeFormatter.ofPattern("yyyy-MM-dd"), DateTimeFormatter.ofPattern("EEE'.' M'/'d")),
HOUR("%%Y-%%m-%%dT%%H", 0, 1, ChronoUnit.HOURS, DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH"), DateTimeFormatter.ofPattern("h a"));
/*******************************************************************************
**
*******************************************************************************/
public interface MillisPer
{
long HOUR = 60 * 60 * 1000;
long DAY = 24 * HOUR;
long WEEK = 7 * DAY;
long MONTH = 30 * DAY;
long YEAR = 365 * DAY;
}
private final String sqlDateFormat;
private final long millisThreshold;
private final int noOfChronoUnitsToAdd;
private final ChronoUnit chronoUnitToAdd;
private final DateTimeFormatter selectedStringFormatter;
private final DateTimeFormatter humanStringFormatter;
/*******************************************************************************
** Constructor
**
*******************************************************************************/
DateTimeGroupBy(String sqlDateFormat, long millisThreshold, int noOfChronoUnitsToAdd, ChronoUnit chronoUnitToAdd, DateTimeFormatter selectedStringFormatter, DateTimeFormatter humanStringFormatter)
{
this.sqlDateFormat = sqlDateFormat;
this.millisThreshold = millisThreshold;
this.noOfChronoUnitsToAdd = noOfChronoUnitsToAdd;
this.chronoUnitToAdd = chronoUnitToAdd;
this.selectedStringFormatter = selectedStringFormatter;
this.humanStringFormatter = humanStringFormatter;
}
/*******************************************************************************
**
*******************************************************************************/
public String getSqlExpression()
{
ZoneId sessionOrInstanceZoneId = ValueUtils.getSessionOrInstanceZoneId();
return (getSqlExpression(sessionOrInstanceZoneId));
}
/*******************************************************************************
**
*******************************************************************************/
public String getSqlExpression(ZoneId targetZoneId)
{
String targetTimezone = targetZoneId.toString();
if("Z".equals(targetTimezone) || !StringUtils.hasContent(targetTimezone))
{
targetTimezone = "UTC";
}
//////////////////////////////////////////////////////////////////////////////////////////////////////
// if we only had a timezone offset (not a zone name/id), then the zoneId's toString will look like //
// UTC-05:00. MySQL doesn't want that, so, strip away the leading UTC, to just get -05:00 //
//////////////////////////////////////////////////////////////////////////////////////////////////////
if((targetTimezone.startsWith("UTC-") || targetTimezone.startsWith("UTC+")) && targetTimezone.length() > 5)
{
targetTimezone = targetTimezone.substring(3);
}
return "DATE_FORMAT(CONVERT_TZ(%s, 'UTC', '" + targetTimezone + "'), '" + sqlDateFormat + "')";
/*
if(this == WEEK)
{
return "YEARWEEK(CONVERT_TZ(%s, 'UTC', '" + targetTimezone + "'), 6)";
}
else
{
return "DATE_FORMAT(CONVERT_TZ(%s, 'UTC', '" + targetTimezone + "'), '" + sqlDateFormat + "')";
}
*/
}
/*******************************************************************************
** get an instance of this enum, based on start & end instants - look at the #
** of millis between them, and return the first enum value w/ a millisThreshold
** under that difference. Default to HOUR.
*******************************************************************************/
public static DateTimeGroupBy selectFromStartAndEndTimes(Instant start, Instant end)
{
long millisBetween = end.toEpochMilli() - start.toEpochMilli();
for(DateTimeGroupBy value : DateTimeGroupBy.values())
{
if(millisBetween > value.millisThreshold)
{
return (value);
}
}
return (HOUR);
}
/*******************************************************************************
** Make an Instant into a string that will match what came out of the database's
** DATE_FORMAT() function
*******************************************************************************/
public String makeSelectedString(Instant time)
{
return (makeSelectedString(time, ValueUtils.getSessionOrInstanceZoneId()));
}
/*******************************************************************************
** Make an Instant into a string that will match what came out of the database's
** DATE_FORMAT() function
*******************************************************************************/
public String makeSelectedString(Instant time, ZoneId zoneId)
{
ZonedDateTime zoned = time.atZone(zoneId);
if(this == WEEK)
{
////////////////////////////////////////////////////////////////////////////////////////////////////
// so, it seems like database is returning, e.g., W00-W52, but java is doing W1-W53... //
// which, apparently we can compensate for by adding a week? not sure, but results seemed right. //
////////////////////////////////////////////////////////////////////////////////////////////////////
zoned = zoned.plusDays(7);
int weekYear = zoned.get(IsoFields.WEEK_BASED_YEAR);
int week = zoned.get(IsoFields.WEEK_OF_WEEK_BASED_YEAR);
return (String.format("%04dW%02d", weekYear, week));
}
return (selectedStringFormatter.format(zoned));
}
/*******************************************************************************
** Make a string to show to a user
*******************************************************************************/
public String makeHumanString(Instant instant)
{
return (makeHumanString(instant, ValueUtils.getSessionOrInstanceZoneId()));
}
/*******************************************************************************
** Make a string to show to a user
*******************************************************************************/
public String makeHumanString(Instant instant, ZoneId zoneId)
{
ZonedDateTime zoned = instant.atZone(zoneId);
if(this.equals(WEEK))
{
DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("M'/'d");
while(zoned.get(ChronoField.DAY_OF_WEEK) != DayOfWeek.SUNDAY.getValue())
{
////////////////////////////////////////
// go backwards until sunday is found //
////////////////////////////////////////
zoned = zoned.minus(1, ChronoUnit.DAYS);
}
return (dateTimeFormatter.format(zoned) + "-" + dateTimeFormatter.format(zoned.plusDays(6)));
/*
int weekOfYear = zoned.get(ChronoField.ALIGNED_WEEK_OF_YEAR);
ZonedDateTime sunday = zoned.with(IsoFields.WEEK_OF_WEEK_BASED_YEAR, weekOfYear).with(TemporalAdjusters.previousOrSame(DayOfWeek.SUNDAY));
ZonedDateTime saturday = sunday.with(TemporalAdjusters.next(DayOfWeek.SATURDAY));
DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("M'/'d");
return (dateTimeFormatter.format(sunday) + "-" + dateTimeFormatter.format(saturday));
*/
}
return (humanStringFormatter.format(zoned));
}
/*******************************************************************************
**
*******************************************************************************/
public Instant roundDown(Instant instant)
{
return roundDown(instant, ValueUtils.getSessionOrInstanceZoneId());
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("checkstyle:indentation")
public Instant roundDown(Instant instant, ZoneId zoneId)
{
ZonedDateTime zoned = instant.atZone(zoneId);
return switch(this)
{
case YEAR -> zoned.with(TemporalAdjusters.firstDayOfYear()).truncatedTo(ChronoUnit.DAYS).toInstant();
case MONTH -> zoned.with(TemporalAdjusters.firstDayOfMonth()).truncatedTo(ChronoUnit.DAYS).toInstant();
case WEEK ->
{
while(zoned.get(ChronoField.DAY_OF_WEEK) != DayOfWeek.SUNDAY.getValue())
{
zoned = zoned.minusDays(1);
}
yield (zoned.truncatedTo(ChronoUnit.DAYS).toInstant());
}
case DAY -> zoned.truncatedTo(ChronoUnit.DAYS).toInstant();
case HOUR -> zoned.truncatedTo(ChronoUnit.HOURS).toInstant();
};
}
/*******************************************************************************
**
*******************************************************************************/
public Instant increment(Instant instant)
{
return (increment(instant, ValueUtils.getSessionOrInstanceZoneId()));
}
/*******************************************************************************
**
*******************************************************************************/
public Instant increment(Instant instant, ZoneId zoneId)
{
ZonedDateTime zoned = instant.atZone(zoneId);
return (zoned.plus(noOfChronoUnitsToAdd, chronoUnitToAdd).toInstant());
}
}

View File

@ -0,0 +1,92 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.util.Map;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.QWidgetData;
/*******************************************************************************
**
*******************************************************************************/
public class DefaultWidgetRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
return new RenderWidgetOutput(new DefaultWidgetData(input));
}
/*******************************************************************************
**
*******************************************************************************/
public static class DefaultWidgetData extends QWidgetData
{
private final String type;
private final Map<String, String> queryParams;
/*******************************************************************************
** Constructor
**
*******************************************************************************/
public DefaultWidgetData(RenderWidgetInput renderWidgetInput)
{
this.type = renderWidgetInput.getWidgetMetaData().getType();
this.queryParams = renderWidgetInput.getQueryParams();
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public String getType()
{
return (type);
}
/*******************************************************************************
** Getter for queryParams
**
*******************************************************************************/
public Map<String, String> getQueryParams()
{
return queryParams;
}
}
}

View File

@ -0,0 +1,47 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.DividerWidgetData;
/*******************************************************************************
** Generic widget for showing a divider
*******************************************************************************/
public class DividerWidgetRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
ActionHelper.validateSession(input);
return (new RenderWidgetOutput(new DividerWidgetData()));
}
}

View File

@ -0,0 +1,153 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.nocode.AbstractWidgetOutput;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.nocode.AbstractWidgetValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.nocode.QNoCodeWidgetMetaData;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.utils.BackendQueryFilterUtils;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
**
*******************************************************************************/
public class NoCodeWidgetRenderer extends AbstractWidgetRenderer
{
private static final QLogger LOG = QLogger.getLogger(NoCodeWidgetRenderer.class);
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
QNoCodeWidgetMetaData widgetMetaData = (QNoCodeWidgetMetaData) input.getWidgetMetaData();
Map<String, Object> context = initContext(input);
context.putAll(input.getQueryParams());
///////////////////////////////////////////////
// populate context by evaluating all values //
///////////////////////////////////////////////
for(AbstractWidgetValueSource valueSource : widgetMetaData.getValues())
{
try
{
LOG.trace("Computing: " + valueSource.getType() + " named " + valueSource.getName() + "...");
Object value = valueSource.evaluate(context, input);
LOG.trace("Computed: " + valueSource.getName() + " = " + value);
context.put(valueSource.getName(), value);
context.put(valueSource.getName() + ".source", valueSource);
}
catch(Exception e)
{
LOG.warn("Error evaluating widget value source", e, logPair("widgetName", input.getWidgetMetaData().getName()), logPair("valueSourceName", valueSource.getName()));
}
}
/////////////////////////////////////////////
// build content by evaluating all outputs //
/////////////////////////////////////////////
List<AbstractWidgetOutput> outputs = widgetMetaData.getOutputs();
String content = renderOutputs(context, outputs);
return (new RenderWidgetOutput(new RawHTML(widgetMetaData.getLabel(), content)));
}
/*******************************************************************************
**
*******************************************************************************/
public Map<String, Object> initContext(RenderWidgetInput input)
{
Map<String, Object> context = new HashMap<>();
context.put("utils", new NoCodeWidgetVelocityUtils(context, input));
context.put("input", input);
return context;
}
/*******************************************************************************
**
*******************************************************************************/
public String renderOutputs(Map<String, Object> context, List<AbstractWidgetOutput> outputs) throws QException
{
StringBuilder content = new StringBuilder();
for(AbstractWidgetOutput output : CollectionUtils.nonNullList(outputs))
{
boolean conditionPassed = true;
if(output.getCondition() != null)
{
conditionPassed = evaluateCondition(output.getCondition(), context);
}
if(conditionPassed)
{
String render = output.render(context);
content.append(render);
LOG.trace("Condition passed, rendered: " + render);
}
else
{
LOG.trace("Condition failed - not rendering this output.");
}
}
return (content.toString());
}
/*******************************************************************************
**
*******************************************************************************/
private boolean evaluateCondition(QFilterCriteria condition, Map<String, Object> context)
{
try
{
Object value = context.get(condition.getFieldName());
return (BackendQueryFilterUtils.doesCriteriaMatch(condition, condition.getFieldName(), (Serializable) value));
}
catch(Exception e)
{
LOG.warn("Error evaluating condition: " + condition, e);
return (false);
}
}
}

View File

@ -0,0 +1,341 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.io.Serializable;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.Instant;
import java.time.ZoneId;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.dashboard.AbstractHTMLWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.values.QValueFormatter;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.nocode.WidgetCount;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
/*******************************************************************************
**
*******************************************************************************/
public class NoCodeWidgetVelocityUtils
{
private static final QLogger LOG = QLogger.getLogger(NoCodeWidgetVelocityUtils.class);
private Map<String, Object> context;
private RenderWidgetInput input;
/*******************************************************************************
** Constructor
**
*******************************************************************************/
public NoCodeWidgetVelocityUtils(Map<String, Object> context, RenderWidgetInput input)
{
this.context = context;
this.input = input;
}
/*******************************************************************************
**
*******************************************************************************/
public String icon(String iconName, String color)
{
return String.format("""
<span class="material-icons-round notranslate MuiIcon-root MuiIcon-fontSizeInherit" style="color: %s; position: relative; top: 6px;" aria-hidden="true">%s</span>
""", color, iconName);
}
/*******************************************************************************
**
*******************************************************************************/
public String helpIcon()
{
return (icon("help_outline", "blue"));
}
/*******************************************************************************
**
*******************************************************************************/
public String errorIcon()
{
return (icon("error_outline", "red"));
}
/*******************************************************************************
**
*******************************************************************************/
public String warningIcon()
{
return (icon("warning", "orange"));
}
/*******************************************************************************
**
*******************************************************************************/
public String checkIcon()
{
return (icon("check", "green"));
}
/*******************************************************************************
**
*******************************************************************************/
public String pendingIcon()
{
return (icon("pending", "#0062ff"));
}
/*******************************************************************************
**
*******************************************************************************/
public String spanColorGreen()
{
return ("""
<span style="color: green;">
""");
}
/*******************************************************************************
**
*******************************************************************************/
public String spanColorOrange()
{
return ("""
<span style="color: orange;">
""");
}
/*******************************************************************************
**
*******************************************************************************/
public String spanColorRed()
{
return ("""
<span style="color: red;">
""");
}
/*******************************************************************************
**
*******************************************************************************/
public String plural(Integer size, String ifOne, String ifNotOne)
{
return StringUtils.plural(size, ifOne, ifNotOne);
}
/*******************************************************************************
**
*******************************************************************************/
public String formatDateTime(Instant i)
{
if(i == null)
{
return ("");
}
return QValueFormatter.formatDateTimeWithZone(i.atZone(ZoneId.of(QContext.getQInstance().getDefaultTimeZoneId())));
}
/*******************************************************************************
**
*******************************************************************************/
public String formatSecondsAsDuration(Integer seconds)
{
StringBuilder rs = new StringBuilder();
if(seconds == null)
{
return ("");
}
int secondsPerDay = 24 * 60 * 60;
if(seconds >= secondsPerDay)
{
int days = seconds / (secondsPerDay);
seconds = seconds % secondsPerDay;
rs.append(days).append(StringUtils.plural(days, " day", " days")).append(" ");
}
int secondsPerHour = 60 * 60;
if(seconds >= secondsPerHour)
{
int hours = seconds / (secondsPerHour);
seconds = seconds % secondsPerHour;
rs.append(hours).append(StringUtils.plural(hours, " hour", " hours")).append(" ");
}
int secondsPerMinute = 60;
if(seconds >= secondsPerMinute)
{
int minutes = seconds / (secondsPerMinute);
seconds = seconds % secondsPerMinute;
rs.append(minutes).append(StringUtils.plural(minutes, " minute", " minutes")).append(" ");
}
if(seconds > 0 || rs.length() == 0)
{
rs.append(seconds).append(StringUtils.plural(seconds, " second", " seconds")).append(" ");
}
if(rs.length() > 0)
{
rs.deleteCharAt(rs.length() - 1);
}
return (rs.toString());
}
/*******************************************************************************
**
*******************************************************************************/
public String formatSecondsAsRoundedDuration(Integer seconds)
{
StringBuilder rs = new StringBuilder();
if(seconds == null)
{
return ("");
}
int secondsPerDay = 24 * 60 * 60;
if(seconds >= secondsPerDay)
{
int days = seconds / (secondsPerDay);
return (days + StringUtils.plural(days, " day", " days"));
}
int secondsPerHour = 60 * 60;
if(seconds >= secondsPerHour)
{
int hours = seconds / (secondsPerHour);
return (hours + StringUtils.plural(hours, " hour", " hours"));
}
int secondsPerMinute = 60;
if(seconds >= secondsPerMinute)
{
int minutes = seconds / (secondsPerMinute);
return (minutes + StringUtils.plural(minutes, " minute", " minutes"));
}
if(seconds > 0 || rs.length() == 0)
{
return (seconds + StringUtils.plural(seconds, " second", " seconds"));
}
return ("");
}
/*******************************************************************************
**
*******************************************************************************/
public String tableCountFilterLink(String countVariableName, String singular, String plural) throws QException
{
try
{
WidgetCount widgetCount = (WidgetCount) context.get(countVariableName + ".source");
Integer count = ValueUtils.getValueAsInteger(context.get(countVariableName));
QQueryFilter filter = widgetCount.getEffectiveFilter(input);
return (AbstractHTMLWidgetRenderer.aHrefTableFilterNoOfRecords(widgetCount.getTableName(), filter, count, singular, plural));
}
catch(Exception e)
{
LOG.warn("Error rendering widget link", e);
return ("");
}
}
/*******************************************************************************
**
*******************************************************************************/
public String format(String displayFormat, Serializable value)
{
return (QValueFormatter.formatValue(displayFormat, value));
}
/*******************************************************************************
**
*******************************************************************************/
public String round(BigDecimal input, int digits)
{
return String.valueOf(input.setScale(digits, RoundingMode.HALF_UP));
}
/*******************************************************************************
**
*******************************************************************************/
public Object ifElse(Object ifObject, Object elseObject)
{
if(StringUtils.hasContent(ValueUtils.getValueAsString(ifObject)))
{
return (ifObject);
}
else if(StringUtils.hasContent(ValueUtils.getValueAsString(elseObject)))
{
return (elseObject);
}
return ("");
}
}

View File

@ -0,0 +1,73 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.ParentWidgetData;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.ParentWidgetMetaData;
/*******************************************************************************
** Generic widget for display a parent widget with children of possible values,
** child widgets, and child actions
*******************************************************************************/
public class ParentWidgetRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
ActionHelper.validateSession(input);
try
{
ParentWidgetMetaData metaData = (ParentWidgetMetaData) input.getWidgetMetaData();
ParentWidgetData widgetData = new ParentWidgetData();
/////////////////////////////////////////////////////////////
// handle any PVSs creating dropdown data for the frontend //
/////////////////////////////////////////////////////////////
boolean dropdownsValid = setupDropdowns(input, metaData, widgetData);
if(dropdownsValid)
{
widgetData.setChildWidgetNameList(metaData.getChildWidgetNameList());
}
widgetData.setLayoutType(metaData.getLayoutType());
return (new RenderWidgetOutput(widgetData));
}
catch(Exception e)
{
throw (new QException("Error rendering parent widget", e));
}
}
}

View File

@ -0,0 +1,73 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.util.HashMap;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.ProcessWidgetData;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
/*******************************************************************************
** Generic widget for displaying a process as a widget
*******************************************************************************/
public class ProcessWidgetRenderer extends AbstractWidgetRenderer
{
public static final String WIDGET_PROCESS_NAME = "processName";
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
ActionHelper.validateSession(input);
try
{
ProcessWidgetData data = new ProcessWidgetData();
if(input.getWidgetMetaData() instanceof QWidgetMetaData widgetMetaData)
{
setupDropdowns(input, widgetMetaData, data);
String processName = (String) widgetMetaData.getDefaultValues().get(WIDGET_PROCESS_NAME);
QProcessMetaData processMetaData = input.getInstance().getProcess(processName);
data.setProcessMetaData(processMetaData);
data.setDefaultValues(new HashMap<>(input.getQueryParams()));
}
return (new RenderWidgetOutput(data));
}
catch(Exception e)
{
throw (new QException("Error rendering process widget", e));
}
}
}

View File

@ -0,0 +1,104 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.QWidgetData;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.QuickSightChart;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QuickSightChartMetaData;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.quicksight.QuickSightClient;
import software.amazon.awssdk.services.quicksight.model.GenerateEmbedUrlForRegisteredUserRequest;
import software.amazon.awssdk.services.quicksight.model.GenerateEmbedUrlForRegisteredUserResponse;
import software.amazon.awssdk.services.quicksight.model.RegisteredUserDashboardEmbeddingConfiguration;
import software.amazon.awssdk.services.quicksight.model.RegisteredUserEmbeddingExperienceConfiguration;
/*******************************************************************************
** Widget implementation for amazon QuickSight charts
**
*******************************************************************************/
public class QuickSightChartRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
ActionHelper.validateSession(input);
try
{
QuickSightChartMetaData quickSightMetaData = (QuickSightChartMetaData) input.getWidgetMetaData();
QuickSightClient quickSightClient = getQuickSightClient(quickSightMetaData);
final RegisteredUserEmbeddingExperienceConfiguration experienceConfiguration = RegisteredUserEmbeddingExperienceConfiguration.builder()
.dashboard(
RegisteredUserDashboardEmbeddingConfiguration.builder()
.initialDashboardId(quickSightMetaData.getDashboardId())
.build())
.build();
final GenerateEmbedUrlForRegisteredUserRequest generateEmbedUrlForRegisteredUserRequest = GenerateEmbedUrlForRegisteredUserRequest.builder()
.awsAccountId(quickSightMetaData.getAccountId())
.userArn(quickSightMetaData.getUserArn())
.experienceConfiguration(experienceConfiguration)
.build();
final GenerateEmbedUrlForRegisteredUserResponse generateEmbedUrlForRegisteredUserResponse = quickSightClient.generateEmbedUrlForRegisteredUser(generateEmbedUrlForRegisteredUserRequest);
String embedUrl = generateEmbedUrlForRegisteredUserResponse.embedUrl();
QWidgetData widget = new QuickSightChart(input.getWidgetMetaData().getName(), quickSightMetaData.getLabel(), embedUrl);
return (new RenderWidgetOutput(widget));
}
catch(Exception e)
{
throw (new QException("Error rendering widget", e));
}
}
/*******************************************************************************
**
*******************************************************************************/
private QuickSightClient getQuickSightClient(QuickSightChartMetaData metaData)
{
AwsBasicCredentials awsCredentials = AwsBasicCredentials.create(metaData.getAccessKey(), metaData.getSecretKey());
QuickSightClient amazonQuickSightClient = QuickSightClient.builder()
.credentialsProvider(StaticCredentialsProvider.create(awsCredentials))
.region(Region.of(metaData.getRegion()))
.build();
return (amazonQuickSightClient);
}
}

View File

@ -0,0 +1,62 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.dashboard.widgets;
import java.math.BigDecimal;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.USMapWidgetData;
/*******************************************************************************
** Generic widget for display a map of the us
*******************************************************************************/
public class USMapWidgetRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
return (new RenderWidgetOutput(
new USMapWidgetData()
.withHeight("250px")
.withMapMarkerList(generateMapMarkerList())
));
}
/*******************************************************************************
**
*******************************************************************************/
protected List<USMapWidgetData.MapMarker> generateMapMarkerList() throws QException
{
return (List.of(new USMapWidgetData.MapMarker("maryville", new BigDecimal("38.725278"), new BigDecimal("-89.957778"))));
}
}

View File

@ -0,0 +1,40 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.interfaces;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOutput;
/*******************************************************************************
** Interface for the Aggregate action.
**
*******************************************************************************/
public interface AggregateInterface extends BaseQueryInterface
{
/*******************************************************************************
**
*******************************************************************************/
AggregateOutput execute(AggregateInput aggregateInput) throws QException;
}

View File

@ -0,0 +1,81 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.interfaces;
import java.time.Instant;
import com.kingsrook.qqq.backend.core.model.querystats.QueryStat;
/*******************************************************************************
** Base class for "query" (e.g., read-operations) action interfaces (query, count, aggregate).
** Initially just here for the QueryStat methods - if we expand those to apply
** to insert/update/delete, well, then rename this maybe to BaseActionInterface?
*******************************************************************************/
public interface BaseQueryInterface
{
/*******************************************************************************
**
*******************************************************************************/
default void setQueryStat(QueryStat queryStat)
{
//////////
// noop //
//////////
}
/*******************************************************************************
**
*******************************************************************************/
default QueryStat getQueryStat()
{
return (null);
}
/*******************************************************************************
**
*******************************************************************************/
default void setQueryStatFirstResultTime()
{
QueryStat queryStat = getQueryStat();
if(queryStat != null)
{
if(queryStat.getFirstResultTimestamp() == null)
{
queryStat.setFirstResultTimestamp(Instant.now());
}
}
}
/*******************************************************************************
**
*******************************************************************************/
default void cancelAction()
{
//////////////////////////////////////////////
// initially at least, a noop in base class //
//////////////////////////////////////////////
}
}

View File

@ -31,7 +31,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountOutput;
** Interface for the Count action.
**
*******************************************************************************/
public interface CountInterface
public interface CountInterface extends BaseQueryInterface
{
/*******************************************************************************
**

View File

@ -50,4 +50,13 @@ public interface DeleteInterface
return (false);
}
/*******************************************************************************
** Specify whether this particular module's delete action can & should fetch
** records before deleting them, e.g., for audits or "not-found-checks"
*******************************************************************************/
default boolean supportsPreFetchQuery()
{
return (true);
}
}

View File

@ -0,0 +1,74 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.interfaces;
import java.util.HashSet;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetOutput;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
/*******************************************************************************
** Interface for the Get action.
**
*******************************************************************************/
public interface GetInterface
{
/*******************************************************************************
**
*******************************************************************************/
GetOutput execute(GetInput getInput) throws QException;
/*******************************************************************************
**
*******************************************************************************/
default void validateInput(GetInput getInput) throws QException
{
if(getInput.getPrimaryKey() != null & getInput.getUniqueKey() != null)
{
throw new QException("A GetInput may not contain both a primary key [" + getInput.getPrimaryKey() + "] and unique key [" + getInput.getUniqueKey() + "]");
}
if(getInput.getUniqueKey() != null)
{
QTableMetaData table = getInput.getTable();
boolean foundMatch = false;
for(UniqueKey uniqueKey : CollectionUtils.nonNullList(table.getUniqueKeys()))
{
if(new HashSet<>(uniqueKey.getFieldNames()).equals(getInput.getUniqueKey().keySet()))
{
foundMatch = true;
break;
}
}
if(!foundMatch)
{
throw new QException("Table [" + table.getName() + "] does not have a unique key defined on fields: " + getInput.getUniqueKey().keySet().stream().sorted().toList());
}
}
}
}

View File

@ -22,7 +22,6 @@
package com.kingsrook.qqq.backend.core.actions.interfaces;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
@ -39,12 +38,4 @@ public interface InsertInterface
*******************************************************************************/
InsertOutput execute(InsertInput insertInput) throws QException;
/*******************************************************************************
**
*******************************************************************************/
default QBackendTransaction openTransaction(InsertInput insertInput) throws QException
{
return (new QBackendTransaction());
}
}

View File

@ -31,10 +31,11 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
** Interface for the Query action.
**
*******************************************************************************/
public interface QueryInterface
public interface QueryInterface extends BaseQueryInterface
{
/*******************************************************************************
**
*******************************************************************************/
QueryOutput execute(QueryInput queryInput) throws QException;
}

View File

@ -37,4 +37,14 @@ public interface UpdateInterface
**
*******************************************************************************/
UpdateOutput execute(UpdateInput updateInput) throws QException;
/*******************************************************************************
** Specify whether this particular module's update action can & should fetch
** records before updating them, e.g., for audits or "not-found-checks"
*******************************************************************************/
default boolean supportsPreFetchQuery()
{
return (true);
}
}

View File

@ -0,0 +1,336 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.metadata;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
/*******************************************************************************
** Object to represent the graph of joins in a QQQ Instance. e.g., all of the
** connections among tables through joins.
*******************************************************************************/
public class JoinGraph
{
private Set<Edge> edges = new HashSet<>();
/*******************************************************************************
** Graph edge (no graph nodes needed in here)
*******************************************************************************/
private record Edge(String joinName, String leftTable, String rightTable)
{
}
/*******************************************************************************
** In this class, we are treating joins as non-directional graph edges - so -
** use this class to "normalize" what may otherwise be duplicated joins in the
** qInstance (e.g., A -> B and B -> A -- in the instance, those are valid, but
** in our graph here, we want to consider those the same).
*******************************************************************************/
private static class NormalizedJoin
{
private String tableA;
private String tableB;
private String joinFieldA;
private String joinFieldB;
/*******************************************************************************
**
*******************************************************************************/
public NormalizedJoin(QJoinMetaData joinMetaData)
{
boolean needFlip = false;
int tableCompare = joinMetaData.getLeftTable().compareTo(joinMetaData.getRightTable());
if(tableCompare < 0)
{
needFlip = true;
}
else if(tableCompare == 0)
{
int fieldCompare = joinMetaData.getJoinOns().get(0).getLeftField().compareTo(joinMetaData.getJoinOns().get(0).getRightField());
if(fieldCompare < 0)
{
needFlip = true;
}
}
if(needFlip)
{
joinMetaData = joinMetaData.flip();
}
tableA = joinMetaData.getLeftTable();
tableB = joinMetaData.getRightTable();
joinFieldA = joinMetaData.getJoinOns().get(0).getLeftField();
joinFieldB = joinMetaData.getJoinOns().get(0).getRightField();
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public boolean equals(Object o)
{
if(this == o)
{
return true;
}
if(o == null || getClass() != o.getClass())
{
return false;
}
NormalizedJoin that = (NormalizedJoin) o;
return Objects.equals(tableA, that.tableA) && Objects.equals(tableB, that.tableB) && Objects.equals(joinFieldA, that.joinFieldA) && Objects.equals(joinFieldB, that.joinFieldB);
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public int hashCode()
{
return Objects.hash(tableA, tableB, joinFieldA, joinFieldB);
}
}
/*******************************************************************************
** Constructor
**
*******************************************************************************/
public JoinGraph(QInstance qInstance)
{
Set<NormalizedJoin> usedJoins = new HashSet<>();
for(QJoinMetaData join : CollectionUtils.nonNullMap(qInstance.getJoins()).values())
{
NormalizedJoin normalizedJoin = new NormalizedJoin(join);
if(usedJoins.contains(normalizedJoin))
{
continue;
}
usedJoins.add(normalizedJoin);
edges.add(new Edge(join.getName(), join.getLeftTable(), join.getRightTable()));
}
}
/*******************************************************************************
**
*******************************************************************************/
public record JoinConnection(String joinTable, String viaJoinName) implements Comparable<JoinConnection>
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public int compareTo(JoinConnection that)
{
Comparator<JoinConnection> comparator = Comparator.comparing((JoinConnection jc) -> jc.joinTable())
.thenComparing((JoinConnection jc) -> jc.viaJoinName());
return (comparator.compare(this, that));
}
}
/*******************************************************************************
**
*******************************************************************************/
public record JoinConnectionList(List<JoinConnection> list) implements Comparable<JoinConnectionList>
{
/*******************************************************************************
**
*******************************************************************************/
public JoinConnectionList copy()
{
return new JoinConnectionList(new ArrayList<>(list));
}
/*******************************************************************************
**
*******************************************************************************/
public int compareTo(JoinConnectionList that)
{
if(this.equals(that))
{
return (0);
}
for(int i = 0; i < Math.min(this.list.size(), that.list.size()); i++)
{
int comp = this.list.get(i).compareTo(that.list.get(i));
if(comp != 0)
{
return (comp);
}
}
return (this.list.size() - that.list.size());
}
/*******************************************************************************
**
*******************************************************************************/
public boolean matchesJoinPath(List<String> joinPath)
{
if(list.size() != joinPath.size())
{
return (false);
}
for(int i = 0; i < list.size(); i++)
{
if(!list.get(i).viaJoinName().equals(joinPath.get(i)))
{
return (false);
}
}
return (true);
}
/*******************************************************************************
**
*******************************************************************************/
public String getJoinNamesAsString()
{
return (StringUtils.join(", ", list().stream().map(jc -> jc.viaJoinName()).toList()));
}
/*******************************************************************************
**
*******************************************************************************/
public List<String> getJoinNamesAsList()
{
return (list().stream().map(jc -> jc.viaJoinName()).toList());
}
}
/*******************************************************************************
**
*******************************************************************************/
public Set<JoinConnectionList> getJoinConnections(String tableName)
{
Set<JoinConnectionList> rs = new TreeSet<>();
doGetJoinConnections(rs, tableName, new ArrayList<>(), new JoinConnectionList(new ArrayList<>()));
return (rs);
}
/*******************************************************************************
**
*******************************************************************************/
private void doGetJoinConnections(Set<JoinConnectionList> joinConnections, String tableName, List<String> path, JoinConnectionList connectionList)
{
for(Edge edge : edges)
{
if(edge.leftTable.equals(tableName) || edge.rightTable.equals(tableName))
{
if(path.contains(edge.joinName))
{
continue;
}
List<String> newPath = new ArrayList<>(path);
newPath.add(edge.joinName);
if(!joinConnectionsContain(joinConnections, newPath))
{
String otherTableName = null;
if(!edge.leftTable.equals(tableName))
{
otherTableName = edge.leftTable;
}
else if(!edge.rightTable.equals(tableName))
{
otherTableName = edge.rightTable;
}
if(otherTableName != null)
{
JoinConnectionList newConnectionList = connectionList.copy();
JoinConnection joinConnection = new JoinConnection(otherTableName, edge.joinName);
newConnectionList.list.add(joinConnection);
joinConnections.add(newConnectionList);
doGetJoinConnections(joinConnections, otherTableName, new ArrayList<>(newPath), newConnectionList);
}
}
}
}
}
/*******************************************************************************
**
*******************************************************************************/
private boolean joinConnectionsContain(Set<JoinConnectionList> joinPaths, List<String> newPath)
{
for(JoinConnectionList joinConnections : joinPaths)
{
List<String> joinConnectionJoins = joinConnections.list.stream().map(jc -> jc.viaJoinName).toList();
if(joinConnectionJoins.equals(newPath))
{
return (true);
}
}
return (false);
}
}

View File

@ -23,21 +23,31 @@ package com.kingsrook.qqq.backend.core.actions.metadata;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionCheckResult;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.metadata.MetaDataInput;
import com.kingsrook.qqq.backend.core.model.actions.metadata.MetaDataOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.AppTreeNode;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendReportMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppChildMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.reporting.QReportMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
/*******************************************************************************
@ -64,34 +74,123 @@ public class MetaDataAction
Map<String, QFrontendTableMetaData> tables = new LinkedHashMap<>();
for(Map.Entry<String, QTableMetaData> entry : metaDataInput.getInstance().getTables().entrySet())
{
tables.put(entry.getKey(), new QFrontendTableMetaData(entry.getValue(), false));
treeNodes.put(entry.getKey(), new AppTreeNode(entry.getValue()));
String tableName = entry.getKey();
QTableMetaData table = entry.getValue();
PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, table);
if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
continue;
}
QBackendMetaData backendForTable = metaDataInput.getInstance().getBackendForTable(tableName);
tables.put(tableName, new QFrontendTableMetaData(metaDataInput, backendForTable, table, false, false));
treeNodes.put(tableName, new AppTreeNode(table));
}
metaDataOutput.setTables(tables);
// addJoinsToTables(tables);
// addJoinedTablesToTables(tables);
////////////////////////////////////////
// map processes to frontend metadata //
////////////////////////////////////////
Map<String, QFrontendProcessMetaData> processes = new LinkedHashMap<>();
for(Map.Entry<String, QProcessMetaData> entry : metaDataInput.getInstance().getProcesses().entrySet())
{
processes.put(entry.getKey(), new QFrontendProcessMetaData(entry.getValue(), false));
treeNodes.put(entry.getKey(), new AppTreeNode(entry.getValue()));
String processName = entry.getKey();
QProcessMetaData process = entry.getValue();
PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, process);
if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
continue;
}
processes.put(processName, new QFrontendProcessMetaData(metaDataInput, process, false));
treeNodes.put(processName, new AppTreeNode(process));
}
metaDataOutput.setProcesses(processes);
//////////////////////////////////////
// map reports to frontend metadata //
//////////////////////////////////////
Map<String, QFrontendReportMetaData> reports = new LinkedHashMap<>();
for(Map.Entry<String, QReportMetaData> entry : metaDataInput.getInstance().getReports().entrySet())
{
String reportName = entry.getKey();
QReportMetaData report = entry.getValue();
PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, report);
if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
continue;
}
reports.put(reportName, new QFrontendReportMetaData(metaDataInput, report, false));
treeNodes.put(reportName, new AppTreeNode(report));
}
metaDataOutput.setReports(reports);
//////////////////////////////////////
// map widgets to frontend metadata //
//////////////////////////////////////
Map<String, QFrontendWidgetMetaData> widgets = new LinkedHashMap<>();
for(Map.Entry<String, QWidgetMetaDataInterface> entry : metaDataInput.getInstance().getWidgets().entrySet())
{
String widgetName = entry.getKey();
QWidgetMetaDataInterface widget = entry.getValue();
PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, widget);
if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
continue;
}
widgets.put(widgetName, new QFrontendWidgetMetaData(metaDataInput, widget));
}
metaDataOutput.setWidgets(widgets);
///////////////////////////////////////////////////////
// sort apps - by sortOrder (integer), then by label //
///////////////////////////////////////////////////////
List<QAppMetaData> sortedApps = metaDataInput.getInstance().getApps().values().stream()
.sorted(Comparator.comparing((QAppMetaData a) -> a.getSortOrder())
.thenComparing((QAppMetaData a) -> a.getLabel()))
.toList();
///////////////////////////////////
// map apps to frontend metadata //
///////////////////////////////////
Map<String, QFrontendAppMetaData> apps = new LinkedHashMap<>();
for(Map.Entry<String, QAppMetaData> entry : metaDataInput.getInstance().getApps().entrySet())
for(QAppMetaData app : sortedApps)
{
apps.put(entry.getKey(), new QFrontendAppMetaData(entry.getValue()));
treeNodes.put(entry.getKey(), new AppTreeNode(entry.getValue()));
String appName = app.getName();
for(QAppChildMetaData child : entry.getValue().getChildren())
PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, app);
if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
apps.get(entry.getKey()).addChild(new AppTreeNode(child));
continue;
}
apps.put(appName, new QFrontendAppMetaData(app, metaDataOutput));
treeNodes.put(appName, new AppTreeNode(app));
if(CollectionUtils.nullSafeHasContents(app.getChildren()))
{
for(QAppChildMetaData child : app.getChildren())
{
if(child instanceof MetaDataWithPermissionRules metaDataWithPermissionRules)
{
PermissionCheckResult childPermissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, metaDataWithPermissionRules);
if(childPermissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
continue;
}
}
apps.get(appName).addChild(new AppTreeNode(child));
}
}
}
metaDataOutput.setApps(apps);
@ -100,15 +199,25 @@ public class MetaDataAction
// organize app tree nodes by their hierarchy //
////////////////////////////////////////////////
List<AppTreeNode> appTree = new ArrayList<>();
for(QAppMetaData appMetaData : metaDataInput.getInstance().getApps().values())
for(QAppMetaData appMetaData : sortedApps)
{
if(appMetaData.getParentAppName() == null)
{
buildAppTree(treeNodes, appTree, appMetaData);
buildAppTree(metaDataInput, treeNodes, appTree, appMetaData);
}
}
metaDataOutput.setAppTree(appTree);
////////////////////////////////////
// add branding metadata if found //
////////////////////////////////////
if(metaDataInput.getInstance().getBranding() != null)
{
metaDataOutput.setBranding(metaDataInput.getInstance().getBranding());
}
metaDataOutput.setEnvironmentValues(metaDataInput.getInstance().getEnvironmentValues());
// todo post-customization - can do whatever w/ the result if you want?
return metaDataOutput;
@ -119,7 +228,7 @@ public class MetaDataAction
/*******************************************************************************
**
*******************************************************************************/
private void buildAppTree(Map<String, AppTreeNode> treeNodes, List<AppTreeNode> nodeList, QAppChildMetaData childMetaData)
private void buildAppTree(MetaDataInput metaDataInput, Map<String, AppTreeNode> treeNodes, List<AppTreeNode> nodeList, QAppChildMetaData childMetaData)
{
AppTreeNode treeNode = treeNodes.get(childMetaData.getName());
if(treeNode == null)
@ -134,7 +243,16 @@ public class MetaDataAction
{
for(QAppChildMetaData child : app.getChildren())
{
buildAppTree(treeNodes, treeNode.getChildren(), child);
if(child instanceof MetaDataWithPermissionRules metaDataWithPermissionRules)
{
PermissionCheckResult permissionResult = PermissionsHelper.getPermissionCheckResult(metaDataInput, metaDataWithPermissionRules);
if(permissionResult.equals(PermissionCheckResult.DENY_HIDE))
{
continue;
}
}
buildAppTree(metaDataInput, treeNodes, treeNode.getChildren(), child);
}
}
}

View File

@ -52,7 +52,7 @@ public class ProcessMetaDataAction
{
throw (new QNotFoundException("Process [" + processMetaDataInput.getProcessName() + "] was not found."));
}
processMetaDataOutput.setProcess(new QFrontendProcessMetaData(process, true));
processMetaDataOutput.setProcess(new QFrontendProcessMetaData(processMetaDataInput, process, true));
// todo post-customization - can do whatever w/ the result if you want

View File

@ -27,6 +27,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
import com.kingsrook.qqq.backend.core.model.actions.metadata.TableMetaDataInput;
import com.kingsrook.qqq.backend.core.model.actions.metadata.TableMetaDataOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.frontend.QFrontendTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@ -52,7 +53,8 @@ public class TableMetaDataAction
{
throw (new QNotFoundException("Table [" + tableMetaDataInput.getTableName() + "] was not found."));
}
tableMetaDataOutput.setTable(new QFrontendTableMetaData(table, true));
QBackendMetaData backendForTable = tableMetaDataInput.getInstance().getBackendForTable(table.getName());
tableMetaDataOutput.setTable(new QFrontendTableMetaData(tableMetaDataInput, backendForTable, table, true, true));
// todo post-customization - can do whatever w/ the result if you want

View File

@ -0,0 +1,204 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
import java.util.Objects;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
/*******************************************************************************
**
*******************************************************************************/
public class AvailablePermission extends QRecordEntity
{
public static final String TABLE_NAME = "availablePermission";
@QField(label = "Permission Name")
private String name;
@QField(label = "Object")
private String objectName;
@QField()
private String objectType;
@QField()
private String permissionType;
/*******************************************************************************
**
*******************************************************************************/
@Override
public boolean equals(Object o)
{
if(this == o)
{
return true;
}
if(o == null || getClass() != o.getClass())
{
return false;
}
AvailablePermission that = (AvailablePermission) o;
return Objects.equals(name, that.name) && Objects.equals(objectName, that.objectName) && Objects.equals(objectType, that.objectType) && Objects.equals(permissionType, that.permissionType);
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public int hashCode()
{
return Objects.hash(name, objectName, objectType, permissionType);
}
/*******************************************************************************
** Getter for name
*******************************************************************************/
public String getName()
{
return (this.name);
}
/*******************************************************************************
** Setter for name
*******************************************************************************/
public void setName(String name)
{
this.name = name;
}
/*******************************************************************************
** Fluent setter for name
*******************************************************************************/
public AvailablePermission withName(String name)
{
this.name = name;
return (this);
}
/*******************************************************************************
** Getter for objectType
*******************************************************************************/
public String getObjectType()
{
return (this.objectType);
}
/*******************************************************************************
** Setter for objectType
*******************************************************************************/
public void setObjectType(String objectType)
{
this.objectType = objectType;
}
/*******************************************************************************
** Fluent setter for objectType
*******************************************************************************/
public AvailablePermission withObjectType(String objectType)
{
this.objectType = objectType;
return (this);
}
/*******************************************************************************
** Getter for permissionType
*******************************************************************************/
public String getPermissionType()
{
return (this.permissionType);
}
/*******************************************************************************
** Setter for permissionType
*******************************************************************************/
public void setPermissionType(String permissionType)
{
this.permissionType = permissionType;
}
/*******************************************************************************
** Fluent setter for permissionType
*******************************************************************************/
public AvailablePermission withPermissionType(String permissionType)
{
this.permissionType = permissionType;
return (this);
}
/*******************************************************************************
** Getter for objectName
*******************************************************************************/
public String getObjectName()
{
return (this.objectName);
}
/*******************************************************************************
** Setter for objectName
*******************************************************************************/
public void setObjectName(String objectName)
{
this.objectName = objectName;
}
/*******************************************************************************
** Fluent setter for objectName
*******************************************************************************/
public AvailablePermission withObjectName(String objectName)
{
this.objectName = objectName;
return (this);
}
}

View File

@ -0,0 +1,67 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
/*******************************************************************************
**
*******************************************************************************/
public class BulkTableActionProcessPermissionChecker implements CustomPermissionChecker
{
private static final QLogger LOG = QLogger.getLogger(BulkTableActionProcessPermissionChecker.class);
/*******************************************************************************
**
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput actionInput, MetaDataWithPermissionRules metaDataWithPermissionRules) throws QPermissionDeniedException
{
String processName = metaDataWithPermissionRules.getName();
if(processName != null && processName.indexOf('.') > -1)
{
String[] parts = processName.split("\\.", 2);
String tableName = parts[0];
String bulkActionName = parts[1];
AbstractTableActionInput tableActionInput = new AbstractTableActionInput();
tableActionInput.setTableName(tableName);
switch(bulkActionName)
{
case "bulkInsert" -> PermissionsHelper.checkTablePermissionThrowing(tableActionInput, TablePermissionSubType.INSERT);
case "bulkEdit" -> PermissionsHelper.checkTablePermissionThrowing(tableActionInput, TablePermissionSubType.EDIT);
case "bulkDelete" -> PermissionsHelper.checkTablePermissionThrowing(tableActionInput, TablePermissionSubType.DELETE);
default -> LOG.warn("Unexpected bulk action name when checking permissions for process: " + processName);
}
}
}
}

View File

@ -0,0 +1,41 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
/*******************************************************************************
**
*******************************************************************************/
public interface CustomPermissionChecker
{
/*******************************************************************************
**
*******************************************************************************/
void checkPermissionsThrowing(AbstractActionInput actionInput, MetaDataWithPermissionRules metaDataWithPermissionRules) throws QPermissionDeniedException;
}

View File

@ -0,0 +1,33 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
/*******************************************************************************
**
*******************************************************************************/
public enum PermissionCheckResult
{
ALLOW,
DENY_HIDE,
DENY_DISABLE;
}

View File

@ -0,0 +1,36 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
/*******************************************************************************
**
*******************************************************************************/
sealed interface PermissionSubType permits PrivatePermissionSubType, TablePermissionSubType
{
/*******************************************************************************
**
*******************************************************************************/
String getPermissionSuffix();
}

View File

@ -0,0 +1,602 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
import java.io.Serializable;
import java.util.Collection;
import java.util.Collections;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.stream.Collectors;
import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaDataInterface;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.DenyBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithName;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.reporting.QReportMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
/*******************************************************************************
**
*******************************************************************************/
public class PermissionsHelper
{
private static final QLogger LOG = QLogger.getLogger(PermissionsHelper.class);
/*******************************************************************************
**
*******************************************************************************/
public static void checkTablePermissionThrowing(AbstractTableActionInput tableActionInput, TablePermissionSubType permissionSubType) throws QPermissionDeniedException
{
checkTablePermissionThrowing(tableActionInput, tableActionInput.getTableName(), permissionSubType);
}
/*******************************************************************************
**
*******************************************************************************/
private static void checkTablePermissionThrowing(AbstractActionInput actionInput, String tableName, TablePermissionSubType permissionSubType) throws QPermissionDeniedException
{
warnAboutPermissionSubTypeForTables(permissionSubType);
QTableMetaData table = QContext.getQInstance().getTable(tableName);
commonCheckPermissionThrowing(getEffectivePermissionRules(table, QContext.getQInstance()), permissionSubType, table.getName());
}
/*******************************************************************************
**
*******************************************************************************/
public static String getTablePermissionName(String tableName, TablePermissionSubType permissionSubType)
{
QInstance qInstance = QContext.getQInstance();
QPermissionRules rules = getEffectivePermissionRules(qInstance.getTable(tableName), qInstance);
String permissionBaseName = getEffectivePermissionBaseName(rules, tableName);
return (getPermissionName(permissionBaseName, permissionSubType));
}
/*******************************************************************************
**
*******************************************************************************/
public static boolean hasTablePermission(AbstractActionInput actionInput, String tableName, TablePermissionSubType permissionSubType)
{
try
{
checkTablePermissionThrowing(actionInput, tableName, permissionSubType);
return (true);
}
catch(QPermissionDeniedException e)
{
return (false);
}
}
/*******************************************************************************
**
*******************************************************************************/
public static PermissionCheckResult getPermissionCheckResult(AbstractActionInput actionInput, MetaDataWithPermissionRules metaDataWithPermissionRules)
{
QPermissionRules rules = getEffectivePermissionRules(metaDataWithPermissionRules, QContext.getQInstance());
String permissionBaseName = getEffectivePermissionBaseName(rules, metaDataWithPermissionRules.getName());
switch(rules.getLevel())
{
case NOT_PROTECTED:
{
/////////////////////////////////////////////////
// if the entity isn't protected, always ALLOW //
/////////////////////////////////////////////////
return PermissionCheckResult.ALLOW;
}
case HAS_ACCESS_PERMISSION:
{
////////////////////////////////////////////////////////////////////////
// if the entity just has a 'has access', then check for 'has access' //
////////////////////////////////////////////////////////////////////////
return getPermissionCheckResult(actionInput, rules, permissionBaseName, metaDataWithPermissionRules, PrivatePermissionSubType.HAS_ACCESS);
}
case READ_WRITE_PERMISSIONS:
{
////////////////////////////////////////////////////////////////
// if the table is configured w/ read/write, check for either //
////////////////////////////////////////////////////////////////
if(metaDataWithPermissionRules instanceof QTableMetaData)
{
return getPermissionCheckResult(actionInput, rules, permissionBaseName, metaDataWithPermissionRules, PrivatePermissionSubType.READ, PrivatePermissionSubType.WRITE);
}
return getPermissionCheckResult(actionInput, rules, permissionBaseName, metaDataWithPermissionRules, PrivatePermissionSubType.HAS_ACCESS);
}
case READ_INSERT_EDIT_DELETE_PERMISSIONS:
{
//////////////////////////////////////////////////////////////////////////
// if the table is configured w/ read/insert/edit/delete, check for any //
//////////////////////////////////////////////////////////////////////////
if(metaDataWithPermissionRules instanceof QTableMetaData)
{
return getPermissionCheckResult(actionInput, rules, permissionBaseName, metaDataWithPermissionRules, TablePermissionSubType.READ, TablePermissionSubType.INSERT, TablePermissionSubType.EDIT, TablePermissionSubType.DELETE);
}
return getPermissionCheckResult(actionInput, rules, permissionBaseName, metaDataWithPermissionRules, PrivatePermissionSubType.HAS_ACCESS);
}
default:
{
return getPermissionDeniedCheckResult(rules);
}
}
}
/*******************************************************************************
**
*******************************************************************************/
public static void checkProcessPermissionThrowing(AbstractActionInput actionInput, String processName) throws QPermissionDeniedException
{
checkProcessPermissionThrowing(actionInput, processName, Collections.emptyMap());
}
/*******************************************************************************
**
*******************************************************************************/
public static void checkProcessPermissionThrowing(AbstractActionInput actionInput, String processName, Map<String, Serializable> processValues) throws QPermissionDeniedException
{
QProcessMetaData process = QContext.getQInstance().getProcess(processName);
QPermissionRules effectivePermissionRules = getEffectivePermissionRules(process, QContext.getQInstance());
if(effectivePermissionRules.getCustomPermissionChecker() != null)
{
/////////////////////////////////////
// todo - avoid stack overflows... //
/////////////////////////////////////
CustomPermissionChecker customPermissionChecker = QCodeLoader.getAdHoc(CustomPermissionChecker.class, effectivePermissionRules.getCustomPermissionChecker());
customPermissionChecker.checkPermissionsThrowing(actionInput, process);
return;
}
commonCheckPermissionThrowing(effectivePermissionRules, PrivatePermissionSubType.HAS_ACCESS, process.getName());
}
/*******************************************************************************
**
*******************************************************************************/
public static boolean hasProcessPermission(AbstractActionInput actionInput, String processName)
{
try
{
checkProcessPermissionThrowing(actionInput, processName);
return (true);
}
catch(QPermissionDeniedException e)
{
return (false);
}
}
/*******************************************************************************
**
*******************************************************************************/
public static void checkAppPermissionThrowing(AbstractActionInput actionInput, String appName) throws QPermissionDeniedException
{
QAppMetaData app = QContext.getQInstance().getApp(appName);
commonCheckPermissionThrowing(getEffectivePermissionRules(app, QContext.getQInstance()), PrivatePermissionSubType.HAS_ACCESS, app.getName());
}
/*******************************************************************************
**
*******************************************************************************/
public static boolean hasAppPermission(AbstractActionInput actionInput, String appName)
{
try
{
checkAppPermissionThrowing(actionInput, appName);
return (true);
}
catch(QPermissionDeniedException e)
{
return (false);
}
}
/*******************************************************************************
**
*******************************************************************************/
public static void checkReportPermissionThrowing(AbstractActionInput actionInput, String reportName) throws QPermissionDeniedException
{
QReportMetaData report = QContext.getQInstance().getReport(reportName);
commonCheckPermissionThrowing(getEffectivePermissionRules(report, QContext.getQInstance()), PrivatePermissionSubType.HAS_ACCESS, report.getName());
}
/*******************************************************************************
**
*******************************************************************************/
public static boolean hasReportPermission(AbstractActionInput actionInput, String reportName)
{
try
{
checkReportPermissionThrowing(actionInput, reportName);
return (true);
}
catch(QPermissionDeniedException e)
{
return (false);
}
}
/*******************************************************************************
**
*******************************************************************************/
public static void checkWidgetPermissionThrowing(AbstractActionInput actionInput, String widgetName) throws QPermissionDeniedException
{
QWidgetMetaDataInterface widget = QContext.getQInstance().getWidget(widgetName);
commonCheckPermissionThrowing(getEffectivePermissionRules(widget, QContext.getQInstance()), PrivatePermissionSubType.HAS_ACCESS, widget.getName());
}
/*******************************************************************************
**
*******************************************************************************/
public static boolean hasWidgetPermission(AbstractActionInput actionInput, String widgetName)
{
try
{
checkWidgetPermissionThrowing(actionInput, widgetName);
return (true);
}
catch(QPermissionDeniedException e)
{
return (false);
}
}
/*******************************************************************************
**
*******************************************************************************/
public static Collection<String> getAllAvailablePermissionNames(QInstance instance)
{
return (getAllAvailablePermissions(instance).stream()
.map(AvailablePermission::getName)
.collect(Collectors.toCollection(LinkedHashSet::new)));
}
/*******************************************************************************
**
*******************************************************************************/
public static Collection<AvailablePermission> getAllAvailablePermissions(QInstance instance)
{
Collection<AvailablePermission> rs = new LinkedHashSet<>();
for(QTableMetaData tableMetaData : instance.getTables().values())
{
if(tableMetaData.getIsHidden())
{
continue;
}
QPermissionRules rules = getEffectivePermissionRules(tableMetaData, instance);
String baseName = getEffectivePermissionBaseName(rules, tableMetaData.getName());
for(TablePermissionSubType permissionSubType : TablePermissionSubType.values())
{
addEffectiveAvailablePermission(rules, permissionSubType, rs, baseName, tableMetaData, "Table");
}
}
for(QProcessMetaData processMetaData : instance.getProcesses().values())
{
if(processMetaData.getIsHidden())
{
continue;
}
QPermissionRules rules = getEffectivePermissionRules(processMetaData, instance);
String baseName = getEffectivePermissionBaseName(rules, processMetaData.getName());
addEffectiveAvailablePermission(rules, PrivatePermissionSubType.HAS_ACCESS, rs, baseName, processMetaData, "Process");
}
for(QAppMetaData appMetaData : instance.getApps().values())
{
QPermissionRules rules = getEffectivePermissionRules(appMetaData, instance);
String baseName = getEffectivePermissionBaseName(rules, appMetaData.getName());
addEffectiveAvailablePermission(rules, PrivatePermissionSubType.HAS_ACCESS, rs, baseName, appMetaData, "App");
}
for(QReportMetaData reportMetaData : instance.getReports().values())
{
QPermissionRules rules = getEffectivePermissionRules(reportMetaData, instance);
String baseName = getEffectivePermissionBaseName(rules, reportMetaData.getName());
addEffectiveAvailablePermission(rules, PrivatePermissionSubType.HAS_ACCESS, rs, baseName, reportMetaData, "Report");
}
for(QWidgetMetaDataInterface widgetMetaData : instance.getWidgets().values())
{
QPermissionRules rules = getEffectivePermissionRules(widgetMetaData, instance);
String baseName = getEffectivePermissionBaseName(rules, widgetMetaData.getName());
addEffectiveAvailablePermission(rules, PrivatePermissionSubType.HAS_ACCESS, rs, baseName, widgetMetaData, "Widget");
}
return (rs);
}
/*******************************************************************************
**
*******************************************************************************/
private static void addEffectiveAvailablePermission(QPermissionRules rules, PermissionSubType permissionSubType, Collection<AvailablePermission> rs, String baseName, MetaDataWithName metaDataWithName, String objectType)
{
PermissionSubType effectivePermissionSubType = getEffectivePermissionSubType(rules, permissionSubType);
if(effectivePermissionSubType != null)
{
rs.add(new AvailablePermission()
.withName(getPermissionName(baseName, effectivePermissionSubType))
.withObjectName(metaDataWithName.getLabel())
.withPermissionType(effectivePermissionSubType.toString())
.withObjectType(objectType));
}
}
/*******************************************************************************
**
*******************************************************************************/
public static QPermissionRules getEffectivePermissionRules(MetaDataWithPermissionRules metaDataWithPermissionRules, QInstance instance)
{
if(metaDataWithPermissionRules.getPermissionRules() == null)
{
LOG.warn("Null permission rules on meta data object [" + metaDataWithPermissionRules.getClass().getSimpleName() + "][" + metaDataWithPermissionRules.getName() + "] - does the instance need enriched? Returning instance default rules.");
return (instance.getDefaultPermissionRules());
}
return (metaDataWithPermissionRules.getPermissionRules());
}
/*******************************************************************************
**
*******************************************************************************/
static boolean hasPermission(QSession session, String permissionBaseName, PermissionSubType permissionSubType)
{
if(permissionSubType == null)
{
return (true);
}
String permissionName = getPermissionName(permissionBaseName, permissionSubType);
return (session.hasPermission(permissionName));
}
/*******************************************************************************
**
*******************************************************************************/
static PermissionCheckResult getPermissionCheckResult(AbstractActionInput actionInput, QPermissionRules rules, String permissionBaseName, MetaDataWithPermissionRules metaDataWithPermissionRules, PermissionSubType... permissionSubTypes)
{
for(PermissionSubType permissionSubType : permissionSubTypes)
{
PermissionSubType effectivePermissionSubType = getEffectivePermissionSubType(rules, permissionSubType);
if(rules.getCustomPermissionChecker() != null)
{
try
{
CustomPermissionChecker customPermissionChecker = QCodeLoader.getAdHoc(CustomPermissionChecker.class, rules.getCustomPermissionChecker());
customPermissionChecker.checkPermissionsThrowing(actionInput, metaDataWithPermissionRules);
return (PermissionCheckResult.ALLOW);
}
catch(QPermissionDeniedException e)
{
return (getPermissionDeniedCheckResult(rules));
}
}
if(hasPermission(QContext.getQSession(), permissionBaseName, effectivePermissionSubType))
{
return (PermissionCheckResult.ALLOW);
}
}
return (getPermissionDeniedCheckResult(rules));
}
/*******************************************************************************
**
*******************************************************************************/
static String getEffectivePermissionBaseName(QPermissionRules rules, String standardName)
{
if(rules != null && StringUtils.hasContent(rules.getPermissionBaseName()))
{
return (rules.getPermissionBaseName());
}
return (standardName);
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("checkstyle:indentation")
static PermissionSubType getEffectivePermissionSubType(QPermissionRules rules, PermissionSubType originalPermissionSubType)
{
if(rules == null || rules.getLevel() == null)
{
return (originalPermissionSubType);
}
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// if the original permission sub-type is "hasAccess" - then this is a check for a process/report/widget. //
// in that case - never return the table-level read/write/insert/edit/delete options //
////////////////////////////////////////////////////////////////////////////////////////////////////////////
if(PrivatePermissionSubType.HAS_ACCESS.equals(originalPermissionSubType))
{
return switch(rules.getLevel())
{
case NOT_PROTECTED -> null;
default -> PrivatePermissionSubType.HAS_ACCESS;
};
}
else
{
////////////////////////////////////////////////////////////////////////////////////////////////////////
// else, this is a table check - so - based on the rules being used for this table, map the requested //
// permission sub-type to what we expect to be set for the table //
////////////////////////////////////////////////////////////////////////////////////////////////////////
return switch(rules.getLevel())
{
case NOT_PROTECTED -> null;
case HAS_ACCESS_PERMISSION -> PrivatePermissionSubType.HAS_ACCESS;
case READ_WRITE_PERMISSIONS ->
{
if(PrivatePermissionSubType.READ.equals(originalPermissionSubType) || PrivatePermissionSubType.WRITE.equals(originalPermissionSubType))
{
yield (originalPermissionSubType);
}
else if(TablePermissionSubType.INSERT.equals(originalPermissionSubType) || TablePermissionSubType.EDIT.equals(originalPermissionSubType) || TablePermissionSubType.DELETE.equals(originalPermissionSubType))
{
yield (PrivatePermissionSubType.WRITE);
}
else if(TablePermissionSubType.READ.equals(originalPermissionSubType))
{
yield (PrivatePermissionSubType.READ);
}
else
{
throw new IllegalStateException("Unexpected permissionSubType: " + originalPermissionSubType);
}
}
case READ_INSERT_EDIT_DELETE_PERMISSIONS -> originalPermissionSubType;
};
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void commonCheckPermissionThrowing(QPermissionRules rules, PermissionSubType permissionSubType, String name) throws QPermissionDeniedException
{
PermissionSubType effectivePermissionSubType = getEffectivePermissionSubType(rules, permissionSubType);
String permissionBaseName = getEffectivePermissionBaseName(rules, name);
if(effectivePermissionSubType == null)
{
return;
}
if(!hasPermission(QContext.getQSession(), permissionBaseName, effectivePermissionSubType))
{
// LOG.debug("Throwing permission denied for: " + getPermissionName(permissionBaseName, effectivePermissionSubType) + " for " + QContext.getQSession().getUser());
throw (new QPermissionDeniedException("Permission denied."));
}
}
/*******************************************************************************
**
*******************************************************************************/
private static String getPermissionName(String permissionBaseName, PermissionSubType permissionSubType)
{
return permissionBaseName + "." + permissionSubType.getPermissionSuffix();
}
/*******************************************************************************
**
*******************************************************************************/
private static PermissionCheckResult getPermissionDeniedCheckResult(QPermissionRules rules)
{
if(rules == null || rules.getDenyBehavior() == null || rules.getDenyBehavior().equals(DenyBehavior.HIDDEN))
{
return (PermissionCheckResult.DENY_HIDE);
}
else
{
return (PermissionCheckResult.DENY_DISABLE);
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void warnAboutPermissionSubTypeForTables(PermissionSubType permissionSubType)
{
if(permissionSubType == null)
{
return;
}
if(permissionSubType == PrivatePermissionSubType.HAS_ACCESS)
{
LOG.warn("PermissionSubType.HAS_ACCESS should not be checked for a table");
}
}
}

View File

@ -0,0 +1,56 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
/*******************************************************************************
**
*******************************************************************************/
enum PrivatePermissionSubType implements PermissionSubType
{
HAS_ACCESS("hasAccess"), // for processes, reports, etc - basically, not tables.
READ("read"), // for a table in read/write mode - or - for read (query, get, count) on a table in full-mode
WRITE("write");
private final String permissionSuffix;
/*******************************************************************************
**
*******************************************************************************/
PrivatePermissionSubType(String permissionSuffix)
{
this.permissionSuffix = permissionSuffix;
}
/*******************************************************************************
** Getter for permissionSuffix
*******************************************************************************/
public String getPermissionSuffix()
{
return (this.permissionSuffix);
}
}

View File

@ -0,0 +1,53 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
/*******************************************************************************
**
*******************************************************************************/
public class ReportProcessPermissionChecker implements CustomPermissionChecker
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput actionInput, MetaDataWithPermissionRules metaDataWithPermissionRules) throws QPermissionDeniedException
{
if(actionInput instanceof RunProcessInput runProcessInput)
{
String reportName = runProcessInput.getValueString("reportName");
if(reportName != null)
{
PermissionsHelper.checkReportPermissionThrowing(actionInput, reportName);
}
}
}
}

View File

@ -0,0 +1,57 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.permissions;
/*******************************************************************************
**
*******************************************************************************/
public enum TablePermissionSubType implements PermissionSubType
{
READ("read"), // for a table in read/write mode - or - for read (query, get, count) on a table in full-mode
INSERT("insert"), // for table-insert.
EDIT("edit"), // for table-edit.
DELETE("delete"); // for table-delete.
private final String permissionSuffix;
/*******************************************************************************
**
*******************************************************************************/
TablePermissionSubType(String permissionSuffix)
{
this.permissionSuffix = permissionSuffix;
}
/*******************************************************************************
** Getter for permissionSuffix
*******************************************************************************/
public String getPermissionSuffix()
{
return (this.permissionSuffix);
}
}

View File

@ -39,10 +39,16 @@ public interface QProcessCallback
/*******************************************************************************
** Get the filter query for this callback.
*******************************************************************************/
QQueryFilter getQueryFilter();
default QQueryFilter getQueryFilter()
{
return (null);
}
/*******************************************************************************
** Get the field values for this callback.
*******************************************************************************/
Map<String, Serializable> getFieldValues(List<QFieldMetaData> fields);
default Map<String, Serializable> getFieldValues(List<QFieldMetaData> fields)
{
return (null);
}
}

View File

@ -0,0 +1,68 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2024. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.processes;
import java.io.Serializable;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
/*******************************************************************************
** Constructor for commonly used QProcessCallback's
*******************************************************************************/
public class QProcessCallbackFactory
{
/*******************************************************************************
**
*******************************************************************************/
public static QProcessCallback forFilter(QQueryFilter filter)
{
return new QProcessCallback()
{
/*******************************************************************************
**
*******************************************************************************/
@Override
public QQueryFilter getQueryFilter()
{
return (filter);
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public Map<String, Serializable> getFieldValues(List<QFieldMetaData> fields)
{
return (Collections.emptyMap());
}
};
}
}

View File

@ -31,6 +31,7 @@ import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
@ -42,8 +43,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMet
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
/*******************************************************************************
@ -52,7 +51,7 @@ import org.apache.logging.log4j.Logger;
*******************************************************************************/
public class RunBackendStepAction
{
private static final Logger LOG = LogManager.getLogger(RunBackendStepAction.class);
private static final QLogger LOG = QLogger.getLogger(RunBackendStepAction.class);
@ -107,7 +106,7 @@ public class RunBackendStepAction
return;
}
List<QFieldMetaData> fieldsToGet = new ArrayList<>();
List<QFieldMetaData> fieldsToGet = new ArrayList<>();
List<QFieldMetaData> requiredFieldsMissing = new ArrayList<>();
for(QFieldMetaData field : inputMetaData.getFieldList())
{
@ -175,8 +174,7 @@ public class RunBackendStepAction
{
if(CollectionUtils.nullSafeIsEmpty(runBackendStepInput.getRecords()))
{
QueryInput queryInput = new QueryInput(runBackendStepInput.getInstance());
queryInput.setSession(runBackendStepInput.getSession());
QueryInput queryInput = new QueryInput();
queryInput.setTableName(inputMetaData.getRecordListMetaData().getTableName());
// todo - handle this being async (e.g., http)

View File

@ -23,37 +23,72 @@ package com.kingsrook.qqq.backend.core.actions.processes;
import java.io.Serializable;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.NoCodeWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessState;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.NoCodeWidgetFrontendComponentMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.processes.implementations.basepull.BasepullConfiguration;
import com.kingsrook.qqq.backend.core.state.InMemoryStateProvider;
import com.kingsrook.qqq.backend.core.state.StateProviderInterface;
import com.kingsrook.qqq.backend.core.state.StateType;
import com.kingsrook.qqq.backend.core.state.UUIDAndTypeStateKey;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import org.apache.commons.lang.BooleanUtils;
/*******************************************************************************
** Action handler for running q-processes (which are a sequence of q-functions).
** Action handler for running q-processes (which are a sequence of q-steps).
*
*******************************************************************************/
public class RunProcessAction
{
private static final Logger LOG = LogManager.getLogger(RunProcessAction.class);
private static final QLogger LOG = QLogger.getLogger(RunProcessAction.class);
public static final String BASEPULL_THIS_RUNTIME_KEY = "basepullThisRuntimeKey";
public static final String BASEPULL_LAST_RUNTIME_KEY = "basepullLastRuntimeKey";
public static final String BASEPULL_TIMESTAMP_FIELD = "basepullTimestampField";
public static final String BASEPULL_CONFIGURATION = "basepullConfiguration";
////////////////////////////////////////////////////////////////////////////////////////////////
// indicator that the timestamp field should be updated - e.g., the execute step is finished. //
////////////////////////////////////////////////////////////////////////////////////////////////
public static final String BASEPULL_READY_TO_UPDATE_TIMESTAMP_FIELD = "basepullReadyToUpdateTimestamp";
public static final String BASEPULL_DID_QUERY_USING_TIMESTAMP_FIELD = "basepullDidQueryUsingTimestamp";
@ -82,16 +117,41 @@ public class RunProcessAction
runProcessOutput.setProcessUUID(runProcessInput.getProcessUUID());
UUIDAndTypeStateKey stateKey = new UUIDAndTypeStateKey(UUID.fromString(runProcessInput.getProcessUUID()), StateType.PROCESS_STATUS);
ProcessState processState = primeProcessState(runProcessInput, stateKey);
ProcessState processState = primeProcessState(runProcessInput, stateKey, process);
/////////////////////////////////////////////////////////
// if process is 'basepull' style, keep track of 'now' //
/////////////////////////////////////////////////////////
BasepullConfiguration basepullConfiguration = process.getBasepullConfiguration();
if(basepullConfiguration != null)
{
///////////////////////////////////////
// get the stored basepull timestamp //
///////////////////////////////////////
persistLastRunTime(runProcessInput, process, basepullConfiguration);
}
// todo - custom routing
List<QStepMetaData> stepList = getAvailableStepList(process, runProcessInput);
try
{
String lastStepName = runProcessInput.getStartAfterStep();
STEP_LOOP:
for(QStepMetaData step : stepList)
while(true)
{
if(step instanceof QFrontendStepMetaData)
///////////////////////////////////////////////////////////////////////////////////////////////////////
// always refresh the step list - as any step that runs can modify it (in the process state). //
// this is why we don't do a loop over the step list - as we'd get ConcurrentModificationExceptions. //
///////////////////////////////////////////////////////////////////////////////////////////////////////
List<QStepMetaData> stepList = getAvailableStepList(processState, process, lastStepName);
if(stepList.isEmpty())
{
break;
}
QStepMetaData step = stepList.get(0);
lastStepName = step.getName();
if(step instanceof QFrontendStepMetaData frontendStep)
{
////////////////////////////////////////////////////////////////
// Handle what to do with frontend steps, per request setting //
@ -100,13 +160,15 @@ public class RunProcessAction
{
case BREAK ->
{
LOG.info("Breaking process [" + process.getName() + "] at frontend step (as requested by caller): " + step.getName());
LOG.trace("Breaking process [" + process.getName() + "] at frontend step (as requested by caller): " + step.getName());
processFrontendStepFieldDefaultValues(processState, frontendStep);
processFrontendComponents(processState, frontendStep);
processState.setNextStepName(step.getName());
break STEP_LOOP;
}
case SKIP ->
{
LOG.info("Skipping frontend step [" + step.getName() + "] in process [" + process.getName() + "] (as requested by caller)");
LOG.trace("Skipping frontend step [" + step.getName() + "] in process [" + process.getName() + "] (as requested by caller)");
//////////////////////////////////////////////////////////////////////
// much less error prone in case this code changes in the future... //
@ -116,7 +178,7 @@ public class RunProcessAction
}
case FAIL ->
{
LOG.info("Throwing error for frontend step [" + step.getName() + "] in process [" + process.getName() + "] (as requested by caller)");
LOG.trace("Throwing error for frontend step [" + step.getName() + "] in process [" + process.getName() + "] (as requested by caller)");
throw (new QException("Failing process at step " + step.getName() + " (as requested, to fail on frontend steps)"));
}
default -> throw new IllegalStateException("Unexpected value: " + runProcessInput.getFrontendStepBehavior());
@ -127,6 +189,7 @@ public class RunProcessAction
///////////////////////
// Run backend steps //
///////////////////////
LOG.debug("Running backend step [" + step.getName() + "] in process [" + process.getName() + "]");
runBackendStep(runProcessInput, process, runProcessOutput, stateKey, backendStepMetaData, process, processState);
}
else
@ -137,6 +200,18 @@ public class RunProcessAction
throw (new QException("Unsure how to run a step of type: " + step.getClass().getName()));
}
}
///////////////////////////////////////////////////////////////////////////
// if 'basepull' style process, update the stored basepull timestamp //
// but only when we've been signaled to do so - i.e., only if we did our //
// query using the timestamp field, and only after an Execute step runs. //
///////////////////////////////////////////////////////////////////////////
if(basepullConfiguration != null
&& BooleanUtils.isTrue(ValueUtils.getValueAsBoolean(runProcessInput.getValue(BASEPULL_DID_QUERY_USING_TIMESTAMP_FIELD)))
&& BooleanUtils.isTrue(ValueUtils.getValueAsBoolean(runProcessInput.getValue(BASEPULL_READY_TO_UPDATE_TIMESTAMP_FIELD))))
{
storeLastRunTime(runProcessInput, process, basepullConfiguration);
}
}
catch(QException qe)
{
@ -165,11 +240,47 @@ public class RunProcessAction
/*******************************************************************************
**
*******************************************************************************/
private void processFrontendComponents(ProcessState processState, QFrontendStepMetaData frontendStep) throws QException
{
for(QFrontendComponentMetaData component : CollectionUtils.nonNullList(frontendStep.getComponents()))
{
if(component instanceof NoCodeWidgetFrontendComponentMetaData noCodeWidgetComponent)
{
NoCodeWidgetRenderer noCodeWidgetRenderer = new NoCodeWidgetRenderer();
Map<String, Object> context = noCodeWidgetRenderer.initContext(null);
context.putAll(processState.getValues());
String html = noCodeWidgetRenderer.renderOutputs(context, noCodeWidgetComponent.getOutputs());
processState.getValues().put(frontendStep.getName() + ".html", html);
}
}
}
/*******************************************************************************
**
*******************************************************************************/
private void processFrontendStepFieldDefaultValues(ProcessState processState, QFrontendStepMetaData step)
{
for(QFieldMetaData formField : CollectionUtils.mergeLists(step.getFormFields(), step.getInputFields(), step.getViewFields(), step.getOutputFields()))
{
if(formField.getDefaultValue() != null && processState.getValues().get(formField.getName()) == null)
{
processState.getValues().put(formField.getName(), formField.getDefaultValue());
}
}
}
/*******************************************************************************
** When we start running a process (or resuming it), get data in the RunProcessRequest
** either from the state provider (if they're found, for a resume).
*******************************************************************************/
ProcessState primeProcessState(RunProcessInput runProcessInput, UUIDAndTypeStateKey stateKey) throws QException
ProcessState primeProcessState(RunProcessInput runProcessInput, UUIDAndTypeStateKey stateKey, QProcessMetaData process) throws QException
{
Optional<ProcessState> optionalProcessState = loadState(stateKey);
if(optionalProcessState.isEmpty())
@ -177,11 +288,14 @@ public class RunProcessAction
if(runProcessInput.getStartAfterStep() == null)
{
///////////////////////////////////////////////////////////////////////////////////
// this is fine - it means its our first time running in the backend. //
// This condition (no state in state-provider, and no start-after-step) means //
// that we're starting a new process! Init the process state here, then //
// Go ahead and store the state that we have (e.g., w/ initial records & values) //
///////////////////////////////////////////////////////////////////////////////////
storeState(stateKey, runProcessInput.getProcessState());
optionalProcessState = Optional.of(runProcessInput.getProcessState());
ProcessState processState = runProcessInput.getProcessState();
processState.setStepList(process.getStepList().stream().map(QStepMetaData::getName).toList());
storeState(stateKey, processState);
optionalProcessState = Optional.of(processState);
}
else
{
@ -227,13 +341,33 @@ public class RunProcessAction
*******************************************************************************/
private void runBackendStep(RunProcessInput runProcessInput, QProcessMetaData process, RunProcessOutput runProcessOutput, UUIDAndTypeStateKey stateKey, QBackendStepMetaData backendStep, QProcessMetaData qProcessMetaData, ProcessState processState) throws Exception
{
RunBackendStepInput runBackendStepInput = new RunBackendStepInput(runProcessInput.getInstance(), processState);
RunBackendStepInput runBackendStepInput = new RunBackendStepInput(processState);
runBackendStepInput.setProcessName(process.getName());
runBackendStepInput.setStepName(backendStep.getName());
runBackendStepInput.setTableName(process.getTableName());
runBackendStepInput.setSession(runProcessInput.getSession());
runBackendStepInput.setCallback(runProcessInput.getCallback());
runBackendStepInput.setFrontendStepBehavior(runProcessInput.getFrontendStepBehavior());
runBackendStepInput.setAsyncJobCallback(runProcessInput.getAsyncJobCallback());
runBackendStepInput.setTableName(process.getTableName());
if(!StringUtils.hasContent(runBackendStepInput.getTableName()))
{
////////////////////////////////////////////////////////////////
// help support generic (e.g., not tied-to-a-table) processes //
////////////////////////////////////////////////////////////////
if(runProcessInput.getValue("tableName") != null)
{
runBackendStepInput.setTableName(ValueUtils.getValueAsString(runProcessInput.getValue("tableName")));
}
}
///////////////////////////////////////////////////////////////
// if 'basepull' values are in the inputs, add to step input //
///////////////////////////////////////////////////////////////
if(runProcessInput.getValues().containsKey(BASEPULL_LAST_RUNTIME_KEY))
{
runBackendStepInput.setBasepullLastRunTime((Instant) runProcessInput.getValues().get(BASEPULL_LAST_RUNTIME_KEY));
}
RunBackendStepOutput lastFunctionResult = new RunBackendStepAction().execute(runBackendStepInput);
storeState(stateKey, lastFunctionResult.getProcessState());
@ -249,41 +383,63 @@ public class RunProcessAction
/*******************************************************************************
** Get the list of steps which are eligible to run.
*******************************************************************************/
private List<QStepMetaData> getAvailableStepList(QProcessMetaData process, RunProcessInput runProcessInput)
private List<QStepMetaData> getAvailableStepList(ProcessState processState, QProcessMetaData process, String lastStep) throws QException
{
if(runProcessInput.getStartAfterStep() == null)
if(lastStep == null)
{
/////////////////////////////////////////////////////////////////////////////
// if the caller did not supply a 'startAfterStep', then use the full list //
/////////////////////////////////////////////////////////////////////////////
return (process.getStepList());
///////////////////////////////////////////////////////////////////////
// if the caller did not supply a 'lastStep', then use the full list //
///////////////////////////////////////////////////////////////////////
return (stepNamesToSteps(process, processState.getStepList()));
}
else
{
////////////////////////////////////////////////////////////////////////////////
// else, loop until the startAfterStep is found, and return the ones after it //
////////////////////////////////////////////////////////////////////////////////
boolean foundStartAfterStep = false;
List<QStepMetaData> rs = new ArrayList<>();
////////////////////////////////////////////////////////////////////////////
// else, loop until the 'lastStep' is found, and return the ones after it //
////////////////////////////////////////////////////////////////////////////
boolean foundLastStep = false;
List<String> validStepNames = new ArrayList<>();
for(QStepMetaData step : process.getStepList())
for(String stepName : processState.getStepList())
{
if(foundStartAfterStep)
if(foundLastStep)
{
rs.add(step);
validStepNames.add(stepName);
}
if(step.getName().equals(runProcessInput.getStartAfterStep()))
if(stepName.equals(lastStep))
{
foundStartAfterStep = true;
foundLastStep = true;
}
}
return (rs);
return (stepNamesToSteps(process, validStepNames));
}
}
/*******************************************************************************
**
*******************************************************************************/
private List<QStepMetaData> stepNamesToSteps(QProcessMetaData process, List<String> stepNames) throws QException
{
List<QStepMetaData> result = new ArrayList<>();
for(String stepName : stepNames)
{
QStepMetaData step = process.getStep(stepName);
if(step == null)
{
throw (new QException("Could not find a step named [" + stepName + "] in this process."));
}
result.add(step);
}
return (result);
}
/*******************************************************************************
** Load an instance of the appropriate state provider
**
@ -329,4 +485,155 @@ public class RunProcessAction
return (getStateProvider().get(ProcessState.class, stateKey));
}
/*******************************************************************************
**
*******************************************************************************/
protected String determineBasepullKeyValue(QProcessMetaData process, BasepullConfiguration basepullConfiguration) throws QException
{
String basepullKeyValue = (basepullConfiguration.getKeyValue() != null) ? basepullConfiguration.getKeyValue() : process.getName();
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// if backend specifies that it uses variants, look for that data in the session and append to our basepull key //
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////
if(process.getSchedule() != null && process.getSchedule().getVariantBackend() != null)
{
QSession session = QContext.getQSession();
QBackendMetaData backendMetaData = QContext.getQInstance().getBackend(process.getSchedule().getVariantBackend());
if(session.getBackendVariants() == null || !session.getBackendVariants().containsKey(backendMetaData.getVariantOptionsTableTypeValue()))
{
LOG.info("Could not find Backend Variant information for Backend '" + backendMetaData.getName() + "'");
}
else
{
basepullKeyValue += "-" + session.getBackendVariants().get(backendMetaData.getVariantOptionsTableTypeValue());
}
}
return (basepullKeyValue);
}
/*******************************************************************************
** Insert or update the last runtime value for this basepull into the backend.
*******************************************************************************/
protected void storeLastRunTime(RunProcessInput runProcessInput, QProcessMetaData process, BasepullConfiguration basepullConfiguration) throws QException
{
String basepullTableName = basepullConfiguration.getTableName();
String basepullKeyFieldName = basepullConfiguration.getKeyField();
String basepullLastRunTimeFieldName = basepullConfiguration.getLastRunTimeFieldName();
String basepullKeyValue = determineBasepullKeyValue(process, basepullConfiguration);
///////////////////////////////////////
// get the stored basepull timestamp //
///////////////////////////////////////
QueryInput queryInput = new QueryInput();
queryInput.setTableName(basepullTableName);
queryInput.setFilter(new QQueryFilter().withCriteria(
new QFilterCriteria()
.withFieldName(basepullKeyFieldName)
.withOperator(QCriteriaOperator.EQUALS)
.withValues(List.of(basepullKeyValue))));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
//////////////////////////////////////////
// get the runtime for this process run //
//////////////////////////////////////////
Instant newRunTime = (Instant) runProcessInput.getValues().get(BASEPULL_THIS_RUNTIME_KEY);
/////////////////////////////////////////////////
// update if found, otherwise insert new value //
/////////////////////////////////////////////////
if(CollectionUtils.nullSafeHasContents(queryOutput.getRecords()))
{
///////////////////////////////////////////////////////////////////////////////
// update the basepull table with 'now' (which is before original query ran) //
///////////////////////////////////////////////////////////////////////////////
QRecord basepullRecord = queryOutput.getRecords().get(0);
basepullRecord.setValue(basepullLastRunTimeFieldName, newRunTime);
////////////
// update //
////////////
UpdateInput updateInput = new UpdateInput();
updateInput.setTableName(basepullTableName);
updateInput.setRecords(List.of(basepullRecord));
new UpdateAction().execute(updateInput);
}
else
{
QRecord basepullRecord = new QRecord()
.withValue(basepullKeyFieldName, basepullKeyValue)
.withValue(basepullLastRunTimeFieldName, newRunTime);
////////////////////////////////
// insert new basepull record //
////////////////////////////////
InsertInput insertInput = new InsertInput();
insertInput.setTableName(basepullTableName);
insertInput.setRecords(List.of(basepullRecord));
new InsertAction().execute(insertInput);
}
}
/*******************************************************************************
** Lookup the last runtime for this basepull, and set it (plus now) in the process's
** values.
*******************************************************************************/
protected void persistLastRunTime(RunProcessInput runProcessInput, QProcessMetaData process, BasepullConfiguration basepullConfiguration) throws QException
{
////////////////////////////////////////////////////////
// if these values were already computed, don't re-do //
////////////////////////////////////////////////////////
if(runProcessInput.getValue(BASEPULL_THIS_RUNTIME_KEY) != null)
{
return;
}
/////////////////////////////////////////////////////////////////////////////////////////////////
// store 'now', which will be used to update basepull record if process completes successfully //
/////////////////////////////////////////////////////////////////////////////////////////////////
Instant now = Instant.now();
runProcessInput.getValues().put(BASEPULL_THIS_RUNTIME_KEY, now);
String basepullTableName = basepullConfiguration.getTableName();
String basepullKeyFieldName = basepullConfiguration.getKeyField();
String basepullLastRunTimeFieldName = basepullConfiguration.getLastRunTimeFieldName();
Integer basepullHoursBackForInitialTimestamp = basepullConfiguration.getHoursBackForInitialTimestamp();
String basepullKeyValue = determineBasepullKeyValue(process, basepullConfiguration);
///////////////////////////////////////
// get the stored basepull timestamp //
///////////////////////////////////////
QueryInput queryInput = new QueryInput();
queryInput.setTableName(basepullTableName);
queryInput.setFilter(new QQueryFilter().withCriteria(
new QFilterCriteria()
.withFieldName(basepullKeyFieldName)
.withOperator(QCriteriaOperator.EQUALS)
.withValues(List.of(basepullKeyValue))));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
///////////////////////////////////////////////////////////////////////////////////////////////////
// get the stored time, if not, default to 'now' unless a number of hours to offset was provided //
///////////////////////////////////////////////////////////////////////////////////////////////////
Instant lastRunTime = now;
if(CollectionUtils.nullSafeHasContents(queryOutput.getRecords()))
{
QRecord basepullRecord = queryOutput.getRecords().get(0);
lastRunTime = ValueUtils.getValueAsInstant(basepullRecord.getValue(basepullLastRunTimeFieldName));
}
else if(basepullHoursBackForInitialTimestamp != null)
{
lastRunTime = lastRunTime.minus(basepullHoursBackForInitialTimestamp, ChronoUnit.HOURS);
}
runProcessInput.getValues().put(BASEPULL_LAST_RUNTIME_KEY, lastRunTime);
runProcessInput.getValues().put(BASEPULL_TIMESTAMP_FIELD, basepullConfiguration.getTimestampField());
runProcessInput.getValues().put(BASEPULL_CONFIGURATION, basepullConfiguration);
}
}

View File

@ -0,0 +1,84 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2023. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.queues;
import java.util.List;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.amazonaws.services.sqs.model.GetQueueAttributesResult;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.queues.QQueueMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.queues.QQueueProviderMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.queues.SQSQueueProviderMetaData;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
**
*******************************************************************************/
public class GetQueueSize
{
private static final QLogger LOG = QLogger.getLogger(GetQueueSize.class);
/*******************************************************************************
**
*******************************************************************************/
public Integer getQueueSize(QQueueProviderMetaData queueProviderMetaData, QQueueMetaData queueMetaData) throws QException
{
try
{
//////////////////////////////////////////////////////////////////
// todo - handle other queue provider types, somewhere, somehow //
//////////////////////////////////////////////////////////////////
SQSQueueProviderMetaData queueProvider = (SQSQueueProviderMetaData) queueProviderMetaData;
BasicAWSCredentials credentials = new BasicAWSCredentials(queueProvider.getAccessKey(), queueProvider.getSecretKey());
final AmazonSQS sqs = AmazonSQSClientBuilder.standard()
.withRegion(queueProvider.getRegion())
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.build();
String queueUrl = queueProvider.getBaseURL();
if(!queueUrl.endsWith("/"))
{
queueUrl += "/";
}
queueUrl += queueMetaData.getQueueName();
GetQueueAttributesResult queueAttributes = sqs.getQueueAttributes(queueUrl, List.of("ApproximateNumberOfMessages"));
String approximateNumberOfMessages = queueAttributes.getAttributes().get("ApproximateNumberOfMessages");
return (Integer.parseInt(approximateNumberOfMessages));
}
catch(Exception e)
{
LOG.warn("Error getting queue size", e, logPair("queueName", queueMetaData == null ? "null" : queueMetaData.getName()));
throw (new QException("Error getting queue size", e));
}
}
}

View File

@ -0,0 +1,221 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.queues;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Supplier;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.amazonaws.services.sqs.model.DeleteMessageBatchRequest;
import com.amazonaws.services.sqs.model.DeleteMessageBatchRequestEntry;
import com.amazonaws.services.sqs.model.Message;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
import com.amazonaws.services.sqs.model.ReceiveMessageResult;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.queues.QQueueMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.queues.SQSQueueProviderMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Class to poll an SQS queue, and run process code for each message found.
*******************************************************************************/
public class SQSQueuePoller implements Runnable
{
private static final QLogger LOG = QLogger.getLogger(SQSQueuePoller.class);
///////////////////////////////////////////////
// todo - move these 2 to a "QBaseRunnable"? //
///////////////////////////////////////////////
private QInstance qInstance;
private Supplier<QSession> sessionSupplier;
private SQSQueueProviderMetaData queueProviderMetaData;
private QQueueMetaData queueMetaData;
/*******************************************************************************
**
*******************************************************************************/
@Override
public void run()
{
QContext.init(qInstance, sessionSupplier.get());
String originalThreadName = Thread.currentThread().getName();
Thread.currentThread().setName("SQSPoller>" + queueMetaData.getName());
LOG.debug("Running " + this.getClass().getSimpleName() + "[" + queueMetaData.getName() + "]");
try
{
BasicAWSCredentials credentials = new BasicAWSCredentials(queueProviderMetaData.getAccessKey(), queueProviderMetaData.getSecretKey());
final AmazonSQS sqs = AmazonSQSClientBuilder.standard()
.withRegion(queueProviderMetaData.getRegion())
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.build();
String queueUrl = queueProviderMetaData.getBaseURL();
if(!queueUrl.endsWith("/"))
{
queueUrl += "/";
}
queueUrl += queueMetaData.getQueueName();
while(true)
{
///////////////////////////////
// fetch a batch of messages //
///////////////////////////////
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();
receiveMessageRequest.setQueueUrl(queueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
receiveMessageRequest.setWaitTimeSeconds(20); // help urge SQS to query multiple servers and find more messages
ReceiveMessageResult receiveMessageResult = sqs.receiveMessage(receiveMessageRequest);
if(receiveMessageResult.getMessages().isEmpty())
{
LOG.debug("0 messages received. Breaking.");
break;
}
LOG.debug(receiveMessageResult.getMessages().size() + " messages received. Processing.");
//////////////////////////////////////////////////////////////////////////////////////////////////////////////
// extract data from the messages into list of bodies to pass into process, and list of delete-batch-inputs //
//////////////////////////////////////////////////////////////////////////////////////////////////////////////
List<DeleteMessageBatchRequestEntry> deleteRequestEntries = new ArrayList<>();
ArrayList<String> bodies = new ArrayList<>();
int i = 0;
for(Message message : receiveMessageResult.getMessages())
{
bodies.add(message.getBody());
deleteRequestEntries.add(new DeleteMessageBatchRequestEntry(String.valueOf(i++), message.getReceiptHandle()));
}
/////////////////////////////////////////////////////////////////////////////////////
// run the process, in a try-catch, so even if it fails, our loop keeps going. //
// the messages in a failed process will get re-delivered, to try-again, up to the //
// number of times configured in AWS //
/////////////////////////////////////////////////////////////////////////////////////
try
{
RunProcessInput runProcessInput = new RunProcessInput();
runProcessInput.setProcessName(queueMetaData.getProcessName());
runProcessInput.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
runProcessInput.addValue("bodies", bodies);
QContext.pushAction(runProcessInput);
RunProcessAction runProcessAction = new RunProcessAction();
RunProcessOutput runProcessOutput = runProcessAction.execute(runProcessInput);
////////////////////////////////////////////////////////////////////////////////////////////
// if there was an exception returned by the process (e.g., thrown in backend step), then //
// warn and leave the messages for re-processing. //
////////////////////////////////////////////////////////////////////////////////////////////
if(runProcessOutput.getException().isPresent())
{
LOG.warn("Exception returned by process when handling SQS Messages. They will not be deleted from the queue.", runProcessOutput.getException().get());
}
else
{
///////////////////////////////////////////////
// else, if no exception, do a batch delete. //
///////////////////////////////////////////////
sqs.deleteMessageBatch(new DeleteMessageBatchRequest()
.withQueueUrl(queueUrl)
.withEntries(deleteRequestEntries));
}
}
catch(Exception e)
{
LOG.warn("Error receiving SQS Messages.", e);
}
finally
{
QContext.popAction();
}
}
}
catch(Exception e)
{
LOG.warn("Error running SQS Queue Poller", e);
}
finally
{
Thread.currentThread().setName(originalThreadName);
QContext.clear();
}
}
/*******************************************************************************
** Setter for queueProviderMetaData
**
*******************************************************************************/
public void setQueueProviderMetaData(SQSQueueProviderMetaData queueProviderMetaData)
{
this.queueProviderMetaData = queueProviderMetaData;
}
/*******************************************************************************
** Setter for queueMetaData
**
*******************************************************************************/
public void setQueueMetaData(QQueueMetaData queueMetaData)
{
this.queueMetaData = queueMetaData;
}
/*******************************************************************************
** Setter for qInstance
**
*******************************************************************************/
public void setQInstance(QInstance qInstance)
{
this.qInstance = qInstance;
}
/*******************************************************************************
** Setter for sessionSupplier
**
*******************************************************************************/
public void setSessionSupplier(Supplier<QSession> sessionSupplier)
{
this.sessionSupplier = sessionSupplier;
}
}

View File

@ -0,0 +1,90 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.reporting;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Subclass of RecordPipe, which uses a buffer in the addRecord method, to avoid
** sending single-records at a time through postRecordActions and to consumers.
*******************************************************************************/
public class BufferedRecordPipe extends RecordPipe
{
private List<QRecord> buffer = new ArrayList<>();
private Integer bufferSize = 100;
/*******************************************************************************
** Constructor - uses default buffer size
**
*******************************************************************************/
public BufferedRecordPipe()
{
}
/*******************************************************************************
** Constructor - customize buffer size.
**
*******************************************************************************/
public BufferedRecordPipe(Integer bufferSize)
{
this.bufferSize = bufferSize;
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public void addRecord(QRecord record) throws QException
{
buffer.add(record);
if(buffer.size() >= bufferSize)
{
addRecords(buffer);
buffer.clear();
}
}
/*******************************************************************************
**
*******************************************************************************/
public void finalFlush() throws QException
{
if(!buffer.isEmpty())
{
addRecords(buffer);
buffer.clear();
}
}
}

View File

@ -27,24 +27,24 @@ import java.nio.charset.StandardCharsets;
import java.util.List;
import com.kingsrook.qqq.backend.core.adapters.QRecordToCsvAdapter;
import com.kingsrook.qqq.backend.core.exceptions.QReportingException;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
/*******************************************************************************
** CSV report format implementation
** CSV export format implementation
*******************************************************************************/
public class CsvReportStreamer implements ReportStreamerInterface
public class CsvExportStreamer implements ExportStreamerInterface
{
private static final Logger LOG = LogManager.getLogger(CsvReportStreamer.class);
private static final QLogger LOG = QLogger.getLogger(CsvExportStreamer.class);
private final QRecordToCsvAdapter qRecordToCsvAdapter;
private ReportInput reportInput;
private ExportInput exportInput;
private QTableMetaData table;
private List<QFieldMetaData> fields;
private OutputStream outputStream;
@ -54,7 +54,7 @@ public class CsvReportStreamer implements ReportStreamerInterface
/*******************************************************************************
**
*******************************************************************************/
public CsvReportStreamer()
public CsvExportStreamer()
{
qRecordToCsvAdapter = new QRecordToCsvAdapter();
}
@ -65,14 +65,14 @@ public class CsvReportStreamer implements ReportStreamerInterface
**
*******************************************************************************/
@Override
public void start(ReportInput reportInput, List<QFieldMetaData> fields) throws QReportingException
public void start(ExportInput exportInput, List<QFieldMetaData> fields, String label) throws QReportingException
{
this.reportInput = reportInput;
this.exportInput = exportInput;
this.fields = fields;
table = reportInput.getTable();
outputStream = this.reportInput.getReportOutputStream();
table = exportInput.getTable();
outputStream = this.exportInput.getReportOutputStream();
writeReportHeaderRow();
writeTitleAndHeader();
}
@ -80,20 +80,29 @@ public class CsvReportStreamer implements ReportStreamerInterface
/*******************************************************************************
**
*******************************************************************************/
private void writeReportHeaderRow() throws QReportingException
private void writeTitleAndHeader() throws QReportingException
{
try
{
int col = 0;
for(QFieldMetaData column : fields)
if(StringUtils.hasContent(exportInput.getTitleRow()))
{
if(col++ > 0)
{
outputStream.write(',');
}
outputStream.write(('"' + column.getLabel() + '"').getBytes(StandardCharsets.UTF_8));
outputStream.write((exportInput.getTitleRow() + "\n").getBytes(StandardCharsets.UTF_8));
}
outputStream.write('\n');
if(exportInput.getIncludeHeaderRow())
{
int col = 0;
for(QFieldMetaData column : fields)
{
if(col++ > 0)
{
outputStream.write(',');
}
outputStream.write(('"' + column.getLabel() + '"').getBytes(StandardCharsets.UTF_8));
}
outputStream.write('\n');
}
outputStream.flush();
}
catch(Exception e)
@ -108,20 +117,28 @@ public class CsvReportStreamer implements ReportStreamerInterface
**
*******************************************************************************/
@Override
public int takeRecordsFromPipe(RecordPipe recordPipe) throws QReportingException
public void addRecords(List<QRecord> qRecords) throws QReportingException
{
List<QRecord> qRecords = recordPipe.consumeAvailableRecords();
LOG.info("Consuming [" + qRecords.size() + "] records from the pipe");
for(QRecord qRecord : qRecords)
{
writeRecord(qRecord);
}
}
/*******************************************************************************
**
*******************************************************************************/
private void writeRecord(QRecord qRecord) throws QReportingException
{
try
{
for(QRecord qRecord : qRecords)
{
String csv = qRecordToCsvAdapter.recordToCsv(table, qRecord, fields);
outputStream.write(csv.getBytes(StandardCharsets.UTF_8));
outputStream.flush(); // todo - less often?
}
return (qRecords.size());
String csv = qRecordToCsvAdapter.recordToCsv(table, qRecord, fields);
outputStream.write(csv.getBytes(StandardCharsets.UTF_8));
outputStream.flush(); // todo - less often?
}
catch(Exception e)
{
@ -131,6 +148,17 @@ public class CsvReportStreamer implements ReportStreamerInterface
/*******************************************************************************
**
*******************************************************************************/
@Override
public void addTotalsRow(QRecord record) throws QReportingException
{
writeRecord(record);
}
/*******************************************************************************
**
*******************************************************************************/

View File

@ -0,0 +1,343 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.reporting;
import java.io.OutputStream;
import java.io.Serializable;
import java.time.Instant;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qqq.backend.core.actions.reporting.excelformatting.ExcelStylerInterface;
import com.kingsrook.qqq.backend.core.actions.reporting.excelformatting.PlainExcelStyler;
import com.kingsrook.qqq.backend.core.exceptions.QReportingException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import org.dhatim.fastexcel.StyleSetter;
import org.dhatim.fastexcel.Workbook;
import org.dhatim.fastexcel.Worksheet;
/*******************************************************************************
** Excel export format implementation
*******************************************************************************/
public class ExcelExportStreamer implements ExportStreamerInterface
{
private static final QLogger LOG = QLogger.getLogger(ExcelExportStreamer.class);
private ExportInput exportInput;
private QTableMetaData table;
private List<QFieldMetaData> fields;
private OutputStream outputStream;
private ExcelStylerInterface excelStylerInterface = new PlainExcelStyler();
private Map<String, String> excelCellFormats;
private Workbook workbook;
private Worksheet worksheet;
private int row = 0;
private int sheetCount = 0;
/*******************************************************************************
**
*******************************************************************************/
public ExcelExportStreamer()
{
}
/*******************************************************************************
** display formats is a map of field name to Excel format strings (e.g., $#,##0.00)
*******************************************************************************/
@Override
public void setDisplayFormats(Map<String, String> displayFormats)
{
this.excelCellFormats = new HashMap<>();
for(Map.Entry<String, String> entry : displayFormats.entrySet())
{
String excelFormat = DisplayFormat.getExcelFormat(entry.getValue());
if(excelFormat != null)
{
excelCellFormats.put(entry.getKey(), excelFormat);
}
}
}
/*******************************************************************************
** Starts a new worksheet in the current workbook. Can be called multiple times.
*******************************************************************************/
@Override
public void start(ExportInput exportInput, List<QFieldMetaData> fields, String label) throws QReportingException
{
try
{
this.exportInput = exportInput;
this.fields = fields;
table = exportInput.getTable();
outputStream = this.exportInput.getReportOutputStream();
this.row = 0;
this.sheetCount++;
/////////////////////////////////////////////////////////////////////////////////////////////////////
// if this is the first call in here (e.g., the workbook hasn't been opened yet), then open it now //
/////////////////////////////////////////////////////////////////////////////////////////////////////
if(workbook == null)
{
String appName = "QQQ";
QInstance instance = exportInput.getInstance();
if(instance != null && instance.getBranding() != null && instance.getBranding().getCompanyName() != null)
{
appName = instance.getBranding().getCompanyName();
}
workbook = new Workbook(outputStream, appName, null);
}
/////////////////////////////////////////////////////////////////////////////////////
// if start is called a second time (e.g., and there's already an open worksheet), //
// finish that sheet, before a new one is created. //
/////////////////////////////////////////////////////////////////////////////////////
if(worksheet != null)
{
worksheet.finish();
}
worksheet = workbook.newWorksheet(Objects.requireNonNullElse(label, "Sheet" + sheetCount));
writeTitleAndHeader();
}
catch(Exception e)
{
throw (new QReportingException("Error starting worksheet", e));
}
}
/*******************************************************************************
**
*******************************************************************************/
private void writeTitleAndHeader() throws QReportingException
{
try
{
///////////////
// title row //
///////////////
if(StringUtils.hasContent(exportInput.getTitleRow()))
{
worksheet.value(row, 0, exportInput.getTitleRow());
worksheet.range(row, 0, row, fields.size() - 1).merge();
StyleSetter titleStyle = worksheet.range(row, 0, row, fields.size() - 1).style();
excelStylerInterface.styleTitleRow(titleStyle);
titleStyle.set();
row++;
worksheet.flush();
}
////////////////
// header row //
////////////////
if(exportInput.getIncludeHeaderRow())
{
int col = 0;
for(QFieldMetaData column : fields)
{
worksheet.value(row, col, column.getLabel());
col++;
}
StyleSetter headerStyle = worksheet.range(row, 0, row, fields.size() - 1).style();
excelStylerInterface.styleHeaderRow(headerStyle);
headerStyle.set();
row++;
worksheet.flush();
}
}
catch(Exception e)
{
throw (new QReportingException("Error starting Excel report"));
}
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public void addRecords(List<QRecord> qRecords) throws QReportingException
{
LOG.info("Consuming [" + qRecords.size() + "] records from the pipe");
try
{
for(QRecord qRecord : qRecords)
{
writeRecord(qRecord);
row++;
worksheet.flush(); // todo? not at all? or just sometimes?
}
}
catch(Exception e)
{
LOG.error("Exception generating excel file", e);
try
{
workbook.finish();
outputStream.close();
}
finally
{
throw (new QReportingException("Error generating Excel report", e));
}
}
}
/*******************************************************************************
**
*******************************************************************************/
private void writeRecord(QRecord qRecord)
{
int col = 0;
for(QFieldMetaData field : fields)
{
Serializable value = qRecord.getValue(field.getName());
if(value != null)
{
if(value instanceof String s)
{
worksheet.value(row, col, s);
}
else if(value instanceof Number n)
{
worksheet.value(row, col, n);
if(excelCellFormats != null)
{
String format = excelCellFormats.get(field.getName());
if(format != null)
{
worksheet.style(row, col).format(format).set();
}
}
}
else if(value instanceof Boolean b)
{
worksheet.value(row, col, b);
}
else if(value instanceof Date d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd").set();
}
else if(value instanceof LocalDate d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd").set();
}
else if(value instanceof LocalDateTime d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd H:mm:ss").set();
}
else if(value instanceof ZonedDateTime d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd H:mm:ss").set();
}
else if(value instanceof Instant i)
{
// todo - what would be a better zone to use here?
worksheet.value(row, col, i.atZone(ZoneId.systemDefault()));
worksheet.style(row, col).format("yyyy-MM-dd H:mm:ss").set();
}
else
{
worksheet.value(row, col, ValueUtils.getValueAsString(value));
}
}
col++;
}
}
/*******************************************************************************
**
*******************************************************************************/
public void addTotalsRow(QRecord record)
{
writeRecord(record);
StyleSetter totalsRowStyle = worksheet.range(row, 0, row, fields.size() - 1).style();
excelStylerInterface.styleTotalsRow(totalsRowStyle);
totalsRowStyle.set();
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public void finish() throws QReportingException
{
try
{
if(workbook != null)
{
workbook.finish();
}
}
catch(Exception e)
{
throw (new QReportingException("Error finishing Excel report", e));
}
}
}

View File

@ -1,214 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2022. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
package com.kingsrook.qqq.backend.core.actions.reporting;
import java.io.OutputStream;
import java.io.Serializable;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZonedDateTime;
import java.util.Date;
import java.util.List;
import com.kingsrook.qqq.backend.core.exceptions.QReportingException;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.dhatim.fastexcel.Workbook;
import org.dhatim.fastexcel.Worksheet;
/*******************************************************************************
** Excel report format implementation
*******************************************************************************/
public class ExcelReportStreamer implements ReportStreamerInterface
{
private static final Logger LOG = LogManager.getLogger(ExcelReportStreamer.class);
private ReportInput reportInput;
private QTableMetaData table;
private List<QFieldMetaData> fields;
private OutputStream outputStream;
private Workbook workbook;
private Worksheet worksheet;
private int row = 1;
/*******************************************************************************
**
*******************************************************************************/
public ExcelReportStreamer()
{
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public void start(ReportInput reportInput, List<QFieldMetaData> fields) throws QReportingException
{
this.reportInput = reportInput;
this.fields = fields;
table = reportInput.getTable();
outputStream = this.reportInput.getReportOutputStream();
workbook = new Workbook(outputStream, "QQQ", null);
worksheet = workbook.newWorksheet("Sheet 1");
writeReportHeaderRow();
}
/*******************************************************************************
**
*******************************************************************************/
private void writeReportHeaderRow() throws QReportingException
{
try
{
int col = 0;
for(QFieldMetaData column : fields)
{
worksheet.value(0, col, column.getLabel());
col++;
}
worksheet.flush();
}
catch(Exception e)
{
throw (new QReportingException("Error starting Excel report"));
}
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public int takeRecordsFromPipe(RecordPipe recordPipe) throws QReportingException
{
List<QRecord> qRecords = recordPipe.consumeAvailableRecords();
LOG.info("Consuming [" + qRecords.size() + "] records from the pipe");
try
{
for(QRecord qRecord : qRecords)
{
int col = 0;
for(QFieldMetaData column : fields)
{
Serializable value = qRecord.getValue(column.getName());
if(value != null)
{
if(value instanceof String s)
{
worksheet.value(row, col, s);
}
else if(value instanceof Number n)
{
worksheet.value(row, col, n);
}
else if(value instanceof Boolean b)
{
worksheet.value(row, col, b);
}
else if(value instanceof Date d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd").set();
}
else if(value instanceof LocalDate d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd").set();
}
else if(value instanceof LocalDateTime d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd H:mm:ss").set();
}
else if(value instanceof ZonedDateTime d)
{
worksheet.value(row, col, d);
worksheet.style(row, col).format("yyyy-MM-dd H:mm:ss").set();
}
else
{
worksheet.value(row, col, ValueUtils.getValueAsString(value));
}
}
col++;
}
row++;
worksheet.flush(); // todo? not at all? or just sometimes?
}
}
catch(Exception e)
{
try
{
workbook.finish();
outputStream.close();
}
finally
{
throw (new QReportingException("Error generating Excel report", e));
}
}
return (qRecords.size());
}
/*******************************************************************************
**
*******************************************************************************/
@Override
public void finish() throws QReportingException
{
try
{
if(workbook != null)
{
workbook.finish();
}
}
catch(Exception e)
{
throw (new QReportingException("Error finishing Excel report", e));
}
}
}

View File

@ -23,37 +23,46 @@ package com.kingsrook.qqq.backend.core.actions.reporting;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import com.kingsrook.qqq.backend.core.actions.ActionHelper;
import com.kingsrook.qqq.backend.core.actions.async.AsyncJobManager;
import com.kingsrook.qqq.backend.core.actions.async.AsyncJobState;
import com.kingsrook.qqq.backend.core.actions.async.AsyncJobStatus;
import com.kingsrook.qqq.backend.core.actions.interfaces.CountInterface;
import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QReportingException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportOutput;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.ExposedJoin;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.SleepUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
/*******************************************************************************
** Action to generate a report.
** Action to generate an export from a table
**
** At this time (future may change?), this action starts a new thread to run
** the query in the backend module. As records are produced by the query,
@ -63,9 +72,9 @@ import org.apache.logging.log4j.Logger;
** time the report outputStream can be closed.
**
*******************************************************************************/
public class ReportAction
public class ExportAction
{
private static final Logger LOG = LogManager.getLogger(ReportAction.class);
private static final QLogger LOG = QLogger.getLogger(ExportAction.class);
private boolean preExecuteRan = false;
private Integer countFromPreExecute = null;
@ -82,27 +91,37 @@ public class ReportAction
** first, in their thread, to catch any validation errors before they start
** the thread (which they may abandon).
*******************************************************************************/
public void preExecute(ReportInput reportInput) throws QException
public void preExecute(ExportInput exportInput) throws QException
{
ActionHelper.validateSession(reportInput);
ActionHelper.validateSession(exportInput);
QBackendModuleDispatcher qBackendModuleDispatcher = new QBackendModuleDispatcher();
QBackendModuleInterface backendModule = qBackendModuleDispatcher.getQBackendModule(reportInput.getBackend());
QBackendModuleInterface backendModule = qBackendModuleDispatcher.getQBackendModule(exportInput.getBackend());
///////////////////////////////////
// verify field names (if given) //
///////////////////////////////////
if(CollectionUtils.nullSafeHasContents(reportInput.getFieldNames()))
if(CollectionUtils.nullSafeHasContents(exportInput.getFieldNames()))
{
QTableMetaData table = reportInput.getTable();
List<String> badFieldNames = new ArrayList<>();
for(String fieldName : reportInput.getFieldNames())
QTableMetaData table = exportInput.getTable();
Map<String, QTableMetaData> joinTableMap = getJoinTableMap(table);
List<String> badFieldNames = new ArrayList<>();
for(String fieldName : exportInput.getFieldNames())
{
try
{
table.getField(fieldName);
if(fieldName.contains("."))
{
String[] parts = fieldName.split("\\.", 2);
joinTableMap.get(parts[0]).getField(parts[1]);
}
else
{
table.getField(fieldName);
}
}
catch(IllegalArgumentException iae)
catch(Exception e)
{
badFieldNames.add(fieldName);
}
@ -119,58 +138,120 @@ public class ReportAction
///////////////////////////////////////////////////////////////////////////////////////////////////////////
// check if this report format has a max-rows limit -- if so, do a count to verify we're under the limit //
///////////////////////////////////////////////////////////////////////////////////////////////////////////
ReportFormat reportFormat = reportInput.getReportFormat();
verifyCountUnderMax(reportInput, backendModule, reportFormat);
ReportFormat reportFormat = exportInput.getReportFormat();
verifyCountUnderMax(exportInput, backendModule, reportFormat);
preExecuteRan = true;
}
/*******************************************************************************
**
*******************************************************************************/
private static Map<String, QTableMetaData> getJoinTableMap(QTableMetaData table)
{
Map<String, QTableMetaData> joinTableMap = new HashMap<>();
for(ExposedJoin exposedJoin : CollectionUtils.nonNullList(table.getExposedJoins()))
{
joinTableMap.put(exposedJoin.getJoinTable(), QContext.getQInstance().getTable(exposedJoin.getJoinTable()));
}
return joinTableMap;
}
/*******************************************************************************
** Run the report.
*******************************************************************************/
public ReportOutput execute(ReportInput reportInput) throws QException
public ExportOutput execute(ExportInput exportInput) throws QException
{
if(!preExecuteRan)
{
/////////////////////////////////////
// ensure that pre-execute has ran //
/////////////////////////////////////
preExecute(reportInput);
preExecute(exportInput);
}
QBackendModuleDispatcher qBackendModuleDispatcher = new QBackendModuleDispatcher();
QBackendModuleInterface backendModule = qBackendModuleDispatcher.getQBackendModule(reportInput.getBackend());
QBackendModuleInterface backendModule = qBackendModuleDispatcher.getQBackendModule(exportInput.getBackend());
//////////////////////////
// set up a query input //
//////////////////////////
QueryInterface queryInterface = backendModule.getQueryInterface();
QueryInput queryInput = new QueryInput(reportInput.getInstance());
queryInput.setSession(reportInput.getSession());
queryInput.setTableName(reportInput.getTableName());
queryInput.setFilter(reportInput.getQueryFilter());
queryInput.setLimit(reportInput.getLimit());
QueryAction queryAction = new QueryAction();
QueryInput queryInput = new QueryInput();
queryInput.setTableName(exportInput.getTableName());
queryInput.setFilter(exportInput.getQueryFilter());
List<QueryJoin> queryJoins = new ArrayList<>();
Set<String> addedJoinNames = new HashSet<>();
if(CollectionUtils.nullSafeHasContents(exportInput.getFieldNames()))
{
for(String fieldName : exportInput.getFieldNames())
{
if(fieldName.contains("."))
{
String[] parts = fieldName.split("\\.", 2);
String joinTableName = parts[0];
if(!addedJoinNames.contains(joinTableName))
{
QueryJoin queryJoin = new QueryJoin(joinTableName).withType(QueryJoin.Type.LEFT).withSelect(true);
queryJoins.add(queryJoin);
/////////////////////////////////////////////////////////////////////////////////////////////
// in at least some cases, we need to let the queryJoin know what join-meta-data to use... //
// This code basically mirrors what QFMD is doing right now, so it's better - //
// but shouldn't all of this just be in JoinsContext? it does some of this... //
/////////////////////////////////////////////////////////////////////////////////////////////
QTableMetaData table = exportInput.getTable();
Optional<ExposedJoin> exposedJoinOptional = CollectionUtils.nonNullList(table.getExposedJoins()).stream().filter(ej -> ej.getJoinTable().equals(joinTableName)).findFirst();
if(exposedJoinOptional.isEmpty())
{
throw (new QException("Could not find exposed join between base table " + table.getName() + " and requested join table " + joinTableName));
}
ExposedJoin exposedJoin = exposedJoinOptional.get();
if(exposedJoin.getJoinPath().size() == 1)
{
queryJoin.setJoinMetaData(QContext.getQInstance().getJoin(exposedJoin.getJoinPath().get(exposedJoin.getJoinPath().size() - 1)));
}
addedJoinNames.add(joinTableName);
}
}
}
}
queryInput.setQueryJoins(queryJoins);
if(queryInput.getFilter() == null)
{
queryInput.setFilter(new QQueryFilter());
}
queryInput.getFilter().setLimit(exportInput.getLimit());
queryInput.setShouldTranslatePossibleValues(true);
/////////////////////////////////////////////////////////////////
// tell this query that it needs to put its output into a pipe //
/////////////////////////////////////////////////////////////////
RecordPipe recordPipe = new RecordPipe();
RecordPipe recordPipe = new BufferedRecordPipe(500);
queryInput.setRecordPipe(recordPipe);
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// set up a report streamer, which will read rows from the pipe, and write formatted report rows to the output stream //
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
ReportFormat reportFormat = reportInput.getReportFormat();
ReportStreamerInterface reportStreamer = reportFormat.newReportStreamer();
reportStreamer.start(reportInput, getFields(reportInput));
ReportFormat reportFormat = exportInput.getReportFormat();
ExportStreamerInterface reportStreamer = reportFormat.newReportStreamer();
List<QFieldMetaData> fields = getFields(exportInput);
reportStreamer.start(exportInput, fields, "Sheet 1");
//////////////////////////////////////////
// run the query action as an async job //
//////////////////////////////////////////
AsyncJobManager asyncJobManager = new AsyncJobManager();
String queryJobUUID = asyncJobManager.startJob("ReportAction>QueryAction", (status) -> (queryInterface.execute(queryInput)));
String queryJobUUID = asyncJobManager.startJob("ReportAction>QueryAction", (status) -> (queryAction.execute(queryInput)));
LOG.info("Started query job [" + queryJobUUID + "] for report");
AsyncJobState queryJobState = AsyncJobState.RUNNING;
@ -207,8 +288,9 @@ public class ReportAction
lastReceivedRecordsAt = System.currentTimeMillis();
nextSleepMillis = INIT_SLEEP_MS;
int recordsConsumed = reportStreamer.takeRecordsFromPipe(recordPipe);
recordCount += recordsConsumed;
List<QRecord> records = recordPipe.consumeAvailableRecords();
processRecords(reportStreamer, fields, records);
recordCount += records.size();
LOG.info(countFromPreExecute != null
? String.format("Processed %,d of %,d records so far", recordCount, countFromPreExecute)
@ -235,8 +317,9 @@ public class ReportAction
///////////////////////////////////////////////////
// send the final records to the report streamer //
///////////////////////////////////////////////////
int recordsConsumed = reportStreamer.takeRecordsFromPipe(recordPipe);
recordCount += recordsConsumed;
List<QRecord> records = recordPipe.consumeAvailableRecords();
processRecords(reportStreamer, fields, records);
recordCount += records.size();
long reportEndTime = System.currentTimeMillis();
LOG.info((countFromPreExecute != null
@ -251,17 +334,17 @@ public class ReportAction
try
{
reportInput.getReportOutputStream().close();
exportInput.getReportOutputStream().close();
}
catch(Exception e)
{
throw (new QReportingException("Error completing report", e));
}
ReportOutput reportOutput = new ReportOutput();
reportOutput.setRecordCount(recordCount);
ExportOutput exportOutput = new ExportOutput();
exportOutput.setRecordCount(recordCount);
return (reportOutput);
return (exportOutput);
}
@ -269,17 +352,83 @@ public class ReportAction
/*******************************************************************************
**
*******************************************************************************/
private List<QFieldMetaData> getFields(ReportInput reportInput)
private static void processRecords(ExportStreamerInterface reportStreamer, List<QFieldMetaData> fields, List<QRecord> records) throws QReportingException
{
QTableMetaData table = reportInput.getTable();
if(reportInput.getFieldNames() != null)
for(QFieldMetaData field : fields)
{
return (reportInput.getFieldNames().stream().map(table::getField).toList());
if(field.getName().endsWith(":possibleValueLabel"))
{
String effectiveFieldName = field.getName().replace(":possibleValueLabel", "");
for(QRecord record : records)
{
String displayValue = record.getDisplayValue(effectiveFieldName);
record.setValue(field.getName(), displayValue);
}
}
}
reportStreamer.addRecords(records);
}
/*******************************************************************************
**
*******************************************************************************/
private List<QFieldMetaData> getFields(ExportInput exportInput)
{
QTableMetaData table = exportInput.getTable();
Map<String, QTableMetaData> joinTableMap = getJoinTableMap(table);
List<QFieldMetaData> fieldList;
if(exportInput.getFieldNames() != null)
{
fieldList = new ArrayList<>();
for(String fieldName : exportInput.getFieldNames())
{
if(fieldName.contains("."))
{
String[] parts = fieldName.split("\\.", 2);
QTableMetaData joinTable = joinTableMap.get(parts[0]);
QFieldMetaData field = joinTable.getField(parts[1]).clone();
field.setName(fieldName);
field.setLabel(joinTable.getLabel() + ": " + field.getLabel());
fieldList.add(field);
}
else
{
fieldList.add(table.getField(fieldName));
}
}
}
else
{
return (new ArrayList<>(table.getFields().values()));
fieldList = new ArrayList<>(table.getFields().values());
}
List<QFieldMetaData> returnList = new ArrayList<>();
for(QFieldMetaData field : fieldList)
{
/////////////////////////////////////////////////////////////////////////////////////////
// skip heavy fields. they aren't fetched, and we generally think we don't want them. //
/////////////////////////////////////////////////////////////////////////////////////////
if(field.getIsHeavy())
{
continue;
}
returnList.add(field);
//////////////////////////////////////////
// add fields for possible value labels //
//////////////////////////////////////////
if(StringUtils.hasContent(field.getPossibleValueSourceName()))
{
returnList.add(new QFieldMetaData(field.getName() + ":possibleValueLabel", QFieldType.STRING).withLabel(field.getLabel() + " Name"));
}
}
return (returnList);
}
@ -287,12 +436,12 @@ public class ReportAction
/*******************************************************************************
**
*******************************************************************************/
private void verifyCountUnderMax(ReportInput reportInput, QBackendModuleInterface backendModule, ReportFormat reportFormat) throws QException
private void verifyCountUnderMax(ExportInput exportInput, QBackendModuleInterface backendModule, ReportFormat reportFormat) throws QException
{
if(reportFormat.getMaxCols() != null)
{
List<QFieldMetaData> fields = getFields(reportInput);
if (fields.size() > reportFormat.getMaxCols())
List<QFieldMetaData> fields = getFields(exportInput);
if(fields.size() > reportFormat.getMaxCols())
{
throw (new QUserFacingException("The requested report would include more columns ("
+ String.format("%,d", fields.size()) + ") than the maximum allowed ("
@ -302,13 +451,12 @@ public class ReportAction
if(reportFormat.getMaxRows() != null)
{
if(reportInput.getLimit() == null || reportInput.getLimit() > reportFormat.getMaxRows())
if(exportInput.getLimit() == null || exportInput.getLimit() > reportFormat.getMaxRows())
{
CountInterface countInterface = backendModule.getCountInterface();
CountInput countInput = new CountInput(reportInput.getInstance());
countInput.setSession(reportInput.getSession());
countInput.setTableName(reportInput.getTableName());
countInput.setFilter(reportInput.getQueryFilter());
CountInput countInput = new CountInput();
countInput.setTableName(exportInput.getTableName());
countInput.setFilter(exportInput.getQueryFilter());
CountOutput countOutput = countInterface.execute(countInput);
countFromPreExecute = countOutput.getCount();
if(countFromPreExecute > reportFormat.getMaxRows())

View File

@ -23,29 +23,46 @@ package com.kingsrook.qqq.backend.core.actions.reporting;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.exceptions.QReportingException;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportInput;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
/*******************************************************************************
** Interface for various report formats to implement.
** Interface for various export formats to implement.
*******************************************************************************/
public interface ReportStreamerInterface
public interface ExportStreamerInterface
{
/*******************************************************************************
** Called once, before any rows are available. Meant to write a header, for example.
*******************************************************************************/
void start(ReportInput reportInput, List<QFieldMetaData> fields) throws QReportingException;
void start(ExportInput exportInput, List<QFieldMetaData> fields, String label) throws QReportingException;
/*******************************************************************************
** Called as records flow into the pipe.
******************************************************************************/
int takeRecordsFromPipe(RecordPipe recordPipe) throws QReportingException;
void addRecords(List<QRecord> recordList) throws QReportingException;
/*******************************************************************************
** Called once, after all rows are available. Meant to write a footer, or close resources, for example.
*******************************************************************************/
void finish() throws QReportingException;
/*******************************************************************************
**
*******************************************************************************/
default void setDisplayFormats(Map<String, String> displayFormats)
{
// noop in base class
}
/*******************************************************************************
**
*******************************************************************************/
default void addTotalsRow(QRecord record) throws QReportingException
{
addRecords(List.of(record));
}
}

Some files were not shown because too many files have changed in this diff Show More