Commit Graph

1670 Commits

Author SHA1 Message Date
624a723b54 CE-781 Initial checkin of filesystem importer meta-data template and process 2024-01-11 07:46:30 -06:00
b64883f34f CE-781 rename beforeEach and afterEach (to help avoid overwriting in sub-classes) 2024-01-10 19:59:40 -06:00
4b1bdebe44 CE-781 Move backend type (name) up to public static final constant 2024-01-10 19:59:11 -06:00
8d668d12ec CE-781 Get fields using getFields().containsKey/get, rather than getField(String) - to avoid it throwing, to cut down on exceptions (and warn if we get a real exception, vs., we'll expect non-fields sometimes, so be okay with that) 2024-01-10 19:58:26 -06:00
c27a2a986a CE-781 Add cases for LinkedHashMap and HashMap in deepCopySimpleMap 2024-01-10 19:57:20 -06:00
fed8cbbb45 CE-781 Make tie-break for sorting do backends earlier than everything else 2024-01-10 19:52:22 -06:00
5147a022fa CE-781 add sortOrder attribute to apps, for sorting them... 2024-01-10 19:51:44 -06:00
615ff6fce5 CE-781 more fluent methods in process meta data builders 2024-01-10 19:51:07 -06:00
f5c4c12388 CE-781 Initial build of mongodb backend module 2024-01-08 20:00:57 -06:00
68911190fa CE-781 Updates for compatibility with corresponding changes, refactoring, in backend-core 2024-01-08 16:40:56 -06:00
56a2949911 CE-781 Remove check for empty record list (has been moved up to core InsertAction) 2024-01-08 14:18:32 -06:00
96013878bc CE-781 Update some of the getValueAs methods to take Object instead of Serializable 2024-01-08 14:17:55 -06:00
f879575b32 CE-781 Gracefully ignore request to add null uniqueKey or recordSecurityLock 2024-01-08 14:16:59 -06:00
bab3c7b374 CE-781 Initial checkin 2024-01-08 14:15:34 -06:00
1c69784897 CE-781 Add method add(TopLevelMetaDataInterface) 2024-01-08 14:15:05 -06:00
8822c1bb99 CE-781 Add overload constructor that takes Collection of values 2024-01-08 14:14:47 -06:00
a5420bff4c CE-781 Add concept of sharded automations - schedule multiple instances of job, filter implicitly by shard value 2024-01-08 14:13:44 -06:00
06259041f8 CE-781 Update to work without a table specified (just getting field names from the json keys) 2024-01-08 12:39:43 -06:00
a00d4f3cbd CE-781 Refactoring this code out of RDBMS update, to be shared with MongoDB update 2024-01-08 12:37:38 -06:00
8473e11444 CE-781 Refactoring of backend actions - moving openTransaction out of insert-action only (up to backendModule); re-using the exit-early-if-0 and set-default-create-and-modify-date logics; 2024-01-08 12:37:22 -06:00
bc3f462d13 CE-781 log (once) & noop for tables w/o integer primary key, as that is required for auditing... 2024-01-08 12:31:37 -06:00
56a2099515 CE-781 Add option to treat CSV headers as field names (rather than working with a table's fields) 2024-01-08 12:30:43 -06:00
93dcee9f61 Add QRecord as a handled type inside deepCopySimpleMap (e.g., so copy constructor won't need to warn about it and do slow serialization-based cloning). 2024-01-04 18:11:05 -06:00
92b052fe59 CE-773 avoid s3 list requests that start with / if backend & table have no basePaths 2023-12-29 19:12:34 -06:00
3f431b39b9 Merge pull request #58 from Kingsrook/feature/CE-773-cartonization-playground
CE-773 Fixing globs for local filesystem by using Files.walkFileTree.…
2023-12-29 10:42:41 -06:00
688e221f9a CE-773 Fixing globs for local filesystem by using Files.walkFileTree. Refactored to share filter matching between s3 & local fs. 2023-12-29 08:20:38 -06:00
959f8c8041 Merge pull request #55 from Kingsrook/feature/CE-773-cartonization-playground
Feature/ce 773 cartonization playground
2023-12-28 18:45:30 -06:00
6e1ea5c8f1 CE-773 fix tables created in here, per new validationing! 2023-12-28 16:46:51 -06:00
872dec3177 CE-773 change fileNameFieldName and contentsFieldName to default as null - add validation to tableBackendDetails, specifically implemented in filesystem module 2023-12-28 16:38:40 -06:00
78d9ec87a2 Merge pull request #57 from Kingsrook/feature/basepull-subtract-seconds
Add option to move timestamps, e.g., to make overlapping windows
2023-12-28 16:24:17 -06:00
01c78534ef Add test for previous commit (Add option to move timestamps, e.g., to make overlapping windows) 2023-12-28 16:20:38 -06:00
cfab10c8e8 Add option to move timestamps, e.g., to make overlapping windows 2023-12-28 15:54:44 -06:00
2da6878e70 Make sure to always return an empty list rather than a null 2023-12-28 10:33:23 -06:00
345d8022c1 CE-773 Feedback from code review 2023-12-28 10:33:23 -06:00
aff4b43296 Merged dev into feature/CE-773-cartonization-playground snapshot-feature-CE-773-cartonization-playground 2023-12-27 20:20:08 -06:00
b805e7645b CE-773 Update for compat. with previous commit, but also, fix all generics and move inputStream into try-with-resources 2023-12-27 16:11:19 -06:00
940080bc86 CE-773 update to support listing/filtering filesystem tables with Cardinality.ONE (single-record per-file) 2023-12-27 16:10:45 -06:00
6c9506d18b Merge pull request #54 from Kingsrook/feature/meta-data-producer-interface-and-is-enabled
Add isEnabled method to meta-data producers; Put interface on top of …
2023-12-22 19:03:45 -06:00
8fc2b548ee Add isEnabled method to meta-data producers; Put interface on top of MetaDataProducer, for times when someone wants that; update MetaDataProducerHelper to work w/ the interface. 2023-12-21 15:28:34 -06:00
a8c30b1bed Merge pull request #53 from Kingsrook/feature/CE-773-cartonization-playground
CE-773 Pass script revision id through
2023-12-21 11:59:36 -06:00
fd18568785 Only apply mysql result set optimization per a system property, default to false. 2023-12-20 14:18:34 -06:00
db2e5fb7fc CE-775 Add some sleep to help timeout test 2023-12-19 15:32:15 -06:00
dceb0ee142 CE-775 add timeouts to outbound http calls 2023-12-19 15:12:52 -06:00
1d022200c5 CE-773 Pass script revision id through 2023-12-18 12:43:55 -06:00
d1bfc834d6 Merge pull request #52 from Kingsrook/feature/gc-perf-debug
Feature/gc perf debug
2023-12-18 11:48:52 -06:00
5f586d30c7 Switch to do mysql optimizations if connection is com.mysql class snapshot-feature-gc-perf-debug 2023-12-18 08:45:20 -06:00
4703d3bb24 Fixed last commit (meant to use backend.vendor, not name, compare to aurora) 2023-12-16 10:27:25 -06:00
2b90d7e4b3 Update to use mysql optimizations for statements on aurora too... 2023-12-15 18:36:17 -06:00
9144754e74 Merge pull request #51 from Kingsrook/feature/CE-752-add-information-to-order
Feature/ce 752 add information to order
2023-12-14 13:13:54 -06:00
fb80c92f73 Merge pull request #47 from Kingsrook/dependabot/maven/qqq-backend-core/org.json-json-20231013
Bump org.json:json from 20230618 to 20231013 in /qqq-backend-core
2023-12-08 16:39:57 -06:00