HBA: A Heuristic Batch Advisor for Self-Tuning Bulk Data Ingestion in the OutSystems Low-Code Platform

Abstract

Low code development platforms like OutSystems allow teams to build data-centric applications using visual models that the platform takes care of the database plumbing [1] [2]. We have demonstrated in an earlier study that the write path is row-wise and fails at scale; and in a previous study we introduced EAVS [25], a C# extension to restore bulk ingestion by using SQL Server's TDS bulk-copy stream. However, EAVS [25] left one decision to the developer, the batch size B, which is not explained in the vendor's documentation, and is only suggested to be tested [10]. This paper first demonstrates that B is not a cosmetic setting. Across an eight-point sweep at N = 100,000 records on the OutSystems Free Cloud, a poorly chosen batch size (B = 500) roughly doubled execution time relative to the efficient region, and even our own informed default of B = 5,000 ran 28 to 47 percent slower than the best measured configurations. A second sweep with payloads about eleven times wider showed the efficient region itself moving by an order of magnitude, from roughly B = 10,000 down to roughly B = 1,000. Interestingly, both workloads converged near a constant batch payload of about 1 MB, which suggests that the quantity worth controlling is bytes per batch rather than rows per batch, and that no single fixed B can suit every workload. Building on that observation we present HBA, the Heuristic Batch Advisor, a self-tuning variant of the EAVS [25] bulk path that takes no batch-size input at all. HBA seeds its first batch from a byte-budget heuristic that targets about 1 MB per batch using the payload's own measured row width, then adjusts B between chunks based on observed throughput, smoothed with an exponential moving average so that a single noisy reading cannot whipsaw the controller. On thin rows at N = 100,000, HBA with zero manual tuning ran 32 percent faster than a naive configuration and matched the designers' informed default, staying within about 27 percent of the best hand-tuned batch size; on wide rows it matched the best fixed configuration outright, within about 1 percent, while beating the naive choice by 49 percent. The seed heuristic was verified directly in a cross workload test, where the controller opened B = 10,485 for 100-byte rows and B = 1,113 for 1.1-KB rows (no knowledge of the data). The instrumentation records the adaptation as it occurs, including runs in which the controller overshot and recognized the drop in throughput and then backed off. HBA is deployed outside database engines and runs in a low-code platform's hermetically sealed extension layer and has only feedback in the form of end-to-end throughput.HBA is deployed outside database engines, and is used in the hermetically sealed extension layer of a low-code platform, with end-to-end throughput being the only feedback signal.