EAVS: An Environment-Aware Vectorized Streaming Architecture for Eliminating the Bulk Write Bottleneck in OutSystems Low-Code Platform

Abstract

Outsystem is a Low-Code Development Platforms. Low-Code Development is an extremely flexible platform for handling database, web based application, mobile based application and process development. Outsystem abstracts the process of coding and makes it a visual modeling using drag and drop. It also has the power to speed the growth of enterprises. Large-scale data migration architectures that builts on Low-Code Development Platforms (LCDPs) such as OutSystems often encounters a major performance bottleneck. The problem is prevalent in industrial settings. There is little research work done on computational overhead it introduces. The main constraint is that the platform's Entity Action loop makes its own SQL Server request for every entity processed. This row-by-row approach requires one round trip over the network, one database lock, and one transaction log entry per row, leading to linear runtime with the number of records to be processed. For the OutSystems 11 Free Cloud, the authors timed it to take 23 seconds to insert 50,000 records through the native path, 44 seconds for 100,000, and didn't get to 150,000 and beyond because the native path had a timeout with the platform (~60 seconds). In this paper, EAVS is introduced, a custom C# extension created with the OutSystems Integration Studio. Whereas the row-by-row execution model is replaced by the native TDS Bulk Copy Protocol that SQL Server supports, data will be sent in batches with a single connection, simplifying the operation complexity from O(N) to O(N/B), where B is the batch size. The authors have measured a factor of 53–56× faster with EAVS for the scales where both paths complete (50,000–100,000 records); and experiments confirm that EAVS can insert 300,000 records in 2.2 seconds, a workload that is beyond the abilities of the native path. The cost of a record is linear, with R2=0.976 over six dataset sizes, and about 7.7 μs per record compared to the native path's cost of 450 μs per record. In addition to the speed gains, EAVS also supports platform-specific requirements such as conditional tenant-column injection, and entity-agnostic column mapping, without impacting any existing application logic.