Why Your Fraud Rules Are Rejecting Good Customers and How to Measure It
DOI:
https://doi.org/10.70917/ijcisim-2026-6100Keywords:
Card-not-present fraud, False declines, Selective labels, Randomized holdout, Off-policy evaluation, E-commerceAbstract
Online merchants who screen card-not-present orders with fraud rules cannot see what happens to the orders they decline. This retrospective examines how merchants and their advisers estimated those false declines in the last half-decade, and proposes a way for a small merchant to measure them. We tested ten claims from the period on DECLINESIM-23, a seeded synthetic benchmark of era-styled order streams in which the true outcome of every declined order is known, across 300 store-level parameter draws. Under red-flag rules that declined a median of 1.5% of orders, genuine customers accounted for a median of 89% of declines, far more than the one-third industry sources reported. Counting customers who reordered after a decline captured about a fifth of declines as genuine, and performance measured on approved orders alone understated the rules’ accuracy. Measured as lost profit rather than lost sales, false declines exceeded fraud losses in only 57% of stores. Approving a random 5% of would-be declines gave nearly unbiased estimates, and in 80% of stores it earned more from recovered sales than it lost to fraud. Holdouts were too small for precise reporting at small-merchant volumes, but large enough for decisions: a 2% holdout lifted a blanket ban on international orders in a median of one month and cut false declines on those orders from 135 to 21 per 1,000 orders.