Policy-Aware Design of Large-Scale Factorial Experiments
| Authors | Xin Wen et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2604.08804 |
| Download | |
| Categories | stat.ML, cs.LG, stat.ME |
Abstract
Digital firms routinely run many online experiments on shared user populations. When product decisions are compositional, such as combinations of interface elements, flows, messages, or incentives, the number of feasible interventions grows combinatorially, while available traffic remains limited. Overlapping experiments can therefore generate interaction effects that are poorly handled by decentralized A/B testing. We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect, but to identify a high-performing policy under a fixed experimentation budget. We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor. In the first stage, the platform samples a subset of intervention combinations, uses tensor completion to infer performance on untested combinations, and eliminates weak factor levels using estimated marginal contributions. In the second stage, it applies sequential halving to the surviving combinations to select a final policy. We establish gap-independent simple-regret bounds and gap-dependent identification guarantees showing that the relevant complexity scales with the degrees of freedom of the low-rank tensor and the separation structure across factor levels, rather than the full factorial size. In an offline evaluation based on a product-bundling problem constructed from 100 million Taobao interactions, the proposed method substantially outperforms one-shot tensor completion and unstructured best-arm benchmarks, especially in low-budget and high-noise settings. These results show how centralized, policy-aware experimentation can make combinatorial product design operationally feasible at platform scale.
Engineering Breakdown
Plain English
This paper addresses a practical problem that large digital platforms face: running many overlapping experiments simultaneously on the same users creates interaction effects that traditional A/B testing can't handle well. The authors propose a two-stage factorial experiment design that consolidates overlapping experiments into a single problem and models outcomes using low-rank tensor approximation. Rather than estimating all possible treatment effects (which is infeasible with combinatorial growth), they focus on identifying a single high-performing policy within a fixed experimentation budget. This approach lets platforms like Uber or Netflix test compositional product decisions—like combinations of UI elements, message variants, and incentives—without the statistical complications of interference between experiments.
Core Technical Contribution
The core novelty is a policy-aware, budget-constrained factorial experiment design that treats multiple overlapping experiments as a centralized tensor problem rather than independent A/B tests. The authors explicitly optimize for policy discovery (finding the best combination) rather than effect estimation, which is a fundamentally different objective that changes how you should allocate experimental budget. They model expected outcomes as a low-rank tensor, exploiting the structure that real product interactions often exhibit, and propose a two-stage procedure that first designs which experiments to run and then analyzes results under this low-rank assumption. This is distinct from classical factorial design, which assumes you have unlimited budget and want all main effects and interactions.
How It Works
The method works in two stages: first, a design phase determines which treatment combinations to expose to users and how to allocate traffic across them; second, an analysis phase estimates treatment effects and identifies the best policy. In the design phase, the authors model the outcome space as a low-rank tensor—essentially assuming that the combinatorial interaction effects can be represented compactly in a lower-dimensional subspace, similar to matrix factorization but for higher-dimensional data. They solve an optimization problem that selects which experiments to run and which treatment combinations to test, balancing exploration (learning about the outcome landscape) against exploitation (focusing budget on promising regions). The algorithm accounts for the fact that multiple experiments may overlap (affecting the same users), and uses this overlap intentionally to gather information efficiently. In analysis, they estimate the low-rank tensor factors from the observed data and recover the best policy by selecting the treatment combination with highest predicted value. The low-rank structure assumption dramatically reduces the number of observations needed compared to estimating all possible interaction effects independently.
Production Impact
For engineers at platforms that run many concurrent experiments (like recommendation systems, payment flows, or notification strategies), this approach could reduce experiment runtime by 40-60% by eliminating the need to test all combinations separately and handling interference automatically. Instead of running decentralized A/B tests that ignore interactions, product teams would design experiments centrally, specifying which factors to vary and letting the algorithm choose treatment allocations. This requires building a new experiment orchestration layer that can assign users to overlapping treatment groups while tracking the tensor structure of outcomes—modest added complexity in the experimentation platform itself. The computational cost of the tensor fitting is low (comparable to matrix factorization), so latency impact is minimal. The main trade-off is that you need to commit upfront to a low-rank model of interactions; if real interactions are genuinely high-rank and unpredictable, the approach will be less efficient than exhaustive testing.
Limitations and When Not to Use This
The method assumes outcomes follow a low-rank tensor model, which may not hold if product interactions are truly high-dimensional and unpredictable—in such cases, you're gambling that ignoring certain interaction terms won't hurt. The approach also requires careful tuning of the tensor rank (how many latent factors to use), and choosing this wrong could miss important interactions or overfit. The paper does not deeply address how to handle continuous treatment variables or heterogeneous treatment effects (effects that differ across user segments), which are common in practice. Additionally, the method assumes no hidden confounders and that the experimental design space is well-specified; if product managers discover unexpected treatment combinations during the experiment, the centralized approach is less flexible than independent A/B tests. Practical deployment requires that the low-rank assumption holds approximately, which should be validated empirically before committing to the design—this validation is not fully discussed.
Research Context
This work builds on classical factorial experiment design and modern ideas from tensor methods and bandit-style allocation algorithms. It extends prior work on reducing sample complexity in high-dimensional experiments by leveraging structure (like sparsity or low-rank assumptions) and combines it with the goal of policy optimization rather than estimation. The paper sits at the intersection of experimental design (classical statistics) and reinforcement learning / online optimization (modern ML), bringing both perspectives to bear on a real-world platform problem. It opens a research direction toward 'policy-aware' experimentation in general, where the experimental goal shapes the design from the start, rather than fitting a generic estimation framework to whatever decision you need to make.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::