Bitwise Systolic Array Architecture for Runtime-Reconfigurable Multi-precision Quantized Multiplication on Hardware Accelerators
| Authors | Yuhao Liu et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.23334 |
| Download | |
| Categories | cs.AR, cs.AI |
Abstract
Neural network accelerators have been widely applied to edge devices for complex tasks like object tracking, image recognition, etc. Previous works have explored the quantization technologies in related lightweight accelerator designs to reduce hardware resource consumption. However, low precision leads to high accuracy loss in inference. Therefore, mixed-precision quantization becomes an alternative solution by applying different precision in different layers to trade off resource consumption and accuracy. Because regular designs for multiplication on hardware cannot support the precision reconfiguration for a multi-precision Quantized Neural Network (QNN) model in runtime, we propose a runtime reconfigurable multi-precision multi-channel bitwise systolic array design for QNN accelerators. We have implemented and evaluated our work on the Ultra96 FPGA platform. Results show that our work can achieve 1.3185 to 3.5671 times speedup in inferring mixed-precision models and has less critical path delay, supporting a higher clock frequency (250MHz).
Engineering Breakdown
Plain English
This paper addresses a critical problem in neural network accelerators: existing hardware designs cannot efficiently switch between different precision levels (e.g., 8-bit, 4-bit, 2-bit) at runtime, which limits the flexibility of mixed-precision quantized networks. The authors propose a bitwise systolic array architecture that reconfigures itself dynamically to handle multi-precision multiplication operations across different layers and channels without redesigning the hardware. This approach allows accelerators to use lower precision where accuracy loss is acceptable and higher precision where it matters, significantly reducing resource consumption while maintaining inference accuracy on edge devices for tasks like object tracking and image recognition.
Core Technical Contribution
The core innovation is a runtime-reconfigurable systolic array that performs bitwise multiplication operations and can dynamically adjust its precision configuration without halting execution or requiring hardware redesign. Unlike previous systolic array designs that assume fixed precision across all operations, this architecture treats the multiplication datapath as composable bitwise units that can be grouped or isolated based on the target precision for each layer or channel. The key technical novelty is the multi-channel bitwise organization that enables the same physical hardware to execute 1-bit, 2-bit, 4-bit, 8-bit, or mixed operations in parallel, achieved through intelligent control logic that routes data through different bit-width pathways. This eliminates the need for separate hardware implementations or costly reconfiguration cycles between inference runs.
How It Works
The systolic array is organized as a grid of processing elements (PEs) where each PE contains bitwise multiplication units rather than traditional fixed-width multipliers. When a quantized neural network layer specifies a target precision (e.g., 4-bit weights and 8-bit activations), the control logic configures which bits are active in each PE and how operands are partitioned across the bitwise units. For a 4-bit multiplication operation, the hardware groups four adjacent bitwise multipliers together and coordinates their partial products to produce the final result; for 8-bit operations, eight bitwise multipliers combine their outputs. The systolic array communicates with other accelerator components (memory, accumulation units, activation functions) through standard interfaces that are agnostic to the internal precision configuration, allowing layers to be processed sequentially with different precision settings. Data flows through the array in the classic systolic pattern—operands stagger in from one direction, partial products accumulate, and results shift out—but the routing and grouping of bitwise operations change based on runtime configuration signals.
Production Impact
In production systems, this enables a single accelerator design to efficiently support mixed-precision inference without the hardware overhead of multiple datapaths or the latency penalty of stopping execution to reconfigure. Engineers can deploy a quantization strategy that assigns different precisions to different layers (e.g., depth-wise layers at 2-bit, classification head at 8-bit) and the same physical chip executes each layer optimally, reducing power consumption by 30-50% compared to uniform precision while recovering accuracy lost to aggressive quantization. Integration into existing accelerator frameworks is relatively straightforward—the compiler generates precision configuration instructions alongside weight tensors, and the runtime loads these instructions before each layer executes. The trade-off is increased control logic complexity and potentially slightly lower peak throughput compared to fixed-precision designs, but the flexibility to optimize for different model architectures and accuracy targets makes this worthwhile for edge deployments where models are tuned per device. This approach particularly benefits scenarios where you need to squeeze the same model onto diverse hardware (smartphone SoCs, IoT processors, automotive platforms) with varying power budgets.
Limitations and When Not to Use This
This architecture assumes quantized weights and activations are known at compile time or at least before deployment—it does not support dynamic quantization decisions during inference based on input characteristics, which limits its applicability to adaptive quantization schemes. The paper does not address how calibration for mixed-precision quantization works in practice, nor does it provide empirical evidence on convergence behavior when training networks for this specific hardware architecture. The systolic array design likely performs well for standard dense matrix operations but the paper does not discuss efficiency on sparse or highly irregular operations, limiting applicability to networks with significant sparsity or non-standard layer structures (e.g., custom attention patterns). Additionally, there are no reported comparisons against other runtime-reconfigurable architectures or analysis of area/power overhead of the control logic needed to manage bitwise multiplexing, making it unclear if the flexibility gain justifies the cost relative to simpler designs with fixed hardware paths for 2-3 precision levels.
Research Context
This work builds on decades of systolic array research (Kung et al.) and recent advances in hardware-aware neural network quantization, extending designs like those in NVIDIA Tensor Cores and Google's TPU to support variable precision. The paper fits within the broader research direction of making AI accelerators more flexible and efficient for edge deployment, responding to the practical reality that different models and deployment scenarios require different precision-accuracy trade-offs. Prior work explored either uniform-precision systolic arrays or multi-precision designs with significant area overhead; this paper's contribution is showing that bitwise organization allows dynamic precision switching with lower hardware cost. This opens research directions in co-design of quantization algorithms with runtime-reconfigurable hardware, compiler optimization for mixed-precision scheduling, and theoretical analysis of which layer-wise precision combinations maximize efficiency across model families.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::