This paper explores a novel paradigm in low-bit (i.e. 4-bits or lower) quantization, differing from existing state-of-the-art methods, by framing optimal quantization as an architecture search problem within convolutional neural networks (ConvNets). Our framework, dubbed CoRa (Optimal Quantization Residual Convolutional Operator Low-Rank Adaptation), is motivated by two key aspects. Firstly, quantization residual knowledge, i.e. the lost information between floating-point weights and quantized weights, has long been neglected by the research community. Reclaiming the critical residual knowledge, with an infinitesimal extra parameter cost, can reverse performance degradation without training. Secondly, state-of-the-art quantization frameworks search for optimal quantized weights to address the performance degradation. Yet, the vast search spaces in weight optimization pose a challenge for the efficient optimization in large models. For example, state-of-the-art BRECQ necessitates 2 ×10^4 iterations to quantize models. Fundamentally differing from existing methods, CoRa searches for the optimal architectures of low-rank adapters, reclaiming critical quantization residual knowledge, within the search spaces smaller compared to the weight spaces, by many orders of magnitude. The low-rank adapters approximate the quantization residual weights, discarded in previous methods. We evaluate our approach over multiple pre-trained ConvNets on ImageNet. CoRa achieves comparable performance against both state-of-the-art quantization-aware training and post-training quantization baselines, in 4-bit and 3-bit quantization, by using less than 250 iterations on a small calibration set with 1600 images. Thus, CoRa establishes a new state-of-the-art in terms of the optimization efficiency in low-bit quantization.