Differentiable architecture search (DARTS) has attracted much attention due to its simplicity and significant improvement in efficiency. However, the excessive accumulation of the skip connection makes it suffer from long-term weak stability and low robustness. Many works attempt to restrict the accumulation of skip connections by indicators or manual design, however, these methods are susceptible to thresholds and human priors. In this work, we suggest a more subtle and direct approach that removes skip connections from the operation space. Then, by introducing an adaptive channel allocation strategy, we redesign the DARTS framework to automatically refill the skip connections in the evaluation stage, resolving the performance degradation caused by the absence of skip connections. Our method, dubbed Adaptive-Channel-Allocation-DARTS (ACA-DRATS), could eliminate the inconsistency in operation strength and significantly expand the architecture diversity. We continue to explore smaller search space under our framework, and offer a direct search on the entire ImageNet dataset. Experiments show that ACA-DRATS improves the search stability and significantly speeds up DARTS by more than ten times while yielding higher accuracy.