Zigzag flattening (ZF) is commonly utilized as a default option to get the image patches ordering in deep models, e.g., vision transformers (ViTs). Notably, when decomposing multi-scale images, ZF could not maintain the invariance of feature point positions. To this end, we investigate Hilbert flattening (HF) as an alternative for sequence ordering in vision tasks. Specifically, we give the theoretical bounds of HF and ZF in maintaining the spatial locality of the input features, showing the superiority of HF over ZF in matrix unfolding. Moreover, we further prove that such locality-preserving capability of HF is robust against the input scale variance in theory. In the application of HF, we design a Hilbert patch embedding for transformers to empirically demonstrate its scale-invariant locality-preserving in image classification and segmentation. This is also validated in feature down-sampling and feature/image interpolation. Extensive experiments demonstrate that it can yield consistent performance boosts for several popular architectures and applications.