Image anomaly detection (IAD) is an urgent issue that needs to be addressed in modern industrial manufacturing (IM). Recently, many advanced algorithms have been released, but their performance varies greatly due to non-uniformed settings. That is, researchers find it difficult to analyze because they are designed for different or specific cases in IM. To eliminate this problem, we first propose a uniform IAD setting to systematically assess the effectiveness of these algorithms, mainly considering three aspects of supervision level (unsupervised, fully supervised), learning paradigm (few-shot, continual, noisy label), and efficiency (memory usage, inference speed). Then, we skillfully construct a comprehensive image anomaly detection benchmark (IM-IAD), which includes 19 algorithms on 7 major datasets with the same setting. Our extensive experiments (17,017 total) provide new insights into the redesign or selection of the IAD algorithm under uniform conditions. Importantly, the proposed IM-IAD presents feasible challenges and future directions for further work. We believe that this work can have a significant impact on the IAD field. To foster reproducibility and accessibility, the source code of IM-IAD is uploaded on the website, https://github.com/M-3LAB/IM-IAD.