This paper proposes an unsupervised anomaly detection technique for image-based plant disease diagnosis. A construction of large and openly available data set on labeled images of healthy and diseased crop plants has led to growing interest in computer vision techniques for automatic plant disease diagnosis. Although supervised image classifiers based on deep learning could be a powerful tool to identify plant diseases, they require huge amount of data sets that have been labeled as healthy and diseased. While, data mining techniques called "anomaly detection" include unsupervised approaches that not require rare samples for training classifiers. The proposed method in this study focuses on the reconstructability of colors on plant images. We expect that a deep encoder decoder network trained for reconstructing colors of healthy plant images certainly fails to reconstruct colors of symptomatic regions. The main contributions of this work are as follows: (i) we propose a new image-based plant disease detection framework utilizing a conditional adversarial network called pix2pix, (ii) we introduce a new anomaly score based on CIEDE2000 color difference. Through experiments using PlantVillage dataset, we demonstrate that our method is superior to an existing anomaly detector called AnoGAN for identifying diseased crop images in terms of accuracy, interpretability and computational efficiency.