For image restoration, the majority of existing deep learning-based algorithms have a tendency to overfit the training data, resulting in poor performance when confronted with unseen degradations. To achieve more robust restoration, generative adversarial network (GAN) prior based methods have been proposed, demonstrating a promising capacity to restore photo-realistic and high-quality results. However, these methods are susceptible to semantic ambiguity, particularly with semantically relevant images such as facial images. In this paper, we propose a semantic-aware latent space exploration method for image restoration (SAIR). By explicitly modeling referenced semantics information, SAIR is able to reliably restore severely degraded images not only to high-resolution highly-realistic looks but also to correct semantics. Quantitative and qualitative experiments collectively demonstrate the effectiveness of the proposed SAIR. Our code can be found in https://github.com/Liamkuo/SAIR.