Image restoration is an important and challenging task in computer vision. Reverting a filtered image to its original image is helpful in various computer vision tasks. We employ a nonlinear activation function free network (NAFNet) for a fast and lightweight model and add a color attention module that extracts useful color information for better accuracy. We propose an accurate, fast, lightweight network with multi-scale and color attention for Instagram filter removal (CAIR). Experiment results show that the proposed CAIR outperforms existing Instagram filter removal networks in fast and lightweight ways, about 11× faster and 2.4× lighter while exceeding 3.69 dB PSNR on IFFI dataset. CAIR can successfully remove the Instagram filter with high quality and restore color information in qualitative results. The source code and pretrained weights are available at https://github.com/HnV-Lab/CAIR.