Image Captioning is a popular vision-and-language task to generate the language description of an image. Recent advances focus on scaling up the model size and the number of training data, significantly increasing the cost of training. As an alternative to these heavy-cost models, we introduce I-Tuning, a lightweight image captioning framework, which contains only a small number of trainable parameters. The novel I-Tuning cross-attention module connects the non-trainable pre-trained language decoder GPT2 and vision encoder CLIP-ViT. Since most parameters are not updated during training, our framework is lightweight and fast. Experimental results on three image captioning benchmarks reveal that our framework achieves comparable or better performance than the large-scale baseline systems. At the same time, our models require up to 10 times fewer trainable parameters and much fewer training data.