Deep learning-based medical volumetric segmentation methods either train the model from scratch or follow the standard ``pre-training then fine-tuning" paradigm. Although fine-tuning a pre-trained model on downstream tasks can harness its representation power, the standard full fine-tuning is costly in terms of computation and memory footprint. In this paper, we present the study on parameter-efficient transfer learning for medical volumetric segmentation and propose a new framework named Med-Tuning based on intra-stage feature enhancement and inter-stage feature interaction. Additionally, aiming at exploiting the intrinsic global properties of Fourier Transform for parameter-efficient transfer learning, a new adapter block namely Med-Adapter with a well-designed Fourier Transform branch is proposed for effectively and efficiently modeling the crucial global context for medical volumetric segmentation. Given a large-scale pre-trained model on 2D natural images, our method can exploit both the crucial spatial multi-scale feature and volumetric correlations along slices for accurate segmentation. Extensive experiments on three benchmark datasets (including CT and MRI) show that our method can achieve better results than previous parameter-efficient transfer learning methods on segmentation tasks, with much less tuned parameter costs. Compared to full fine-tuning, our method reduces the fine-tuned model parameters by up to 4x, with even better segmentation performance. The code will be made publicly available at https://github.com/jessie-chen99/Med-Tuning.