We demonstrate that neural network layers that explicitly combine frequency and image feature representations are a versatile building block for analysis of imaging data acquired in the frequency space. Our work is motivated by the challenges arising in MRI acquisition where the signal is a corrupted Fourier transform of the desired image. The joint learning schemes proposed and analyzed in this paper enable both correction of artifacts native to the frequency space and manipulation of image space representations to reconstruct coherent image structures. This is in contrast to most current deep learning approaches for image reconstruction that apply learned data manipulations solely in the frequency space or solely in the image space. We demonstrate the advantages of joint convolutional learning on three diverse tasks: image reconstruction from undersampled acquisitions, motion correction, and image denoising in brain and knee MRI. We further demonstrate advantages of the joint learning approaches across training schemes using a wide variety of loss functions. Unlike purely image based and purely frequency based architectures, the joint models produce consistently high quality output images across all tasks and datasets. Joint image and frequency space feature representations promise to significantly improve modeling and reconstruction of images acquired in the frequency space. Our code is available at https://github.com/nalinimsingh/interlacer.