3D reconstruction and novel view rendering can greatly benefit from geometric priors when the input views are not sufficient in terms of coverage and inter-view baselines. Deep learning of geometric priors from 2D images often requires each image to be represented in a 2D canonical frame and the prior to be learned in a given or learned 3D canonical frame. In this paper, given only the relative poses of the cameras, we show how to learn priors from multiple views equivariant to coordinate frame transformations by proposing an SE(3)-equivariant convolution and transformer in the space of rays in 3D. This enables the creation of a light field that remains equivariant to the choice of coordinate frame. The light field as defined in our work, refers both to the radiance field and the feature field defined on the ray space. We model the ray space, the domain of the light field, as a homogeneous space of SE(3) and introduce the SE(3)-equivariant convolution in ray space. Depending on the output domain of the convolution, we present convolution-based SE(3)-equivariant maps from ray space to ray space and to R^3. Our mathematical framework allows us to go beyond convolution to SE(3)-equivariant attention in the ray space. We demonstrate how to tailor and adapt the equivariant convolution and transformer in the tasks of equivariant neural rendering and 3D reconstruction from multiple views. We demonstrate SE(3)-equivariance by obtaining robust results in roto-translated datasets without performing transformation augmentation.