Scroll wave chaos is thought to underlie life-threatening ventricular fibrillation. However, currently there is no direct way to measure action potential wave patterns transmurally throughout the thick ventricular heart muscle. Consequently, direct observations of three-dimensional electrical scroll waves remains elusive. Here, we study whether it is possible to reconstruct simulated scroll waves and scroll wave chaos using deep learning. We trained encoding-decoding convolutional neural networks to predict three-dimensional scroll wave dynamics inside bulk-shaped excitable media from two-dimensional observations of the wave dynamics on the bulk's surface. We tested whether observations from one or two opposing surfaces would be sufficient, and whether transparency or measurements of surface deformations enhances the reconstruction. Further, we evaluated the approach's robustness against noise and tested the feasibility of predicting the bulk's thickness. We distinguished isotropic and anisotropic, as well as opaque and transparent excitable media as models for cardiac tissue and the Belousov-Zhabotinsky chemical reaction, respectively. While we demonstrate that it is possible to reconstruct three-dimensional scroll wave dynamics, we also show that it is challenging to reconstruct complicated scroll wave chaos and that prediction outcomes depend on various factors such as transparency, anisotropy and ultimately the thickness of the medium compared to the size of the scroll waves. In particular, we found that anisotropy provides crucial information for neural networks to decode depth, which facilitates the reconstructions. In the future, deep neural networks could be used to visualize intramural action potential wave patterns from epi- or endocardial measurements.