It is increasingly common for satellites to lose connection with the ground stations on Earth with which they communicate, due to signal interruptions from the Earth's ionosphere and magnetosphere. Given the important roles that satellites play in national defense, public safety, and worldwide communications, finding ways to determine satellite trajectories in such situations is a crucially important task. In this paper, we demonstrate the efficacy of a novel computer vision based approach, which relies on earth imagery taken by the satellite itself, to determine the orbit of a satellite that has lost contact with its ground stations. We empirically observe significant improvements by more than an order of magnitude, over the present state of the art approach, namely, the Gibbs method for an initial orbit estimate with the Kalman filter for differential error correction. We further investigate the performance of the approach by comparing various neural networks, namely, ResNet50, ResNet101, VGG19, VGG16, AlexNet, and CoAtNet4.