In medical applications, deep learning methods are built to automate diagnostic tasks, often formulated as single-target classification problems. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first transformer in this model aims to analyze the imaging data, and the second one leverages its internal states as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes, and forecasting Alzheimer's disease clinical status. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics, including calibration, which is desired from a medical decision support system. An open source implementation of our method is made publicly available at https://github.com/MIPT-Oulu/CLIMAT.