Recent advancements in computer vision, particularly by making use of deep learning, have drastically improved human motion analysis in videos. However, these improvements have not yet fully translated into improved performance in clinical in-bed scenarios due to the lack of public datasets representative of this scenario. To address this issue, we introduce BlanketSet, an RGB-IR-D action recognition dataset of sequences performed in a hospital bed. This dataset has the potential to help bridge the improvements attained in general use cases to these clinical scenarios. The dataset is available at https://rdm.inesctec.pt/dataset/nis-2022-004