Convolutional Neural Networks (CNNs) have become the state of the art method for image classification in the last ten years. Despite the fact that they achieve superhuman classification accuracy on many popular datasets, they often perform much worse on more abstract image classification tasks. We will show that these difficult tasks are linked to relational concepts from the field of concept learning. We will review deep learning research that is linked to this area, even if it was not originally presented from this angle. Reviewing the current literature, we will argue that used datasets lead to an overestimate of system performance by providing data in a pre-attended form, by overestimating the true variability and complexity of the given tasks, and other shortcomings. We will hypothesise that iterative processing of the input, together with attentional shifts, will be needed to efficiently and reliably solve relational reasoning tasks with deep learning methods.