Real-world application requires affect perception models to be sensitive to individual differences in expression. As each user is different and expresses differently, these models need to personalise towards each individual to adequately capture their expressions and thus, model their affective state. Despite high performance on benchmarks, current approaches fall short in such adaptation. In this work, we propose the use of Continual Learning (CL) for affective computing as a paradigm for developing personalised affect perception.