While neural networks allow highly accurate predictions in many tasks, their lack in robustness towards even slight input perturbations hampers their deployment in many real-world applications. Recent research towards evaluating the robustness of neural networks such as the seminal projected gradient descent (PGD) attack and subsequent works and benchmarks have therefore drawn significant attention. Yet, such methods focus predominantly on classification tasks, while only a few approaches specifically address the analysis of pixel-wise prediction tasks such as semantic segmentation, optical flow, or disparity estimation. One notable exception is the recently proposed SegPGD attack, which could showcase the importance of pixel-wise attacks for evaluating semantic segmentation. While SegPGD is limited to pixel-wise classification (i.e. segmentation), in this work, we propose CosPGD, a novel white-box adversarial attack that allows to optimize dedicated attacks for any pixel-wise prediction task in a unified setting. It leverages the cosine similarity between the predictions and ground truth to extend directly from classification tasks to regression settings. Further, we empirically show the superior performance of CosPGD for semantic segmentation as well as for optical flow and disparity estimation.