Reports regarding the misuse of Generative AI (GenAI) to create harmful deepfakes are emerging daily. Recently, defensive watermarking, which enables GenAI providers to hide fingerprints in their images to later use for deepfake detection, has been on the rise. Yet, its potential has not been fully explored. We present UnMarker -- the first practical universal attack on defensive watermarking. Unlike existing attacks, UnMarker requires no detector feedback, no unrealistic knowledge of the scheme or similar models, and no advanced denoising pipelines that may not be available. Instead, being the product of an in-depth analysis of the watermarking paradigm revealing that robust schemes must construct their watermarks in the spectral amplitudes, UnMarker employs two novel adversarial optimizations to disrupt the spectra of watermarked images, erasing the watermarks. Evaluations against the SOTA prove its effectiveness, not only defeating traditional schemes while retaining superior quality compared to existing attacks but also breaking semantic watermarks that alter the image's structure, reducing the best detection rate to 43% and rendering them useless. To our knowledge, UnMarker is the first practical attack on semantic watermarks, which have been deemed the future of robust watermarking. UnMarker casts doubts on the very penitential of this countermeasure and exposes its paradoxical nature as designing schemes for robustness inevitably compromises other robustness aspects.