Watermarking serves as a widely adopted approach to safeguard media copyright. In parallel, the research focus has extended to watermark removal techniques, offering an adversarial means to enhance watermark robustness and foster advancements in the watermarking field. Existing watermark removal methods often rely on UNet architectures with multiple decoder branches -- one for watermark localization and the other for background image restoration. These methods involve complex module designs to guide information flow for respective tasks, which can lead to suboptimal performance and an overly cumbersome model. To simplify the existing framework, we propose a novel Transformer-based approach with a unified decoder branch, treating watermark extraction and background restoration as a single task and allowing thenetwork to learn information flow between them without artificial design patterns. Additionally, we utilize nested structures to facilitate multi-scale feature fusion, forming a parallel ensemble of nested structures that constitute the UNet. Supervision is applied to UNets with varying depths to facilitate knowledge learning across all levels. Extensive experiments are conducted on various challenging benchmarks to validate the effectiveness of our proposed method. The results demonstrate that our approach achieves state-of-the-art performance and produces high-quality images.