Hyperspectral images (HSIs) usually suffer from different types of pollution. This severely reduces the quality of HSIs and limits the accuracy of subsequent processing tasks. HSI denoising can be modeled as a low-rank tensor denoising problem. Tensor nuclear norm (TNN) induced by tensor singular value decomposition plays an important role in this problem. In this letter, we first reconsider three inconspicuous but crucial phenomenons in TNN. In the Fourier transform domain of HSIs, different frequency slices (FS) contain different information; different singular values (SVs) of each FS also represent different information. The two physical phenomenons lie not only in the spectral mode but also in the spatial modes. Then based on them, we propose a multi-modal and double-weighted TNN. It can adaptively shrink the FS and SVs according to their physical meanings in all modes of HSIs. In the framework of the alternating direction method of multipliers, we design an effective alternating iterative strategy to optimize our proposed model. Denoised experiments on both synthetic and real HSI datasets demonstrate their superiority against related methods.