Universal domain adaptation (UDA) aims to transfer the knowledge of common classes from source domain to target domain without any prior knowledge on the label set, which requires to distinguish the unknown samples from the known ones in the target domain. Recent methods preferred to increase the inter-sample affinity within a known class, while they ignored the inter-sample affinity between the unknown samples and the known ones. This paper reveals that exploiting such inter-sample affinity can significantly improve the performance of UDA and proposes a knowability-aware UDA framework based on it. First, we estimate the knowability of each target sample by searching its neighboring samples in the source domain. Then, we propose an auto-thresholding scheme applied to the estimated knowability to determine whether a target sample is unknown or known. Next, in addition to increasing the inter-sample affinity within each known class like previous methods, we design new losses based on the estimated knowability to reduce the inter-sample affinity between the unknown target samples and the known ones. Finally, experiments on four public datasets demonstrate that our method significantly outperforms existing state-of-the-art methods.