Over the years, a number of biometric template protection schemes, primarily based on the notion of "cancelable biometrics" (CB) have been proposed. An ideal cancelable biometric algorithm possesses four criteria, i.e., irreversibility, revocability, unlinkability, and performance preservation. Cancelable biometrics employed an irreversible but distance preserving transform to convert the original biometric templates to the protected templates. Matching in the transformed domain can be accomplished due to the property of distance preservation. However, the distance preservation property invites security issues, which are often neglected. In this paper, we analyzed the property of distance preservation in cancelable biometrics, and subsequently, a pre-image attack is launched to break the security of cancelable biometrics under the Kerckhoffs's assumption, where the cancelable biometrics algorithm and parameters are known to the attackers. Furthermore, we proposed a framework based on mutual information to measure the information leakage incurred by the distance preserving transform, and demonstrated that information leakage is theoretically inevitable. The results examined on face, iris, and fingerprint revealed that the risks origin from the matching score computed from the distance/similarity of two cancelable templates jeopardize the security of cancelable biometrics schemes greatly. At the end, we discussed the security and accuracy trade-off and made recommendations against pre-image attacks in order to design a secure biometric system.