Existing deep convolutional neural network (CNN) architectures frequently rely upon batch normalization (BatchNorm) to effectively train the model. BatchNorm significantly improves model performance in centralized training, but it is unsuitable for federated learning and differential privacy settings. Even in centralized learning, BatchNorm performs poorly with smaller batch sizes. To address these limitations, we propose kernel normalization and kernel normalized convolutional layers, and incorporate them into kernel normalized convolutional networks (KNConvNets) as the main building blocks. We implement KNConvNets corresponding to the state-of-the-art CNNs such as VGGNets and ResNets while forgoing BatchNorm layers. Through extensive experiments, we illustrate KNConvNets consistently outperform their batch, group, and layer normalized counterparts in terms of both accuracy and convergence rate in centralized, federated, and differentially private learning settings.