Deep learning models are used to minimize the number of polyps that goes unnoticed by the experts and to accurately segment the detected polyps during interventions. Although state-of-the-art models perform well, they require too many parameters, which can pose a problem with real-time applications. To address this problem, we propose a novel segmentation model called PlutoNet which requires only 2,626,537 parameters while outperforming state-of-the-art models on several datasets. With PlutoNet, we propose a novel consistency training approach that consists of a shared encoder, the modified partial decoder, and the auxiliary decoder that are trained with a combined loss to enforce consistency. We use a lightweight architecture and propose the modified partial decoder, which is a combination of the partial decoder and full-scale connections that further reduces the number of parameters required, as well as captures semantic details. We use asymmetric convolutions to handle varying polyp sizes and then we weigh each feature map by using a squeeze and excitation block. Then, we enforce consistency by combining the loss of the modified partial decoder and the auxiliary decoder, which helps improve the encoder's representations without losing learned relevant semantic details. This way we are able to reduce false positive rates. We perform extensive experiments to show that our model outperforms the state-of-the-art models and is able to generalize well to several datasets. Our model outperforms the state-of-the-art models on the EndoScene dataset, Etis dataset, 2018 Data Science Bowl Challenge dataset, and Kvasir Instrument dataset. We also perform an ablation study to show the effectiveness of each component of our model.