Salient object detection (SOD) is viewed as a pixel-wise saliency modeling task by traditional deep learning-based methods. Although great progress has been made, a challenge of modern SOD models is the insufficient utilization of inter-pixel information, which usually results in imperfect segmentations near the edge regions. As we demonstrate, using a saliency map as the network output is a sub-optimal choice. To address this problem, we propose a connectivity-based approach named bilateral connectivity network (BiconNet), which uses a connectivity map instead of a saliency map as the network output for effective modeling of inter-pixel relationships and object saliency. Moreover, we propose a bilateral voting module to enhance the output connectivity map and a novel edge feature enhancement method that efficiently utilizes edge-specific features with negligible parameter increase. We show that our model can use any existing saliency-based SOD framework as its backbone. Through comprehensive experiments on five benchmark datasets, we demonstrate that our proposed method outperforms state-of-the-art SOD approaches.