Recent end-to-end multi-object detectors simplify the inference pipeline by removing the hand-crafted process such as the duplicate bounding box removal using non-maximum suppression (NMS). However, in the training, they require bipartite matching to calculate the loss from the output of the detector. Contrary to the directivity, which is at the heart of end-to-end learning, the bipartite matching makes the training of the end-to-end detector complex, heuristic, and reliant. In this paper, we propose a method to train an end-to-end multi-object detector without bipartite matching. To this end, we approach end-to-end multi-object detection as a density estimation problem using a mixture model. Our proposed detector, called Sparse Mixture Density Object Detector (Sparse MDOD), estimates the distribution of bounding boxes using a mixture model. Sparse MDOD is trained by minimizing the negative log-likelihood and our proposed regularization term, maximum component maximization (MCM) loss that prevents duplicated predictions. During training, no additional procedure such as bipartite matching is needed, and the loss is directly computed from the network outputs. Moreover, our Sparse MDOD outperforms the existing detectors on MS-COCO, a renowned multi-object detection benchmark.