Dual energy X-ray Computed Tomography (DECT) enables to automatically decompose materials in clinical images without the manual segmentation using the dependency of the X-ray linear attenuation with energy. In this work we propose a deep learning procedure called End-to-End Material Decomposition (E2E-DEcomp) for quantitative material decomposition which directly convert the CT projection data into material images. The algorithm is based on incorporating the knowledge of the spectral model DECT system into the deep learning training loss and combining a data-learned prior in the material image domain. Furthermore, the training does not require any energy-based images in the dataset but rather only sinogram and material images. We show the effectiveness of the proposed direct E2E-DEcomp method on the AAPM spectral CT dataset (Sidky and Pan, 2023) compared with state of the art supervised deep learning networks.