Low-cost thermal cameras are inaccurate (usually ±3^∘C) and have space-variant nonuniformity across their detector. Both inaccuracy and nonuniformity are dependent on the ambient temperature of the camera. The goal of this work was to estimate temperatures with low-cost infrared cameras, and rectify the nonuniformity. A nonuniformity simulator that accounts for the ambient temperature was developed. An end-to-end neural network that incorporates both the physical model of the camera and the ambient camera temperature was introduced. The neural network was trained with the simulated nonuniformity data to estimate the object's temperature and correct the nonuniformity, using only a single image and the ambient temperature measured by the camera itself. Results of the proposed method significantly improved the mean temperature error compared to previous works by up to 0.5^∘C. In addition, constraining the physical model of the camera with the network lowered the error by an additional 0.1^∘C. The mean temperature error over an extensive validation dataset was 0.37^∘C. The method was verified on real data in the field and produced equivalent results.