Owing to resource limitations, efficient computation systems have long been a critical demand for those designing autonomous vehicles. In addition, sensor cost and size have restricted the development of self-driving cars. To overcome these restrictions, this study proposed an efficient framework for the operation of vision-based automatic vehicles; a front-facing camera and a few inexpensive radars are the required sensors for driving environment perception. The proposed algorithm comprises a multi-task UNet (MTUNet) network for extracting image features and constrained iterative linear quadratic regulator (CILQR) modules for rapid lateral and longitudinal motion planning. The MTUNet is designed to simultaneously solve lane line segmentation, ego vehicle heading angle regression, road type classification, and traffic object detection tasks at an approximate speed of 40 FPS (frames per second) when an RGB image of size 228 x 228 is fed into it. The linear CILQR controllers then apply processed MTUNet outputs and radar data as inputs to produce driving commands for lateral and longitudinal guidance related to autonomous vehicle operation; optimal control problems can be solved within 1 ms. The linear CILQR approaches are more efficient than the standard sequential quadratic programming (SQP) methods and can be combined with MTUNet for autonomous vehicle operation in simulated environments for lane-keeping and car-following maneuvers without the use of high-definition (HD) maps. Our experiments demonstrate that the proposed autonomous driving system is applicable to current automobile technology.