The usage of Unmanned Aerial Vehicles (UAVs) in the context of structural health inspection is recently gaining tremendous popularity. Camera mounted UAVs enable the fast acquisition of a large number of images often used for mapping, 3D model reconstruction, and as an assisting tool for inspectors. Due to the number of images captured during large scale UAV surveys, a manual image-based inspection analysis of entire assets cannot be efficiently performed by qualified engineers. Additionally, comparing defects to past inspections requires the retrieval of relevant images which is often impractical without extensive metadata or computer-vision-based algorithms. In this paper, we propose an end-to-end method for automated structural inspection damage analysis. Using automated object detection and segmentation we accurately localize defects, bridge utilities and elements. Next, given the high overlap in UAV imagery, points of interest are extracted, and defects are located and matched throughout the image database, considerably reducing data redundancy while maintaining a detailed record of the defects. Our technique not only enables fast and robust damage analysis of UAV imagery, as we show herein, but is also effective for analyzing manually acquired images.