With the growth of residential rooftop PV adoption in recent decades, the problem of effective layout design has become increasingly important in recent years. Although a number of automated methods have been introduced, these tend to rely on simplifying assumptions and heuristics to improve computational tractability. We demonstrate a fully automated layout design pipeline that attempts to solve a more general formulation with greater geometric flexibility that accounts for shading losses. Our approach generates rooftop areas from satellite imagery and uses MINLP optimization to select panel positions, azimuth angles and tilt angles on an individual basis rather than imposing any predefined layouts. Our results demonstrate that shading plays a critical role in automated rooftop PV optimization and significantly changes the resulting layouts. Additionally, they suggest that, although several common heuristics are often effective, they may not be universally suitable due to complications resulting from geometric restrictions and shading losses. Finally, we evaluate a few specific heuristics from the literature and propose a potential new rule of thumb that may help improve rooftop solar energy potential when shading effects are considered.