The horizon line is a fundamental semantic feature in several maritime video processing tasks, such as digital video stabilization, camera calibration, target tracking, and target distance estimation. Visible range Electro-Optical (EO) sensors capture richer information in the daytime, which often comes with challenging clutter. The best methods rely on tailored filters to keep, ideally, only horizon edge pixels. These methods work well but often fail in the case of edge-degraded horizons. Our first aim is to solve this problem while taking the real-time constraint into account; we propose a tailored edge filter that relies on growing line segments with a low edge threshold and filters them based on their slope, length, and relative position. Next, we build the filtered edge map by computing Cartesian coordinates of pixels across line segments that survived the filter. We infer the horizon from the filtered edge map using line fitting techniques and simple temporal information. We consider the real-time constraint by vectorizing the computations and proposing a better way to leverage image downsizing. Extensive experiments on 26,125 visible range frames show that the proposed method achieves significant robustness while satisfying the real-time constraint.