The automatic creation of geometric models from point clouds has numerous applications in CAD (e.g., reverse engineering, manufacturing, assembling) and, more in general, in shape modelling and processing. Given a segmented point cloud representing a man-made object, we propose a method for recognizing simple geometric primitives and their interrelationships. Our approach is based on the Hough transform (HT) for its ability to deal with noise, missing parts and outliers. In our method we introduce a novel technique for processing segmented point clouds that, through a voting procedure, is able to provide an initial estimate of the geometric parameters characterizing each primitive type. By using these estimates, we localize the search of the optimal solution in a dimensionally-reduced parameter space thus making it efficient to extend the HT to more primitives than those that are generally found in the literature, i.e. planes and spheres. Then, we extract a number of geometric descriptors that uniquely characterize a segment, and, on the basis of these descriptors, we show how to aggregate parts of primitives (segments). Experiments on both synthetic and industrial scans reveal the robustness of the primitive fitting method and its effectiveness for inferring relations among segments.