Egocentric hand-object segmentation (EgoHOS) is a promising new task aiming at segmenting hands and interacting objects in egocentric images. Although EgoHOS has the potential to enable various applications, current methods struggle to achieve both high performance and end-to-end optimization simultaneously. Moreover, existing approaches fail to fully leverage hand cues to assist the interacting-object segmentation and overlook the coupled relationships between diverse interacting-object categories, resulting in performance deficiencies. To address these limitations, this paper proposes a novel Object-centric Relationship Modeling Network (ORMNet) to fulfill end-to-end and effective EgoHOS by modeling relationships between hands and objects as well as objects and objects. Specifically, a Hand-Object Relation (HOR) module is introduced to capture the correlation between hands and objects, which uses hand features to guide the network to extract more distinguishing interacting-object features. Besides, we find the coupling relations between diverse interacting-object categories and design the Object Relation Decoupling (ORD) strategy to disentangle them, emphasizing learning of the interaction between hands and objects and reducing the confusion of interacting-object classification. In-domain experiments show that ORMNet has notably exceptional segmentation performance compared with state-of-the-art methods, while out-of-domain experiments further exhibit its robust generalization capability. The project is available at https://github.com/yuggiehk/ORMNet/