Domain generalization (DG) aims to learn from multiple known source domains a model that can generalize well to unknown target domains. The existing DG methods usually exploit the fusion of shared multi-source data for capturing domain invariance and training a generalizable model. However, tremendous data is distributed across lots of places nowadays that can not be shared due to strict privacy policies. A dilemma is thus raised between the generalization learning with shared multi-source data and the privacy protection of real-world sensitive data. In this paper, we introduce a separated domain generalization task with separated source datasets that can only be accessed locally for data privacy protection. We propose a novel solution called Collaborative Semantic Aggregation and Calibration (CSAC) to enable this challenging task. To fully absorb multi-source semantic information while avoiding unsafe data fusion, we conduct data-free semantic aggregation by fusing the models trained on the separated domains layer-by-layer. To address the semantic dislocation problem caused by domain shift, we further design cross-layer semantic calibration with an elaborate attention mechanism to align each semantic level and enhance domain invariance. We unify multi-source semantic learning and alignment in a collaborative way by repeating the semantic aggregation and calibration alternately, keeping each dataset localized, and the data privacy is thus carefully protected. Extensive experiments show the significant performance of our method in addressing this challenging task, which is even comparable to the previous DG methods with shared source data.