We present a system for object recognition based on a semantic graph representation, which the system can learn from image examples. This graph is based on intrinsic properties of objects such as structure and geometry, so it is more robust than the current machine learning methods that can be fooled by changing a few pixels. Current methods have proved to be powerful but brittle because they ignore the structure and semantics of the objects. We define semantics as a form of abstraction, in terms of the intrinsic properties of the object, not in terms of human perception. Thus, it can be learned automatically. This is facilitated by the graph having two distinct hierarchies: abstraction and parts, which also makes its representation of objects more accurate and versatile. Previous semantic networks had only one amorphous hierarchy and were difficult to build and traverse. Our system performs both the learning and recognition by an algorithm that traverses both hierarchies at the same time, combining the advantages of top-down and bottom-up strategies. This reduces dimensionality and obviates the need for the brute force of big data training.