The local descriptors have gained wide range of attention due to their enhanced discriminative abilities. It has been proved that the consideration of multi-scale local neighborhood improves the performance of the descriptor, though at the cost of increased dimension. This paper proposes a novel method to construct a local descriptor using multi-scale neighborhood by finding the local directional order among the intensity values at different scales in a particular direction. Local directional order is the multi-radius relationship factor in a particular direction. The proposed local directional order pattern (LDOP) for a particular pixel is computed by finding the relationship between the center pixel and local directional order indexes. It is required to transform the center value into the range of neighboring orders. Finally, the histogram of LDOP is computed over whole image to construct the descriptor. In contrast to the state-of-the-art descriptors, the dimension of the proposed descriptor does not depend upon the number of neighbors involved to compute the order; it only depends upon the number of directions. The introduced descriptor is evaluated over the image retrieval framework and compared with the state-of-the-art descriptors over challenging face databases such as PaSC, LFW, PubFig, FERET, AR, AT&T, and ExtendedYale. The experimental results confirm the superiority and robustness of the LDOP descriptor.