An accurate and unbiased examination of skin lesions is critical for the early diagnosis and treatment of skin cancers. The visual feature of the skin lesions varies significantly because skin images are collected from patients with different skin colours by using various devices. Recent studies have developed ensembled convolutional neural networks (CNNs) to classify the images for early diagnosis. However, the practical use of CNNs is limited because their network structures are heavyweight and neglect contextual information. Vision transformers (ViTs) learn the global features by self-attention mechanisms, but they also have comparatively large model sizes (more than 100M). To address these limitations, we introduce HierAttn, a lite and effective neural network with hierarchical and self attention. HierAttn applies a novel strategy based on learning local and global features by a multi-stage and hierarchical network. The efficacy of HierAttn was evaluated by using the dermoscopy images dataset ISIC2019 and smartphone photos dataset PAD-UFES-20. The experimental results show that HierAttn achieves the best top-1 accuracy and AUC among state-of-the-art mobile networks, including MobileNetV3 and MobileViT. The code is available at https://github.com/anthonyweidai/HierAttn.