Sparse adversarial perturbations received much less attention in the literature compared to l_2- and l_∞-attacks. However, it is equally important to accurately assess the robustness of a model against sparse perturbations. Motivated by this goal, we propose a versatile framework based on random search, Sparse-RS, for score-based sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on substitute models and achieves state-of-the-art success rate and query efficiency for multiple sparse attack models: l_0-bounded perturbations, adversarial patches, and adversarial frames. Unlike existing methods, the l_0-version of untargeted Sparse-RS achieves almost 100% success rate on ImageNet by perturbing only 0.1% of the total number of pixels, outperforming all existing white-box attacks including l_0-PGD. Moreover, our untargeted Sparse-RS achieves very high success rates even for the challenging settings of 20×20 adversarial patches and 2-pixel wide adversarial frames for 224×224 images. Finally, we show that Sparse-RS can be applied for universal adversarial patches where it significantly outperforms transfer-based approaches. The code of our framework is available at https://github.com/fra31/sparse-rs.