Event cameras are neuromorphic vision sensors representing visual information as sparse and asynchronous event streams. Most state-of-the-art event-based methods project events into dense frames and process them with conventional learning models. However, these approaches sacrifice the sparsity and high temporal resolution of event data, resulting in a large model size and high computational complexity. To fit the sparse nature of events and sufficiently explore the relationship between them, we develop a novel attention-aware model named Event Voxel Set Transformer (EVSTr) for spatiotemporal representation learning on event streams. It first converts the event stream into voxel sets and then hierarchically aggregates voxel features to obtain robust representations. The core of EVSTr is an event voxel transformer encoder to extract discriminative spatiotemporal features, which consists of two well-designed components, including a Multi-Scale Neighbor Embedding Layer (MNEL) for local information aggregation and a Voxel Self-Attention Layer (VSAL) for global feature interactions. Enabling the network to incorporate a long-range temporal structure, we introduce a segment modeling strategy to learn motion patterns from a sequence of segmented voxel sets. We evaluate the proposed model on two event-based recognition tasks: object classification and action recognition. Comprehensive experiments show that EVSTr achieves state-of-the-art performance while maintaining low model complexity. Additionally, we present a new dataset (NeuroHAR) recorded in challenging visual scenarios to complement the lack of real-world event-based datasets for action recognition.