Class incremental learning has attracted much attention, but most existing works still continually fine-tune the entire representation model, inevitably resulting in much catastrophic forgetting. Instead of struggling to fight against such forgetting by replaying or distillation like most of the existing methods, we take a novel pre-train-and-prompt-tuning paradigm to sequentially learn new visual concepts based on a fixed semantic-rich pre-trained representation model. In detail, we incrementally prompt-tune category prototypes for classification and example prototypes to compensate for semantic drift, the problem caused by learning bias at different phases. Extensive experiments conducted on the mainstream incremental learning benchmarks demonstrate that our method outperforms other state-of-the-art methods.