Class incremental learning has attracted much attention, but most existing related works focus on fine-tuning the entire representation model, which inevitably results in much catastrophic forgetting. Instead of struggling to fight against such forgetting by replaying or distilling like most of the existing methods, we take a novel pre-train-and-prompt-tuning paradigm to sequentially learn new visual concepts based on a fixed semantic-rich pre-trained representation model. In detail, we incrementally prompt-tune category prototypes for classification and example prototypes to compensate for semantic drift, the problem caused by learning bias at different learning phases. Extensive experiments conducted on the mainstream incremental learning benchmarks demonstrate that our method outperforms other state-of-the-art methods.