Benefiting from the continuous representation ability, deep implicit functions can represent a shape at infinite resolution. However, extracting high-resolution iso-surface from an implicit function requires forward-propagating a network with a large number of parameters for numerous query points, thus preventing the generation speed. Inspired by the Taylor series, we propose Taylo3DNet to accelerate the inference of implicit shape representations. Taylor3DNet exploits a set of discrete landmark points and their corresponding Taylor series coefficients to represent the implicit field of a 3D shape, and the number of landmark points is independent of the resolution of the iso-surface extraction. Once the coefficients corresponding to the landmark points are predicted, the network evaluation for each query point can be simplified as a low-order Taylor series calculation with several nearest landmark points. Based on this efficient representation, our Taylor3DNet achieves a significantly faster inference speed than classical network-based implicit functions. We evaluate our approach on reconstruction tasks with various input types, and the results demonstrate that our approach can improve the inference speed by a large margin without sacrificing the performance compared with state-of-the-art baselines.