The development of a three-dimensional (3D) molecular generative model based on protein pockets has recently attracted a lot of attention. This type of model aims to achieve the simultaneous generation of molecular graphs and 3D binding conformation under the constraint of protein binding. Various pocket-based generative models have been proposed; however, currently, there is a lack of systematic and objective evaluation metrics for these models. To address this issue, a comprehensive benchmark data set, named POKMOL-3D, is proposed to evaluate protein pocket-based 3D molecular generative models. It includes 32 protein targets together with their known active compounds as a test set to evaluate the versatility of generation models to mimic the real-world scenario. Additionally, a series of two-dimensional (2D) and 3D evaluation metrics with some newly created ones was integrated to assess the quality of generated molecular structures and their binding conformations. It is expected that this work can enhance our comprehension of the effectiveness and weakness of current 3D generative models and stimulate the discussion on challenges and useful guidance for developing the next wave of molecular generative models.