The image nonlocal self-similarity (NSS) prior refers to the fact that a local patch often has many nonlocal similar patches to it across the image. In this paper we apply such NSS prior to enhance the robust quaternion matrix completion (QMC) method and significantly improve the inpainting performance. A patch group based NSS prior learning scheme is proposed to learn explicit NSS models from natural color images. The NSS-based QMC algorithm computes an optimal low-rank approximation to the high-rank color image, resulting in high PSNR and SSIM measures and particularly the better visual quality. A new joint NSS-base QMC method is also presented to solve the color video inpainting problem based quaternion tensor representation. The numerical experiments on large-scale color images and videos indicate the advantages of NSS-based QMC over the state-of-the-art methods.