FPDIP-GAN: Feature Patch Discriminator Based Image Inpainting Using Generative Adversarial Network

Abstract

Image inpainting is the art of reconstructing damaged pixels in an image such that the completed image is realistic-looking and follows the original context. Over the last few years, many deep learning technique-based approaches have been proposed for image inpainting. however, those techniques come up short to recreate sensible structures due to the brokenness of the nearby pixels so the methods generate contents with blurry textures and misshaped structures. To handle those challenges, we investigate the human behavior to reconstruct the damaged picture and propose Feature patch discriminator-based Image Inpainting using a GAN-based approach which can not only preserve contextual structure but also make more effective predictions of missing parts by modeling the semantic relevance between the holes features. The architecture is designed based on an attention mechanism(CSA and Self-attention)to maintain global semantic structure and generate realistic texture details for the missing regions. The architecture task is divided into rough and refinement as two steps and model each step with a neural network under U-Net architecture. The architecture utilized a skip connection that was introduced between the same level of encoder and decoder to improve localization information and tackle the vanishing gradient problem. To improve the execution of the network with small computational complexity EfficientNet-B7 is utilized as a feature extractor in the generator network. The architecture takes use of a feature patch discriminator, which not only speeds up training and maintains stability but also helps the refinement network provide more significant high-frequency details. The self-attention mechanism was utilized in this work to capture long-range dependencies between channels and pixels instead of relying only on local convolution filters. Moreover, the PSNR and SSIM scores of the proposed solution along with different experiments have been measured and the proposed model improves the PSNR value from 26.54 to 29.80 and SSIM value from 0.931 to 0.952. Finally, from the evaluation results,the added EfficientNet-B7, self-attention, and residual blocks have improved the quality of the image being reconstructed.