Perceptually-Informed No-Reference Image Harmonisation

Dolhasz, A ORCID: https://orcid.org/0000-0002-6520-8094, Harvey, C ORCID: https://orcid.org/0000-0002-4809-1592 and Williams, I ORCID: https://orcid.org/0000-0002-0651-0963 (2022) Perceptually-Informed No-Reference Image Harmonisation. In: 15th International Joint Conference: VISIGRAPP 2020, 27 February 2020 – 29 February 2020, Valletta, Malta.

Accepted Version Available under License In Copyright. Download (14MB)

Abstract

Many image synthesis tasks, such as image compositing, rely on the process of image harmonisation. The goal of harmonisation is to create a plausible combination of component elements. The subjective quality of this combination is directly related to the existence of human-detectable appearance differences between these component parts, suggesting that consideration for human perceptual tolerances is an important aspect of designing automatic harmonisation algorithms. In this paper, we first investigate the impact of a perceptually-calibrated composite artifact detector on the performance of a state-of-the-art deep harmonisation model. We first evaluate a two-stage model, whereby the performance of both pre-trained models and their naive combination is assessed against a large data-set of 68128 automatically generated image composites. We find that without any task-specific adaptations, the two-stage model achieves comparable results to the baseline harmoniser fed with ground truth composite masks. Based on these findings, we design and train an end-to-end model, and evaluate its performance against a set of baseline models. Overall, our results indicate that explicit modeling and incorporation of image features conditioned on a human perceptual task improves the performance of no-reference harmonisation algorithms. We conclude by discussing the generalisability of our approach in the context of related work.