Natural scene statistics based blind image quality assessment and repair

Abstract

Progress in multimedia technologies has resulted in a plethora of services and devices that capture, compress, transmit and display audiovisual stimuli. Humans -- the ultimate receivers of such stimuli -- now have access to visual entertainment at their homes, their workplaces as well as on mobile devices. With increasing visual signals being received by human observers, in the face of degradations that occur to due the capture, compression and transmission processes, an important aspect of the quality of experience of such stimuli is the \emph{perceived visual quality}. This dissertation focuses on algorithm development for assessing such visual quality of natural images, without need for the `pristine' reference image, i.e., we develop computational models for no-reference image quality assessment (NR IQA).

Our NR IQA model stems from the theory that natural images have certain statistical properties that are violated in the presence of degradations, and quantifying such deviations from \emph{naturalness} leads to a blind estimate of quality. The proposed modular and easily extensible framework is distortion-agnostic, in that it does not need to have knowledge of the distortion afflicting the image (contrary to most present-day NR IQA algorithms) and is not only capable of quality assessment with high correlation with human perception, but also is capable of identifying the distortion afflicting the image. This additional distortion-identification, coupled with blind quality assessment leads to a framework that allows for blind general-purpose image repair, which is the second major contribution of this dissertation. The blind general-purpose image repair framework, and its exemplar algorithm described here stem from a revolutionary perspective on image repair, where the framework does not simply attempt to ameliorate the distortion in the image, but to ameliorate the distortion, so that visual quality at the output is maximized.

Lastly, this dissertation describes a large-scale human subjective study that was conducted at UT to assess human behavior and opinion on visual quality of videos when viewed on mobile devices. The study lead to a database of 200 distorted videos, which incorporates previously studied distortions such as compression and wireless packet-loss, and also dynamically varying distortions that change as a function of time, such as frame-freezes and temporally varying compression rates. This study -- the first of its kind -- involved over 50 human subjects and resulted in 5,300 summary subjective scores and time-sampled subjective traces of quality for multiple displays. The last part of this dissertation analyzes human behavior and opinion on time-varying video quality, opening up an extremely interesting and relevant field for future research in the area of quality assessment and human behavior.