Browsing by Subject "Neural rendering"

Now showing 1 - 1 of 1

Bridging virtual and physical world by recreating and manipulating visual contents
(2024-05) Jiang, Yifan, active 21st century; Wang, Zhangyang; Zhangyang Wang
Benefiting from the rapid development of deep neural networks, modern computer vision algorithms are capable of solving various high-level tasks, including classification, detection, and segmentation. In addition to understanding our physical world, another intriguing topic lies in the path of restoring the visual world in virtual applications. This includes but is not limited to accurately reconstructing 2D/3D visual content, creating new visual content based on the conditions, and manipulating visual content. Toward this goal, various approaches have gained more popularity and attention, such as the probabilistic generative model, neural rendering techniques, and the design of scalable architectures. Displaying a visual world in virtual applications, commonly referred to as "rendering", has been one of the most important goals in the area of computer graphics. Conventional methodologies build tools to render each pixel, such as rasterization or ray tracing, each offering different levels of realism and efficiency. These are achieved by formulating the physical world via a series of mathematical equations that are capable of simulating real-world textures and lighting conditions. However, modern computer vision algorithms have introduced a paradigm shift by replacing the aforementioned mathematical equations with deep neural networks, where the display formulation is directly learned from real-world data. Compared to traditional computer graphics tools, the learning-based "renderer" delivers more photo-realistic output with less computational costs. My PhD research contributes to building stronger learning frameworks with less computation and data demands, as well as improving the fidelity of generative content and enabling further manipulation. In this manuscript, we strive to develop learning-based algorithms with the capacity to faithfully recreate our visual reality to bridge the gap between the visual contents from the physical world and virtual applications. We aim to contribute to the advancement of the aforementioned techniques to enable more effective restoration of our visual world in digital formats, as well as facilitate the seamless rendering of artistic effects with greater ease and finesse. The manuscript covers: (1) Learning to infer the physical world from corrupted measurements; (2) Learning to reconstruct 3D visual content under limited conditions and manipulate it further; (3) Learning to efficiently/effectively formulate the distributions of the visual world and create new contents further.