PAPR in Motion: Seamless Point-level 3D Scene Interpolation

We propose the problem of point-level 3D scene interpolation, which aims to reconstruct a 3D scene in two different states from multiple views, synthesize a plausible smooth point-level interpolation between the 3D scenes in the two states, and render the 3D scene at any point in time from a novel view, all without any supervision in-between the states. The primary challenge lies in producing a smooth transition between the two states which can exhibit substantial changes in geometry. To tackle it, we leverage recent advances in point renderers, which are naturally suited to representing Lagrangian motion. Our approach works by initially learning a point-based representation of the scene in its starting state, followed by finetuning this model towards the end state. Critical to achieving smooth interpolation of both the scene's geometry and appearance is the choice of the point rendering technique. Different techniques excel along different performance dimensions, and we propose leveraging the recent Proximity Attention Point Rendering (PAPR) technique, which is designed to learn point clouds from scratch and support novel view synthesis of scenes after they undergo non-rigid geometric deformations. Our method, which we dub ``PAPR in Motion'', builds on PAPR's strengths and addresses its weaknesses by developing various regularization techniques specifically for the task. The end result is a method that can effectively bridge large scene changes, producing plausible and smooth geometry and appearance interpolations. When evaluating under diverse motion types, we find that PAPR in Motion achieves superior performance relative to the leading point renderer for dynamic scenes, thereby demonstrating an effective solution to this new and challenging problem of point-level 3D scene interpolation.