New AI technology enables 3D capture and editing of real-world objects

Imagine sweeping around an object with your smartphone and getting a realistic, fully editable 3D model that you can view from any angle. Thanks to advances in AI, this is rapidly becoming a reality.

Researchers at Simon Fraser University (SFU) in Canada have announced a new AI technology to do just that. Soon, the everyday consumer will not only be able to simply take his 2D photo, but he will also be able to take his 3D capture of a real object and view its shape and shape as easily as he takes a regular 2D photo of him today. You will be able to freely edit the appearance.

In a new paper published in arXiv Utilizing a preprint server and presenting at the 2023 Neural Information Processing Systems Conference (NeurIPS) in New Orleans, Louisiana, researchers developed Proximity Attention Point Rendering (PAPR), which can transform a set of 2D photos of an object into a cloud. ) demonstrated a new technology called 3D points that represent the shape and appearance of an object.

Each point provides the user with a knob to control the object. Dragging a point changes the object's shape, and editing point properties changes the object's appearance. Then, in a process known as “rendering,” the 3D point cloud is viewed from any angle and transformed into a 2D photo that shows the edited object as if it were actually photographed from that angle.

Researchers have shown how to bring statues to life using new AI technology. The technology automatically converts a series of photos of a statue into a 3D point cloud and animates it. The end result is a video of the statue turning its head from side to side as the viewer is guided through the path around it.

“AI and machine learning are really driving a paradigm shift in reconstructing 3D objects from 2D images.The remarkable success of machine learning in fields such as computer vision and natural language is encouraging researchers to It encourages us to investigate how graphics pipelines can be redesigned,” said Dr. Ke Li, Assistant Professor of Computer Science, Director of the APEX Lab, and Senior Research Scientist at Simon Fraser University (SFU). He said: Author of the paper.

“We've found that making this successful is much harder than expected and requires overcoming several technical challenges. What excites me most is what this will bring to consumer technology. There are many possibilities. 3D may become as common as ever as a medium for visual communication and expression. 2D is today. ”

One of the biggest challenges in 3D is how to represent 3D shapes in a way that allows users to easily and intuitively edit them. The previous approach, known as Neural Radiance Field (NeRF), requires the user to describe what happens to every continuous coordinate and does not allow for easy editing of the shape. A more recent approach known as 3D Gaussian splatting (3DGS) is also not well suited for shape editing, as the surface of the shape can become shattered or disjointed after editing.