Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Neural Radiance Field

•
•
Dong Heon Cho
Dong Heon Cho

Neural Radiance Fields (NeRF) represent scenes as neural networks that map 5D input (3D position and 2D viewing direction) to a 4D output (RGB color and opacity). NeRF uses an MLP that is trained to predict volumetric density and color for a scene from many camera views. Key aspects of NeRF include using positional encodings as inputs to help model view-dependent effects, and training to optimize for integrated color and density values along camera rays. NeRF has enabled novel applications beyond novel view synthesis, including pose estimation, dense descriptors, and self-supervised segmentation.

1 of 24
Download to read offline
Neural Radiance Fields Neural Fields in Computer Vision
Field 스칼라장 : 온도, 압력, 오디오 등 벡터장 : 유체속도장, 자기장 등 텐서장 : 영상 등
Neural Field
3D Synthesize : COLMAP - Structure-from-Motion and Multi-View Stereo https://colmap.github.io/
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://neuralfields.cs.brown.edu/cvpr22 https://keras.io/examples/vision/nerf/
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf https://github.com/NakuraMino/Minimal-NeRF 1. MLP 기반의 NeRF * input의 Spherical Vector(2D)를 Cartesian(3D) 으로 변경 * input의 viewing direction : sigma, pi (Cartesian)가 3*2*L_embed로 확장 ( positional encoding ) * output은 RGBa (a는 volume density) => !!중요한점, 1 view는 1 pixel에 대응 8 Depth FC (with 256 dim - Relu)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py MLP 기반의 NeRF (Dataset from Blender)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py https://docs.blender.org/api/blender_python_api_2_71_release/ https://github.com/bmild/nerf/issues/78 https://www.3dgep.com/understanding-the-view-matrix/#:~:text=The%20world%20transformation%20matrix%20is,%2Dspace%20to%20view%2Dspace. MLP 기반의 NeRF (Dataset from Blender) camera_angle_x : Camera lens horizontal field of view <- 카메라의 Focal 값 (모두 동일값) rotation : radians (degree) <- 카메라 제자리 회전 값… (모두 동일값) transform_matrix (Worldspace transformation matrix, matrix_world (blender)) <- Camera 월 좌표계 및 상대 Rotation 정보
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) transform_matrix (Look-At 행렬) - Camera 2 World focal = .5 * W / np.tan(.5 * camera_angle_x)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) - rays_o는 rays_original (position) : 각 HxWx3 pixel의 original ray point를 의미 == 모든 점이 camera x,y,z로 똑같음 (각 픽셀마다의 camera position) rays_d는 direction: dirs는 HxW내에서의 상대 좌표. c2w를 곱해줌으로서 camera viewing direction(+depth)을 계산 (각 픽셀마다의 camera view direction)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) http://sgvr.kaist.ac.kr/~sungeui/GCG_S22/Student_Presentations/CS580_PaperPresentation_dgkim.pdf
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) - z_vals <- Random Accumulate Point 지정 pts <- z_vals위치의 point들을 생성 pts input에 대한 network output 값의 합산이 rgb가 될 수 있도록 함 (weights 값 합산은 near 1) => raw[:3]는 rgb_map과 mse, raw[4]는 직접 mse가 없고 raw[:3]와 곱해져 간접 업데이트 (본 코드에는 raw[4] - a는 업데이트 없음 fix z_vals)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://m.blog.naver.com/jetarrow82/221252045857 1. MLP 기반의 NeRF - Viewing Direction의 중요성 실제 물체는 non-Lambertian effects 특성을 가짐 Lambertian Surface는 ViewPoint에 상관없이 똑같은 빛의 양을 반사함
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.matthewtancik.com/nerf 1. MLP 기반의 NeRF - Positional Encoding
NeRF Result https://phog.github.io/snerg/viewer/index.html?dir=https://storage.googleapis.com/snerg/750/lego <= 같이 보자!
NeRF: Synthesizer로서의 가치 Small Size Model Rendering 도메인이 한정되어 있지 않음
Texture https://learnopengl.com/Advanced-OpenGL/Cubemaps https://lifeisforu.tistory.com/375 CubeMap - 보통 ray 영향을 크게 받지 않는 원경에 사용 Spherical Harmonics( SH, 구면 조화 ) - irradiance map(난반사 정도)을 저장하는 효율적 방식
Plenoxels : ReRF without Neural Networks NeRF key point is differentiable Volume Renderer MLP Volume Rendering을 Voxel Grid와 Interpolation의 조합으로 치환
Beyond Synthesize - iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020) Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS) https://yenchenlin.me/inerf/ Trained Nerf를 통해 Generation하고 Pose Estimation을 Iterative하게 Gradient Based Optimization을 하여 Estimated Pose를 취득
Beyond Synthesize - iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020) Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS) https://yenchenlin.me/inerf/
Beyond Synthesize - Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA) https://yenchenlin.me/nerf-supervision/ Keypoint detection과 유사. Keypoint는 sparse하게 pose 정보를 추출한다면, Dense Descriptor는 Dense하게 Pose 정보를 추출
Beyond Synthesize - Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA) https://yenchenlin.me/nerf-supervision/
Beyond Synthesize - NERF-SOS: ANY-VIEW SELF-SUPERVISED OBJECT SEGMENTATION ON COMPLEX SCENES (2022 Pre) - 셀링포인트는 다양한 각도에서 영상을 촬영하면 NeRF 샘플에서 Unsupervised로 Segmentation을 만들어줌 => 한계점 - K-Means/Pre-trained Network에 의존적 (BG가 명확하게 분리되고, Object의 Semantic 정보가 명확해야 할 것으로 보임)
NeRFs… https://kakaobrain.github.io/NeRF-Factory/ <= Nerf 성능 비교 Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs <= 넓은 공간 모델링 Instant Neural Graphics Primitive <= NeRF, SDF(signed distance functions), Gigapixel image등 (Computer graphics primitives)을 처리할 수 있는 네트워크 https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4 Dex-NeRF : Using a Neural Radiance Field to Grasp Transparent Objects - Transparent, Light-sensitive한 물체를 잘 다룰 수 있지 않을까? https://sites.google.com/view/dex-nerf vision-only robot navigation in a neural radiance world - Vision Based Planning https://mikh3x4.github.io/nerf-navigation/ Relighting the Real World With Neural Rendering - https://www.unite.ai/adobe-neural-rendering-relighting-2021/ NeRF-Studio https://docs.nerf.studio/en/latest/index.html

More Related Content

Neural Radiance Field

  • 1. Neural Radiance Fields Neural Fields in Computer Vision
  • 2. Field 스칼라장 : 온도, 압력, 오디오 등 벡터장 : 유체속도장, 자기장 등 텐서장 : 영상 등
  • 4. 3D Synthesize : COLMAP - Structure-from-Motion and Multi-View Stereo https://colmap.github.io/
  • 5. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://neuralfields.cs.brown.edu/cvpr22 https://keras.io/examples/vision/nerf/
  • 6. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf https://github.com/NakuraMino/Minimal-NeRF 1. MLP 기반의 NeRF * input의 Spherical Vector(2D)를 Cartesian(3D) 으로 변경 * input의 viewing direction : sigma, pi (Cartesian)가 3*2*L_embed로 확장 ( positional encoding ) * output은 RGBa (a는 volume density) => !!중요한점, 1 view는 1 pixel에 대응 8 Depth FC (with 256 dim - Relu)
  • 7. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py MLP 기반의 NeRF (Dataset from Blender)
  • 8. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py https://docs.blender.org/api/blender_python_api_2_71_release/ https://github.com/bmild/nerf/issues/78 https://www.3dgep.com/understanding-the-view-matrix/#:~:text=The%20world%20transformation%20matrix%20is,%2Dspace%20to%20view%2Dspace. MLP 기반의 NeRF (Dataset from Blender) camera_angle_x : Camera lens horizontal field of view <- 카메라의 Focal 값 (모두 동일값) rotation : radians (degree) <- 카메라 제자리 회전 값… (모두 동일값) transform_matrix (Worldspace transformation matrix, matrix_world (blender)) <- Camera 월 좌표계 및 상대 Rotation 정보
  • 9. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) transform_matrix (Look-At 행렬) - Camera 2 World focal = .5 * W / np.tan(.5 * camera_angle_x)
  • 10. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) - rays_o는 rays_original (position) : 각 HxWx3 pixel의 original ray point를 의미 == 모든 점이 camera x,y,z로 똑같음 (각 픽셀마다의 camera position) rays_d는 direction: dirs는 HxW내에서의 상대 좌표. c2w를 곱해줌으로서 camera viewing direction(+depth)을 계산 (각 픽셀마다의 camera view direction)
  • 11. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) http://sgvr.kaist.ac.kr/~sungeui/GCG_S22/Student_Presentations/CS580_PaperPresentation_dgkim.pdf
  • 12. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) - z_vals <- Random Accumulate Point 지정 pts <- z_vals위치의 point들을 생성 pts input에 대한 network output 값의 합산이 rgb가 될 수 있도록 함 (weights 값 합산은 near 1) => raw[:3]는 rgb_map과 mse, raw[4]는 직접 mse가 없고 raw[:3]와 곱해져 간접 업데이트 (본 코드에는 raw[4] - a는 업데이트 없음 fix z_vals)
  • 13. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://m.blog.naver.com/jetarrow82/221252045857 1. MLP 기반의 NeRF - Viewing Direction의 중요성 실제 물체는 non-Lambertian effects 특성을 가짐 Lambertian Surface는 ViewPoint에 상관없이 똑같은 빛의 양을 반사함
  • 14. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.matthewtancik.com/nerf 1. MLP 기반의 NeRF - Positional Encoding
  • 16. NeRF: Synthesizer로서의 가치 Small Size Model Rendering 도메인이 한정되어 있지 않음
  • 17. Texture https://learnopengl.com/Advanced-OpenGL/Cubemaps https://lifeisforu.tistory.com/375 CubeMap - 보통 ray 영향을 크게 받지 않는 원경에 사용 Spherical Harmonics( SH, 구면 조화 ) - irradiance map(난반사 정도)을 저장하는 효율적 방식
  • 18. Plenoxels : ReRF without Neural Networks NeRF key point is differentiable Volume Renderer MLP Volume Rendering을 Voxel Grid와 Interpolation의 조합으로 치환
  • 19. Beyond Synthesize - iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020) Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS) https://yenchenlin.me/inerf/ Trained Nerf를 통해 Generation하고 Pose Estimation을 Iterative하게 Gradient Based Optimization을 하여 Estimated Pose를 취득
  • 20. Beyond Synthesize - iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020) Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS) https://yenchenlin.me/inerf/
  • 21. Beyond Synthesize - Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA) https://yenchenlin.me/nerf-supervision/ Keypoint detection과 유사. Keypoint는 sparse하게 pose 정보를 추출한다면, Dense Descriptor는 Dense하게 Pose 정보를 추출
  • 22. Beyond Synthesize - Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA) https://yenchenlin.me/nerf-supervision/
  • 23. Beyond Synthesize - NERF-SOS: ANY-VIEW SELF-SUPERVISED OBJECT SEGMENTATION ON COMPLEX SCENES (2022 Pre) - 셀링포인트는 다양한 각도에서 영상을 촬영하면 NeRF 샘플에서 Unsupervised로 Segmentation을 만들어줌 => 한계점 - K-Means/Pre-trained Network에 의존적 (BG가 명확하게 분리되고, Object의 Semantic 정보가 명확해야 할 것으로 보임)
  • 24. NeRFs… https://kakaobrain.github.io/NeRF-Factory/ <= Nerf 성능 비교 Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs <= 넓은 공간 모델링 Instant Neural Graphics Primitive <= NeRF, SDF(signed distance functions), Gigapixel image등 (Computer graphics primitives)을 처리할 수 있는 네트워크 https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4 Dex-NeRF : Using a Neural Radiance Field to Grasp Transparent Objects - Transparent, Light-sensitive한 물체를 잘 다룰 수 있지 않을까? https://sites.google.com/view/dex-nerf vision-only robot navigation in a neural radiance world - Vision Based Planning https://mikh3x4.github.io/nerf-navigation/ Relighting the Real World With Neural Rendering - https://www.unite.ai/adobe-neural-rendering-relighting-2021/ NeRF-Studio https://docs.nerf.studio/en/latest/index.html