Cited By
View all- Guo DChang KTang YLing MLi M(2023)CInvISP: Conditional Invertible Image Signal Processing PipelineNeural Information Processing10.1007/978-981-99-8073-4_42(548-562)Online publication date: 20-Nov-2023
Reversed Image Signal Processing and RAW Reconstruction. AIM 2022 Challenge Report
Authors: 
Marcos V. Conde, Radu Timofte, Yibin Huang, Jingyang Peng, Chang Chen, Cheng Li, Eduardo Pérez-Pellitero, Fenglong Song, Furui Bai, Shuai Liu, Chaoyu Feng, Xiaotao Wang, Lei Lei, Yu Zhu, Chenghua Li, Yingying Jiang, Yong A, Peisong Wang, Cong Leng, Jian Cheng, Xiaoyu Liu, Zhicun Yin, Zhilu Zhang, Junyi Li, Ming Liu, Wangmeng Zuo, Jun Jiang, Jinha Kim, Yue Zhang, Beiji Zou, Zhikai Zong, Xiaoxiao Liu, Juan Marín Vega, Michael Sloth, Peter Schneider-Kamp, Richard Röttger, Furkan Kınlı, Barış Özcan, Furkan Kıraç, Li Leyi, S. M. Nadim Uddin, Dipon Kumar Ghosh, Yong Ju JungAuthors Info & Claims
Marcos V. Conde, Radu Timofte, Yibin Huang, Jingyang Peng, Chang Chen, Cheng Li, Eduardo Pérez-Pellitero, Fenglong Song, Furui Bai, Shuai Liu, Chaoyu Feng, Xiaotao Wang, Lei Lei, Yu Zhu, Chenghua Li, Yingying Jiang, Yong A, Peisong Wang, Cong Leng, Jian Cheng, Xiaoyu Liu, Zhicun Yin, Zhilu Zhang, Junyi Li, Ming Liu, Wangmeng Zuo, Jun Jiang, Jinha Kim, Yue Zhang, Beiji Zou, Zhikai Zong, Xiaoxiao Liu, Juan Marín Vega, Michael Sloth, Peter Schneider-Kamp, Richard Röttger, Furkan Kınlı, Barış Özcan, Furkan Kıraç, Li Leyi, S. M. Nadim Uddin, Dipon Kumar Ghosh, Yong Ju JungAuthors Info & ClaimsPages 3 - 26
Abstract
Cameras capture sensor RAW images and transform them into pleasant RGB images, suitable for the human eyes, using their integrated Image Signal Processor (ISP). Numerous low-level vision tasks operate in the RAW domain (e.g. image denoising, white balance) due to its linear relationship with the scene irradiance, wide-range of information at 12bits, and sensor designs. Despite this, RAW image datasets are scarce and more expensive to collect than the already large and public RGB datasets. This paper introduces the AIM 2022 Challenge on Reversed Image Signal Processing and RAW Reconstruction. We aim to recover raw sensor images from the corresponding RGBs without metadata and, by doing this, “reverse” the ISP transformation. The proposed methods and benchmark establish the state-of-the-art for this low-level vision inverse problem, and generating realistic raw sensor readings can potentially benefit other tasks such as denoising and super-resolution.
References
[1]
Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1692–1700 (2018)
[2]
Abdelhamed, A., Timofte, R., Brown, M.S.: NTIRE 2019 challenge on real image denoising: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
[3]
Afifi M, Abdelhamed A, Abuolaim A, Punnappurath A, and Brown MS CIE XYZ Net: unprocessing images for low-level computer vision tasks IEEE Trans. Pattern Anal. Mach. Intell. 2021 44 9 4688-4700
[4]
Afifi, M., Abuolaim, A.: Semi-supervised raw-to-raw mapping. arXiv preprint arXiv:2106.13883 (2021)
[5]
Arad, B., Timofte, R., Ben-Shahar, O., Lin, Y.T., Finlayson, G.D.: NTIRE 2020 challenge on spectral reconstruction from an RGB image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 446–447 (2020)
[6]
Bhat, G., Danelljan, M., Timofte, R.: NTIRE 2021 challenge on burst super-resolution: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 613–626 (2021)
[7]
Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237 (2018)
[8]
Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11036–11045 (2019)
[9]
Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. arXiv preprint arXiv:2204.04676 (2022)
[10]
Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: HINet: half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 182–192 (2021)
[11]
Chen, X., Zhang, Z., Ren, J.S., Tian, L., Qiao, Y., Dong, C.: A new journey from SDRTV to HDRTV. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4500–4509 (2021)
[12]
Chu, X., Chen, L., Chen, C., Lu, X.: Improving image restoration by revisiting global information aggregation. arXiv preprint arXiv:2112.04491 (2021)
[13]
Conde, M.V., McDonagh, S., Maggioni, M., Leonardis, A., Pérez-Pellitero, E.: Model-based image signal processors via learnable dictionaries. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 481–489 (2022)
[14]
Conde, M.V., Timofte, R., et al.: Reversed image signal processing and RAW reconstruction. AIM 2022 challenge report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[15]
Delbracio, M., Kelly, D., Brown, M.S., Milanfar, P.: Mobile computational photography: a tour. arXiv preprint arXiv:2102.09000 (2021)
[16]
Dong, X., Zhu, Y., Li, C., Wang, P., Cheng, J.: RISPNet: a network for reversed image signal processing. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[17]
Gharbi M, Chaurasia G, Paris S, and Durand F Deep joint demosaicking and denoising ACM Trans. Graph. (ToG) 2016 35 6 1-12
[18]
He K, Zhang X, Ren S, and Sun J Leibe B, Matas J, Sebe N, and Welling M Identity mappings in deep residual networks Computer Vision – ECCV 2016 2016 Cham Springer 630-645
[19]
Heide F et al. FlexISP: a flexible camera image processing framework ACM Trans. Graph. (ToG) 2014 33 6 1-13
[20]
Ignatov, A., Timofte, R., Denna, M., Younes, A., et al.: Efficient and accurate quantized image super-resolution on mobile NPUs, mobile AI & AIM 2022 challenge: report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[21]
Ignatov, A., Timofte, R., Kuo, H.K., Lee, M., Xu, Y.S., et al.: Real-time video super-resolution on mobile NPUs with deep learning, mobile AI & AIM 2022 challenge: report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[22]
Ignatov A et al. Bartoli A, Fusiello A, et al. AIM 2020 challenge on learned image signal processing pipeline Computer Vision – ECCV 2020 Workshops 2020 Cham Springer 152-170
[23]
Ignatov, A., Timofte, R., et al.: Efficient bokeh effect rendering on mobile GPUs with deep learning, mobile AI & AIM 2022 challenge: report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[24]
Ignatov, A., Timofte, R., et al.: Efficient single-image depth estimation on mobile devices, mobile AI & AIM 2022 challenge: report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[25]
Ignatov, A., Timofte, R., et al.: Learned smartphone ISP on mobile GPUs with deep learning, mobile AI & AIM 2022 challenge: report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[26]
Ignatov, A., Van Gool, L., Timofte, R.: Replacing mobile camera ISP with a single deep learning model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2020)
[27]
Jiang, J., Kim, J., Gu, J.: Overexposure mask fusion: generalizable reverse ISP multi-step refinement. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[28]
Karaimer HC and Brown MS Leibe B, Matas J, Sebe N, and Welling M A software platform for manipulating the camera imaging pipeline Computer Vision – ECCV 2016 2016 Cham Springer 429-444
[29]
Kınlı, F., Özcan, B., Kıraç, F.: Patch-wise contrastive style learning for instagram filter removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 578–588 (2022)
[30]
Kinli, F., Ozcan, B., Kirac, F.: Instagram filter removal on fashionable images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 736–745 (2021)
[31]
Kobyzev I, Prince SJ, and Brubaker MA Normalizing flows: an introduction and review of current methods IEEE Trans. Pattern Anal. Mach. Intell. 2020 43 11 3964-3979
[32]
Kınlı, F.O., Menteş, S., Özcan, B., Kirac, F., Timofte, R., et al.: AIM 2022 challenge on instagram filter removal: methods and results. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[33]
Kınlı, F.O., Özcan, B., Kirac, F.: Reversing image signal processors by reverse style transferring. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[34]
Nam, S., Punnappurath, A., Brubaker, M.A., Brown, M.S.: Learning SRGB-to-raw-RGB de-rendering with content-aware metadata. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17704–17713 (2022)
[35]
Nguyen, R.M.H., Brown, M.S.: Raw image reconstruction using a self-contained SRGB-jpeg image with only 64 kb overhead. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
[36]
Punnappurath A and Brown MS Learning raw image reconstruction-aware deep image compressors IEEE Trans. Pattern Anal. Mach. Intell. 2019 42 4 1013-1019
[37]
Punnappurath, A., Brown, M.S.: Spatially aware metadata for raw reconstruction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 218–226 (2021)
[38]
Qian, G., et al.: Rethinking the pipeline of demosaicing, denoising and super-resolution. arXiv preprint arXiv:1905.02538 (2019)
[39]
Ronneberger O, Fischer P, and Brox T Navab N, Hornegger J, Wells WM, and Frangi AF U-Net: convolutional networks for biomedical image segmentation Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 2015 Cham Springer 234-241
[40]
Schwartz E, Giryes R, and Bronstein AM DeepISP: toward learning an end-to-end image processing pipeline IEEE Trans. Image Process. 2018 28 2 912-923
[41]
Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934–8943 (2018)
[42]
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
[43]
Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)
[44]
Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
[45]
Xing, Y., Qian, Z., Chen, Q.: Invertible image signal processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6287–6296 (2021)
[46]
Yang, R., Timofte, R., et al.: AIM 2022 challenge on super-resolution of compressed image and video: dataset, methods and results. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
[47]
Zamir, S.W., et al.: CycleISP: real image restoration via improved data synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2696–2705 (2020)
[48]
Zeng H, Cai J, Li L, Cao Z, and Zhang L Learning image-adaptive 3D lookup tables for high performance photo enhancement in real-time IEEE Trans. Pattern Anal. Mach. Intell. 2020 44 4 2058-2073
[49]
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
[50]
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018)
[51]
Zhang, Z., Wang, H., Liu, M., Wang, R., Zhang, J., Zuo, W.: Learning raw-to-SRGB mappings with inaccurately aligned supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4348–4358 (2021)
[52]
Zhu Yu et al. Bartoli A, Fusiello A, et al. EEDNet: enhanced encoder-decoder network for AutoISP Computer Vision – ECCV 2020 Workshops 2020 Cham Springer 171-184
[53]
Zou, B., Zhang, Y.: Learned reverse ISP with soft supervision. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
Recommendations
AIM 2020 Challenge on Image Extreme Inpainting
Computer Vision – ECCV 2020 WorkshopsAbstractThis paper reviews the AIM 2020 challenge on extreme image inpainting. This report focuses on proposed solutions and results for two different tracks on extreme image inpainting: classical image inpainting and semantically guided image inpainting. ...
Restoration and enhancement on low exposure raw images by joint demosaicing and denoising
AbstractRestoring high quality images from raw data in low light is challenging due to various noises caused by limited photon count and complicated Image Signal Process (ISP). Although several restoration and enhancement approaches are ...
Highlights- We propose a Short2Long raw restoration strategy to process low quality raw images and restore pseudo raw ones for the subsequent subnet. It extracts ...
ELMformer: Efficient Raw Image Restoration with a Locally Multiplicative Transformer
MM '22: Proceedings of the 30th ACM International Conference on MultimediaIn order to get raw images of high quality for downstream Image Signal Process (ISP), in this paper we present an Efficient Locally Multiplicative Transformer called ELMformer for raw image restoration. ELMformer contains two core designs especially for ...
Comments
Information & Contributors
Information
Published In
Oct 2022
796 pages
ISBN:978-3-031-25065-1
DOI:10.1007/978-3-031-25066-8
- Editors:
- Leonid Karlinsky,
- Tomer Michaeli,
- Ko Nishino
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 18 February 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Aug 2024
Other Metrics
Citations
Cited By
View all- Guo DChang KTang YLing MLi M(2023)CInvISP: Conditional Invertible Image Signal Processing PipelineNeural Information Processing10.1007/978-981-99-8073-4_42(548-562)Online publication date: 20-Nov-2023
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in