HIGUCHI, Atsushi, TAKENAKA, Hideaki, and TOYOSHIMA, Koichi:
Center for Environmental Remote Sensing (CEReS), Chiba University, Japan
pdf version release note is here(H08_09_gridded_FD_V02_V20190123_readme_en.pdf)
Observation periods by HIMAWARI 8 and 9 ware as follows:
- HIMAWARI 8 Advanced Himawari Imager [AHI]: 07 July 2015 – 0450 UTC 13 Dec 2022, with several times are switch by HIMAWARI 9.
- HIMAWARI 9 AHI: 0250 UTC 13 Feb 2018 – 0710 UTC 14 Feb 2018, 0450 UTC 13 Dec 2022 –
# Please check observation periods released by JMA.
CEReS released gridded-processed (latitude-longitude coordinated with geo-correction) Version 02 (V20190123) dataset of HIMAWARI 8/9 Advanced Himawari Imager (AHI) Full-Disk (FD) mode. Highlights of updated Version 02 are as follows.
In Version 02 (V20190123) gridded dataset, we apply the technologies for the solar radiation product “AMATERASS” into a gridded dataset. Specifically, Version 02 improves geo-correction accuracy and adds 4-km products such as physical variables and geometric information.
Version 02 slightly improves the accuracy of geo-correction compared with Version 01 (V20151105). Users feel the benefit of using Version 02, particularly EXT utilization (0.005 degree) during the daytime. On the other hand, Version 02’s geo-correction accuracy during the nighttime is the same as Version 01.
We add the 4-km datasets, containing already converted physical variables and solar-sensor zenith and azimuth angles geometric information (see section 3.3.5.).
In the same manner as Version 01, users can get data files to access the anonymous ftp server
(ftp://hmwr829gr.cr.chiba-u.ac.jp/).
After anonymous login, please change directory to gridded/FD/V20190123/.
Alternatively, users can also access directly
ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20190123/.
There are sub-directories named with the following conventions.
YYYYMM
YYYY: YYYY: Year (4 digits), MM: Month（always ２ digits）,
e.g., August 2015 --> 201508.
Each YYYYMM directory has five sub-directories as below.
EXT VIS SIR TIR 4KM
Gridded data specifications stored in each sub-directory will describe in 3.3.3.
Our gridded data has different bands naming rule with JMA official Himawari 8/9 bands names. Tables 1 and 2 show the relation between our gridded data and JMA's Himawari 8/9 bands, and the common specifications for EXT, VIS, SIR, and TIR.
| CEReS gridded | JMA AHI Bands | Pixel x Line | Gridded spatial resolution |
|---|---|---|---|
| EXT 01 | Band03 (0.64 μｍ) | 24000 x 24000 | 0.005 degree |
| VIS 01 | Band01 (0.47 μｍ) | 12000 x 12000 | 0.01 degree |
| VIS 02 | Band02 (0.51 μｍ) | 12000 x 12000 | 0.01 degree |
| VIS 03 | Band04 (0.86 μｍ) | 12000 x 12000 | 0.01 degree |
| SIR 01 | Band05 (1.6 μｍ) | 6000 x 6000 | 0.02 degree |
| SIR 02 | Band06 (2.3 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 01 | Band13 (10.4 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 02 | Band14 (11.2 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 03 | Band15 (12.4 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 04 | Band16 (13.3 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 05 | Band07 (3.9 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 06 | Band08 (6.2 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 07 | Band09 (6.9 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 08 | Band10 (7.3 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 09 | Band11 (8.8 μｍ) | 6000 x 6000 | 0.02 degree |
| TIR 10 | Band12 (9.6 μｍ) | 6000 x 6000 | 0.02 degree |
| Covered area | 85E – 205E (155W), 60N – 60S |
| Data type | 2 byte “unsigned short” binary data without any hearder and footer information, with “big endian” byte order. |
| Data writing orders | West to East (left to right) and North to South (up to down) |
| Stored data | Himawari Standard (HS) data themselves (CCT count like). Note that missing-data pixels are filled by the value of 65535 as the exception. |
File name specifications are as follows;
YYYYMMDDHHMN.XXX.ZZ.fld.geoss.bz2
YYYY: Year (4 digits), e.g., Year of 2015 -> 2015.，MM: Month (always 2 digits), e.g., August -> 08.
DD: Day (always 2 digits), e.g., 12th -> 12. HH: Hour (always 2 digits; Unit is UTC), e.g., 02 UTC -> 02，
MN: Minute (always 2 digits; Unit is UTC), e.g., 10 minutes -> 10.
Note that the timestamp in the file name represents "observation start time (start time to scan)".
XXX: CEReS gridded data wavelength-divided list; i.e., ext, vis, sir, and tir.
Note that those are all capital (EXT, VIS, SIR, and TIR) in the sub-directory names, but are small letters in filenames.
ZZ: CEReS gridded band number as listed in Table 1 (always 2 digits).
fld: FD version.
geoss: no meaning. Simply following the CEReS geostationary gridded data's name rule.
Table 3 shows EXT (0.005 degree, 500-m) specifications and file naming rule.
| CEReS gridded | JMA AHI Bands | Pixel x Line | Gridded spatial resolution |
|---|---|---|---|
| EXT 01 | Band03 (0.64 μｍ) | 24000 x 24000 | 0.005 degree (500-m at nadir) |
| File Name：YYYYMMDDHHMN.ext.01.fld.geoss.bz2 YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), ext: CEReS grided data ‘ext’ represented, 01: CEReS gridded data band number (01 only for EXT), geoss: no meaning，bz2: bzip2 compressed file. |
| Only for the EXT, static information for geo-correction are recorded in the log files YYYYMMDDHHMN.ext.(coff or loff).fld.txt.bz2. Normally do not need to see these files. |
Table 4 shows VIS (0.01 degree, 1-km) specifications and file naming rule.
| CEReS gridded | JMA AHI Bands | Pixel x Line | Gridded spatial resolution |
|---|---|---|---|
| VIS 01 | Band01 (0.47 μｍ) | 12000 x 12000 | 0.01 degree (1-km at nadir) |
| VIS 02 | Band02 (0.51 μｍ) | 12000 x 12000 | 0.01 degree (1-km at nadir) |
| VIS 03 | Band04 (0.86 μｍ) | 12000 x 12000 | 0.01 degree (1-km at nadir) |
| File Name：YYYYMMDDHHMN.vis.ZZ.fld.geoss.bz2 YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), vis: CEReS grided data ‘vis’ represented, ZZ: CEReS gridded data VIS band number (thus 01, 02, and 03), geoss: no meaning，bz2: bzip2 compressed file. |
Table 5 shows SIR (0.02 degree, 2-km) specifications and file naming rule.
| CEReS gridded | JMA AHI Bands | Pixel x Line | Gridded spatial resolution |
|---|---|---|---|
| SIR 01 | Band05 (1.6 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| SIR 02 | Band06 (2.3 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| File Name：YYYYMMDDHHMN.sir.ZZ.fld.geoss.bz2 YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), sir: CEReS grided data ‘sir’ represented, ZZ: CEReS gridded data SIR band number (thus 01 and 02), geoss: no meaning，bz2: bzip2 compressed file. |
Table 6 shows TIR (0.02 degree, 2-km) specifications and file naming rule.
| CEReS gridded | JMA AHI Bands | Pixel x Line | Gridded spatial resolution |
|---|---|---|---|
| TIR 01 | Band13 (10.4 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 02 | Band14 (11.2 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 03 | Band15 (12.4 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 04 | Band16 (13.3 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 05 | Band07 (3.9 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 06 | Band08 (6.2 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 07 | Band09 (6.9 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 08 | Band10 (7.3 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 09 | Band11 (8.8 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| TIR 10 | Band12 (9.6 μｍ) | 6000 x 6000 | 0.02 degree (2-km at nadir) |
| File Name：YYYYMMDDHHMN.tir.ZZ.fld.geoss.bz2 YYYY: Year (4 digits), MM: Month (always 2 digits), DD: Day (always 2 digits), HH: Hour (always 2 digits; Unit is UTC), MN: Minute (always 2 digits; Unit is UTC), tir: CEReS grided data ‘tir’ represented, ZZ: CEReS gridded data TIR band number (thus from 01 to 10), geoss: no meaning，bz2: bzip2 compressed file. |
4KM data is Version 02’s new product. 4KM data stored in daily sub-directories such as; ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/V20190123/ YYYYMM/4KM/YYYYMMDD/． 4KM data specifications and file naming rules are represented in Table7.
| Pixel x Line, resolution | 3000 x 3000, 0.04 degree (4-km at nadir) |
| Covered area | 85E – 205E (155W), 60N – 60S (same as the other dataset) |
| Data type | 4 byte “float” binary data with “big endian” byte order. |
| Data writing orders | West to East (left to right) and North to South (up to down) |
| Stored data | Already converted to physical variables |
| File name example | Physical variable |
|---|---|
| YYYYMMDDHHMN.xxx.ZZ.rad.fld.4km.bin.bz2 (xxx: ext, vis, sir, tir; ZZ: CEReS gridded data band number) | ext, vis, sir, tir irradiance (unit: W m-2 sr-1 μm-1) |
| YYYYMMDDHHMN.xxx.ZZ.rfc.fld.4km.bin.bz2 (xxx: ext, vis, sir; ZZ: CEReS gridded data band number) | ext, vis, sir spectral reflectance (dimensionless) |
| YYYYMMDDHHMN.xxx.ZZ.rfy.fld.4km.bin.bz2 (xxx: ext, vis, sir; ZZ: CEReS gridded data band number) | ext, vis, sir spectral reflectance (%) |
| YYYYMMDDHHMN.tir.ZZ.tbb.fld.4km.bin.bz2 (ZZ: CEReS gridded data band number) | tir (only) brightness temperature (Tbb (K)) |
| File name example | Physical variable |
|---|---|
| YYYYMMDDHHMN.sun.azm.fld.4km.bin.bz2 | Solar azimuth angle (Degree, South direction is zero, clockwise rotation) |
| YYYYMMDDHHMN.sun.zth.fld.4km.bin.bz2 | Solar zenith angle (Degree) |
| YYYYMMDDHHMN.sat.azm.fld.4km.bin.bz2 | Sensor azimuth angle (Degree, South direction is zero, clockwise rotation) |
| YYYYMMDDHHMN.sat.zth.fld.4km.bin.bz2 | Sensor zenith angle (Degree) |
| YYYYMMDDHHMN.grd.time.mjd.hms.fld.4km.bin.bz2 | Scanning time（Normalized 0 to 1, i.e., 12:00 UTC is 0.5) |
| YYYYMMDDHHMN.lat.fld.4km.bin.bz2 | Latitude (Degree) |
| YYYYMMDDHHMN.lng.fld.4km.bin.bz2 | Longitude (Degree) |
| YYYYMMDDHHMN.cap.flg.fld.bin.bz2*1 | Cloud flag (Daytime and over ocean only. More than 1 represents cloud) |
Except for the 4KM product, the conversion process will be needed to get physical variables, using he calibration tables of which produced by JMA.
We provide sample programs (Fortran90, C, and python codes) and calibration tables.
The programs are compressed in a tarball,
count2tbb_v103.tgz,
and can download at
ftp://hmwr829gr.cr.chiba-u.ac.jp/gridded/FD/support/.
Version 103 sample programs are that add LUT for HIMAWARI 09 from version 102.
If you have version 102 or older sample programs, please switch version 103.
Also, PC-based OS such as Windows, Intel MacOS, Linux (i386, x86_64), standard binary data reading order is little-endian.
Thus, before processing our sample programs, the user needs to byte swapping process.
In the case of Linux, dd command can do such byte swap process as follows;
$dd if=(input binary) of=(output binary) conv=swab
Such a byte swap does not need native 64bit environments such as Power PC, Intel Itanium (IA-64), SPARC, HP-UX, etc.
[Note] Even though 64-bit OS on intel/AMD CPU, such as Linux x86_64, MacOS, Windows 64-bit,
our sample programs for EXT (24000 pixels x 24000 lines) band data process the data as followings;
a) The input data file splitting into two files as the Northern area and Southern area.
b) Then C/F90 program convert CCT to reflectance for each splitting file.
c) After conversion, two files are merged into one (please see the detail processes in shell scripts in the tarball) .
This process is mainly due to the limitation of malloc (memory allocation).
Moreover, under the 32-bit OS, due to 32-bit OS (limitation of 32-bit),
it is impossible to generate any file larger than 2-GB.
If users try to analyze EXT data on 32-bit OS, the users need to treat EXT physical converted data as split files
(e.g., Northern and Southern areas).
If you want to read more detail on reading gridded data, the below URL is maybe useful.
http://quicklooks.cr.chiba-u.ac.jp/~himawari_movie/rd_gridded.html
In addition, our gridded data user made Python sample codes. These programs are out of our support, but maybe useful.
https://github.com/zxdawn/Himawari-8-gridded
CEReS look-up tables (LUTs) are based on the original calibration coefficients.
Due to sensor degradation of the AHI,
JMA's Meteorological Satellite Center (MSC) updates AHI calibration coefficients annually.
Professor MIURA, Tomoaki of the University of Hawai'i created new LUTs to reflect these new calibration coefficients.
If the user analyzes the long-term trend, it is better to apply these LUTs.
Through Professor Miura's courtesy, we can release updated LUTs for EXT, VIS, and SIR bands
Updated_LUTs_20200308_zip.zip
Useful reference：Update of Calibration Information Used to Correct Himawari-8 AHI Sensitivity Trend
（
https://www.data.jma.go.jp/mscweb/en/oper/eventlog/Update_of_Calibration_Information_2020.pdf;
accessed on 03 September 2020).
Global Space-based Inter-Calibration System (GSICS) is one of the space components of the World Meteorological Organization (WMO).
Its mission is to provide users with high-quality and inter-calibrated measurements from operational satellites.
JMA MSC operationally has been providing information about inter-calibration results at Himawari Calibration Portal:
https://www.data.jma.go.jp/mscweb/en/oper/calibration/calibration_portal.html (accessed on 01 September 2020)
Please cite two papers listed as below．
Geo-correction algorithm：Takenaka, H., T. Sakashita, A. Higuchi, T. Nakajima (2020):
Development of geolocation correction for geostationary satellite observation by phase only correlation method using visible channel,
Remote Sensing, 12 (15), 2472
doi:10.3390/rs12152472.
Geo-correction accuracy validation：Yamamoto, Y., K. Ichii, A. Higuchi, H. Takenaka (2020):
Geolocation accuracy assessment of Himawari-8/AHI imagery for application to terrestrial monitoring,
Remote Sensing, 12 (9), 1372
doi:10.3390/rs12091372.
Besides, please describe below sentence in Acknowledgements;
“Himawari 8/9 gridded data are distributed by the Center for Environmental Remote Sensing (CEReS), Chiba University, Japan.”
CEReS Himawari 8/9 gridded data is limited for non-commercial use.
This data policy is the same as JMA's one for academic-data-releasement.
We also prohibit the distribution of our gridded data to third parties, except among the research communities.
Due to the data policy of JMA,
quasi-real-time data (in the definition of JMA, real-time data is within 24 hours data since now) cannot open without any restriction.
Thus, in the CEReS ftp server, we control the access for quasi-real-time data by IP address.
If the user wants to access quasi-real-time data for non-commercial use only,
please send an e-mail with the accessed global IP address and its objective to below e-mail address.
request4himawaridata _AT_ ceres.cr.chiba-u.ac.jp.
We have no plan to change the configuration on the CEReS ftp server by user registration.
We will do via IP based restriction only.
This activity is partly supported by the Virtual Laboratory (VL) project funded by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan, JST/ CREST TEEDA, and the President leaderships of Chiba University. We thank Prof. HAMADA, Atsushi of University of Toyama, who pointed out our sample programs bug and contributed to sample programs improvements. Thanks are extended to Prof. MIURA, Tomoaki of the University of Hawai'i who created new LUTs to reflect these new calibration coefficients. We also thank Mr. LIU, Zhiyan, Ph.D student of Ichii Lab., CEReS, Chiba University, who improved python code in a sample programs package.
If you want to see the releasenote for Version 01 (V20151105), visit here (index_en_V20151105.html).