CREST User Manual v2.1.3 - MATLAB
CREST User Manual v2.1.3 - MATLAB
edu)
CREST
Coupled Routing and Excess STorage
User Manual
Version 2.1 (released in Jul. 2015)
(f) (g)
Cover: CREST—Coupled Routing and Excess Storage User Manual Version 2.1
Table of Contents
CREST ................................................................................................................... 1
2 INSTALLATION ................................................................................................ 4
6 CONTACT US .................................................................................................. 26
I
1 Model Overview
2
Figure 1-1 Core Components of the CREST model
(a) Vertical profile of a cell including rainfall-runoff generation, evapotranspiration, sub-grid cell routing
and feedbacks from routing; (b) variable infiltration curve of a cell; (c) plane view of cells and flow
directions; and (d) vertical profile along several cells including sub-grid cell routing, downstream routing,
The major upgrade is on the routing scheme. The cell-to-cell routing scheme used
in previous versions of CREST is a quasi-distributed linear reservoir routing (QDLRR)
method which we found problematic to apply to a distributed hydrological model in
practice. In CREST v2.1, a fully distributed LRR method (FDLRR) is proposed and
used to replace the QDLRR module of CREST. The conception of the QDLRR and the
FDLRR is shown in Figure 1-2. Suppose that in one time step, water moves from A to
3
D, B to E and C to F and take C as the observation cell. In previous versions as shown
in Figure 1-2(a), only the water from C to F contributes to the final runoff (discharge)
of C while water from A to D and B to F is denied of contribution as if the water jumps
over cell C. On the contrary, in CREST v2.1, all these three terms contributes to the
runoff of cell C because they either sets off from or passed via cell C.
(a) (b)
Figure 1-2 Routing Conception of v2.0 and v2.1. (a) Linear reservoir routing (LRR) method used in
V2.0 and (b) Fully distributed linear reservoir (DLRR) used in v2.1.
Minor upgrade includes: 1) the full vectorization of the computation which boosts
the efficiency by nearly one order (no routines loop cell by cell in the new version); 2)
acceptance of more advanced geographic data formats 3) automatic decompression,
reprojection, resampling and clipping of the forcing data to accommodate data in
different formats, coordinate system and resolution; 4) adding a flood event mode and
5) switching on and off a) the feedback mechanism, b) the existence of interflow in
channels.
2 Installation
CREST v2.1 is written in Matlab that is OS (operating systems) independent. It is
compatible with R2013a and later versions of matlab. However, it integrates the GDAL
libraries to implements I/O (input/output) functionality, which is OS dependent. Please
install CREST v2.1 using the following steps:
Compile and install the GDAL
For Linux OS, the components and links are provided blow. Please install them in
order. Specifically, GDAL needs to be no older than 1.11.0.
1) Geos
4
via http://trac.osgeo.org/geos/
2) proj.4
via https://trac.osgeo.org/proj/
3) GDAL
via http://trac.osgeo.org/gdal/wiki/DownloadSource
and compile following the same commands below
http://trac.osgeo.org/gdal/wiki/BuildingOnUnix
For Windows OS, compiled x64 binaries are provided with the model, users do not
have to compile GDAL themselves unless they need to support some additional image
formats.
Compile the matlab GDAL interface
Similar to the GDAL binaries, windows-x64 users can skip this subsection because
the compiled *.mexw64 are provided with CREST v2.1. For Linux users, please follow
the following orders to compile all *.cpp/*.c files in the MEX folder using the “mex”
command in matlab environment as the example below. The order is arbitrary.
5
that 1) the files in the MEX folder end with “.amd64” and 2) the DLL folder is not
required. The reason is that the GDAL library and its dependency must be compiled by
the user before installing CREST. It’s not able for us to distribute a compatible binary
for any version of Linux. Besides an image format of interest, users must install the
minimum libraries found below. We provided the links of the source codes but users are
free to try other source or newer versions of the same functionality. If a user is not
familiar to the installation of these libraries, please contact the server administrators or
simply visit gdal.org. CREST developers are not responsible for the instruction on
GDAL installation.
6
options and physical locations of all other data files needed by CREST and is usually
put in the root of the project folder. Other data files, as shown in Figure 3-1, are
distributed in several folders of the project according to their categories. These folders
are specified in the control file including “basic”, “rains”, “PETs” “Param”,
“obs”,”calib” and “results”.
The control file and these folders will be described in the following subsections.
Setting up a basin is to create and fill these folders and the control file.
After a basin is setup, users can run CREST using the following matlab command
>> CREST(globalCtl, opt, nCore);
where
opt=’mean’|’real’ refers to use the presumed mean height difference or the real one at
the outlet. The mean height difference is usually used because the clipped geographic
data contained in the basic folder usually lacks the information of the next downstream
cell of the basin outlet and it can also be a “sink” at the outlet cell.
gdalPath is the path where gdal_csharp.dll is stored. By default, it should be in the
subfolder ”.\ gdal_1110_dll” in the decompressed CREST folder.
nCore is the number of allowed parallel workers. It is only effective in the calibration
mode and can be ignored in the simulation mode.
The control file described in this subsection contains model settings and data
directories.
Note:
The statements in the control file should be listed in order in a keywords-value
manner as following:
Keyword = Value
The statement appearing on the same line should be tab-separated.
Comment should begin with a pound sign, #.
Keyword is not case sensitive.
Note 1. Keyword and value format
Keywords with * is new in CREST v2.1.
7
Model Area (obsolete)
CREST v2.1 accepts common geographic data formats that contain information of
coordinate systems and projections. Therefore the “Model Area” is removed.
Figure 3-2 A sample of temporal settings section in the control file (regular mode).
8
Figure 3-3 Sample temporal settings in the control file (flood event mode)
9
in the flood event mode.
EndDate_i integer The End date time to load state variables to run CREST in the
flood event mode, where 0<i<=NLoad. Its format is defined
by “TimeFormat”.
Users set the running style and model options in this section in the control file.
Figure 3-4 A sample of the Style and Options Section in the control file
Table 3-2 Style And Options Section in the control file
Keyword Value Default Description
RunStyle simu|ca N/A “simu” stands for the simulation mode
lib_SC while “calib_SCEUA” stands for the
EUA automatic calibration mode using the SCE-
UA method.
Feekback* Yes|No Yes “Yes” means that the routing process feeds
back the rainfall-runoff process
hasRiverInterflow Yes|No No “No” means that all interflow turns to
* surface flow in channels
useLAI* Yes|No No “Yes” is reserved for later versions.
In this section in the control file, directory of all input and output data files is specified.
10
Figure 3-5 A sample of Model Directory in the control file
As shown in Figure 3-5, CREST separates the input and output data into 9
categories: “Basic”, “Param”, “States”, “ICS”, “Rains”, “PETs”, “Result”, “Calib” and
“OBS”. Each category has a standalone folder denoted by “*Path”, for example, the
“BasicPath”. The name of the folders is user-specified while the keywords are fixed.
The statements in this section is written in the keyword/value format defined in Note 1.
The basic folder contains the raster files of the same format that stores the
geographic information of the basin. The full path of these files is “known” by CREST
using the information specified in this section as described in Table 3-3. For e.g., the
full path of the DEM file is specified as BasicPath +”dem”+ BasicFormat.
Table 3-3 Basic section in the control file.
Keyword Value Description
BasicFormat The extension of an image file. file name extension
Default is Geotiff ‘.tif’
BasicPath A string of a valid directory Physical path of the basic
that ends with a ‘\’ folder
11
3.2.3.2 Param, State and ICS Sections
The “Param” and “ICS” folders contain text files of model parameters and the
initial condition respectively. The key/value format of these files appears the same as
in the control file defined in Note 1. The file format in “Param” and “ICS” folders is
fixed to ‘.txt’ while the format in the States folder is fixed to “.mat”.
The forcing Sections are the trickiest. It includes the rainfall, PET and the LAI
subsections. The three forcing sections have the same structure with the rainfall section
being shown in Figure 3-6 as an example. CREST incorporates two mechanisms to
efficiently read the forcing file: reading from external formats and from the internal
format. The external format can be in arbitrary image format of a standard coordinate
system. The internal format is in matlab’s “.mat” matrix. When the model runs in the
simulation mode, it first check the existence of internal files, if they exist, the model
uses read the internal files; otherwise, it tries to read the external file on the missing
date time and then save it in the internal format for next time use. Reading the external
forcing file can be significantly slower than the internal one because it may involve the
decompression, resampling, reprojection and clipping. As a result, it is recommended
to run the model in the simulation mode for the first time and then to play the model at
any styles the user desire.
Table 3-4 lists the variables in the rainfall section as an example. In this example,
daily stageIV data is used, the file name without directory is “yyyymmdd12.24h.Z”. It
should be noted that different forcing data source has its own label (file name)
convention. Since stageIV data follows the “End” time convention, in the previous
example, yyyymmdd12 means the rainfall time step starts from yyyy year mm month
dd-1 day 12 am. and ends at yyyy year mm month dd day 12am.. Therefore, it centers
at yyyy year mm month dd day 0 am. Please refer 3.2.1 to the definition of different
time conventions.
Considering the external forcing time step/convention can be different from the
model’s, users are required to provide the NEAREST date time of the forcing to the
model’s actual start time as ForcingStart (RainStart, PETStart, LAIStart). Please
compare the forcing and temporal settings sections in the control file of the example
basin for clarity.
12
Figure 3-6 A sample of the Forcing Section in the control file.
Table 3-4 Basic section in the control file
Keyword Example Value Description
RainFormat .24h.Z The extension of the EXTERNAL forcing
file. Note that the extension means all
content after the date time part
The file can be a compressed file. CREST
will identify this by its extension
(.zip, .rar, .z, .7z) and decompress it using
the user specified decompression
software if necessary. WinRAR is
currently supported in windows
platforms.
RainDateFormat yyyymmddHH The time format used by RainStart, and
RainDateInterval.
The calibration Section specifies the directory of the calibration folder and the
calibration mode which is either parallel or sequential as in
The observation section specifies the directory of the “OBS” folder, the observation
date format and the no-data value of the runoff records. Please refer to Chapter 3.2.1
for the requirement of observation time lines.
The shape file contains the information of all sites and thus the outpix information
is not needed anymore.
CREST v2.1 retains the hydrograph from multiple sites whose location is provided
in the shape file specified in Section 3.2.5. The sub-basin masks for internal sites are
automatically generated by CREST and stored in the result folder.
Instead of outputting selected pixel information, CREST v2.1 outputs the selected
variables of the entire river network whose location is read from the stream file in the
basic folder.
CREST v2.1 uses a point feature in the ESRI shape file format to represent the
location of the outlet rather than texts of the latitude and longitude. Therefore, only the
file name of the shape file is specified in the control file. In addition, the shape file
MUST contain a projection.
14
first field of the site (a point feature) in the shape file.
OutletShpFile 02083500.shp File name of the shape file that contains the outlet
location as a point feature. The first field of the point
feature must be the OutletName. The default directory
of the shape file is the “obs” folder.
Grid Outputs is used to select the 2-D gridded variables to output at EVERY time
step. The selected (of Yes value) variables will be output to the result folder and the file
name will be suffixed by the date time in model’s temporal format. The default format
of the 2-D gridded files is GeoTiff (.tif). Grid outputs is time consuming and not
recommended during the calibration.
15
GOVar_RI The depth of interflow flow in mm
CREST v2.1 is able to run at a flood event (FE) mode, in which the initial state of
each event must be reloaded. These states were saved during a previous simulation. The
previous run can be at a different time step while save the states as at an offset time (in
its file name) to adjust to the time line in the FE mode. For instance, in Figure 3-11, the
first save date time is at 0:00 am., Oct. 8th, 2002. To adjust to a FE mode at hourly scale
that centered at XX:30, a minimum 30 min offset is added to make the first saving date
time 00:30 am., Oct. 8th 2002.
This section provides the decompression software strings and the information of
operational system (OS). The OS is either windows or linux. The string specified by
DecompBeforeSrc stores the command needed before the compressed file name while
16
the string specified by DecompBeforeDst stores the command needed before the
uncompressed file. The complete command of decompression of the example in Figure
3-12 would be,
Figure 3-12 A sample of the Unzip Software and OS Section in the control file
CREST v2.1 can read more than 200 the raster file formats supported by GDAL.
Users only need to prepare the basic files and other text files. The decompression,
reprojection, resample and clipping of the forcing file according to the configuration of
the basic file is automatically conduct by CREST. Therefore, Users can save their space
and time in preparing forcing files for each basin.
This folder contains the raster files that represent the geographic information of the
basin and a text file that defines the average height difference: a DEM (Digital elevation
model) file, an FDR (Flow Direction) file an FAC (Flow Accumulation) file a stream
file. All files except the slope file is in a geographic data format with a projection. From
CREST v2.1, the model accepts any commonly used raster formats supported by GDAL.
Raster files in this folder only contains grid-cell values within the basin area while the
grids out of the basin is marked by Null value which is explicitly recorded in the each
raster file, as done by the SetNull function of the ArcGIS Map Algebra tool. In addition,
the regions in all four raster files MUST have exactly the same size and basin area.
Users can use a GIS tool to generate the files in this folder. We also attached a python
script that calls ArcGIS routines to prepare all raster files for CREST.
Table 3-6 Contents in the basic folder
Name File name Format Optional Description
by default
DEM dem.* any No The digital elevation model
FDR fdr.* any No Flow direction (code defined as in
ArcGIS, 1-128)
FAC fac.* any No Flow accumulation (value defined as
in HydroSHEDS, i.e., the high
ends(minimum value) is 1) in pixel
17
stream stream.* any No Value in the river is 1, otherwise is 0
slope Slope.def .def (text) No Contains the GM value that defines
the pre-defined mean height
difference. It is used for calculate the
slope at the outlet or other places
where the slope value is invalid from
the DEM map.
A second value is the height of the
adjacent downstream cell of the
outlet. The second value is optional.
Mask, GridArea and AreaFact files are obsolete since CREST v2.1.
This folder contains a parameter.txt file that records all 15 model parameters that
are categorized as physical and conceptual types (see Table 3-7). The model parameters
in CREST v2.1.0 remains the same as in CREST v2.0. The parameter.txt file also
18
Table 3-7 Classification in CREST v2.1
Type Parameter Description Min Default Max Unit
RainFact The multiplier on the precipitation field 0.5 1.0 1.2
Ksat The Soil saturate hydraulic conductivity 0 500 3000 mm/day
WM The Mean Water Capacity 80 120 200 mm
Physical
The exponent of the variable infiltration 0.05 0.25 1.5
B
curve
Parameters
IM The impervious area ratio 0 0.05 0.2
KE The factor to convert the PET to local actual 0.1 0.95 1.5
coeM The overland runoff velocity coefficient 1 90 150
expM The overland flow speed exponent 0.1 0.5 2
The multiplier used to convert overland flow 1 2 3
coeR
Conceptual The speed to channel flow speed
The multiplier used to convert overland flow 0.001 0.3 1
coeS
Parameters The speed to interflow flow speed
KS The overland reservoir Discharge Parameter 0 0.6 1
KI The interflow Reservoir Discharge Parameter 0 0.25 1
19
follows the keyword/value format defined in Note 2. Furthermore, each variable is not
only defined by its value, but also defined by its type, i.e., the “varNameType”
keyword. The type can only be uniform or distributed. If the type of a parameter is
distributed, it’s the value should be a file name in the “Param” folder of a raster file
that exactly matches size and basin area defined by the files in the basic folder. The
limits and the default value of uniform parameters are also listed in Table 3-7.
This folder contains the saved model state files named by the date time specified in
Section 3.2.7. The state files are in matlab “.mat” format. The State folder is an output
folder for regular running mode whereas an input folder for the FE mode. In the FE
mode, CREST loads the state variables saved by a previous simulation in the regular
mode. However, the date time to load is specified in the “temporal Settings” section in
the control file rather than the “State to Save” section.
This folder has an “initilalCondition.txt” file that contains uniform variables listed in
Table 3-8 as the model initial condition. The initial condition file is written in the same
format as the parameter file. To let CREST be well distributed, a sufficient warm up
period is necessary. This folder is ineffective in the FE mode.
Table 3-8 Classification in CREST v2.1
Keywords Unit Description
W0 % Initial Value of Soil Moisture
SS0 mm Initial value of Overland Reservoir
SI0 mm Initial value of Interflow reservoir
This folder contains the shape file (location) of the outlet and the observed runoff
data excel file. For the model calibration or validation. The file name of the observed
20
runoff is “OutletName_Obs.csv” (“.csv” is the comma delimited file) where the
“OutletName” is specified in the project file and the file name of the shape file is
specified in the control file. The OutletName_Obs.csv has two columns WITH head.
The first and second columns record the date time and runoff respectively. The date
time must follow the model time step. The location shape file specified in Section 3.2.5
contain the position of outlets as point features. Each point feature contains one site.
Only the site of the outlet name specified by the OutletName keyword is acknowledged
as the outlet used to evaluate the model performance and calibrate.
The time zone of CREST is arbitrary but is usually UTC because the date time of
most forcing data is measured in UTC. Users should carefully adjust the time zone of
their observation data to the model’s if their observation data is measured in local time
zone. This is especially important for sub-daily running.
This folder has a “calibration.txt” file that contains the calibration configuration
site name and ratio limits of the model parameters as shown in Figure 3-11 and detailed
in Table 3-9.
21
Table 3-9 content in the Calibrations.txt
keyword Type Description
iseed SCE_UA The initial random seed
The max no. of trials allowed before
maxn SCE_UA
optimization is terminated
The mumber of shuffling loops in which
the criterion value must change by the
kstop SCE_UA
given percentage before optimization is
terminated
The Percentage by which the criterion
This folder only contains the INTERNAL potential evaporation data files named by
their date time. Please refer to Chapter 3.2.3.3 on the data time format and directory of
this folder. And. In the first time simulation of the basin, if the PETs empty, CREST
v2.1 automatically decompresses (if necessary), re-project, resample and clip the
22
external forcing file to the projection, region and resolution defined in your basic
(geographic) files. The processed forcing files will be saved in internal “.mat” format
in this folder. For later runs, if CREST finds the internal forcing files, it will ignore the
external ones and use the internal ones directly. In practice, we encourage you to store
your global/regional external forcing files in one fixed location and let model convert
between the external and internal files to save your space and preprocessing time. Due
to the complex procedure of importing external files, it can be time consuming in
simulating a basin for the first time. However, for later simulations or calibrations, the
model runs significantly faster.
This folder only contains the internal rainfall files named by their date time. Please
refer to Chapter 3.2.3.3 on the data time format and directory of this folder. All rules
in Chapter 3.3.7 apply in this folder as well.
23
4.1 Outputs in the Simulation Style
24
Figure 4-2 snapshot of the hydrograph file.
Since CREST v2.0, SCE-UA (Duan et al., 1992) is selected as the kernel algorithm
in calibration process. In CREST v2.1, SCE-UA is parallelized using the shared
memory multiprocessing (OpenMP) approach.
CREST v2.1 directly uses the screen output of SCE-UA codes in matlab written by Duan et al.,
1992. The objective function value is the NSCE of each simulation. CREST v2.1 also outputs
the calibration process to a log file in the “Results” folder, named as “log.txt”. The
calibration result is output both to the screen and to a file named
“SCE_UAyyyymmdd_hh.txt” in the same folder.
Users can run CREST in their own basins after installation. The CREST model
automatically runs over the region defined by geographic images in the basic folder. All
files in this folder must be prepared before running the models. Forcing files are also
necessary but CREST deals with most of the preprocessing.
Please follow the steps below to setup a basin of a user’s own.
1. Create a project folder that contains all folders described in Chapter 3.2.8.
25
The name of the project folder is arbitrary.
2. Create a control file by either
a. Copying the existing project file provided in the example Tar basin
folder and modifying the content according to your own basin.
or
b. Filling the content following the instructions in Chapter 3.2.
Please note that all sections in the control file are necessary for CREST.
Please use the switchers to mute the sections not needed rather than
removing those sections
3. Generate the geographic (basic) files required in the basic. Please also refer
to Chapter 3.3.1 for the definition and generation of basic files.
4. Create and fill the files needed in the “Param” and “ICS” folders. Please
refer to Chapters 3.3.2 and 3.3.4 for the files in these folders.
5. Prepare the observation files needed in the “OBS” folder. Please refer to
Chapter 3.2.3.6 and 3.3.5 for these files.
6. Run the model in simulation style using the commands introduced in
Chapter 3.1.
7. If a calibration process is needed, please also create and fill the files in the
“Calibs” folder and switch the running style in the control file accordingly.
Please refer to Chapter 3.3.6 for the calibration file and the parameter
difference in the calibration and simulation styles. The parameters used in
the simulation style and calibration style are determined by different files.
Note that a lot of users failed to calibrate the model by failing to notice this
difference.
8. A user must remember to multiply the calibrated “ratios” and magnitude
values in the parameter file used in the calibration process provided he
needs to simulate the basin using his calibrated model parameters.
6 Contact us
The official version of the OU-NASA CREST model has been developing and
26
maintaining in the Hydrometeorology and Remote Sensing Laboratory, University of
Oklahoma (http://hydro.ou.edu) and Atmospheric Radar Research Center (ARRC)
located in the National Weather Center (http://nwc.ou.edu). For the information about
the current release of the CREST model, the source code of beta versions and technical
support, please send e-mail to Prof. Yang Hong (yanghong@ou.edu) and Dr. Xinyi Shen
(shen.xinyi@ou.edu).
Xinyi Shen, Yang Hong, Ke Zhang and Zengchao Hao, (2014). “Refine a Distributed
Linear Reservoir Routing Method to Improve Performance of the CREST
Model” (minor revision).
Wang. J., Y. Hong, L. Li, J.J. Gourley, K. Yilmaz, S. I. Khan, F.S. Policelli, R.F. Adler,
S. Habib, D. Irwn, S.A. Limaye, T. Korme, and L. Okello, (2011). “The Coupled
Routing and Excess STorage (CREST) distributed hydrological model”. Hydrol.
Sciences Journal, 56, 84-98.
Xue X, Hong Y, Limaye AS, Gourley JJ, Huffman GJ, Khan SI, et al., 2013: Statistical
and hydrological evaluation of TRMM-based Multi-satellite Precipitation
Analysis over the Wangchu Basin of Bhutan: Are the latest satellite precipitation
products 3B42V7 ready for use in ungauged basins? Journal of Hydrology,
499(0): 91-99.
Khan, S. I., Y. Hong, J. Wang, K.K. Yilmaz, J.J. Gourley, R.F. Adler, G.R. Brakenridge,
F. Policelli, S. Habib, and D. Irwin, 2011: Satellite Remote Sensing and
Hydrologic Modeling for Flood Inundation Mapping in Lake Victoria Basin:
Implications for Hydrologic Prediction in Ungauged Basins, IEEE Transactions
on Geosciences and Remote Sensing, 49(1), 85-95, Jan. 2011, doi:
27
10.1109/TGRS.2010.2057513
Wu H, Adler RF, Hong Y, Tian Y, Policelli F., 2012: Evaluation of Global Flood
Detection Using Satellite-Based Rainfall and a Hydrologic Model. Journal of
Hydrometeorolog, 13(4): 1268-1284.
Duan, Q., Sorooshian, S., Gupta, V., 1992. Effective and efficient global optimization
for conceptual rainfall‐runoff models. Water Resour. Res., 28(4): 1015-1031.
28