8. Calibration tool

Calibration tool for hydrological models
in ../CWATM/calibration
using a distributed evolutionary algorithms in python: DEAP library

Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Marc Parizeau and Christian Gagné, “DEAP: Evolutionary Algorithms Made Easy”, Journal of Machine Learning Research, vol. 13, pp. 2171-2175, jul 2012

The calibration tool was created by Hylke Beck 2014 (JRC, Princeton) hylkeb@princeton.edu
Thanks Hylke for making it available for use and modification
Modified by Peter Burek
The submodule Hydrostats was created 2011 by:
Sat Kumar Tomer (modified by Hylke Beck)
Please see his book Python in Hydrology

Calibration tool structure

calibration
│-  readme.txt
│-  readme.txt
│
└--observed_data
│   └- lobith2006.cvs, ...
│
└--templates
│   └-- runpy.bat
│   └-- settings.ini

How it works

The calibration tool builds up a single-objective obtimization framework using the Python libray DEAP For each run it triggers the run of the hydrological model:

  • using a template of the settings file
  • replacing the output folder in this template file
  • replace placeholders with the values of calibration parameters, the limit of the parameter range is given in the file: ParamRanges.csv

After each run the model run is compared to observed values (e.g. observed_data/lobith2006.csv)

After the calibration, statistics and the best run is printed output

What is needed

  1. The template files in ../templates have to be adjusted
  • runpy.bat: the path to cwatm.py have to be set correctly (for linux a .sh file has to be created)
  • The actual version of a cwatm settings file has to modified:
  • replacing the output folder with the placeholder: %run_rand_id
640
OUT_Dir = %run_rand_id
  • replacing calibration parameter values with a placeholder: e.g. %SnowMelt
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
[CALIBRATION]

# These are parameter which are used for calibration
# could be any parameter, but for an easier overview, tehey are collected here
# in the calibration template a placeholder (e.g. %arnoBeta) instead of value

# Snow  SnowMeltCoef = 0.004
SnowMeltCoef = %SnowMelt
# Cropf factor correction
crop_correct =  %crop
#Soil
soildepth_factor = %soildepthF
#Soil preferentialFlowConstant = 4.0, arnoBeta_factor = 1.0
preferentialFlowConstant = %pref
arnoBeta_add = %arnoB
# groundwater recessionCoeff_factor = 1.0
recessionCoeff_factor = %reces
# runoff concentration factor runoffConc_factor = 1.0
runoffConc_factor = %runoff
#Routing manningsN = 0.04
manningsN = %CCM
# reservoir  normal storage limit (fraction of total storage, [-]) [0.15 - 0.85] default 0.5
normalStorageLimit = %normalStorageLimit
# lake parameter - factor to alpha: parameter of of channel width and weir coefficient  [0.33 - 3.] dafault 1.
lakeAFactor = %lakeAFactor
  1. the range of parameter space has to be defined in ParamRanges.csv
ParameterName,MinValue,MaxValue
SnowMelt,0.001,0.007
crop,0.8,2
soildepthF, 0.5,2
pref,0.5,8
arnoB,0.1,10
reces,0.1,10
runoff,0.1,5
CCM,0.1,10.0
normalStorageLimit,0.15,0.85
lakeAFactor,0.333,5.0
No,1,100
  1. The observed discharge has to be provided in an .cvs file e.g. observed_data/lobith2006.csv
  • In the template settings the date has to be set, so that the period of observed discharge is between SpinUp and StepEnd
  1. And empty ../catchments directory needs to be created
  2. A few option in the settings.txt have to be adjusted (how many runs?, a first run with standard parameters? etc)
[DEFAULT]
Root = C:\work\CWATM\calibration
ForcingStart = 1/1/2003
ForcingEnd = 12/31/2010

[ObservedData]
Qtss = %(Root)s\observed_data\lobith2006.csv
Column = lobith

[Path]
Templates = %(Root)s\templates
SubCatchmentPath = %(Root)s\catchments
ParamRanges = %(Root)s\ParamRanges.csv

[Templates]
ModelSettings = settings.ini
RunModel = runpy.bat

[Option]
firstrun = True
para_first = [0.0035, 1.0, 1.0, 3.0, 1.0, 1.0, 1.0, 1.0,0.5,1., 1.]
# Snowmelt, crop KC, soil depth,pref. flow, arno beta, groundwater recession, runoff conc., routing, manning factor, normalStorageLimit, lakeAFactor,No of run
bestrun = True

[DEAP]
maximize = True
use_multiprocessing = 1
ngen = 2
mu = 16
lambda_ = 16
  1. run python calibration_single.py settings.txt

Recommondations

  1. Run the model first to store the pot. evaporation results
Afterwards use the stored evaporation to run the calibration
calc_evaporation = False
  1. Run the model and store the last day to be used as initial condition for the calibration runs
Best is to use a long term run for this.
146
147
148
149
150
151
152
153
154
155
156
157
158
[INITITIAL CONDITIONS]
#-------------------------------------------------------

# for a warm start initial variables a loaded
# e.g for a start on 01/01/2010 load variable from 31/12/2009
load_initial = False
initLoad = $(BASICS:PathRoot)/init/Rhine_20030131.nc

# saving variables from this run, to initiate a warm start next run
# StepInit = saving date, can be more than one: 10/01/1973 20/01/1973
save_initial = True
initSave = $(BASICS:PathRoot)/init/Rhine
StepInit = 31/12/1995 31/12/2010
load_initial = False
save_initial = True
During calibration use:
load_initial = True
save_initial = False
  1. Use a long SpinUp time (> 5years to give groundwater enough time)