==========
20180103:
1. Compiled the code using the bug-corrected compute_rc.F90. Continued the clm4.5 run.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4.5/.
cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5
set exp=clm4.5_DE720
set num=33
/bin/cp -r ${exp}_template ${exp}
cd ${exp}/input/restart
ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5/${exp}_${num}/output
/bin/rm rst_clm4.5
ls -l
lrwxrwxrwx 1 fzeng g0620 80 2018-01-03 13:25 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_33/output/
cd ../../run
ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 71781481 2018-01-03 13:19 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x*
qsub lenkf.0.j
==========
20180108:
1. Used read_fields.m and calculate_global.m in ~/Catchment/matlab/clm4-to-clm4.5 to extract the CLM4.5 spinup output and compute 1980-2016 mean global GPP and NEE to monitor the CLM4.5.
==========
20180116:
1. Checked CNFireMod.F90 to investigate why burned area in CLM4.5 Catchment-CN is so high.
Catchment-CN orig CLM4.5
fsr_col: m/s m/s
fd_col: sec sec
forc_rain: mm/s mm/s
forc_snow: mm/s mm/s
forc_rh: %? %
fsr_pft: m/s m/s
fd_pft: hr hr
prec60_col: mm/s
prec10_col: mm/s
fd_col: sec
fd_pft: hr
fd_col(c) = fd_col(c) + fd_pft(ivt(p))*wtcol(p)*secsphr/(1.0_r8-cropf_col(c))
Unit conversion seems correct.
Checked fd_pft and fsr_pft values in subroutine CN_init in CN_DriverMod.F90.
data fd_pfx / 0., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 24., 0., 0., 0., 0., 0., 0., 0., 0./
data fsr_pfx / 0., 0.4, 0.43, 0.43, 0.4, 0.4, 0.4, 0.4, 0.4, 0.46, 0.46, 0.46, 0.46, 0.55, 0.55, 0.55, 0.55, 0.55, 0.55, 0.55, 0., 0., 0., 0., 0., 0., 0., 0./
fd_pft and fsr_pft values in original CLM4.5: /discover/nobackup/fzeng/clm4-to-clm4.5/data/pftdata4.5/pft-physiology.c130503.nc
Values in CN_init are correct.
==========
20180123: (Copied from /discover/nobackup/fzeng/notes/201801)
1. Run clm4.5 on DDT to investigate why burned area is so high.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
ls -l exec/clm4.5/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 71777037 2018-01-10 11:59 exec/clm4.5/Linux/bin/LDASsaCN_mpi.x*
ls -l exec/clm4.5_debug/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 99760874 2017-12-27 13:16 exec/clm4.5_debug/Linux/bin/LDASsaCN_mpi.x*
The executable in exec/clm4.5_debug needs to be updated.
"gmake clean" and "gmake install BOPT=g" in GEOScatchCN_GridComp, GEOSlana_GridComp and Applications/LDAS_App.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4.5_debug/.
Set up the debug run following ~/Catchment/CLM4.5/submit_next_batch_DE720:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5
/bin/cp -r clm4.5_DE720_template clm4.5_debug
cd clm4.5_debug/input/restart
ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_39/output
/bin/rm rst_clm4.5
cd ../../run
nedit lenkf.0.j &
(1) Change "/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x" to "/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/build/Linux/bin/LDASsaCN_mpi.x"
(2) Change "-exp_id clm4.5_DE720 \" to "-exp_id clm4.5_debug \"
nedit driver_inputs_0.5D_GLOBAL.nml & (identified the pixel with high burned area in Sahel using GrADS)
minlon = -1. ! min longitude
maxlon = -0.5 ! max longitude
minlat = 17. ! min latitude
maxlat = 17.5 ! max latitude
cp lenkf.0.j lenkf.0.i
nedit lenkf.0.i & (refer to /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_clm4fsun_point/run/lenkf.0.i)
Make build point to the right executable:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug
rm -f build
ln -s /gpfsm/dnb31/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4.5_debug build
ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 99751106 2018-01-23 15:10 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/build/Linux/bin/LDASsaCN_mpi.x*
Check the restart file:
ls -l input/restart/
total 0
lrwxrwxrwx 1 fzeng g0620 80 2018-01-23 15:17 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_39/output/
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/run
./lenkf.0.i
Found a bug!!
Vegetation types in this grid cell:
11: broadleaf deciduous temperate shrub [moisture stress only], fveg = 0.002
17: warm c4 grass [moisture stress only], fveg = 0.998
However, cropf_col (cropland frac.) = 0.998. Warm c4 grass is counted as crop incorrectly!! This is because of this block
if( ivt(p) > nc4_grass )then
cropf_col(c) = cropf_col(c) + wtcol(p)
end if
And nc4_grass = 16!
I should have changed nc4_grass here to nc4_grass2 (=17) after I added the split types to pftvarcon.F90!!
==========
20180124: (Copied from /discover/nobackup/fzeng/notes/201801)
1. Fixed the bug in CNFireMod.F90. Also checked all the other indices in pftvarcon.F90 to make sure they are used correctly.
"gmake clean" and "gmake install BOPT=g" in GEOScatchCN_GridComp, GEOSlana_GridComp and Applications/LDAS_App.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4.5_debug/.
Run on DDT again. farea_burned is now 0 in all 3 columns for the first 3 CN time steps that I checked. Looks correct now.
Wait until the current cycle to finish before recompiling the code.
==========
20180202:
1. The bug fix in late Jan removed the large burned area in the Sahel region. However, burned area is still too large in SHSA, MIDE and TENA. Will do point runs on DDT for these grid cells:
(1) Central Asia: 35.25N, 62.25E
(2) S America: 14.75S, 49.75W
(3) Central US: 33.25N, 89.75W
First, "gmake clean" and "gmake install BOPT=g" to compile the code.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4.5_debug/.
(1) Central Asia 35.25N, 62.25E: tile id 31292
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug
ls -l build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 99751106 2018-02-02 16:04 build/Linux/bin/LDASsaCN_mpi.x*
cd input/restart/
rm output
ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_41/output
cd ../../run/
nedit driver_inputs_0.5D_GLOBAL.nml &
minlon = 62. ! min longitude
maxlon = 62.5 ! max longitude
minlat = 35. ! min latitude
maxlat = 35.5 ! max latitude
/discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/clsm/CLM_veg_typs_fracs:
31292 199594 18 19 14 15 23.09 47.95 9.41 19.55 16 14 71.04 28.96
The CLM_veg_typs_fracs file shows that this grid cell is mostly generic crops (i.e. 18 c3_crop and 19 c3_irrigated). Why do totvegc and leafc suggest that there is no 18 or 19 but there is 22 (i.e. sprint temperate cereal)? And 22 has 91684.5859 gC/m2 in totvegc, much much larger to 14 and 15 (0.75 gC/m2 and 0.76 gC/m2, respectively).
Another question: why is totvegc so high?
Set a breakpoint in CN_init to see if the ityps are correct.
==========
20180205:
1. Continued working on fire debugging.
wtcol in CNFireArea shows that:
14: 0.1110
15: 0.2305
22: 0.6585
This matches totvegc and leafc.
I should have compared to /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/clsm/CLM4.5_veg_typs_fracs (not CLM_veg_typs_fracs). They match each other (see below)!! Great!
31292 199594 22 22 14 15 65.85 0.00 11.10 23.05 20 14 65.85 34.15
L424-429 in CNFireArea:
! For crop PFT's
if( ivt(p) > nc4_grass2 .and. wtcol(p) > 0._r8 .and. leafc_col(c) > 0._r8 )then
fuelc_crop(c)=fuelc_crop(c) + (leafc(p) + leafc_storage(p) + &
leafc_xfer(p))*wtcol(p)/cropf_col(c) + &
totlitc(c)*leafc(p)/leafc_col(c)*wtcol(p)/cropf_col(c)
end if
So among totvegc (91684.5859 gC/m2 for itpy 22 sprint temperate cereal), only leafc and totlitc (both <10 gC/m2) are counted in fuelc_crop (2.66 gC/m2).
L647-651 in CNFireArea:
if( cropf_col(c) .lt. 1.0 )then
fuelc(c) = totlitc(c)+totvegc_col(c)-rootc_col(c)-fuelc_crop(c)*cropf_col(c)
do j = 1, nlevdecomp
fuelc(c) = fuelc(c)+decomp_cpools_vr(c,j,i_cwd) * dzsoi_decomp(j)
end do
totvegc_col:
1: 60374
2: 332190
3: 894
totlitc:
1: 11.6
2: 17.8
3: 2.6
crootc_col:
1: 0.1
2: 0.01
3: 0.001
Since totvegc_col is very large for ityp 22 sprint temperate cereal, fuelc is very large.
/discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/clsm/CLM4.5_abm_peatf_gdp_hdm_fc:
abm peatf gdp hdm fc
31292 199594 7 0.0000 0.10 22.95 0.2437
What does totvegc include? Why is it so large while leafc and rootc are so low?
CNSummaryMod.F90, L876-918:
! displayed vegetation carbon, excluding storage and cpool (DISPVEGC)
dispvegc(p) = &
leafc(p) + &
frootc(p) + &
livestemc(p) + &
deadstemc(p) + &
livecrootc(p) + &
deadcrootc(p)
! stored vegetation carbon, excluding cpool (STORVEGC)
storvegc(p) = &
cpool(p) + &
leafc_storage(p) + &
frootc_storage(p) + &
livestemc_storage(p) + &
deadstemc_storage(p) + &
livecrootc_storage(p) + &
deadcrootc_storage(p) + &
leafc_xfer(p) + &
frootc_xfer(p) + &
livestemc_xfer(p) + &
deadstemc_xfer(p) + &
livecrootc_xfer(p) + &
deadcrootc_xfer(p) + &
gresp_storage(p) + &
gresp_xfer(p)
if ( crop_prog .and. ivt(p) >= npcropmin )then
storvegc(p) = storvegc(p) + &
grainc_storage(p) + &
grainc_xfer(p)
agnpp(p) = agnpp(p) + &
cpool_to_grainc(p) + &
grainc_xfer_to_grainc(p)
litfall(p) = litfall(p) + &
livestemc_to_litter(p) + &
grainc_to_food(p)
dispvegc(p) = dispvegc(p) + &
grainc(p)
end if
! total vegetation carbon, excluding cpool (TOTVEGC)
totvegc(p) = dispvegc(p) + storvegc(p)
In CN_init:
Almost all totvegc is cpool. The others are 0 or very close to 0 (negligible).
cpool:
1: 91684
2: 594464
3: 1358
What exactly is cpool?
CNCStateUpdate1Mod.F90: psnsun and psunsha --> cpool --> leafc, stemc, etc.
Why is it so high for crops? Just for ityp 22 sprint temperate cereal in this grid cell. ityp 14 and 15 have cpool close to 0.
Tried running the same point in CLM4:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_point/run
nedit driver_inputs_0.5D_GLOBAL.nml &
minlon = 62. ! min longitude
maxlon = 62.5 ! max longitude
minlat = 35. ! min latitude
maxlat = 35.5 ! max latitude
/bin/rm ../output/point/rc_out/*
./lenkf.0.i
Set a breakpoint at the end of CN_init.
cpool here is close to 0, which makes sense.
What's wrong with cpool in CLM4.5?
Run the point in CLM4.5 again:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/run
/bin/rm ../output/global/rc_out/*
./lenkf.0.i
Set breakpoints in CNCStateUpdate1Mod.F90 to investigate why cpool is so large for ityp 22 sprint temperate cereal.
cpool(p) = cpool(p) + psnsun_to_cpool(p)*dt
cpool(p) = cpool(p) + psnshade_to_cpool(p)*dt
! maintenance respiration fluxes from cpool
cpool(p) = cpool(p) - cpool_to_xsmrpool(p)*dt
cpool(p) = cpool(p) - leaf_curmr(p)*dt
cpool(p) = cpool(p) - froot_curmr(p)*dt
if (woody(ivt(p)) == 1._r8) then
cpool(p) = cpool(p) - livestem_curmr(p)*dt
cpool(p) = cpool(p) - livecroot_curmr(p)*dt
end if
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
cpool(p) = cpool(p) - livestem_curmr(p)*dt
cpool(p) = cpool(p) - grain_curmr(p)*dt
end if
! allocation fluxes
cpool(p) = cpool(p) - cpool_to_leafc(p)*dt
leafc(p) = leafc(p) + cpool_to_leafc(p)*dt
cpool(p) = cpool(p) - cpool_to_leafc_storage(p)*dt
leafc_storage(p) = leafc_storage(p) + cpool_to_leafc_storage(p)*dt
cpool(p) = cpool(p) - cpool_to_frootc(p)*dt
frootc(p) = frootc(p) + cpool_to_frootc(p)*dt
cpool(p) = cpool(p) - cpool_to_frootc_storage(p)*dt
frootc_storage(p) = frootc_storage(p) + cpool_to_frootc_storage(p)*dt
if (woody(ivt(p)) == 1._r8) then
cpool(p) = cpool(p) - cpool_to_livestemc(p)*dt
livestemc(p) = livestemc(p) + cpool_to_livestemc(p)*dt
cpool(p) = cpool(p) - cpool_to_livestemc_storage(p)*dt
livestemc_storage(p) = livestemc_storage(p) + cpool_to_livestemc_storage(p)*dt
cpool(p) = cpool(p) - cpool_to_deadstemc(p)*dt
deadstemc(p) = deadstemc(p) + cpool_to_deadstemc(p)*dt
cpool(p) = cpool(p) - cpool_to_deadstemc_storage(p)*dt
deadstemc_storage(p) = deadstemc_storage(p) + cpool_to_deadstemc_storage(p)*dt
cpool(p) = cpool(p) - cpool_to_livecrootc(p)*dt
livecrootc(p) = livecrootc(p) + cpool_to_livecrootc(p)*dt
cpool(p) = cpool(p) - cpool_to_livecrootc_storage(p)*dt
livecrootc_storage(p) = livecrootc_storage(p) + cpool_to_livecrootc_storage(p)*dt
cpool(p) = cpool(p) - cpool_to_deadcrootc(p)*dt
deadcrootc(p) = deadcrootc(p) + cpool_to_deadcrootc(p)*dt
cpool(p) = cpool(p) - cpool_to_deadcrootc_storage(p)*dt
deadcrootc_storage(p) = deadcrootc_storage(p) + cpool_to_deadcrootc_storage(p)*dt
end if
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
cpool(p) = cpool(p) - cpool_to_livestemc(p)*dt
livestemc(p) = livestemc(p) + cpool_to_livestemc(p)*dt
cpool(p) = cpool(p) - cpool_to_livestemc_storage(p)*dt
livestemc_storage(p) = livestemc_storage(p) + cpool_to_livestemc_storage(p)*dt
cpool(p) = cpool(p) - cpool_to_grainc(p)*dt
grainc(p) = grainc(p) + cpool_to_grainc(p)*dt
cpool(p) = cpool(p) - cpool_to_grainc_storage(p)*dt
grainc_storage(p) = grainc_storage(p) + cpool_to_grainc_storage(p)*dt
end if
! growth respiration fluxes for current growth
cpool(p) = cpool(p) - cpool_leaf_gr(p)*dt
cpool(p) = cpool(p) - cpool_froot_gr(p)*dt
if (woody(ivt(p)) == 1._r8) then
cpool(p) = cpool(p) - cpool_livestem_gr(p)*dt
cpool(p) = cpool(p) - cpool_deadstem_gr(p)*dt
cpool(p) = cpool(p) - cpool_livecroot_gr(p)*dt
cpool(p) = cpool(p) - cpool_deadcroot_gr(p)*dt
end if
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
cpool(p) = cpool(p) - cpool_livestem_gr(p)*dt
cpool(p) = cpool(p) - cpool_grain_gr(p)*dt
end if
! growth respiration at time of storage
cpool(p) = cpool(p) - cpool_leaf_storage_gr(p)*dt
cpool(p) = cpool(p) - cpool_froot_storage_gr(p)*dt
if (woody(ivt(p)) == 1._r8) then
cpool(p) = cpool(p) - cpool_livestem_storage_gr(p)*dt
cpool(p) = cpool(p) - cpool_deadstem_storage_gr(p)*dt
cpool(p) = cpool(p) - cpool_livecroot_storage_gr(p)*dt
cpool(p) = cpool(p) - cpool_deadcroot_storage_gr(p)*dt
end if
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
cpool(p) = cpool(p) - cpool_livestem_storage_gr(p)*dt
cpool(p) = cpool(p) - cpool_grain_storage_gr(p)*dt
end if
! growth respiration stored for release during transfer growth
cpool(p) = cpool(p) - cpool_to_gresp_storage(p)*dt
The first time step is not day time in local time. Run a couple of time steps until psn is not 0.
In the 2nd time step, date_time_new 19800101_030000z, for zone 1 (take zone 1 as an example):
dt = 5400
Input to cpool:
14 15 22
psnsun_to_cpool: 1.88e-9 1.88e-9 1.68e-10
psnshade_to_cpool: 0 0 0
Output from cpool:
14 15 22
cpool_to_xsmrpool: 0 0 0
leaf_curmr: 0 0 0
froot_curmr: 1.88e-9 1.88e-9 0 (why did nothing go to froot for ityp 22?)
livestem_curmr: 0 0 -5.42e-07 (why is livestem_curmr negative for ityp 22?!)
grain_curmr: 0 0 -2.99e-13 (why negative again?!)
cpool_to_leafc: 0 0 1.34e-10
cpool_to_leafc_storage: 0 0 0
cpool_to_frootc: 0 0 0
cpool_to_frootc_storage: 0 0 0
cpool_to_livestemc: 0 0 0
cpool_to_livestemc_storage: 0 0 0
cpool_to_grainc: 0 0 0
cpool_to_grainc_storage: 0 0 0
cpool_leaf_gr: 0 0 3.356e-11
cpool_froot_gr: 0 0 0
cpool_livestem_gr: 0 0 0
cpool_grain_gr: 0 0 0
cpool_leaf_storage_gr: 0 0 0
cpool_froot_storage_gr: 0 0 0
cpool_livestem_storage_gr: 0 0 0
cpool_grain_storage_gr: 0 0 0
cpool_to_gresp_storage: 0 0 0
The sum of cpool_to_leafc and cpool_leaf_gr is about the same as the sum of psnsun_to_cpool and psnshade_to_cpool (i.e. input to cpool). However, livestem_curmr and grain_curmr are negative, and they are output fluxes, making a gain for cpool. Does this happen every time step? If so, cpool will keep accumulating and can became very large number (like what we see for cpool of ityp 22 here).
==========
20180206:
1. Compared our CNFireMod.F90 to the original CLM4.5 one. All the changes made are necessary. No bug found.
2. Investigate why livestem_curmr and grain_curmr are negative for ityp 22, particularly livestem_curmr which is large.
livestem_curmr: live stem maintenance respiration from current GPP (gC/m2/s)
CNAllocationMod.F90
199: real(r8), pointer :: livestem_curmr(:)
430: livestem_curmr => pcf%livestem_curmr
621: livestem_curmr(p) = livestem_mr(p) * curmr_ratio
622: livestem_xsmr(p) = livestem_mr(p) - livestem_curmr(p)
In the 1st time step, date_time_new 19800101_013000z, for zone 1:
availc(p) = gpp(p) - mr
if (mr > 0._r8 .and. availc(p) < 0._r8) then
curmr = gpp(p)
curmr_ratio = curmr / mr
else
curmr_ratio = 1._r8
end if
mr = 0, so
curmr_ratio = 1
14 15 22
livestem_mr: 0 0 -5.41e-7
livestem_curmr: 0 0 -5.41e-7
In the 2nd time step, date_time_new 19800101_030000z, for zone 1:
14 15 22
psnsun_to_cpool: 1.88e-9 1.88e-9 1.68e-10
psnshade_to_cpool: 0 0 0
curmr_ratio = 1
14 15 22
livestem_mr: 0 0 -5.42e-7
livestem_curmr: 0 0 -5.42e-7
livestem_curmr is negative because livestem_mr is negative.
Why is livestem_mr negative?
CNMRespMod.F90
87: real(r8), pointer :: livestem_mr(:)
118: livestem_mr => pcf%livestem_mr
175: livestem_mr(p) = livestemn(p)*br*tc
178: livestem_mr(p) = livestemn(p)*br*tc
CNMRespMod.F90:
br = 2.525e-6_r8 constant
q10 = 1.5_r8 constant
t_ref2m = 274.4 K for all PFTs.
tc = q10**((t_ref2m(p)-SHR_CONST_TKFRZ - 20.0_r8)/10.0_r8)
if (woody(ivt(p)) == 1) then
livestem_mr(p) = livestemn(p)*br*tc
livecroot_mr(p) = livecrootn(p)*br*tc
else if (ivt(p) >= npcropmin) then
livestem_mr(p) = livestemn(p)*br*tc
grain_mr(p) = grainn(p)*br*tc
end if
In the 1st time step, date_time_new 19800101_013000z, for zone 1:
tc = 0.4674
14 15 22
livestemn: 0 0 -0.4584
For ityp 22, livestemn is negative, so livestem_mr is negative.
Why is livestemn negative?
Check CN_init:
14 15 22
pns%livestemn: 0 0 -0.4584
So pns%livestemn is negative in the restart file, i.e. at the end of last cycle!!
==========
20180207:
1. Used matlab to plot pns%livestemn in the first restart file for CLM4.5 Catchment-CN that Sarith created (or I created using Sarith's program). See if it's positive or negative.
No. There is no negative numbers. The restart file is good.
It must be some bug in the code!
==========
20180209:
1. Investigated why livestemn is negative in this grid cell Central Asia 35.25N, 62.25E: tile id 31292
Routines that modify livestemn:
CNNStateUpdate1Mod.F90
CNNStateUpdate2Mod.F90
CNNStateUpdate3Mod.F90
CNPrecisionControlMod.F90
Do the point run on DDT using the first restart file for CLM4.5 Catchment-CN generated by Sarith's program. pns%livestemn is correct, i.e. positive, in this restart file.
CLM4:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_point/input/restart
already has
catchcn_internal_rst -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/OutData2/catchcn_internal_rst
cd ../../run/
lenkf.0.i already has "-restart_path ../input/restart/ \"
/bin/rm ../output/point/rc_out/*
./lenkf.0.i
14 15 18 19
clm3%g%l%c%p%pns%livestemn: 0 0 0 0
CLM4.5:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/input/restart
ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/OutData2/catchcn_internal_clm45 catchcn_internal_rst
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/run
nedit lenkf.0.i &
changed "-restart_path ../input/restart/output \" to "-restart_path ../input/restart/ \"
/bin/rm ../output/global/rc_out/*
./lenkf.0.i
14 15 22
pns%livestemn: 0 0 0
So this is consistent with CLM4 above.
CNNStateUpdate1Mod.F90:
L486-492:
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
! lines here for consistency; the transfer terms are zero
livestemn(p) = livestemn(p) + livestemn_xfer_to_livestemn(p)*dt
livestemn_xfer(p) = livestemn_xfer(p) - livestemn_xfer_to_livestemn(p)*dt
grainn(p) = grainn(p) + grainn_xfer_to_grainn(p)*dt
grainn_xfer(p) = grainn_xfer(p) - grainn_xfer_to_grainn(p)*dt
end if
L511-518:
if (ivt(p) >= npcropmin) then ! Beth adds retrans from froot
frootn(p) = frootn(p) - frootn_to_retransn(p)*dt
retransn(p) = retransn(p) + frootn_to_retransn(p)*dt
livestemn(p) = livestemn(p) - livestemn_to_litter(p)*dt
livestemn(p) = livestemn(p) - livestemn_to_retransn(p)*dt
retransn(p) = retransn(p) + livestemn_to_retransn(p)*dt
grainn(p) = grainn(p) - grainn_to_food(p)*dt
end if
L554-563:
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
npool(p) = npool(p) - npool_to_livestemn(p)*dt
livestemn(p) = livestemn(p) + npool_to_livestemn(p)*dt
npool(p) = npool(p) - npool_to_livestemn_storage(p)*dt
livestemn_storage(p) = livestemn_storage(p) + npool_to_livestemn_storage(p)*dt
npool(p) = npool(p) - npool_to_grainn(p)*dt
grainn(p) = grainn(p) + npool_to_grainn(p)*dt
npool(p) = npool(p) - npool_to_grainn_storage(p)*dt
grainn_storage(p) = grainn_storage(p) + npool_to_grainn_storage(p)*dt
end if
Set a breakpoint at the end of subroutine NStateUpdate1. Ran for one day but pns%livestemn is still 0 for all 3 itypes (14, 15 and 22).
Added the print statement below to CNNStateUpdate1Mod.F90 and recompiled.
if (livestemn(p)<0) then
print *, 'p,livestemn(p),livestemn_xfer_to_livestemn(p)',p,livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
stop
end if
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4.5_debug/.
It stopped here (simulation started from 19800101):
date_time_new 19810416_120000z
p,livestemn(p),livestemn_xfer_to_livestemn(p): 23 -2.3759935E-02 0.0000000E+00
livestemn_to_litter(p),livestemn_to_retransn(p) 1.0972248E-05 0.0000000E+00
npool_to_livestemn(p) 2.3184464E-08
beginning_livestemn(p) + livestemn_xfer_to_livestemn(p)*dt - livestemn_to_litter(p)*dt - livestemn_to_retransn(p)*dt + npool_to_livestemn(p)*dt = end_livestemn(p)
If beginning_livestemn(p) = 0:
0 + 0*5400 - 1.0972248E-05*5400 - 0*5400 + 2.3184464E-08*5400 = -5.8E-2
-5.8E-2 is < -2.3759935E-02, so maybe the beginning_livestemn(p) was about 3.424E-2.
Why is livestemn_to_litter(p) so much larger than npool_to_livestemn(p)? Is livestemn_to_litter(p) too large or npool_to_livestemn(p) too low?
CNPhenologyMod.F90, L2407-2410, subroutine CNOffsetLitterfall:
if (ivt(p) >= npcropmin) then
livestemn_to_litter(p) = livestemc_to_litter(p) / livewdcn(ivt(p))
grainn_to_food(p) = grainc_to_food(p) / graincn(ivt(p))
end if
livewdcn is 50 for trees and prognostic crops.
Since livestemn_to_litter = 1.0972248E-05 gC/m2/s, it means that livestemc_to_litter = 1.0972248E-05*50 = 5.486124E-04 gC/m2/s. This is 2.9625 gC/m2/1.5hr (i.e. per time step).
Is this rate normal or too high? If too high, why did it suddenly become so high?
CNAllocationMod.F90, L1446-1458:
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
cng = graincn(ivt(p))
npool_to_livestemn(p) = (nlc * f3 * f4 / cnlw) * fcur
npool_to_livestemn_storage(p) = (nlc * f3 * f4 / cnlw) * (1._r8 - fcur)
npool_to_deadstemn(p) = (nlc * f3 * (1._r8 - f4) / cndw) * fcur
npool_to_deadstemn_storage(p) = (nlc * f3 * (1._r8 - f4) / cndw) * (1._r8 - fcur)
npool_to_livecrootn(p) = (nlc * f2 * f3 * f4 / cnlw) * fcur
npool_to_livecrootn_storage(p) = (nlc * f2 * f3 * f4 / cnlw) * (1._r8 - fcur)
npool_to_deadcrootn(p) = (nlc * f2 * f3 * (1._r8 - f4) / cndw) * fcur
npool_to_deadcrootn_storage(p) = (nlc * f2 * f3 * (1._r8 - f4) / cndw) * (1._r8 - fcur)
npool_to_grainn(p) = (nlc * f5 / cng) * fcur
npool_to_grainn_storage(p) = (nlc * f5 / cng) * (1._r8 -fcur)
end if
f4 = flivewd(ivt(p))
cnlw = livewdcn(ivt(p))
For ityp=22:
fcur = 1.0
cnlw = 50
f4 = 1
nlc = plant_calloc(p) / c_allometry(p)
if (ivt(p) >= npcropmin) then ! skip 2 generic crops
if (croplive(p)) then
f1 = aroot(p) / aleaf(p)
f3 = astem(p) / aleaf(p)
f5 = arepr(p) / aleaf(p)
g1 = 0.25_r8
else
f1 = 0._r8
f3 = 0._r8
f5 = 0._r8
g1 = 0.25_r8
end if
end if
real(r8), pointer :: aleaf(:) ! leaf allocation coefficient
real(r8), pointer :: astem(:) ! stem allocation coefficient
==========
20180213:
1. Compared CNPhenologyMod.F90 and CNAllocationMod.F90 to the original CLM4.5 code.
Is modification in CNPhenologyMod.F90 about nyrs correct?
Is modification in CropPhenologyInit about NHplantdate and SHplantdate correct?
2. Check CNPhenologyMod.F90 to understand why the transfer from livestem to litter occurred in spring (19810416_120000z)
==========
20180221:
1. See notes on 20171215 and 20171218 for how to do an original CLM4.5 run.
==========
20180301:
1. See notes on 20171215 and 20171218 for how to do an original CLM4.5 run.
/discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts > /bin/rm -rf test1 test2 test3 clm45cn_test1 clm45cn_test2 clm45cn_test3
Created a new case using compset I1850CRUCLM45BGC:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts
create_newcase -case clm45bgc_01 -res f09_g16 -compset I1850CRUCLM45BGC -mach discover
cd clm45bgc_01
./cesm_setup
./clm45bgc_01.build
Added this to CNNStateUpdate1Mod.F90:
if (livestemn(p)<0) then
print *, 'p,livestemn(p),livestemn_xfer_to_livestemn(p)',p,livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
stop
end if
Then:
./clm45bgc_01.build
Modified the env_run.xml to run from 1901 throught 1920:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01
./xmlchange RUN_STARTDATE='1901-01-01',STOP_OPTION=nyears,STOP_N=20,RESUBMIT=5,DATM_CLMNCEP_YR_ALIGN=1901,CCSM_CO2_PPMV=390
# Didn't do this below.
#Get restart file:
# cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc/run
# svn export --force https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/initdata_map/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc
Run it:
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01
./clm45bgc_01.run
# Didn't do this below.
#Looks like it's running. Submitted the full job.
# sbatch clm45bgc.run
==========
20180307:
1. Continued on 20180301.
In /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01/run/lnd.log.180307-155810:
define run:
source = Community Land Model CLM4.0
model_version = cesm1_2_2
run type = initial
case title = UNSET
username = fzeng
Why is it CLM4.0? Ask Justin tomorrow, also ask about model time step.
==========
20180308:
1. About my questions on 20180307, Justin said:
It looks like that's just a typo that's being written out, no need to worry. There's a README.case in your experiment directory that describes the component set at the top: "clm4.5 physics: clm4.5 bgc (cn and methane)".
It's in the lnd.log.* file, as "Timestep size (seconds): 1800".
It's also in lnd_in as dtime. 30 min is the default.
2. Tried running the original CLM4.5 with the print statement I added again:
/discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts > /bin/rm -rf clm45bgc_01
Checked and made sure the current CNNStateUpdate1Mod.F90 has the print statement below:
if (livestemn(p)<0) then
print *, 'p,livestemn(p),livestemn_xfer_to_livestemn(p)',p,livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
stop
end if
Created a new case using compset I1850CRUCLM45BGC:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts
create_newcase -case clm45bgc_01 -res f09_g16 -compset I1850CRUCLM45BGC -mach discover
cd clm45bgc_01
./cesm_setup
./clm45bgc_01.build
Modified the env_run.xml to run from 1901 throught 1920:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01
./xmlchange RUN_STARTDATE='1901-01-01',STOP_OPTION=nyears,STOP_N=20,RESUBMIT=5,DATM_CLMNCEP_YR_ALIGN=1901,CCSM_CO2_PPMV=390
# Didn't do this below.
#Get restart file:
# cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc/run
# svn export --force https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/initdata_map/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc
# Didn't do this below.
#Run it:
# interactive.py -A sp3 -n 96 -a g0620 -X --debug
# cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01
# ./clm45bgc_01.run
Submitted the full job.
sbatch clm45bgc_01.run
==========
20180312:
1. Checked the original CLM4.5 run:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01/run
The run was from 1901-01-01 to 1920-12-31, but the last output file is for 1919-10, i.e. the run is not completed. Is this due to time out?
Checked the lnd.log.180308-181621 but couldn't tell.
Asked Justin.
Justin said I can take a look at the cesm.log.180308-181621 in the same directory. This file shows that the run stopped due to some error. A lot of error messages were printed out in this file:
Negative conc. in ch4tran. c,j,deficit (mol): 18812 1
1.235977678297454E-003
Opened file ./clm45bgc_01.rtm.h0.1901-02.nc to write 131072
Opened file ./clm45bgc_01.clm2.h0.1901-02.nc to write 131072
Negative conc. in ch4tran. c,j,deficit (mol): 1460 2
1.163709876623323E-003
Negative conc. in ch4tran. c,j,deficit (mol): 48707 1
1.916877879986152E-003
Negative conc. in ch4tran. c,j,deficit (mol): 37400 1
1.016681316199017E-003
Negative conc. in ch4tran. c,j,deficit (mol): 33493 2
1.368567966872083E-003
Negative conc. in ch4tran. c,j,deficit (mol): 60816 1
1.896129279369293E-003
...
-10^-12 < smin_nh4 < 0. resetting to zero.
smin_nh4_vr(c,j), c, j: -4.235164736271502E-022 31500 9
Negative conc. in ch4tran. c,j,deficit (mol): 34126 2
2.368904236616107E-003
-10^-12 < smin_nh4 < 0. resetting to zero.
smin_nh4_vr(c,j), c, j: -4.235164736271502E-022 32255 9
...
He has never seen this before.
Could it be because I didn't use the latest restart file clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc.
To use this latest restart file, this is what I shoud do:
(1) put this file in the "run" directory
(2) in one level up: add the line below to the file "user_nl_clm"
finidat = '/discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/CASE_NAME/run/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc'
When it starts to run, I can check this file "lnd_in" in the "run" directory to see if the desired restart file is being used.
2. Do another original CLM4.5 run:
Checked and made sure the current CNNStateUpdate1Mod.F90 has the print statement below:
if (livestemn(p)<0) then
print *, 'p,livestemn(p),livestemn_xfer_to_livestemn(p)',p,livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
stop
end if
Created a new case using compset I1850CRUCLM45BGC:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts
create_newcase -case clm45bgc_02 -res f09_g16 -compset I1850CRUCLM45BGC -mach discover
cd clm45bgc_02
./cesm_setup
./clm45bgc_02.build
Modified the env_run.xml to run from 1901 throught 1920:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02
./xmlchange RUN_STARTDATE='1901-01-01',STOP_OPTION=nyears,STOP_N=20,RESUBMIT=5,DATM_CLMNCEP_YR_ALIGN=1901,CCSM_CO2_PPMV=390
Get restart file:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02/run
ln -s /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc/run/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc
[or: svn export --force https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/initdata_map/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc]
Make the run use this restart file:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02
nedit user_nl_clm &
Add this line "finidat = '/discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02/run/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc'"
Run it interactively to test:
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02
./clm45bgc_02.run
Check if it uses the right restart file:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02/run
nedit lnd_in &
[YES. It's using /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02/run/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc.]
#Submitted the full job.
# sbatch clm45bgc_02.run
The run stopped. The error message in /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_02/run/lnd.log.180312-161321 is
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
reading restart file
clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc
Reading restart dataset
check_dim ERROR: mismatch of input dimension 77378 with expected value
35204 for variable landunit
ENDRUN: called without a message string
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Later on Justin said this restart file clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c160127.nc may be for CLM5, and I was correct to use the default restart file /discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/initdata_map/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c130531.nc
==========
20180322:
1. Read CNNStateUpdate1Mod.F90
Checked code about livestemn_to_litter.
CNPhenologyMod.F90:
2408: livestemn_to_litter(p) = livestemc_to_litter(p) / livewdcn(ivt(p))
CNPhenologyMod.F90:
2392: livestemc_to_litter(p) = t1 * livestemc(p) + cpool_to_livestemc(p)
Read CNPhenologyMod.F90, including subroutine CNOffsetLitterfall (same as the original CLM4.5).
offset_flag and offset_counter in this subroutine?
Note this assumption in thsi subroutine: "this assumes that offset_counter == dt for crops" [actually only for prognostic crops]
CN_DriverMod.F90
1292: pepv%offset_counter (np) = cnpft(nc,nz,nv, 31)
1294: pepv%offset_flag (np) = cnpft(nc,nz,nv, 33)
So the starting values of offset_counter and offset_flag are from the restart file.
They are modified in CNPhenologyMod.F90:
(1) offset_counter:
501: real(r8), pointer :: offset_counter(:) ! offset counter (seconds)
596: offset_counter => pepv%offset_counter
680: ! update offset_counter and test for the end of the offset period
683: offset_counter(p) = offset_counter(p) - dt
687: if (offset_counter(p) <= 0.0_r8) then
692: offset_counter(p) = 0._r8
832: offset_counter(p) = ndays_off * secspday
922: real(r8), pointer :: offset_counter(:) ! offset counter (seconds)
1022: offset_counter => pepv%offset_counter
1100: ! update offset_counter and test for the end of the offset period
1103: offset_counter(p) = offset_counter(p) - dt
1107: if (offset_counter(p) <= 0._r8) then
1111: offset_counter(p) = 0._r8
1303: offset_counter(p) = ndays_off * secspday
1436: real(r8), pointer :: offset_counter(:) ! offset counter
1499: offset_counter => pepv%offset_counter
1804: ! offset_counter relevant only at time step of harvest
1837: offset_counter(p) = dt
2304: real(r8), pointer :: offset_counter(:) ! offset days counter
2347: offset_counter => pepv%offset_counter
2384: if (offset_counter(p) == dt) then
2388: ! this assumes that offset_counter == dt for crops
2395: t1 = dt * 2.0_r8 / (offset_counter(p) * offset_counter(p))
2396: leafc_to_litter(p) = prev_leafc_to_litter(p) + t1*(leafc(p) - prev_leafc_to_litter(p)*offset_counter(p))
2397: frootc_to_litter(p) = prev_frootc_to_litter(p) + t1*(frootc(p) - prev_frootc_to_litter(p)*offset_counter(p))
(2) offset_flag:
128: ! phenology type - they depend only on onset_flag, offset_flag, bglfr, and bgtr
500: real(r8), pointer :: offset_flag(:) ! offset flag
595: offset_flag => pepv%offset_flag
681: if (offset_flag(p) == 1.0_r8) then
691: offset_flag(p) = 0._r8
818: else if (offset_flag(p) == 0.0_r8) then
831: offset_flag(p) = 1._r8
921: real(r8), pointer :: offset_flag(:) ! offset flag
1021: offset_flag => pepv%offset_flag
1101: if (offset_flag(p) == 1._r8) then
1110: offset_flag(p) = 0._r8
1256: else if (offset_flag(p) == 0._r8) then
1268: if (offset_swi(p) >= crit_offset_swi .and. onset_flag(p) == 0._r8) offset_flag(p) = 1._r8
1290: if (offset_fdd(p) > crit_offset_fdd .and. onset_flag(p) == 0._r8) offset_flag(p) = 1._r8
1295: offset_flag(p) = 1._r8
1300: if (offset_flag(p) == 1._r8) then
1322: if (offset_flag(p) == 1._r8) then
1435: real(r8), pointer :: offset_flag(:) ! offset flag
1497: offset_flag => pepv%offset_flag
1783: offset_flag(p) = 0._r8 ! carbon and nitrogen transfers
1836: offset_flag(p) = 1._r8
2303: real(r8), pointer :: offset_flag(:) ! offset flag
2346: offset_flag => pepv%offset_flag
2382: if (offset_flag(p) == 1._r8) then
Do the point (35.25N, 62.25E) run on DDT to see the values of offset_flag and offset_counter in the restart file.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/run
ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 90 2018-02-09 12:22 catchcn_internal_rst -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/OutData2/catchcn_internal_clm45
/bin/rm ../output/global/rc_out/*
./lenkf.0.i
Set a breakpoint in CN_init
14 15 22
pepv%offset_counter 0 0 0
pepv%offset_flag 0 0 0
###How to find the corresponding values from the original CLM4.5 restart file /discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/initdata_map/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c130531.nc?
Also check if the modifications to subroutine CNPhenologyClimate (by setting breakpoints in this subroutine) are working as expected.
Beginning of the 1st time step:
14 15 22
t_ref2m 274.40 274.40 274.40
tempavg_t2m 40.51 40.51 40.51 ! temp. avg 2m air temperature (K), why so low?!!
gdd0 0 0 0
gdd8 0 0 0
gdd10 0 0 0
gdd020 0 0 0
gdd820 0 0 0
gdd1020 0 0 0
After "tempavg_t2m(p) = tempavg_t2m(p) + t_ref2m(p) * (fracday/dayspyr)"
14 15 22
tempavg_t2m 40.56 40.56 40.56
End of the 1st time step:
14 15 22
gdd020 0 0 0
gdd820 0 0 0
gdd1020 0 0 0
gdd020, gdd820 and gdd1020 are only changed at the end of each year.
Double checked subroutine CNPhenologyClimate. It looks to me that it's functioning the same as the original CLM4.5.
###Also find the values of gdd020, gdd820 and gdd1020 from the original CLM4.5 restart file /discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/initdata_map/clmi.I1850CRUCLM45BGC.0241-01-01.0.9x1.25_g1v6_simyr1850_c130531.nc.
Read subroutine CropPhenology tomorrow.
==========
20180323:
1. Read subroutine CropPhenology in CNPhenologyMod.F90.
2. Did the point run and stopped it at 19810101 to get a restart file, then from there to 19810416 to get another restart file, and then debug from there.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 > /bin/cp -pr clm4.5_debug clm4.5_debug_01
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug
/bin/rm output/global/rc_out/*
cd run
nedit lenkf.0.i & [to run from 19800101 through 19811231 to get the restart file for 19810101.
./lenkf.0.i
/bin/rm ../output/global/rc_out/*
cp lenkf.0.i lenkf.1.i
nedit lenkf.1.i & [to run from 19810101 through 19810415]
./lenkf.1.i
2. Talked to Justin and found that in the clm45bgc_01 run actually the crop model is OFF. Do another run in which crop is ON, following instructions on http://www.cesm.ucar.edu/models/cesm1.2/clm/models/lnd/clm/doc/UsersGuide/c12455.html.
> cd scripts
> ./create_newcase -case clm45bgcCROP -res f19_g16 -compset I1850CRUCLM45BGC -mach discover
> cd clm45bgcCROP
# Append "-crop on" to CLM_CONFIG_OPTS in env_build.xml (you could also use an editor)
> ./xmlchange CLM_CONFIG_OPTS="-crop on" -append
# Change to startup type so uses spunup initial conditions file for crop if it exists
# By default the model will do a hybrid startup with an initial condition file
# incompatible with the crop surface dataset.
> ./xmlchange RUN_TYPE=startup [NO need! Already so in env_run.xml.]
> ./xmlchange STOP_OPTION=nyears,STOP_N=20,REST_OPTION=nmonths,REST_N=1,RESUBMIT=1,CCSM_CO2_PPMV=390
> ./cesm_setup
# Now build and run normally
> ./clm45bgcCROP.build
> ./clm45bgcCROP.submit
Got error after "./cesm_setup":
Creating Macros file for discover
/gpfsm/dnb31/fzeng/clm_orig/cesm1_2_2/scripts/ccsm_utils/Machines/config_compilers.xml intel18 discover
Creating batch script clm45bgcCROP.run
Locking file env_mach_pes.xml
Creating user_nl_xxx files for components and cpl
Running preview_namelist script
infile is /gpfsm/dnb31/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgcCROP/Buildconf/cplconf/cesm_namelist
CLM configure done.
CLM adding use_case 1850_control defaults for var sim_year with val 1850
CLM adding use_case 1850_control defaults for var sim_year_range with val constant
CLM adding use_case 1850_control defaults for var stream_year_first_ndep with val 1850
CLM adding use_case 1850_control defaults for var stream_year_first_popdens with val 1850
CLM adding use_case 1850_control defaults for var stream_year_last_ndep with val 1850
CLM adding use_case 1850_control defaults for var stream_year_last_popdens with val 1850
CLM adding use_case 1850_control defaults for var use_case_desc with val Conditions to simulate 1850 land-use
CLM build-namelist ERROR: No default value found for fsurdat.
Are defaults provided for this resolution and land mask?
ERROR: clm.buildnml.csh failed
ERROR: /gpfsm/dnb31/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgcCROP/preview_namelists failed: 25344
3. Read "20.2.3.4 Harvest" in CLM4.5 Tech Note:
###Is GDDT2m computed correctly in Catchment-CN?
Are these below relevant?
crit_onset_fdd = 7.0 ! was 15.0 in original CLM4.5.
Write a MATLAB script to extract the info of interest from the original CLM4.5 restart file.
subroutine CNOffsetLitterfall in CNPhenologyMod.F90: why offset_flag becomes 1 and offset_counter becomes dt in mid-April for this spring temperate cereal?
==========
20180326:
1. Asked Justin where I can find the default surface data for the clm45bgcCROP case I created on 20180323.
Justin:
It looks like there's none for I1850CRUCLM45BGC compset with crop. Instead of historical (year 1850), I tried the corresponding Year 2000 compset: ICRUCLM45BGCCROP (still using CRU-NCEP forcing).
Checked and made sure the current CNNStateUpdate1Mod.F90 has the print statement below:
if (livestemn(p)<0) then
print *, 'p,livestemn(p),livestemn_xfer_to_livestemn(p)',p,livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
stop
end if
Follow his notes to create a new case:
> cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts
> ./create_newcase -case ICRUCLM45BGCCROP -res f19_g16 -compset ICRUCLM45BGCCROP -mach discover
> cd ICRUCLM45BGCCROP
> ./cesm_setup
> # at this point, run/lnd_in is showing default values for finidat and fsurfdat
> ./ICRUCLM45BGCCROP.build # at this point, it should try to download the missing data, including the forcing data set, and the modern cruncep forcing. This may take a while, and some space...
Modified the env_run.xml to run from 1901 throught 1920:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICRUCLM45BGCCROP
./xmlchange RUN_STARTDATE='1901-01-01',STOP_OPTION=nyears,STOP_N=20,REST_OPTION=nmonths,REST_N=1,RESUBMIT=5,DATM_CLMNCEP_YR_ALIGN=1901,CCSM_CO2_PPMV=390
Run it:
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICRUCLM45BGCCROP
./ICRUCLM45BGCCROP.run
It's running. Stopped it.
Submitted the full job.
sbatch ICRUCLM45BGCCROP.run
==========
20180327:
1. Continued working on 1 above on 20180326.
2. Checked the restart file used in ICRUCLM45BGCCROP
finidat = '/discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/initdata_map/clmi.ICRUCLM45BGCCROPmp24.0241-01-01.1.9x2.5_g1v6_simyr2000_c130515.nc'
The number of PFTs is 24, looks correct.
It stopped at "clm2: completed timestep 16734" (see run/lnd.log.180327-114322)
In run/cesm.log.180327-114322:
p,livestemn(p),livestemn_xfer_to_livestemn(p) 25125
-1.843356756513877E-003 0.000000000000000E+000
livestemn_to_litter(p),livestemn_to_retransn(p) 1.740304630169850E-006
1.024087086952154E-006
npool_to_livestemn(p) 7.616311387259969E-008
Compared to what I got from Catchment-CN: (see notes on 20180209)
date_time_new 19810416_120000z
p,livestemn(p),livestemn_xfer_to_livestemn(p): 23 -2.3759935E-02 0.0000000E+00
livestemn_to_litter(p),livestemn_to_retransn(p) 1.0972248E-05 0.0000000E+00
npool_to_livestemn(p) 2.3184464E-08
According to Justin:
~~~~~~~~~~~~~~~~~~~~~
We can back it out by doing a little math. Scrolling up from the bottom of the lnd.log, the last written restart statement says:
> writing restart file ./ICRUCLM45BGCCROP.clm2.r.1901-12-01-00000.nc
> for model date = 1901-12-01-00000
...
> Successfully wrote out restart data at nstep = 16032
and the last line is:
> clm2: completed timestep 16734
So looks like CLM got 702 (1800sec) timesteps past the beginning of 1901-12-01-00000, or 14 days , 15 hours, 00 min.
~~~~~~~~~~~~~~~~~~~~~
The PFT with this negative livestemn seems to be 16, c3 irrigated, which I think is one of the prognostic crop types. Earlier in my /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01 run I also have the same stop and print statements, but it was able to finish >19 years of simulation before it stopped due to time out. It seems to me that the crop model is not stable. It can run but this particular nitrogen pool can go negative. In our Catchment-CN, this has caused the cpool (which is supposed to be close to 0) to keep increasing and become huge. The result of this is too much fire burned area.
Justin:
~~~~~~~~~~~~~~~~~~~~~
I came across this recent (Dec.-Feb.) bug report:
https://github.com/ESCOMP/ctsm/issues/66
Although, its about an development version of CLM5, it seems related to your very issue.
To paraphrase: It looks like what they've done is let pools have rather large negative C or N values before crashing, and that is how it's been behaving for a while.
I should also add, it looks like they aren't comfortable with this, and plan to address it. Also, CLM's fire model had trouble about too much burning in the past, but I'm not sure if it was exactly related, or something to do with lightning, as I recall. Perhaps both?
~~~~~~~~~~~~~~~~~~~~~
==========
20180328:
1. Justin:
~~~~~~~~~~~~~~~~~~~~~
Perhaps we're both wrong about the pft index. The 25125th pft as you define is 0% of the land unit and 0% of the gridcell, shown in the variables PFT_WTLUNIT and PFT_WTGCELL. Perhaps I was wrong in asserting the the pft printed in the log can be directly matched to the index in pfts1d_itypveg. It's probable in the loop of that print statement, there's a pft filter (like p = filter_SOMTHING"). If it doesn't take too long to rerun, you could expand the print statement to write out ivt(p) to get the vegetation type (making sure ivt => pft%itype is a pointer in the module).
~~~~~~~~~~~~~~~~~~~~~
Modified the print statement in /discover/nobackup/fzeng/clm_orig/cesm1_2_2/models/lnd/clm/src/clm4_5/biogeochem/CNNStateUpdate1Mod.F90 to also print out ivt(p).
Set CONTINUE_RUN in env_run.xml to TRUE to start from the latest restart file and save time:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICRUCLM45BGCCROP
./xmlchange CONTINUE_RUN=TRUE
Re-build:
./ICRUCLM45BGCCROP.build
Run it:
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICRUCLM45BGCCROP
./ICRUCLM45BGCCROP.run > run/lnd.log.180328 [actually no need to add "> run/lnd.log.180328". The log files will be automatically written out.]
In run/lnd.log.180328-112230:
begin continuation run at:
nstep= 16033 year= 1901 month= 12 day=
1 seconds= 1800
And it stopped at timestep 16734, 702 timesteps (1800sec) past the beginning of 1901-12-01-00000, or 14 days , 15 hours, 00 min. This is the same as what I got yesterday.
In run/cesm.log.180328-112230:
p,ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 25125 23
-1.843356756513877E-003 0.000000000000000E+000
livestemn_to_litter(p),livestemn_to_retransn(p) 1.740304630169850E-006
1.024087086952154E-006
npool_to_livestemn(p) 7.616311387259970E-008
So the problematic pft is 23 (soybean).
==========
20180329:
1. Removed the stop statement and only kept the print statement to let it run so that we know whether the issue also occurs to other PFTs.
CNNStateUpdate1Mod.F90:
if (livestemn(p)<0) then
print *, 'p,ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p)',p,ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
end if
Checked and made sure that CONTINUE_RUN in env_run.xml is TRUE to start from the latest restart file and save time:
Re-build:
./ICRUCLM45BGCCROP.build
Run it:
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICRUCLM45BGCCROP
./ICRUCLM45BGCCROP.run
run/cesm.log.180329-143222: these three crop types below have negative livestemn in many tiles:
17: corn
19: spring temperate cereal
23: soybean
Since the crop model has such issue, stopped the clm4.5_DE720 run.
==========
20180426:
1. Due to the negative livestemn issue in the original CLM4.5 prognostic crops, we decided to remove prognostic crops from Catchment-CN.
What is the land cover map used in the original CLM4.5 run with compset I1850CRUCLM45BGC in which the crop model is OFF? The experiment is /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc.
In both /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/clm45bgc_01/run/lnd_in and /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICRUCLM45BGCCROP/run/lnd_in, it shows that the land cover map used is
fpftcon = '/discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/pftdata/pft-physiology.c130503.nc'
clm45bgc_01:
fsurdat = '/discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/surfdata_map/surfdata_0.9x1.25_simyr1850_c130415.nc'
ICRUCLM45BGCCROP:
fsurdat = '/discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/surfdata_map/surfdata_1.9x2.5_mp24_simyr2000_c130419.nc'
surfdata_0.9x1.25_simyr1850_c130415.nc:
:Vegetation_type_raw_data_filename = "mksrf_landuse_rc1850_c090630.nc" ;
surfdata_1.9x2.5_mp24_simyr2000_c130419.nc:
:Vegetation_type_raw_data_filename = "mksrf_24pftNT_landuse_rc2000_c121207.nc" ;
mksrf_24pftNT_landuse_rc2000_c121207.nc is the 3-minute raw data file that I obtained from Keith W. Oleson on 8/11/2016. Sarith used it to generate the new CLM_veg_typs_fracs for the upgraded Catchment-CN version CLM4.5. (See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes)
Need to get the raw data file that doesn't have prognostic crops: mksrf_landuse_rc1850_c090630.nc
Catchment-CN version CLM4 used the land cover map in
https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/surfdata/surfdata_0.23x0.31_simyr2000_c100406.nc
(see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes)
What's the raw land cover data file for it?
A copy of surfdata_0.23x0.31_simyr2000_c100406.nc is in land01:/terra/fzeng/clm/surfdata_map.
:Vegetation_type_raw_data_filename = "mksrf_landuse_rc2000_c090630.nc" ;
==========
20180427:
1. What's the difference between mksrf_landuse_rc1850_c090630.nc (used in the original CLM4.5 run with CROP OFF) and mksrf_landuse_rc2000_c090630.nc (used in Catchment-CN CLM4)?
Can we use mksrf_landuse_rc2000_c090630.nc for Catchment-CN CLM4.5 as well?
==========
20180430:
1. Both mksrf_landuse_rc1850_c090630.nc and mksrf_landuse_rc2000_c090630.nc can be found in https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/rawdata/pftlandusedyn.0.5x0.5.simyr1850-2005.c090630/ (thanks to Justin, see his email on 4/30/2018).
According to Justin: CESM uses 1850 as the nominal pre-industrial conditions year, and 2000 corresponds to present-day.
Fanwei: Since mksrf_land_use_rc1850_c090630.nc is used for the original CLM4.5 with CROP OFF for pre-industrial conditions year, I think mksrf_landuse_rc2000_c090630.nc is suitable for the CLM4.5 version of Catchment-CN without prognostic crops. Catchment-CN is usually used for present-day simulations. Therefore, we can just use the CLM4 Catchment-CN restart file for the CLM4.5 Catchment-CN without prognostic crops.
==========
20180503:
1. Which part of the code is specific to c3_irrigated?
(1) discover32:/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > ~bmauer/bin/ack nc3irrig
pftvarcon.F90
41: integer, parameter :: nc3irrig = 19 ! C3_irrigated
CNVegStructUpdateMod.F90
42: use pftvarcon , only: noveg, nc3crop, nc3irrig, nbrdlf_evr_shrub, nbrdlf_dcd_brl_shrub
202: if (ivt(p) == nc3crop .or. ivt(p) == nc3irrig) then ! generic crops [for updating sai]
nc3irrig is only used in CNVegStructUpdateMod.F90, and it's treated the same as generic crop nc3crop.
(2) What about parameters for nc3crop vs. nc3irrig?
Checked subroutine CN_init in CN_DriverMod.F90. The parameter values are the same for nc3crop vs. nc3irrig.
(3) How is the code different for the two split types?
L65 of Greg's notes /discover/nobackup/fzeng/notes/cn.txt.20150827:
To disable the temperature stress trigger for the four types (broadleaf deciduous temperate shrub, cool c3 grass, warm c4 grass, and crop, the freezing degree-day critical value was changed from 15 days to 999 days, ensuring the threshold is never met. To enable seasonal deciduous trigger for these four "split" types, the offset (leaf-fall) flag is triggered if the length of daylight is less than the critical value (39300 seconds; ~10.9 hours), with a mechanism to prevent growth onset between offset and the winter solstice, and to allow check for new growth (the growing degree-day summation) after the winter solstice. These changes are in subroutine CNStressDecidPhenology in code CNPhenologyMod.F90. The split types are defined in pftvarcon.F90, where the new PFT were added, shifting several existing PFT index values (the new types could not be added ad the end of the list due to the use of "ncrop" in CNVegStructUpdateMod.F90).
In L904-906 of CLM4 Catchment-CN CNPhenologyMod.F90:
crit_offset_fdd = 15.0
if(ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) crit_offset_fdd = 999.0 ! no T stress trigger
if(ivt(p)==15 .or. ivt(p)==19 .or. ivt(p)==11 .or. ivt(p)==17) crit_offset_fdd = 999.0 ! no T stress trigger
But I don't have this block in CLM4.5 Catchment-CN Phenology!! Need to add this in.
(4) Search for how nc3crop is used in the code.
CNCStateUpdate1Mod.F90
127: use pftvarcon , only: npcropmin, nc3crop
nc3crop is not used in this file at all. Removed it.
CNNStateUpdate1Mod.F90
49: use pftvarcon , only: npcropmin, nc3crop
nc3crop is not used in this file at all. Removed it.
pftvarcon.F90
40: integer, parameter :: nc3crop = 18 ! C3_crop
CNPrecisionControlMod.F90
46: use pftvarcon, only: nc3crop
451: if ( crop_prog .and. ivt(p) >= nc3crop )then
717: if ( crop_prog .and. ivt(p) >= nc3crop )then
CNVegStructUpdateMod.F90
42: use pftvarcon , only: noveg, nc3crop, nc3irrig, nbrdlf_evr_shrub, nbrdlf_dcd_brl_shrub
202: if (ivt(p) == nc3crop .or. ivt(p) == nc3irrig) then ! generic crops
CNFireMod.F90
150: use pftvarcon , only: nc4_grass2, nc3crop, ndllf_evr_tmp_tree, &
459: if( ivt(p) .lt. nc3crop .and. cropf_col(c) .lt. 1.0_r8 )then
747: use pftvarcon, only: nc3crop
1181: if( ivt(p) .lt. nc3crop )then
(5) Irrigation code in original CLM4.5?
Whether or not the pft is irrigated is determined by the irrigated parameter. The value of this parameter can be found in land01:/terra/fzeng/clm/pftdata/pft-physiology.c130503.nc.
pftnum = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24 ;
irrigated = 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1,
0, 1, 0, 1 ;
The irrigation code is in
CanopyFluxesMod.F90
Hydrology2Mod.F90
BalanceCheckMod.F90
SLakeHydrologyMod.F90
Hydrology1Mod.F90
BiogeophysRestMod.F90
SoilHydrologyMod.F90
(5.1) CanopyFluxesMod.F90:
L758-817:
! Determine if irrigation is needed (over irrigated soil columns)
! First, determine in what grid cells we need to bother 'measuring' soil water, to see if we need irrigation
! Also set n_irrig_steps_left for these grid cells
! n_irrig_steps_left(p) > 0 is ok even if irrig_rate(p) ends up = 0
! in this case, we'll irrigate by 0 for the given number of time steps
call get_prev_date(yr, mon, day, time) ! get time as of beginning of time step
do f = 1, fn
p = filterp(f)
c = pcolumn(p)
g = pgridcell(p)
if (irrigated(ivt(p)) == 1._r8 .and. elai(p) > irrig_min_lai .and. btran(p) < irrig_btran_thresh) then
! see if it's the right time of day to start irrigating:
local_time = modulo(time + nint(grc%londeg(g)/degpsec), isecspday)
if (modulo(local_time - irrig_start_time, isecspday) < dtime) then
! it's time to start irrigating
check_for_irrig(p) = .true.
n_irrig_steps_left(c) = irrig_nsteps_per_day
irrig_rate(c) = 0._r8 ! reset; we'll add to this later
else
check_for_irrig(p) = .false.
end if
else ! non-irrig pft or elai<=irrig_min_lai or btran>irrig_btran_thresh
check_for_irrig(p) = .false.
end if
end do
! Now 'measure' soil water for the grid cells identified above and see if the soil is dry enough to warrant irrigation
frozen_soil(:) = .false.
do j = 1,nlevgrnd
do f = 1, fn
p = filterp(f)
c = pcolumn(p)
if (check_for_irrig(p) .and. .not. frozen_soil(c)) then
! if level L was frozen, then we don't look at any levels below L
if (t_soisno(c,j) <= SHR_CONST_TKFRZ) then
frozen_soil(c) = .true.
else if (rootfr(p,j) > 0._r8) then
! determine soil water deficit in this layer:
! Calculate vol_liq_so - i.e., vol_liq at which smp_node = smpso - by inverting the above equations
! for the root resistance factors
vol_ice = min(watsat(c,j), h2osoi_ice(c,j)/(dz(c,j)*denice)) ! this duplicates the above equation for vol_ice
eff_porosity = watsat(c,j)-vol_ice ! this duplicates the above equation for eff_porosity
vol_liq_so = eff_porosity * (-smpso(ivt(p))/sucsat(c,j))**(-1/bsw(c,j))
! Translate vol_liq_so and eff_porosity into h2osoi_liq_so and h2osoi_liq_sat and calculate deficit
h2osoi_liq_so = vol_liq_so * denh2o * dz(c,j)
h2osoi_liq_sat = eff_porosity * denh2o * dz(c,j)
deficit = max((h2osoi_liq_so + irrig_factor*(h2osoi_liq_sat - h2osoi_liq_so)) - h2osoi_liq(c,j), 0._r8)
! Add deficit to irrig_rate, converting units from mm to mm/sec
irrig_rate(c) = irrig_rate(c) + deficit/(dtime*irrig_nsteps_per_day)
end if ! else if (rootfr(p,j) .gt. 0)
end if ! if (check_for_irrig(p) .and. .not. frozen_soil(c))
end do ! do f
end do ! do j
(5.2) Hydrology1Mod.F90:
L454-468:
! Determine whether we're irrigating here; set qflx_irrig appropriately
if (n_irrig_steps_left(c) > 0) then
qflx_irrig(c) = irrig_rate(c)
n_irrig_steps_left(c) = n_irrig_steps_left(c) - 1
else
qflx_irrig(c) = 0._r8
end if
! Add irrigation water directly onto ground (bypassing canopy interception)
! Note that it's still possible that (some of) this irrigation water will runoff (as runoff is computed later)
qflx_prec_grnd_rain(p) = qflx_prec_grnd_rain(p) + qflx_irrig(c)
! Done irrigation
qflx_prec_grnd(p) = qflx_prec_grnd_snow(p) + qflx_prec_grnd_rain(p)
2. Modifications needed to remove prognostic crops from CLM4.5 Catchment-CN.
(1) pftvarcon.F90 (refer to CLM4 Catchment-CN version):
L41:
"integer, parameter :: nc3irrig = 19 ! C3_irrigated"
should be
"integer, parameter :: nc3crop2 = 19 ! C3_crop [moisture stress only]"
And more for this file.
Check the impact of the changes!!
(2) clm_varpar.F90:
L29 "numpft = 27"
should be
"numpft = 19?"
Modify L101 "integer, parameter, PUBLIC :: map_cat(0:numpft) = (/4,3,3,3,1,1,2,2,2,5,5,5,6,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4/)" accordingly.
Check the impact of the changes!!
(3) CNVegStructUpdateMod.F90:
L42: use pftvarcon , only: noveg, nc3crop, nc3irrig, nbrdlf_evr_shrub, nbrdlf_dcd_brl_shrub
L202: if (ivt(p) == nc3crop .or. ivt(p) == nc3irrig) then ! generic crops
In both lines, replace "nc3irrig" with "nc3crop2".
(4) Set crop_prog to .false.
(5) Change nc3crop to nc3crop2 for the lines below in CNPrecisionControlMod.F90
46: use pftvarcon, only: nc3crop
451: if ( crop_prog .and. ivt(p) >= nc3crop )then
717: if ( crop_prog .and. ivt(p) >= nc3crop )then
(6) subroutine CN_init in CN_DriverMod.F90
fcur: use the same values in our CLM4 Catchment-CN
(7) To disable the temperature stress trigger and enable seasonal deciduous trigger for the four "split" types:
Add this block below to CNPhenologyMod.F90 right before L1097 "crit_onset_gdd = exp(4.8_r8 + 0.13_r8*(annavg_t2m(p) - SHR_CONST_TKFRZ))":
! set flag for solstice period (winter->summer = 1, summer->winter = 0)
if (dayl(p) >= prev_dayl(p)) then
ws_flag = 1.
else
ws_flag = 0.
end if
Add this block below to CNPhenologyMod.F90 right after L1097 "crit_onset_gdd = exp(4.8_r8 + 0.13_r8*(annavg_t2m(p) - SHR_CONST_TKFRZ))":
crit_offset_fdd = 15.0
if(ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) crit_offset_fdd = 999.0 ! no T stress trigger
if(ivt(p)==15 .or. ivt(p)==19 .or. ivt(p)==11 .or. ivt(p)==17) crit_offset_fdd = 999.0 ! no T stress trigger
Add this block below to CNPhenologyMod.F90 right after the "if (onset_fdd(p) > crit_onset_fdd) then" block (L1181-1185):
if (ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) then ! gkw: special case; seasonal deciduous
! after winter solstice, allow check for new growth
if (onset_gddflag(p) == 0. .and. ws_flag == 1.) then
onset_gddflag(p) = 1.
onset_gdd(p) = 0.
onset_fdd(p) = 999.
onset_swi(p) = 0.
end if
! before winter solstice, prevent growth onset
if (ws_flag == 0.) then
if (onset_flag(p) == 1. .or. dormant_flag(p) == 1. .or. onset_gddflag(p) == 1.) then
onset_flag(p) = 0.
onset_gddflag(p) = 0.
onset_gdd(p) = 0.
onset_fdd(p) = 999.
onset_swi(p) = 0.
end if
endif
endif
Add this block below to CNPhenologyMod.F90 right after the "if (dayl(p) <= secspqtrday) then" block (L1294-1296):
! only begin to test for offset daylength once past the summer sol
if( ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) then ! gkw: special case
if (ws_flag == 0. .and. dayl(p) < crit_dayl) then
offset_flag(p) = 1.
end if
endif
Carefully compare the CNPhenologyMod.F90 between CLM4 Catchment-CN and CLM4.5 Catchment-CN to make sure the code for the two split types is included in CNPhenologyMod.F90 of CLM4.5 Catchment-CN!!!
Also read through Greg's notes /discover/nobackup/fzeng/notes/cn.txt.20150827 carefully to see if there is anything else I missed!!
Also compare all the CN files between CLM4 Catchment-CN and original CLM4 for this purpose.
Things to note in cn.txt:
L143:
CNPhenologyMod.F90: "soilt" in CLM was set to soil layer 3; we are using a single soil layer, so changed this to 1; also, we are using max(t_grnd,t_soisno) for the determination of an "onset" trigger, which was probably a mistake... in future versions, t_grnd should be removed from this check (this is the unified model's TG).
==========
20180504:
1. Continued working on 1 and 2 above on 20180503.
==========
20180507:
1. Continued working on 1 and 2 above on 20180503.
==========
20180508:
1. Continue working on 1 and 2 above on 20180503.
==========
20180510:
1. Follow notes on 20180503 to modify the code to remove prognostic crops.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
mkdir progCROP
(1) pftvarcon.F90 (refer to CLM4 Catchment-CN version):
cp -p pftvarcon.F90 progCROP/.
L41: changed
"integer, parameter :: nc3irrig = 19 ! C3_irrigated"
to
"integer, parameter :: nc3crop2 = 19 ! C3_crop [moisture stress only]"
(2) clm_varpar.F90:
cp -p clm_varpar.F90 progCROP/.
L29: changed
"integer, parameter :: numpft = 27"
to
"integer, parameter :: numpft = 19"
L101: changed
"integer, parameter, PUBLIC :: map_cat(0:numpft) = (/4,3,3,3,1,1,2,2,2,5,5,5,6,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4/)"
to
"integer, parameter, PUBLIC :: map_cat(0:numpft) = (/4,3,3,3,1,1,2,2,2,5,5,5,6,4,4,4,4,4,4,4/)"
L90-91: (modified on 20180522)
integer, parameter, PUBLIC :: VAR_COL=35 ! number of CN column restart variables
integer, parameter, PUBLIC :: VAR_PFT=75 ! number of CN PFT variables per column
(3) CNVegStructUpdateMod.F90:
cp -p CNVegStructUpdateMod.F90 progCROP/. [actually forgot to save a copy before the modifications below]
L42: changed
"use pftvarcon , only: noveg, nc3crop, nc3irrig, nbrdlf_evr_shrub, nbrdlf_dcd_brl_shrub"
to
"use pftvarcon , only: noveg, nc3crop, nc3crop2, nbrdlf_evr_shrub, nbrdlf_dcd_brl_shrub"
L202: changed
"if (ivt(p) == nc3crop .or. ivt(p) == nc3irrig) then ! generic crops"
to
"if (ivt(p) == nc3crop .or. ivt(p) == nc3crop2) then ! generic crops"
(4) Set crop_prog to .false.
cp -p clm_varctl.F90 progCROP/.
L72: changed
"logical, public :: crop_prog = .true."
to
"logical, public :: crop_prog = .false."
(5) Change nc3crop to nc3crop2 for the lines below in CNPrecisionControlMod.F90
cp -p CNPrecisionControlMod.F90 progCROP/.
Changed "nc3crop" to "nc3crop2" in these 3 lines below:
46: use pftvarcon, only: nc3crop
451: if ( crop_prog .and. ivt(p) >= nc3crop )then
717: if ( crop_prog .and. ivt(p) >= nc3crop )then
(6) subroutine CN_init in CN_DriverMod.F90
cp -p CN_DriverMod.F90 progCROP/.
Removed the parameters for the prognostic crops.
fcur: use the same values in our CLM4 Catchment-CN
What else? Variables in the restart file? See the slides in CLM4 vs. CLM4.5.pptx.
More modifications are needed in subroutine CN_init!!!
(7) CNPhenologyMod.F90: see 20180521.
==========
20180516:
1. Continued working on 1(6) above on 20180510.
New (relative to CLM4 Catchment-CN) variables added to the restart file of CLM4.5 Catchment-CN. Which of them are only for prognostic crops and should be removed from the restart file?
The ones denoted by "*" below are for prognostic crops only and can be removed from the restart file.
The ones denoted by "**" below are for vertical transport of soil carbon (currently OFF) only and can be removed from the restart file.
Tile level:
RHM: same as rhm and forc_rh, only used in CNFireMod.F90.
WINDM: same as windm and forc_wind, only used in CNFireMod.F90.
RAINFM: same as rainfm and forc_rain, only used in CNFireMod.F90.
SNOWFM: same as snowfm and forc_snow, only used in CNFireMod.F90.
RUNSRFM: same as qflx_surf, only used in CNNDynamicsMod.F90 for VERTSOILC.
AR1M: same as car1m, used to compute fsat, fsat is only used in CNFireMod.F90.
T2M10D: same as t10d and t10, used in subroutine Photosynthesis to compute jmax25top for all PFTs, and in subroutine CropPhenology which is only called if doalb and num_pcropp > 0 (see CNPhenologyMod.F90). doalb = .true. in CN_DriverMod.F90. num_pcropp will be 0 if we use the same vegetation maps as CLM4 Catchment-CN. Therefore, subroutine CropPhenology is only called for prognostic crops.
TPREC10D: same as prec10d and prec10, only used in CNFireMod.F90.
TPREC60D: same as prec60d and prec60, only used in CNFireMod.F90.
*T2M1H: used to compute T2MMAXI and T2MMINI in process_cn.F90.
*T2MMAXI: used to compute T2MMAX in process_cn.F90.
*T2MMAX: same as tm_max and t_ref2m_max, only used in subroutine vernalization in CNPhenologyMod.F90. Subroutine vernalization is only called for winter temperate cereal (one of the prognostic crops).
*T2MMINI: used to compute T2MMIN, T2MIN5D and T2MIN10D in process_cn.F90.
*T2MMIN: same as tm_min and t_ref2m_min, only used in subroutine CropPhenology and subroutine vernalization, both of which are only for prognostic crops.
*T2MIN5D: same as tdm5d and a5tmin, only used in subroutine CropPhenology.
*T2MIN10D: same as tdm10d and a10tmin, only used in subroutine CropPhenology.
*GDD0: same as gdd0c and gdd0, used to compute gdd020 for prognostic crops in subroutine CNPhenologyClimate (in CNPhenologyMod.F90). Need to comment out related declaration and pointers?
*GDD8: same as gdd8c and gdd8, used to compute gdd820 for prognostic crops in subroutine CNPhenologyClimate (in CNPhenologyMod.F90). Need to comment out related declaration and pointers?
*GDD10: same as gdd10c and gdd10, used to compute gdd1020 for prognostic crops in subroutine CNPhenologyClimate (in CNPhenologyMod.F90). Need to comment out related declaration and pointers?
Column level:
**! 30 cps%altmax: used for VERTSOILC in subroutine CNSoilLittVertTransp in CNSoilLittVertTranspMod.F90.
**! 35 cps%altmax_lastyear: same as cps%altmax.
**! 36 cps%altmax_indx: used to compute altmax_lastyear_indx in CN_DriverMod.F90.
**! 39 cps%altmax_lastyear_indx: used in subroutine decomp_vertprofiles in CNVerticalProfileMod.F90.
PFT level:
*! 75 pps%gddplant: same as hui, hui is used to compute GDDfrac in CNNDynamicsMod.F90, and used in subroutine CropPhenology, and used for prognostic crops in subroutine CNAllocation. GDDfrac is only used for soybean in CNNDynamicsMod.F90.
*! 76 pps%gddtsoi: same as leafout, used in subroutine CropPhenology, and in CNAllocationMod.F90 for prognostic crops.
*! 77 pps%peaklai: used in CNAllocationMod.F90 for prognostic crops.
*! 78 pps%idop: used in subroutine CropPhenology.
*! 79 pps%aleaf: used for prognostic crops in subroutine CNAllocation.
*! 80 pps%aleafi: used for prognostic crops in subroutine CNAllocation.
*! 81 pps%astem: used for prognostic crops in subroutine CNAllocation.
*! 82 pps%astemi: used for prognostic crops in subroutine CNAllocation.
*! 83 pps%htmx: used to compute htop for prognostic crops in CNVegStructUpdateMod.F90.
*! 84 pps%hdidx: used in subroutine vernalization in CNPhenologyMod.F90. Subroutine vernalization is only called for winter temperate cereal (one of the prognostic crops).
*! 85 pps%vf: vernalization factor for cereal, used in subroutine CropPhenology and subroutine vernalization in CNPhenologyMod.F90.
*! 86 pps%cumvd: cumulative vernalization dependence? used in subroutine vernalization in CNPhenologyMod.F90.
*! 87 pps%croplive: used in CNNDynamicsMod.F90 for soybean, in subroutine CropPhenology, and for prognostic crops in CNAllocationMod.F90.
*! 88 pps%cropplant: used in subroutine CropPhenology.
*! 89 pps%harvdate: used for prognostic crops in CNCStateUpdate1Mod.F90, in subroutine CropPhenology, and for prognostic crops in CNVegStructUpdateMod.F90.
*! 90 pps%gdd1020: used for soybean and soybeanirrig in subroutine CropPhenology.
*! 91 pps%gdd820: used for corn and cornirrig in subroutine CropPhenology.
*! 92 pps%gdd020: used for scereal and scerealirrig in subroutine CropPhenology.
*! 93 pps%gddmaturity: used for soybean in CNNDynamicsMod.F90, in subroutine CropPhenology and subroutine vernalization, and for prognostic crops in CNAllocationMod.F90.
*! 94 pps%huileaf: used in subroutine CropPhenology, and for prognostic crops in CNAllocationMod.F90.
*! 95 pps%huigrain: used in subroutine CropPhenology and subroutine vernalization, and for prognostic crops in CNAllocationMod.F90.
*! 96 pcs%grainc: used for prognostic crops in CNCStateUpdate1Mod.F90, CNSummaryMod.F90, CNPrecisionControlMod.F90, and CNiniTimeVar.F90.
*! 97 pcs%grainc_storage: same as grainc.
*! 98 pcs%grainc_xfer: same as grainc.
*! 99 pns%grainn: used for prognostic crops in CNMRespMod.F90, CNNStateUpdate1Mod.F90, CNSummaryMod.F90, CNPrecisionControlMod.F90, and CNiniTimeVar.F90.
*!100 pns%grainn_storage: same as grainn.
*!101 pns%grainn_xfer: same as grainn.
*!102 pepv%fert_counter: used in subroutine CropPhenology.
*!103 pnf%fert: used in subroutine CNNFert (only called for prognostic crops in CNEcosystemDynMod.F90) in CNNDynamicsMod.F90, and in subroutine CropPhenology.
*!104 pepv%grain_flag: used for prognostic crops in CNAllocationMod.F90.
!105 pepv%plant_ndemand: used for soybean in CNNDynamicsMod.F90, and for all PFTs in CNAllocationMod.F90.
Relevant routines that need to modify when the restart file is changed: any more?
GEOSlana_GridComp/catch_types.F90
GEOSlana_GridComp/clsm_ensdrv_init_routines.F90
GEOSlana_GridComp/lsm_calib_routines.F90
GEOSlana_GridComp/process_cn.F90
GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/CN_DriverMod.F90
GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90
NOTE: Before start the spinup run again, ask Melanie what additional variables she would like to write to output.
==========
20180517:
1. Continued working on 1 above on 20180516.
==========
20180518:
1. Continued working on 1 above on 20180516.
==========
20180521:
1. Continued working on 1 above on 20180516.
2. Sent Sarith a list of variables to be removed from the CLM4.5 Catchment-CN restart.
Tile level:
T2M1H
T2MMAXI
T2MMAX
T2MMINI
T2MMIN
T2MIN5D
T2MIN10D
GDD0
GDD8
GDD10
Column level:
! 30 cps%altmax
! 35 cps%altmax_lastyear
! 36 cps%altmax_indx
! 39 cps%altmax_lastyear_indx
PFT level:
! 75 pps%gddplant
! 76 pps%gddtsoi
! 77 pps%peaklai
! 78 pps%idop
! 79 pps%aleaf
! 80 pps%aleafi
! 81 pps%astem
! 82 pps%astemi
! 83 pps%htmx
! 84 pps%hdidx
! 85 pps%vf
! 86 pps%cumvd
! 87 pps%croplive
! 88 pps%cropplant
! 89 pps%harvdate
! 90 pps%gdd1020
! 91 pps%gdd820
! 92 pps%gdd020
! 93 pps%gddmaturity
! 94 pps%huileaf
! 95 pps%huigrain
! 96 pcs%grainc
! 97 pcs%grainc_storage
! 98 pcs%grainc_xfer
! 99 pns%grainn
!100 pns%grainn_storage
!101 pns%grainn_xfer
!102 pepv%fert_counter
!103 pnf%fert
!104 pepv%grain_flag
Sarith modified the dummy file and told me to try. Will do this after I finish modifying the driver routines.
3. Modified CNPhenologyMod.F90 to disable the temperature stress trigger and enable seasonal deciduous trigger for the four "split" types. These changes are in subroutine CNStressDecidPhenology.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
cp -p CNPhenologyMod.F90 CNPhenologyMod.F90.NOsplit
Add this block below to CNPhenologyMod.F90 right before L1097 "crit_onset_gdd = exp(4.8_r8 + 0.13_r8*(annavg_t2m(p) - SHR_CONST_TKFRZ))":
! set flag for solstice period (winter->summer = 1, summer->winter = 0)
if (dayl(p) >= prev_dayl(p)) then
ws_flag = 1.
else
ws_flag = 0.
end if
Add this block below to CNPhenologyMod.F90 right after L1097 "crit_onset_gdd = exp(4.8_r8 + 0.13_r8*(annavg_t2m(p) - SHR_CONST_TKFRZ))":
crit_offset_fdd = 15.0
if(ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) crit_offset_fdd = 999.0 ! no T stress trigger
if(ivt(p)==15 .or. ivt(p)==19 .or. ivt(p)==11 .or. ivt(p)==17) crit_offset_fdd = 999.0 ! no T stress trigger
Add this block below to CNPhenologyMod.F90 right after the "if (onset_fdd(p) > crit_onset_fdd) then" block (L1181-1185):
if (ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) then ! gkw: special case; seasonal deciduous
! after winter solstice, allow check for new growth
if (onset_gddflag(p) == 0. .and. ws_flag == 1.) then
onset_gddflag(p) = 1.
onset_gdd(p) = 0.
onset_fdd(p) = 999.
onset_swi(p) = 0.
end if
! before winter solstice, prevent growth onset
if (ws_flag == 0.) then
if (onset_flag(p) == 1. .or. dormant_flag(p) == 1. .or. onset_gddflag(p) == 1.) then
onset_flag(p) = 0.
onset_gddflag(p) = 0.
onset_gdd(p) = 0.
onset_fdd(p) = 999.
onset_swi(p) = 0.
end if
endif
endif
Add this block below to CNPhenologyMod.F90 right after the "if (dayl(p) <= secspqtrday) then" block (L1294-1296):
! only begin to test for offset daylength once past the summer sol
if( ivt(p)==14 .or. ivt(p)==18 .or. ivt(p)==10 .or. ivt(p)==16) then ! gkw: special case
if (ws_flag == 0. .and. dayl(p) < crit_dayl) then
offset_flag(p) = 1.
end if
endif
4. Carefully compared the CNPhenologyMod.F90 between CLM4 Catchment-CN and CLM4.5 Catchment-CN to make sure the code for the two split types is included in CNPhenologyMod.F90 of CLM4.5 Catchment-CN!!!
Found that in subroutine CNStressDecidPhenology in the CNPhenologyMod.F90 of original CLM4 and CLM4 Catchment-CN:
real, pointer :: leafc_xfer(:) ! (kgC/m2) leaf C transfer
real, pointer :: frootc_xfer(:) ! (kgC/m2) fine root C transfer
real, pointer :: livestemc_xfer(:) ! (kgC/m2) live stem C transfer
real, pointer :: deadstemc_xfer(:) ! (kgC/m2) dead stem C transfer
real, pointer :: livecrootc_xfer(:) ! (kgC/m2) live coarse root C transfer
real, pointer :: deadcrootc_xfer(:) ! (kgC/m2) dead coarse root C transfer
real, pointer :: leafn_xfer(:) ! (kgN/m2) leaf N transfer
real, pointer :: frootn_xfer(:) ! (kgN/m2) fine root N transfer
real, pointer :: livestemn_xfer(:) ! (kgN/m2) live stem N transfer
real, pointer :: deadstemn_xfer(:) ! (kgN/m2) dead stem N transfer
real, pointer :: livecrootn_xfer(:) ! (kgN/m2) live coarse root N transfer
real, pointer :: deadcrootn_xfer(:) ! (kgN/m2) dead coarse root N transfer
While in subroutine CNStressDecidPhenology in the CNPhenologyMod.F90 of original CLM4.5 and CLM4.5 Catchment-CN:
real(r8), pointer :: leafc_xfer(:) ! (gC/m2) leaf C transfer
real(r8), pointer :: frootc_xfer(:) ! (gC/m2) fine root C transfer
real(r8), pointer :: livestemc_xfer(:) ! (gC/m2) live stem C transfer
real(r8), pointer :: deadstemc_xfer(:) ! (gC/m2) dead stem C transfer
real(r8), pointer :: livecrootc_xfer(:) ! (gC/m2) live coarse root C transfer
real(r8), pointer :: deadcrootc_xfer(:) ! (gC/m2) dead coarse root C transfer
real(r8), pointer :: leafn_xfer(:) ! (gN/m2) leaf N transfer
real(r8), pointer :: frootn_xfer(:) ! (gN/m2) fine root N transfer
real(r8), pointer :: livestemn_xfer(:) ! (gN/m2) live stem N transfer
real(r8), pointer :: deadstemn_xfer(:) ! (gN/m2) dead stem N transfer
real(r8), pointer :: livecrootn_xfer(:) ! (gN/m2) live coarse root N transfer
real(r8), pointer :: deadcrootn_xfer(:) ! (gN/m2) dead coarse root N transfer
The units are 1000 times in difference!! Is it just a typo in one of the two?
Checked the CLM4 Catchment-CN code:
leafc_xfer is in gC/m2 in all other routines (e.g. CN_DriverMod.F90, clmtype.F90, CNCStateUpdate1Mod.F90, CNCStateUpdate2Mod.F90, CNCStateUpdate3Mod.F90, CNFireMod.F90, CNHarvestMod.F90, CNiniTimeVar.F90, CNPrecisionControlMod.F90, CNSummaryMod.F90, and CNGapMortalityMod.F90) except CNPhenologyMod.F90 where it's in kgC/m2. And I don't see any unit conversion. Therefore, I think the kgC/m2 in CNPhenologyMod.F90 is a typo.
leafc_xfer is in gC/m2 in all routines in the CLM4.5 Catchment-CN code.
5. Modified mk_LDASsaRestarts.F90 for the new CLM4.5 Catchment-CN restart (the old one was created on 2017-08-09).
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts
mv 0.5D 0.5D_20170809
mkdir 0.5D
cd 0.5D
cp /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 .
nedit mk_LDASsaRestarts.F90 & [Do an xxdiff between the modified one and the original one to see the modifications made.]
integer, parameter :: npft_clm45 = 19 [Use the same vegetation map as CLM4 Catchment-CN]
integer, parameter :: VAR_COL_CLM45 = 35 ! number of CN column restart variables
integer, parameter :: VAR_PFT_CLM45 = 75 ! number of CN PFT variables per column
InCNRestart = '/gpfsm/dnb42/projects/p16/ssd/land/l_data/LandRestarts_for_Regridding/CatchCN/M09/20151231/catchcn_internal_rst', &
InCNTilFile = '/discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til', &
logical, parameter :: clm45 = .true.
SUBROUTINE read_bcs_data:
if(clm45) then
open(unit=29, file=trim(DataDir)//'CLM_veg_typs_fracs',form='formatted')
open(unit=30, file=trim(DataDir)//'CLM4.5_abm_peatf_gdp_hdm_fc' ,form='formatted')
endif
SUBROUTINE regrid_carbon_vars_clm45:
integer :: iclass(npft_clm45) = &
(/1,1,2,3,3,4,5,5,6,7,8,9,10,11,12,11,12,11,12/)
i = 1
do nv = 1,VAR_COL_CLM45
do nz = 1,nzone
STATUS = NF_PUT_VARA_REAL(OutID,VarID(OutID,'CNCOL'), (/1,i/), (/NTILES,1 /),var_col_out(:, nz,nv))
if(nv == 30) STATUS = NF_PUT_VARA_REAL(OutID,VarID(OutID,'CNCOL'), (/1,i/), (/NTILES,1 /),var_col_out(:, nz,37)) ! Move fpg from 37 in CLM4 to 30 in CLM4.5
if(nv == 35) STATUS = NF_PUT_VARA_REAL(OutID,VarID(OutID,'CNCOL'), (/1,i/), (/NTILES,1 /),var_col_out(:, nz,38)) ! Move fpi from 38 in CLM4 to 35 in CLM4.5
i = i + 1
end do
end do
i = 1
do iv = 1,VAR_PFT_CLM45
do nv = 1,nveg
do nz = 1,nzone
if(iv <= 74) then
STATUS = NF_PUT_VARA_REAL(OutID,VarID(OutID,'CNPFT'), (/1,i/), (/NTILES,1 /),var_pft_out(:, nz,nv,iv))
else
if(iv == 75) then ! plant_ndemand
var_dum = 0.
STATUS = NF_PUT_VARA_REAL(OutID,VarID(OutID,'CNPFT'), (/1,i/), (/NTILES,1 /),var_dum)
endif
endif
i = i + 1
end do
end do
end do
==========
20180522:
1. Continued working on 5 on 20180521.
2. Compiled and ran the modified mk_LDASsaRestarts.F90.
(2.1) Compile:
cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
source $ESMADIR/src/g5_modules
gmake install
(2.2) Run mk_LDASsaRestarts:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D
ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
nedit mkLDASsa.j & [Create a job file]
mkLDASsa.j reads:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/csh -f
#SBATCH --account=g0620
#SBATCH --time=1:00:00
#SBATCH --qos=debug
#SBATCH --ntasks=56
#SBATCH --job-name=mkLDAS
#SBATCH --constraint=hasw
#SBATCH --output=mkLDAS.o
#SBATCH --error=mkLDAS.e
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05
source $ESMADIR/src/g5_modules
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
limit stacksize unlimited
setenv SPONSORID g0620
setenv BCSDIR /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/
setenv TILFILE DE_00720x00360_PE_0720x0360.til
mkdir -p OutData1/
mkdir -p OutData2/
ln -s $BCSDIR/$TILFILE OutData1/OutTileFile
ln -s $BCSDIR/$TILFILE OutData2/OutTileFile
ln -s $BCSDIR/clsm OutData2/clsm
mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y
/bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst
mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y
bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50
bin/Scale_CatchCN OutData1/catchcn_internal_clm45 OutData2/catchcn_internal_clm45 catchcn_internal_clm45 50
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
chmod 755 mkLDASsa.j
sbatch mkLDASsa.j
Successfully created catchcn_internal_rst and catchcn_internal_clm45 in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D > diff catchcn_internal_rst ../0.5D_20170809/.
Nothing returned. So the catchcn_internal_rst created is identical to the previous one. Great!!
3. Compared fpg and fpi between catchcn_internal_rst and catchcn_internal_clm45 using /home/fzeng/matlab/plot_catchcn_internal_rst_vars.m.
They are the same between the two. Great!!
4. Modified subroutine CN_init and subroutine CN_exit in CN_DriverMod.F90 to be consistent with /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/mk_LDASsaRestarts.F90.
5. Stripped out from CN_DriverMod.F90 anything related to the restart variables removed (see the list on 20180521).
6. Modified the routines in GEOSlana_GridComp.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
mkdir progCROP
(1) process_cn.F90:
cp -p process_cn.F90 progCROP/.
Stripped out anything related to the restart variables removed (see the list on 20180521).
(2) catch_types.F90:
cp -p catch_types.F90 progCROP/.
Stripped out anything related to the restart variables removed (see the list on 20180521).
And
integer, parameter :: N_cn_progn = 15 + NUM_ZON*VAR_COL + NUM_ZON*NUM_VEG*VAR_PFT + &
3*NUM_ZON + 2*NUM_ZON*NUM_VEG
(3) clsm_ensdrv_init_routines.F90:
cp -p clsm_ensdrv_init_routines.F90 progCROP/.
Stripped out anything related to the restart variables removed (see the list on 20180521).
(4) lsm_calib_routines.F90:
cp -p lsm_calib_routines.F90 progCROP/.
Stripped out anything related to the restart variables removed (see the list on 20180521).
7. Removed the parameters only for prognostic crops from subroutine CN_init:
pftcon%fertnitro
pftcon%lfemerg
pftcon%grnfill
pftcon%mxmat
pftcon%hybgdd
pftcon%laimx
pftcon%ztopmx
pftcon%gddmin
pftcon%graincn
pftcon%fleafcn
pftcon%ffrootcn
pftcon%fstemcn
pftcon%rootprof_beta
pftcon%aleaff
pftcon%allconsl
pftcon%allconss
pftcon%arootf
pftcon%arooti
pftcon%astemf
pftcon%bfact
pftcon%declfact
pftcon%fleafi
pftcon%baset
pftcon%mxtmp
pftcon%planttemp
pftcon%minplanttemp
pftcon%mnNHplantdate
pftcon%mxNHplantdate
pftcon%mnSHplantdate
pftcon%mxSHplantdate
==========
20180523:
1. Continued working on 7 on 20180522.
2. Read through the relevant part of Greg's notes /discover/nobackup/fzeng/notes/cn.txt.20150827 carefully to see if there is anything else I missed.
3. Also compared all the CN files between CLM4 Catchment-CN and original CLM4 to see if there is anything else I missed.
CNAllocationMod.F90
CNAnnualUpdateMod.F90
CNBalanceCheckMod.F90
CNCStateUpdate1Mod.F90
CNCStateUpdate2Mod.F90
CNCStateUpdate3Mod.F90
CNDecompMod.F90
CNEcosystemDynMod.F90
CNGapMortalityMod.F90
CNGRespMod.F90
CNMRespMod.F90
CNNDynamicsMod.F90
CNNStateUpdate1Mod.F90
CNNStateUpdate2Mod.F90
CNNStateUpdate3Mod.F90
CNPhenologyMod.F90
CNPrecisionControlMod.F90
CNSummaryMod.F90
CNVegStructUpdateMod.F90
CNWoodProductsMod.F90
4. Compile with BOPT=g and do test run.
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
source $ESMADIR/src/g5_modules
"gmake clean" and "gmake install BOPT=g" in GEOScatchCN_GridComp, GEOSlana_GridComp, and Applications/LDAS_App
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4.5_debug/.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/run
Check the executable:
ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 99152569 2018-05-23 13:20 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/build/Linux/bin/LDASsaCN_mpi.x [correct!]
Check the restart file:
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_debug/input/restart
rm catchcn_internal_rst
ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/catchcn_internal_clm45 catchcn_internal_rst [This is the one created on 20180522.]
rm output
Modify driver_inputs_0.5D_GLOBAL.nml to run on a small region (82W-77W, 37N-42N):
minlon = -82. ! min longitude
maxlon = -77. ! max longitude
minlat = 37. ! min latitude
maxlat = 42. ! max latitude
Check the job file lenkf.0.i:
-restart_path ../input/restart/ \
Run lenkf.0.i:
/bin/rm ../output/global/rc_out/*
./lenkf.0.i
It's running.
Test it globally next.
5. Contacted Melanie about what new diagnostics we would like to add to Catchment-CN. Answered her questions.
==========
20180524:
1. The regional test run finished 4 years of simulation successfully. Try a 4-year global test run.
Run it on DDT first.
date_time_new 19800101_013000z
p,livestemn(p),livestemn_xfer_to_livestemn(p) 28672 -1.8516766E-05
5.7520826E-15
p,livestemn(p),livestemn_xfer_to_livestemn(p) 52365 -5.1838271E-07
0.0000000E+00
p,livestemn(p),livestemn_xfer_to_livestemn(p) 154373 -1.5335548E-08
0.0000000E+00
p,livestemn(p),livestemn_xfer_to_livestemn(p) 115009 -2.7214586E-05
0.0000000E+00
forrtl: severe (408): fort: (2): Subscript #1 of the array LIVESTEMN_TO_LITTER has value 154373 which is greater than the upper bound of -1
Image PC Routine Line Source
libmpi_usempif08. 00002AAAAC2D2093 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000197D75B cnnstateupdate1mo 590 CNNStateUpdate1Mod.F90
LDASsaCN_mpi.x 0000000001633872 cnecosystemdynmod 211 CNEcosystemDynMod.F90
LDASsaCN_mpi.x 00000000010B24D8 cn_drivermod_mp_c 680 CN_DriverMod.F90
LDASsaCN_mpi.x 0000000000B3C3DF process_cn_mp_pro 1480 process_cn.F90
LDASsaCN_mpi.x 00000000009124A9 ens_driver_routin 375 ens_driver_routines.F90
LDASsaCN_mpi.x 00000000004FACF8 MAIN__ 2045 cnlsm_ensdrv_main.F90
Used /home/fzeng/matlab/plot_catchcn_internal_rst_vars.m to check livestemn in the restart files. It's identical between catchcn_internal_rst and catchcn_internal_clm45 in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D. So livestemn in the restart file catchcn_internal_clm45 is good.
Don't know on which tile(s) this negative livestemn occurred.
Try the one that had this negative livestemn before when prognostic crops were included.
minlon = 62. ! min longitude
maxlon = 62.5 ! max longitude
minlat = 35. ! min latitude
maxlat = 35.5 ! max latitude
This tile has NO negative livestemn now. It was able to run!
Need to add tile id to the routines to print out the problematic tile id!
date_time_new 19800101_013000z
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 61797
11 -1.8516766E-05 5.7520826E-15
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 0
4 -5.1838271E-07 0.0000000E+00
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 0
12 -1.5335548E-08 0.0000000E+00
forrtl: severe (408): fort: (2): Subscript #1 of the array LIVESTEMN_TO_LITTER has value 28672 which is greater than the upper bound of -1
Image PC Routine Line Source
libmpi_usempif08. 00002AAAAC2D2093 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000197EB1F cnnstateupdate1mo 591 CNNStateUpdate1Mod.F90
LDASsaCN_mpi.x 00000000016344DB cnecosystemdynmod 212 CNEcosystemDynMod.F90
Why do I get 0 tileid?
Tile 61797:
61797 148057 12 12 13 13 58.76 0.00 41.24 0.00 12 13 58.76 41.24
There is no veg type 11 in tile 61797 or the ones next to it.
Changed "tileid_soilp(num_soilp) = tile_id(nc)" to "tileid_soilp(p) = tile_id(nc)" in CN_DriverMod.F90 and it is working now.
date_time_new 19800101_013000z
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11814
11 -1.8516766E-05 5.7520826E-15
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 13284
4 -5.1838271E-07 0.0000000E+00
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 40341
8 -2.7214586E-05 0.0000000E+00
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 66701
12 -1.5335548E-08 0.0000000E+00
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 45542
2 -1.5835078E-06 0.0000000E+00
forrtl: severe (408): fort: (2): Subscript #1 of the array LIVESTEMN_TO_LITTER has value 169063 which is greater than the upper bound of -1
Image PC Routine Line Source
libmpi_usempif08. 00002AAAAC2D2093 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000197E9DF cnnstateupdate1mo 591 CNNStateUpdate1Mod.F90
Investigate this one:
tileid_soilp(p),ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 13284
4 -5.1838271E-07 0.0000000E+00
CLM_veg_typs_frac:
13284 100697 4 4 18 19 63.02 0.00 0.00 36.98 5 16 63.02 36.98
catchment.def:
13284 100697 29.0000 29.5000 -1.5000 -1.0000 1609.2594
minlon = 29. ! min longitude
maxlon = 29.5 ! max longitude
minlat = -1.5 ! min latitude
maxlat = -1. ! max latitude
In the restart file, livestemn (gN/m2) is as follow for this tile.
type 4 type 19
zone 1: 2.23922729 0
zone 2: 1.80735159 0
zone 3: -5.18382592e-07 0
/home/fzeng/matlab/plot_catchcn_internal_rst_vars.m shows that these two below are identical in grid level between catchcn_internal_rst and catchcn_internal_clm45 in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D:
livestemn: from 0.0 to 2.3933 gN/m2.
cpool: can be negative, generally from -0.8886 to 15.6462 gC/m2. Close to 0 so it's fine.
However, not sure about pft-level values of these two.
Does this mean that livestemn could also go negative in CLM4 Catchment-CN? Try to run CLM4 Catchment-CN on the this tile?
==========
20180525:
1. Modified and ran read_catchcn_internal_rst.m and plot_catchcn_internal_rst_vars.m in /home/fzeng/matlab. Found negative livestemn in both CLM4 and CLM4.5 restart files (same tile, zone and ityp between CLM4 and CLM4.5):
Negative PFT level variable in tile id, nz, ityp: 1077 3 10
Negative PFT level variable in tile id, nz, ityp: 1077 3 11
Negative PFT level variable in tile id, nz, ityp: 1141 3 11
Negative PFT level variable in tile id, nz, ityp: 2118 2 11
Negative PFT level variable in tile id, nz, ityp: 3407 3 11
Negative PFT level variable in tile id, nz, ityp: 4067 3 11
Negative PFT level variable in tile id, nz, ityp: 4076 3 11
Negative PFT level variable in tile id, nz, ityp: 5851 3 11
Negative PFT level variable in tile id, nz, ityp: 5851 3 6
Negative PFT level variable in tile id, nz, ityp: 6400 2 6
Negative PFT level variable in tile id, nz, ityp: 6733 3 11
Negative PFT level variable in tile id, nz, ityp: 6918 3 11
Negative PFT level variable in tile id, nz, ityp: 7037 3 11
Negative PFT level variable in tile id, nz, ityp: 7134 3 6
Negative PFT level variable in tile id, nz, ityp: 7232 3 11
Negative PFT level variable in tile id, nz, ityp: 7917 3 4
Negative PFT level variable in tile id, nz, ityp: 11814 3 11
Negative PFT level variable in tile id, nz, ityp: 12833 3 11
Negative PFT level variable in tile id, nz, ityp: 13257 3 11
Negative PFT level variable in tile id, nz, ityp: 13284 3 4
Negative PFT level variable in tile id, nz, ityp: 13895 3 11
Negative PFT level variable in tile id, nz, ityp: 14435 3 11
Negative PFT level variable in tile id, nz, ityp: 17293 2 11
Negative PFT level variable in tile id, nz, ityp: 18739 3 6
Negative PFT level variable in tile id, nz, ityp: 19880 2 6
Negative PFT level variable in tile id, nz, ityp: 20359 3 6
Negative PFT level variable in tile id, nz, ityp: 21887 3 5
Negative PFT level variable in tile id, nz, ityp: 22940 2 6
Negative PFT level variable in tile id, nz, ityp: 22948 3 5
Negative PFT level variable in tile id, nz, ityp: 22989 3 1
Negative PFT level variable in tile id, nz, ityp: 23788 2 11
Negative PFT level variable in tile id, nz, ityp: 23788 3 11
Negative PFT level variable in tile id, nz, ityp: 24357 3 4
Negative PFT level variable in tile id, nz, ityp: 25015 3 1
Negative PFT level variable in tile id, nz, ityp: 25254 3 1
Negative PFT level variable in tile id, nz, ityp: 25255 3 5
Negative PFT level variable in tile id, nz, ityp: 25337 3 6
Negative PFT level variable in tile id, nz, ityp: 26902 3 1
Negative PFT level variable in tile id, nz, ityp: 27223 3 1
Negative PFT level variable in tile id, nz, ityp: 29845 3 1
Negative PFT level variable in tile id, nz, ityp: 29858 3 10
Negative PFT level variable in tile id, nz, ityp: 30166 3 11
Negative PFT level variable in tile id, nz, ityp: 30969 3 1
Negative PFT level variable in tile id, nz, ityp: 31245 2 11
Negative PFT level variable in tile id, nz, ityp: 31801 3 11
Negative PFT level variable in tile id, nz, ityp: 32103 2 1
Negative PFT level variable in tile id, nz, ityp: 32192 3 11
Negative PFT level variable in tile id, nz, ityp: 33489 3 2
Negative PFT level variable in tile id, nz, ityp: 33825 2 1
Negative PFT level variable in tile id, nz, ityp: 34398 1 11
Negative PFT level variable in tile id, nz, ityp: 34542 3 2
Negative PFT level variable in tile id, nz, ityp: 34900 3 2
Negative PFT level variable in tile id, nz, ityp: 35262 3 2
Negative PFT level variable in tile id, nz, ityp: 35610 3 2
Negative PFT level variable in tile id, nz, ityp: 40341 3 8
Negative PFT level variable in tile id, nz, ityp: 45113 3 8
Negative PFT level variable in tile id, nz, ityp: 45542 3 2
Negative PFT level variable in tile id, nz, ityp: 65516 3 12
Negative PFT level variable in tile id, nz, ityp: 66701 3 12
Negative PFT level variable in tile id, nz, ityp: 67284 3 8
Negative PFT level variable in tile id, nz, ityp: 67287 3 8
Negative PFT level variable in tile id, nz, ityp: 67288 3 8
Double checked:
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts > diff 0.5D/catchcn_internal_rst 0.5D_20170809/catchcn_internal_rst
Returned nothing --> so the new restart file I created a few days ago is identical to the one I used for the CLM4 spinup run.
Want to check cpool in the restart files from the CLM4 Catchment-CN runs.
Modified and ran read_catchcn_internal_rst.m and plot_catchcn_internal_rst_vars.m in /home/fzeng/matlab to plot elai (as a test, cpool is the one I want to plot later) in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_42/output/global/rs/ens0000/Y2017/M01/clm4_DE720.ens0000.catchcn_ldas_rst.20170101_0000z. The tile id is 0 everywhere in this file, so need to read tile id from /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_42/output/global/rc_out/clm4_DE720.ldas_domain.txt. The tile id is needed to re-arrange the data so that they are in ascending tile id order. However, the global elai plotted doesn't look correct. Is there anything wrong with my re-ordering?
==========
20180529:
1. Created /home/fzeng/Catchment/read_CN_restart.f90 to read and print out variables in CN_restart in both /discover/nobackup/fzeng/Catchment/princeton/p0005s_63_1948 and /discover/nobackup/elee15/offline/sims/co2var_spinup/M2.n5P.HRv2.280ppm.c39. Both CN_restart files have negative livestemn and huge cpool values in a lot of tiles!!
2. Added this print statement to CNNStateUpdate1Mod.F90 in CLM4 Catchment-CN and compiled.
if (livestemn(p)<0) then
print *, 'ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p)',ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p)
stop
end if
Compiled:
"gmake clean" and "gmake install" in GEOScatchCN_GridComp, and then GEOSlana_GridComp. Error message when compiling GEOSlana_GridComp.
../process_route.F90(497): error #6784: The number of actual arguments cannot be greater than the number of dummy arguments. [RIVER_ROUTING]
CALL RIVER_ROUTING (N_Active,ROUTE_DT, RUNOFF,AREACAT,LENGSC, &
------------^
../process_route.F90(497): error #6633: The type of the actual argument differs from the type of the dummy argument. [3600]
CALL RIVER_ROUTING (N_Active,ROUTE_DT, RUNOFF,AREACAT,LENGSC, &
-------------------------------------^
../process_route.F90(497): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic. [RUNCATCH]
CALL RIVER_ROUTING (N_Active,ROUTE_DT, RUNOFF,AREACAT,LENGSC, &
------------^
../process_route.F90(1370): error #6784: The number of actual arguments cannot be greater than the number of dummy arguments. [RIVER_ROUTING]
CALL RIVER_ROUTING (N_Pfaf_D, ROUTE_INTERVAL, &
---------------------^
../process_route.F90(1370): error #6633: The type of the actual argument differs from the type of the dummy argument. [3600]
CALL RIVER_ROUTING (N_Pfaf_D, ROUTE_INTERVAL, &
------------------------------------------------^
../process_route.F90(1370): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic. [RUNCATCH]
CALL RIVER_ROUTING (N_Pfaf_D, ROUTE_INTERVAL, &
---------------------^
compilation aborted for ../process_route.F90 (code 1)
gmake[1]: *** [process_route.o] Error 1
gmake[1]: Leaving directory `/gpfsm/dnb31/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/mpi'
gmake: *** [install] Error 2
Had to copy routing_model.F90 (then compiled GEOSroute_GridComp) and process_route.F90 (then compiled GEOSlana_GridComp) from clm4.5 to make it work. These two files are newer in clm4.5. Old copies were kept.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOSroute_GridComp > cp -p /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOSroute_GridComp/routing_model.F90 .
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > cp -p /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/process_route.F90 .
Then "gmake clean" and "gmake install" in Applications/LDAS_App.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec > mv clm4 clm4_orig
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec > mkdir clm4
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4/.
3. Continued to run clm4_DE720 and see if there is negative livestemn.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720
ls -l build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 69968545 2018-02-27 11:49 build/Linux/bin/LDASsaCN_mpi.x [Wrong!]
Why is it not updated? It's because I forgot to setenv to /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3, so the executable was copied to /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 and messed up the CLM4.5 executable.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > source $ESMADIR/src/g5_modules
"gmake clean" and "gmake install" in GEOScatchCN_GridComp, GEOSlana_GridComp, and then Applications/LDAS_App.
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4/.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720
ls -l build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70744331 2018-05-29 16:51 build/Linux/bin/LDASsaCN_mpi.x* [Correct!]
discover18:/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720 > ls -l output/global/rs/ens0000/Y2003/M11
total 284640
-rw-r--r-- 1 fzeng g0620 291445863 2018-04-14 09:40 clm4_DE720.ens0000.catchcn_ldas_rst.20031101_0000z
discover18:/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720 > ls -l output/global/rs/ens0000/Y2003/M12
total 0
-rw-r--r-- 1 fzeng g0620 48 2018-04-14 09:49 clm4_DE720.ens0000.catchcn_ldas_rst.20031201_0000z
The clm4_DE720 stopped in mid April due to disk space exceeded.
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720/run
cp lenkf.2.j lenkf.2b.j
nedit lenkf.2b.j &
qsub lenkf.2b.j
date_time_new 20031101_013000z
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 4 -2.0746938E-05
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 12 -8.6905159E-09
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -1.3411276E-10
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -3.0819955E-09
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -1.8086140E-10
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -1.7981962E-06
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 2 -4.3481343E-08
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 10 -6.7742144E-07
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 8 -4.4882492E-07
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -2.2807149E-06
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 2 -2.4904669E-08
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 1 -1.2151677E-08
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -8.3364461E-07
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 7 -1.4454433E-08
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -1.6135593E-06
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -1.2544183E-09
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 10 -5.2325369E-05
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 1 -9.0889893E-07
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 7 -5.8316263E-10
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 11 -3.8626004E-05
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 1 -5.8726208E-08
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 10 -4.5838502E-09
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 10 -8.7907773E-07
0.0000000E+00
ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 2 -1.1419068E-08
0.0000000E+00
...
==========
20180530:
1. Checked livestemn and cpool in the CN_restart in one of Greg's runs.
cd /discover/nobackup/fzeng/Catchment/princeton
cp /archive/u/gkwalker/Catchment/princeton/p0005s_38.tgz .
tar -xf p0005s_38.tgz
/bin/rm p0005s_38.tgz
mv p0005s_38 p0005s_38_gkw
Modified /home/fzeng/Catchment/read_CN_restart.f90 to read the CN_restart in /discover/nobackup/fzeng/Catchment/princeton/p0005s_38_gkw and print out negative livestemn and large cpool values.
livestemn:
/discover/nobackup/fzeng/Catchment/princeton/p0005s_38_gkw/
n,nz,nv,value: 4369 2 3 -2.0210076E-05
n,nz,nv,value: 8861 3 1 -5.7664074E-10
n,nz,nv,value: 11808 3 1 -1.9410731E-10
n,nz,nv,value: 12705 3 1 -6.8678045E-12
n,nz,nv,value: 12710 3 1 -3.5061887E-10
n,nz,nv,value: 12711 3 1 -1.0238758E-10
n,nz,nv,value: 12991 3 1 -3.4913728E-11
n,nz,nv,value: 13045 3 1 -2.3386235E-10
n,nz,nv,value: 13435 3 3 -1.0407103E-05
n,nz,nv,value: 13444 3 3 -4.7770150E-06
n,nz,nv,value: 13447 3 3 -1.0029362E-05
n,nz,nv,value: 14128 3 1 -1.9675260E-11
n,nz,nv,value: 14746 3 3 -1.3701764E-08
n,nz,nv,value: 22492 1 4 -1.8794437E-06
n,nz,nv,value: 22493 1 4 -6.2009026E-06
n,nz,nv,value: 22496 1 4 -6.0499265E-06
n,nz,nv,value: 22506 1 4 -8.0085421E-08
n,nz,nv,value: 25434 1 4 -3.2802660E-07
n,nz,nv,value: 25627 1 4 -5.8760691E-07
n,nz,nv,value: 25629 1 4 -1.7946679E-06
cpool:
/discover/nobackup/fzeng/Catchment/princeton/p0005s_38_gkw/
n,nz,nv,value: 66535 2 2 245.6748
n,nz,nv,value: 66535 3 4 96868.35
n,nz,nv,value: 66536 2 3 273.4533
n,nz,nv,value: 66538 2 2 98001.55
n,nz,nv,value: 66538 2 4 280.9594
n,nz,nv,value: 66538 3 2 243.7362
n,nz,nv,value: 66542 2 4 97499.25
n,nz,nv,value: 66542 3 4 281.0563
n,nz,nv,value: 66544 1 3 264.8176
n,nz,nv,value: 66546 2 2 159.9011
n,nz,nv,value: 66548 1 3 89701.70
n,nz,nv,value: 66560 2 4 284.7641
n,nz,nv,value: 66561 3 2 271.5475
n,nz,nv,value: 66563 1 3 235.2253
n,nz,nv,value: 66563 3 2 96368.21
n,nz,nv,value: 66563 3 4 280.1057
n,nz,nv,value: 66565 2 2 252.0963
n,nz,nv,value: 66565 3 4 97040.46
n,nz,nv,value: 66567 2 2 99094.14
n,nz,nv,value: 66567 2 4 280.9594
n,nz,nv,value: 66567 3 2 254.4470
n,nz,nv,value: 66569 2 4 97499.25
n,nz,nv,value: 66569 3 4 289.5081
n,nz,nv,value: 66570 1 3 264.8176
n,nz,nv,value: 66571 2 2 186.5992
n,nz,nv,value: 66572 1 3 89701.70
==========
20180605:
1. Tried to figure out how to run the original CLM4 and see if there is the negative livestemn issue.
Followed Greg's /discover/nobackup/fzeng/clm_orig/ccsm4_0/steps. Also needed to retrieve /archive/u/gkwalker/ccsm4_0/ccsm4_0.tar and follow the modifications Greg made to set up the run.
module load comp/intel-15.0.2.164
module load lib/mkl-15.0.2.164
module load other/mpi/openmpi/1.8.4-intel-15.0.2.164
> cd /discover/nobackup/fzeng/clm_orig/ccsm4_0/scripts
> setenv CCSMROOT /discover/nobackup/fzeng/clm_orig/ccsm4_0
> create_newcase --list [Can skip this.]
> mkdir -p /discover/nobackup/fzeng/clm_orig/ccsm4_0/data
> create_newcase -case ICN_f19_g16 -mach generic_linux_intel -compset ICN -res f19_g16 -scratchroot /discover/nobackup/fzeng/clm_orig/ccsm4_0/run -din_loc_root_csmdata /discover/nobackup/fzeng/clm_orig/ccsm4_0/data -max_tasks_per_node 8
Note: ICN will build dynamic land model with CN physics, data atmosphere
> cd ICN_f19_g16
> nedit env_mach_pes.xml &
L61:
L65:
L69:
L73:
L77:
L81:
L92:
L94:
L96:
L98:
> nedit env_run.xml &
L21:
L40:
L209:
> nedit Macros.generic_linux_intel &
NETCDF_PATH := /usr/local/other/netcdf/3.6.2_intel-9.1.042
INC_NETCDF := $(NETCDF_PATH)/include
LIB_NETCDF := $(NETCDF_PATH)/lib
MOD_NETCDF := $(NETCDF_PATH)/include
MPICH_PATH := /usr/local/intel/mpi/3.2.011
INC_MPI := $(MPICH_PATH)/include64
LIB_MPI := $(MPICH_PATH)/lib64
MPI_LIB_NAME := mpich
PNETCDF_PATH :=
INC_PNETCDF :=
LIB_PNETCDF :=
LAPACK_LIBDIR :=
> ./configure -case
> mkdir -p /discover/nobackup/fzeng/clm_orig/ccsm4_0/data/ccsm4_init/I2000CN_f19_g16_c100305/0001-01-01
> cd /discover/nobackup/fzeng/clm_orig/ccsm4_0/data/ccsm4_init/I2000CN_f19_g16_c100305
> svn export --force https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ccsm4_init/I2000CN_f19_g16_c100305/0001-01-01
#> setenv NETCDF_PATH /discover/swdev/mathomp4/Baselibs/ESMA-Baselibs-5.0.6/x86_64-unknown-linux-gnu/ifort_18.0.1.163-intelmpi_18.0.1.163/Linux
It downloaded I2000CN_f19_g16_c100305.clm2.r.0001-01-01-00000.nc to /discover/nobackup/fzeng/clm_orig/ccsm4_0/data/ccsm4_init/I2000CN_f19_g16_c100305/0001-01-01.
> cd /discover/nobackup/fzeng/clm_orig/ccsm4_0/scripts/ICN_f19_g16
> ICN_f19_g16.generic_linux_intel.build
It downloaded the necessary initial condition files.
I keep getting this error message:
cp: cannot stat `Makefile.conf.old': No such file or directory
cp: cannot stat `Makefile.conf': No such file or directory
Makefile:25: Makefile.conf: No such file or directory
gmake: *** No rule to make target `Makefile.conf'. Stop.
Running ./configure is supposed to create Makefile.conf according to models/utils/pio/README.config. Why don't I see Makefile.conf created after running ./configure?
I compared all the files below between /discover/nobackup/fzeng/clm_orig/ccsm4_0/scripts/ICN_f19_g16 and /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts/test.
check_input_data*
configure*
create_production_test*
env_build.xml
env_case.xml
env_conf.xml
env_derived
env_mach_pes.xml
env_mach_specific*
env_run.xml
Macros.generic_linux_intel
README.case
xmlchange*
ICN_f19_g16.generic_linux_intel.build*
ICN_f19_g16.generic_linux_intel.clean_build*
ICN_f19_g16.generic_linux_intel.run*
But could not find what I missed that could cause the Makefile.conf issue.
Tried this below but didn't work.
> nedit env_mach_specific &
if (-e /usr/share/modules/init/csh) then
source /usr/share/modules/init/csh
module purge
module load comp/intel-18.0.0.128
module load mpi/impi-5.1.3.258
module load lib/mkl-18.0.0.128
module load other/cmake-3.8.2
endif
limit stacksize unlimited
set path=($path . $HOME/bin /usr/bin /usr/local/bin $SHARE/dasilva/opengrads/Contents)
set ARCH = `uname -s`
setenv BASEDIR5 /discover/nobackup/projects/gmao/share/gmao_ops/Baselibs/v4.0.3_build1/x86_64-unknown-linux-gnu/ifort_13.1.2.183-mvapich2_1.8.1
setenv BASEDIR $BASEDIR5
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${BASEDIR}/${ARCH}/lib:/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3/lib
setenv PATH $PATH":/usr/local/other/SLES11.1/netcdf4/4.1.3/intel-12.1.0.233/bin"
setenv NETCDF_PATH /discover/swdev/mathomp4/Baselibs/ESMA-Baselibs-5.0.6/x86_64-unknown-linux-gnu/ifort_18.0.1.163-intelmpi_18.0.1.163/Linux
==========
20180606:
1. Continued working on 1 on 20180605 above.
==========
20180607:
1. Continued working on 1 on 20180605 above.
Checked config_pes.xml and config_machines.xml. They are identical between /discover/nobackup/fzeng/clm_orig/ccsm4_0/scripts/ccsm_utils/Machines and /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts/ccsm_utils/Machines.
Files under ccsm_utils/Build, ccsm_utils/Components, and ccsm_utils/Tools are also identical between mine and Greg's.
Tomorrow: try to set up a new case under /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts and see if it can build.
==========
20180608:
1. Tried to set up a new case under /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts.
> module load comp/intel-9.1.042
comp/intel-9.1.042(12):ERROR:150: Module 'comp/intel-9.1.042' conflicts with the currently loaded module(s) 'comp/intel-15.0.3.187'
comp/intel-9.1.042(12):ERROR:102: Tcl command execution failed: conflict comp
> module load lib/mkl-9.1.023
> module load mpi/impi-3.2.011
ModuleCmd_Load.c(208):ERROR:105: Unable to locate a modulefile for 'mpi/impi-3.2.011'
> cd /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts
> setenv CCSMROOT /discover/nobackup/fzeng/gkwalker/ccsm4_0
> mkdir -p /discover/nobackup/fzeng/gkwalker/ccsm4_0/data [Skipped. Already there.]
> create_newcase -case ICN01 -mach generic_linux_intel -compset ICN -res f19_g16 -scratchroot /discover/nobackup/fzeng/gkwalker/ccsm4_0/run -din_loc_root_csmdata /discover/nobackup/fzeng/gkwalker/ccsm4_0/data -max_tasks_per_node 8
> cd ICN01
> mv env_mach_pes.xml env_mach_pes.xml.orig
> cp ../test/env_mach_pes.xml .
> configure -case
> setenv NETCDF_PATH /usr/local/other/netcdf/3.6.2_intel-9.1.042
> mv Macros.generic_linux_intel Macros.generic_linux_intel.orig
> cp ../test/Macros.generic_linux_intel .
> nedit env_run.xml &
L21:
L40:
L209:
> ICN01.generic_linux_intel.build
Got the same error message:
cp: cannot stat `Makefile.conf': No such file or directory
Makefile:25: Makefile.conf: No such file or directory
gmake: *** No rule to make target `Makefile.conf'. Stop.
2. Tried search for solutions online but only found the solutions for the same issue in cesm (not ccsm).
3. Tried starting from very beginning -- check out the code.
> cd /discover/nobackup/fzeng/clm_orig
> mv ccsm4_0 ccsm4_0_gkw # This ccsm4_0 was copied from Greg's directory about 3 years ago.
> svn co https://svn-ccsm-release.cgd.ucar.edu/model_versions/ccsm4_0
> t # temporally accept
> module load comp/intel-9.1.042 # This is what Greg did
comp/intel-9.1.042(12):ERROR:150: Module 'comp/intel-9.1.042' conflicts with the currently loaded module(s) 'comp/intel-15.0.2.164'
comp/intel-9.1.042(12):ERROR:102: Tcl command execution failed: conflict comp
> module load lib/mkl-9.1.023
> module load mpi/impi-3.2.011 # This is what Greg did
ModuleCmd_Load.c(208):ERROR:105: Unable to locate a modulefile for 'mpi/impi-3.2.011'
> cd ccsm4_0
> setenv CCSMROOT /discover/nobackup/fzeng/clm_orig/ccsm4_0
> mkdir /discover/nobackup/fzeng/clm_orig/ccsm4_0/data
> cd scripts
> create_newcase -case ICN01 -mach generic_linux_intel -compset ICN -res f19_g16 -scratchroot /discover/nobackup/fzeng/clm_orig/ccsm4_0/run -din_loc_root_csmdata /discover/nobackup/fzeng/clm_orig/ccsm4_0/data -max_tasks_per_node 8
> mkdir -p /discover/nobackup/fzeng/clm_orig/ccsm4_0/data/ccsm4_init/I2000CN_f19_g16_c100305/0001-01-01
> cd /discover/nobackup/fzeng/clm_orig/ccsm4_0/data/ccsm4_init/I2000CN_f19_g16_c100305
> svn export --force https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ccsm4_init/I2000CN_f19_g16_c100305/0001-01-01
> cd /discover/nobackup/fzeng/clm_orig/ccsm4_0/scripts/ICN01
> mv env_mach_pes.xml env_mach_pes.xml.orig
> cp /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts/test/env_mach_pes.xml .
> nedit env_run.xml &
L21:
L40:
L209:
> mv Macros.generic_linux_intel Macros.generic_linux_intel.orig
> cp /discover/nobackup/fzeng/gkwalker/ccsm4_0/scripts/test/Macros.generic_linux_intel .
> configure -case
> ICN01.generic_linux_intel.build
Error:
using NETCDF_PATH from environment
configure: error: netcdf.mod not found in NETCDF_PATH/include check the environment variable NETCDF_PATH
gmake: *** [configure] Error 1
cp: cannot stat `Makefile.conf': No such file or directory
Makefile:25: Makefile.conf: No such file or directory
gmake: *** No rule to make target `Makefile.conf'. Stop.
==========
20180613:
1. Emailed Matt asking for help to make the original CLM4 (ccsm4_0) run on Discover.
Met with Matt and described the issue with running ccsm4_0 on Discover. He said he will check out the code and try it himself.
2. Meanwhile, trying to run the CLM4 in cesm1_2_2.
Justin's email on 6/6/2018:
-----------------
The easiest way would be to use the same cesm1.2 code you have, but a compset that uses uses clm4.0 instead of clm4.5.
http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/modelnl/compsets.html
However, I thing all the compsets with bgc are clm4.5 only. You can force clm4.0 physics by making sure CLM CONFIG OPTS ="phys clm4 " by xmlchange or otherwise in env_build.xml before you build.
-----------------
On http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/modelnl/compsets.html, this is the compset Greg used:
Compset Longname and description Short Name Alias
-------------------------------------------------------------------------
2000_DATM%QIA_CLM40%CN_SICE_SOCN_RTM_SGLC_SWAV I_2000_CN ICN
DATM:
CLM:
RTM:
SICE:
SOCN:
SGLC:
SWAV:
present day:
clm4.0 physics:
clm4.0 cn:
QIAN atm input data for 1972-2004:
Created a new case using compset ICN: see my notes on 20180308.
Added this to /discover/nobackup/fzeng/clm_orig/cesm1_2_2/models/lnd/clm/src/clm4_0/biogeochem/CNNStateUpdate1Mod.F90:
if (livestemn(p)<0) then
print *, 'p,livestemn(p),livestemn_xfer_to_livestemn(p)',p,livestemn(p),livestemn_xfer_to_livestemn(p)
print *, 'livestemn_to_litter(p),livestemn_to_retransn(p)',livestemn_to_litter(p),livestemn_to_retransn(p)
print *, 'npool_to_livestemn(p)',npool_to_livestemn(p)
stop
end if
NOTE: clm4_0 in cesm1_2_2 operationally supports prognostic crops while ccsm4_0 does not. This can be seen in CNNStateUpdate1Mod.F90.
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts
create_newcase -case ICN_01 -res f19_g16 -compset ICN -mach discover
cd ICN_01
./cesm_setup
./ICN_01.build # This step downloads a lot of files needed for the simulations. It took a long time.
# This "export" instruction below was printed out on screen when I tried to do an iterative run. Did it and rebuilt.
> mkdir -p /discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/ccsm4_init/I2000CN_f19_g16_c100503/0001-01-01
> cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/ccsm4_init/I2000CN_f19_g16_c100503/0001-01-01
> cd ..
> svn export --force https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ccsm4_init/I2000CN_f19_g16_c100503/0001-01-01
./ICN_01.build
Modified the env_run.xml to run from 1901 throught 1920:
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICN_01
./xmlchange RUN_STARTDATE='1980-01-01',STOP_OPTION=nyears,STOP_N=20,RESUBMIT=5,DATM_CLMNCEP_YR_ALIGN=1980,CCSM_CO2_PPMV=390
Run it interactively:
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICN_01
./ICN_01.run
It's running. Submitted the full job.
sbatch ICN_01.run
The run stopped when it's simulating 198006 due to negative livestemn (see /discover/nobackup/fzeng/clm_orig/cesm1_2_2/scripts/ICN_01/run/cesm.log.180614-100735):
p,ivt(p),livestemn(p),livestemn_xfer_to_livestemn(p) 8037 4
-1.913666074285143E-005 0.000000000000000E+000
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
cesm.exe 00000000010326CD Unknown Unknown Unknown
libpthread-2.11.3 00002AAAB11CD850 Unknown Unknown Unknown
cesm.exe 00000000006A9919 cnnstateupdate1mo 534 CNNStateUpdate1Mod.F90
cesm.exe 0000000000691EA6 cnecosystemdynmod 186 CNEcosystemDynMod.F90
cesm.exe 00000000004D1B29 clm_driver_mp_clm 604 clm_driver.F90
cesm.exe 00000000004A9C6E lnd_comp_mct_mp_l 589 lnd_comp_mct.F90
cesm.exe 0000000000423513 ccsm_comp_mod_mp_ 3281 ccsm_comp_mod.F90
cesm.exe 00000000004421E8 MAIN__ 91 ccsm_driver.F90
cesm.exe 000000000041FE8E Unknown Unknown Unknown
libc-2.11.3.so 00002AAAB13F9C36 __libc_start_main Unknown Unknown
cesm.exe 000000000041FD99 Unknown Unknown Unknown
The PFT physiology data used is /discover/nobackup/fzeng/clm_orig/cesm1_2_2/data/lnd/clm2/pftdata/pft-physiology.clm40.c130424.nc.
In this file, there are 21 PFTs.
pftnum: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
pftname =
"not_vegetated ",
"needleleaf_evergreen_temperate_tree ",
"needleleaf_evergreen_boreal_tree ",
"needleleaf_deciduous_boreal_tree ",
"broadleaf_evergreen_tropical_tree ",
"broadleaf_evergreen_temperate_tree ",
"broadleaf_deciduous_tropical_tree ",
"broadleaf_deciduous_temperate_tree ",
"broadleaf_deciduous_boreal_tree ",
"broadleaf_evergreen_shrub ",
"broadleaf_deciduous_temperate_shrub ",
"broadleaf_deciduous_boreal_shrub ",
"c3_arctic_grass ",
"c3_non-arctic_grass ",
"c4_grass ",
"c3_crop ",
"c3_irrigated ",
"corn ",
"spring_temperate_cereal ",
"winter_temperate_cereal ",
"soybean " ;
}
So ivt 4 is broadleaf_evergreen_tropical_tree.
Why does ICN include prognostic CROP?
==========
20180815:
1. Discover is down for maintenance all day.
2. Met with Melanie. I will try this below in CNFireMod.F90 to see if it would improve burned area:
Current:
fuelc(c) = totlitc(c)+totvegc_col(c)-rootc_col(c)-fuelc_crop(c)*cropf_col(c)
Change to this below:
fuelc(c) = totlitc(c)+totvegc_col(c)-rootc_col(c)-fuelc_crop(c)*cropf_col(c)-cpool_col(c)
Asked Weiyuan:
Is the speed of
running it linearly related to the number of processes used? For
example, if we double the number of processes, would it double the speed
of running it (or reduce the run time to about half)?
Weiyuan's reply:
In ideal world, the speed should double if the number of processes doubles. But I don't think GEOSldas gets that scalability. The main issue here is the input or reading forcing. In GEOSldas, one PE reads and shares with the other PEs among the same node. The reading is sequential and has not been parallelized ( As far as I know, parallel reading is not effective either. We will use Extdata in the future) . When the number of process is below 112 ( it depends on the grid resolution though, higher resolution, bigger the number), you can see very good scalability. If you keep increasing the number of processes, you can still get the improvement, but not that significant.
==========
20180816:
1. Modified CNFireMod.F90:
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> cp -p CNFireMod.F90 CNFireMod.F90.20180124
> nedit CNFireMod.F90&
Change L648 from
fuelc(c) = totlitc(c)+totvegc_col(c)-rootc_col(c)-fuelc_crop(c)*cropf_col(c)
to
fuelc(c) = totlitc(c)+totvegc_col(c)-rootc_col(c)-fuelc_crop(c)*cropf_col(c)-cpool_col(c)
Search for cpool for more modifications.
Also modified these files below to define, initiate and compute cpool_col (following what's done for leafc):
clmtype.F90
clmtypeInitMod.F90
CNSetValueMod.F90
Search for cpool_col for the modifications.
2. Add 2 diagnostics: nfire and somc_fire
Files modified:
CN_DriverMod.F90 [to aggregate nfire and somc_fire from column-level to grid-level]
clsm_ensdrv_out_routines.F90
catch_types.F90 [update 'N_cn_diagn' and follow 'closs']
process_cn.F90 [follow 'closs']
clsm_ensdrv_glob_param.F90 [change 'N_CatchCN_OutFields = 35' to 'N_CatchCN_OutFields = 37']
After adding a new diagnostic, the model should be rebuilt from a clean state:
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src
> setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
> source $ESMADIR/src/g5_modules
> gmake realclean
> gmake install |& tee make.install.log
3. Run one cycle of clm4.5_DE720 (the 53rd cycle):
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720
> ls -l build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70928420 2018-08-16 13:18 build/Linux/bin/LDASsaCN_mpi.x*
> ls -l input/restart/
lrwxrwxrwx 1 fzeng g0620 80 2018-08-16 13:24 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_52/output/
Did an interactive run:
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> ./lenkf.0.j
It's running. Stopped it at date_time_new 19800101_040730z.
> qsub lenkf.0.j
==========
20180820:
1. The clm4.5_DE720 stopped in simulating 19840828.
*log*:
date_time_new 19840828_073730z
column cbalance error = -0.1162157 614
Latdeg,Londeg= -34.75000 138.2500
begcb = 540730.9
endcb = 540731.1
delta store = 0.1875000
*out*:
CBalance
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libirc.so 00002AAAAB590961 Unknown Unknown Unknown
libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2B2B73 Unknown Unknown Unknown
libc.so.6 00002AAAABA16910 Unknown Unknown Unknown
libc.so.6 00002AAAABAB4BBD Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000016B93EA Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000016A9E19 Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000016975B5 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000169A185 Unknown Unknown Unknown
LDASsaCN_mpi.x 0000000001871DB6 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000165FD09 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000164168B Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000186C2A7 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000186C386 Unknown Unknown Unknown
LDASsaCN_mpi.x 0000000001873C06 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000187B6AA Unknown Unknown Unknown
LDASsaCN_mpi.x 0000000001875E9E Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000016727F6 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000157E6B8 Unknown Unknown Unknown
LDASsaCN_mpi.x 0000000001590154 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000159023A Unknown Unknown Unknown
LDASsaCN_mpi.x 0000000001532487 Unknown Unknown Unknown
LDASsaCN_mpi.x 000000000153397E Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000014B6C2F Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000014A6724 gfio_getvar1_.R 2365 gfio___.F
LDASsaCN_mpi.x 00000000014A85C7 gfio_getvar_ 2044 gfio___.F
LDASsaCN_mpi.x 00000000006E02CE clsm_ensdrv_force 2796 clsm_ensdrv_force_routines.F90
LDASsaCN_mpi.x 00000000006C3D6B clsm_ensdrv_force 335 clsm_ensdrv_force_routines.F90
LDASsaCN_mpi.x 00000000004A9E11 MAIN__ 1859 cnlsm_ensdrv_main.F90
Investigate the issue.
2. Also noticed that I forgot to do this on 20180816:
clsm_ensdrv_glob_param.F90 [change 'N_CatchCN_OutFields = 35' to 'N_CatchCN_OutFields = 37']
That's why I don't see a change in the output file size.
Did this and compiled:
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
> source $ESMADIR/src/g5_modules
> gmake install
> cd ../../Applications/LDAS_App/
> gmake install
> cd ../../../
> /bin/cp -pr Linux/ exec/clm4.5/.
3. Set up the new cycle from scratch, following ~/Catchment/CLM4.5/submit_next_batch_DE720.
> set exp = clm4.5_DE720
> set num = 52
> cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5
> /bin/cp -r ${exp}_template ${exp}
> cd ${exp}/input/restart
> ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5/${exp}_${num}/output
> /bin/rm rst_clm4.5
> cd ../../run
> ls -l ../build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70928420 2018-08-21 11:45 ../build/Linux/bin/LDASsaCN_mpi.x*
Did an interactive run:
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> ./lenkf.0.j
It's running. Stopped it at date_time_new 19800101_100730z.
> /bin/rm ../output/global/rc_out/*
> qsub lenkf.0.j
==========
20180821:
1. The clm4.5_DE720 run stopped at 19800201.
forrtl: error (65): floating invalid
Image PC Routine Line Source
libirc.so 00002AAAAB590961 Unknown Unknown Unknown
libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2B2E82 Unknown Unknown Unknown
libc.so.6 00002AAAABA16910 Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000006B9E30 tile_coord_routin 2076 tile_coord.F90
LDASsaCN_mpi.x 0000000000569E72 clsm_ensdrv_out_r 1618 clsm_ensdrv_out_routines.F90
LDASsaCN_mpi.x 00000000004B6CC8 MAIN__ 2655 cnlsm_ensdrv_main.F90
L2076 in tile_coord.F90:
w = tile_coord(n)%frac_cell
==========
20180823:
1. Continued investigating the issue in the clm4.5_DE720 run.
The tile file /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/til/DE_00720x00360_PE_0720x0360.til was modified to also work for GEOSldas. However, I was able to use it for the clm4.5_DE720_52 which was finished on 20180327. So I don't understand why it complains about tile_coord.F90.
It wrote the output for 198001.
Updated ~/Catchment/CLM4.5/grid_restore_hdeg.f90 to include the 2 new diagnostics. Processed the 198001 output.
~/Catchment/CLM4.5 > grid_restore_hdeg clm4.5_DE720
1980 1
LatLon 1 1 1
720 287 0 61 -180.0000 -59.50000
180.0000 84.00000 0.5000000 0.5000000
N_land_mask= 67704
81 -1
18
This means that the 81th (last in this case) field was not written out correctly.
2. Set up the new cycle from scratch, following ~/Catchment/CLM4.5/submit_next_batch_DE720.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5
> rm -rf clm4.5_DE720
> set exp = clm4.5_DE720
> set num = 52
> cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5
> /bin/cp -r ${exp}_template ${exp}
> cd ${exp}/input/restart
> ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5/${exp}_${num}/output
> /bin/rm rst_clm4.5
> cd ../../run
> ls -l ../build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70928420 2018-08-21 11:45 ../build/Linux/bin/LDASsaCN_mpi.x*
Did an interactive run:
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> ./lenkf.0.j >> logfile
It's running.
It stopped again at date_time_new 19800201_000000z giving the same error message.
Added some print statements to clsm_ensdrv_out_routines.F90.
i,min(tile_data_f),max(tile_data_f): 78 0.0000000E+00 4.9818773E-05
i,min(tile_data_f),max(tile_data_f): 79 0.0000000E+00 7.5700441E-08
i,min(tile_data_f),max(tile_data_f): 80 0.0000000E+00 1.8518449E+32
The 80th field is nfire. The maximum value is huge!
Added some print statement to process_cn.F90 right after "call CN_Driver".
date_time_new 19800101_030000z
min(nfire),max(nfire): 0.0000000E+00 1.8518517E+32
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
date_time_new 19800101_043000z
min(nfire),max(nfire): 0.0000000E+00 1.8518517E+32
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
date_time_new 19800101_060000z
min(nfire),max(nfire): 0.0000000E+00 1.8518517E+32
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
So there must be something wrong with the nfire calculation in CN_DriverMod.F90 or the CN model.
Compiled with "gmake clean" and "gmake install BOPT=g" in (1) GEOScatchCN_GridComp, (2) GEOSlana_GridComp, and (3) Applications/LDAS_App.
minlon = -82. ! min longitude
maxlon = -80. ! max longitude
minlat = 37. ! min latitude
maxlat = 38. ! max latitude
> cp lenkf.0.j lenkf.j.ddt
> nedit lenkf.j.ddt &
lenkf.j.ddt reads:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/csh -fx
limit stacksize unlimited
source ../build/Linux/bin/g5_modules
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
setenv MYNAME `finger $USER | cut -d':' -f3 | head -1`
module load tool/allinea-tools-7.0.4
ddt /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x \
-work_path ../output \
-run_path . \
-exp_id clm4.5_DE720 \
-exp_domain global \
-start_year 1980 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \
-end_year 1984 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \
-N_ens 1 \
-spin .false. \
-force_dtstep 3600 \
-first_ens_id 0 \
-resolution DE_00720x00360_PE_0720x0360 \
-restart .true. \
-restart_path ../input/restart/output \
-restart_domain global \
-restart_id clm4.5_DE720 \
-met_tag M2COR_cross__precCPCUGPCP22clim_MERRA2_BMTXS \
-met_path ../input/met_forcing/MERRA2_land_forcing \
-driver_inputs_path . \
-driver_inputs_file driver_inputs_0.5D_GLOBAL.nml &
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> /bin/rm ../output/global/rc_out/*
> ./lenkf.j.ddt
All the 24 columns (of 8 tiles) have nfire=0 in the first CN time step in CNFireArea.
At the end of the first time step in CN_Driver:
cps%nfire = 0 for all 24 columns;
ccf%somc_fire = nan for all 24 columns. --> This is wrong!
In CNFireFluxes:
do fc = 1,num_soilc
c = filter_soilc(fc)
g = cgridcell(c)
if( grc%latdeg(g) .lt. borealat)then
somc_fire(c)= totsomc(c)*baf_peatf(c)/dt*6.0_r8/33.9_r8
else
somc_fire(c)= baf_peatf(c)/dt*2.2e3_r8
end if
end do
For all 24 columns:
baf_peatf = 0
totsomc = nan --> This is wrong!
ccs%totsomc is nan for all 24 columns in the 1st time step. Why?
totsomc is computed in CSummary, but CSummary is called after CNFireFluxes is called in CNEcosystemDyn. Also totsomc is not included in the restart file.
In CN_init, added the calculation of the initial totsomc value following the method in CSummary. This resolved the nan totsomc issue.
date_time_new 19800101_163000z
min(nfire),max(nfire): 0.0000000E+00 8.8692456E-11
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
date_time_new 19800101_180000z
min(nfire),max(nfire): 0.0000000E+00 1.9501317E-10
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
date_time_new 19800101_193000z
min(nfire),max(nfire): 2.0880690E-11 2.2343874E-10
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
The magnitude of nfire looks correct now.
Now run it globally interactively (not using DDT).
The global run still writes out huge nfire values:
date_time_new 19800101_013000z
min(nfire),max(nfire): 0.0000000E+00 1.8518517E+32
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
date_time_new 19800101_030000z
min(nfire),max(nfire): 0.0000000E+00 1.8518517E+32
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
==========
20180824:
1. Restored clsm_ensdrv_out_routines.F90 and added print statements to process_cn.F90 to locate the tiles with huge nfire, compiled and did a global run.
Did an interactive run:
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> /bin/rm ../output/global/rc_out/*
> ./lenkf.0.j >> logfile
min(nfire),max(nfire): 0.0000000E+00 1.8518517E+32
min(som_closs),max(som_closs): 0.0000000E+00 0.0000000E+00
lat,lon,value: 40.75000 63.25000 1.8518517E+32
lat,lon,value: 43.75000 63.25000 1.8518517E+32
lat,lon,value: 43.75000 63.75000 1.8518517E+32
lat,lon,value: 41.25000 64.25000 1.8518517E+32
lat,lon,value: 42.25000 63.75000 1.8518517E+32
lat,lon,value: 42.25000 63.25000 1.8518517E+32
lat,lon,value: 45.25000 60.75000 1.8518517E+32
lat,lon,value: 45.25000 62.75000 1.8518517E+32
lat,lon,value: 45.75000 59.25000 1.8518517E+32
lat,lon,value: 42.75000 58.75000 1.8518517E+32
lat,lon,value: 43.25000 60.25000 1.8518517E+32
lat,lon,value: 42.75000 59.75000 1.8518517E+32
lat,lon,value: 42.75000 60.75000 1.8518517E+32
lat,lon,value: 43.25000 60.75000 1.8518517E+32
lat,lon,value: 42.75000 61.25000 1.8518517E+32
lat,lon,value: 42.25000 61.25000 1.8518517E+32
lat,lon,value: 42.25000 61.75000 1.8518517E+32
lat,lon,value: 41.75000 61.75000 1.8518517E+32
lat,lon,value: 44.25000 60.75000 1.8518517E+32
lat,lon,value: 44.75000 60.75000 1.8518517E+32
lat,lon,value: 44.25000 61.75000 1.8518517E+32
lat,lon,value: 44.75000 62.25000 1.8518517E+32
lat,lon,value: 44.25000 62.25000 1.8518517E+32
lat,lon,value: 43.75000 60.75000 1.8518517E+32
lat,lon,value: 43.25000 61.25000 1.8518517E+32
lat,lon,value: 43.75000 61.25000 1.8518517E+32
lat,lon,value: 43.75000 61.75000 1.8518517E+32
lat,lon,value: 43.25000 62.25000 1.8518517E+32
lat,lon,value: 43.75000 62.25000 1.8518517E+32
lat,lon,value: 43.75000 62.75000 1.8518517E+32
These are in central Asia!!
==========
20180827:
1. Did a regional clm4.5_DE720 run on DDT:
minlon = 63. ! min longitude
maxlon = 63.5 ! max longitude
minlat = 40.5 ! min latitude
maxlat = 41. ! max latitude
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> /bin/rm ../output/global/rc_out/*
> lenkf.j.ddt &
The calculation of nfire is inside this 'if' statement:
if( cropf_col(c) .lt. 1.0 )then
Checked the crop fraction of this tile:
DE_00720x00360_PE_0720x0360.til:
100 4 63.2500 40.7500 487 262 1.000000000000 4 0.2368E+03 101112300000 35088
L35088 in CLM_veg_typs_fracs (we don't use CLM4.5_veg_typs_fracs anymore because we don't want to use prognostic crops):
4 18 19 18 19 87.50 12.50 0.00 0.00 16 16 100.00 0.00
So cropf_col(1), cropf_col(2) and cropf_col(3) are all 1. Therefore, the condition is not met and nfire is not computed.
After initiated to 1.e36, nfire is only updated for natural vegetation (i.e. "if( cropf_col(c) .lt. 1.0" ), and is only used to compute burned area for natural vegetation. Therefore, nfire is only valid for natural vegetation (i.e. Region C defined in CLM4.5).
For tiles 100% covered by crops, nfire should be 1.e36. Why is it 1.8518517E+32 in the model run?
It's because 1.e36 is fire counts per CN model time step (5400s), and 1.8518517E+32 is fire counts per second. 1.8518517E+32 * 5400 = 1.e36.
2. Now that the somc_fire issue is fixed and nfire is understood. Re-compiled the code using "gmake clean" and "gmake install" in GEOScatchCN_GridComp, GEOSlana_GridComp and Applications/LDAS_App.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
> /bin/cp -pr Linux/ exec/clm4.5/.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720
> mv build build2
> mv build1 build
> ls -l
lrwxrwxrwx 1 fzeng g0620 99 2018-08-23 10:58 build -> /gpfsm/dnb31/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4.5/
lrwxrwxrwx 1 fzeng g0620 105 2018-08-23 14:26 build2 -> /gpfsm/dnb31/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4.5_debug/
Changed the domain to global in driver_inputs_0.5D_GLOBAL.nml.
Did an interactive run:
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> ls -l ../build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70933376 2018-08-27 14:58 ../build/Linux/bin/LDASsaCN_mpi.x*
> /bin/rm ../output/global/rc_out/*
> /bin/rm logfile
> ./lenkf.0.j >> logfile &
It is running. Stopped it at date_time_new 19800405_200730z.
Removed the print statement I added to process_cn.F90 and recompiled the code: "gmake install" in GEOSlana_GridComp and Applications/LDAS_App.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
> /bin/cp -pr Linux/ exec/clm4.5/.
3. Set up the new cycle from scratch, following ~/Catchment/CLM4.5/submit_next_batch_DE720.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5
> rm -rf clm4.5_DE720
> set exp = clm4.5_DE720
> set num = 52
> cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5
> /bin/cp -r ${exp}_template ${exp}
> cd ${exp}/input/restart
> ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4.5/${exp}_${num}/output
> /bin/rm rst_clm4.5
> cd ../../run
> ls -l ../build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70928420 2018-08-27 15:51 ../build/Linux/bin/LDASsaCN_mpi.x*
Did an interactive run:
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> ./lenkf.0.j >> logfile &
It's running. Stopped it at date_time_new 19800102_000730z.
> /bin/rm ../output/global/rc_out/*
> /bin/rm ../output/global/rc_out/Y1980/M01/*
> qsub lenkf.0.j
==========
20180828:
1. Checked the clm4.5_DE720 run:
It crashed at:
date_time_new 19800605_000730z
get_forcing(): assuming GEOS-5 forcing data set
column cbalance error = -0.1139601 1026
Latdeg,Londeg= -34.75000 139.2500
begcb = 144447.3
endcb = 144447.4
delta store = 0.1250000
CBalance
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libirc.so 00002AAAAB590961 Unknown Unknown Unknown
libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown
libmpi_usempif08. 00002AAAAC2B2B73 Unknown Unknown Unknown
libc.so.6 00002AAAABA16910 Unknown Unknown Unknown
libpthread.so.0 00002AAAABD6C3D3 Unknown Unknown Unknown
libmlx5-rdmav2.so 00002AAABA1C686F Unknown Unknown Unknown
mca_btl_openib.so 00002AAAB30F6094 Unknown Unknown Unknown
libopen-pal.so.13 00002AAAAD1FC88B Unknown Unknown Unknown
mca_pml_ob1.so 00002AAAB3DC51B0 Unknown Unknown Unknown
libmpi.so.12 00002AAAACA9B9A1 Unknown Unknown Unknown
libmpi_mpifh.so.1 00002AAAAC80C582 Unknown Unknown Unknown
LDASsaCN_mpi.x 00000000004AA07B MAIN__ 1864 cnlsm_ensdrv_main.F90
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/output/global/rs/ens0000 > ls -l Y1980/M06
-rw-r--r-- 1 fzeng g0620 303909763 2018-08-27 20:19 clm4.5_DE720.ens0000.catchcn_ldas_rst.19800601_0000z
This clm4.5_DE720 cycle is using CLM4 vegetation types, but I am still using the restart from the end of clm4.5_DE720_52 (which used CLM4.5 vegetation types). This could be the reason for this C imbalance issue.
On 20180522 I have created the new restart file (from the M09 restart file) for the new clm4.5_DE720 run. This new restart file used CLM4 vegetation types to remove prognostic crops. There are 35 column-level and 75 PFT-level variables. I should have used this restart file.
Ask Randy if I should spin up the clm4.5_DE720 now after the removal of prognostic crops, and continue investigating the large burned area issue. He said yes.
Need to ask Sarith how to regrid the restart file from clm4.5_DE720_52 that used 27 PFTs for the new clm4.5_DE720 cycles that will use 19 PFTs.
==========
20180829:
1. Talked to Sarith about regridding the restart file from clm4.5_DE720_52 that used 27 PFTs for the new clm4.5_DE720 cycles that will use 19 PFTs. He said he would work on that.
==========
20180830:
1. For mapping from 19 Catchment-CN PFTs to the 5 types that Pete and Melanie have coefficients for trace gases:
CNSummaryMod.F90 and subgridAveMod.F90 are 2 of the files I need to modify.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> cp -p subgridAveMod.F90 subgridAveMod.F90.20170627
> cp -p CNSummaryMod.F90 CNSummaryMod.F90.20170417
==========
20180831:
1. Continued working on 6 of 20180830.
Added subroutine p2c_closs to subgridAveMod.F90.
Added "call p2c_closs" in CNSummaryMod.F90.
Restored these two files for now.
> mv CNSummaryMod.F90 CNSummaryMod.F90.closs
> mv subgridAveMod.F90 subgridAveMod.F90.closs
> cp -p subgridAveMod.F90.20170627 subgridAveMod.F90
> cp -p CNSummaryMod.F90.20170417 CNSummaryMod.F90
Need to first do a run to make sure I added these two new diagnostics nfire and som_closs correctly.
Waiting for Sarith to create the new restart file.
==========
20181231: (yes, 2018, not 2017!)
1. Checked the Catchment-CN with CLM4.5 code and CLM4.5 Tech Note to investigate why GPP of Catchment-CN with CLM4.5 is so low.
vcmax is much lower in CLM4.5 compared to CLM4. Is it causing the low GPP in CLM4.5?
Do more point runs to investigate the issue!
Things to note in cn.txt:
L143:
CNPhenologyMod.F90: "soilt" in CLM was set to soil layer 3; we are using a single soil layer, so changed this to 1; also, we are using max(t_grnd,t_soisno) for the determination of an "onset" trigger, which was probably a mistake... in future versions, t_grnd should be removed from this check (this is the unified model's TG).