========== 20191001: 1. Prepared slides for the fire meeting this afternoon. 2. September report. 3. Fire meeting. ========== 20191002: 1. Experiment to investigate low GPP in SH Africa. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit CN_DriverMod.F90 & Use PFT 14 parameter values for FPT 6. > cp -p CN_DriverMod.F90 CN_DriverMod.F90.PFT6NOW14 > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp/ > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > /bin/cp -pr Linux exec/clm4.5_Kn11_Ndiags_fire/. > ls -l exec/clm4.5_Kn11_Ndiags_fire/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-02 10:38 exec/clm4.5_Kn11_Ndiags_fire/Linux/bin/LDASsaCN_mpi.x* Restore CN_DriverMod.F90: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp CN_DriverMod.F90.restore CN_DriverMod.F90 > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_fire_DE720 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-02 10:38 build/Linux/bin/LDASsaCN_mpi.x* > cd input/restart/ > ls -l ../../../clm4.5_DE720_01/input/restart/ lrwxrwxrwx 1 fzeng g0620 81 2018-09-17 16:24 catchcn_internal_rst -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/catchcn_internal_clm45 > rm output > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_20180917/catchcn_internal_clm45 catchcn_internal_rsts > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_fire_DE720/run > nedit lenkf.0.j & changed -restart_path ../input/restart/output \ to -restart_path ../input/restart/ \ Did an interactive run: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_fire_DE720/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800102_010730z. > ./ldas_batchrun.j 2. Now that the automating procedure is working, try using more cores to see if it will speed up the simulation. Currently: for DE720 tile, using 168 cores, it took 2 hours 15 minutes to finish 1 year of simulation. > cd /discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas4.5_DE720 GEOSldas4.5_DE720_tmp > mv GEOSldas4.5_DE720_1980 GEOSldas4.5_DE720_1980_168cores Following notes on 20190923. > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > nedit GEOSldas4.5_DE720.bat GEOSldas4.5_DE720.exe & GEOSldas4.5_DE720.bat: ntasks : 252 GEOSldas4.5_DE720.exe: BEG_DATE : 19800101 000000 END_DATE : 19810101 000000 <------ for test only!! > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/exec/clm4.5_Kn11/install/bin > ls -l GEOSldas.x -rwxr-xr-x 1 fzeng g0620 66173350 2019-09-03 10:51 GEOSldas.x* > source g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.bat No cores have 0 tiles. Good! > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720 Checked executable: > ls -l build/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 66173350 2019-09-03 10:51 build/bin/GEOSldas.x* For restart: > set exp = GEOSldas4.5_DE720 > set num = 01 > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rs/ens0000/Y1980/M01 > /bin/rm ${exp}.catchcn_internal_rst.19800101_0000 > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/Y2019/M01/${exp}.catchcn_internal_rst.20190101_0000 ${exp}.catchcn_internal_rst.19800101_0000 > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rs/ens0000 > /bin/rm ${exp}.vegdyn_internal_rst > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/${exp}.vegdyn_internal_rst > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/input/restart/ > diff catchcn_internal_rst /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/Y2019/M01/${exp}.catchcn_internal_rst.20190101_0000 > diff vegdyn_internal_rst /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/${exp}.vegdyn_internal_rst Both are identical. Great! > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/run > /bin/cp /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/run/HISTORY.rc . > nedit LDAS.rc & [DO NOT copy from ${exp}_${num} due to different number of cores!!] Added to the end (L53): LNFM_FILE: ../input/lnfm.data > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rc_out > cp /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/lnfm_clim_720x360_DE.data . > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/input > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rc_out/lnfm_clim_720x360_DE.data lnfm.data Test: > interactive.py -A sp3 -n 252 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run > cp lenkf.j lenkf.j.orig > nedit lenkf.j & Add "exit" after L284 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > ./lenkf.j It's running. Stopped it at 1980-01-02T02:00:00. Submit the full jobs: > mv lenkf.j.orig lenkf.j > ./ldas_batchrun.j Using 252 cores, it took 2 hours 9 minutes to finish 1 year of simulation. This is only a little faster than the run using 168 cores. Talked to Weiyuan. He said no need to try more cores. Using 168 cores may be even better than using 252 cores because using fewer cores can shorten the wait time. Tried 196 cores. It took 2 hours 15 minutes to finish 1 year of simulation. Tried 224 cores on 20191003. It took 3 hours (02:58:43) to finish 1 year of simulation! ========== 20191003: 1. Drought experiment for Eunjee's proposal: (1) Data preprocessing, following notes on 20190909. > cd ~/Catchment/M2n5P_m0002 > nedit grid2tile_monthly.f90 & [No need to change.] > grid2tile_monthly deltaT2M_45member_1deg_avg_dryS_ctrl > grid2tile_monthly deltaPRCP_45member_1deg_avg_dryS_ctrl --> output not used in the dryS experiment > grid2tile_monthly deltaCO2SC_45member_1deg_avg_dryS_ctrl ##verify!! NOT DONE YET! (2) Model code modification for the drought experiment. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > mkdir dryS > cp -p ens_driver_routines.F90 ens_driver_routines.F90.restore > cp -p ens_driver_routines.F90 dryS/. > cp process_cn.F90 dryS/. > cd dryS > nedit ens_driver_routines.F90 & > nedit process_cn.F90 & > cp ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/compute_rc.F90 . > nedit compute_rc.F90 & Made sure ens_driver_routines.F90, process_cn.F90 and compute_rc.F90 each has its own *.restore file. > cp ens_driver_routines.F90 ../. > cp process_cn.F90 ../. > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/. Compile: > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp/ > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > mkdir -p exec/clm4.5_Kn11_dryS > /bin/cp -pr Linux exec/clm4.5_Kn11_dryS/. Set up experiment: Check the restart for the control (i.e. the current clm4.5_DE720 cycle which will be renamed to clm4.5_DE720_67): > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 > ls -l clm4.5_DE720/input/restart lrwxrwxrwx 1 fzeng g0620 80 2019-09-24 10:39 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5_cn.exe D0.5.bat & D0.5.bat: job-name = clm4.5_DE720_dryS D0.5_cn.exe: exp_id = clm4.5_DE720_dryS start_time = 1983-01-01-00-00-00 end_time = 1984-01-01-00-00-00 restart_path = /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/ restart_id = clm4.5_DE720 > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_dryS/Linux/bin > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dryS Check the executable: > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71273854 2019-10-04 11:42 build/Linux/bin/LDASsaCN_mpi.x* Check the restart: > ls -l input/restart lrwxrwxrwx 1 fzeng g0620 77 2019-10-04 12:12 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/output/ > cd run/ > nedit lenkf.0.j & #SBATCH --time=03:00:00 Did an interactive run: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dryS/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19830102_140730z. > qsub lenkf.0.j Restored the files: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > cp ens_driver_routines.F90.restore ens_driver_routines.F90 > cp process_cn.F90.restore process_cn.F90 > cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/ > cp compute_rc.F90.restore compute_rc.F90 ========== 20191004: 1. Continued working on the drought experiment for Eunjee's proposal. Notes included above. 2. Processed GMAO Oct forecast. 3. Reviewed git. 4. Processed the output of GEOSldas4.5_DE720_01. > cd ~/Catchment/CLM4.5 Do NOT change tile2grid_GEOSldas.py because it will be used for spinup, so make a copy and rename to tile2grid_GEOSldas_01.py to process the output of GEOSldas4.5_DE720_01. > cp tile2grid_GEOSldas.py tile2grid_GEOSldas_01.py > nedit tile2grid_GEOSldas_01.py & > python tile2grid_GEOSldas_01.py > cd ~/python/clm4-to-clm4.5 Created compare_ts_GEOSldas_DE720.py to calculate multi-year mean annual GPP and NEE. > python compare_ts_GEOSldas_DE720.py 5. Set up the full (i.e. 1980-2018) cycle for the GEOSldas4.5 spinup: Following notes on 20190923. > cd /discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas4.5_DE720_template GEOSldas4.5_DE720_template_1980 > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > nedit GEOSldas4.5_DE720.bat GEOSldas4.5_DE720.exe & GEOSldas4.5_DE720.bat: ntasks : 168 GEOSldas4.5_DE720.exe: BEG_DATE : 19800101 000000 END_DATE : 20190101 000000 > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/exec/clm4.5_Kn11/install/bin > source g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.bat > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720 Checked executable: > ls -l build/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 66173350 2019-09-03 10:51 build/bin/GEOSldas.x* For restart: > set exp = GEOSldas4.5_DE720 > set num = 01 > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rs/ens0000/Y1980/M01 > /bin/rm ${exp}.catchcn_internal_rst.19800101_0000 > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/Y2019/M01/${exp}.catchcn_internal_rst.20190101_0000 ${exp}.catchcn_internal_rst.19800101_0000 > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rs/ens0000 > /bin/rm ${exp}.vegdyn_internal_rst > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/${exp}.vegdyn_internal_rst > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/input/restart/ > diff catchcn_internal_rst /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/Y2019/M01/${exp}.catchcn_internal_rst.20190101_0000 > diff vegdyn_internal_rst /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/${exp}.vegdyn_internal_rst Both are identical. Great! > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/run > /bin/cp /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/run/HISTORY.rc . > nedit LDAS.rc & [DO NOT copy from ${exp}_${num} due to different number of cores!!] Added to the end (L53): LNFM_FILE: ../input/lnfm.data > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rc_out > cp /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/lnfm_clim_720x360_DE.data . > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/input > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rc_out/lnfm_clim_720x360_DE.data lnfm.data > cd ~/Catchment/CLM4.5 > nedit submit_next_batch_GEOSldas_DE720 & [update it to check if the restart file for 20190101 is found] > cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/run > nedit ldas_batchrun.j & Changed all "afterany" to "afterok". Added after the last jobid (which is jobid9 in this case): jobid10=$(echo $(sbatch --dependency=afterok:$jobid9 lenkf_submit_next_batch.j) | cut -d' ' -f 4) echo $jobid10 > cp -p ../../GEOSldas4.5_DE720_template_1980/run/lenkf_submit_next_batch.j . > cd /discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas4.5_DE720 GEOSldas4.5_DE720_template > /bin/cp -r GEOSldas4.5_DE720_template GEOSldas4.5_DE720 Test: > interactive.py -A sp3 -n 168 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run > cp lenkf.j lenkf.j.orig > nedit lenkf.j & Add "exit" after L284 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > ./lenkf.j It's running. Stopped it at 1980-01-02T04:00:00. Submit the full jobs: > mv lenkf.j.orig lenkf.j > ./ldas_batchrun.j ========== 20191007: 1. Processed and analyzed the output of clm4.5_fire_DE720 set up on 20191002. > cd ~/Catchment/CLM4.5 > grid_restore_hdeg clm4.5_fire_DE720 LAI is much lower in Jan 1980 compared to the clm4.5_DE720_01 and clm4_DE720_01 for the Africa grid that I plot. Is this because LAI dropped quickly because this grid now is cool c3 grass and no longer type 6? or I made a mistake? Saved a copy and re-do the run: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 > /bin/cp -pr clm4.5_fire_DE720 clm4.5_fire_DE720_PFT6NOW14 NOTE: The initial conditions of clm4.5_fire_DE720_PFT6NOW14 are the same as those of clm4.5_DE720_01. > cd clm4.5_fire_DE720 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-02 10:38 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 90 2019-10-02 10:46 catchcn_internal_rst -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_20180917/catchcn_internal_clm45 > /bin/rm output/global/rc_out/* > /bin/rm output/global/rc_out/Y*/M*/* > /bin/rm output/global/cat/ens_avg/Y*/M*/* > cd run > ./ldas_batchrun.j Compared LAI between clm4.5_fire_DE720_PFT6NOW14 and clm4.5_DE720_01. For Jan 1980, the difference is only found in the type 6 regions. It seems that the clm4.5_fire_DE720_PFT6NOW14 run was done correctly. For Jul 1980, the difference is found in many more places. This should be due to the fire code changes and therefore there are more fires in clm4.5_fire_DE720_PFT6NOW14 than in clm4.5_DE720_01. Deleted the jobs for clm4.5_fire_DE720 from the queue. > qdel 35111190 35111191 35111192 35111193 35111194 Made clm4.5_fire_DE720 start from the final states of clm4.5_DE720_66 (so I should compare clm4.5_fire_DE720 to clm4.5_DE720_67 later): > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_fire_DE720/input/restart > rm catchcn_internal_rst > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output > cd ../../run/ > nedit lenkf.0.j & Did an interactive run: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_fire_DE720/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800102_040730z. > ./ldas_batchrun.j Analyzed the output of clm4.5_fire_DE720_PFT6NOW14: ~/Catchment/CLM4.5 > grads -lc "run check_output.gs" 2. Drought experiment for Eunjee's proposal: > cd ~/Catchment/CLM4.5 > cp clm4.5_DE720.ctl clm4.5_DE720_dryS.ctl > nedit clm4.5_DE720.ctl clm4.5_DE720_dryS.ctl & > cp grid_restore_hdeg.f90 grid_restore_hdeg_singleYr.f90 > ifort grid_restore_hdeg_singleYr.f90 > a.out clm4.5_DE720_dryS Couldn't find the output for 198303. Checked the experiment log file. The run crashed at date_time_new 19830401_000000z and gave a lot of these error messages below: date_time_new 19830401_000000z THRU= -1.6897316E-08 0.0000000E+00 0.0000000E+00 -1.8189886E-09 0.0000000E+00 THRU= -4.1569862E-08 0.0000000E+00 0.0000000E+00 -2.7093967E-09 0.0000000E+00 THRU= -5.9915230E-08 0.0000000E+00 0.0000000E+00 -2.7124769E-09 0.0000000E+00 THRU= -1.0388538E-07 0.0000000E+00 0.0000000E+00 -4.6786850E-09 0.0000000E+00 THRU= -1.3323104E-07 0.0000000E+00 0.0000000E+00 -3.7562984E-09 0.0000000E+00 THRU= -3.0176213E-06 -1.5900580E-06 -4.9885367E-09 -5.4036715E-08 0.0000000E+00 THRU= -1.1620014E-05 -7.1816794E-06 -2.2325661E-08 -1.7966343E-07 0.0000000E+00 THRU= -3.7870948E-06 -9.6238603E-07 -2.9869891E-09 -5.7203113E-08 0.0000000E+00 THRU= -6.1463666E-05 -1.5604037E-05 -4.3838867E-08 -3.6677824E-07 0.0000000E+00 In GEOSsurface_GridComp/GEOSland_GridComp/Shared/lsm_routines.F90: L375-376: IF (THRUL+THRUC .LT. -1.e-8) WRITE(*,*) 'THRU= ', & THRUL, THRUC, TRAINC(CHNO), TRAINL(CHNO), SMELT(CHNO) (1) Tried this: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > cp -p ens_driver_routines.F90 ens_driver_routines.F90.v1 > nedit ens_driver_routines.F90 & Commented out this block: ! totprcp = met_force_ntp(n)%Rainf+met_force_ntp(n)%Snowf ! if(totprcp>totprcp_max) then ! met_force_ntp(n)%Rainf = met_force_ntp(n)%Rainf * totprcp_max / totprcp ! met_force_ntp(n)%Rainf_C = met_force_ntp(n)%Rainf_C * totprcp_max / totprcp ! met_force_ntp(n)%Snowf = met_force_ntp(n)%Snowf * totprcp_max / totprcp ! endif Compile: > cp ens_driver_routines.F90 ../. > cp process_cn.F90 ../. > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/. > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp/ > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > /bin/cp -pr Linux exec/clm4.5_Kn11_dryS/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dryS Check the executable: > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71291773 2019-10-07 14:50 build/Linux/bin/LDASsaCN_mpi.x* For the restart: > cd input/restart > rm output > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output > ls -l lrwxrwxrwx 1 fzeng g0620 80 2019-10-07 14:53 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output/ > cd ../../run/ Did an interactive run: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dryS/run > ./lenkf.0.j Let it run until it passed the point where it crashed earlier. It again crashed at the same date_time giving the same error messages. Restored the files: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > cp ens_driver_routines.F90.restore ens_driver_routines.F90 > cp process_cn.F90.restore process_cn.F90 > cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/ > cp compute_rc.F90.restore compute_rc.F90 (2) Tried this: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > cp ens_driver_routines.F90 ens_driver_routines.F90.v2 > mv ens_driver_routines.F90 ens_driver_routines.F90.dryS > nedit ens_driver_routines.F90.dryS & [Following /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2/ ens_driver_routines.F90.dryS.] > mv process_cn.F90 process_cn.F90.dCO2SC_S Compile: > cp process_cn.F90.dCO2SC_S ../process_cn.F90 > cp ens_driver_routines.F90.dryS ../ens_driver_routines.F90 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/. > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp/ > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > mkdir -p exec/clm4.5_Kn11_dco2sc_dryS > /bin/cp -pr Linux exec/clm4.5_Kn11_dco2sc_dryS/. Followed /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the 1983 experiment: (x.1) Modified D0.5.bat and D0.5_cn.exe in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run for this run. (x.2) Run ldsetup: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_dco2sc_dryS/Linux/bin source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 9 --landmodel catchCN (x.3) Check the restart file: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/input/restart ls -l lrwxrwxrwx 1 fzeng g0620 80 2019-10-07 16:55 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output/ Looks correct. > cd ../.. > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71274121 2019-10-07 16:01 build/Linux/bin/LDASsaCN_mpi.x* (x.4) Check the job file and modify if needed cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run lenkf.0.j looks correct. No modification needed. /bin/rm lenkf.1.j (it's not needed) (x.5) Check the executable > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71274121 2019-10-07 16:01 build/Linux/bin/LDASsaCN_mpi.x* Correct. (x.6) Do an interactive run to make sure it runs and uses the right restart file: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ./lenkf.0.j (x.7) qsub lenkf.0.j ========== 20191008: 1. Drought experiment for Eunjee's proposal: The clm4.5_DE720_dco2sc_dryS_1983 run crashed again: date_time_new 19830401_000000z THRU= -1.3361371E-04 0.0000000E+00 0.0000000E+00 -7.3947979E-07 0.0000000E+00 THRU= -6.5215410E-08 0.0000000E+00 0.0000000E+00 -2.9211225E-10 0.0000000E+00 THRU= -9.3582207E-03 0.0000000E+00 0.0000000E+00 -5.3862597E-05 0.0000000E+00 THRU= -2.6260035E-03 0.0000000E+00 0.0000000E+00 -1.6911019E-05 0.0000000E+00 THRU= -4.8777883E-07 -5.8923163E-03 -1.5285281E-05 -1.9026629E-09 0.0000000E+00 THRU= -2.0361463E-06 -4.3872267E-02 -1.1015252E-04 -6.9048838E-09 0.0000000E+00 THRU= -8.3370404E-03 0.0000000E+00 0.0000000E+00 -5.3418247E-05 0.0000000E+00 THRU= -6.0094864E-04 0.0000000E+00 0.0000000E+00 -4.9875862E-06 0.0000000E+00 THRU= -1.7809725E-03 0.0000000E+00 0.0000000E+00 -1.4830332E-05 0.0000000E+00 Ball-Berry error check - stomatal conductance error: 1.7224414E+07 1.7223934E+07 Ball-Berry error check - stomatal conductance error: 1.7209494E+07 1.7209642E+07 THRU= -8.1596691E-03 0.0000000E+00 0.0000000E+00 -2.3019114E-04 0.0000000E+00 THRU= -4.0305298E-05 0.0000000E+00 0.0000000E+00 -3.1498807E-06 0.0000000E+00 THRU= -2.2404829E-06 0.0000000E+00 0.0000000E+00 -2.7329057E-07 0.0000000E+00 THRU= -1.0281198E-08 0.0000000E+00 0.0000000E+00 -2.6045164E-08 0.0000000E+00 THRU= -4.6305768E-02 0.0000000E+00 0.0000000E+00 -1.1946417E-03 0.0000000E+00 THRU= -5.3310591E-06 0.0000000E+00 0.0000000E+00 -6.7810851E-07 0.0000000E+00 THRU= -5.7990651E-06 0.0000000E+00 0.0000000E+00 -7.1351184E-07 0.0000000E+00 THRU= -3.9063036E-02 0.0000000E+00 0.0000000E+00 -9.8778727E-04 0.0000000E+00 THRU= -1.3608367E-06 -1.9790674E-02 -6.5251072E-05 -1.4213583E-07 0.0000000E+00 Talked to Randy. Randy suggests to add print statements to check if there is any negative precipitation after the precipitation is scaled. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > nedit ens_driver_routines.F90 & Added ! check if there is any negative precipitation if (met_force_ntp(n)%Rainf<0. .or. met_force_ntp(n)%Rainf_C<0. .or. met_force_ntp(n)%Snowf<0.) then print *, 'Tile ID: ', tile_coord(n)%tile_id print *, 'Tile lat: ', tile_coord(n)%com_lat print *, 'Tile long: ', tile_coord(n)%com_lon print *, 'Rainf: ', met_force_ntp(n)%Rainf print *, 'Rainf_C: ', met_force_ntp(n)%Rainf_C print *, 'Snowf: ', met_force_ntp(n)%Snowf print *, 'PRCP_AGCM_dryS: ', PRCP_AGCM_dryS(n) print *, 'PRCP_Offline: ', PRCP_Offline(n) stop endif to the do loop which does precipiation scaling. > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > /bin/cp -pr Linux exec/clm4.5_Kn11_dco2sc_dryS/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71298063 2019-10-08 11:40 ../build/Linux/bin/LDASsaCN_mpi.x* Run it interactively: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ./lenkf.0.j Found that in quite a few tiles AGCM dryS precipitation climatology is negative. MERRA2 precipitation climatology is positive. Will check whether I did something wrong when I converted the AGCM dryS precipitation climatology from lat-lon grid-space to tile-space. Checked Eunjee's PRCP_dryS_45member_1deg_avg_04.txt in /discover/nobackup/elee15/analysis/geos5/4offline_input. Minimum is 0, which is correct. > cd ~/Catchment/M2n5P_m0002 > cp grid2tile_monthly.f90 grid2tile_monthly_test.f90 > nedit grid2tile_monthly_test.f90 & Added these blocks: do j = 1,jG5 do i = 1,iG5 if (data_grid(i,j)<0) then print *, data_grid(i,j) stop 'negative data_grid' endif end do end do if (raster(i,j)<0) then print *, raster(i,j) stop 'negative raster' endif do n = 1,nland if (data_tile(n)<0) then print *, data_tile(n) stop 'negative data_tile' endif end do > ifort -o grid2tile_monthly_test -mcmodel=medium grid2tile_monthly_test.f90 > grid2tile_monthly_test PRCP_dryS_45member_1deg_avg 04: Grid-space: 68.00831 0.0000000E+00 2.626695 Raster: 68.00831 0.0000000E+00 2.345011 Tile-space: 68.00850 0.0000000E+00 3.209420 05: Grid-space: 70.82157 0.0000000E+00 2.689141 Raster: 70.82157 0.0000000E+00 2.343802 Tile-space: 70.82181 0.0000000E+00 3.054273 06: Grid-space: 72.92512 0.0000000E+00 2.740868 Raster: 72.92512 0.0000000E+00 2.326402 Tile-space: 72.92546 0.0000000E+00 3.244992 07: Grid-space: 74.59188 0.0000000E+00 2.818000 Raster: 74.59188 0.0000000E+00 2.363467 Tile-space: 74.59317 0.0000000E+00 3.601331 08: Grid-space: 75.13960 0.0000000E+00 2.795789 Raster: 75.13960 0.0000000E+00 2.346531 Tile-space: 75.14013 0.0000000E+00 3.583776 09: Grid-space: 75.70292 0.0000000E+00 2.743450 Raster: 75.70292 0.0000000E+00 2.334161 Tile-space: 75.70311 0.0000000E+00 3.404108 The 3 numbers are max, min and avg, respectively. There is no negative numbers in the generated tile-space output. > nedit check_PRCP_dryS_45member_1deg_avg.f90 & ~~~~~~~~~~~~ implicit none integer, parameter :: N_catd = 648421 ! number of land tiles real, allocatable :: PRCP_AGCM_dryS(:) integer :: n, mon character :: cmo*2, datapath*200 datapath = '/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology/' allocate(PRCP_AGCM_dryS(N_catd)) do mon = 4,9 write(cmo,'(i2.2)') mon open(7593,file=trim(datapath)//'/PRCP_dryS_45member_1deg_avg_'//cmo//'.dat',& form='unformatted',status='old',access='direct',recl=1) do n = 1,N_catd read(7593,rec=n) PRCP_AGCM_dryS(n) if (PRCP_AGCM_dryS(n)<0) then print *, cmo print *, PRCP_AGCM_dryS(n) stop endif end do close(7593) end do print *, 'no negative values' deallocate(PRCP_AGCM_dryS) end ~~~~~~~~~~~~ > ifort check_PRCP_dryS_45member_1deg_avg.f90 > a.out 04 -5.9350864E+37 This is because I did not use "convert='big_endian'," when I read the data in check_PRCP_dryS_45member_1deg_avg.f90. After I added "convert='big_endian'," to the open statement when reading the data in check_PRCP_dryS_45member_1deg_avg.f90: open(7593,file=trim(datapath)//'/PRCP_dryS_45member_1deg_avg_'//cmo//'.dat',& form='unformatted',convert='big_endian',status='old',access='direct',recl=1) And I ran the program again, I got: no negative values > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > nedit ens_driver_routines.F90 & Added "convert='big_endian'," to the data reading statements in ens_driver_routines.F90. > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > /bin/cp -pr Linux exec/clm4.5_Kn11_dco2sc_dryS/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71298199 2019-10-08 14:19 ../build/Linux/bin/LDASsaCN_mpi.x* -rwxr-xr-x 1 fzeng g0620 71304047 2019-10-08 14:49 ../build/Linux/bin/LDASsaCN_mpi.x* [added more print statements] Still crashed and giving the same error messages. -rwxr-xr-x 1 fzeng g0620 71304047 2019-10-08 15:13 ../build/Linux/bin/LDASsaCN_mpi.x* [changed when to read the data: now at YYYYMM01_000000z, was YYYYMM01_000730z] This works! Run it interactively: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ./lenkf.0.j Also replaced if(date_time%day==1 .and. date_time%hour==0 .and. & date_time%min==7 .and. date_time%sec==30) then ! assumes 7.5 minute time step with if(date_time%day==1 .and. date_time%hour==0 .and. & date_time%min==0 .and. date_time%sec==0) then ! For the runs starting from Jan 01 in process_cn.F90 and compiled. > qsub lenkf.0.j > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > nedit create_symbolic_links_2 & > chmod 755 create_symbolic_links_2 > create_symbolic_links_2 2. Control for the drought experiment for Eunjee's proposal: Modify the code and set up the control run. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90 > cp process_cn.F90.ctrl ../process_cn.F90 > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp compute_rc.F90.restore compute_rc.F90 Already set ESMADIR and source g5_modules earlier. > gmake install > cd ../../../GEOSlana_GridComp/ > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_ctrl > /bin/cp -pr Linux exec/clm4.5_Kn11_ctrl/. Followed notes on 20191007 to set up the 1983 control: (x.1) Modified D0.5.bat and D0.5_cn.exe in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run for this run. (x.2) Run ldsetup: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_ctrl/Linux/bin source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 9 --landmodel catchCN (x.3) Check the restart file: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/input/restart ls -l lrwxrwxrwx 1 fzeng g0620 80 2019-10-08 16:37 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output/ Looks correct. (x.4) Check the job file and modify if needed cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run lenkf.0.j looks correct. No modification needed. /bin/rm lenkf.1.j (it's not needed) (x.5) Check the executable > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71294785 2019-10-08 16:33 build/Linux/bin/LDASsaCN_mpi.x* Correct. (x.6) Do an interactive run to make sure it runs and uses the right restart file: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run > ./lenkf.0.j It's running and passed 19830401. Stopped it at date_time_new 19830401_160730z. (x.7) qsub lenkf.0.j ========== 20191009: 1. For the drought experiment for Eunjee's proposal: I forgot that I am using the DE720 tile system here, so I can not use the data in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology which is for the DC0576xPC0361_DE0360xPE0180 tile system. (1) Recomputed MERRA2 T2M and precipitation climatology: > cd ~/Catchment/CLM4.5 > cp ../M2n5P_m0002/calculate_clim_tile_monthly_Tair.f90 calculate_clim_grid_monthly_Tair.f90 > nedit calculate_clim_grid_monthly_Tair.f90 & > ifort -o calculate_clim_grid_monthly_Tair calculate_clim_grid_monthly_Tair.f90 > calculate_clim_grid_monthly_Tair clm4.5_DE720_67 clm4.5_DE720 > cp calculate_clim_grid_monthly_Tair.f90 calculate_clim_grid_monthly_precip.f90 > nedit calculate_clim_grid_monthly_precip.f90 & > ifort -o calculate_clim_grid_monthly_precip calculate_clim_grid_monthly_precip.f90 > calculate_clim_grid_monthly_precip clm4.5_DE720_67 clm4.5_DE720 Checked the calculated MERRA2 climatology. > cd ~/Catchment/CLM4.5 > cp clm4.5_DE720.ctl MERRA2_T2M_2000_2014_avg.ctl > nedit MERRA2_T2M_2000_2014_avg.ctl & Checked on GrADS. Looks correct. > cp MERRA2_T2M_2000_2014_avg.ctl MERRA2_precip_2000_2014_avg.ctl > nedit MERRA2_precip_2000_2014_avg.ctl & Checked on GrADS. Looks correct. (2) For the control run: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > nedit ens_driver_routines.F90 & Compile: > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > /bin/cp -pr Linux exec/clm4.5_Kn11_ctrl/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71298931 2019-10-09 16:01 ../build/Linux/bin/LDASsaCN_mpi.x* Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run > ./lenkf.0.j It's running and has passed 19830401. Stopped it at date_time_new 19830401_230730z. > qsub lenkf.0.j (3) For the experiment run: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > nedit ens_driver_routines.F90.dryS & > nedit process_cn.F90.dCO2SC_S & > cp ens_driver_routines.F90.dryS ../ens_driver_routines.F90 > cp process_cn.F90.dCO2SC_S ../process_cn.F90 > cp compute_rc.F90.dry ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/compute_rc.F90 > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > /bin/cp -pr Linux exec/clm4.5_Kn11_dco2sc_dryS/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71308459 2019-10-09 16:59 ../build/Linux/bin/LDASsaCN_mpi.x* Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ./lenkf.0.j > exp_log It's running and has passed 19830401. Stopped it at date_time_new 19830419_160730z. ========== 20191010: 1. For the drought experiment for Eunjee's proposal: > cd ~/Catchment/CLM4.5 > cp grid_restore_hdeg.f90 grid_restore_hdeg_drought.f90 > ifort -o grid_restore_hdeg_drought grid_restore_hdeg_drought.f90 > grid_restore_hdeg_drought clm4.5_DE720_ctrl 1983 > grid_restore_hdeg_drought clm4.5_DE720_dco2sc_dryS 1983 > cp clm4.5_DE720.ctl clm4.5_DE720_ctrl.ctl > nedit clm4.5_DE720_ctrl.ctl & > cp clm4.5_DE720_ctrl.ctl clm4.5_DE720_dco2sc_dryS.ctl > nedit clm4.5_DE720_dco2sc_dryS.ctl & > nedit compare_drought_ctrlVSexp.gs & Analyzed the output of clm4.5_DE720 and clm4.5_DE720_dco2sc_dryS using compare_drought_ctrlVSexp.gs. 2. Processed and analyzed the output of clm4.5_fire_DE720: ~/Catchment/CLM4.5 > grid_restore_hdeg clm4.5_fire_DE720 Analyzed using check_output.gs and check_fire_bug_fix.gs in ~/Catchment/CLM4.5. 3. Prepared meeting slides. 4. Met with Randy, Eunjee and Jana about the proposal. 5. Met with Randy and Eunjee about the fire experiments. ========== 20191011: 1. For the drought experiment for Eunjee's proposal: Changed the cap for the precipitation to 200mm/d. (1) Control: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > nedit ens_driver_routines.F90.ctrl & real, parameter :: totprcp_max = 200./86400 ! cap the precipitation at 200 mm/d, convert it to mm/s > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90 > cp process_cn.F90.ctrl ../process_cn.F90 > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp compute_rc.F90.restore compute_rc.F90 > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > /bin/cp -pr Linux exec/clm4.5_Kn11_ctrl/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71298999 2019-10-11 09:19 ../build/Linux/bin/LDASsaCN_mpi.x* > qsub lenkf.0.j (2) Experiment: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > nedit ens_driver_routines.F90.dryS & real, parameter :: totprcp_max = 200./86400 ! cap the precipitation at 200 mm/d, convert it to mm/s > cp ens_driver_routines.F90.dryS ../ens_driver_routines.F90 > cp process_cn.F90.dCO2SC_S ../process_cn.F90 > cp compute_rc.F90.dry ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/compute_rc.F90 > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > /bin/cp -pr Linux exec/clm4.5_Kn11_dco2sc_dryS/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71308191 2019-10-11 09:30 ../build/Linux/bin/LDASsaCN_mpi.x* Restored the files: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp compute_rc.F90.restore compute_rc.F90 > cd ../../../GEOSlana_GridComp/ > cp process_cn.F90.restore process_cn.F90 > cp ens_driver_routines.F90.restore ens_driver_routines.F90 2. Proposal writing. 3. Fire experiment: (1) Changed FLNR from 0.0716 to 0.096 for type 6: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit CN_DriverMod.F90 & Changed FLNR from 0.0716 to 0.096 for type 6. > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_PFT6Vcmax55 > /bin/cp -pr Linux exec/clm4.5_Kn11_PFT6Vcmax55/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5.bat D0.5_cn.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_PFT6Vcmax55/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_PFT6Vcmax55 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-11 14:43 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-11 14:50 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > cp ../../clm4.5_fire_DE720/run/ldas_batchrun.j . Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_PFT6Vcmax55/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800102_030730z. > ./ldas_batchrun.j Restored the file: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp CN_DriverMod.F90.restore CN_DriverMod.F90 (2) Change fm of leaves and cc of cwd and litter: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit CN_DriverMod.F90 & ! data fm_leax / 0., 0.8, 0.8, 0.8, 0.8, 0.8, 0.80, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8/ data fm_leax / 0., 0.5, 0.5, 0.5, 0.5, 0.5, 0.50, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5/ > nedit CNFireMod.F90 & do j = 1, nlevdecomp ! carbon fluxes do l = 1, ndecomp_pools if ( is_litter(l) ) then ! m_decomp_cpools_to_fire_vr(c,j,l) = decomp_cpools_vr(c,j,l) * f * 0.5_r8 m_decomp_cpools_to_fire_vr(c,j,l) = decomp_cpools_vr(c,j,l) * f * 0.9_r8 end if if ( is_cwd(l) ) then m_decomp_cpools_to_fire_vr(c,j,l) = decomp_cpools_vr(c,j,l) * & ! (f-baf_crop(c)) * 0.25_r8 (f-baf_crop(c)) * 0.5_r8 end if end do ! nitrogen fluxes do l = 1, ndecomp_pools if ( is_litter(l) ) then ! m_decomp_npools_to_fire_vr(c,j,l) = decomp_npools_vr(c,j,l) * f * 0.5_r8 m_decomp_npools_to_fire_vr(c,j,l) = decomp_npools_vr(c,j,l) * f * 0.9_r8 end if if ( is_cwd(l) ) then m_decomp_npools_to_fire_vr(c,j,l) = decomp_npools_vr(c,j,l) * & ! (f-baf_crop(c)) * 0.25_r8 (f-baf_crop(c)) * 0.5_r8 end if end do end do > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_fmleaf_CCcwdNlitter > /bin/cp -pr Linux exec/clm4.5_Kn11_fmleaf_CCcwdNlitter/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5.bat D0.5_cn.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_fmleaf_CCcwdNlitter/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_fmleaf_CCcwdNlitter > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-11 16:42 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-11 16:42 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > cp ../../clm4.5_fire_DE720/run/ldas_batchrun.j . Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_fmleaf_CCcwdNlitter/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800103_070730z. > ./ldas_batchrun.j Restored the file: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp CN_DriverMod.F90.restore CN_DriverMod.F90 > cp CNFireMod.F90.restore CNFireMod.F90 ========== 20191015: 1. For the drought experiment for Eunjee's proposal: Both clm4.5_DE720_ctrl_1983 and clm4.5_DE720_dco2sc_dryS_1983 crashed. This is because I modified process_cn.F90 and ens_driver_routines.F90 when I compiled the code but I forgot to save the final copies, so the ones in GEOSlana_GridComp/dryS are still the buggy ones!! So when I used them on 20191011 and didn't have a chance to do a test run first, they crashed in the batch jobs. (1) Decided to run another cycle of clm4.5_DE720 to write out Apr01 restart files for each year, so that the test runs do not have to start from Jan01 and takes a long time to get to Apr01 where problem/crash may happen. Made sure the files process_cn.F90, ens_driver_routines.F90 and compute_rc.F90 are restored. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > nedit clsm_ensdrv_out_routines.F90 & > cp clsm_ensdrv_out_routines.F90 dryS/clsm_ensdrv_out_routines.F90.wApr01rst > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_wApr01rst > /bin/cp -pr Linux exec/clm4.5_Kn11_wApr01rst/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output/global/rs/ens0000/Y1980/M01 ls -l lrwxrwxrwx 1 fzeng g0620 68 2019-10-04 21:33 clm4.5_DE720.ens0000.catchcn_ldas_rst.19800101_0000z -> ../../Y2017/M01/clm4.5_DE720.ens0000.catchcn_ldas_rst.20170101_0000z Good! > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720 > rm build > ln -s /gpfsm/dnb31/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_wApr01rst build > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-15 10:51 build/Linux/bin/LDASsaCN_mpi.x* > cd input/restart/ > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output > ls -l lrwxrwxrwx 1 fzeng g0620 80 2019-10-15 10:59 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_67/output/ Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run > ./lenkf.0.j > ./ldas_batchrun.j Restore: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > cp clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90 (2) Control: Since the current clm4.5_DE720 cycle that will write out Apr01 restart files is still in the queue, the clm4.5_DE720_ctrl_1983 still have to start from Jan01. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > nedit ens_driver_routines.F90.ctrl & L469-470: open(7594,file=trim(DE720path)//'/MERRA2_precip_2000_2014_avg_'//cmo//'.dat',& form='unformatted',status='old') > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90 > cp process_cn.F90.ctrl ../process_cn.F90 > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp compute_rc.F90.restore compute_rc.F90 > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > /bin/cp -pr Linux exec/clm4.5_Kn11_ctrl/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71298931 2019-10-15 11:26 ../build/Linux/bin/LDASsaCN_mpi.x* Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_ctrl/clm4.5_DE720_ctrl_1983/run > ./lenkf.0.j It's running and passed Apr 01. Stopped it at date_time_new 19830402_073000z. > /bin/rm ../output/global/rc_out/* > /bin/rm ../output/global/rc_out/Y1983/M01/* > qsub lenkf.0.j Restore: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp > cp ens_driver_routines.F90.restore ens_driver_routines.F90 > cp process_cn.F90.restore process_cn.F90 (3) Experiment: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSlana_GridComp/dryS > nedit process_cn.F90.dCO2SC_S & > cp ens_driver_routines.F90.dryS ../ens_driver_routines.F90 > cp process_cn.F90.dCO2SC_S ../process_cn.F90 > cp compute_rc.F90.dry ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/compute_rc.F90 > cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > /bin/cp -pr Linux exec/clm4.5_Kn11_dco2sc_dryS/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71308459 2019-10-15 12:03 ../build/Linux/bin/LDASsaCN_mpi.x* Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_dco2sc_dryS/clm4.5_DE720_dco2sc_dryS_1983/run > ./lenkf.0.j It's running and passed Apr 01. Stopped it at date_time_new 19830401_230730z. > /bin/rm ../output/global/rc_out/* > /bin/rm ../output/global/rc_out/Y1983/M01/* > qsub lenkf.0.j Restored the files: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp compute_rc.F90.restore compute_rc.F90 > cd ../../../GEOSlana_GridComp/ > cp process_cn.F90.restore process_cn.F90 > cp ens_driver_routines.F90.restore ens_driver_routines.F90 2. Processed and analyzed the output of clm4.5_DE720_ctrl_1983 and clm4.5_DE720_dco2sc_dryS_1983. > cd ~/Catchment/CLM4.5 > grid_restore_hdeg_drought clm4.5_DE720_ctrl 1983 > grid_restore_hdeg_drought clm4.5_DE720_dco2sc_dryS 1983 Analyzed the output of clm4.5_DE720 and clm4.5_DE720_dco2sc_dryS using compare_drought_ctrlVSexp.gs. 3. Processed and analyzed the output of clm4.5_DE720_PFT6Vcmax55 and clm4.5_DE720_fmleaf_CCcwdNlitter: > cd ~/Catchment/CLM4.5 > grid_restore_hdeg clm4.5_DE720_PFT6Vcmax55 > grid_restore_hdeg clm4.5_DE720_fmleaf_CCcwdNlitter > cp clm4.5_DE720.ctl clm4.5_DE720_PFT6Vcmax55.ctl > nedit clm4.5_DE720_PFT6Vcmax55.ctl & > cp clm4.5_DE720_PFT6Vcmax55.ctl clm4.5_DE720_fmleaf_CCcwdNlitter.ctl > nedit clm4.5_DE720_fmleaf_CCcwdNlitter.ctl & Analyzed the output using ~/Catchment/CLM4.5/check_fire_bug_fix.gs. ========== 20191016: 1. Read Bonan et al., 2012 to see why they use a higher Vcmax25 for PFT 4 (broadleaf evergreen tropical tree) that Kattge et al., 2009. 2. Checked the spinup of GEOSldas4.5_DE720. The first cycle was completed and post-processed. The second cycle was successfully submitted and it ran through the end of 2018, but its output was not post-processed and the next cycle was not set up. Tried submitting lenkf_submit_next_batch.j manually: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run > sbatch lenkf_submit_next_batch.j It's running and post-processing the output of the second cycle. I don't know what happened. Since I didn't provide a SLURM_JOB_ID, it may not be able to set up the next cycle. > cd /home/fzeng/Catchment/CLM4.5 > nedit submit_next_batch_GEOSldas_DE720 & #set num = ${SLURM_JOB_ID} set num = 03 #python $HOME/Catchment/CLM4.5/tile2grid_GEOSldas.py #if ($status != 0) exit 12 > set SLURM_JOB_ID = 03 > submit_next_batch_GEOSldas_DE720 GEOSldas4.5_DE720 > nedit submit_next_batch_GEOSldas_DE720 & Uncomment these lines: set num = ${SLURM_JOB_ID} python $HOME/Catchment/CLM4.5/tile2grid_GEOSldas.py if ($status != 0) exit 12 However, none of these are needed. It DID get the SLURM_JOB_ID and submit the next cycle. 3. For hindcast data bias correction: 2007, following notes in /discover/nobackup/fzeng/notes/bias_correct_hindcast (1) Bias correct monthly hindcast data. Done on 20191016. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2007 FCST_EYR=2007 > PART2_BCSD-Calc.H.sh (2) Downscale bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Done on 20191017. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2007 # SYR and EYR should be the same as FCST_EYR=2007 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh (3) Combine the individual fields into one file for each month. Done on 20191017. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2007 FCST_EYR=2007 > PART4_Combine.H.sh (4) Set up the 4 ensemble experiments. Done on 20191018. Take e0004s_transientCO2_BChindcast_YYYYJan01_ens1 as an example. Check if the restart file is available: > ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2007/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20070101_0000z (4.1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2007Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2007-01-01-00-00-00 end_time = 2007-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2007_ens1 (4.2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (4.3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2007Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-10-18 16:56 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4.4) Create year_co2.txt: > cd run > echo 2007 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2007 [Correct!] (4.5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (4.6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2016Jan01_ens1/run/LDAS.rc (4.7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (4.8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2007Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2007/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20070101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2007/jan01/ens1/GEOS5.200701.nc4 Stopped it at date_time_new 20070101_080730z. (4.9) Submit the job: > qsub lenkf.0.j (5) Process the four ensemble run output and check the output on GrADS. Done on 20191021. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2007Jan01_ens1 2007 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2007Jan01_ens2 2007 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2007Jan01_ens3 2007 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2007Jan01_ens4 2007 Checked July GPP and NEE of the 4 ensembles using e0004s_transientCO2_BChindcast_monthly.ctl on GrADS. Looks correct. 4. Deleted some files to release some disk space. /discover/nobackup/fzeng/Catchment/M2n5P/m0002 is more than 2TB! Checked and made sure the files to be deleted have been archived. > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002 > rm -rf CTco2_3hrly_dco2sc_L CTco2_3hrly_T2M_L CTco2_3hrly_PRCP_L CTco2_3hrly_dco2sc_dryL CTco2_3hrly_ctrl_L CTco2_3hrly_dco2sc_dryL_ctrl_d_rst Deleted the restart files in CTco2_3hrly_ctrl CTco2_3hrly_PRCP_S CTco2_3hrly_T2M_S CTco2_3hrly_dco2sc_S CTco2_3hrly_dco2sc_dryS by modifying and running /discover/nobackup/fzeng/Catchment/M2n5P/m0002/delete_rst. This released quite a lot of disk space. 5. Prepared meeting slides. ========== 20191017: 1. For hindcast data bias correction: 2008, following notes in /discover/nobackup/fzeng/notes/bias_correct_hindcast (1) Bias correct monthly hindcast data. Done on 20191017. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2008 FCST_EYR=2008 > PART2_BCSD-Calc.H.sh (2) Downscale bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Done on 20191017. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2008 # SYR and EYR should be the same as FCST_EYR=2008 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh (3) Combine the individual fields into one file for each month. Done on 20191018. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2008 FCST_EYR=2008 > PART4_Combine.H.sh (4) Set up the 4 ensemble experiments. Done on 20191018. Take e0004s_transientCO2_BChindcast_YYYYJan01_ens1 as an example. Check if the restart file is available: > ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2008/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20080101_0000z (4.1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2008Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2008-01-01-00-00-00 end_time = 2008-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2008_ens1 (4.2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (4.3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2008Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-10-18 17:23 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4.4) Create year_co2.txt: > cd run > echo 2008 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2008 [Correct!] (4.5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (4.6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2016Jan01_ens1/run/LDAS.rc (4.7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (4.8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2008Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2008/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20080101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2008/jan01/ens1/GEOS5.200801.nc4 Stopped it at date_time_new 20080101_060000z. (4.9) Submit the job: > qsub lenkf.0.j (5) Process the four ensemble run output and check the output on GrADS. Done on 20191021. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2008Jan01_ens1 2008 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2008Jan01_ens2 2008 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2008Jan01_ens3 2008 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2008Jan01_ens4 2008 Checked July GPP and NEE of the 4 ensembles using e0004s_transientCO2_BChindcast_monthly.ctl on GrADS. Looks correct. 2. The GEOSldas4.5_DE720 crashed because it couldn't find the restart files. The restart files are symbolic links to the previous cycles, and I already removed them from NOBACKUP after I archived them. Now need to copy these cycles back to NOBACKUP to resolve the issue. > cd ~/Catchment/CLM4.5 Modified /home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas_DE720 to not create symbolic links but copy the rst files from the previous cycle instead, so that removing the previous cycle after archiving it will not affect the current cycle. > cp -p submit_next_batch_GEOSldas_DE720 submit_next_batch_GEOSldas_DE720_tmp > nedit submit_next_batch_GEOSldas_DE720_tmp & > submit_next_batch_GEOSldas_DE720_tmp [to create the 4th cycle of GEOSldas4.5_DE720] Test run GEOSldas4.5_DE720: > interactive.py -A sp3 -n 168 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run > cp lenkf.j lenkf.j.orig > nedit lenkf.j & Add "exit" after L284 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > ./lenkf.j It's running. Stopped it at 1980-01-03T00:00:00. Submit the full jobs: > mv lenkf.j.orig lenkf.j > ./ldas_batchrun.j 3. Calculated 1980-2018 mean annual global GPP and NEE of the following cycles using ~/python/clm4-to-clm4.5/compare_ts_GEOSldas_DE720.py: GEOSldas4.5_DE720: 02, 03 4. Fire meeting 5. Proposal meeting 6. For hindcast data bias correction: 2009, following notes in /discover/nobackup/fzeng/notes/bias_correct_hindcast (1) Bias correct monthly hindcast data. Done on 20191017. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2009 FCST_EYR=2009 > PART2_BCSD-Calc.H.sh (2) Downscale bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Done on 20191018. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2009 # SYR and EYR should be the same as FCST_EYR=2009 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh (3) Combine the individual fields into one file for each month. Done on 20191018. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2009 FCST_EYR=2009 > PART4_Combine.H.sh (4) Set up the 4 ensemble experiments. Done on 20191021. Take e0004s_transientCO2_BChindcast_YYYYJan01_ens1 as an example. Check if the restart file is available: > ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2009/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20090101_0000z (4.1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2009Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2009-01-01-00-00-00 end_time = 2009-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2009_ens1 (4.2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (4.3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2009Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-10-21 10:47 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4.4) Create year_co2.txt: > cd run > echo 2009 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2009 [Correct!] (4.5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (4.6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2016Jan01_ens1/run/LDAS.rc (4.7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (4.8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2009Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2009/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20090101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2009/jan01/ens1/GEOS5.200901.nc4 Stopped it at date_time_new 20090101_060000z. (4.9) Submit the job: > qsub lenkf.0.j (5) Process the four ensemble run output and check the output on GrADS. Done on 20191023. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2009Jan01_ens1 2009 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2009Jan01_ens2 2009 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2009Jan01_ens3 2009 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2009Jan01_ens4 2009 Checked July GPP and NEE of the 4 ensembles using check_e0004s_transientCO2_BChindcast_monthly.gs on GrADS. Looks correct. ========== 20191018: 1. For hindcast data bias correction: 2010, following notes in /discover/nobackup/fzeng/notes/bias_correct_hindcast (1) Bias correct monthly hindcast data. Done on 20191018. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2010 FCST_EYR=2010 > PART2_BCSD-Calc.H.sh (2) Downscale bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Done on 20191021. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2010 # SYR and EYR should be the same as FCST_EYR=2010 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh (3) Combine the individual fields into one file for each month. Done on 20191021. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2010 FCST_EYR=2010 > PART4_Combine.H.sh (4) Set up the 4 ensemble experiments. Done on 20191023. Take e0004s_transientCO2_BChindcast_YYYYJan01_ens1 as an example. Check if the restart file is available: > ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2010/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20100101_0000z (4.1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2010Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2010-01-01-00-00-00 end_time = 2010-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2010_ens1 (4.2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (4.3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2010Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-10-23 10:57 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4.4) Create year_co2.txt: > cd run > echo 2010 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2010 [Correct!] (4.5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (4.6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2016Jan01_ens1/run/LDAS.rc (4.7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (4.8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2010Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2010/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20100101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2010/jan01/ens1/GEOS5.201001.nc4 Stopped it at date_time_new 20100101_060000z. (4.9) Submit the job: > qsub lenkf.0.j (5) Process the four ensemble run output and check the output on GrADS. Done on 20191025. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2010Jan01_ens1 2010 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2010Jan01_ens2 2010 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2010Jan01_ens3 2010 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2010Jan01_ens4 2010 Checked July GPP and NEE of the 4 ensembles using check_e0004s_transientCO2_BChindcast_monthly.gs on GrADS. Looks correct. 2. Read Eunjee's proposal 3. Fire experiment: changed FLNR from 0.0716 to 0.096 for type 6: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit CN_DriverMod.F90 & Changed FLNR from 0.0716 to 0.1138 for type 6. > nedit CNFireMod.F90 & Changed the btran lower threshold from 0.4 back to 0.3, and the btran upper threshold from 0.8 back to 0.7. spread_m = (1.0_r8 - max(0._r8,min(1._r8,(btran_col(c)/wtlf(c)-0.3_r8)/ & (0.7_r8-0.3_r8))))*(1.0-max(0._r8, & min(1._r8,(forc_rh(g)-30._r8)/(80._r8-30._r8)))) ! spread_m = (1.0_r8 - max(0._r8,min(1._r8,(btran_col(c)/wtlf(c)-0.4_r8)/ & ! (0.8_r8-0.4_r8))))*(1.0-max(0._r8, & ! min(1._r8,(forc_rh(g)-30._r8)/(80._r8-30._r8)))) > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_PFT6Vcmax65_origBtran > /bin/cp -pr Linux exec/clm4.5_Kn11_PFT6Vcmax65_origBtran/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5.bat D0.5_cn.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_PFT6Vcmax65_origBtran/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_PFT6Vcmax65_origBtran > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-18 12:47 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-18 12:58 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > cp ../../clm4.5_fire_DE720/run/ldas_batchrun.j . Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_PFT6Vcmax65_origBtran/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800102_030730z. > ./ldas_batchrun.j Restored the file: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp CN_DriverMod.F90.restore CN_DriverMod.F90 > cp CNFireMod.F90.restore CNFireMod.F90 4. Proposal meeting ========== 20191021: 1. For hindcast data bias correction: 2012, following notes in /discover/nobackup/fzeng/notes/bias_correct_hindcast (1) Bias correct monthly hindcast data. Done on 20191021. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2012 FCST_EYR=2012 > PART2_BCSD-Calc.H.sh (2) Downscale bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Done on 20191021. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2012 # SYR and EYR should be the same as FCST_EYR=2012 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh (3) Combine the individual fields into one file for each month. Done on 20191023. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2012 FCST_EYR=2012 > PART4_Combine.H.sh (4) Set up the 4 ensemble experiments. Done on 20191023. Take e0004s_transientCO2_BChindcast_YYYYJan01_ens1 as an example. Check if the restart file is available: > ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2012/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20120101_0000z (4.1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2012Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2012-01-01-00-00-00 end_time = 2012-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2012_ens1 (4.2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (4.3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2012Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-10-23 14:48 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4.4) Create year_co2.txt: > cd run > echo 2012 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2012 [Correct!] (4.5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (4.6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2016Jan01_ens1/run/LDAS.rc (4.7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (4.8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2012Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2012/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20120101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2012/jan01/ens1/GEOS5.201201.nc4 Stopped it at date_time_new 20120101_042230z. (4.9) Submit the job: > qsub lenkf.0.j (5) Process the four ensemble run output and check the output on GrADS. Done on 20191025. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2012Jan01_ens1 2012 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2012Jan01_ens2 2012 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2012Jan01_ens3 2012 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2012Jan01_ens4 2012 Checked July GPP and NEE of the 4 ensembles using check_e0004s_transientCO2_BChindcast_monthly.gs on GrADS. Looks correct. 2. Processed and analyzed the output of clm4.5_DE720_PFT6Vcmax65_origBtran: > cd ~/Catchment/CLM4.5 > grid_restore_hdeg clm4.5_DE720_PFT6Vcmax65_origBtran > cp clm4.5_DE720.ctl clm4.5_DE720_PFT6Vcmax65_origBtran.ctl > nedit clm4.5_DE720_PFT6Vcmax65_origBtran.ctl & Analyzed the output using ~/Catchment/CLM4.5/check_fire_bug_fix.gs. 3. For hindcast data bias correction: 2013, following notes in /discover/nobackup/fzeng/notes/bias_correct_hindcast (1) Bias correct monthly hindcast data. Done on 20191021. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2013 FCST_EYR=2013 > PART2_BCSD-Calc.H.sh (2) Downscale bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Done on 20191023. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2013 # SYR and EYR should be the same as FCST_EYR=2013 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh (3) Combine the individual fields into one file for each month. Done on 20191023. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2013 FCST_EYR=2013 > PART4_Combine.H.sh (4) Set up the 4 ensemble experiments. Done on 20191024. Take e0004s_transientCO2_BChindcast_YYYYJan01_ens1 as an example. Check if the restart file is available: > ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2013/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20130101_0000z (4.1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2013Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2013-01-01-00-00-00 end_time = 2013-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2013_ens1 (4.2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (4.3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2013Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-10-24 16:28 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4.4) Create year_co2.txt: > cd run > echo 2013 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2013 [Correct!] (4.5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (4.6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2016Jan01_ens1/run/LDAS.rc (4.7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (4.8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2013Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2013/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20130101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2013/jan01/ens1/GEOS5.201301.nc4 Stopped it at date_time_new 20130101_030000z. (4.9) Submit the job: > qsub lenkf.0.j (5) Process the four ensemble run output and check the output on GrADS. Done on 20191025. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2013Jan01_ens1 2013 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2013Jan01_ens2 2013 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2013Jan01_ens3 2013 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2013Jan01_ens4 2013 Checked July GPP and NEE of the 4 ensembles using check_e0004s_transientCO2_BChindcast_monthly.gs on GrADS. Looks correct. ========== 20191022: 1. Modified ~/Catchment/CLM4.5/check_fire_bug_fix.gs and used it to analyze the output of clm4.5_DE720_PFT6Vcmax55 and clm4.5_DE720_PFT6Vcmax65_origBtran. ========== 20191023: 1. Continued analyzing the output of clm4.5_DE720_PFT6Vcmax55 and clm4.5_DE720_PFT6Vcmax65_origBtran using ~/Catchment/CLM4.5/check_fire_bug_fix.gs and ~/Catchment/CLM4.5/check_output.gs. 2. Fire experiment: try turning off fire to see the effect of vcmax25top on GPP (1) Turned off fire without any other change: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp -p clm_varctl.F90 clm_varctl.F90.restore > nedit clm_varctl.F90 & ! logical, public :: use_nofire = .false. logical, public :: use_nofire = .true. > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_nofire > /bin/cp -pr Linux exec/clm4.5_Kn11_nofire/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5.bat D0.5_cn.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_nofire/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_nofire > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-23 14:20 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-23 14:25 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > cp ../../clm4.5_fire_DE720/run/ldas_batchrun.j . Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_nofire/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800101_230730z. > ./ldas_batchrun.j (2) Turned off fire and changed FLNR from 0.0716 to 0.1138 for type 6: With the modification to clm_varctl.F90 above to turn off fire: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit CN_DriverMod.F90 & Changed FLNR from 0.0716 to 0.1138 for type 6. > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_nofire_PFT6Vcmax65 > /bin/cp -pr Linux exec/clm4.5_Kn11_nofire_PFT6Vcmax65/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5.bat D0.5_cn.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_nofire_PFT6Vcmax65/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_nofire_PFT6Vcmax65 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-10-23 14:35 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-23 14:39 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > cp ../../clm4.5_fire_DE720/run/ldas_batchrun.j . Do an interactive run to make sure it runs: > interactive.py -A sp3 -n 308 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_nofire_PFT6Vcmax65/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800101_190730z. > ./ldas_batchrun.j Restored the files and compile: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp clm_varctl.F90.restore clm_varctl.F90 > cp CN_DriverMod.F90.restore CN_DriverMod.F90 > gmake install ========== 20191024: 1. Created ~/python/clm4-to-clm4.5/plot_gpp_bias.py and used it to plot the GPP biases. 2. Prepared slides for the fire meeting. 3. Fire meeting. ========== 20191025 and 20191028: this method is obsolete. Moved to the new method. See notes on 20191029. 1. Created 19940101, 20000101 and 20120101 restart files for Huisheng. Referred to notes on 20170808 and 20170822. The 19940101 restart file I created for Huisheng on 20170822 used the BCs files in Sarith's directory which no longer exists. Now use the BCs in Larry's directory to create a new one. > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng > mv Icarus-NL_Reynolds_CF90 Icarus-NL_Reynolds_CF90_20170822 > mv SMAP_EASEv2_M09_reordered SMAP_EASEv2_M09_reordered_20170808 Talked to Sarith. Sarith copied a few files from /discover/nobackup/smahanam/GEOS-ESM/GEOSgcm/src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ to /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/: SaltImpConverter.F90 SaltIntSplitter.F90 Scale_Catch.F90 Scale_CatchCN.F90 mk_CICERestart.F90 mk_CatchCNRestarts.F90 mk_CatchRestarts.F90 mk_LDASsaRestarts.F90 mk_LakeLandiceSaltRestarts.F90 (1) Reorder: (1.1) Create path: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > mkdir -p rst_Huisheng/SMAP_EASEv2_M09_reordered (1.2) Create shell script "reorder_restarts" for reordering: > cd rst_Huisheng/SMAP_EASEv2_M09_reordered > nedit reorder_restarts & reorder_restarts reads: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/bin/csh set YYYY = 2000 set YYYYMMDD=`echo ${YYYY}0101` mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/ -d ${YYYYMMDD} -e e0004s_transientCO2 -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/ -m catchcn -s 50 -r Y -t SMAP_EASEv2_M09_3856x1624.til ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > chmod 755 reorder_restarts (1.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: > cp /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 . > nedit mk_LDASsaRestarts.F90 & [NO changes made!] > cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. [skipped here] > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ > source $ESMADIR/src/g5_modules > gmake install (1.4) Run "reorder_restarts" to reorder the offline restart file: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/SMAP_EASEv2_M09_reordered > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/Linux/bin > reorder_restarts (2) Regrid from SMAP_EASEv2_M09 to Huisheng's AGCM grid/resolution: (2.1) Create path: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng > mkdir Icarus-NLv3_Reynolds_CF90 (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: > cd Icarus-NLv3_Reynolds_CF90 > cp ../SMAP_EASEv2_M09_reordered/mk_LDASsaRestarts.F90 . > nedit mk_LDASsaRestarts.F90 & L67-68: InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.20000101', & InCNTilFile = '/discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til', & Add to L210: write(10,'(a)')'#SBATCH --qos=debug' L229-230: write(10,'(a)')'bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst '//trim(SFL)//' 13 13' if(clm45) write(10,'(a)')'bin/Scale_CatchCN OutData1/catchcn_internal_clm45 OutData2/catchcn_internal_clm45 catchcn_internal_clm45 '//trim(SFL)//' 13 13' Uncomment L779: if(MOD(n,10000) == 0) print *,'In HYD', myid,n Uncomment L1618: if(MOD(n,10000) == 0) print *,'In Carbon', myid,n Add to L2210-2225: VAR_DUM = 0.1 do i = 1,4 STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'WW'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) end do VAR_DUM = 0.25 do i = 1,4 STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'FR'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) end do VAR_DUM = 0.001 do i = 1,4 STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CH'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CM'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CQ'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) end do > cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ > source $ESMADIR/src/g5_modules > gmake install (2.3) Run mk_LDASsaRestarts to convert to desired tile system: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NLv3_Reynolds_CF90 > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/Linux/bin > mpirun -np 1 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_Reynolds/CF0090x6C_DE0360xPE0180/ -m catchcn -s 50 -t CF0090x6C_DE0360xPE0180-Pfafstetter.til > sbatch mkLDASsa.j ========== 20191028: 1. Continued 1 on 20191025. 2. To use ncdiff, trying "module load other/comp/gcc-5.3-sp3" before using it. ========== 20191029 to 20191031: 1. Sarith told me to use the new method to create restart files for Huisheng's AGCM runs. There is no ndvi_clim_3856x1624_DE.data in Larry's bcs /discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/, so do this: > cd /discover/nobackup/fzeng/bcs > mkdir -p Heracles-NL/SMAP_EASEv2_M09 > cd Heracles-NL/SMAP_EASEv2_M09 > nedit create_symbolic_links & > chmod 755 create_symbolic_links > create_symbolic_links > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2000/M01 > ln -s e0004s_transientCO2.ens0000.catchcn_ldas_rst.20000101_0000z e0004s_transientCO2.catchcn_internal_rst.20000101_0000 (1) Reorder: > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > cp GEOSldas4.5_DE720.exe rst_Huisheng.exe > cp GEOSldas4.5_DE720.bat rst_Huisheng.bat > nedit rst_Huisheng.exe & EXP_ID : rst_Huisheng_reorder BEG_DATE : 20000101 000000 RESTART : 1 RESTART_ID : e0004s_transientCO2 RESTART_PATH : /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ RESTART_DOMAIN : SMAP_EASEv2_M09_GLOBAL BCS_PATH : /discover/nobackup/fzeng/bcs/Heracles-NL/SMAP_EASEv2_M09/ Run ldas_setup to reorder the SMAP_EASEv2_M09 restart file: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/install/bin > source g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/rst_Huisheng.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/rst_Huisheng.bat The reordered M09 restart file is: /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_reorder/output/global/rs/ens0000/Y2000/M01/rst_Huisheng_reorder.catchcn_internal_rst.20000101_0000 Check whether the reordered restart file created using the new method is the same as the one created using the old method. This only works for 20000101 rst. > cd /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_reorder/output/global/rs/ens0000/Y2000/M01 > ~smahanam/bin/cdo -diffn rst_Huisheng_reorder.catchcn_internal_rst.20000101_0000 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.20000101 cdo diffn: Processed 3847911900 values from 160 variables over 2 timesteps ( 85.01s ) So they are the same. Great! (2) Regrid: This preparation is one-time thing. No need to do it again. Copied a few files from /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ to /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts: SaltImpConverter.F90 SaltIntSplitter.F90 Scale_Catch.F90 Scale_CatchCN.F90 mk_CICERestart.F90 mk_CatchCNRestarts.F90 mk_CatchRestarts.F90 mk_LDASsaRestarts.F90 mk_LakeLandiceSaltRestarts.F90 These ones are for git! Although these files were originally copied from /discover/nobackup/smahanam/GEOS-ESM/GEOSgcm/src/Components/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ on Monday Oct 28, the current ones in Sarith's directory were modified by Sarith on Oct 30 and they are not for git! > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > mv mk_LDASsaRestarts.F90 mk_LDASsaRestarts.F90.v1 > mv SaltImpConverter.F90 SaltImpConverter.F90.orig > mv SaltIntSplitter.F90 SaltIntSplitter.F90.orig > mv Scale_Catch.F90 Scale_Catch.F90.orig > mv Scale_CatchCN.F90 Scale_CatchCN.F90.orig > mv mk_CICERestart.F90 mk_CICERestart.F90.orig > mv mk_CatchCNRestarts.F90 mk_CatchCNRestarts.F90.orig > mv mk_CatchRestarts.F90 mk_CatchRestarts.F90.orig > mv mk_LDASsaRestarts.F90 mk_LDASsaRestarts.F90.orig [skipped because it's already mk_LDASsaRestarts.F90.orig] > mv mk_LakeLandiceSaltRestarts.F90 mk_LakeLandiceSaltRestarts.F90.orig > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ > cp -p SaltImpConverter.F90 SaltIntSplitter.F90 Scale_Catch.F90 Scale_CatchCN.F90 mk_CICERestart.F90 mk_CatchCNRestarts.F90 mk_CatchRestarts.F90 mk_LDASsaRestarts.F90 mk_LakeLandiceSaltRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. Also copy getids.H from Sarith's nobackup: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > mv getids.H getids.H.orig > cp -p /discover/nobackup/smahanam/getids.H . (2.1) Modify mk_LDASsaRestarts.F90: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > nedit mk_LDASsaRestarts.F90 & L67-68: InCNRestart = '/discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_reorder/output/global/rs/ens0000/Y2000/M01/rst_Huisheng_reorder.catchcn_internal_rst.20000101_0000', & InCNTilFile = '/discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til', & L230-231: write(10,'(a)')'bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst '//trim(SFL)// if(clm45) write(10,'(a)')'bin/Scale_CatchCN OutData1/catchcn_internal_clm45 OutData2/catchcn_internal_clm45 catchcn_internal_clm45 '//trim(SFL)// Add to L2211-2226: VAR_DUM = 0.1 do i = 1,4 STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'WW'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) end do VAR_DUM = 0.25 do i = 1,4 STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'FR'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) end do VAR_DUM = 0.001 do i = 1,4 STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CH'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CM'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CQ'), (/1,i/), (/NTILES,1 /),VAR_DUM(:)) end do (2.2) Modify process_rst.csh: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Applications/LDAS_App > cp -p process_rst.csh process_rst.csh.orig > nedit process_rst.csh & L93: if($LSM_CHOICE == 2) sed -i '$ a\bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 13 13 \' mkLDASsa.j (2.3) Have to build the whole model: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas > ./parallel_build.csh [used the default (c) clean: runs "make clean"] (2.4) Check and make sure the code is compiled succssfully. > ls -l install/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 66173350 2019-10-31 11:52 install/bin/GEOSldas.x* (2.5) Modify the .exe file: > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > nedit rst_Huisheng.exe & EXP_ID : rst_Huisheng_regrid RESTART : 0 BCS_PATH : /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_Reynolds/CF0090x6C_DE0360xPE0180/ (2.6) Run ldas_setup: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/install/bin > source g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/rst_Huisheng.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/rst_Huisheng.bat creating dir structure creating restart and bc Correct the tile file if it is an old EASE tile format... cmd: ./preprocess_ldas.x correctease /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/output/global/rc_out//CF0090x6C_DE0360xPE0180-Pfafstetter.til /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/output/global/rc_out//MAPL_CF0090x6C_DE0360xPE0180-Pfafstetter.til Creating f2g.txt.... cmd: ./preprocess_ldas.x c_f2g /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/output/global/rc_out//CF0090x6C_DE0360xPE0180-Pfafstetter.til LDAS_domain_def.nml /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/output/global /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_Reynolds/CF0090x6C_DE0360xPE0180/clsm/catchment.def rst_Huisheng_regrid 200001010000 50.0 Creating domain..., reading white and black lists if there have ones... Finish domain setup... SURFLAY: 50.00000 Writing catparam file : /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_r egrid/output/global/rc_out///Y2000/M01/rst_Huisheng_regrid.ldas_catparam.200001 01_0000z.bin linking bcs... Creating and lining restart... cmd: ./process_rst.csh g0620 rst_Huisheng_regrid /discover/nobackup/fzeng/Catchment/M2n5P /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_Reynolds/CF0090x6C_DE0360xPE0180/ CF0090x6C_DE0360xPE0180-Pfafstetter.til 2 0 20000101 none none none/ 1 0 ######################################################## NOTE: IMPORTANT IMPORTANT IMPROTANT In the absence of a restart file, a global catch(cn)_internal_rst file is being produced by regridding SMAP_Nature_v05 restarts from SMAP 9km grid to the BCS grid. The start date will be forced January 1st. PLEASE SPIN UP BEFORE USE ######################################################## Submitted batch job 35467350 Please hold on for a while until the restart file is created ..... - |restart: 0 catchRstFile1: /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/mk_restarts/catchcn_internal_rst.20000101 catchRstFile: /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/output/global/rs/ens0000/Y2000/M01/rst_Huisheng_regrid.catchcn_internal_rst.20000101_0000 Updating restart path... creating RC Files Optimizing... decomposition of processes.... cmd: ./preprocess_ldas.x optimize /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/input/tile.data 168 total tiles 475330 ... (2.7) Check the restart file created: > cd /discover/nobackup/fzeng/Catchment/M2n5P/rst_Huisheng_regrid/mk_restarts Checked mkLDAS.o and mkLDAS.e to make sure there is no error message. This only works for 20000101 rst. > ~smahanam/bin/cdo -diffn catchcn_internal_rst /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NLv3_Reynolds_CF90/catchcn_internal_rst.20000101 cdo diffn: Processed 1085653720 values from 160 variables over 2 timesteps ( 28.01s ) So they are the same. Great! (2.8) Rename the experiment directories and save some files: > cd /discover/nobackup/fzeng/Catchment/M2n5P > mv rst_Huisheng_reorder rst_Huisheng_reorder_20000101 > mv rst_Huisheng_regrid rst_Huisheng_regrid_20000101 > cd rst_Huisheng_regrid_20000101 > cp -p /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 . 2. Saved the new method in notes. > cd /discover/nobackup/fzeng/notes > mv convert_rst_offline2AGCM convert_rst_offline2AGCM_old > nedit convert_rst_offline2AGCM & 3. Met with Sarith and Huisheng almost the entire day. Finally we resolved Huisheng's AGCM issue. The new land updates are good. It's Huisheng's modifications that caused the crash. ========== 20191030: 1. Continued working on regridding in 1 on 20191029. 2. Talked to Randy. 3. Set up some sensitivity experiments: point runs (1) Control: Use the right executable: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_point > rm build > ln -s /gpfsm/dnb31/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11 build > ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_point/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-09-04 15:54 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_point/build/Linux/bin/LDASsaCN_mpi.x* > ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263362 2019-09-04 15:54 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x* Same. Good! Use the right restart file: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_point/input/restart > rm catchcn_internal_rst > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output Set the domain: > cd ../../run > nedit driver_inputs_0.5D_point.nml & out_select_ensavg%tile%inst = .false. out_select_ensavg%tile%xhourly = .false. out_select_ensavg%tile%daily = .false. out_select_ensavg%tile%pentad = .false. out_select_ensavg%tile%monthly = .true. minlon = 30 ! min longitude maxlon = 30.5 ! max longitude minlat = -10.5 ! min latitude maxlat = -10 ! max latitude Set the simulation period: > nedit lenkf.0.j & -start_year 1980 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 1991 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -restart_path ../input/restart/output \ -restart_id clm4.5_DE720 \ -restart_domain global \ Clear some files: > /bin/rm ../output/point/rc_out/* > /bin/rm ../output/point/rc_out/Y1980/M01/* Do an interactive run to make sure it runs: > ./lenkf.0.j It's running. Stopped it at date_time_new 19800101_230730z. > sbatch lenkf.0.j (2) Increase PFT 6 vcmax25top by 1%: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit compute_rc.F90 & Added to L434: print *, 'rc (including rb):', rc, 'rb:', rb > nedit CN_DriverMod.F90 & Change flnr of PFT 6 from 0.0716 to 0.0723 > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_PFT6VcmaxUp1pct > /bin/cp -pr Linux exec/clm4.5_Kn11_PFT6VcmaxUp1pct/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > cp -p ../clm4.5_point/run/driver_inputs_0.5D_point.nml . > nedit D0.5_clm4.5_point.bat D0.5_clm4.5_point.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_PFT6VcmaxUp1pct/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_clm4.5_point.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_clm4.5_point.bat --runmodel --monthsperjob 120 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_PFT6VcmaxUp1pct_point > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263758 2019-10-30 15:19 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-30 15:30 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800102_020730z. > sbatch lenkf.0.j (3) Decrease PFT 6 vcmax25top by 1%: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit CN_DriverMod.F90 & Change flnr of PFT 6 from 0.0716 to 0.0709 > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_PFT6VcmaxDn1pct > /bin/cp -pr Linux exec/clm4.5_Kn11_PFT6VcmaxDn1pct/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5_clm4.5_point.bat D0.5_clm4.5_point.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_PFT6VcmaxDn1pct/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_clm4.5_point.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_clm4.5_point.bat --runmodel --monthsperjob 120 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_PFT6VcmaxDn1pct_point > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263758 2019-10-30 15:41 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-30 15:30 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800102_170730z. > sbatch lenkf.0.j Restored CN_DriverMod.F90: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp CN_DriverMod.F90.restore CN_DriverMod.F90 (4) Change both terms 1.4 and 1.6 which are the ratios of diffusivity of CO2 to H2O for the leaf boundary layer resistance and stomatal resistance to 5: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit compute_rc.F90 & subroutine Photosynthesis: !cs = cair(p) - 1.4_r8/gb_mol(p) * an(p,iv) * forc_pbot(g) cs = cair(p) - 5._r8/gb_mol(p) * an(p,iv) * forc_pbot(g) !ci_z(p,iv) = cair(p) - an(p,iv) * forc_pbot(g) * (1.4_r8*gs_mol(p,iv)+1.6_r8*gb_mol(p)) / (gb_mol(p)*gs_mol(p,iv)) ci_z(p,iv) = cair(p) - an(p,iv) * forc_pbot(g) * (5._r8*gs_mol(p,iv)+5._r8*gb_mol(p)) / (gb_mol(p)*gs_mol(p,iv)) subroutine ci_func: !cs = cair - 1.4_r8/gb_mol * an * forc_pbott cs = cair - 5._r8/gb_mol * an * forc_pbot !fval = ci - cair + an * forc_pbot * (1.4_r8*gs_mol+1.6_r8*gb_mol) / (gb_mol*gs_mol) fval = ci - cair + an * forc_pbot * (5._r8*gs_mol+5._r8*gb_mol) / (gb_mol*gs_mol) > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05 > source $ESMADIR/src/g5_modules > gmake install > cd ../../../GEOSlana_GridComp > gmake install > cd ../../Applications/LDAS_App/ > gmake install > cd ../../.. > mkdir -p exec/clm4.5_Kn11_CO2_H2O_5 > /bin/cp -pr Linux exec/clm4.5_Kn11_CO2_H2O_5/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > nedit D0.5_clm4.5_point.bat D0.5_clm4.5_point.exe & > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec/clm4.5_Kn11_CO2_H2O_5/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_clm4.5_point.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_clm4.5_point.bat --runmodel --monthsperjob 120 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_CO2_H2O_5_point > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 71263758 2019-10-30 16:04 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 80 2019-10-30 16:06 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_66/output/ > cd run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800103_000730z. > sbatch lenkf.0.j Restored compute_rc.F90: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > cp compute_rc.F90.restore compute_rc.F90 > xdiff compute_rc.F90.restore compute_rc.F90 & Identical. Good. > xdiff CN_DriverMod.F90 CN_DriverMod.F90.restore & Identical. Good. ========== 20191031: 1. Analyzed the output of the experiments set up on 20191030 using ~/Catchment/CLM4.5/check_output.gs.