========== 20161201: 1. Checked the output of the corrected M2.n5P.mmco2.glb and made sure that it differs from that of the M2.n5P.mco2.glb run. 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161202: 1. November report. 2. Discover dnb42 is still unavailable. Can't create restart file for offline test runs. Read CLM4.5 Tech Note and Bonan et al., JGR 2011. ========== 20161205: 1. The M2.n5P.mmco2.glb run stopped at Friday night (2016-12-02 20:25) when it's simulating date_time_new 20140120_001000z. No error message or any information about why it stopped is given in the exp log file. dali15:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-12-02 20:13 CN_restart dali15:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb > ls -l RUN/rs/ens0000/Y2014/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-12-02 20:13 mM2.n5P.mmco2.glb.ens0000.catch_ldas_rst.20140101_0000z.bin So the CN_restart is for 20140101. dali15:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb > ls -l RUN/cat/ens_avg/Y2013/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-12-02 20:13 mM2.n5P.mmco2.glb.ens_avg.ldas_tile_monthly_out.201312.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb > ls -1 RUN/rs/ens0000/*/*/*.bin | tail -1 | cut -c17-20 2014 The lenkf.j can detect the last year when land restart file of Jan 01 was written out. So just need to submit lenkf.j again to make the simulation resume from 20140101. cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb cp -p CN_restart CN_restart_20140101 (keep a copy just in case) qsub lenkf.j It's running. Checked the exp log file. It started from date_time_new 20140101_001000z. Correct. Processed the output: ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.mmco2.glb 2. The CF0180 run also stopped about the same time (2016-12-02 20:29) when it's simulating date_time_new 20000815_020730z. No error message or any information about why it stopped is given in the exp log file. dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C > ls -l output/CF0180x6C_GLOBAL/rs/ens0000/Y2000/M08 total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-12-02 20:15 CF0180x6C.ens0000.catchcn_ldas_rst.20000801_0000z dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL > ls -l cat/ens_avg/Y2000/M07 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-12-02 20:15 CF0180x6C.ens_avg.ldas_tile_monthly_out.200007.bin cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/run cp lenkf.10.j lenkf.10new.j nedit lenkf.10new.j These lines become: #SBATCH --output=../output/CF0180x6C_GLOBAL/rc_out/Y2000/M08/CF0180x6C.ldas_log.20000801_0000.txt #SBATCH --error=../output/CF0180x6C_GLOBAL/rc_out/Y2000/M08/CF0180x6C.ldas_out.20000801_0000.txt -start_year 2000 -start_month 8 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ qsub lenkf.10new.j It's running. Checked the exp log file. It started from date_time_new 20000801_000730z. Correct. Earlier when I tried to open lenkf.10.j, I forgot to type "nedit" and it became "lenkf.10.j &" and started the run from 20000101 in the background. Didn't notice it's still running in the background until after the run above starting from 20000801 began to run. Did this to stop it. xterm & (to open a new terminal window on the same node) ps -u (to show a list of jobs under my account) find the job number for "mpirun ..." and then kill it using "kill jobnumber". Although this unexpected run hasn't messed up the restart files or the output, it has modified some files under rc_out/Y2000/M01 (see below). dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rc_out/Y2000/M01 > ls -ltr total 239392 -rw-r--r-- 1 fzeng g0620 21928 2016-12-02 17:37 CF0180x6C.ldas_out.20000101_0000.txt -rw-r--r-- 1 fzeng g0620 27254728 2016-12-02 20:29 CF0180x6C.ldas_log.20000101_0000.txt -rw-r--r-- 1 fzeng g0620 5798 2016-12-05 11:00 CF0180x6C.ldas_driver_inputs.20000101_0000z.nml -rw-r--r-- 1 fzeng g0620 11588 2016-12-05 11:00 CF0180x6C.ldas_ensprop_inputs.20000101_0000z.nml -rw-r--r-- 1 fzeng g0620 17161213 2016-12-05 11:02 CF0180x6C.ldas_domdecomp.20000101_0000z.txt -rw-r--r-- 1 fzeng g0620 142781600 2016-12-05 11:05 CF0180x6C.ldas_catparam.20000101_0000z.bin -rw-r--r-- 1 fzeng g0620 49424412 2016-12-05 11:05 CF0180x6C.ldas_mwRTMparam.20000101_0000z.bin Decided to resume the run from 20000101. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rc_out/Y2000/M01 /bin/rm * cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/run qsub lenkf.10.j It's running. It started from date_time_new 20000101_000730z. Correct. 3. The e0004s_preindustrialCO2 also stopped about the same time (2016-12-02 20:24) when it's simulating date_time_new 20050708_020730z. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rc_out > ls -l ../rs/ens0000/Y2005/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-12-02 14:19 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20050101_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2005 > ls -l M03 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-12-02 16:05 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20050301_0000z Since the last restart file written out is for 20050301. Have to restart from 20050301. qsub lenkf.5.j It's running. It started from date_time_new 20050301_000730z. Correct. 4. Updated higgins rain data through Dec 4. 5. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161206: 1. Processed GMAO GEOS5 seasonal forecast for December. 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161207: 1. Calculated global GPP monthly and annual time series for the 6 CO2 experiments for Eunjee. /home/fzeng/Catchment/M2n5P_m0001/calculate_global_sum_monthly.f90 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161208: 1. Calculated global NEE monthly and annual time series for the 6 CO2 experiments for Eunjee. /home/fzeng/Catchment/M2n5P_m0001/calculate_global_sum_monthly.f90 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161209: 1. Due to some issue on Discover, the e0004s_preindustrialCO2 run stopped at 2016-12-08 18:52 when it's simulating 20110731_230730z. Luckily the CF0180 run had finished the 10th cycle by the time the issue occurred. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2011/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-12-08 12:25 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20110101_0000z Need to start the run from 20110101. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/run qsub lenkf.12.j (lenkf.12.j starts from 20110101) It's running. 2. The CF0180 run finished the 10th cycle. (1) Checked and processed the output: dali15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-12-08 18:09 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-12-08 18:09 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_10 cd CF0180x6C_10 rm -rf lenkf_orig cd output/CF0180x6C_GLOBAL/rs/ens0000/ mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec mv CF0180x6C CF0180x6C_01to10 mkdir CF0180x6C (2) Modified the code to make it write out additional restart files for 00z Apr 16 every year (Yehui wants restart files for 00z Jul 01 and 00z Apr 16 each year. See his emails on Dec 2, 2016): Only these two routines were modified earlier for CF0180: clsm_ensdrv_out_routines.F90 and tile_coord.F90. The current (updated) tile_coord.F90 includes the CF0180 tile structure, so no need to modify. Modified clsm_ensdrv_out_routines.F90: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp cp CF0180x6C/clsm_ensdrv_out_routines.F90 . mv CF0180x6C/clsm_ensdrv_out_routines.F90 CF0180x6C/clsm_ensdrv_out_routines.F90.01to10 nedit clsm_ensdrv_out_routines.F90 & cp -p clsm_ensdrv_out_routines.F90 CF0180x6C/. Made sure that process_cn.F90 and compute_rc.F90 are the default ones. Compiled: setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/ source $ESMADIR/src/g5_modules "gmake clean" and "gmake install" under GEOScatchCN_GridComp, GEOSlana_GridComp and Applications/LDAS_App. (Need to "gmake clean" and "gmake install" under GEOScatchCN_GridComp here because it's not compiled after the default compute_rc.F90 was put back.) cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec cp -pr ../Linux CF0180x6C/. ls -l CF0180x6C/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69784076 2016-12-09 12:09 CF0180x6C/Linux/bin/LDASsaCN_mpi.x (3) Set up the 11th cycle run. cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C mkdir rst_Yehui_11 cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../CF0180x6C_09/run/edit_lenkf.csh . nedit edit_lenkf.csh & (make it copy the restart files before deleting them and then add 'qsub lenkf.N+1.j') ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j Checked the output restart files. Restart file for 19800416_0000z was written out. Good! 3. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161212: 1. Found a bug in edit_lenkf.csh that I used to modify the job scripts in the current 11th CF0180x6C cycle. Therefore, the restart files were not copied to rst_Yehui_11 before they were removed. Set up the 11th cycle run again. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C rm -rf CF0180x6C cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../edit_lenkf.csh . nedit edit_lenkf.csh & (make it copy the restart files before deleting them and then add 'qsub lenkf.N+1.j') ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j Checked the output restart files. Restart file for 19800416_0000z was written out. Good! /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M04 > diff CF0180x6C.ens0000.catchcn_ldas_rst.19800416_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/rst_backup/CF0180x6C.ens0000.catchcn_ldas_rst.19800416_0000z Files CF0180x6C.ens0000.catchcn_ldas_rst.19800416_0000z and /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/rst_backup/CF0180x6C.ens0000.catchcn_ldas_rst.19800416_0000z differ diff CF0180x6C.ens0000.catchcn_ldas_rst.19800701_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/rst_backup/CF0180x6C.ens0000.catchcn_ldas_rst.19800701_0000z Files CF0180x6C.ens0000.catchcn_ldas_rst.19800701_0000z and /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/rst_backup/CF0180x6C.ens0000.catchcn_ldas_rst.19800701_0000z differ /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > diff CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/CF0180x6C_GLOBAL/rs/ens0000/Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z Files CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z and /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/CF0180x6C_GLOBAL/rs/ens0000/Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z differ Why? Set up another run repeating the 11th cycle above and see if I get restart files identical to the above 11th cycle. cd /discover/nobackup/fzeng/Catchment/M2n5P/run cp CF0180x6C_CN.exe CF0180x6C_CN_check.exe cp CF0180x6C.bat CF0180x6C_check.bat nedit CF0180x6C_CN_check.exe CF0180x6C_check.bat & CF0180x6C_CN_check.exe: exp_id = CF0180x6C_check restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/ restart_id = CF0180x6C CF0180x6C_check.bat: job-name = CF0180x6C_check cd /discover/nobackup/fzeng/Catchment/M2n5P/ mkdir CF0180x6C_check cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN_check.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_check.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../../CF0180x6C/CF0180x6C/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > diff CF0180x6C_check.ens0000.catchcn_ldas_rst.19800101_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/CF0180x6C_GLOBAL/rs/ens0000/Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z Files CF0180x6C_check.ens0000.catchcn_ldas_rst.19800101_0000z and /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/CF0180x6C_GLOBAL/rs/ens0000/Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z differ dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > diff CF0180x6C_check.ens0000.catchcn_ldas_rst.19800101_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z Files CF0180x6C_check.ens0000.catchcn_ldas_rst.19800101_0000z and /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z differ /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01 > diff CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198001.bin /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C.ens_avg.ldas_tile_monthly_out.198001.bin Return nothing. Good!! /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M02 > diff CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198002.bin /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M02/CF0180x6C.ens_avg.ldas_tile_monthly_out.198002.bin Return nothing. Good!! Asked Sarith for help. He suggested me to try ~smahanam/bin/cdo diffn file1 file2 /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > ~smahanam/bin/cdo diffn CF0180x6C_check.ens0000.catchcn_ldas_rst.19800101_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 1 of 84 records differ cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 46.73s ) /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M02 > ~smahanam/bin/cdo diffn CF0180x6C_check.ens0000.catchcn_ldas_rst.19800201_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M02/CF0180x6C.ens0000.catchcn_ldas_rst.19800201_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 32 : 2001-01-02 21:00:00 0 609565824 0 2186 : F T 4.3945e-42 0.16969 : CNPFT 2 of 84 records differ 1 of 84 records differ more than 0.001 cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 45.57s ) /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M03 > ~smahanam/bin/cdo diffn CF0180x6C_check.ens0000.catchcn_ldas_rst.19800301_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M03/CF0180x6C.ens0000.catchcn_ldas_rst.19800301_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 32 : 2001-01-02 21:00:00 0 609565824 0 3930 : F T 8.8786e-42 0.00037739 : CNPFT 2 of 84 records differ 1 of 84 records differ more than 0.001 cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 45.87s ) /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M04 > ~smahanam/bin/cdo diffn CF0180x6C.ens0000.catchcn_ldas_rst.19800416_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/rst_backup/CF0180x6C.ens0000.catchcn_ldas_rst.19800416_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 32 : 2001-01-02 21:00:00 0 609565824 0 45267310 : T T 1.8334e+09 1.0000 : CNPFT 2 of 84 records differ cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 45.77s ) "~smahanam/bin/cdo diffv" gives exactly the same results as "~smahanam/bin/cdo diffn" above. Found this MATLAB function https://www.mathworks.com/matlabcentral/fileexchange/47857-comparing-two-netcdf-files/content//nccmp.m. Saved a copy in /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > cp -p CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z CF0180x6C_ens0000_catchcn_ldas_rst_19800101_0000z.nc Did "diff" to make sure that they are identical. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > cp -p CF0180x6C_check.ens0000.catchcn_ldas_rst.19800101_0000z CF0180x6C_check_ens0000_catchcn_ldas_rst_19800101_0000z.nc Did "diff" to make sure that they are identical. In MATLAB: nccmp('/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_ens0000_catchcn_ldas_rst_19800101_0000z.nc', '/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_check_ens0000_catchcn_ldas_rst_19800101_0000z.nc') Got error message: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Struct contents reference from a non-struct array object. Error in nccmp (line 102) attrName1 = {finfo1.Attributes.Name}; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > mv CF0180x6C_check_ens0000_catchcn_ldas_rst_19800101_0000z.nc /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C > mv CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_ens0000_catchcn_ldas_rst_19800101_0000z.nc . In MATLAB: ncfile1 = 'CF0180x6C_ens0000_catchcn_ldas_rst_19800101_0000z.nc'; ncfile2 = 'CF0180x6C_check_ens0000_catchcn_ldas_rst_19800101_0000z.nc'; ncid1 = netcdf.open(ncfile1,'NC_NOWRITE'); ncid2 = netcdf.open(ncfile2,'NC_NOWRITE'); [numdims1, numvars1, numglobalatts1, unlimdimID1] = netcdf.inq(ncid1); [numdims2, numvars2, numglobalatts2, unlimdimID2] = netcdf.inq(ncid2); finfo1 = ncinfo(ncfile1); finfo2 = ncinfo(ncfile2); attrName1 = {finfo1.Attributes.Name}; attrName2 = {finfo2.Attributes.Name}; There is nothing in finfo1.Attributes. Therefore "attrName1 = {finfo1.Attributes.Name};" gives an error message. Wrote a MATLAB script /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/cmp_ncfiles.m. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C > diff CF0180x6C_ens0000_catchcn_ldas_rst_19800101_0000z.nc CF0180x6C_check_ens0000_catchcn_ldas_rst_19800101_0000z.nc Files CF0180x6C_ens0000_catchcn_ldas_rst_19800101_0000z.nc and CF0180x6C_check_ens0000_catchcn_ldas_rst_19800101_0000z.nc differ But cmp_ncfiles.m didn't find difference between the two in CH. discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C > cp -p CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M02/CF0180x6C.ens0000.catchcn_ldas_rst.19800201_0000z CF0180x6C_ens0000_catchcn_ldas_rst_19800201_0000z.nc Did "diff" to make sure that they are identical. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C > cp -p ../CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M02/CF0180x6C_check.ens0000.catchcn_ldas_rst.19800201_0000z CF0180x6C_check_ens0000_catchcn_ldas_rst_19800201_0000z.nc Did "diff" to make sure that they are identical. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C > ~smahanam/bin/cdo diffn CF0180x6C_ens0000_catchcn_ldas_rst_19800201_0000z.nc CF0180x6C_check_ens0000_catchcn_ldas_rst_19800201_0000z.nc Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 32 : 2001-01-02 21:00:00 0 609565824 0 2186 : F T 4.3945e-42 0.16969 : CNPFT 2 of 84 records differ 1 of 84 records differ more than 0.001 cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 45.68s ) 2. The e0004s_preindustrialCO2 finished the 3rd cycle. dali15:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-12-11 19:24 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z dali15:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-12-11 19:24 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_03 cd e0004s_preindustrialCO2_03 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-5] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that in M09_CN_e0004s_preindustrialCO2.exe: "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_03/output/" "restart_id = e0004s_preindustrialCO2" Run ldsetup to set up the next cycle: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN Run edit_lenkf.csh to modify the job scripts: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../e0004s_preindustrialCO2_03/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 3. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161213: 1. Continued 1 on 20161212. Used /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/cmp_ncfiles.m to look at the difference. Re-do CF0180x6C_check and make it run longer for more comparison with CF0180x6C. /discover/nobackup/fzeng/Catchment/M2n5P > ls -l CF0180x6C/CF0180x6C/build lrwxrwxrwx 1 fzeng g0620 75 2016-12-12 10:13 CF0180x6C/CF0180x6C/build -> /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/ /discover/nobackup/fzeng/Catchment/M2n5P > ls -l CF0180x6C_check/CF0180x6C_check/build lrwxrwxrwx 1 fzeng g0620 75 2016-12-12 12:57 CF0180x6C_check/CF0180x6C_check/build -> /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/ /discover/nobackup/fzeng/Catchment/M2n5P > ls -l CF0180x6C/CF0180x6C/input/restart/output lrwxrwxrwx 1 fzeng g0620 70 2016-12-12 10:13 CF0180x6C/CF0180x6C/input/restart/output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/ /discover/nobackup/fzeng/Catchment/M2n5P > ls -l CF0180x6C_check/CF0180x6C_check/input/restart/output lrwxrwxrwx 1 fzeng g0620 70 2016-12-12 12:57 CF0180x6C_check/CF0180x6C_check/input/restart/output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/ /discover/nobackup/fzeng/Catchment/M2n5P > xdiff CF0180x6C/CF0180x6C/run/lenkf.0.j CF0180x6C_check/CF0180x6C_check/run/lenkf.0.j & Found no differece in setting. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C > mkdir rst_Yehui_11_check /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run > diff /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/build/Linux/bin/LDASsaCN_mpi.x /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/build/Linux/bin/LDASsaCN_mpi.x Returned nothing. So both the executables and the restart files are identical between the two (CF0180x6C and CF0180x6C_check). 2. Worked on compute_rc.F90 for upgrading to CLM4.5: radiation partitioning. ========== 20161214: 1. Set up two same experiments to debug. cd /discover/nobackup/fzeng/Catchment/M2n5P/run cp CF0180x6C.bat CF0180x6C_test1.bat cp CF0180x6C.bat CF0180x6C_test2.bat nedit CF0180x6C_test1.bat CF0180x6C_test2.bat & cp CF0180x6C_CN.exe CF0180x6C_test1.exe cp CF0180x6C_CN.exe CF0180x6C_test2.exe nedit CF0180x6C_test1.exe CF0180x6C_test2.exe & cp driver_inputs_CF0180x6C_GLOBAL.nml driver_inputs_CF0180x6C_GLOBAL_test.nml nedit driver_inputs_CF0180x6C_GLOBAL_test.nml & cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_test1.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_test1.bat --runmodel --daysperjob 1 --landmodel catchCN ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_test2.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_test2.bat --runmodel --daysperjob 1 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/run nedit lenkf.0.j (to make it a debug run) cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2/run nedit lenkf.0.j (to make it a debug run) Made sure that the two lenkf.0.j files only differ in the exp id, then submitted both. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > ~smahanam/bin/cdo diffn CF0180x6C_test1.ens0000.catchcn_ldas_rst.19800101_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_test2.ens0000.catchcn_ldas_rst.19800101_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 1 of 84 records differ cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 47.00s ) /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01 > ~smahanam/bin/cdo diffn CF0180x6C_test1.ens0000.catchcn_ldas_rst.19800102_0000z /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_test2.ens0000.catchcn_ldas_rst.19800102_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 28 : 2001-01-02 21:00:00 0 2745792 0 686449 : F T 4330.0 0.0000 : CH 32 : 2001-01-02 21:00:00 0 609565824 0 176636 : T T 70000. 1.0000 : CNPFT 2 of 84 records differ cdo diffn: Processed 1567847232 values from 160 variables over 2 timesteps ( 45.94s ) Tried Matt's script. cd /discover/nobackup/fzeng/Catchment/M2n5P/ source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ~mathomp4/SI_Team/scripts/systest/LDAS_Cmp.py CF0180x6C_test1 CF0180x6C_test2 Comparing LDASsa output files... BASELINE: CF0180x6C_test1 CURRENT: CF0180x6C_test2 ens_avg.ldas_tile_xhourly_out.19800101_0130z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_0430z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_0730z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_1030z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_1330z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_1630z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_1930z: PASSED. ens_avg.ldas_tile_xhourly_out.19800101_2230z: PASSED. ldas_tilecoord: PASSED. ldas_tilegrids: PASSED. ldas_catparam.19800101_0000z: PASSED. ldas_mwRTMparam.19800101_0000z: PASSED. However, it didn't compare the rst files. Worked with Sarith to debug. ========== 20161215: Discover down the whole day. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161216: 1. Continued debugging. Sarith suggested (see his email on 20161215 and this morning): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible that after all we are just worried for nothing. CN model runs only on places where fraction of vegetation is > 0. so CNPFT array elements don't get updated by the CN model if veg fraction is zero. After the CN_exit call in process_cn.F90. when you update cn_progn%CNPFT, and cn_progn%CNCOL, first initialize those arrays to zero., then use fveg information, and update only the relevant elements then do the same 2 runs for a day and compare ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Take a look at /discover/nobackup/smahanam/process_cn.F90 Here is the modification (so copy paste) - it may work, does it make sense, we will see Even if nobody cares about fveg =0. fractions, if this works, I will commit these changes to process_cn and GEOS_CatchCNGridComp to ensure zero-diff call CN_exit(N_cat,num_veg,num_zon,ityp,fveg,col,var_col,pft,var_pft) i = 1 do nv = 1,VAR_COL do nz = 1,NUM_ZON cn_progn(:)%CNCOL(i) = col(:, nz,nv) i = i + 1 end do end do i = 1 do iv = 1,VAR_PFT do nv = 1,NUM_VEG do nz = 1,NUM_ZON do k = 1, N_cat if(fveg (k,nv,nz) > 0.) then cn_progn(k)%CNPFT(i) = pft(k, nz,nv,iv) else cn_progn(k)%CNPFT(i) = 0. endif end do i = i + 1 end do end do end do Basically, just redo the 2 experiments with this modification (that we set up on a zoomed in window If it gives a zero-diff, just continue for 2 months and diff the files in cat/ens_avg ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C_debug > rm -rf Linux cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp Made sure that tile_coord.F90 and clsm_ensdrv_out_routines.F90 are for CF0180x6C (>=11th cycle). cp -p /discover/nobackup/smahanam/process_cn.F90 process_cn.F90.Sarith cp process_cn.F90.Sarith process_cn.F90 Compiled: setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/ source $ESMADIR/src/g5_modules then "gmake clean" and "gmake install" in GEOSlana_GridComp and Applications/LDAS_App. -rwxr-xr-x 1 fzeng g0620 69763384 2016-12-16 10:56 Linux/bin/LDASsaCN_mpi.x cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 /bin/cp -pr Linux exec/CF0180x6C_debug/. -rwxr-xr-x 1 fzeng g0620 69763384 2016-12-16 10:56 exec/CF0180x6C_debug/Linux/bin/LDASsaCN_mpi.x /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/run > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763384 2016-12-16 10:56 /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/build/Linux/bin/LDASsaCN_mpi.x cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2 rm build ln -s /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C_debug build /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2 > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763384 2016-12-16 10:56 /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2/build/Linux/bin/LDASsaCN_mpi.x Run interactively: ssh discover-sp3 interactive.py -A sp3 -n 96 -a g0620 --debug ("interactive.py -A sp3 -n 96 -a g0620 -t 03:00" is taking too long.) cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test1/run /bin/rm ../output/CF0180x6C_GLOBAL/rc_out/* ./lenkf.0.i Wait until it's done. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_test2/run /bin/rm ../output/CF0180x6C_GLOBAL/rc_out/* ./lenkf.0.i Compared the rst files: /discover/nobackup/fzeng/Catchment/M2n5P > ~smahanam/bin/cdo diffn CF0180x6C_test1/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_test1.ens0000.catchcn_ldas_rst.19800102_0000z CF0180x6C_test2/output/CF0180x6C_GLOBAL/rs/ens0000/Y1980/M01/CF0180x6C_test2.ens0000.catchcn_ldas_rst.19800102_0000z Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 33 : 2001-01-02 21:00:00 0 52 0 2 : F T 6.7730e+14 9.6552e-05 : CNSUM 1 of 84 records differ cdo diffn: Processed 118768 values from 160 variables over 2 timesteps ( 0.03s ) /discover/nobackup/fzeng/Catchment/M2n5P > diff CF0180x6C_test1/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C_test1.ens_avg.ldas_tile_xhourly_out.19800101_0130z.bin CF0180x6C_test2/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C_test2.ens_avg.ldas_tile_xhourly_out.19800101_0130z.bin Returned nothing, so the output again is identical. CNSUM is not used in offline at all. Sarith explained in his email: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ unused arrays (CNSUM, CN, CH, etc) or array elements (for e.g. fveg = 0. elements in CNPFT) get filled in by the computer with some crazy numbers. You can just set to zero to ensure zero-diff. To make sure this change has not affected your simulations, do a diff with an exisiting simulation for few files in cat/ens_avg ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2. Set up a global test run using the new executable and the restart file from the 10th cycle. Ran for one year and saved monthly output. Compared the output files in cat/ens_avg with the current 11th cycle. cd /discover/nobackup/fzeng/Catchment/M2n5P/ rm -rf CF0180x6C_check mkdir CF0180x6C_check cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C_debug/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN_check.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_check.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run nedit lenkf.0.j & (make it run for one year) /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763384 2016-12-16 10:56 /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/build/Linux/bin/LDASsaCN_mpi.x /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run > ls -l ../input/restart/output lrwxrwxrwx 1 fzeng g0620 70 2016-12-16 13:30 ../input/restart/output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/ qsub lenkf.0.j Compared output: cd /discover/nobackup/fzeng/Catchment/M2n5P diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C.ens_avg.ldas_tile_monthly_out.198001.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198001.bin Files CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C.ens_avg.ldas_tile_monthly_out.198001.bin and CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198001.bin differ The output is different?! Tried Matt's script. cd /discover/nobackup/fzeng/Catchment/M2n5P/ source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ~mathomp4/SI_Team/scripts/systest/LDAS_Cmp.py CF0180x6C/CF0180x6C CF0180x6C_check/CF0180x6C_check Comparing LDASsa output files... BASELINE: CF0180x6C/CF0180x6C CURRENT: CF0180x6C_check/CF0180x6C_check -WARNING----------- num of BAS bin files [246] != num of CUR bin files [7] ------------------- ens_avg.ldas_tile_monthly_out.198001: [FAILED]*!?#$* ens_avg.ldas_tile_monthly_out.198002: [FAILED]*!?#$* ens_avg.ldas_tile_monthly_out.198003: [FAILED]*!?#$* Double-check if I made any mistake: Cmp CF0180x6C_CN.exe and CF0180x6C_CN_check.exe. Only differs in exp_id. Correct. They both used the restart file from /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/. Same met forcing. Same time period. Executable correct: /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763384 2016-12-16 10:56 build/Linux/bin/LDASsaCN_mpi.x Found that I forgot to change "firefac" from -1 to 0.1 after I copied Sarith's process_cn.F90. Correct this and compiled. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 /bin/cp -pr Linux exec/CF0180x6C_debug/. -rwxr-xr-x 1 fzeng g0620 69763684 2016-12-16 15:53 exec/CF0180x6C_debug/Linux/bin/LDASsaCN_mpi.x Set up another global test run: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check rm -rf CF0180x6C_check cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C_debug/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN_check.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_check.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run nedit lenkf.0.j & (make it run for one year) /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763684 2016-12-16 15:53 /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/build/Linux/bin/LDASsaCN_mpi.x /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C_check/CF0180x6C_check/run > ls -l ../input/restart/output lrwxrwxrwx 1 fzeng g0620 70 2016-12-16 15:54 ../input/restart/output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_10/output/ qsub lenkf.0.j Compared output: cd /discover/nobackup/fzeng/Catchment/M2n5P diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C.ens_avg.ldas_tile_monthly_out.198001.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M01/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198001.bin Returned nothing. So they are identical. Great! Tried Matt's script. cd /discover/nobackup/fzeng/Catchment/M2n5P/ source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ~mathomp4/SI_Team/scripts/systest/LDAS_Cmp.py CF0180x6C/CF0180x6C CF0180x6C_check/CF0180x6C_check Comparing LDASsa output files... BASELINE: CF0180x6C/CF0180x6C CURRENT: CF0180x6C_check/CF0180x6C_check -WARNING----------- num of BAS bin files [250] != num of CUR bin files [5] ------------------- ens_avg.ldas_tile_monthly_out.198001: PASSED. ldas_tilecoord: PASSED. ldas_tilegrids: PASSED. ldas_catparam.19800101_0000z: PASSED. ldas_mwRTMparam.19800101_0000z: PASSED. CF0180x6C_check has only finished the first month so far. I think that's why we see the WARNING. diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M02/CF0180x6C.ens_avg.ldas_tile_monthly_out.198002.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M02/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198002.bin Returned nothing. So they are identical. Great! diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M03/CF0180x6C.ens_avg.ldas_tile_monthly_out.198003.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M03/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198003.bin Returned nothing. So they are identical. Great! ========== 20161219: 1. CF0180x6C_check ran for one year (1980) and finished on Friday (20161216) night. diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M12/CF0180x6C.ens_avg.ldas_tile_monthly_out.198012.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1980/M12/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198012.bin Returned nothing. So they are identical. Great! 2. Sarith suggests to use the restart file from CF0180x6C_check to do a run and then compare the output to that of CF0180x6C. Therefore, extended the CF0180x6C_check run which used the 19810101 restart file from CF0180x6C_check. cd /discover/nobackup/fzeng/Catchment/M2n5P/ diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1981/M01/CF0180x6C.ens_avg.ldas_tile_monthly_out.198101.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1981/M01/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198101.bin Returned nothing. So they are identical. Great! diff CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1981/M02/CF0180x6C.ens_avg.ldas_tile_monthly_out.198102.bin CF0180x6C_check/CF0180x6C_check/output/CF0180x6C_GLOBAL/cat/ens_avg/Y1981/M02/CF0180x6C_check.ens_avg.ldas_tile_monthly_out.198102.bin Returned nothing. So they are identical. Great! 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161220: 1. The CF0180 run finished the 11th cycle. (1) Checked and processed the output: dali16:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-12-19 23:41 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali16:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-12-19 23:41 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_11 cd CF0180x6C_11 rm -rf lenkf_orig cd output/CF0180x6C_GLOBAL/rs/ens0000/ mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec mv CF0180x6C CF0180x6C_11 mv CF0180x6C_debug CF0180x6C (3) Set up the 12th cycle run. cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_11/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin ls -l LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763684 2016-12-16 15:53 LDASsaCN_mpi.x source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C mkdir rst_Yehui_12 cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../CF0180x6C_11/run/edit_lenkf.csh . nedit edit_lenkf.csh & (make it copy the restart files to rst_Yehui_12) ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j Checked the output restart files. Restart file for 19800416_0000z was written out. Good! 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. 3. The e0004s_preindustrialCO2 finished the 4th cycle. dali11:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-12-20 14:04 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z dali11:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-12-20 14:04 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_04 cd e0004s_preindustrialCO2_04 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-5] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that in M09_CN_e0004s_preindustrialCO2.exe: "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_04/output/" "restart_id = e0004s_preindustrialCO2" Modified GEOSlana_GridComp/e0004s_preindustrialCO2/process_cn.F90 to include the recent modification (see notes on 20161216), and copied it to GEOSlana_GridComp. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp cp e0004s_preindustrialCO2/clsm_ensdrv_out_routines.F90 . cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp cp /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/e0004s_preindustrialCO2/compute_rc.F90 . Recompiled: "gmake clean" and "gmake install" in GEOScatchCN_GridComp, GEOSlana_GridComp and Applications/LDAS_App. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_01to04 mkdir e0004s_preindustrialCO2 /bin/cp -pr ../Linux e0004s_preindustrialCO2/. ls -l e0004s_preindustrialCO2/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69805590 2016-12-20 16:25 e0004s_preindustrialCO2/Linux/bin/LDASsaCN_mpi.x Run ldsetup to set up the next cycle: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN Run edit_lenkf.csh to modify the job scripts: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../e0004s_preindustrialCO2_04/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. Double-checked the executable and the restart file. They are both correct. qsub lenkf.0.j Checked the log file. Looks correct. Restored clsm_ensdrv_out_routines.F90, compute_rc.F90 and process_cn.F90. ========== 20161221: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161222: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161223: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161227: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161228: 1. The CF0180 run finished the 12th cycle. (1) Checked and processed the output: discover15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-12-27 22:20 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z discover15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-12-27 22:20 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_12 cd CF0180x6C_12 rm -rf lenkf_orig cd output/CF0180x6C_GLOBAL/rs/ens0000/ mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z (2) Set up the 13th cycle run. cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_12/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C mkdir rst_Yehui_13 cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763684 2016-12-16 15:53 build/Linux/bin/LDASsaCN_mpi.x* ls -l input/restart/ lrwxrwxrwx 1 fzeng g0620 70 2016-12-28 11:11 output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_12/output/ mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../CF0180x6C_12/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161229: 1. The e0004s_preindustrialCO2 finished the 5th cycle. dali17:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-12-28 12:29 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z dali17:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-12-28 12:29 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_05 cd e0004s_preindustrialCO2_05 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-5] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that in M09_CN_e0004s_preindustrialCO2.exe: "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_05/output/" "restart_id = e0004s_preindustrialCO2" Run ldsetup to set up the next cycle: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN Run edit_lenkf.csh to modify the job scripts: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../e0004s_preindustrialCO2_05/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. Double-checked the executable and the restart file. They are both correct. qsub lenkf.0.j 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily.