========== 20161101: 1. Continued working on 4 of 20161031. Modified process_cn.F90 to read climatological monthly-mean CO2 diurnal cycle and interpolate to daily for the new SMAP M09 e0004s preindustrial CO2 spinup run. Compiled: "gmake install" in GEOSlana_GridComp and Applications/LDAS_App. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . 2. Set up a regional test run e0004s_preindustrialCO2_neUS: experiment domain: 37N-40N, 75W-78W (893 tiles total) cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2 cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run cp driver_inputs_M09_GLOBAL.nml driver_inputs_M09_neUS.nml nedit driver_inputs_M09_neUS.nml & (modified the output option and the region to simulate) cp M09_CN_e0004s_spinup.exe M09_CN_neUS.exe cp M09.bat M09_neUS.bat nedit M09_neUS.bat M09_CN_neUS.exe & Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_neUS.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_neUS.bat --runmodel --monthsperjob 15 --landmodel catchCN The test run crashed due to floating invalid in the interpolation of climatological monthly mean to daily step: "CO2_dy = wt * CO2_mo1 + (1 - wt) * CO2_mo2" in process_cn.F90. ========== 20161102: 1. Added print statements to process_cn.F90. Compiled: "gmake install" in GEOSlana_GridComp and Applications/LDAS_App. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . Set up a new regional test run e0004s_preindustrialCO2_neUS: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2_neUS Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_neUS.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_neUS.bat --runmodel --monthsperjob 15 --landmodel catchCN The problem is in the read CO2 data statement. Adding "convert='little_endian'" fixed the problem. Another problem occurred in "co2v(n) = CO2_tmp(iCT(n),jCT(n))" in process_cn.F90. Finally, adding "if (.not. allocated(CO2_tmp)) allocate(CO2_tmp(CT_grid_N_lon,CT_grid_N_lat))" right before "co2v(n) = CO2_tmp(iCT(n),jCT(n))" in process_cn.F90 solved the issue. Commented out the print statements. Recompiled. Set up a new regional test run e0004s_preindustrialCO2_neUS to run for 2 years. Save 3hrly and monthly tile output. 2. October report. ========== 20161103: 1. The e0004s_preindustrialCO2_neUS test run stopped at 20010116_000730z when the Feb mean CO2 diurnal cycle is read and the error message points to L1256 "CO2_mo2(:,:,hr/3+1) = CO2_tmp(:,:)" in process_cn.F90. It took about 5 minutes to simulate 15 days, so 2 years of simulation needs less than 5 hours. Only need to request for 6 hours next time. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_neUS/output/SMAP_EASEv2_M09_neUS/rc_out/e0004s_preindustrialCO2_neUS.ldas_domain.txt: 893 266541 266541 266540 266540 266542 266542 266543 266543 265855 265855 265854 265854 265858 265858 265602 265602 266296 266296 265174 265174 265857 265857 266546 266546 266544 266544 266545 266545 265172 265172 265171 265171 264490 264490 264491 264491 265856 265856 265173 265173 ... So there are 893 tiles in the experiment domain. Plotted the 3hrly CO2 (i.e. sifl which is borrowed to save CO2) output of tile 266541 for the 15 days finished. Looks correct. Modified process_cn.F90. Compiled: "gmake install" in GEOSlana_GridComp and Applications/LDAS_App. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2_neUS cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . Run ldsetup to set up a new regional test run e0004s_preindustrialCO2_neUS: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_neUS.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_neUS.bat --runmodel --monthsperjob 15 --landmodel catchCN Run interactively: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_neUS/run cp lenkf.0.j lenkf.0.i nedit lenkf.0.i & lenkf.0.i: ~~~~~~~~~~~~~~~~~~~~~ #!/bin/csh -fx limit stacksize unlimited source ../build/Linux/bin/g5_modules setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2 setenv MYNAME `finger $USER | cut -d':' -f3 | head -1` mpirun -np 96 \ /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_neUS/build/Linux/bin/LDASsaCN_mpi.x \ -work_path ../output \ -run_path . \ -exp_id e0004s_preindustrialCO2_neUS \ -exp_domain SMAP_EASEv2_M09_neUS \ -start_year 2001 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2002 -end_month 4 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -N_ens 1 \ -spin .false. \ -force_dtstep 3600 \ -first_ens_id 0 \ -resolution SMAP_EASEv2_M09 \ -restart .true. \ -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s \ -met_tag M2COR_cross__precCPCUGPCP22clim_MERRA2_BMTXS \ -met_path ../input/met_forcing/MERRA2_land_forcing \ -driver_inputs_path . \ -driver_inputs_file driver_inputs_M09_neUS.nml ~~~~~~~~~~~~~~~~~~~~~ ssh discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_neUS/run ./lenkf.0.i It's running. It passed Jan 16 2001 and Feb 1 2001 and finished 10.5 months of simulation. Plotted the 3hrly CO2 output of tile 266541 for Jan through Apr. CO2 was not interpolated after Jan 16, 2001!! date_time_new 20010115_000730z: dist for 2001 1 15 : 31.00000 wt for 2001 1 15 : 3.2258064E-02 range of CO2_mo1: 372.4386 451.7278 range of CO2_mo2: 374.5602 451.9049 range of CO2_dy: 374.5965 451.8954 calculate CT co2 at 2001 1 15 0 7 30 mean co2v (fraction): 2.8857225E-04 dist for 2001 1 15 : 31.00000 wt for 2001 1 15 : 3.2258064E-02 range of CO2_mo1: 372.4386 451.7278 range of CO2_mo2: 374.5602 451.9049 range of CO2_dy: 374.5965 451.8954 calculate CT co2 at 2001 1 15 0 7 30 mean co2v (fraction): 2.8857225E-04 dist for 2001 1 15 : 31.00000 wt for 2001 1 15 : 3.2258064E-02 range of CO2_mo1: 372.4386 451.7278 range of CO2_mo2: 374.5602 451.9049 range of CO2_dy: 374.5965 451.8954 calculate CT co2 at 2001 1 15 0 7 30 mean co2v (fraction): 2.8857225E-04 date_time_new 20010116_000730z: dist for 2001 1 16 : 29.00000 wt for 2001 1 16 : 0.0000000E+00 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.5602 451.9049 calculate CT co2 at 2001 1 16 0 7 30 mean co2v (fraction): 2.8850304E-04 dist for 2001 1 16 : 29.00000 wt for 2001 1 16 : 0.0000000E+00 range of CO2_mo1: 373.6144 450.0688 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 373.6144 450.0688 calculate CT co2 at 2001 1 16 0 7 30 mean co2v (fraction): 2.8900799E-04 dist for 2001 1 16 : 29.00000 wt for 2001 1 16 : 0.0000000E+00 range of CO2_mo1: 373.6144 450.0688 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 373.6144 450.0688 calculate CT co2 at 2001 1 16 0 7 30 mean co2v (fraction): 2.8900799E-04 "range of CO2_mo1" and "range of CO2_mo2" were printed out 3 times at 20010116_000730z. The first time they are different, which is expected. The 2nd and third times they are identical, which is not what we want. It's probably that, due to the mpi, "CO2_mo1 = CO2_mo2" was executed three time (like the print statement was executed three time as seen above), while we want it to be executed only once. The 2nd and 3rd times "CO2_mo1 = CO2_mo2" was executed makes CO2_mo1 identical to CO2_mo2 and messes up the run. Therefore, have to read data of two months, instead of just the next month, in the middle of each month. After the correction, now the CO2_mo1 and CO2_mo2 ranges look correct: date_time_new 20010117_000730z: dist for 2001 1 17 : 29.00000 wt for 2001 1 17 : 3.4482758E-02 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.6566 451.6776 calculate CT co2 at 2001 1 17 0 7 30 mean co2v (fraction): 2.8852044E-04 dist for 2001 1 17 : 29.00000 wt for 2001 1 17 : 3.4482758E-02 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.6566 451.6776 calculate CT co2 at 2001 1 17 0 7 30 mean co2v (fraction): 2.8852044E-04 dist for 2001 1 17 : 29.00000 wt for 2001 1 17 : 3.4482758E-02 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.6566 451.6776 calculate CT co2 at 2001 1 17 0 7 30 mean co2v (fraction): 2.8852044E-04 date_time_new 20010118_000730z: dist for 2001 1 18 : 29.00000 wt for 2001 1 18 : 6.8965517E-02 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.7528 451.4502 calculate CT co2 at 2001 1 18 0 7 30 mean co2v (fraction): 2.8853791E-04 dist for 2001 1 18 : 29.00000 wt for 2001 1 18 : 6.8965517E-02 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.7528 451.4502 calculate CT co2 at 2001 1 18 0 7 30 mean co2v (fraction): 2.8853791E-04 dist for 2001 1 18 : 29.00000 wt for 2001 1 18 : 6.8965517E-02 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.7528 451.4502 calculate CT co2 at 2001 1 18 0 7 30 mean co2v (fraction): 2.8853791E-04 date_time_new 20010119_000730z: dist for 2001 1 19 : 29.00000 wt for 2001 1 19 : 0.1034483 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.8491 451.2229 calculate CT co2 at 2001 1 19 0 7 30 mean co2v (fraction): 2.8855525E-04 dist for 2001 1 19 : 29.00000 wt for 2001 1 19 : 0.1034483 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.8491 451.2229 calculate CT co2 at 2001 1 19 0 7 30 mean co2v (fraction): 2.8855525E-04 dist for 2001 1 19 : 29.00000 wt for 2001 1 19 : 0.1034483 range of CO2_mo1: 374.5602 451.9049 range of CO2_mo2: 373.6144 450.0688 range of CO2_dy: 374.8491 451.2229 calculate CT co2 at 2001 1 19 0 7 30 mean co2v (fraction): 2.8855525E-04 Looks good. Removed the print statements in process_cn.F90. Recompiled. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2_neUS cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . Run ldsetup to set up a new regional test run e0004s_preindustrialCO2_neUS: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_neUS.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_neUS.bat --runmodel --monthsperjob 15 --landmodel catchCN Save 3hrly and monthly tile output for the first 3 months of 2001, then save monthly output through Dec 2002. 2. 2-page CV. ========== 20161104: 1. There is another bug in my modification in process_cn.F90. The bug is in "if(first) then read data of Dec and Jan". This is wrong when the start time of the simulation is not Jan 1st such as the 2nd job script above which starts from Apr 1st. Fixed the bug. Recompiled. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2_neUS cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . Run ldsetup to set up a new regional test run e0004s_preindustrialCO2_neUS: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_neUS.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_neUS.bat --runmodel --monthsperjob 15 --landmodel catchCN Save 3hrly and monthly tile output for the first 3 months of 2001, then save monthly output through Dec 2002. Output looks correct. Commented out the print statements in process_cn.F90. Recompiled. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Run ldsetup to set up e0004s_preindustrialCO2: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 6 --landmodel catchCN Run the modified version of Sarith's edit_lenkf.csh to modify the job scripts (now no need to further modify the first two and last job scripts manually after running edit_lenkf.csh): cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../../M2n5P/CF0180x6C/CF0180x6C/run/edit_lenkf.csh . nedit edit_lenkf.csh ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 2. Processed GMAO GEOS5 seasonal forecast for November. 3. Modified LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/ens_driver_routines.F90 to make the precipitation only fall between 12am and 2am local time. Used the original fwetl and fwetc setting: /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp > cp unified_rc3f_matrix_calc.F90.restore unified_rc3f_matrix_calc.F90 Compiled using cleanup without "gmake clean". 4. Set up a monthly interpolated precipitation global test run to run for 5 days: save hourly grid output to check whether the precipitation only falls for 2 hours during 12am and 2am save daily tile output to compared to the input daily data /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthly_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp ../m0001_Prcp_monthly_interp_defaultFWET/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l RUN/rs/ens0000/Y2001/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j It's running. Exp log file: FWETL = 1.000000 FWETC = 0.2000000 SATCAPFR = 0.2000000 Correct. ========== 20161107: 1. The CF0180 run finished the 7th cycle. dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-11-04 18:56 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-11-04 18:57 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_07 cd CF0180x6C_07/output/CF0180x6C_GLOBAL/rs/ens0000/ mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_07/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_07/run/edit_lenkf.csh . (copy edit_lenkf.csh from the previous cycle) ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j. 2. Checked the output of the e0004s_preindustrialCO2. rc_out/e0004s_preindustrialCO2.ldas_domain.txt: 1684725 1118127 1118127 1118126 1118126 1118547 1118547 1118549 1118549 1118975 1118975 1120244 1120244 1120243 1120243 1119820 1119820 1120668 1120668 1120245 1120245 1120666 1120666 1120667 1120667 Monthly CO2 time series output for the first tile above checked looks correct - show seasonal cycle without year-to-year variability. ##Will check the maps as well. 3. Checked the output of the monthly interpolated precipitation global test run. rc_out/m0001.ldas_domain.txt: 370121 66533 66533 66535 66535 66536 66536 66538 66538 66540 66540 66542 66542 66544 66544 66546 66546 66548 66548 66560 66560 66561 66561 66562 66562 66563 66563 66564 66564 66565 66565 66566 66566 66567 66567 66568 66568 66569 66569 66570 66570 Made global maps of hourly precip output to see if precip occurs in a particular longitudinal zone only in each of the 24 hours within a day. Maps look correct. Made time series plots of daily tile precip output of selected tiles and compared to the input daily tile precip data. Looks correct for the first 5 days of 1 Jan 2001 simulated. Extended the simulation to the end of Feb 2001, save daily tile output. See /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp/lenkf.2.j Exp log file: FWETL = 1.000000 FWETC = 0.2000000 SATCAPFR = 0.2000000 Correct. Plotted time series of daily tile precip output and compared to the input again. Looks correct. Did another run to save monthly grid output to check the eint/precip ratio. cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mv m0001_Prcp_monthly_interp m0001_Prcp_monthly_interp_hrlyNdailyOUT mkdir -p m0001_Prcp_monthly_interp/RUN/rs/ens0000/Y2001/M01 cd m0001_Prcp_monthly_interp cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin cp ../m0001_Prcp_monthly_interp_hrlyNdailyOUT/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l RUN/rs/ens0000/Y2001/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j (to run for 4 months as a debug run and save monthly grid output) qsub lenkf.j ========== 20161108: 1. Updated higgins rain data through Nov 5. Noticed that there is no valid data for Jan 1 2016 in the higgins_2016.gdat file created. Checked the original precip.V1.0.2016.nc file downloaded from ftp://ftp.cdc.noaa.gov/Datasets/cpc_us_precip/RT. Data for Jan 1 2016 is all missing values in this netCDF file. Will check and see if the netCDF file is updated tomorrow. ========== 20161109: 1. Set up the CO2 experiments. (1) 3hrly: cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mkdir -p M2.n5P.3hco2.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.3hco2.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.3hco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/elee15/offline/sims/M2.n5P.3hco2.glb/lenkf.j . dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.3hco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin (2) daily: mkdir -p M2.n5P.dco2.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.dco2.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.dco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb/lenkf.j . (3) monthly interpolated: mkdir -p M2.n5P.mco2.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.mco2.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.mco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb/lenkf.j . (4) monthly mean interpolated: mkdir -p M2.n5P.mmco2.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.mmco2.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.mmco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb/lenkf.j . (5) annual mean: mkdir -p M2.n5P.aco2.glb/RUN/rs/ens0000/Y2001/M01 cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.aco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb/lenkf.j . (6) constant: mkdir -p M2.n5P.cco2.glb/RUN/rs/ens0000/Y2001/M01 cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.cco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2.glb/lenkf.j . Eunjee's post processing script: /discover/nobackup/elee15/offline/postprc/tile2grid_FV_144x91_monthly.f90 Modified /home/fzeng/Catchment/M2n5P_m0001/tile2grid_FV_144x91_monthly.f90 to match hers. 2. Checked precip.V1.0.2016.nc on ftp://ftp.cdc.noaa.gov/Datasets/cpc_us_precip/RT. It's updated - the time stamp of the file is now "precip.V1.0.2016.nc 8813 KB 11/9/16 12:16:00 PM" However, the data for 1 Jan 2016 is still all missing values. Updated Randy and asked if I should contact the Data Management of NOAA/ESRL/PSD about this. 3. Further verified the e0004s_preindustrialCO2 output: Made monthly CO2 time series plot for the first tile 1118127 in the output for the first 4 years finished. It looks correct - show seasonal cycle without year-to-year variability. Converted tile-space output to grid-space output and made CO2 maps. ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Made map of Jan CO2 output and compared to the input. Patterns are similar. ========== 20161110: 1. Checked the precip.V1.0.2016.nc file on ftp://ftp.cdc.noaa.gov/Datasets/cpc_us_precip/RT. It's been updated since yesterday. Now it has data through Nov 9, 2016 and there is no missing data in 2016. 1 Jan 2016 now has valid data. Updated higgins rain data through Nov 9, 2016. 2. Organized the modified (for the mP night precip and SMAP preindustrial CO2 runs) offline routines. Restored the offline model code to its default settings. 3. Completed 2016 GSFC-Building Emergency Plan (BEP). 4. GMAO Theme Meeting 5. Read materials for Open Enrollment ========== 20161111: 1. Worked on upgrading the CN model from CLM4 to CLM4.5. Since Sarith's point driver for LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 has to compile on NCCS (see notes on 20160818), copy the previous work done on land01 (land01:~/point_driver_LDASsa/cesm1_2_2-testing) to NCCS. /discover/nobackup/fzeng/clm4-to-clm4.5 > ls catchcn4_0/ catchcn4_5/ clm4_0/ clm4_5/ /discover/nobackup/fzeng/clm4-to-clm4.5/pftdata4: scp -p land01:/terra/fzeng/clm/pftdata/pft-physiology.c100226 . /discover/nobackup/fzeng/clm4-to-clm4.5/pftdata4.5: scp -p land01:/home/fzeng/point_driver_LDASsa/cesm1_2_2-testing/pft-physiology.c130503 . /discover/nobackup/fzeng/clm4-to-clm4.5/surfdata_map4.5: scp -p land01:/terra/fzeng/clm/mksrf_24pftNT_landuse_rc2000_c121207.nc . /discover/nobackup/fzeng/clm4-to-clm4.5/notes: (wasn't able to copy multiple files from land01 to NCCS) scp -p land01:/home/fzeng/point_driver_LDASsa/cesm1_2_2-testing/notes . scp -p land01:/home/fzeng/point_driver_LDASsa/cesm1_2_2-testing/notes_daily . scp -p land01:/home/fzeng/point_driver_LDASsa/cesm1_2_2-testing/notes.20150825.gkw . ##More to copy. How to do this faster? Reviewed notes in late Aug 2016 to see what I did about using Sarith's point driver to test what would happen if we don't split vegetation types in the current CN model (which has CLM4 physics). Reviewed notes in late Aug 2015 and late Aug 2016 to see what Greg and I did about the upgrade. 2. Noticed that e0004s_preindustrialCO2 is no longer in the queue. It successfully finished til the end of Jun 2007 and wrote restart file for 20070701. The next job script was successfully submitted and started to run but stopped at the very beginning. There is only the experiment log file in rc_out for this job segment (there should be a few more for a successful run). Also, I can not open the experiment log file - I don't even have read permission. See below. dali14:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2007/M07 > ls -l total 32 -rw------- 1 root root 2321 2016-11-11 12:59 e0004s_preindustrialCO2.ldas_log.20070701_0000.txt Checked the job scripts and they look correct. Tried: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2007/M07 > rm e0004s_preindustrialCO2.ldas_log.20070701_0000.txt /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/run > qsub lenkf.13.j Now e0004s_preindustrialCO2 is in the queue again. ========== 20161114: 1. The CO2 experiments have finished. dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.3hco2.glb/RUN/rs/ens0000/Y2015/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-13 02:59 mM2.n5P.3hco2.glb.ens0000.catch_ldas_rst.20150101_0000z.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.3hco2.glb/RUN/cat/ens_avg/Y2014/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-11-13 02:59 mM2.n5P.3hco2.glb.ens_avg.ldas_tile_monthly_out.201412.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.dco2.glb/RUN/rs/ens0000/Y2015/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-13 01:15 mM2.n5P.dco2.glb.ens0000.catch_ldas_rst.20150101_0000z.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.dco2.glb/RUN/cat/ens_avg/Y2014/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-11-13 01:15 mM2.n5P.dco2.glb.ens_avg.ldas_tile_monthly_out.201412.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.mco2.glb/RUN/rs/ens0000/Y2015/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-12 22:55 mM2.n5P.mco2.glb.ens0000.catch_ldas_rst.20150101_0000z.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.mco2.glb/RUN/cat/ens_avg/Y2014/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-11-12 22:55 mM2.n5P.mco2.glb.ens_avg.ldas_tile_monthly_out.201412.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.mmco2.glb/RUN/rs/ens0000/Y2015/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-13 01:15 mM2.n5P.mmco2.glb.ens0000.catch_ldas_rst.20150101_0000z.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.mmco2.glb/RUN/cat/ens_avg/Y2014/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-11-13 01:16 mM2.n5P.mmco2.glb.ens_avg.ldas_tile_monthly_out.201412.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.aco2.glb/RUN/rs/ens0000/Y2015/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-13 01:13 mM2.n5P.aco2.glb.ens0000.catch_ldas_rst.20150101_0000z.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.aco2.glb/RUN/cat/ens_avg/Y2014/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-11-13 01:13 mM2.n5P.aco2.glb.ens_avg.ldas_tile_monthly_out.201412.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.cco2.glb/RUN/rs/ens0000/Y2015/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-13 01:02 mM2.n5P.cco2.glb.ens0000.catch_ldas_rst.20150101_0000z.bin dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001 > ls -l M2.n5P.cco2.glb/RUN/cat/ens_avg/Y2014/M12 total 117120 -rw-r--r-- 1 fzeng g0620 119919852 2016-11-13 01:03 mM2.n5P.cco2.glb.ens_avg.ldas_tile_monthly_out.201412.bin Processed the output: ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.3hco2.glb ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.dco2.glb ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.mco2.glb ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.mmco2.glb ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.aco2.glb ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.cco2.glb 2. Looked at carbon state convergence for the CF0180 spinup run. 3. Compared LAI, FPAR and GPP from e0004s_preindustrialCO2 to e0004s_43. ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Used /home/fzeng/Catchment/SMAP_M09/compare_new_old_e0004s.gs for comparison. 4. Talked to Sarith about the PFT. Sarith copied the raw data file and will re-do the BCs maps. 5. Read cn.txt to understand which routine is from where. 6. For upgrading to CLM4.5, decided to use the m3-15 version of the point driver on land01 to make every change first and to do testing (because it's much easier to run compared to Sarith's point drive, and it's the same or very close to Sarith's m3-16 version in terms of physics). Once the change works for the point driver, make the same change to the offline and test, and finally to GEOS-5 and test. See notes land01:~/point_driver_LDASsa/cesm1_2_2-testing/notes_daily. ========== 20161115: 1. Asked Sarith if his point driver can compile on land01. 2. Made some GPP maps from the output of the new CO2 experiments. (1) Compared the other 5 cases with the 3hco2 case; (2) Compared these new output with Eunjee's old output. For some reason Eunjee didn't process the mco2 and mmco2 output after interpolation, so I used the results from /discover/nobackup/elee15/offline/sims/M2.n5P.mco2.glb.old and /discover/nobackup/elee15/offline/sims/M2.n5P.mmco2.glb.old in my comparison here. dali10:/discover/nobackup/elee15/offline/sims > ls -l M2.n5P.mco2.glb total 1494976 lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 12:14 alb_b_1_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_b_1_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 12:14 alb_b_2_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_b_2_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 12:14 alb_w_1_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_w_1_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 12:14 alb_w_2_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_w_2_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 76 2016-09-02 12:14 cli_t2m.dat -> /discover/nobackup/fzeng/Catchment/t2m_cli_FV_144x91_SMAP_TestBed_370121.dat -rw-r--r-- 1 elee15 s1583 1530820456 2016-09-04 01:05 CN_restart lrwxrwxrwx 1 elee15 s1583 3 2016-09-02 12:14 GLOBAL_FV_144x91_DC -> RUN/ -rwxr-xr-x 1 elee15 s1583 3801 2016-09-02 12:14 lenkf.j* -rw-r--r-- 1 elee15 s1583 11 2016-09-04 01:05 lenkf_job_completed.txt lrwxrwxrwx 1 elee15 s1583 73 2016-09-02 12:14 ndep.dat -> /discover/nobackup/fzeng/Catchment/ndep_FV_144x91_SMAP_TestBed_370121.dat lrwxrwxrwx 1 elee15 s1583 77 2016-09-02 12:14 pft.dat -> /discover/nobackup/fzeng/Catchment/pft_FV_144x91_SMAP_TestBed-test_370121.dat drwxr-xr-x 5 elee15 s1583 512 2016-09-03 19:56 RUN/ dali10:/discover/nobackup/elee15/offline/sims > ls -l M2.n5P.mmco2.glb total 1494976 lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 16:29 alb_b_1_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_b_1_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 16:29 alb_b_2_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_b_2_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 16:29 alb_w_1_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_w_1_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 96 2016-09-02 16:29 alb_w_2_tile.dat -> /discover/nobackup/fzeng/modis_soil_albedo_cmg_v3/alb_w_2_FV_144x91_SMAP_TestBed_370121_tile.dat lrwxrwxrwx 1 elee15 s1583 76 2016-09-02 16:29 cli_t2m.dat -> /discover/nobackup/fzeng/Catchment/t2m_cli_FV_144x91_SMAP_TestBed_370121.dat -rw-r--r-- 1 elee15 s1583 1530820456 2016-09-04 05:02 CN_restart lrwxrwxrwx 1 elee15 s1583 3 2016-09-02 16:29 GLOBAL_FV_144x91_DC -> RUN/ -rwxr-xr-x 1 elee15 s1583 3812 2016-09-02 12:14 lenkf.j* -rw-r--r-- 1 elee15 s1583 11 2016-09-04 05:03 lenkf_job_completed.txt lrwxrwxrwx 1 elee15 s1583 73 2016-09-02 16:29 ndep.dat -> /discover/nobackup/fzeng/Catchment/ndep_FV_144x91_SMAP_TestBed_370121.dat lrwxrwxrwx 1 elee15 s1583 77 2016-09-02 16:29 pft.dat -> /discover/nobackup/fzeng/Catchment/pft_FV_144x91_SMAP_TestBed-test_370121.dat drwxr-xr-x 5 elee15 s1583 512 2016-09-03 23:55 RUN/ 2. Worked on compute_rc_01.F90 for upgrading to CLM4.5. See notes land01:~/point_driver_LDASsa/cesm1_2_2-testing/notes_daily. ========== 20161116: 1. The e0004s_preindustrialCO2 finished the 1st cycle. dali10:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-11-16 04:33 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z dali10:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-11-16 04:33 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_01 cd e0004s_preindustrialCO2_01 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-5] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that in M09_CN_e0004s_preindustrialCO2.exe: "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_01/output/" "restart_id = e0004s_preindustrialCO2" Run ldsetup to set up the next cycle: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN Run the modified version of Sarith's edit_lenkf.csh to modify the job scripts (now no need to further modify the first two and last job scripts manually after running edit_lenkf.csh): cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../e0004s_preindustrialCO2_01/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 2. Worked with Sarith on the PFT (for CLM4.5) issue: different PFT maps from the raw 0.05 deg data mksrf_24pftNT_landuse_rc2000_c121207.nc and from surfdata_0.125x0.125_mp24_simyr2000_c150114.nc. 3. Worked on compute_rc_01.F90 for upgrading to CLM4.5. See notes land01:~/point_driver_LDASsa/cesm1_2_2-testing/notes_daily. ========== 20161117: 1. The CF0180 run finished the 8th cycle. dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-11-16 22:38 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-11-16 22:39 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_08 cd CF0180x6C_08 rm -rf lenkf_orig cd output/CF0180x6C_GLOBAL/rs/ens0000/ mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_08/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../CF0180x6C_08/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 2. Worked on compute_rc_01.F90 for upgrading to CLM4.5. See notes land01:~/point_driver_LDASsa/cesm1_2_2-testing/notes_daily. ========== 20161118: 1. Worked on compute_rc_01.F90 for upgrading to CLM4.5. See notes land01:~/point_driver_LDASsa/cesm1_2_2-testing/notes_daily. ========== 20161121: 1. Updated higgins rain data through Nov 20. 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161122: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161123: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. Set up offline test runs. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161125: 1. Checked the e0004s_preindustrialCO2 run and found it no longer in the queue. Thought it finished the 2nd cycle, but found that it stopped at the beginning of the 201209 job segment at 2016-11-24 01:40. Error message: Reading vegetation class and, if available, height Cannot find file /mosaic_veg_typs_fracs in: /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//SMAP_EASEv2_ M09/clsm /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//SMAP_EASEv2_ M09/VEGETATION-GSWP2 LDAS ERROR (3000) from open_land_param_file: ERROR opening file Checked Sarith's bcs and found that Sarith modified mosaic_veg_typs_fracs on 2016-11-23 and this file became unreadible to other users: -rw------- 1 smahanam g0610 85920975 2016-11-23 14:54 /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//SMAP_EASEv2_M09/clsm//mosaic_veg_typs_fracs Will talk to Sarith next week. ========== 20161128: 1. The CF0180 run finished the 9th cycle. dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-11-26 04:23 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali17:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-11-26 04:23 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_09 cd CF0180x6C_09 rm -rf lenkf_orig cd output/CF0180x6C_GLOBAL/rs/ens0000/ mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_09/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../CF0180x6C_09/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 2. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. 3. Talked to Sarith about /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//SMAP_EASEv2_M09/clsm//mosaic_veg_typs_fracs. He chmod 644 this file. The e0004s_preindustrialCO2 run stopped at 20120901, and lenkf.14.j starts from 20120901, so /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/run > qsub lenkf.14.j ========== 20161129: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. ========== 20161130: 1. Worked on compute_rc.F90 for upgrading to CLM4.5. See notes land01:/terra/fzeng/cesm1_2_2-testing/notes_daily. 2. The e0004s_preindustrialCO2 finished the 2nd cycle. dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-11-30 08:40 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-11-30 08:40 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2 Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_02 cd e0004s_preindustrialCO2_02 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-5] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that in M09_CN_e0004s_preindustrialCO2.exe: "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_02/output/" "restart_id = e0004s_preindustrialCO2" Run ldsetup to set up the next cycle: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN Run the modified version of Sarith's edit_lenkf.csh to modify the job scripts (now no need to further modify the first two and last job scripts manually after running edit_lenkf.csh): cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp ../../e0004s_preindustrialCO2_02/run/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j 3. Eunjee noticed that the .gdat output files in the M2.n5P.mco2.glb and M2.n5P.mmco2.glb runs I did are identical. I found that it's because the two executables LDASsa_mpi_M2n5P_m0001CTco2_monthly_interpolated.x and LDASsa_mpi_M2n5P_m0001CTco2_monthlymean_interpolated.x in /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec are identical. I must have forgotten to change process_cat.F90 for the monthlymean one. We didn't notice this because, although Eunjee updated her mco2 and mmco2 simulations she didn't process the output, so in her plots the results from the old mco2 and mmco2 (monthly CO2 data was used without being interpolated to daily) were used instead of the results from these updated mco2 and mmco2 runs. cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec mv LDASsa_mpi_M2n5P_m0001CTco2_monthlymean_interpolated.x LDASsa_mpi_M2n5P_m0001CTco2_monthlymean_interpolated.x.wrong cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp /bin/cp M2n5P_m0001CTco2/process_cat.F90.monthlymeanCO2_interpolated process_cat.F90 /bin/cp M2n5P_m0001CTco2/compute_rc.F90 ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/. Compiled using cleanup without "gmake clean". Now the two executables differ. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec > cmp LDASsa_mpi_M2n5P_m0001CTco2_monthly_interpolated.x LDASsa_mpi_M2n5P_m0001CTco2_monthlymean_interpolated.x LDASsa_mpi_M2n5P_m0001CTco2_monthly_interpolated.x LDASsa_mpi_M2n5P_m0001CTco2_monthlymean_interpolated.x differ: byte 41, line 1 Set up the corrected mmco2 run: cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mv M2.n5P.mmco2.glb M2.n5P.mmco2.glb.wrong mkdir -p M2.n5P.mmco2.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.mmco2.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.mmco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb.wrong/lenkf.j . dali15:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart dali15:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.mmco2.glb > ls -l RUN/rs/ens0000/Y2001/M01/ total 38912 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.mmco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin qsub lenkf.j