========== 20180904: 1. Processed GEOS5 September forecast. 2. August report sent to Qing. 3. Tair issue. Eunjee found some problem in the Tair from my GEOSldas_CF90 run. Tair in 2001-2017 is systematically lower than that in 1981-2000. For example, 2001 Jun Tair is much lower than 2000 Jun Tair. Compared 2001 Jun Tair between my GEOSldas_CF90 run and my e0004s_transientCO2 run. The patterns are the same but the former is 10 degree lower than the latter. The latter looks correct. The 2000 Jun Tair from GEOSldas_CF90 looks correct. /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/cat/ens0000/Y2000/M06 > ls -l total 24512 -rw-r--r-- 1 fzeng g0620 20109220 2018-08-03 20:44 GEOSldas_CF90.tavg1_2D_lnd_Nx.monthly.200006.nc4 -rw-r--r-- 1 fzeng g0620 4971980 2018-08-03 20:44 GEOSldas_CF90.tavg1_2D_mf-st.monthly.200006.nc4 /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/cat/ens0000/Y2001/M06 > ls -l total 24512 -rw-r--r-- 1 fzeng g0620 20109220 2018-08-06 22:21 GEOSldas_CF90.tavg1_2D_lnd_Nx.monthly.200106.nc4 -rw-r--r-- 1 fzeng g0620 4971980 2018-08-06 22:21 GEOSldas_CF90.tavg1_2D_mf-st.monthly.200106.nc4 /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90 > ls -l build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50726775 2018-07-25 12:52 build/Linux/bin/GEOSldas.x* The same executable was used for both 2000 and 2001. This is correct! Checked my notes. This is what I did on 20180806: ~~~~~~~~~~~~~~~~~~~~~~~ 3. Extended the GEOSldas_CF90 run to 20180101_0000z. > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run > cat cap_restart 20001231 210000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 141 2018-08-04 01:00 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000/M12/GEOSldas_CF90.catchcn_internal_rst.20001231_2100 lrwxrwxrwx 1 fzeng g0620 116 2018-07-25 13:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/GEOSldas_CF90.vegdyn_internal_rst > nedit CAP.rc & END_DATE: 20180101 000000 > qsub lenkf.j ~~~~~~~~~~~~~~~~~~~~~~~ Don't see anything wrong. Checked the met_forcing path in the exp_id/input/met_forcing. They are the same between the 2 experiments. Talked to Sarith. Found that I missed one '0' in CAP.rc: JOB_SGMT: 0000600 000000 It should be: JOB_SGMT: 00000600 000000 However, it didn't seem to affect the frequency of writing out the restart files though. The restart file was written out every 6 months. Corrected this. Checked Tair of some other months. Good: 200101 through 200105, 200107 Bad: 200106 Sarith said since after 20001231_21z, the 1st job segment is from 20001231_21z through 20010630_21z --> missing the last 3 hours of 20010630. This affects the daily mean of 20010630 and therefore the monthly mean of 200106. Decided to re-run from 20000701_0000z, the last 0000z restart file available. /discover/nobackup/fzeng/Catchment/M2n5P > /bin/cp -pr GEOSldas_CF90 GEOSldas_CF90_old [Just in case] > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run > nedit cap_restart & 20000701 000000 > cd ../input/restart/ > rm catchcn_internal_rst > ln -s /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000/M07/GEOSldas_CF90.catchcn_internal_rst.20000701_0000 catchcn_internal_rst > ls -l total 0 lrwxrwxrwx 1 fzeng g0620 141 2018-09-04 13:30 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000/M07/GEOSldas_CF90.catchcn_internal_rst.20000701_0000 lrwxrwxrwx 1 fzeng g0620 116 2018-07-25 13:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/GEOSldas_CF90.vegdyn_internal_rst Double check: > cd ../../run > cat CAP.rc BEG_DATE: 19810101 000000 END_DATE: 20180801 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 2 > cat cap_restart 20000701 000000 > ls -l ../input/restart/ total 0 lrwxrwxrwx 1 fzeng g0620 141 2018-09-04 13:30 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000/M07/GEOSldas_CF90.catchcn_internal_rst.20000701_0000 lrwxrwxrwx 1 fzeng g0620 116 2018-07-25 13:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/GEOSldas_CF90.vegdyn_internal_rst Everything looks correct, so submit the job: > qsub lenkf.j 4. Extended the e0004s_transientCO2_05 run to 20180731 so that we can use the output here to examine the impact of 2018 summer heat wave, since Tair from the GEOSldas_CF90 run is having some issue now. > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/run > cat year_co2.txt 2018 > cp lenkf.45.j lenkf.46.j > nedit lenkf.46.j & #SBATCH --time=05:00:00 #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2018/M05/e0004s_transientCO2.ldas_log.20180501_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2018/M05/e0004s_transientCO2.ldas_out.20180501_0000.txt -start_year 2018 -start_month 5 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2018 -end_month 8 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ > mkdir -p ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2018/M05 > qsub lenkf.46.j 5. Checked the output of clm4_pPRCP_DE720_20 using ~/matlab/princetonVSmerra2/compare_princeton_merra2.m and ~/matlab/m_map/fz_plot_princeton_merra2_diff.m ========== 20180905: 1. GEOSldas tutorial meeting. 2. Prepared slides for 2pm meeting with Randy and Eunjee. 3. Talked to Weiyuan about the GEOSldas issue: The run that I started from 20000701 wrote out daily output files, but these daily output was not averaged to create new monthly output -- the old monthly output files are still there. 4. Investigated why the difference in total precipitation between clm4_pPRCP_DE720_20 and p0007s_67 is larger than I expected. ========== 20180906: 1. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4/Linux/bin > ls -l LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70033364 2018-07-13 16:45 LDASsaCN_mpi.x* My notes on 20180713 show that I restored catchment.F90 and compiled GEOScatch_GridComp, GEOSlana_GridComp and Applications/LDAS_App, and then saved Linux directory in exec/clm4. I should have modified catchmentCN.F90 instead of catchment.F90 at that time. That's why I restored catchment.F90 on 20180713. -rwxr-xr-x 1 fzeng g0620 23806 2018-07-13 13:41 CNNStateUpdate1Mod.F90* The print statement about livestemn I added is commented out in this CNNStateUpdate1Mod.F90, and this CNNStateUpdate1Mod.F90 is slightly older than the executable in exec/clm4/Linux/bin (2018-07-13 16:45), so this executable is good for the standard control run with clm4. 2. Set up a clm4_Princeton_DE720 run that uses Princeton forcing. Modified D0.5_cn.exe and D0.5.bat in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/run > nedit lenkf.0.j & changed "-restart_path ../input/restart/rst_clm4 \" to "-restart_path ../input/restart/output \" > cp -p ../../clm4_DE720_42/run/edit_lenkf.csh . > cp -p ../../clm4_DE720_42/run/LSM_Calib.rc . > mkdir ../lenkf_orig > cp -p lenkf.*.j ../lenkf_orig/. > ./edit_lenkf.csh Remove the last line "/home/fzeng/Catchment/CLM4/submit_next_batch_DE720 clm4_DE720" in lenkf.4.j because I will do only one cycle. > cd ../input/restart > ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output > /bin/rm rst_clm4 > cd ../../run > ls -l ../build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70033364 2018-07-13 16:45 ../build/Linux/bin/LDASsaCN_mpi.x* > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 76 2018-09-06 11:05 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output/ Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800103_030730z. > /bin/rm ../output/global/rc_out/* > qsub lenkf.0.j 3. Sarith provided me this shell script to average daily output to monthly: /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run/daily2monthly.csh Ran it to process the daily GEOSldas_CF90 output to get monthly files and remove the daily files. 4. Seminar by Yao Zhang. 5. Meeting with Eunjee. 6. Meeting with Randy, Joanna, Yao and Eunjee. ========== 20180907: 1. Ran /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run/daily2monthly.csh to process the 20050101-20061231 daily GEOSldas_CF90 output to get monthly files and remove the daily files. 2. Created and used /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run/delete_old_monthly.csh to delete the old monthly files from 200701 through 201807. 3. Re-do the GEOSldas_CF90_satcap run because it started from 20001231_21z so the monthly output will have the same issue in the last month of each job segment as what we found for the GEOSldas_CF90 run on 20180904: /discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas_CF90_satcap GEOSldas_CF90_satcap_old > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > nedit CF0090x6C.exec & EXP_ID : GEOSldas_CF90_satcap EXP_DOMAIN : CF0090x6C NUM_ENSEMBLE : 1 BEG_DATE : 20010101 000000 END_DATE : 20020101 000000 RESTART : 1 RESTART_ID : GEOSldas_CF90 RESTART_PATH : /discover/nobackup/fzeng/Catchment/M2n5P/ RESTART_DOMAIN : CF0090x6C > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/satcap/Linux/bin > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/ > source $ESMADIR/src/g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0090x6C.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0090x6C.bat > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/run/ Check restart file: > cat cap_restart 20010101 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 155 2018-09-07 10:57 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/output/CF0090x6C/rs/ens0000/Y2001/M01/GEOSldas_CF90_satcap.catchcn_internal_rst.20010101_0000 lrwxrwxrwx 1 fzeng g0620 130 2018-09-07 10:57 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_satcap.vegdyn_internal_rst Check executable: > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50418780 2018-08-10 13:19 ../build/Linux/bin/GEOSldas.x* CAP.rc: check the END_DATE, JOB_SGMT and NUM_SGMT. No change needed. HISTORY.rc: tavg1_2D_mf-st.resolution: 360 181, tavg1_2D_lnd_Nx.resolution: 360 181, Added to L236: 'CNCO2' , 'CATCHCN' , LDAS.rc: added "IMPOSE_CT_CO2: 1" to the end (L38). Did an interactive run: > interactive.py -A sp3 -n 168 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/run > cp lenkf.j lenkf.j.orig > nedit lenkf.j & Add "exit" after L179 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > ./lenkf.j It's running. Stopped it at 2001-01-01T04:00:00. > mv lenkf.j.orig lenkf.j > qsub lenkf.j 4. The clm4_Princeton_DE720 run has finished simulating the period from 19800101 through 20121231. Processed the output: ~/Catchment/CLM4 > grid_restore_hdeg clm4_Princeton_DE720 Compared PRCP between clm4_Princeton_DE720 and clm4_pPRCP_DE720_20 using ~/matlab/princetonVSmerra2/compare_princeton_merra2.m and ~/matlab/m_map/fz_plot_princeton_merra2_diff.m Also compared T2M between clm4_Princeton_DE720 and clm4_pTair_DE720_33. Found something strange: Princeton is higher than pTair in NH and lower in SH for the first half of the year, and the difference is the opposite in the 2nd half of the year. Checked ~/matlab/princetonVSmerra2/compare_princeton_merra2.m and ~/matlab/m_map/fz_plot_princeton_merra2_diff.m, but couldn't anything wrong. Checked ~/Catchment/princeton_monthly_cli.f90. It takes into account leap year. Correct! ~/Catchment/merra2_monthly_cli.f90 deals with monthly files, so no need to consider leap year. Next: check these pre-processing scripts more carefully; also check my model code modifications 5. Modified mk_LDASsaRestarts.F90 to create new restart file (19 PFTs, CNCOL=35, CNPFT=75) for Catchment-CN with CLM4.5 from /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_52/output/global/rs/ens0000/Y2017/M01/clm4.5_DE720.ens0000.catchcn_ldas_rst.20170101_0000z (27 PFTs, CNCOL=39, CNPFT=104). > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts > mkdir 0.5D_clm4.5 > cd 0.5D_clm4.5 > cp -p ../0.5D/mk_LDASsaRestarts.F90 . (1) First I need to re-order the clm4.5_DE720.ens0000.catchcn_ldas_rst.20170101_0000z so that the tiles are in ascending order of tile id. Followed notes on 20180626. (1.1) Create path: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5 mkdir reordered (1.2) Create shell script "reorder_restarts" for reordering: cd reordered nedit reorder_restarts & chmod 755 reorder_restarts reorder_restarts reads: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/bin/csh set YYYY = 2017 set YYYYMMDD=`echo ${YYYY}0101` mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/ -d ${YYYYMMDD} -e clm4.5_DE720 -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_52/output/global/ -m catchcn -s 50 -r Y -t DE_00720x00360_PE_0720x0360.til ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: cp -p ../mk_LDASsaRestarts.F90 . nedit mk_LDASsaRestarts.F90 & [Actually no change in this case.] In SUBROUTINE reorder_LDASsa_restarts: (a) L298: tile_coord = trim(EXPDIR)//'rc_out/'//trim(expname)//'.ldas_tilecoord.bin' (b) L540: rst_file = trim(EXPDIR)//'rs/ens0000/Y'//YYYYMMDD(1:4)//'/M'//YYYYMMDD(5:6)//'/'//trim(ExpName)//& '.ens0000.catchcn_ldas_rst.'//trim(YYYYMMDD)//'_0000z' cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ source $ESMADIR/src/g5_modules gmake install (1.4) Run "reorder_restarts" to reorder the offline restart file: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5/reordered ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin reorder_restarts Issue 1: Complained about "Index exceeds dimension bound". Need Sarith to make another dummy file for the old CLM4.5 rst under Rolf's directory. Solution: Talked to Sarith. He said I can just use the clm4.5_DE720.ens0000.catchcn_ldas_rst.20170101_0000z as the dummy file. L307-309: if(trim(MODEL) == 'CATCHCN') & ! InNCIO = MAPL_NCIOOpen('/discover/nobackup/rreichle/l_data/LandRestarts_for_Regridding/CatchCN/catchcn_internal_dummy' , rc=rc) InNCIO = MAPL_NCIOOpen('/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_52/output/global/rs/ens0000/Y2017/M01/clm4.5_DE720.ens0000.catchcn_ldas_rst.20170101_0000z' , rc=rc) Issue 2: BF1: NetCDF: Not a valid ID Stopped Solution: set clm45 to false, also L899-901 became if(NewLand) then open(unit=27, file=trim(DataDir)//'CLM4.5_veg_typs_fracs' ,form='formatted') open(unit=28, file=trim(DataDir)//'CLM_NDep_SoilAlb_T2m' ,form='formatted') Also made some other modifications. See /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5/reordered/mk_LDASsaRestarts.F90 Checked ABM and HDM in the catchcn_internal_rst.20170101 created. They look correct. (2) Regrid from old CLM4.5 rst (nPFT=27, CNCOL=39, CNPFT=105) to the new CLM4.5 rst (nPFT=19, CNCOL=35, CNPFT=75), no change in tile system: (2.1) Create path: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5 mkdir regridded (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: cd regridded cp ../reordered/mk_LDASsaRestarts.F90 . nedit mk_LDASsaRestarts.F90 & cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ source $ESMADIR/src/g5_modules gmake install (2.3) Run mk_LDASsaRestarts to convert to desired tile system: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5/regridded ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin nedit mkLDASsa.j & [Create a job file] mkLDASsa.j reads: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/bin/csh -f #SBATCH --account=g0620 #SBATCH --time=1:00:00 #SBATCH --qos=debug #SBATCH --ntasks=56 #SBATCH --job-name=mkLDAS #SBATCH --constraint=hasw #SBATCH --output=mkLDAS.o #SBATCH --error=mkLDAS.e setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05 source $ESMADIR/src/g5_modules setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2 limit stacksize unlimited setenv SPONSORID g0620 setenv BCSDIR /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/ setenv TILFILE DE_00720x00360_PE_0720x0360.til mkdir -p OutData1/ mkdir -p OutData2/ ln -s $BCSDIR/$TILFILE OutData1/OutTileFile ln -s $BCSDIR/$TILFILE OutData2/OutTileFile ln -s $BCSDIR/clsm OutData2/clsm mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y /bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 bin/Scale_CatchCN OutData1/catchcn_internal_clm45 OutData2/catchcn_internal_clm45 catchcn_internal_clm45 50 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ chmod 755 mkLDASsa.j sbatch mkLDASsa.j Finished. mv catchcn_internal_rst catchcn_internal_rst.20150301_0000z mv OutData1 OutData1_20150301_0000z mv OutData2 OutData2_20150301_0000z ========== 20180910: 1. Continued working on (1.4) and (2) in 5 of 20180907. 2. The clm4_pPRCP_DE720 run stopped when it's simulating 19870707 due to time out. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/run > ls -l ../output/global/rs/ens0000/Y1987/M07 total 284640 -rw-r--r-- 1 fzeng g0620 291445863 2018-09-06 20:36 clm4_pPRCP_DE720.ens0000.catchcn_ldas_rst.19870701_0000z Tried starting from 19870701 but got "float divide by zero" in catchmentCN.F90. This restart file may have some problem. Tried starting from 19870601 and it worked. > cp lenkf.0.j lenkf.0b.j > nedit lenkf.0b.j & #SBATCH --time=01:00:00 #SBATCH --qos=debug #SBATCH --output=../output/global/rc_out/Y1980/M01/clm4_pPRCP_DE720.ldas_log.19870601_0000.txt #SBATCH --error=../output/global/rc_out/Y1980/M01/clm4_pPRCP_DE720.ldas_out.19870601_0000.txt -start_year 1987 -start_month 6 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 1988 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -restart_path ../output \ > qsub lenkf.0b.j ========== 20180911: Full day IDS meeting at UMD. ========== 20180912: 1. IDS meeting at UMD until ~1:15pm. 2. Read /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5/regridded/mk_LDASsaRestarts.F90 and ~gkwalker/geos5/convert_offline_cn_restart.f90 to better understand how the regridding works. ========== 20180913: 1. Read and modified /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D_clm4.5/regridded/mk_LDASsaRestarts.F90. Continued working on 5(2) of 20180907. ========== 20180914: 1. Continued working on 5(2) of 20180907. 2. GMAO Carbon Meeting #13. 3. Processed the clm4_pPRCP_DE720_20 through 24 output. ========== 20180917: 1. Continued working on 5 of 20180907 to convert the old CLM4.5 rst to the new CLM4.5 rst (that uses CLM4 veg map). However, for a lot of tiles that have type 19 in the CLM4 veg map (which the new CLM4.5 rst uses), it's very hard to find a tile in the original/old CLM4.5 veg map (i.e. with prognostic crops) that has a crop type or a type in the same class. Therefore, it keeps seeking and the job is hanging there. This could be due to the difference in the vegetation maps between CLM4 and CLM4.5. Gave up and tried to create the new CLM4.5 rst from the CLM4 M09 rst in Rolf directory. Actually back in May 2018 (2018-05-22) I already did this. However, there is a bug in the mk_LDASsaRestarts.F90, which is "sub_ityp1(i)>fmin" in L2326. It should be "sub_fevg1(i)>fmin" instead. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts > cp -pr 0.5D 0.5D_20180522 > cd 0.5D > nedit mk_LDASsaRestarts.F90 & [changed "sub_ityp1(i)>fmin" to "sub_fevg1(i)>fmin" in L2326.] > cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ > source $ESMADIR/src/g5_modules > gmake install > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D > ls -l bin/mk_LDASsaRestarts -rwxr-xr-x 1 fzeng g0620 27811798 2018-09-17 15:24 bin/mk_LDASsaRestarts* > rm -rf catchcn_internal_clm45 catchcn_internal_rst mkLDAS.e mkLDAS.o OutData1/ OutData2/ > sbatch mkLDASsa.j It took less than 8 minutes. > ~smahanam/bin/cdo -diffn catchcn_internal_clm45 ../0.5D_20180522/catchcn_internal_clm45 Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter name 33 : 2001-02-01 21:00:00 0 7108920 0 489 : F T 4.8530 0.87403 : CNCOL 34 : 2001-02-01 21:00:00 0 60933600 0 726 : T T 0.0000 0.99995 : CNPFT 2 of 98 records differ cdo diffn: Processed 156396240 values from 188 variables over 2 timesteps ( 3.31s ) Checked the LAI data using ~/matlab/plot_catchcn_internal_rst_pftvars.m. 2. Test run: try to use different number of processors and see what's the best number for the fastest run speed. Removed some old directories: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 > rm -rf clm4.5_DE720_53_old > rm -rf clm4.5_DE720 Set up the new CLM4.5 1st cycle: > /bin/cp -r clm4.5_DE720_template clm4.5_DE720 > cd clm4.5_DE720 > cd input/restart/ > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/catchcn_internal_clm45 catchcn_internal_rst > /bin/rm rst_clm4.5 > ls -l total 0 lrwxrwxrwx 1 fzeng g0620 81 2018-09-17 16:24 catchcn_internal_rst -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D/catchcn_internal_clm45 > cd ../.. > cd run > cp lenkf.0.j.1stcycle lenkf.0a.j > nedit lenkf.0a.j & #SBATCH --time=01:00:00 #SBATCH --qos=debug -start_year 1980 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 1980 -end_month 2 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ removed "qsub lenkf.1.j" > ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70928420 2018-08-27 15:51 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/build/Linux/bin/LDASsaCN_mpi.x* Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run > ./lenkf.0a.j It's running. Stopped it at date_time_new 19800101_080730z. > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:56. > nedit lenkf.0a.j & #SBATCH --ntasks=168 mpirun -np 168 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:50. > nedit lenkf.0a.j & #SBATCH --ntasks=196 mpirun -np 196 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:44. > nedit lenkf.0a.j & #SBATCH --ntasks=224 mpirun -np 224 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:30. > nedit lenkf.0a.j & #SBATCH --ntasks=252 mpirun -np 252 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:34. > nedit lenkf.0a.j & #SBATCH --ntasks=280 mpirun -np 280 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:17. > nedit lenkf.0a.j & #SBATCH --ntasks=308 mpirun -np 308 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:10 > nedit lenkf.0a.j & #SBATCH --ntasks=336 mpirun -np 336 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:17. > nedit lenkf.0a.j & #SBATCH --ntasks=364 mpirun -np 364 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:24 > nedit lenkf.0a.j & #SBATCH --ntasks=392 mpirun -np 392 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0a.j Finished in 00:06:18 3. Processed the clm4_pPRCP_DE720_25 output. ========== 20180918: 1. Continued working on 2 of 20180917. 2. Use 308 processors for Catchment-CN with CLM4.5 simulations. Started the 1st cycle of the new Catchment-CN with CLM4.5 spinup run. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run > cp lenkf.0.j.1stcycle lenkf.0.j > nedit lenkf.*.j & #SBATCH --ntasks=308 mpirun -np 308 \ > /bin/rm ../output/global/rc_out/* > qsub lenkf.0.j To save spinup time, changed restart output frequency from monthly to annual in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/clsm_ensdrv_out_routines.F90 (L2121). Recompiled: "gmake install" in GEOSlana_GridComp and Applications/LDAS_App. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec > mv clm4.5 clm4.5_20180827 > /bin/cp -pr Linux exec/clm4.5/. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > ls -l exec/clm4.5/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70928420 2018-09-18 13:53 exec/clm4.5/Linux/bin/LDASsaCN_mpi.x* /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720 > ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70928420 2018-09-18 13:53 build/Linux/bin/LDASsaCN_mpi.x* Did an interactive run to make sure it works. 3. Re-analyzed the GEOSldas_CF90_satcap output using ~/matlab/plot_eint_evap.m All these maps I showed Randy on 20180822 were updated on 18 Sep 2018 after re-running the control and experiment (the previous runs started from 21z and could have some issue in the monthly output). No visible change (eyeballing). 4. Continued working on 4 of 20180907. Compared PRCP between clm4_Princeton_DE720 and clm4_pPRCP_DE720_20 using ~/matlab/princetonVSmerra2/compare_princeton_merra2.m and ~/matlab/m_map/fz_plot_princeton_merra2_diff.m Also compared T2M between clm4_Princeton_DE720 and clm4_pTair_DE720_33. Found something strange: Princeton is higher than pTair in NH and lower in SH for the first half of the year, and the difference is the opposite in the 2nd half of the year. Double checked ~/matlab/princetonVSmerra2/compare_princeton_merra2.m and ~/matlab/m_map/fz_plot_princeton_merra2_diff.m, but still couldn't anything wrong. ~/Catchment/merra2_monthly_cli.f90 deals with monthly files, so no need to consider leap year. Double checked ~/Catchment/princeton_monthly_cli.f90. It takes into account leap year, but there may be a bug in computing dofyr. This bug would affect the climatology of March through December. ~/Catchment > cp -p princeton_monthly_cli.f90 princeton_monthly_cli.f90.gkw Fixed the bug in princeton_monthly_cli.f90 and compiled the program. > cd /discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > mkdir old > mv tas* old/ Ran the program to re-compute the Tair climatology: ~/Catchment > princeton_monthly_cli discover17:/discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > diff tas_01.dat old/. discover17:/discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > diff tas_02.dat old/. discover17:/discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > diff tas_03.dat old/. Files tas_03.dat and old/./tas_03.dat differ discover17:/discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > diff tas_04.dat old/. Files tas_04.dat and old/./tas_04.dat differ Extend the clm4_pTair_DE720 run: > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 > rm -rf clm4_pTair_DE720 > set exp = clm4_pTair_DE720 > set num = 33 > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 > /bin/cp -r ${exp}_template ${exp} > cd ${exp}/input/restart > ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num}/output > /bin/rm rst_clm4 > ls -l total 0 lrwxrwxrwx 1 fzeng g0620 82 2018-09-18 17:32 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720_33/output/ > cd ../../run [Did an interative run. It's running. Stopped it at date_time_new 19800101_060730z.] > nedit lenkf.4.j & #/home/fzeng/Catchment/CLM4/submit_next_batch_DE720 clm4_pTair_DE720 > qsub lenkf.0.j 5. Set up and automate the clm4_Princeton_DE720 spinup run that uses Princeton forcing. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 > mv clm4_Princeton_DE720 clm4_Princeton_DE720_20 (1) Create the template: Checked and made sure D0.5_cn.exe and D0.5.bat in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run are identical to those in clm4_Princeton_DE720_20. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4/Linux/bin > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/ > mv clm4_Princeton_DE720 clm4_Princeton_DE720_template > cd clm4_Princeton_DE720_template/run > nedit lenkf.0.j & changed "-restart_path ../input/restart/rst_clm4 \" to "-restart_path ../input/restart/output \" -restart_id clm4_Princeton_DE720 \ > cp -p ../../clm4_DE720_42/run/edit_lenkf.csh . > cp -p ../../clm4_DE720_42/run/LSM_Calib.rc . > mkdir ../lenkf_orig > cp -p lenkf.*.j ../lenkf_orig/. > ./edit_lenkf.csh Change the last line to "/home/fzeng/Catchment/CLM4/submit_next_batch_DE720 clm4_Princeton_DE720" in lenkf.4.j (2) Set up the clm4_Princeton_DE720_21 cycle following steps in /home/fzeng/Catchment/CLM4/submit_next_batch_DE720: > set exp = clm4_Princeton_DE720 > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 > /bin/cp -r ${exp}_template ${exp} > cd ${exp}/input/restart > ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720_20/output > /bin/rm rst_clm4 > cd ../../run Check the executable: > ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70033364 2018-07-13 16:45 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/build/Linux/bin/LDASsaCN_mpi.x* Matches that for the 20th cycle (see 20180906). Check the restart file: /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/input/restart > ls -l lrwxrwxrwx 1 fzeng g0620 86 2018-09-18 15:09 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720_20/output/ > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/run > nedit lenkf.0.j & [No change.] Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720/run > ./lenkf.0.j It's running. Stopped it at date_time_new 19800101_183730z. > /bin/rm ../output/global/rc_out/* > qsub lenkf.0.j ========== 20180919: 1. Checked the clm4.5_DE720 run. Using 308 processes: It was waiting in the queue for 8 hours before it started to run. Emailed Melanie and Pete and asked if there is a way to shorten the wait time. It was able to finish 4 years of simulation within 5 hours, so it can finish at least 8 years of simulation within 12 hours. Changed the job scripts to make the run more efficient. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/output/global > ls -l rs/ens0000/Y1988/M01 total 296800 -rw-r--r-- 1 fzeng g0620 303909763 2018-09-19 09:40 clm4.5_DE720.ens0000.catchcn_ldas_rst.19880101_0000z So lenkf.0.j and lenkf.1.j are done. lenkf.2.j is now in the queue. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run > nedit lenkf.*.j & lenkf.2.j: -start_year 1988 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 1996 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ lenkf.3.j: -start_year 1996 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2004 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ lenkf.4.j: -start_year 2004 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2012 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ lenkf.5.j: -start_year 2012 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2017 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ /home/fzeng/Catchment/CLM4.5/submit_next_batch_DE720 clm4.5_DE720 Delete the old lenkf.2.j from the queue and submit the new lenkf.2.j: > qdel 26873529 > qsub lenkf.2.j 2. Updated the clm4.5_DE720_template. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_template/run > nedit lenkf.*.j & #SBATCH --ntasks=308 mpirun -np 308 \ lenkf.0.j: -start_year 1980 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 1988 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ lenkf.1.j: -start_year 1988 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 1996 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ lenkf.2.j: -start_year 1996 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2004 -end_month 1 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ However, this will be messy because the log files will not be in the right places they are supposed to be (this will be the case for the new clm4.5_DE720_01). Undo all these changes I just made. Now that the clm4.5_DE720 is still waiting in the queue. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 > mv clm4.5_DE720 clm4.5_DE720_currentcycle > mv clm4.5_DE720_template clm4.5_DE720_template_20170809 [Note: I forgot to do this before the following few steps. Therefore, when I did "mv clm4.5_DE720 clm4.5_DE720_template" below, the newly generated clm4.5_DE720 was moved into clm4.5_DE720_template. I then renamed clm4.5_DE720_template to clm4.5_DE720_template_20170809, and moved clm4.5_DE720 one-level up and renamed it to clm4.5_DE720_template. This is why the time stamp of clm4.5_DE720_template was changed to 2018-09-19 11:42.] Create a new template: Checked D0.5_cn.exe and D0.5.bat in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run. D0.5.bat: ntasks = 308 > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4.5/Linux/bin > ls -l LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 70928420 2018-09-18 13:53 LDASsaCN_mpi.x > source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/ > mv clm4.5_DE720 clm4.5_DE720_template > mv clm4.5_DE720_currentcycle clm4.5_DE720 clm4.5_DE720 is still in the queue, so what I did will not affect it. > cd clm4.5_DE720_template/run > nedit lenkf.0.j & changed "-restart_path ../input/restart/rst_clm4 \" to "-restart_path ../input/restart/output \" > cp -p ../../../clm4/clm4_DE720_42/run/edit_lenkf.csh . > cp -p ../../../clm4/clm4_DE720_42/run/LSM_Calib.rc . > mkdir ../lenkf_orig > cp -p lenkf.*.j ../lenkf_orig/. > ./edit_lenkf.csh Change the last line to "/home/fzeng/Catchment/CLM4.5/submit_next_batch_DE720 clm4.5_DE720" in lenkf.4.j 3. Processed the clm4_pPRCP_DE720_26 output. 4. Python HW2. Learn python. 5. Contacted Kristi (617) about disaggregating S2S hindcasts data from daily to subdaily. ========== 20180920: 1. > cd /archive/u/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5 > mkdir old > mv * old/ 2. Prepared meeting slides -- will report work to Randy next week. 3. Investigated the high LAI41 issue. Created plot_catchcn_internal_rst_pftvars_M09.m and M09tile2HD.m in ~/matlab to plot LAI41 in /gpfsm/dnb42/projects/p16/ssd/land/l_data/LandRestarts_for_Regridding/CatchCN/M09/20151231/catchcn_internal_rst. This catchcn_internal_rst used my /gpfsm/dnb31/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_41. Next will plot LAI41 in this file. ========== 20180921: 1. Weiyuan created a python script for me to speed up the spinup. His email on 20180920: Fanwei, I have created a python run script for you. /discover/nobackup/wjiang/bin/ldas.py Copy this file to any of your folder. Before you run it, you only need to change "nSegments = 5" to fit your situation. For example, if you have ten jobs, from lenkf.0.j, lenkf.1.j, .... lenkf.9.j, then change it to "nSegments = 10" After you run it, ./ldas.py it will produce a file "ldas_batchrun.j". Copy this file to your run directory with lenkf.0.j, lenkf.1.j, ... ... Then you only need to run ./ldas_batchrun.j Let me know if you have any question. Cheers Weiyuan > /discover/nobackup/fzeng > cp /discover/nobackup/wjiang/bin/ldas.py . > mv ldas.py ldas_WJ.py > mkdir Weiyuan > mv ldas_WJ.py Weiyuan/ > cd Weiyuan Set nSegments = 5 in ldas_WJ.py > ./ldas_WJ.py > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_template/run > nedit lenkf.*.j & Removed "qsub lenkf.N+1.j" from the end of lenkf.N.j. > cp -p /discover/nobackup/fzeng/Weiyuan/ldas_batchrun.j . > cd /home/fzeng/Catchment/CLM4.5 > cp -p submit_next_batch_DE720 submit_next_batch_DE720_old > nedit submit_next_batch_DE720 & L45: changed "qsub lenkf.0.j" to "./ldas_batchrun.j" 2. The 34th cycle of clm4_pTair_DE720 has finished. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720 > ls -l output/global/rs/ens0000/Y2017/M01 total 284640 -rw-r--r-- 1 fzeng g0620 291445863 2018-09-21 07:17 clm4_pTair_DE720.ens0000.catchcn_ldas_rst.20170101_0000z > set exp = clm4_pTair_DE720 > set num = 34 > $HOME/Catchment/CLM4/grid_restore_hdeg clm4_pTair_DE720 > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 > mv ${exp} ${exp}_${num} > cd ${exp}_${num}/output/global/rs/ens0000 > /bin/rm -rf Y19?? Y200? Y201[0-6] > mkdir -p Y1980/M01 > cd Y1980/M01 > ln -s ../../Y2017/M01/${exp}.ens0000.catchcn_ldas_rst.20170101_0000z ${exp}.ens0000.catchcn_ldas_rst.19800101_0000z > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num} > /bin/rm -rf lenkf_orig Analyzed the output using: ~/matlab/princetonVSmerra2/compare_princeton_merra2.m ~/matlab/m_map/fz_plot_princeton_merra2_diff.m Now the temperature climatology difference between clm4_pTair_DE720 and clm4_Princeton_DE720 is very small. Great! 3. Because of the bug in ~/Catchment/princeton_monthly_cli.f90 found on 20180918, the Princeton precipitation climatology needs to be re-calculated. The clm4_pPRCP_DE720 run is using the Princeton precipitation climatology now, so can't do the re-calculation now. Made clm4_pPRCP_DE720 stop when this current cycle is done. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/run > nedit lenkf.4.j & #/home/fzeng/Catchment/CLM4/submit_next_batch_DE720 clm4_pPRCP_DE720 4. Continued investigating the high LAI41 issue. /gpfsm/dnb31/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_41 used LDASsa_m3_16, so the restart file is already catchcn_internal_rst and the tiles are not in ascending order. e0004s_28 is the 1st cycle I did following Greg's e0004s_27. Its restart file is CN_restart. Create ~/matlab/plot_CN_restart_pftvars_M09.m to plot LAI41 in /gpfsm/dnb31/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_28. LAI41 looks good here. Modified ~/matlab/plot_catchcn_internal_rst_pftvars_M09.m so that it also works for restart files in which tiles are not in ascending order of tile id. Used it to plot LAI41 in the restart file of /gpfsm/dnb31/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_41. 5. Took a quick look at the output of the new clm4.5_DE720_01. ========== 20180924: 1. The clm4.5_DE720 run stopped because the bcs directory in driver_inputs_0.5D_GLOBAL.nml is still Sarith's directory which no longer exists. Fixed the issue and re-submitted the job. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run > cp ../../clm4.5_DE720_26918138/run/driver_inputs_0.5D_GLOBAL.nml . > ./ldas_batchrun.j Also corrected it in the template and one more place. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720_template/run > cp ../../clm4.5_DE720_26918138/run/driver_inputs_0.5D_GLOBAL.nml . > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/run > cp -p ../clm4.5_DE720_26918138/run/driver_inputs_0.5D_GLOBAL.nml . 2. Re-calculate the Princeton PRCP climatology. > cd /discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > mv prcp* old/ Modified princeton_monthly_cli.f90 and re-ran it to calculate the Princeton PRCP climatology. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 > mv clm4_pPRCP_DE720_26906141 clm4_pPRCP_DE720_27 3. Processed the clm4_pPRCP_DE720 cycle that finished over the weekend and resumed the clm4_pPRCP_DE720 spinup run. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 > set exp = clm4_pPRCP_DE720 > set num = 28 > ls -l $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}/output/global/rs/ens0000/Y2017/M01/${exp}.ens0000.catchcn_ldas_rst.20170101_0000z -rw-r--r-- 1 fzeng g0620 291445863 2018-09-23 07:27 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/output/global/rs/ens0000/Y2017/M01/clm4_pPRCP_DE720.ens0000.catchcn_ldas_rst.20170101_0000z > $HOME/Catchment/CLM4/grid_restore_hdeg clm4_pPRCP_DE720 > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 > mv ${exp} ${exp}_${num} > cd ${exp}_${num}/output/global/rs/ens0000 > /bin/rm -rf Y19?? Y200? Y201[0-6] > mkdir -p Y1980/M01 > cd Y1980/M01 > ln -s ../../Y2017/M01/${exp}.ens0000.catchcn_ldas_rst.20170101_0000z ${exp}.ens0000.catchcn_ldas_rst.19800101_0000z > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num} > /bin/rm -rf lenkf_orig # set up the next cycle > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_template/run/ > cp -p $NOBACKUP/Weiyuan/ldas_batchrun.j . > nedit lenkf.*.j & Removed "qsub lenkf.N+1.j" from lenkf.N.j. > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 > /bin/cp -r ${exp}_template ${exp} > cd ${exp}/input/restart > ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num}/output > /bin/rm rst_clm4 > ls -l total 0 lrwxrwxrwx 1 fzeng g0620 82 2018-09-24 11:16 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720_28/output/ > cd ../../run > ./ldas_batchrun.j > nedit /home/fzeng/Catchment/CLM4/submit_next_batch_DE720 & L45: Changed from "qsub lenkf.0.j" to "./ldas_batchrun.j" 4. Applied Weiyuan's method to the clm4_Princeton_DE720 spinup run. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_Princeton_DE720_template/run > cp -p $NOBACKUP/Weiyuan/ldas_batchrun.j . > nedit lenkf.*.j & Removed "qsub lenkf.N+1.j" from lenkf.N.j. > nedit /home/fzeng/Catchment/CLM4/submit_next_batch_Princeton_DE720 & L45: Changed from "qsub lenkf.0.j" to "./ldas_batchrun.j" 5. Processed the output of the spinup runs. clm4_pTair_DE720: 34 clm4_pPRCP_DE720: 27 clm4_Princeton_DE720: 20 ========== 20180925: 1. Re-computed the Princeton dswrf climatology. > cd /discover/nobackup/fzeng/princeton/v2/cli_monthly_1980_2012 > mv dswrf* old/ Currently the last cycle of clm4_pIPAR_DE720 is 34. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720_template/run (was accidently done to clm4_IPAR_DE720_template) > cp -p $NOBACKUP/Weiyuan/ldas_batchrun.j . > nedit lenkf.*.j & Removed "qsub lenkf.N+1.j" from lenkf.N.j. > cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 > set exp = clm4_pIPAR_DE720 > set num = 34 > cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 > /bin/cp -r ${exp}_template ${exp} > cd ${exp}/input/restart > ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num}/output > /bin/rm rst_clm4 > cd ../../run > ./ldas_batchrun.j 2. Prepared slides for meeting with Randy and Eunjee. 3. GMAO seminar. 4. Meeting with Joanna, Yasuko and Eunjee. 5. Meeting with Randy and Eunjee. ========== 20180926: 1. Meeting with Sarith on the pLAI issue and the high LAI41 issue. For the high LAI41 issue, we have to re-create a restart file from the CN_restart files in which LAI41 looks correct. Sarith needs a 19810101 restart file at M09 grid for some experiment(s). 2. Meeting with Joe on PEP. ========== 20180927: 1. Meeting with Abheera and Kristi. 2. Modified Greg's convert_offline_cn_restart.f90 to create a restart file from Eunjee's 2x2.5 degree run. Don't have permission to access Eunjee's files. Emailed her (she is on leave today and tomorrow). > cd ~/Catchment > cp /home/gkwalker/geos5/convert_offline_cn_restart.f90 regrid_offline_cn_restart.f90 > nedit regrid_offline_cn_restart.f90 & 3. SMAP nature run percentile data from Qing: (see her email on 20180921) L4 version OL4001 which is an ensemble version of the Nature Run 7.2 discover:/gpfsm/dnb02/projects/p51/SMAP_L4/L4_SM/OL4001/gph ========== 20180928: 1. Eunjee copied her files to her nobackup. Copied the M2.n5P.HRv2.transCO2.c4.tgz2 from there. > cd /discover/nobackup/fzeng/Eunjee > cp -p /discover/nobackup/elee15/offline/sims/co2var_transCO2/M2.n5P.HRv2.transCO2.c4.tgz . > tar -xf M2.n5P.HRv2.transCO2.c4.tgz > rm M2.n5P.HRv2.transCO2.c4.tgz 2. Continued to modifiy regrid_offline_cn_restart.f90. Compiled: ifort -o regrid_offline_cn_restart -traceback -mcmodel=medium -openmp regrid_offline_cn_restart.f90 Ran OPENMP interactive batch: salloc --ntasks=1 --cpus-per-task=28 --constraint=hasw --time=01:00:00 --qos=debug setenv OMP_NUM_THREADS 28 regrid_offline_cn_restart Crashed: output: 1684725 tiles fveg_new L486 in regrid_offline_cn_restart.f90: if(fveg_new(n,nv)<0..or. fveg_new(n,nv)>1.00) stop 'fveg_new' Added some print statement. output: 1684725 tiles 60.00000 fveg_new It's because fveg is now in percent instead of fraction. Added a line to do the conversion in regrid_offline_cn_restart.f90. This fixed the problem. It has finished, but printed out thousands of these: col error: 40010 1 31 3.1528352E+07 3.1525516E+07 3.1524884E+07 col error: 40010 1 37 1.000100 1.000010 2.5623973E-02 col error: 40010 2 31 3.1528352E+07 3.1525516E+07 3.1524884E+07 col error: 40010 2 37 1.000100 1.000010 2.5623973E-02 col error: 40010 2 38 1.000100 1.000010 2.5623973E-02 col error: 40010 3 31 3.1528352E+07 3.1525516E+07 3.1524884E+07 col error: 40010 3 37 1.000100 1.000010 2.5623973E-02 col error: 40010 3 38 1.000100 1.000010 2.5623973E-02 col error: 43691 1 31 3.1528352E+07 3.1525516E+07 3.1524884E+07 col error: 43691 1 37 1.000100 1.000010 2.5623973E-02 col error: 43691 1 38 1.000100 1.000010 2.5623973E-02 col error: 43691 2 31 3.1528352E+07 3.1525516E+07 3.1524884E+07 col error: 43691 2 37 1.000100 1.000010 2.5623973E-02 col error: 43691 2 38 1.000100 1.000010 2.5623973E-02 col error: 43691 3 31 3.1528352E+07 3.1525516E+07 3.1524884E+07 col error: 43691 3 37 1.000100 1.000010 2.5623973E-02 col error: 43691 3 38 1.000100 1.000010 2.5623973E-02 They are all related to ! 31 clm3%g%l%c%cps%annsum_counter ! 37 clm3%g%l%c%cps%fpg ! 38 clm3%g%l%c%cps%fpi Are these acceptable? Checked the LAI for the 4 veg types on Matlab. No. There must be something wrong -- the created M09 LAI11 has almost nothing (see my slides sent to Sarith). 3. Checked the GPP data from Joanna