========== 20161003: 1. The CF0180 run finished the 3rd cycle. /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-03 06:51 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-10-03 06:52 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_03 cd CF0180x6C_03/output/CF0180x6C_GLOBAL/rs/ens0000/ /bin/rm -rf Y19?? Y200[0-9] Y201[0-5] mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_03/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN --submit Checked the carbon pools for level of equilibrium using ~/Catchment/CF0180x6C/converge.gs. Still need at least a couple of cycles. 2. Sarith suggested me to check what N_catd really is in ens_driver_routines.F90: clsm_ensdrv_main.F90: N_catg = number of catchments in *g*lobal GEOS5 domain N_catf = number of catchments in *f*ull domain N_catl = number of catchments in *l*ocal processor's subdomain call propagate_ensemble( & N_ens, N_catl, N_force_pert, N_progn_pert, tile_coord_l, ...) ens_driver_routines.F90: subroutine propagate_ensemble( & N_ens, N_catd, N_force_pert, N_progn_pert, tile_coord, & Therefore, N_catd in subroutine propagate_ensemble is N_catl which is the number of catchments in local processor’s subdomain. N_catd in subroutine propagate_ensemble is the same as N_catl. And because it's N_catl instead of N_catg, when the model is run on multiple processors the mean values printed out by the master processor are different from those printed out by tmp.F90 which calculates global averages. I think the modifications I made to ens_driver_routines.F90 are correct. The output Rainf, Rainf_C and Snowf are very close to the input. 3. September report. 4. Met with Randy. Randy suggests to pick a small area and see if the total precipitation of this area for a particular date is the same or close between MERRA-2 and the tile-space data (coverted back to grid-space for verification purpose) I produced. ========== 20161004: 1. The e0004s spinup run finished another cycle. dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2015/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-10-03 23:43 e0004s.ens0000.catchcn_ldas_rst.20150101_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2014/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-10-03 23:44 e0004s.ens_avg.ldas_tile_monthly_out.201412.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_40 cd e0004s_40/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that the cycle number in the restart path in M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_40/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit Checked the carbon state convergence using ~/Catchment/SMAP_M09/convergeSMAP.gs. Compared to Greg's 26vs27 differences. Need at lease a couple of more cycles. 2. Worked on scripts to compare total precipitation of a small area for a particular date between MERRA-2 and the tile-space data (converted back to grid-space for verification purpose) I produced. To choose a small area which has integer number of grid cells for both MERRA-2 and the tile-space data: ~/Catchment/M2n5P_m0001/select_area_dailyPrcp_testing.f90 However, the MERRA-2 grids and the 2x2.5 grids do not overlap anywhere in either latitude or longitude direction. This means that it’s impossible to find a small area that contains integer number of MERRA-2 grids and integer number of 2x2.5 grids for comparison. Compared the tile-space daily mean precipitation data (converted back to grid-space) to the daily grid-space output of the previous daily CO2 and 3-hourly CO2 test runs. They look identical by eyeballing. 3. Processed GMAO GEOS5 seasonal forecast for October. ========== 20161005: 1. Emailed Randy the results of comparing the tile-space daily mean precipitation data (converted back to grid-space) to the daily grid-space output of the previous daily CO2 and 3-hourly CO2 test runs and asked for his opinion. Met with Randy. Will do another test run, save monthly grid-space output and compare to Eunjee's output to see if the precipitation data is identical. From /discover/nobackup/fzeng/Catchment/M2n5P/m0001: mv m0001_dailyPrcp m0001_dailyPrcp_4days mkdir -p m0001_dailyPrcp/RUN/rs/ens0000/Y2001/M01 cd m0001_dailyPrcp cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin cp ../m0001_dailyPrcp_4days/lenkf.j . nedit lenkf.j & (change the namelist) Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin sbatch lenkf.j 2. Completed 2016 Performance Review. 3. Compared my monthly grid-space precipitation output from the test run in 1 above to the monthly grid-space output from Eunjee's daily CO2 and 3-hourly CO2 runs. There is some difference in some grid cells. ========== 20161006: 1. Took a look at Eunjee's science snapshot. Helped her find papers that used Jung's GPP as a reference to evaluate models. 2. Investigated why my monthly grid-space precipitation output from the test run in 1 on 20161005 above is slightly different from Eunjee's output. I was wondering whether the difference has anything to do with subroutine check_forcing_nodata and subroutine repair_forcing in the model that I didn't apply when I averaged the hourly precipitation data to daily. However, the experiment log file of my daily CO2 test run shows that there is no missing data in the hourly data and the precipitation data does not need any repair. Therefore, I haven't figured out why there is this difference. However, since the difference is generally very small, I will just move on with the current method that I developed. Compared my daily grid-space precipitation output from the same test run to the daily output of my daily CO2 test run. Again there are a few grid cells showing some differences. ####3. Archive /discover/nobackup/fzeng/forcing/data/M2n5P/daily_grid to release some space from nobackup. 4. Found that I didn't check if there is any negative values in the original hourly SMAP nature 5 precipitation data in /discover/nobackup/fzeng/forcing/code/nc2bin_hrly2daily_M2n5P_precip.f90. Modified nc2bin_hrly2daily_M2n5P_precip.f90 to check this and re-ran it. /discover/nobackup/fzeng/forcing/data/M2n5P > mv daily_grid daily_grid_old Have finished 2008 so far. dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2001 daily_grid_old/2001 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2002 daily_grid_old/2002 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2003 daily_grid_old/2003 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2004 daily_grid_old/2004 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2005 daily_grid_old/2005 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2006 daily_grid_old/2006 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2007 daily_grid_old/2007 dali17:/discover/nobackup/fzeng/forcing/data/M2n5P > diff daily_grid/2008 daily_grid_old/2008 All returned nothing. So they are identical. Also modified /discover/nobackup/fzeng/forcing/code/grid2tile_M2n5P_precip_daily_LDAS.f90 to check whether there is any negative values in the created tile-space files. /discover/nobackup/fzeng/forcing/data/M2n5P > mv daily_tile_LDAS daily_tile_LDAS_old ####Compare files: daily_grid vs. daily_grid_old, daily_tile_LDAS vs. daily_tile_LDAS_old ####If identical, remove the old ones, then archive daily_grid 5. Worked on /discover/nobackup/fzeng/forcing/code/daily2monthly_tile_M2n5P_precip.f90 to average tile-space precipitation from daily to monthly. ========== 20161007: 1. After adding print and stop statements to check whether there is missing data in MERRA-2 precipitation, the output is the same as what I got earlier. /discover/nobackup/fzeng/forcing/data/M2n5P > diff --brief -r daily_grid daily_grid_old /discover/nobackup/fzeng/forcing/data/M2n5P > diff --brief -r daily_tile_LDAS daily_tile_LDAS_old Both returned nothing. /discover/nobackup/fzeng/forcing/data/M2n5P > /bin/rm -rf daily_grid_old daily_tile_LDAS_old Archived daily_grid and daily_tile_LDAS 2. Made more plots to verify the tile-space data I produced. Although it's impossible to pick a small area and compare the total precipitation, I picked a few grid cells and made time series plots. The grid cells in the tile-space data have lower values compared to the grid cells in MERRA-2, and the grid cells in the tile-space data is larger (i.e. 2x2.5 degree vs. MERRA-2 0.5x0.625 degree), so the total precipitation could be the same or very close between the two. Passed the daily mean precipitation run to Eunjee. 3. Worked on scripts to average tile-space precipitation from daily to monthly, and then linearly interpolate from monthly to daily. /discover/nobackup/fzeng/forcing/code: daily2monthly_tile_M2n5P_precip.f90 monthly2monthly_interp_tile_M2n5P_precip.f90 4. For the monthly interpolated precipitation run, modify ens_driver_routines.F90 in GEOSlana_GridComp, and compiled: "gmake clean" and "gmake install" in Components/GEOSlana_GridComp and then Applications/LDAS_App. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Applications/LDAS_App: cp -p LDASsa_mpi.x ../../../exec/LDASsa_mpi_M2n5P_m0001_Prcp_monthly_interp.x 5. Set up a monthly interpolated precipitation global test run to run for 3 months and save daily and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthly_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp ../m0001_dailyPrcp/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l RUN/rs/ens0000/Y2001/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j It's running. 6. Processed the output of the monthly interpolated precipitation global test run. ~/Catchment/M2n5P_m0001 > grid_restore_M2n5P_daily 0001 _Prcp_monthly_interp ~/Catchment/M2n5P_m0001 > grid_restore_M2n5P 0001 _Prcp_monthly_interp ========== 20161011: 1. The e0004s spinup run finished another cycle. dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2015/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-10-11 03:13 e0004s.ens0000.catchcn_ldas_rst.20150101_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2014/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-10-11 03:13 e0004s.ens_avg.ldas_tile_monthly_out.201412.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_41 cd e0004s_41/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that the cycle number in the restart path in M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_41/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit Checked the carbon state convergence using ~/Catchment/SMAP_M09/convergeSMAP.gs. Compared to Greg's 26vs27 differences. Need at lease a couple of more cycles. 2. The CF0180 run finished the 4th cycle. dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-10 09:02 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-10-10 09:02 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_04 cd CF0180x6C_04/output/CF0180x6C_GLOBAL/rs/ens0000/ /bin/rm -rf Y19?? Y200[0-9] Y201[0-5] mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_04/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN --submit Checked the carbon pools for level of equilibrium using ~/Catchment/CF0180x6C/converge.gs. Still need at least a couple of cycles. 3. Updated higgins rain data through 20161010. 4. Noticed that LAI and GPP are very low in Myanmar in the CF0180x6C_04 output. Checked my SMAP_M09 (e0004s_41). It's the same. Checked Greg's SMAP_M09 (e0004s_27). Same too. Why? 5. Verified the output of the monthly interpolated precipitation global test run. Compared to Eunjee’s monthly output Picked a grid cell to look at daily time series 6. Worked on scripts to calculate monthly mean SMAP nature 5 precipitation from monthly time series, and to interpolate to daily from monthly mean. /discover/nobackup/fzeng/forcing/code/monthly2monthlymean_tile_M2n5P_precip.f90 /discover/nobackup/fzeng/forcing/code/monthlymean2monthlymean_interpolated_tile_M2n5P_precip.f90 ========== 20161012: 1. Verified the monthly mean interpolated SMAP nature 5 precipitation data. 2. For the monthly mean interpolated precipitation test run, modified ens_driver_routines.F90 and compiled: "gmake clean" and "gmake install" in Components/GEOSlana_GridComp and then Applications/LDAS_App. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Applications/LDAS_App: cp -p LDASsa_mpi.x ../../../exec/LDASsa_mpi_M2n5P_m0001_Prcp_monthlymean_interp.x 3. Set up a monthly mean interpolated precipitation global test run to run for 3 months, and save daily tile output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthlymean_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp > cp ../m0001_Prcp_monthly_interp/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j 4. Verify output of the monthly mean interpolated precipitation global test run. ~/Catchment/M2n5P_m0001/verify_m0001_Prcp_monthlymean_interp.gs 5. Passed the year-specific monthly mean interpolated and climatological monthly mean interpolated precipitation runs to Eunjee. 6. Worked on script to calculate annual mean from monthly mean SMAP nature 5 precipitation. /discover/nobackup/fzeng/forcing/code/monthlymean2annualmean_tile_M2n5P_precip.f90 7. For the annual mean precipitation test run, modified ens_driver_routines.F90 and compiled: "gmake clean" and "gmake install" in Components/GEOSlana_GridComp and then Applications/LDAS_App. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Applications/LDAS_App: cp -p LDASsa_mpi.x ../../../exec/LDASsa_mpi_M2n5P_m0001_Prcp_annualmean.x 8. Set up an annual mean precipitation global test run to run for 2 months, and save daily tile output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_annualmean/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean > cp ../m0001_Prcp_monthlymean_interp/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j 9. Verify output of the annual mean precipitation global test run. ~/Catchment/M2n5P_m0001/verify_m0001_Prcp_annualmean.gs 10. Passed the annual mean precipitation run to Eunjee. ========== 20161013: 1. Worked on scripts to calculate 3hrly MERRA-2 surface air temperature (TLML) from the original hourly data, and convert 3hrly data from grid-space to tile-space. /discover/nobackup/fzeng/forcing/code/nc2bin_hrlyTO3hrly_M2n5P_Tair.f90 /discover/nobackup/fzeng/forcing/code/grid2tile_M2n5P_Tair_3hrly_LDAS.f90 2. Emailed Rolf the CF0180 and CF1440 offline test run results that have been shown to Randy. 3. Sarith asked me to create catchcn_internal_rst using the old offline CN_restart and land restart files (his current code that went through some modifications is very slow). Use mk_catchcn_bin2nc4.F90 and mk_LDASsaRestarts.F90 to create catchcn_internal_rst from the CN and land restarts of my SMAP_EASEv2_M09/e0004s_32. (1) Retrieve my e0004s_32 cycle from archive: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > cp -p /archive/u/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_32.tgz . /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > tar -xf e0004s_32.tgz (2) Copied mk_catchcn_bin2nc4.F90 and mk_LDASsaRestarts.F90 to a compiled GEOS-5 code for compile: /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > mv mk_LDASsaRestarts.F90 mk_LDASsaRestarts.F90.20160829 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > cp mk_catchcn_bin2nc4.F90 mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. Checked and made sure that they use the CN and land restart from e0004s_32, so no modification was made. To compile mk_catchcn_bin2nc4.F90 and mk_LDASsaRestarts.F90, from /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts: cp -p GNUmakefile GNUmakefile.orig nedit GNUmakefile & (to add mk_catchcn_bin2nc4) setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Heracles-5_2 source $ESMADIR/src/g5_modules gmake install (3) run mk_catchcn_bin2nc4: ssh discover-sp3 interactive.py -A sp3 -n 36 -a g0620 -X --debug borgt099:/discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > ./mk_catchcn_bin2nc4 Now I got catchcn_internal_rst under this directory: -rw-r--r-- 1 fzeng g0620 6718476686 2016-10-13 15:40 /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/catchcn_internal_rst /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > diff catchcn_internal_rst_e0004s_32 /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/catchcn_internal_rst Returned nothing. So the catchcn_internal_rst I just created is zero-diff from /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/catchcn_internal_rst_e0004s_32. So just use the current /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/catchcn_internal_rst_e0004s_32 in running mk_LDASsaRestarts below. (4) run mk_LDASsaRestarts interactively following the procedures in mk_LDASsaRestarts.F90: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > mkdir restarts_SM ssh discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X -t 02:00:00 (Took a long time and still in the queue when requesting for 12 hours) borgo053:~ > setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Heracles-5_2 borgo053:~ > source $ESMADIR/src/g5_modules borgo053:~ > limit stacksize unlimited borgo053:~ > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM (removed everything, if any, under this directory) borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > ln -s $ESMADIR/Linux/bin borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > mkdir -p OutData1/ OutData2/ borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > setenv BCSDIR /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/SMAP_EASEv2_M09/ borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > ln -s $BCSDIR/SMAP_EASEv2_M09_3856x1624.til OutData1/OutTileFile borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > ln -s $BCSDIR/SMAP_EASEv2_M09_3856x1624.til OutData2/OutTileFile borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > ln -s $BCSDIR/clsm OutData2/clsm borgo053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > mpirun -np 96 bin/mk_LDASsaRestarts M09 50 When done, scale: borgu053:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts_SM > bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 SURFLAY: 50.00000 Total Tiles: 1684725 Scaled Tiles: 342823 (20%) Performing Sanity Check ... CatDef Tiles: 0 (00%) SrfExc Tiles: 0 (00%) SrfExc Tiles: 0 (00%) ========== 20161014: 1. Both of my e0004s and CF0180 runs stopped around 6am this morning. Why? Maybe there was some issue with NCCS. Have to resubmit the jobs. (1) e0004s: dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2006/M08 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-10-14 04:14 e0004s.ens0000.catchcn_ldas_rst.20060801_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2006/M09 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-10-14 05:11 e0004s.ens0000.catchcn_ldas_rst.20060901_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2006/M10 total 0 dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2006/M08 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-10-14 05:11 e0004s.ens_avg.ldas_tile_monthly_out.200608.bin dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2006/M09 total 0 So it was simulating Y2006/M09 when it stopped. The log file in rc_out shows that it stopped when it's simulating 20060922. Tried this: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > cp lenkf.6.j lenkf.6new.j /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > nedit lenkf.6new.j & These lines become: #SBATCH --time=03:00:00 #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2006/M01/e0004s.ldas_log_new.20060101_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2006/M01/e0004s.ldas_out_new.20060101_0000.txt -start_year 2006 -start_month 9 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ New line added: sbatch lenkf.7.j Then: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > sbatch lenkf.6new.j It seems to be working. Add "sbatch lenkf.N+1.j" to the end of the rest of each lenkf.N.j. Do the same for the CF0180 run. (2) CF0180: dali15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y1993/M01 total 5998336 -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-14 00:46 CF0180x6C.ens0000.catchcn_ldas_rst.19930101_0000z -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-14 01:10 CF0180x6C.ens0000.catchcn_ldas_rst.19930131_2100z dali15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y1994/M01 total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-14 05:27 CF0180x6C.ens0000.catchcn_ldas_rst.19940101_0000z dali15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y1993/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-10-14 05:27 CF0180x6C.ens_avg.ldas_tile_monthly_out.199312.bin dali15:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y1994/M01 total 0 So it was simulating Y1994/M01 when it stopped. The log file in rc_out shows that it stopped when it's simulating 19940129. Did this: /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/run > cp lenkf.7.j lenkf.7new.j /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/run > nedit lenkf.7new.j & These lines become: #SBATCH --output=../output/CF0180x6C_GLOBAL/rc_out/Y1994/M01/CF0180x6C.ldas_log_new.19940101_0000.txt #SBATCH --error=../output/CF0180x6C_GLOBAL/rc_out/Y1994/M01/CF0180x6C.ldas_out_new.19940101_0000.txt New line added: sbatch lenkf.8.j Then: /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/run > sbatch lenkf.7new.j It's running and the new log file shows that it's working. Add "sbatch lenkf.N+1.j" to the end of the rest of each lenkf.N.j. 2. Worked on script to calculate year-specific monthly-mean 3hrly diurnal cycle of MERRA-2 surface air temperature (TLML) from 3hrly time series. Both are tile-space. /discover/nobackup/fzeng/forcing/code/monthly_diurnalcycle_tile_M2n5P_Tair.f90 ========== 20161017: 1. Converted tile-space 201411 and 201412 monthly diurnal cycle M2n5P Tair back to grid-space and made plots on GrADS to verify the program monthly_diurnalcycle_tile_M2n5P_Tair.f90 finished successfully on late Friday night of 20161014. 2. Worked on pre-processing MERRA-2 surface air temperature data: created and ran script to interpolate year-specific monthly-mean diurnal cycle to daily diurnal cycle, both are 3hrly: /discover/nobackup/fzeng/forcing/code/monthly_interp_diurnalcycle_tile_M2n5P_Tair.f90 created and ran script to calculate climatological monthly-mean diurnal cycle from year-specific monthly-mean diurnal cycle: /discover/nobackup/fzeng/forcing/code/monthlymean_diurnalcycle_tile_M2n5P_Tair.f90 created and ran script to interpolate climatological monthly-mean diurnal cycle to daily diurnal cycle, both are 3hrly: /discover/nobackup/fzeng/forcing/code/monthlymean_interp_diurnalcycle_tile_M2n5P_Tair.f90 3. Corrected the description of sndz and Rainf in all GrADS control files. ========== 20161018: 1. Made a silly mistake in the control file for the monthly mean interpolated T2m diurnal cycle data files and spent the whole moring trying to figure out what happened. Finally David helped me resolve the issue - I missed one % sign in the control file!!! 2. Forgot to put "_LDAS" at the end of directories earlier, now adding it: dali12:/discover/nobackup/fzeng/forcing/data/M2n5P > mv monthly_diurnalcycle_tile monthly_diurnalcycle_tile_LDAS dali12:/discover/nobackup/fzeng/forcing/data/M2n5P > mv monthly_interp_diurnalcycle_tile monthly_interp_diurnalcycle_tile_LDAS Also added "_LDAS" to the related FORTRAN programs. 3. For the monthly interpolated MERRA2 surface air temperature (hereafter T2m) diurnal cycle test run, modified ens_driver_routines.F90 and compiled: "gmake clean" and "gmake install" in Components/GEOSlana_GridComp and then Applications/LDAS_App. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Applications/LDAS_App: cp -p LDASsa_mpi.x ../../../exec/LDASsa_mpi_M2n5P_m0001_Tair_diurnalcycle_monthly_interp.x 4. Attended the NCCS User Forum. 5. Met with Eunjee. ========== 20161019: 1. The e0004s spinup run finished another cycle yesterday. dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2015/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-10-18 15:34 e0004s.ens0000.catchcn_ldas_rst.20150101_0000z dali13:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2014/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-10-18 15:34 e0004s.ens_avg.ldas_tile_monthly_out.201412.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_42 cd e0004s_42/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that the cycle number in the restart path in M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_42/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit Checked the carbon state convergence using ~/Catchment/SMAP_M09/convergeSMAP.gs. Compared to Greg's 26vs27 differences. Need at lease a couple of more cycles. 2. The CF0180 run finished the 5th cycle. dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-18 20:40 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-10-18 20:40 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_05 cd CF0180x6C_05/output/CF0180x6C_GLOBAL/rs/ens0000/ /bin/rm -rf Y19?? Y200[0-9] Y201[0-5] mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_05/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN --submit Checked the carbon pools for level of equilibrium using ~/Catchment/CF0180x6C/converge.gs. Still need at least a couple of cycles. 3. Set up a monthly interpolated T2m diurnal cycle global test run to run for 1 month, and save 3hourly tile output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Tair_diurnalcycle_monthly_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthly_interp > cp ../m0001_Prcp_monthly_interp/lenkf.j . Made sure that CN and land restarts match: dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthly_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthly_interp > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j Found that the print statement prints out more frequently than wanted. Stopped the test run. Modified ens_driver_routines.F90, re-compiled and re-submitted the test run. 4. For the monthly mean interpolated T2m diurnal cycle test run, modified ens_driver_routines.F90 and compiled: "gmake clean" and "gmake install" in Components/GEOSlana_GridComp and then Applications/LDAS_App. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Applications/LDAS_App: cp -p LDASsa_mpi.x ../../../exec/LDASsa_mpi_M2n5P_m0001_Tair_diurnalcycle_monthlymean_interp.x 5. Set up a monthly mean interpolated T2m diurnal cycle global test run to run for 1 month, and save 3hourly tile output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Tair_diurnalcycle_monthlymean_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthlymean_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthlymean_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthlymean_interp > cp ../m0001_Tair_diurnalcycle_monthly_interp/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthlymean_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Tair_diurnalcycle_monthlymean_interp > ls -l RUN/rs/ens0000/Y2001/M01/ total 37888 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j 6. Rewrite the year-specific, and climatological, monthly-mean interpolated M2n5P tile-space Tair diurnal cycle files from one file containing 8 fields to 8 files each containing one field (each field is one 3hrly mean within one day). This is to verify the output of the Tair test runs. 7. Verified output of the monthly and monthly mean interpolated T2m diurnal cycle global test runs. ~/Catchment/M2n5P_m0001/verify_m0001_Tair.gs ========== 20161020: 1. Used /home/fzeng/Catchment/princeton_tiles_location.f90 to locate tiles of interest US: 224576 37 66 -90.10 40.05 0.01852 18 19 1 1 0.4954 0.1175 0.3872 0.0000 2. Made time series plots of dP and mP of these tiles. 3. Reviewed my NDVI paper and tried to understand why GPP in the monthly interpolated precipitation case is lower than that in the daily precipitation case. 4. Archive: e0004s_42 CF0180_05 some Prcp and Tair data files 5. Eunjee's plots show that eint (interception evaporation) in the mP case is much higher than that in the dP case. This might have caused the lower GPP in the mP case compared to the dP case. Randy suggests to change the parameter values of FWETC and FWETL to smaller numbers and see if the excessive eint issue in the mP case can be resolved. For this, modified unified_rc3f_matrix_calc.F90, used 0.02 for both FWETC and FWETL and compiled: "gmake clean" and then "gmake install" in: GEOScatch_GridComp GEOSlana_GridComp Applications/LDAS_App However, got some error message when compiling GEOScatch_GridComp. It complained about clsm_ensdrv_mpi.mod. Used cleanup with "gmake clean" to compile instead. 6. Set up a monthly interpolated precipitation global test run to run for 1 month and save daily and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mv m0001_Prcp_monthly_interp m0001_Prcp_monthly_interp_defaultFWET /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthly_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp ../m0001_Prcp_monthly_interp_old/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l RUN/rs/ens0000/Y2001/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j It's running. Checked the exp log file. The FWETL and FWETC values are still the original ones. Why? Double checked and recompiled using cleanup with "gmake clean". ========== 20161021: 1. Re-do the monthly interpolated precipitation global test run m0001_Prcp_monthly_interp and run for 1 month. The FWETL and FWETC values printed out in the exp log are both 2.0000000E-02 which is what we want. Compared the eint/precip ratio of this new mP case to the dP case for Jan 2001. The ratio is still too large in the mP case in the subtropics. Randy suggested to use 0.002 for both FWETC and FWETL and try again. For this, modified unified_rc3f_matrix_calc.F90, used 0.002 for both FWETC and FWETL, and compiled using cleanup with "gmake clean". 2. Set up a monthly interpolated precipitation global test run to run for 3 months and save daily and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mv m0001_Prcp_monthly_interp m0001_Prcp_monthly_interp_FWETpt02 /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthly_interp/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp ../m0001_Prcp_monthly_interp_FWETpt02/lenkf.j . Made sure that CN and land restarts match: dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > ls -l RUN/rs/ens0000/Y2001/M01 total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j It's running. The exp log: FWETL = 2.0000001E-03 FWETC = 2.0000001E-03 SATCAPFR = 0.2000000 Correct. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mv m0001_Prcp_monthly_interp m0001_Prcp_monthly_interp_FWETpt002 evint/precip ratio in the 0.002 case is too low compared to the dP case. 0.02 case looks promising. Extended the dP and 0.02 FWET mP runs til May 2001 (only ran for one month Jan 2001 in both runs earlier). Wanted to extend both runs to the end of 2001 but NCCS is busy, so used debug runs and was able to simulate 4 more months within 1 hour. Therefore, have output for a total of 5 months. Compared evint/precip ratio between the dP and 0.02 FWET mP case for Feb through May of 2001. The ratio in the 0.02 FWET mP case still looks too high for all 5 months. 3. Tried using 0.01 FWET. Modified unified_rc3f_matrix_calc.F90 and compiled using cleanup with "gmake clean". Set up a monthly interpolated precipitation global test run to run for 5 months and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthly_interp_FWETpt01/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp > cp ../m0001_Prcp_monthly_interp_FWETpt02/lenkf.j . dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp_FWETpt01 > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthly_interp_FWETpt01 > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j The exp log: FWETL = 9.9999998E-03 FWETC = 9.9999998E-03 SATCAPFR = 0.2000000 Correct. 4. Passed both runs FWET 0.02 and FWET 0.01 to Eunjee. ========== 20161024: #### add more details: 1. Convergence: talk to Randy Randy suggests to do new SMAP M09 spinup and the following transient and real-time runs: (1) for pre-industrial period (before 1851): (2) transient (1851 - 2000): (3) 2001-2016: 2. Low lai, fpar, gpp and npp in Myanmar issue in the SMAP M09 and CF0180 runs. Talked to Sarith. Sarith showed me on Google Earth that central Myanmar is no/barely vegetated and that's why. He noticed this earlier. So what we see is correct. 3. Updated higgins rain data through 20161023. 4. Prepared for meeting with seminar speaker tomorrow. ========== 20161025: 1. For the new SMAP M09 e0004s spinup run: (1) modified process_cn.F90 and compute_rc.F90 to use the spatially and temporally varying CT CO2. (2) modified clsm_ensdrv_out_routines.F90 to write out restart files once a year instead of every month. (3) remove GEOS_CatchCNGridComp.F90 from GNUmakefile_openmp so that we don't need to make any modification to GEOS_CatchCNGridComp.F90. Talked to Sarith on how to save a single restart file and it gets override when the new one is written out during model run. Sarith said we can save restart file once a year or at end_time, and modify the job script to delete the old restart files that are not needed. In this case, don't use "--submit" option when running ldsetup. Sarith will provide me a shell script to make such modification and to add "sbatch lenkf.N+1.j" to the end of lenkf.N.j. 2. Seminar about carbon and water cycles by Pierre Gentine from Columbia. 3. Met with Pierre, Randy and Eunjee. ========== 20161026: 1. Copied Sarith's shell script that edits the job scripts to automatically remove old restart files not needed. /discover/nobackup/fzeng/Sarith > cp -p /discover/nobackup/smahanam/edit_lenkf.csh . 2. Compiled the offline code modified yesterdy: "gmake clean" and "gmake install" in GEOScatchCN_GridComp, GEOSlana_GridComp, and Applications/LDAS_App. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 > cp -pr ../../Linux/ . 3. Copied NDVI data to Discover for Manuela. /discover/nobackup/fzeng > mkdir GIMMS_NDVI gs6101-fzengm1:gimms_ndvi_qdeg_monthly_mat fzeng$ pwd /Volumes/G_DRIVE/02_Research_Postdoc/CorrelationProject/data/gimms_ndvi_qdeg_monthly_mat gs6101-fzengm1:gimms_ndvi_qdeg_monthly_mat fzeng$ scp ndvi_qdeg_20*.mat fzeng@dali.nccs.nasa.gov:/discover/nobackup/fzeng/GIMMS_NDVI gs6101-fzengm1:gimms_ndvi_qdeg_monthly_ascii fzeng$ pwd /Volumes/G_DRIVE/02_Research_Postdoc/CorrelationProject/data/gimms_ndvi_2014/gimms_ndvi_qdeg_monthly_ascii gs6101-fzengm1:gimms_ndvi_qdeg_monthly_ascii fzeng$ scp * fzeng@dali.nccs.nasa.gov:/discover/nobackup/fzeng/GIMMS_NDVI gs6101-fzengm1:gimms_ndvi_qdeg_monthly_ascii fzeng$ pwd /Volumes/G_DRIVE/02_Research_Postdoc/CorrelationProject/data/gimms_ndvi_2015_forjim/qdeg/gimms_ndvi_qdeg_monthly_ascii gs6101-fzengm1:gimms_ndvi_qdeg_monthly_ascii fzeng$ scp * fzeng@dali.nccs.nasa.gov:/discover/nobackup/fzeng/GIMMS_NDVI 4. The e0004s spinup run finished another cycle. dali12:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2015/M01 total 7354336 -rw-r--r-- 1 fzeng g0620 7547615847 2016-10-26 12:33 e0004s.ens0000.catchcn_ldas_rst.20150101_0000z dali12:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2014/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-10-26 12:34 e0004s.ens_avg.ldas_tile_monthly_out.201412.bin Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. Remove unneeded restart files and set up restart file for the next cycle: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_43 cd e0004s_43/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z Modify scripts for the next cycle: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run (1) M09.bat: job-name = e0004s_preindustrialCO2 (2) M09_CN_e0004s_spinup.exe: exp_id = e0004s_preindustrialCO2 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_43/output/ Run ldsetup to set up the experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 6 --landmodel catchCN (NOTE: Don't use the "--submit" option because the job files need to be modified manually before submitting the jobs). Run the modified version of Sarith's edit_lenkf.csh to modify the job scripts (now no need to further modify the first two and last job scripts manually after running edit_lenkf.csh): cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp /discover/nobackup/fzeng/Sarith/edit_lenkf.csh . ./edit_lenkf.csh submit a debug run: sbatch lenkf.0.j It's running, but the experiment log file shows that the CarbonTracker CO2 data is not read. (Why is it still running while there is no value in co2v?) The bug is in process_cn.F90 - I assume 10 minute model time step (same for the runs based on m3-15-CN) but the model time step here is actually 7.5 minutes. Modified process_cn.F90, compiled: "gmake install" in GEOSlana_GridComp, and Applications/LDAS_App. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2 Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 6 --landmodel catchCN (NOTE: Don't use the "--submit" option because the job files need to be modified manually before submitting the jobs). Run the modified version of Sarith's edit_lenkf.csh to modify the job scripts (now no need to further modify the first two and last job scripts manually after running edit_lenkf.csh): cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp /discover/nobackup/fzeng/Sarith/edit_lenkf.csh . ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j (full run). ========== 20161027: 1. The CF0180 run finished the 6th cycle. dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l Y2016/M01/ total 2999168 -rw-r--r-- 1 fzeng g0620 3075334887 2016-10-26 21:26 CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z dali14:/discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C/output/CF0180x6C_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12 total 214528 -rw-r--r-- 1 fzeng g0620 219664000 2016-10-26 21:26 CF0180x6C.ens_avg.ldas_tile_monthly_out.201512.bin Process the output (covert from tile-space to grid-space): ~/Catchment/CF0180x6C > tile2grid_CF0180x6C_monthly CF0180x6C Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/ mv CF0180x6C CF0180x6C_06 cd CF0180x6C_06/output/CF0180x6C_GLOBAL/rs/ens0000/ /bin/rm -rf Y19?? Y200[0-9] Y201[0-5] mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2016/M01/CF0180x6C.ens0000.catchcn_ldas_rst.20160101_0000z CF0180x6C.ens0000.catchcn_ldas_rst.19800101_0000z cd /discover/nobackup/fzeng/Catchment/M2n5P/run Make sure thats the restart path in CF0180x6C_CN.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C_06/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0180x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0180x6C.bat --runmodel --monthsperjob 24 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/CF0180x6C mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp /discover/nobackup/fzeng/Sarith/edit_lenkf.csh . (In the future copy edit_lenkf.csh from the previous cycle) nedit edit_lenkf.csh (to make it work for CF0180 which writes out restart files monthly) ./edit_lenkf.csh Checked the job files and they look correct. qsub lenkf.0.j (full run). Didn't check convergence. Just let it run until Yehui needs the restart files. 2. The e0004s_preindustrialCO2 run stopped. Error message is forrtl: severe (36): attempt to access non-existent record, unit 7593, file /discover/nobackup/fzeng/noaaCO2/data/monthlymean_interp_diurnalcycle_tile/SMAP_TestBed_FV_144x91/2001/co2_0101_00z.dat However, the file exists and is readable: ls -l /discover/nobackup/fzeng/noaaCO2/data/monthlymean_interp_diurnalcycle_tile/SMAP_TestBed_FV_144x91/2001/co2_0101_00z.dat -rw-r--r-- 1 fzeng g0620 1480484 2016-09-16 11:07 /discover/nobackup/fzeng/noaaCO2/data/monthlymean_interp_diurnalcycle_tile/SMAP_TestBed_FV_144x91/2001/co2_0101_00z.dat Tried adding "convert='big_endian'" to the open file command in process_cn.F90 and compiled. "gmake install" in GEOSlana_GridComp, and Applications/LDAS_App. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2 rm -rf Linux cp -pr ../../Linux/ . cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 rm -rf e0004s_preindustrialCO2 Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 6 --landmodel catchCN (NOTE: Don't use the "--submit" option because the job files need to be modified manually before submitting the jobs). Run the modified version of Sarith's edit_lenkf.csh to modify the job scripts (now no need to further modify the first two and last job scripts manually after running edit_lenkf.csh): cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2 mkdir lenkf_orig cp run/lenkf.*.j lenkf_orig/. (save a copy of the original job scripts in case anything wrong happens) cd run cp /discover/nobackup/fzeng/Sarith/edit_lenkf_e0004s.csh edit_lenkf.csh ./edit_lenkf.csh Checked the job files and they look correct. Modified lenkf.0.j and submitted it for a debug run. Again got the above error message. Sarith helped me take a look. His suggestions remind me that the CO2 data file is for 2x2.5 deg tile system and I am using it for SMAP M09!!! Need to convert the grid-space CO2 data to SMAP M09 tile space! 3. Use FWET = 0.01 for the hP, dP, mmP and aP cases. (1) hP: cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp cp ens_driver_routines.F90.restore ens_driver_routines.F90 cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp cp unified_rc3f_matrix_calc.F90.FWETpt01 unified_rc3f_matrix_calc.F90 Checked and made sure that process_cat.F90 is the default one. Compiled using cleanup with "gmake clean". The executable is LDASsa_mpi_M2n5P_m0001_FWETpt01.x Set up a global test run to run for 1 month and save monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_FWETpt01/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_FWETpt01 > cp ../m0001_Prcp_monthly_interp_FWETpt01/lenkf.j . dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_FWETpt01 > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_FWETpt01 > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j The exp log: (a) FWETL = 9.9999998E-03 FWETC = 9.9999998E-03 SATCAPFR = 0.2000000 Correct. (b) No "co2", "SMAP nature 5" or "MERRA-2 surface air temperature". Correct. Processed the output: ~/Catchment/M2n5P_m0001 > grid_restore_M2n5P 0001 _FWETpt01 (2) dP: cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp cp M2n5P_m0001CTco2/ens_driver_routines.F90.dailyPrcp ens_driver_routines.F90 Checked and made sure that fwetl and fwetc are both 0.01 in /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/unified_rc3f_matrix_calc.F90 (i.e. same as the above hP case). Checked and made sure that process_cat.F90 is the default one (i.e. same as the above hP case). Compiled using cleanup withOUT "gmake clean". The executable is LDASsa_mpi_M2n5P_m0001_dailyPrcp_FWETpt01.x Set up a global test run to run for 3 months and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_dailyPrcp_FWETpt01/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp_FWETpt01 > cp ../m0001_FWETpt01/lenkf.j . dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp_FWETpt01 > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_dailyPrcp_FWETpt01 > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j The exp log: (a) FWETL = 9.9999998E-03 FWETC = 9.9999998E-03 SATCAPFR = 0.2000000 Correct. (b) prints out "use daily mean SMAP nature 5 precipitation". No "co2" or "MERRA-2 surface air temperature". Correct. Processed the output: ~/Catchment/M2n5P_m0001 > grid_restore_M2n5P 0001 _dailyPrcp_FWETpt01 Compared the eint/precip ratio using ~/Catchment/M2n5P_m0001/verify_m0001_Prcp_FWET.gs (3) mmP: cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp /bin/cp M2n5P_m0001CTco2/ens_driver_routines.F90.monthlymeanPrcp_interp ens_driver_routines.F90 Checked and made sure that fwetl and fwetc are both 0.01 in /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/unified_rc3f_matrix_calc.F90 (i.e. same as the above hP and dP cases). Checked and made sure that process_cat.F90 is the default one (i.e. same as the above hP and dP cases). Compiled using cleanup withOUT "gmake clean". The executable is LDASsa_mpi_M2n5P_m0001_Prcp_monthlymean_interp_FWETpt01.x Set up a global test run to run for 3 months and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_monthlymean_interp_FWETpt01/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp_FWETpt01 > cp ../m0001_dailyPrcp_FWETpt01/lenkf.j . dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp_FWETpt01 > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_monthlymean_interp_FWETpt01 > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j The exp log: (a) FWETL = 9.9999998E-03 FWETC = 9.9999998E-03 SATCAPFR = 0.2000000 (b) prints out "use daily linearly interpolated from monthly mean SMAP nature 5 precipitation". No "co2" or "MERRA-2 surface air temperature". Correct. Processed the output: ~/Catchment/M2n5P_m0001 > grid_restore_M2n5P 0001 _Prcp_monthlymean_interp_FWETpt01 Compared the eint/precip ratio using ~/Catchment/M2n5P_m0001/verify_m0001_Prcp_FWET.gs (4) aP: cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp /bin/cp M2n5P_m0001CTco2/ens_driver_routines.F90.annualmeanPrcp ens_driver_routines.F90 Checked and made sure that fwetl and fwetc are both 0.01 in /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/unified_rc3f_matrix_calc.F90 (i.e. same as the above cases). Checked and made sure that process_cat.F90 is the default one (i.e. same as the above cases). Compiled using cleanup withOUT "gmake clean". The executable is LDASsa_mpi_M2n5P_m0001_Prcp_annualmean_FWETpt01.x Set up a global test run to run for 3 months and monthly grid output. /discover/nobackup/fzeng/Catchment/M2n5P/m0001 > mkdir -p m0001_Prcp_annualmean_FWETpt01/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean_FWETpt01 > cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.HRv2.tCO2.c6/RUN/rs/ens0000/Y2016/M01/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean_FWETpt01 > cp ../m0001_dailyPrcp_FWETpt01/lenkf.j . dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean_FWETpt01 > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali12:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_Prcp_annualmean_FWETpt01 > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin nedit lenkf.j & sbatch lenkf.j The exp log: (a) FWETL = 9.9999998E-03 FWETC = 9.9999998E-03 SATCAPFR = 0.2000000 (b) prints out "use annual mean SMAP nature 5 precipitation". No "co2" or "MERRA-2 surface air temperature". Correct. Processed the output: ~/Catchment/M2n5P_m0001 > grid_restore_M2n5P 0001 _Prcp_annualmean_FWETpt01 The mP case has already been passed to Eunjee on 20161021. ========== 20161028: 1. Continued working on 3 of 20161027 above. Post-processing and compare eint/precip. 2. Asked Sarith and Qing why the fwetl and fwetc values changed largely from 1.0 and 0.2 respectively to both 0.02. It was because the 1.0 and 0.2 values led to too high interception evaporation in MERRA and 0.02 gives a much better agreement to the observation. See this paper: Rolf H. Reichle, Randal D. Koster, Gabrielle J.M. De Lannoy, Bart A. Forman, Qing Liu, Sarith Mahanama, Ally Toure (2011): Assessment and enhancement of MERRA land surface hydrology estimates, Journal of Climate (MERRA Special Issue), 24, 6322-6338, doi:10.1175/JCLI-D-10-05033.1 3. Met with Randy and Eunjee to talk about what we should do for the mP case. Randy suggests me to learn from the sifl technique on how to make the rain fall at night only. 4. Worked on script to calculate CO2 monthly mean diurnal cycle at CarbonTracker native grid-space: /discover/nobackup/fzeng/noaaCO2/code/monthlymean_diurnalcycle_grid_CO2.f90 ========== 20161031: 1. Eunjee's last cycle of the transient CO2 run crashed due to the job script. The problem is in the job script I provided her. It works for all previous cycles except the last cycle when CO2 years are the same as meteorology years. For the last cycle, we need to modify the job script a little bit. This is the job script for the last cycle: /discover/nobackup/fzeng/Catchment/M2n5P/m0001/m0001_transientCO2_neUS/lenkf.lastcycle_eunjee.j 2. From Sarith: "As for the history of the unified model; This is what I think that happened. We had the catchment model (and its predecessor, the Mosaic model), and other NASA groups had their own models (SiB, SiB-Hyd, PLACE or something). Then, there was a push from NASA to take advantage of all superior parameterizations schemes from different models and develop a better land surface model. So we (Randy) took the stomatal conductance parameterization from Jim's model and modified to make it efficient (may be some other parts too) and called it the unified model. Maybe, that push died without making much progress. So the unified model was never really used in any application - I did some experiments to test (both LDASsa and AGCM), and never really became a part of GEOS5 modeling system Then, later Greg used that to when he started working on CLM-carbon paraneterization - Talk to Randy for a more accurate explanation" "Using the unified model (not the catchment model) to couple the CLM carbon model (for catchmentCN) was the plan, I am sure. When we made FWETL, FWETC changes to the catchment model, I am sure Greg was buried in a much more strenuous task - developing the catchmentCN model. In fact, maybe nobody bothered to communicate that change to Greg. That was the model development stage - now that we have consolidated everything, this kind of thing should not happen again" 3. Verified the CO2 monthly mean diurnal cycle generated by /discover/nobackup/fzeng/noaaCO2/code/monthlymean_diurnalcycle_grid_CO2.f90. See /discover/nobackup/fzeng/noaaCO2/code/verify_monthlymean_diurnalcycle_CO2.gs. 4. The EASE grid is different from lat-lon grid, so can't use the method that I used for the 2x2.5 degree tile to convert the CO2 data from grid space to tile space here. Also, the interpolated diurnal cycle in EASE 9km tile will take huge disk space. Sarith suggests to modify process_cn.F90 to do the grid to tile conversion and interpolation. First, calculate the lat and lon indices of the CarbonTraker 2x3 degree grid in which the center of a particular tile falls: /discover/nobackup/fzeng/noaaCO2/code/get_indices_for_tiles.f90 tile coordinate file land tiles: 1684725 3856 1624 range of tile id: 2 291284 <-- Why 291284 instead of 1684725? Ask Sarith! range of lon index: 1 120 range of lat index: 16 87 Second, modify process_cn.F90.