==========
20190903:

1. The GEOSldas4.5_DE720 run crashed.

The log file:

1980-01-24T05:00:00
end Time -------------------------------

 get_forcing(): assuming GEOS-5 forcing data set
 opening file: ../input/met_forcing/MERRA2_land_forcing/precip_corr_CPCUGPCP22cl
 im_MERRA2_BMTXS//MERRA2_100/diag/Y1980/M01/MERRA2_100.tavg1_2d_lfo_Nx_corr.1980
 0124_0530z.nc4
repair_forcing: SWnet > SWdown in tile ID             7659: met_force(i)%SWnet =   0.13918E+03, met_force(i)%SWdown =   0.12873E+03
repair_forcing: SWnet > SWdown in tile ID             8198: met_force(i)%SWnet =   0.70078E+02, met_force(i)%SWdown =   0.67133E+02
repair_forcing: SWnet > SWdown in tile ID             8547: met_force(i)%SWnet =   0.79243E+02, met_force(i)%SWdown =   0.78016E+02
 LDAS ERROR (3000) from repair_forcing: Tair too low
GEOSldas Run Status: -1

The err file:

forrtl: severe (66): output statement overflows record, unit -5, file Internal List-Directed Write
Image              PC                Routine            Line        Source             
GEOSldas.x         000000000188F64E  Unknown               Unknown  Unknown
GEOSldas.x         00000000018E8BAA  Unknown               Unknown  Unknown
GEOSldas.x         00000000018E653D  Unknown               Unknown  Unknown
GEOSldas.x         00000000011BB950  ldas_forcemod_mp_        5141  LDAS_Forcing.F90

Tried these:

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Components/GEOSldas_GridComp/GEOSmetforce_GridComp
> mv LDAS_Forcing.F90 LDAS_Forcing.F90.git
> cp /discover/nobackup/wjiang/develop_ldas/GEOSldas_UNSTABLE/src/GEOSldas_GridComp/GEOSmetforce_GridComp/LDAS_Forcing.F90 .

Compiled:
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas
> ./parallel_build.csh 

Checked the executable:
> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> ls -l ../build/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 66173350 2019-09-03 10:51 ../build/bin/GEOSldas.x*

Checked the restart file:
> cat cap_restart 
19800101 000000d
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 146 2019-08-30 16:14 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/Y1980/M01/GEOSldas4.5_DE720.catchcn_internal_rst.19800101_0000
lrwxrwxrwx 1 fzeng g0620 121 2019-08-30 16:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/GEOSldas4.5_DE720.vegdyn_internal_rst

Fixed ldas_batchrun.j and submitted the jobs:
> nedit ldas_batchrun.j &
Changed "afterany" to "afterok"
> ./ldas_batchrun.j 

2. Processed GMAO September forecast.

3. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

==========
20190904:

1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

2. For the drought-impact offline experiments:

Need to re-do the offline simulations using Eunjee's CF90 AGCM couple run output.

(1) Pre-processed the CF90 AGCM couple run output:

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology
> mkdir C48
> mv *.dat C48
> mv *.gdat C48

(1.1) T2M:

> cd ~/Catchment/M2n5P_m0002 
> nedit grid2tile_monthly.f90 &
> ifort -o grid2tile_monthly -mcmodel=medium grid2tile_monthly.f90
> grid2tile_monthly T2M_ctrl_48member_1deg_avg
> grid2tile_monthly T2M_dryS_48member_1deg_avg

> ifort -o tile2grid_DC0576xPC0361_monthly_clim -mcmodel=medium tile2grid_DC0576xPC0361_monthly_clim.f90
> tile2grid_DC0576xPC0361_monthly_clim T2M_ctrl_48member_1deg_avg
> tile2grid_DC0576xPC0361_monthly_clim T2M_dryS_48member_1deg_avg

Created control files and modified verify_monthly_dT2M.gs.

> grads -lc "run verify_monthly_dT2M.gs"
> cd /discover/nobackup/fzeng/grads 
> gx2pdf verify_monthly_dT2M
> scp -p verify_monthly_dT2M.pdf gs6101-fzengm1.ndc.nasa.gov:/Volumes/G_DRIVE/03_Research_GMAO/Projects/02_Drought_impact/.

(1.2) PRCP:

> cd ~/Catchment/M2n5P_m0002 
> grid2tile_monthly PRCP_ctrl_48member_1deg_avg
> grid2tile_monthly PRCP_dryS_48member_1deg_avg

> tile2grid_DC0576xPC0361_monthly_clim PRCP_ctrl_48member_1deg_avg
> tile2grid_DC0576xPC0361_monthly_clim PRCP_dryS_48member_1deg_avg

Created control files and modified verify_monthly_dPRCP.gs.

> grads -lc "run verify_monthly_dPRCP.gs"
> cd /discover/nobackup/fzeng/grads 
> gx2pdf verify_monthly_dPRCP
> scp -p verify_monthly_dPRCP.pdf gs6101-fzengm1.ndc.nasa.gov:/Volumes/G_DRIVE/03_Research_GMAO/Projects/02_Drought_impact/.

(1.3) deltaCO2SC:

> cd ~/Catchment/M2n5P_m0002 
> grid2tile_monthly deltaCO2SC_48member_1deg_avg_dryS_ctrl
> tile2grid_DC0576xPC0361_monthly_clim deltaCO2SC_48member_1deg_avg_dryS_ctrl

Created control files and modified verify_monthly_dCO2SC.gs.

> grads -lc "run verify_monthly_dCO2SC.gs"
> cd /discover/nobackup/fzeng/grads
> gx2pdf verify_monthly_dCO2SC 
> scp -p verify_monthly_dCO2SC.pdf gs6101-fzengm1.ndc.nasa.gov:/Volumes/G_DRIVE/03_Research_GMAO/Projects/02_Drought_impact/.

==========
20190905:

1. For the drought-impact offline experiments:

Continued the work to re-do the offline simulations using Eunjee's CF90 AGCM couple run output (previously I used her CF48 run output).

My offline runs were from 2001-2014, because the CT CO2 data from Eunjee is available for this period.

Eunjee just said she also have CT CO2 data for 2000.

(1) Pre-process 2000 CT CO2 data:

> cd /discover/nobackup/fzeng/noaaCO2/data
> cp /archive/u/fzeng/data/noaaCO2/data/daily_grid.tgz .
> tar -xvf daily_grid.tgz
> rm daily_grid.tgz

> cd ~/Catchment/noaaCO2
> nedit nc2bin_CT_CO2_3hrlyNdaily.f90 &
> ifort `nc-config --fflags` nc2bin_CT_CO2_3hrlyNdaily.f90 `nc-config --flibs`
> a.out

(2) Created restart file for 20000101: 

Regridded my 20000101 e0004s_transientCO2 restart file to /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0576xPC0361_DE0360xPE0180/DC0576xPC0361_DE0360xPE0180-Pfafstetter.til following notes on 20170906.

  (2.1) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  > cd /discover/nobackup/fzeng/offline_restarts/DC0576xPC0361_DE0360xPE0180
  > mv catchcn_internal_clm45 catchcn_internal_clm45.20010101
  > mv catchcn_internal_clm45 catchcn_internal_clm45.20010101
  > mv vegdyn_internal_rst vegdyn_internal_rst.20010101
  #> cp /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NL_Reynolds_CF90/mk_LDASsaRestarts.F90 . [No need to do this here]

  > nedit mk_LDASsaRestarts.F90 &
  InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2000/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20000101_0000z', &

  > cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  > source $ESMADIR/src/g5_modules
  > gmake install

  (2.2) Run mk_LDASsaRestarts to convert to desired tile system:
  
  > cd /discover/nobackup/fzeng/offline_restarts/DC0576xPC0361_DE0360xPE0180
  > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  > rm -rf OutData1 OutData2
  > mpirun -np 1 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0576xPC0361_DE0360xPE0180/ -m catchcn -s 50 -t DC0576xPC0361_DE0360xPE0180-Pfafstetter.til 
  > nedit mkLDASsa.j &
  Added "#SBATCH --qos=debug"      
  > sbatch mkLDASsa.j  

This doesn't work because the file /discover/nobackup/rreichle/l_data/LandRestarts_for_Regridding/CatchCN/catchcn_internal_dummy has been changed (2018-10-12) since last time (20170906) I used this mk_LDASsaRestarts.F90. 

Need to use the new mk_LDASsaRestarts.F90.

Reorder and regrid restart files: 

Example, reorder and regrid /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2012/M03/e0004s_transientCO2_2012.ens0000.catchcn_ldas_rst.20120331_2100z to CF90 for Eunjee. See notes on 20180604. 

(A) Reorder:

  (A.1) Create path: 

  > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 
  > mkdir -p rst_DroughtImpact/SMAP_EASEv2_M09_reordered

  (A.2) Create shell script "reorder_restarts" for reordering:

  > cd rst_DroughtImpact/SMAP_EASEv2_M09_reordered
  > nedit reorder_restarts &
  
  reorder_restarts reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
  #!/bin/csh

  set YYYY = 2000
  set YYYYMMDD=`echo ${YYYY}0101`          
          
  mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/ -d ${YYYYMMDD} -e e0004s_transientCO2 -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/ -m catchcn -s 50 -r Y -t SMAP_EASEv2_M09_3856x1624.til
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  > chmod 755 reorder_restarts

  (A.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:
  
  > cp /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 .
  
  > nedit mk_LDASsaRestarts.F90 &
    In SUBROUTINE  reorder_LDASsa_restarts:
    (a) L298:
       tile_coord = trim(EXPDIR)//'rc_out/'//trim(expname)//'.ldas_tilecoord.bin'
    (b) L540:
       rst_file = trim(EXPDIR)//'rs/ens0000/Y'//YYYYMMDD(1:4)//'/M'//YYYYMMDD(5:6)//'/'//trim(ExpName)//&
            '.ens0000.catchcn_ldas_rst.'//trim(YYYYMMDD)//'_0000z' 

  > cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  > source $ESMADIR/src/g5_modules
  > gmake install
  
  (A.4) Run "reorder_restarts" to reorder the offline restart file:
 
  > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_DroughtImpact/SMAP_EASEv2_M09_reordered
  > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  > reorder_restarts
  
(B) Regrid from SMAP_EASEv2_M09 to DC0576xPC0361_DE0360xPE0180: (this part is mostly done on 20190906)

  (B.1) Create path:

  > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_DroughtImpact
  > mkdir -p DC0576xPC0361_DE0360xPE0180
  
  (B.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  > cd DC0576xPC0361_DE0360xPE0180  
  > cp ../SMAP_EASEv2_M09_reordered/mk_LDASsaRestarts.F90 .
  > nedit mk_LDASsaRestarts.F90 &
       L67: InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_DroughtImpact/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.20000101', &
       L68: InCNTilFile = '/discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til',         &
       [Do "xdiff mk_LDASsaRestarts.F90 ../SMAP_EASEv2_M09_reordered/. &" for more modifications made.]
  > cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  > cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  > setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  > source $ESMADIR/src/g5_modules
  > gmake install
  
  (B.3) Run mk_LDASsaRestarts to convert to desired tile system:
  
  > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_DroughtImpact/DC0576xPC0361_DE0360xPE0180
  > ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  > nedit mkLDASsa.j & [Create a job file]

  mkLDASsa.j reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
#!/bin/csh -f
 
#SBATCH --account=g0620
#SBATCH --time=1:00:00
#SBATCH --qos=debug
#SBATCH --ntasks=56
#SBATCH --job-name=mkLDAS
#SBATCH --constraint=hasw
#SBATCH --output=mkLDAS.o
#SBATCH --error=mkLDAS.e
 
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05
source $ESMADIR/src/g5_modules
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
limit stacksize unlimited

setenv SPONSORID g0620
setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0576xPC0361_DE0360xPE0180/
setenv TILFILE DC0576xPC0361_DE0360xPE0180-Pfafstetter.til

mkdir -p OutData1/
mkdir -p OutData2/
ln -s $BCSDIR/$TILFILE OutData1/OutTileFile
ln -s $BCSDIR/$TILFILE OutData2/OutTileFile
ln -s $BCSDIR/clsm OutData2/clsm

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

/bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 
  
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
           
  > chmod 755 mkLDASsa.j
  > sbatch mkLDASsa.j
  
  > mv catchcn_internal_rst catchcn_internal_rst.20000101

2. The GEOSldas4.5_DE720 run stopped due to time limit. 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run 
> cat cap_restart
19950101 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 146 2019-09-05 04:59 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/Y1995/M01/GEOSldas4.5_DE720.catchcn_internal_rst.19950101_0000
lrwxrwxrwx 1 fzeng g0620 121 2019-08-30 16:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/GEOSldas4.5_DE720.vegdyn_internal_rst
> nedit lenkf.j &
#SBATCH --ntasks=112  [was 84]
> ./ldas_batchrun.j 

==========
20190906:

1. The GEOSldas4.5_DE720 run crashed at the very beginning, probably due to the change of number of cores.  

Weiyuan modified my lenkf.j and ran it. 
 
/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/scratch > cp -p LDAS.rc IMS.rc ../run/.

There is still some problem, propably because after changing numprocess from 84 to 112, now 2 cores has 0 tiles.

Weiyuan will think about how to avoid 0 tiles on any cores. 

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/src/Applications/LDAS_App 
> cp /discover/nobackup/wjiang/develop_ldas/GEOSldas/src/Applications/LDAS_App/preprocess_ldas.F90 .
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas 
> ./parallel_build.csh 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run 
> nedit lenkf.j &
Added "exit" in L115.
Changed ntasks from 112 to 84.
> ./lenkf.j
This is to test if the updated preprocess_ldas.F90 works. It worked. 
> nedit lenkf.j &
Changed ntasks from 84 to 112.
> ./lenkf.j
> cp ../scratch/IMS.rc .
> cp ../scratch/LDAS.rc .
> nedit lenkf.j &
Removed "exit" in L115.

> cat cap_restart 
19950101 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 146 2019-09-05 04:59 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/Y1995/M01/GEOSldas4.5_DE720.catchcn_internal_rst.19950101_0000
lrwxrwxrwx 1 fzeng g0620 121 2019-08-30 16:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/GEOSldas4.5_DE720.vegdyn_internal_rst

> ls ../output/global/cat/ens0000/Y1995/M01
There is nothing there. Good.

> ls -l ../output/global/rs/ens0000/Y1995/M01
-rw-r--r-- 1 fzeng g0620 307429314 2019-09-05 04:52 GEOSldas4.5_DE720.catchcn_internal_rst.19950101_0000
> ls -l ../output/global/rs/ens0000/Y1994/M01
-rw-r--r-- 1 fzeng g0620 307429314 2019-09-05 01:57 GEOSldas4.5_DE720.catchcn_internal_rst.19940101_0000
The size of the 19950101 rst file looks correct.

> ./ldas_batchrun.j 

2. For the drought-impact offline experiments:

See notes on 20190905.

3. Met with Randy and Eunjee. We decided to use CT2015 2000 CO2 data and add year 2000 to the simulations. The other simulations need to be re-done due to switching to the 1deg (CF90) AGCM results.

==========
20190909:

1. Processed and checked the clm4.5_fire_DE720 output.

~/Catchment/CLM4.5 > grid_restore_hdeg clm4.5_fire_DE720
~/python/clm4-to-clm4.5 > python plot_burn_bias.py
~/Catchment/CLM4.5 > grads -lc "run check_fire_bug_fix.gs"

2. The GEOSldas4.5_DE720 run finished 19800101-20181213.

3. For the drought-impact offline experiments:

(1) CTco2_3hrly used MERRA2 forcing and wrote out restart files on April 1st for each year from 2001 through 2014. 

Now need to add year 2000.

catchcn_internal_rst.20000101 was generated from EASE M09 restart file on 20190905 and 20190906.

2000 CT2015 CO2 data was preprocessed on 20190905.

No change to the executable.

Don't do a full year run. Only run until July 1, 2000.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/input/restart
> ls -l
lrwxrwxrwx 1 fzeng g0620 90 2017-09-07 16:43 catchcn_internal_rst -> /discover/nobackup/fzeng/offline_restarts/DC0576xPC0361_DE0360xPE0180/catchcn_internal_rst
This restart file was for 20010101.
> rm catchcn_internal_rst
> ln -s /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_DroughtImpact/DC0576xPC0361_DE0360xPE0180/catchcn_internal_rst.20000101 catchcn_internal_rst

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/run
> cp lenkf.0.j lenkf.-1.j
> nedit lenkf.-1.j &

#SBATCH --output=../output/GLOBAL/rc_out/Y2000/M01/CTco2_3hrly.ldas_log.20000101_0000.txt
#SBATCH --error=../output/GLOBAL/rc_out/Y2000/M01/CTco2_3hrly.ldas_out.20000101_0000.txt

 -start_year 2000 -start_month  1 -start_day  1 -start_hour  0 -start_min  0 -start_sec  0 \
 -end_year   2000 -end_month    7 -end_day    1 -end_hour    0 -end_min    0 -end_sec    0 \

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/GLOBAL/rc_out
> mkdir -p Y2000/M01

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/GLOBAL/cat/ens_avg
> mkdir Y2000
> cd Y2000
> mkdir M01 M02 M03 M04 M05 M06 M07

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/GLOBAL/rs/ens0000
> mkdir Y2000
> cd Y2000
> mkdir M01 M02 M03 M04 M05 M06 M07 M08 M09 M10 M11 M12

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/run/

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/run/
> ./lenkf.-1.j

It's running. Stopped it at date_time_new 20000101_231500z.

> qsub lenkf.-1.j

Processed and checked the available output of Catchment/M2n5P/m0002/CTco2_3hrly. 
On 20170907 I used tile2grid_DC0576xPC0361_monthly.f90 to process the output of CTco2_3hrly. However, since then I have modified this file a few times. Now modify it to work for CTco2_3hrly. 

> cd ~/Catchment/M2n5P_m0002 
> cp -p tile2grid_DC0576xPC0361_monthly.f90 tile2grid_DC0576xPC0361_monthly.f90.20170915
> mv tile2grid_DC0576xPC0361_monthly.f90 tile2grid_DC0576xPC0361_monthly_1.f90
> nedit tile2grid_DC0576xPC0361_monthly_1.f90 &
> ifort -o tile2grid_DC0576xPC0361_monthly_1 tile2grid_DC0576xPC0361_monthly_1.f90
> tile2grid_DC0576xPC0361_monthly_1 CTco2_3hrly

> nedit CTco2_3hrly.ctl &
Checked LAI and GPP of Jun 2000. Look correct.

Checked the log file. The run used 2000 CT CO2. Good.

Forgot that I need to run at least to the end of September because I need to compute MERRA2 climatology of T2M and precipiation for April through September. 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/run
> cp lenkf.-1.j lenkf.-2.j
> nedit lenkf.-2.j &

#SBATCH --output=../output/GLOBAL/rc_out/Y2000/M07/CTco2_3hrly.ldas_log.20000701_0000.txt
#SBATCH --error=../output/GLOBAL/rc_out/Y2000/M07/CTco2_3hrly.ldas_out.20000701_0000.txt

 -start_year 2000 -start_month  7 -start_day  1 -start_hour  0 -start_min  0 -start_sec  0 \
 -end_year   2000 -end_month   12 -end_day    1 -end_hour    0 -end_min    0 -end_sec    0 \
 
 -restart_path ../output \
 
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/GLOBAL/rc_out
> mkdir -p Y2000/M07

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/GLOBAL/cat/ens_avg/Y2000
> mkdir M08 M09 M10 M11 M12 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/run/

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/run/
> ./lenkf.-2.j

It's running. Stopped it at date_time_new 20000701_223000z.

> qsub lenkf.-2.j

Processed the Jan through Nov 2000 output of Catchment/M2n5P/m0002/CTco2_3hrly. 

> cd ~/Catchment/M2n5P_m0002 
> nedit tile2grid_DC0576xPC0361_monthly_1.f90 &
> ifort -o tile2grid_DC0576xPC0361_monthly_1 tile2grid_DC0576xPC0361_monthly_1.f90
> tile2grid_DC0576xPC0361_monthly_1 CTco2_3hrly

(2) Pre-processed the CF90 AGCM couple run output: earlier we used 16 years (2000-2015) 48 ensembles, now we use 15 years (2000-2014) 45 ensembles.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology
> mkdir C90_48member
> mv *.dat C90_48member

(2.1) T2M:

> cd ~/Catchment/M2n5P_m0002 
> nedit grid2tile_monthly.f90 & [No need to change.]
> grid2tile_monthly T2M_ctrl_45member_1deg_avg
> grid2tile_monthly T2M_dryS_45member_1deg_avg

Checked the data:

> tile2grid_DC0576xPC0361_monthly_clim T2M_ctrl_45member_1deg_avg
> tile2grid_DC0576xPC0361_monthly_clim T2M_dryS_45member_1deg_avg

Created control files and modified verify_monthly_dT2M.gs.

> grads -lc "run verify_monthly_dT2M.gs"
> cd /discover/nobackup/fzeng/grads 
> gx2pdf verify_monthly_dT2M
> scp -p verify_monthly_dT2M.pdf gs6101-fzengm1.ndc.nasa.gov:/Volumes/G_DRIVE/03_Research_GMAO/Projects/02_Drought_impact/C90_45member/.

(2.2) PRCP:

> cd ~/Catchment/M2n5P_m0002 
> grid2tile_monthly PRCP_ctrl_45member_1deg_avg
> grid2tile_monthly PRCP_dryS_45member_1deg_avg

Checked the data: 

> tile2grid_DC0576xPC0361_monthly_clim PRCP_ctrl_45member_1deg_avg
> tile2grid_DC0576xPC0361_monthly_clim PRCP_dryS_45member_1deg_avg

Created control files and modified verify_monthly_dPRCP.gs.

> grads -lc "run verify_monthly_dPRCP.gs"
> cd /discover/nobackup/fzeng/grads 
> gx2pdf verify_monthly_dPRCP
> scp -p verify_monthly_dPRCP.pdf gs6101-fzengm1.ndc.nasa.gov:/Volumes/G_DRIVE/03_Research_GMAO/Projects/02_Drought_impact/C90_45member/.

(2.3) deltaCO2SC:

> cd ~/Catchment/M2n5P_m0002 
> grid2tile_monthly deltaCO2SC_45member_1deg_avg_dryS_ctrl

Checked the data:

> tile2grid_DC0576xPC0361_monthly_clim deltaCO2SC_45member_1deg_avg_dryS_ctrl

Created control files and modified verify_monthly_dCO2SC.gs.

> grads -lc "run verify_monthly_dCO2SC.gs"
> cd /discover/nobackup/fzeng/grads
> gx2pdf verify_monthly_dCO2SC 
> scp -p verify_monthly_dCO2SC.pdf gs6101-fzengm1.ndc.nasa.gov:/Volumes/G_DRIVE/03_Research_GMAO/Projects/02_Drought_impact/C90_45member/.

4. Tried to automate GEOSldas4.5_DE720 spinup:

> cd /discover/nobackup/fzeng/Catchment/M2n5P
> mv GEOSldas4.5_DE720 GEOSldas4.5_DE720_01

Create the template.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/run
> nedit GEOSldas4.5_DE720.exe GEOSldas4.5_DE720.bat &

Stopped here. Need to ask Weiyuan some questions before I can further modify GEOSldas4.5_DE720.exe. Have NOT done these below yet:
See notes on 20190830.
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/install/bin
> ls -l GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 66169420 2019-08-30 15:33 GEOSldas.x*
> source g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.bat

==========
20190910:

1. For the drought-impact offline experiments:

(1) Recomputed MERRA2 T2M and precipitation climatology: 

> cd ~/Catchment/M2n5P_m0002
> nedit calculate_clim_tile_monthly_Tair.f90 &
> ifort -o calculate_clim_tile_monthly_Tair calculate_clim_tile_monthly_Tair.f90
> calculate_clim_tile_monthly_Tair CTco2_3hrly

> nedit calculate_clim_tile_monthly_precip.f90 &
> ifort -o calculate_clim_tile_monthly_precip calculate_clim_tile_monthly_precip.f90
> calculate_clim_tile_monthly_precip CTco2_3hrly

Checked the calculated MERRA2 climatology.

> cd ~/Catchment/M2n5P_m0002 
No need to modify or compile tile2grid_DC0576xPC0361_monthly_clim.f90.
> tile2grid_DC0576xPC0361_monthly_clim MERRA2_T2M_2000_2014_avg
> tile2grid_DC0576xPC0361_monthly_clim MERRA2_precip_2000_2014_avg

> cp verify_MERRA2_T2M_2001_2014_avg.ctl verify_MERRA2_T2M_2000_2014_avg.ctl
> nedit verify_MERRA2_T2M_2000_2014_avg.ctl &
Check T2M Jul clim. Pattern looks correct.

> cp verify_MERRA2_precip_2001_2014_avg.ctl verify_MERRA2_precip_2000_2014_avg.ctl
> nedit verify_MERRA2_precip_2000_2014_avg.ctl & 
Check precip Jul clim. Magnitude and pattern look correct.

(2) Created the symbolic links using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology/create_symbolic_links so that I don't need to modify and re-compile the model code of each of the experiments.

(3) CTco2_3hrly_ctrl_YYYY: following notes on 20170918.

For YYYY = 2001:

  (3.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 3 places in DC0576xPC0361_CN.exe. 

  (3.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_ctrl/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (3.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/input/restart
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 65 2019-09-10 12:02 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  Looks correct.
  
  (3.4) Check the job file and modify if needed:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/run
  > nedit lenkf.0.j &     
  lenkf.0.j looks correct.  No modification needed. 
  > /bin/rm lenkf.1.j (it's not needed)
  
  (3.5) Check the executable:
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69802173 2017-10-05 16:08 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (3.6) Do an interactive run to make sure it runs and uses the right restart file:  
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/run/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it at date_time_new 20000402_030730z.
  > qsub lenkf.0.j

For YYYY from 2001 through 2014: 

  The log files show that the restart files were written out correctly. However, I don't see anything under rs/ens0000 now. May be I removed the restart files at some point to free up some space in my NOBACKUP. Now need to create the directory for the restart file:
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002
  > nedit mkdir_rst &
  > mkdir_rst
  This creates restart file directories for all the years. 

Take 2001 as an example.

  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2001/run/
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69802173 2017-10-05 16:08 ../build/Linux/bin/LDASsaCN_mpi.x*  
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 65 2017-10-05 16:17 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it at date_time_new 20010402_000730z.
  > qsub lenkf.0.j

(4) CTco2_3hrly_PRCP_S_YYYY: 

For YYYY = 2000: following notes (9) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README.

  (4.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 3 places in DC0576xPC0361_CN.exe.

  (4.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_PRCP_S/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (4.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2000/input/restart
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 65 2019-09-10 14:28 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  Looks correct.
  
  (4.4) Check the job file and modify if needed:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2000/run
  lenkf.0.j looks correct.  No modification needed.
  > /bin/rm lenkf.1.j (it's not needed)
  
  (4.5) Check the executable:
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69802183 2017-10-10 14:59 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (4.6) Do an interactive run to make sure it runs and uses the right restart file:
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2000/run
  > ./lenkf.0.j
  It's running and used the right restart file. 
  Stopped it at date_time_new 20000404_050730z.
  > qsub lenkf.0.j

For YYYY from 2001 through 2014: 

  The log files show that the restart files were written out correctly. However, I don't see anything under rs/ens0000 now. May be I removed the restart files at some point to free up some space in my NOBACKUP. Now need to create the directory for the restart file:
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002
  > nedit mkdir_rst &
  > mkdir_rst
  This creates restart file directories for all the years. 

Take 2001 as an example.
 
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_PRCP_S/CTco2_3hrly_PRCP_S_2001/run
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69802183 2017-10-10 14:59 ../build/Linux/bin/LDASsaCN_mpi.x*
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 65 2017-10-10 15:02 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it at date_time_new 20010402_000730z.
  > qsub lenkf.0.j

(5) CTco2_3hrly_T2M_S_YYYY: 

For YYYY = 2000: following notes (8) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README.

  (5.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 3 places in DC0576xPC0361_CN.exe.

  (5.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_T2M_S/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (5.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2000/input/restart
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 65 2019-09-10 14:28 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  Looks correct.
  
  (5.4) Check the job file and modify if needed:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2000/run
  lenkf.0.j looks correct.  No modification needed.
  > /bin/rm lenkf.1.j (it's not needed)
  
  (5.5) Check the executable:
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69802183 2017-10-10 12:54 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (5.6) Do an interactive run to make sure it runs and uses the right restart file:
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2000/run
  > ./lenkf.0.j
  It's running and used the right restart file. 
  Stopped it at date_time_new 20000402_060730z.
  > qsub lenkf.0.j

For YYYY from 2001 through 2014: 

  The log files show that the restart files were written out correctly. However, I don't see anything under rs/ens0000 now. May be I removed the restart files at some point to free up some space in my NOBACKUP. Now need to create the directory for the restart file:
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002
  > nedit mkdir_rst &
  > mkdir_rst
  This creates restart file directories for all the years. 

Take 2001 as an example.
 
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_T2M_S/CTco2_3hrly_T2M_S_2001/run
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69802183 2017-10-10 12:54 ../build/Linux/bin/LDASsaCN_mpi.x*
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 65 2017-10-10 13:54 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it at date_time_new 20010401_090730z.
  > qsub lenkf.0.j

==========
20190911:

1. For the drought-impact offline experiments:

(1) CTco2_3hrly_dCO2SC_S_YYYY: 

For YYYY = 2000: following notes (7) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README.

  (1.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 3 places in DC0576xPC0361_CN.exe.

  (1.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_dco2sc_S/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (1.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2000/input/restart
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 65 2019-09-11 10:29 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  Looks correct.
  
  (1.4) Check the job file and modify if needed
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2000/run
  lenkf.0.j looks correct.  No modification needed.
  > /bin/rm lenkf.1.j (it's not needed)
  
  (1.5) Check the executable
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69817133 2017-10-06 17:44 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (1.6) Do an interactive run to make sure it runs and uses the right restart file
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2000/run
  > ./lenkf.0.j  
  It's running and used the right restart file. 
  Stopped it.
  > qsub lenkf.0.j

For YYYY from 2001 through 2014: 

  The log files show that the restart files were written out correctly. However, I don't see anything under rs/ens0000 now. May be I removed the restart files at some point to free up some space in my NOBACKUP. Now need to create the directory for the restart file:
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002
  > nedit mkdir_rst &
  > mkdir_rst
  This creates restart file directories for all the years. 

Take 2001 as an example.
 
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_S/CTco2_3hrly_dco2sc_S_2001/run
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69817133 2017-10-06 17:44 ../build/Linux/bin/LDASsaCN_mpi.x*
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 65 2017-10-06 17:53 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it.
  > qsub lenkf.0.j

(2) CTco2_3hrly_dco2sc_dryS_YYYY:

For YYYY = 2000: following notes (10) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README.

  (2.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 3 places in DC0576xPC0361_CN.exe.

  (2.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_dco2sc_dryS/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (2.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2000/input/restart
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 65 2019-09-11 11:17 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  Looks correct.
  
  (2.4) Check the job file and modify if needed
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2000/run
  lenkf.0.j looks correct.  No modification needed.
  > /bin/rm lenkf.1.j (it's not needed)
  
  (2.5) Check the executable
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69817153 2017-10-10 16:25 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (2.6) Do an interactive run to make sure it runs and uses the right restart file
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2000/run
  > ./lenkf.0.j 
  It's running and used the right restart file. 
  Stopped it.
  > qsub lenkf.0.j

For YYYY from 2001 through 2014: 

  The log files show that the restart files were written out correctly. However, I don't see anything under rs/ens0000 now. May be I removed the restart files at some point to free up some space in my NOBACKUP. Now need to create the directory for the restart file:
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002
  > nedit mkdir_rst &
  > mkdir_rst
  This creates restart file directories for all the years. 

Take 2001 as an example.
 
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS/CTco2_3hrly_dco2sc_dryS_2001/run
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69817153 2017-10-10 16:25 ../build/Linux/bin/LDASsaCN_mpi.x*
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 65 2017-10-10 16:28 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it.
  > qsub lenkf.0.j

(3) CTco2_3hrly_ctrl_S_YYYY:

For YYYY = 2000: following notes (11) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README.

  (3.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 3 places in DC0576xPC0361_CN.exe.

  (3.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_ctrl_w_daily_rst/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (3.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2000/input/restart/
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 65 2019-09-11 12:04 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  Looks correct.
  
  (x.4) Check the job file and modify if needed
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2000/run 
  lenkf.0.j looks correct. No modification needed. 
  > /bin/rm lenkf.1.j (it's not needed)
  
  > nedit driver_inputs_DC0576xPC0361_GLOBAL.nml &
  minlon               =   -97.       ! min longitude
  maxlon               =   -90.	    ! max longitude
  minlat               =    30.	    ! min latitude
  maxlat               =    37.	    ! max latitude
  Correct.
  
  (3.5) Check the executable
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69802173 2017-10-24 15:49 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (3.6) Do an interactive run to make sure it runs and uses the right restart file
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2000/run
  > ./lenkf.0.j 
  It's running and used the right restart file, and it writes out restart file every day. Stopped it.
  > qsub lenkf.0.j

For YYYY from 2001 through 2014: take 2001 as an example.
 
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2001/run
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69802173 2017-10-24 15:49 ../build/Linux/bin/LDASsaCN_mpi.x*
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 65 2017-10-24 16:02 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it.
  > qsub lenkf.0.j

(4) Found that CTco2_3hrly_ctrl_2000 was not done for global land but done only for region S, because at the beginning I forgot to modify driver_inputs_DC0576xPC0361_GLOBAL.nml to cover the entire global land. I later modified driver_inputs_DC0576xPC0361_GLOBAL.nml to cover the entire global land, but I didn't remove the tile information files in rc_out. 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/output/GLOBAL/rc_out
> rm CTco2_3hrly_ctrl_2000.ldas_domain.txt  CTco2_3hrly_ctrl_2000.ldas_tilecoord.bin  CTco2_3hrly_ctrl_2000.ldas_tilegrids.bin
> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/run
> ./lenkf.0.j
It's running and covers global land. Stopped it at date_time_new 20000401_230730z.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/output/GLOBAL/rc_out/Y2000/M04
> rm *log* *out*

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl/CTco2_3hrly_ctrl_2000/run
> qsub lenkf.0.j

(5) Processed output:

> cd ~/Catchment/M2n5P_m0002 
> nedit grid_restore_DC0576xPC0361_monthly.f90 &
Changed the ntiles to cover global land.
> ifort -o grid_restore_DC0576xPC0361_monthly grid_restore_DC0576xPC0361_monthly.f90

(5.1) CTco2_3hrly_T2M_S_YYYY:

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_T2M_S CTco2_3hrly_T2M_S_2000  [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_T2M_S.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs" 

(5.2) CTco2_3hrly_PRCP_S_YYYY:

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_PRCP_S CTco2_3hrly_PRCP_S_2000  [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_PRCP_S.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs" 

(5.3) CTco2_3hrly_dco2sc_S_YYYY: 

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_S CTco2_3hrly_dco2sc_S_2000  [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_dco2sc_S.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs" 

==========
20190912:

1. For the drought-impact offline experiments:

(1) CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_YYYY:

For YYYY = 2000: following notes (12) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README.

  (1.1) Modified DC0576xPC0361.bat, DC0576xPC0361_CN.exe and driver_inputs_DC0576xPC0361_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run for this run.
  
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2001/run
  > cp DC0576xPC0361.bat DC0576xPC0361_CN.exe driver_inputs_DC0576xPC0361_GLOBAL.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/.
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run
  > nedit DC0576xPC0361.bat DC0576xPC0361_CN.exe &
  Changed the year at 1 place in DC0576xPC0361.bat and at 5 places in DC0576xPC0361_CN.exe.

  (1.2) Run ldsetup:

  > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/M2n5P_m0002CTco2_dco2sc_dryS_ctrl_d_rst/Linux/bin
  > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
  > ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0002/run/DC0576xPC0361.bat --runmodel --monthsperjob 6 --landmodel catchCN
  
  (1.3) Check the restart file:
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2000/input/restart
  > ls -l
  lrwxrwxrwx 1 fzeng g0620 96 2019-09-12 10:27 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2000/output/
  Looks correct.
  
  (1.4) Check the job file and modify if needed
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2000/run
  lenkf.0.j looks correct.  No modification needed.
  > /bin/rm lenkf.1.j (it's not needed)
  
  (x.5) Check the executable
  > ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2000/build/Linux/bin/LDASsaCN_mpi.x
  -rwxr-xr-x 1 fzeng g0620 69835364 2017-11-01 12:58 /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2000/build/Linux/bin/LDASsaCN_mpi.x*
  Correct.
  
  (1.6) Do an interactive run to make sure it runs and uses the right restart file
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2000/run
  > ./lenkf.0.j
  It's running and used the right restart file. 
  Stopped it at date_time_new 20000402_050730z.  
  > qsub lenkf.0.j


For YYYY from 2001 through 2014: take 2001 as an example.
 
  > interactive.py -A sp3 -n 140 -a g0620 -X --debug
  > cd /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst/CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2001/run
  Check the executable:
  > ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
  -rwxr-xr-x 1 fzeng g0620 69835364 2017-11-01 12:58 ../build/Linux/bin/LDASsaCN_mpi.x*
  Check the restart file:
  > ls -l ../input/restart/
  lrwxrwxrwx 1 fzeng g0620 96 2017-11-01 13:04 output -> /discover/nobackup/fzeng/Catchment/M2n5P/m0002/CTco2_3hrly_ctrl_S/CTco2_3hrly_ctrl_S_2001/output/
  > ./lenkf.0.j  
  It's running and used the right restart file. Stopped it.
  > qsub lenkf.0.j

(2) Continued processing output:

> cd ~/Catchment/M2n5P_m0002 

(2.1) CTco2_3hrly_ctrl_YYYY:

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_ctrl CTco2_3hrly_ctrl_2000 [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_ctrl.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs"

(2.2) CTco2_3hrly_dco2sc_dryS_YYYY: 

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_dryS CTco2_3hrly_dco2sc_dryS_2000  [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_dco2sc_dryS.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs"

(2.3) CTco2_3hrly_ctrl_S_YYYY: 

> cd ~/Catchment/M2n5P_m0002 
> nedit grid_restore_DC0576xPC0361_monthly.f90 &
Changed the ntiles to cover only region S.
> ifort -o grid_restore_DC0576xPC0361_monthly grid_restore_DC0576xPC0361_monthly.f90

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_ctrl_S CTco2_3hrly_ctrl_S_2000  [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_ctrl_S.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs" 

(2.4) CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_YYYY: 

Checked and made sure that all the output files in output/GLOBAL/cat/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_cat. 

> grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_dryS_ctrl_d_rst CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2000  [do this for 2000 through 2014]

Checked and made sure that all the files in post/ are updated using /discover/nobackup/fzeng/Catchment/M2n5P/m0002/check_output_in_post.

Updated CTco2_3hrly_dco2sc_dryS_ctrl_d_rst.ctl and check_output.gs, and checked LAI, GPP and NEE:
> ~/Catchment/M2n5P_m0002
> grads -lc "run check_output.gs" 

(3) Checked the output by anaylzing it and comparing to previous results. 

> ~/Catchment/M2n5P_m0002
> nedit compare_all.gs &
> grads -lc "run compare_all.gs"

2. Met with Randy, Sieg, Eunjee and Lesley: Eunjee's work

3. GMAO Theme Meeting

4. Updated PEP goals

==========
20190913:

1. Did some analysis on the clm4.5_fire_DE720 output and prepared meeting slides.

2. Met with Randy and Eunjee: drought paper

3. Create post-processing script to convert tile-space output of GEOSldas4.5_DE720 to grid-space. 

==========
20190916:

1. For the fire project: include the change to cc and fm as permanent changes. 

> nedit /home/fzeng/Catchment/CLM4.5/submit_next_batch_DE720 &
Commented out ./ldas_batchrun.j

To Do: done on 20190917.

Once the current cycle of clm4.5_DE720 is completed:

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_V24_C05/exec
> mv clm4.5_Kn11 clm4.5_Kn11_20190904
> /bin/cp -pr clm4.5_Kn11_Ndiags_fire clm4.5_Kn11

> nedit /home/fzeng/Catchment/CLM4.5/submit_next_batch_DE720 &
Uncomment out ./ldas_batchrun.j

Check the restart file.

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/run
> ls -l ../build/Linux/bin/LDASsaCN_mpi.x 
-rwxr-xr-x 1 fzeng g0620 71263362 2019-09-04 15:54 ../build/Linux/bin/LDASsaCN_mpi.x*
> ./ldas_batchrun.j

2. Copied some CASA-GFED3 output to Discover for comparison with Catchment-CN.

gs6101-fzengm1:output fzeng$ pwd
/Volumes/G_DRIVE/02_Research_Postdoc/CASA/CASA_Model_08_MC/CASA_GFED3/data/MC_v6/output

gs6101-fzengm1:output fzeng$ scp -p soilpool_*_453runs_mean.mat biomass_*_453runs_mean.mat AGB_*_453runs_sd.mat biomass_*_453runs_sd.mat soilpool_*_453runs_sd.mat fzeng@discover.nccs.nasa.gov:/discover/nobackup/fzeng/CASA_GFED3/output/MC_v6/.

gs6101-fzengm1:output fzeng$ scp -p FIRE_*_453runs_mean.mat FIRE_*_453runs_sd.mat GPP_*_453runs_mean.mat GPP_*_453runs_sd.mat fzeng@discover.nccs.nasa.gov:/discover/nobackup/fzeng/CASA_GFED3/output/MC_v6/.

3. Met with Lesley and Eunjee.

4. Talked to Lei Ma about MERRA2 precipitation data correction. 

5. Removed some files from ARCHIVE to release some space. 

==========
20190917:

1. GMAO seminar about GEOS-Chem.

2. Created ~/python/clm4-to-clm4.5 > python compare_CatCN_GFED3_biomass.py and used it to compare fields between CatCN4.5 and GFED3.

3. Found that CatCN4.5 vegc and leafc are very low in SH Africa where we have negative biases in fire C emission. Also found in this region that CatCN4.5 GPP is very low, much lower than CatCN4.0 and MTE GPP. Investigated why. 

4. Met with Eunjee.

==========
20190918:

1. Continued investigating why CatCN GPP is so low in SH Africa.

2. Met with Joe about PEP. 

==========
20190919:

1. Continued investigating why CatCN GPP is so low in SH Africa. Found that low LAI and GPP in SH Africa is likely due to too many fires! Discussed with Eunjee. Will see if we can tune fire mortality to increase GPP while keep fire carbon emission about the same. 

2. Processed and archived spinup output.

clm4.5_DE720: 58 through 63

==========
20190923: 

1. Continued investigating why CatCN GPP is so low in SH Africa: made more plots.

2. Continued working on ~/python/clm4-to-clm4.5/tile2grid_GEOSldas.py so that it works for all 39 years from 1980 through 2018.

3. Automated GEOSldas spinup: refer to notes on 20190830 for how to set up an experiment

> cd /home/fzeng/Catchment/CLM4.5
> nedit submit_next_batch_GEOSldas4.5_DE720 &

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/install/bin
> ls -l GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 66173350 2019-09-03 10:51 GEOSldas.x*

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas 
> mkdir -p exec/clm4.5_Kn11
> /bin/cp -pr install exec/clm4.5_Kn11/.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/run
> nedit GEOSldas4.5_DE720.bat GEOSldas4.5_DE720.exe &
GEOSldas4.5_DE720.bat:
ntasks                  : 168
GEOSldas4.5_DE720.exe:
BEG_DATE                : 19800101 000000
END_DATE                : 19810101 000000       <------ for test only!!

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4.5/GEOSldas/exec/clm4.5_Kn11/install/bin
> source g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.exe /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas4.5_DE720.bat

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720

Checked executable:
> ls -l build/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 66173350 2019-09-03 10:51 build/bin/GEOSldas.x*

For restart:
> set exp = GEOSldas4.5_DE720
> set num = 01
> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rs/ens0000/Y1980/M01
> /bin/rm ${exp}.catchcn_internal_rst.19800101_0000
> ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/Y2019/M01/${exp}.catchcn_internal_rst.20190101_0000 ${exp}.catchcn_internal_rst.19800101_0000
> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rs/ens0000
> /bin/rm ${exp}.vegdyn_internal_rst
> ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/${exp}.vegdyn_internal_rst

> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/input/restart/
> diff catchcn_internal_rst /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/Y2019/M01/${exp}.catchcn_internal_rst.20190101_0000
> diff vegdyn_internal_rst /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/output/global/rs/ens0000/${exp}.vegdyn_internal_rst
Both are identical. Great!

> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/run
> /bin/cp /discover/nobackup/fzeng/Catchment/M2n5P/${exp}_${num}/run/HISTORY.rc .
> nedit LDAS.rc &  [DO NOT copy from ${exp}_${num} due to different number of cores!!]
Added to the end (L53):
LNFM_FILE:                          ../input/lnfm.data

> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rc_out
> cp /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/lnfm_clim_720x360_DE.data .
> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/input
> ln -s /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/output/global/rc_out/lnfm_clim_720x360_DE.data lnfm.data

> cd /discover/nobackup/fzeng/Catchment/M2n5P/${exp}/run
> nedit ldas_batchrun.j &
Added to the end:
/home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas4.5_DE720 GEOSldas4.5_DE720

> cd /discover/nobackup/fzeng/Catchment/M2n5P
> mv GEOSldas4.5_DE720 GEOSldas4.5_DE720_template

> /bin/cp -r GEOSldas4.5_DE720_template GEOSldas4.5_DE720 

Test:
> interactive.py -A sp3 -n 168 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &
Add "exit" after L284 "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x"
> ./lenkf.j

It's running. Stopped it at 1980-01-02T04:00:00.

Submit the full jobs:
> mv lenkf.j.orig lenkf.j
> ./ldas_batchrun.j

However, the command line "/home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas4.5_DE720 GEOSldas4.5_DE720" is executed right after the lenkf.j is submitted. Of course it failed. 

==========
20190924:

1. Processed output:

clm4.5_DE720: 64, 65, 66

2. Randy needs one more daily variable gwetprof to extract from MERRA2. 

> cd ~/merra2 
> nedit extract_daily.gs & 
> grads -lbc "run extract_daily.gs"

Updated merra2.ctl under the same directory.

3. Prepared slides for Friday's meeting.

4. Continued automating GEOSldas spinup:

Tried to see how to can solve the issue so that "/home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas4.5_DE720 GEOSldas4.5_DE720" is executed only after the entire cycle is completed. Couldn't figure out. 

Asked Weiyuan for help.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> nedit ldas_batchrun.j & [followed Weiyuan's instructions]

~~~~~~~~~~~~~~~
#!/bin/bash -f

sed -i 's/if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/#if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/g' lenkf.j

jobid0=$(echo $(sbatch lenkf.j) | cut -d' ' -f 4)
echo $jobid0
jobid1=$(echo $(sbatch --dependency=afterok:$jobid0 /home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas_DE720 GEOSldas4.5_DE720) | cut -d' ' -f 4)
echo $jobid1

sed -i 's/#if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/g' lenkf.j
~~~~~~~~~~~~~~~

> ./ldas_batchrun.j 

5. Fire mortality in CatCN4.5. Read papers and code about fire in GFED3.

==========
20190925:

1. Prepared Statement of Work, 1pg CV and a list of current and pending proposals for Eunjee's proposal, and sent them to Robin.

2. Continued automating GEOSldas spinup:

The GEOSldas4.5_DE720 run finished the 1st cycle (currently set to only one year 1980), but the new cycle is not created or submitted. Investigated why.

The "post" subdirectly is not there, so probably the post-processing failed. 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> /home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas_DE720 GEOSldas4.5_DE720
set exp = GEOSldas4.5_DE720
SLURM_JOB_ID: Undefined variable.

It couldn't find SLURM_JOB_ID.

/discover/nobackup/fzeng/Catchment/M2n5P > python $HOME/Catchment/CLM4.5/tile2grid_GEOSldas.py
This is working. Good!

/discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas4.5_DE720 GEOSldas4.5_DE720_1980

Tried this:

> cd /discover/nobackup/fzeng/Catchment/M2n5P
> /bin/cp -r GEOSldas4.5_DE720_template GEOSldas4.5_DE720 

> cd GEOSldas4.5_DE720/run
> nedit ldas_batchrun.j &

~~~~~~~~~~~~~~~
#!/bin/bash -f

sed -i 's/if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/#if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/g' lenkf.j

jobid0=$(echo $(sbatch lenkf.j) | cut -d' ' -f 4)
echo $jobid0

sed -i 's/#if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/g' lenkf.j

jobid1=$(echo $(sbatch --dependency=afterok:$jobid0 /home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas_DE720 GEOSldas4.5_DE720) | cut -d' ' -f 4)
echo $jobid1
~~~~~~~~~~~~~~~

Test:
> interactive.py -A sp3 -n 168 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &
Add "exit" after L284 "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x"
> ./lenkf.j

It's running. Stopped it at 1980-01-02T04:00:00.

Submit the full jobs:
> mv lenkf.j.orig lenkf.j
> ./ldas_batchrun.j

Talked to Weiyuan. Weiyuan said the submit_next_batch_GEOSldas_DE720 might not work in batch job because there is no environment setup (the lines start with "#SBATCH"). 

Cancelled the jobs submitted, which had not started to run yet.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> cp lenkf.j lenkf_submit_next_batch.j
> nedit lenkf_submit_next_batch.j &
 
~~~~~~~~~~~~~~
#!/bin/csh -f

#SBATCH --output=../scratch/submit_next_batch_log_txt
#SBATCH --error=../scratch/submit_next_batch_err_txt
#SBATCH --account=g0620
#SBATCH --time=01:00:00
#SBATCH --qos=debug
#SBATCH --ntasks=1
#SBATCH --job-name=submit_next_batch
#SBATCH --constraint=hasw

/home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas_DE720 GEOSldas4.5_DE720
~~~~~~~~~~~~~~

> nedit ldas_batchrun.j &

~~~~~~~~~~~~~~
#!/bin/bash -f

sed -i 's/if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/#if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/g' lenkf.j

jobid0=$(echo $(sbatch lenkf.j) | cut -d' ' -f 4)
echo $jobid0
jobid1=$(echo $(sbatch --dependency=afterok:$jobid0 lenkf_submit_next_batch.j) | cut -d' ' -f 4)
echo $jobid1

sed -i 's/#if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/if($capdate<$enddate) sbatch $HOMDIR\/lenkf.j/g' lenkf.j
~~~~~~~~~~~~~~

> ./ldas_batchrun.j

==========
20190926:

1. All my Discover nodes got logged out. Re-logged in and

> cd ~/merra2 
> nedit extract_daily.gs & [made it start from 2002 where it stopped]
> grads -lbc "run extract_daily.gs"

2. Continued automating GEOSldas spinup:

The ldas_batchrun.j is working, but I forgot to change the end year from 2019 to 1981, so I got this error message: 

ls /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/Y2019/M01/GEOSldas4.5_DE720.catchcn_internal_rst.20190101_0000
ls: cannot access /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/output/global/rs/ens0000/Y2019/M01/GEOSldas4.5_DE720.catchcn_internal_rst.20190101_0000: No such file or directory
if ( 2 == 0 ) then
echo job not finished
exit

Changed the end year in /home/fzeng/Catchment/CLM4.5/submit_next_batch_GEOSldas_DE720 to 1981. 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720/run
> cp -p ldas_batchrun.j lenkf_submit_next_batch.j /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas4.5_DE720_template/run/.
> cd ../..
> mv GEOSldas4.5_DE720 GEOSldas4.5_DE720_1980_2

> cd /discover/nobackup/fzeng/Catchment/M2n5P
> /bin/cp -r GEOSldas4.5_DE720_template GEOSldas4.5_DE720 

> cd GEOSldas4.5_DE720/run
> ./ldas_batchrun.j 
 
3. Read and revised Eunjee's MS. 

==========
20190927:

1. Edited Eunjee's MS.

==========
20190930:

1. Edited Eunjee's MS.