==========
20180801:

1. The e0004s_transientCO2 run crashed and I couldn't find the log or error file anywhere. 

Did an interactive run:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run
> ./lenkf.44.j

limit stacksize unlimited
source ../build/Linux/bin/g5_modules
../build/Linux/bin/g5_modules: No such file or directory.

> ls -l ../build
lrwxrwxrwx 1 fzeng g0620 85 2017-07-10 14:05 ../build -> /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2

The /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2 no longer exists! It was renamed to /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_HNL.

> cd ..
> mv build build_old
> ln -s /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_HNL build

> cd run
> cp -p driver_inputs_M09_GLOBAL.nml driver_inputs_M09_GLOBAL.nml.old
> nedit driver_inputs_M09_GLOBAL.nml &
veg_path             = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
alb_path             = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
soil_path            = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
top_path             = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'

mwRTM_param_path     = ''

tile_coord_path      = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
catchment_def_path   = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'

tile_coord_file      = 'SMAP_EASEv2_M09_3856x1624.til'
catchment_def_file   = 'catchment.def'

> ./lenkf.44.j

It is running. Stopped it at date_time_new 20170101_091500z.

> qsub lenkf.44.j

2. Describe the the executables in the following subdirectories in /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec (see notes in 201805 and particularly notes in 201806):

e0004s_transientCO2_HNL: 

time stamp '2017-04-24 14:45', used in the e0004s_transientCO2 (01 through 05) run. Only works for BCs in /discover/nobackup/ltakacs/bcs/Heracles-NL/.

e0004s_transientCO2_INL: 

time stamp '2018-06-04 11:44'. Not used in any run so far. Works for BCs in both /discover/nobackup/ltakacs/bcs/Heracles-NL/ and /discover/nobackup/ltakacs/bcs/Icarus-NL/. 

For this executable, two files in the source code were updated since 2017-04-24. 
(1) lsm_routines.F90 (see notes on 20170714 and notes on 20180604). The change to this file affects model output.
  Also fixed a bug in lsm_routines.F90 (L422):
  Was: bflow(n)=amin1(cond(n),bflow(n))
  Now: bflow(n)=amin1(1000.*cond(n),bflow(n)) 
(2) clsm_ensdrv_init_routines.F90 (see notes on 20180507). This file was changed so that it works for the mosaic_veg_typs_fracs file of both HNL and INL BCs.  

e0004s_transientCO2_INL_20180502: same as e0004s_transientCO2_INL but with BOPT=g

To avoid confusion, delete e0004s_transientCO2_INL_20180502:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec
> rm -rf e0004s_transientCO2_INL_20180502

Record this in /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/README. 

3. Talked to Randy about the bug fix in lsm_routines.F90 (see 2 above). Randy told me to use the corrected lsm_routines.F90 in the e0004s_transientCO2 run starting from 20170101 where it ended previously. 

The e0004s_transientCO2 run submitted in 1 above is already running. Stopped it. 

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/
> rm build
> ln -s /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_INL build 
> cd run

Did an interactive run:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run
> ./lenkf.44.j

It's running. Stopped it at date_time_new 20170101_052230z.

> qsub lenkf.44.j 

Updated /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/README.

4. Sanity checked the hindcast/2015Dec12 run using ~/matlab/verify_hindcast_runs.m.

5. Made the executable for the hindcast runs more generic:

> cd /discover/nobackup/fzeng/Catchment/hindcast
> rm -rf 2015Mar01_old 2015Dec12_old

> cd /discover/nobackup/fzeng/hindcasts
> mkdir -p GEOS_S2S/seasonal/Y2015/dec12/ens1/
> mv *.nc4 *.loc* GEOS_S2S/seasonal/Y2015/dec12/ens1/ 

Next: modify LDAS_Forcing.F90 and the *.exec file

> cd /discover/nobackup/fzeng/Catchment/hindcast/run 
> nedit hindcast.exec &
MET_PATH                : /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015/dec12/ens1/
Set NUM_SGMT to 6 based on the test run done at the end of July.
Set MET_HINTERP to 1 (default). 

==========
20180802:

1. Continued working on 5 of 20180801.

Modified LDAS_Forcing.F90 to make it works for all the 4 ensembles (dec12, dec17, dec22 and dec27 of 2015) that are currently available.

Compiled:

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSmetforce_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6
> source $ESMADIR/src/g5_modules 
> gmake clean
> gmake install

> cd ../../Applications/LDAS_App/
> gmake clean
> gmake install

> cd ../../..
> /bin/cp -pr Linux/ exec/hindcast/.

Set up the run fresh:

> cd /discover/nobackup/fzeng/Catchment/hindcast
> mv 2015Dec12 2015Dec12_old

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.exec /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.bat

> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/run/

Check restart file:
> cat cap_restart 
  20151212 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-02 11:08 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Dec12.catchcn_internal_rst.20151212_0000
lrwxrwxrwx 1 fzeng g0620 111 2018-08-02 11:08 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/output/CF0090x6C/rs/ens0000/2015Dec12.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50737484 2018-08-02 10:50 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the END_DATE, JOB_SGMT and NUM_SGMT. 

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 120 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &

In lenkf.j:
  add "exit" in L179, i.e. after "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x" (so even if you delete lines nothing would happen)

> ./lenkf.j 

For some reason it's complaining about the restart file. Weiyuan found that the restart file in /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/input/restart is either crashed or binary. 

Something went wrong when I set up the run?

2. The e0004s_transientCO2 run finished lenkf.44.j (20170101 - 20171101). Somehow it didn't submit lenkf.45.j. 

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/rc_out > ls -l ../rs/ens0000/Y2017/M11
total 7354336
-rw-r--r-- 1 fzeng g0620 7547615847 2018-08-01 22:50 e0004s_transientCO2.ens0000.catchcn_ldas_rst.20171101_0000z

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL > ls cat/ens_avg/Y2017/M10
e0004s_transientCO2.ens_avg.ldas_tile_monthly_out.201710.bin

Did an interactive run:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run
> ./lenkf.45.j

It's running. Stopped it at date_time_new 20171101_070730z.

> qsub lenkf.45.j

3. Read Eunjee's AMS abstract and provided comments.

==========
20180803:

1. July report.

2. Continued working on 1 of 20180802.

Set up the run fresh:

> cd /discover/nobackup/fzeng/Catchment/hindcast
> rm -rf 2015Dec12 

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.exec /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.bat

> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/run/

Check restart file:
> cat cap_restart 
  20151212 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-03 12:27 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Dec12.catchcn_internal_rst.20151212_0000
lrwxrwxrwx 1 fzeng g0620 111 2018-08-03 12:27 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/output/CF0090x6C/rs/ens0000/2015Dec12.vegdyn_internal_rst
> ncdump -h ../input/restart/catchcn_internal_rst [to make sure the restart file is correct -- nc4 and not crashed.] 

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738315 2018-08-02 15:29 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the END_DATE, JOB_SGMT and NUM_SGMT. 

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 120 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &

In lenkf.j:
  add "exit" in L179, i.e. after "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x" (so even if you delete lines nothing would happen)

> ./lenkf.j  

It's running. Stopped it at 2015-12-14T02:37:30.

Commented out a print statement in LDAS_Forcing.F90 and recompiled (only "gmake install"). 

/discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/run > ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738268 2018-08-03 12:45 ../build/Linux/bin/GEOSldas.x*

Test run interactively again.

Crashed.

  Character Resource Parameter MAPL_ENABLE_BOOTSTRAP: YES
 NOT using buffer I/O for file: ../input/restart/vegdyn_internal_rst
server shmget():: Invalid argument
 MAPL_ShmemMod in line                 1056
 MAPL_ShmemMod in line                  557
MAPL_TileMaskGet                              4944
MAPL_StateVarRead                              851
MAPL_ESMFStateReadFromFile                    5114
 StatePrint:
  State name: VEGDYN_INTERNAL
server shmget():: Invalid argument
     status: Unspecified intent direction, object count: 3
     reconcile needed: T
     Root Attribute
server shmget():: Invalid argument
server shmget():: Invalid argument

Recompiled with "gmake clean" and "gmake install".

/discover/nobackup/fzeng/Catchment/hindcast/2015Dec12/run > ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738268 2018-08-03 12:53 ../build/Linux/bin/GEOSldas.x*

Test run interactively again.

Crashed again, giving the same error messages.

Repeat the step above the set up the experiment again. 

Crashed again.

Build:

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src
> gmake realclean
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6
> source g5_modules
> make -j 8 install |& tee tmpfile.txt

Verify build:

> ./Config/gmh.pl tmpfile.txt
    All 44 packages compiled successfully.

Activate V24_C05 updates:

> cd GEOSldas_GridComp/GEOSsurface_GridComp/
> gmake clean 
> gmake install FOPT='-DLAND_UPD'

> cd ../../Applications/LDAS_App/ 
> gmake install
    
> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> /bin/cp -pr Linux exec/hindcast/.
> ls -l exec/hindcast/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738268 2018-08-03 14:47 exec/hindcast/Linux/bin/GEOSldas.x*

Repeat the step above the set up the experiment again. 

Did an interactive run. It's running. Stopped it at 2015-12-12T04:30:00.

> mv lenkf.j.orig lenkf.j
> qsub lenkf.j

It's running.

3. The e0004s_transientCO2 run finished simulating 20171101 - 20180501. 

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL > ls -l rs/ens0000/Y2018/M05
-rw-r--r-- 1 fzeng g0620 7547615847 2018-08-02 21:16 e0004s_transientCO2.ens0000.catchcn_ldas_rst.20180501_0000z
/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL > ls -l cat/ens_avg/Y2018/M04
-rw-r--r-- 1 fzeng g0620 539112640 2018-08-02 21:16 e0004s_transientCO2.ens_avg.ldas_tile_monthly_out.201804.bin

Processed the output:
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2

==========
20180806: 

1. The hindcast/2015Dec12 run is complete. Sanity checked the forcing in the output by comparing it to the original monthly hindcast data. Looks correct!

Emailed Eunjee the path to the output for her comparison to AGCM results.

Emailed and asked Lesley if she has extracted the dec17, dec22 and dec27 ensembles to her project disk.

2. The GEOSldas_CF90 run has finished simulating through 20001231_21z. 

/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000 > ls -l M12
-rw-r--r-- 1 fzeng g0620 2184676441 2018-08-04 00:59 GEOSldas_CF90.catchcn_internal_rst.20001231_2100

Used Sarith's IDL program to add these 5 variables that are needed for AGCM simulations: CH, CM, CQ, FR, and WW.

> cd /discover/nobackup/fzeng/idl
> cp -p /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000/M12/GEOSldas_CF90.catchcn_internal_rst.20001231_2100 .
> nedit put_var.pro &
  L2: ;filename = 'GEOSldas_CF90.catchcn_internal_rst.20001231_2100'
> module load tool/idl-8.5
> idl
> put_var,file ='GEOSldas_CF90.catchcn_internal_rst.20001231_2100'
> exit  

Emailed Eunjee the path to this restart file.

3. Extended the GEOSldas_CF90 run to 20180101_0000z.

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run
> cat cap_restart 
  20001231 210000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 141 2018-08-04 01:00 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2000/M12/GEOSldas_CF90.catchcn_internal_rst.20001231_2100
lrwxrwxrwx 1 fzeng g0620 116 2018-07-25 13:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/GEOSldas_CF90.vegdyn_internal_rst
> nedit CAP.rc &
  END_DATE: 20180101 000000
> qsub lenkf.j  

4. Processed August GEOS_S2S forecast.

5. The SMAP project: e0005s_transientCO2 run for the 20150301-20150401 period

Following notes in /discover/nobackup/fzeng/notes/SMAP_procedures:

Set up the e0005s_transientCO2 run for the 20150301-20150401 period (following notes on 20180601):

(1) Prepare files for experiment setup: 

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
> cp M09_e0004s_transientCO2.bat M09_e0005s_transientCO2.bat
> cp M09_CN_e0004s_transientCO2.exe M09_CN_e0005s_transientCO2.exe
> nedit M09_e0005s_transientCO2.bat M09_CN_e0005s_transientCO2.exe &
  M09_e0005s_transientCO2.bat:
    job-name                = e0005s_trCO2
  M09_CN_e0005s_transientCO2.exe:
    exp_id                  = e0005s_transientCO2
    start_time              = 2015-03-01-00-00-00
    end_time                = 2015-04-01-00-00-00
    restart_path            = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/
    restart_domain          = SMAP_EASEv2_M09_GLOBAL
    restart_id              = e0004s_transientCO2
    driver_inputs_file      = driver_inputs_M09_GLOBAL_HNL.nml

(2) Run ldsetup:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_HNL/Linux/bin
> source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
> ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0005s_transientCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0005s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN    
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

(3) Check the executable, restart file and create year_co2.txt:

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s_transientCO2 
> ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
    -rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 build/Linux/bin/LDASsaCN_mpi.x* [Correct!]
> ls -l input/restart/ (to make sure the restart file is the right one)
    lrwxrwxrwx 1 fzeng g0620 80 2018-08-06 12:51 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/

(4) Create year_co2.txt:
> cd run
> echo 2015 > year_co2.txt
> cat year_co2.txt (to make sure the year of CO2 is correct)
    2015 [Correct!]

(5) Modify lenkf.0.j:

lenkf.0.j: 

    #SBATCH --time=02:00:00
     -restart_path ../input/restart/output \             [Need modification for the other lenkf.*.j]
     -restart_id e0004s_transientCO2 \                   [Need modification for the other lenkf.*.j]

    Added to the end:
    cat year_ts.txt >> year_ts_all.txt
    cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt
    cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt

Do an interactive run to make sure it runs:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s_transientCO2/run 
> ./lenkf.0.j

It's running. Stopped it at date_time_new 20150301_070000z.

> /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
> qsub lenkf.0.j

6. The SMAP project: e0005s_transientCO2 run from 20150401 onward.

Following notes in /discover/nobackup/fzeng/notes/SMAP_procedures:

Read process_smap and the related GrADS (SMAP.gs and the SMAP.ctl) and Fortran programs (wet2_percentile_SMAP.f90).

==========
20180807:

1. The e0005s_transientCO2 has finished 20150301 through 20150401.

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s_transientCO2 
> ls -l output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M04
-rw-r--r-- 1 fzeng g0620 7547615847 2018-08-06 15:32 e0005s_transientCO2.ens0000.catchcn_ldas_rst.20150401_0000z
> mv build build_20150301-20150401

2. Continued 6 of 20180806.

(1) wet2_percentile_SMAP.f90:

Updated ~/Catchment/SMAP_M09/wet2_percentile_SMAP.f90.

Checked /home/fzeng/Catchment/SMAP_M09/rzmc.f90 and verified that the rzmc data in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_rzmc are in original (i.e. ascending) tile order.

Compiled:
~/Catchment/SMAP_M09 > ifort -o wet2_percentile_SMAP wet2_percentile_SMAP.f90

(2) Updated /home/fzeng/Catchment/SMAP_M09/SMAP.ctl

(3) Updated process_smap

(4) Modified and compiled the driver for the experiment, following the files in /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/e0004s-use_wet2.

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> /bin/cp e0004s_transientCO2/* e0005s_transientCO2/.
> cd e0005s_transientCO2

> nedit process_cn.F90 &

> cp ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/catchmentCN.F90 .
> nedit catchmentCN.F90 &

For the experiment, these 4 files will be modified:
clsm_ensdrv_out_routines.F90
compute_rc.F90
process_cn.F90
catchmentCN.F90

Saved the default copy of catchmentCN.F90 as catchmentCN.F90.restore. The other 3 already have their default copies saved.

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/e0005s_transientCO2
> cp catchmentCN.F90 compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.
> cp clsm_ensdrv_out_routines.F90 process_cn.F90 ../.

> cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
> source $ESMADIR/src/g5_modules
> gmake install

> cd ../../../GEOSlana_GridComp/
> gmake install

> cd ../../Applications/LDAS_App/
> gmake install

> cd ../../..
> mkdir -p exec/e0005s_transientCO2
> /bin/cp -pr Linux exec/e0005s_transientCO2/.

Also, e0004s_transientCO2_INL will be used as default in the future, so:
> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec 
> mv e0004s_transientCO2_INL e0004s_transientCO2

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2
> rm build
> ln -s /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2 build
> rm build_old

Emailed and asked Abhishek if we need to write sifl in the model output. Currently for both the control (e0004s_transientCO2) and experiment (e0005s_transientCO2) sifl is borrowed to save surface atmopheric CO2. Waiting for his reply.

Restored the 4 files modified for e0005s_transientCO2.

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> cp clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90
> cp process_cn.F90.restore process_cn.F90

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/
> cp compute_rc.F90.restore compute_rc.F90
> cp catchmentCN.F90.restore catchmentCN.F90

(5) lenkf.j template

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run
> cp lenkf.45.j lenkf.j.template
> nedit lenkf.j.template &
  #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/YYEARB/MMONTHB/e0004s_transientCO2.ldas_log.YEARBMONTHBDAYB_0000.txt
  #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/YYEARB/MMONTHB/e0004s_transientCO2.ldas_out.YEARBMONTHBDAYB_0000.txt
   /discover/nobackup/UNAME/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/build/Linux/bin/LDASsaCN_mpi.x \
   -start_year YEARB -start_month MONTHB -start_day  DAYB -start_hour  0 -start_min  0 -start_sec  0 \
   -end_year   YERRE -end_month   MONTHE -end_day    DAYE -end_hour    0 -end_min    0 -end_sec    0 \

Next:
Try the last part of process_smap to see if the lenkf.j template works or not.

3. Created symbolic links for the hindcast data:

> cd /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015
> mkdir dec17
> cd dec17
> ln -s /discover/nobackup/projects/gmao/geos_carb/lott/tmp/hindcast_forcing/dec17 ens1

> cd /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015
> mkdir dec22
> cd dec22
> ln -s /discover/nobackup/projects/gmao/geos_carb/lott/tmp/hindcast_forcing/dec22 ens1

> cd /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015
> mkdir dec27
> cd dec27
> ln -s /discover/nobackup/projects/gmao/geos_carb/lott/tmp/hindcast_forcing/dec27 ens1

4. Removed the old and incomplete 2015Dec12 run. 

> cd /discover/nobackup/fzeng/Catchment/hindcast 
> rm -rf 2015Dec12_old

5. Extended the 2015Mar01 run to create initial conditions for the 2015Dec17, 2015Dec22, and 2015Dec27 runs.

(1) Create initial conditions for the 2015Dec17 run:
 
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/run
> cat cap_restart 
20151212 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-07-26 13:59 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Mar01.catchcn_internal_rst.20151212_0000

> nedit CAP.rc &
  END_DATE: 20151217 000000
> nedit lenkf.j &
  #SBATCH --output=/discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rc_out/Y2015/M12/2015Mar01.ldas_log.20151217.txt
  #SBATCH --error=/discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rc_out/Y2015/M12/2015Mar01.ldas_err.20151217.txt  
> qsub lenkf.j 

(2) Create initial conditions for the 2015Dec22 run:
 
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/run
> cat cap_restart 
20151217 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-07 20:45 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Mar01.catchcn_internal_rst.20151217_0000

> nedit CAP.rc &
  END_DATE: 20151222 000000
> nedit lenkf.j &
  #SBATCH --output=/discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rc_out/Y2015/M12/2015Mar01.ldas_log.20151222.txt
  #SBATCH --error=/discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rc_out/Y2015/M12/2015Mar01.ldas_err.20151222.txt  
> qsub lenkf.j

(3) Create initial conditions for the 2015Dec27 run:
 
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/run
> cat cap_restart 
20151222 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-07 21:03 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Mar01.catchcn_internal_rst.20151222_0000

> nedit CAP.rc &
  END_DATE: 20151227 000000
> nedit lenkf.j &
  #SBATCH --output=/discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rc_out/Y2015/M12/2015Mar01.ldas_log.20151227.txt
  #SBATCH --error=/discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rc_out/Y2015/M12/2015Mar01.ldas_err.20151227.txt  
> qsub lenkf.j

All finished.

==========
20180808:
 
1. Set up the 2015Dec17 run:

> cd /discover/nobackup/fzeng/Catchment/hindcast/run
> nedit hindcast.exec & 
EXP_ID                  : 2015Dec17
BEG_DATE                : 20151217 000000
MET_PATH                : /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015/dec17/ens1/

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.exec /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.bat

> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec17/run/

Check restart file:
> cat cap_restart 
  20151217 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-08 09:52 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec17/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Dec17.catchcn_internal_rst.20151217_0000
lrwxrwxrwx 1 fzeng g0620 111 2018-08-08 09:52 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec17/output/CF0090x6C/rs/ens0000/2015Dec17.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738268 2018-08-03 14:47 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the BEG_DATE, END_DATE, JOB_SGMT and NUM_SGMT. 

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 120 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec17/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &

In lenkf.j:
  add "exit" in L179, i.e. after "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x" (so even if you delete lines nothing would happen)

> ./lenkf.j 

It's running. Stopped it at 2015-12-17T04:37:30.

> mv lenkf.j.orig lenkf.j
> qsub lenkf.j

2. Set up the 2015Dec22 run:

> cd /discover/nobackup/fzeng/Catchment/hindcast/run
> nedit hindcast.exec & 
EXP_ID                  : 2015Dec22
BEG_DATE                : 20151222 000000
MET_PATH                : /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015/dec22/ens1/

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.exec /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.bat

> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec22/run/

Check restart file:
> cat cap_restart 
  20151222 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-08 10:04 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec22/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Dec22.catchcn_internal_rst.20151222_0000
lrwxrwxrwx 1 fzeng g0620 111 2018-08-08 10:04 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec22/output/CF0090x6C/rs/ens0000/2015Dec22.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738268 2018-08-03 14:47 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the BEG_DATE, END_DATE, JOB_SGMT and NUM_SGMT. 

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 120 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec22/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &

In lenkf.j:
  add "exit" in L179, i.e. after "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x" (so even if you delete lines nothing would happen)

> ./lenkf.j 

Crashed:

  Character Resource Parameter VEGDYN_INTERNAL_RESTART_FILE: ../input/restart/vegdyn_internal_rst
   Character Resource Parameter MAPL_ENABLE_BOOTSTRAP: YES
 NOT using buffer I/O for file: ../input/restart/vegdyn_internal_rst
server shmget():: Invalid argument
 MAPL_ShmemMod in line                 1056

> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec22/input/restart
> diff vegdyn_internal_rst ../../../2015Dec17/input/restart/. 
Nothing returned, so shouldn't be any issue with vegdyn_internal_rst. 

Don't know why it crashed.

Followed the steps about and re-ran ldsetup again.

Crashed again, giving the same error message.

Tried on Weiyuan's compute. It was able to run without any change! 

Weiyuan said it could be due to distributing the job to different nodes and cores during the interactive run itself. I may terminte the current interactive nodes and re-run "interactive.py -A sp3 -n 120 -a g0620 -X --debug" to try again. Also, I can use Weiyuan's method to get the interactive nodes. However, don't have access to /home/wjiang.

> qsub lenkf.j

3. Set up the 2015Dec27 run:

> cd /discover/nobackup/fzeng/Catchment/hindcast/run
> nedit hindcast.exec & 
EXP_ID                  : 2015Dec27
BEG_DATE                : 20151227 000000
MET_PATH                : /discover/nobackup/fzeng/hindcasts/GEOS_S2S/seasonal/Y2015/dec27/ens1/

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.exec /discover/nobackup/fzeng/Catchment/hindcast/run/hindcast.bat

> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec27/run/

Check restart file:
> cat cap_restart 
  20151227 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-08-08 11:41 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec27/output/CF0090x6C/rs/ens0000/Y2015/M12/2015Dec27.catchcn_internal_rst.20151227_0000
lrwxrwxrwx 1 fzeng g0620 111 2018-08-08 11:41 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Dec27/output/CF0090x6C/rs/ens0000/2015Dec27.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50738268 2018-08-03 14:47 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the BEG_DATE, END_DATE, JOB_SGMT and NUM_SGMT. 

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 120 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/hindcast/2015Dec27/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &

In lenkf.j:
  add "exit" in L179, i.e. after "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x" (so even if you delete lines nothing would happen)

> ./lenkf.j 

It's running. Stopped it at 2015-12-27T04:30:00.

> mv lenkf.j.orig lenkf.j  
> qsub lenkf.j

4. Continued 2 of 20180807.

(1) Tried the last part of process_smap to see if the lenkf.j template in e0004s_transientCO2/run works or not. Fixed a typo in lenkf.j template. It's working.

Also created lenkf.j template for e0005s_transientCO2:

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s_transientCO2/run
> cp ../../e0004s_transientCO2/run/lenkf.j.template .
> nedit lenkf.j.template &
  Replaced "e0004" with "e0005".

(2) Modified and compiled ~/Catchment/SMAP_M09/tile2grid_ease_daily.f90

(3) Copied and updated archive-smap and tile2grid_ease_daily2monthly.f90
 
> cd ~/Catchment 
> cp -p SMAP_M09_old/archive-smap SMAP_M09/.
> nedit archive-smap & 

> cd ~/Catchment/SMAP_M09
> mv tile2grid_ease_monthly.f90 tile2grid_ease_daily2monthly.f90
> nedit tile2grid_ease_daily2monthly.f90 &
> ifort -o tile2grid_ease_daily2monthly tile2grid_ease_daily2monthly.f90

5. Will set up two simulations e0004s_trCO2 (control) and e0005s_trCO2 (experiment) for the SMAP project. e0004s_trCO2 will be the same as e0004s_transientCO2 (symbolic link to e0004s_transientCO2_05) but lai11 (as opposed to sifl in e0004s_transientCO2) will be used to save co2v. e0005s_trCO2 will be the same as e0004s_trCO2 but the rzmc is scaled by SMAP rzmc percentile instead. For both e0004s_trCO2 and e0005s_trCO2, also save sif at 13:30 local time. The current sifl is sif at 09:30 local time.

Modified process_cn.F90 for e0004s_trCO2: now borrow lai11 to save co2v, and borrow lai12 to save 13:30 local time sif; sifl is 9:30am local time sif as it's supposed to be.

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> mkdir e0004s_trCO2
> cp e0004s_transientCO2/* e0004s_trCO2/.
> cd e0004s_trCO2
> nedit process_cn.F90 &
> cp process_cn.F90 ../.
> cp clsm_ensdrv_out_routines.F90 ../.
> cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

NOTE: modification of process_cn.F90 to write out 13:30 sif was actually done on 20180809, so as compilation below.

Compiled:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
> source $ESMADIR/src/g5_modules 
> gmake install

> cd ../../../GEOSlana_GridComp/
> gmake install

> cd ../../Applications/LDAS_App/
> gmake install

> cd ../../..
> mkdir -p exec/e0004s_trCO2
> /bin/cp -pr Linux/ exec/e0004s_trCO2/.

Restored these 3 files on 20180809:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> cp clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90
> cp process_cn.F90.restore process_cn.F90
> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/
> cp compute_rc.F90.restore compute_rc.F90

==========
20180809:

1. The 2015Dec17, 2015Dec22 and 2015Dec27 runs are complete. Emailed Eunjee and Lesley to let them know.

2. Set up e0004s_trCO2 for the 20090101-20150401 period. Abhishek wants the 9:30 and 13:30 local time sif starting from Jan 2009.

(1) Prepare files for experiment setup: 

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
> cp M09_e0004s_transientCO2.bat M09_e0004s_trCO2.bat
> cp M09_CN_e0004s_transientCO2.exe M09_CN_e0004s_trCO2.exe
> nedit M09_e0004s_trCO2.bat M09_CN_e0004s_trCO2.exe &
  M09_e0004s_trCO2.bat:
    job-name                = e0004s_trCO2
  M09_CN_e0004s_trCO2.exe:
    exp_id                  = e0004s_trCO2
    start_time              = 2009-01-01-00-00-00
    end_time                = 2015-04-01-00-00-00
    restart_path            = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/
    restart_domain          = SMAP_EASEv2_M09_GLOBAL
    restart_id              = e0004s_transientCO2
    driver_inputs_file      = driver_inputs_M09_GLOBAL_HNL.nml

(2) Run ldsetup: 

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_trCO2/Linux/bin
> source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
> ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_trCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_trCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN    
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

(3) Check the executable, restart file and create year_co2.txt:

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2 
> ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
    -rwxr-xr-x 1 fzeng g0620 69823800 2018-08-09 11:24 build/Linux/bin/LDASsaCN_mpi.x* [Correct!]
> ls -l input/restart/ (to make sure the restart file is the right one)
    lrwxrwxrwx 1 fzeng g0620 80 2018-08-09 11:41 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/

(4) Create year_co2.txt:
> cd run
> echo 2009 > year_co2.txt
> cat year_co2.txt (to make sure the year of CO2 is correct)
    2009 [Correct!]

(5) Modify lenkf.*.j:

> cp -p ../../e0004s_transientCO2_05/run/edit_lenkf.csh* .
> mkdir ../lenkf.j.orig
> cp -p lenkf.*.j ../lenkf.j.orig/.
> ./edit_lenkf.csh
> nedit lenkf.*.j & (to check these job files)
  Mannually added this block below to the end of the last job file (lenkf.7.j in this case):
  
  cat year_ts.txt >> year_ts_all.txt
  cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt
  cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt

Do an interactive run to make sure it runs:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/run 
> ./lenkf.0.j

It's running. Stopped it at date_time_new 20090101_074500z.

> /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
> qsub lenkf.0.j

3. Created ~/Catchment/SMAP_M09/e0004s_trCO2_monthly.ctl for the monthly output of e0004s_trCO2.

4. Modified e0005s_trCO2/process_cn.F90 to write out sif at 13:30 local time.

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> mv e0005s_transientCO2 e0005s_trCO2
> nedit e0005s_trCO2/process_cn.F90 &

Compile:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/e0005s_trCO2
> cp catchmentCN.F90 compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.
> cp clsm_ensdrv_out_routines.F90 process_cn.F90 ../.

> cd ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
> source $ESMADIR/src/g5_modules
> gmake install

> cd ../../../GEOSlana_GridComp/
> gmake install

> cd ../../Applications/LDAS_App/
> gmake install

> cd ../../..
> mkdir -p exec/e0005s_trCO2
> /bin/cp -pr Linux/ exec/e0005s_trCO2/.

Restored the 4 files:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> cp process_cn.F90.restore process_cn.F90
> cp clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/
> cp catchmentCN.F90.restore catchmentCN.F90
> cp compute_rc.F90.restore compute_rc.F90

To avoid confusion:
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec > rm -rf e0005s_transientCO2
/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > rm -rf e0005s_transientCO2

5. Processed 200901 output of e0004s_trCO2:

~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_trCO2

Verified the CO2, sif at 9:30am and 13:30 local time using ~/Catchment/SMAP_M09/verify_e0004s_trCO2.gs
 
6. Emailed Matt and Ben asking for help to get CESM2 running on Discover.

==========
20180810:

1. The executable in hindcast0 is no longer used. Deleted it to avoid confusion.

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec
> rm -rf hindcast0

2. Checked with Sarith. He doesn't need DE_02880x01440 in my nobackup anymore, and I have a copy in $ARCHIVE. Deleted it to release some space from my NOBACKUP.

> cd /discover/nobackup/fzeng/Catchment/M2n5P
> rm -rf DE_02880x01440

3. Modified catchmentCN.F90 for the interception experiment Randy suggested om 7/31/2018 (see his email).

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> cp -p catchmentCN.F90 catchmentCN.F90.restore
> nedit catchmentCN.F90 &

!       SATCAP(N) = SATCAPFR * ZLAI0 + 1.e-5
        
        ! to fix the issue of too much interception loss due to high LAI biases, rkoster+fzeng, 10 Aug 2018
        SATCAPFR = 0.1
        SATCAP(N) = SATCAPFR * AMIN1( 10., ZLAI0 ) + 1.e-5

> cp -p catchmentCN.F90 catchmentCN.F90.SATCAP

GEOS_CatchCNGridComp.F90 has the transient CO2 feature.

Compiled:

> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules 
> gmake clean
> gmake install

> cd ../../../../Applications/LDAS_App/
> gmake clean
> gmake install

> cd ../../..
> mkdir -p exec/satcap
> /bin/cp -pr Linux/ exec/satcap/.

Set up the experiment: 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/run
> nedit CF0090x6C.exec &
EXP_ID                  : GEOSldas_CF90_satcap
BEG_DATE                : 19810101 000000
END_DATE                : 19820101 000000

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/satcap/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0090x6C.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0090x6C.bat

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/run/

Check restart file:
> cat cap_restart 
  19810101 000000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 155 2018-08-10 14:28 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_satcap.catchcn_internal_rst.19810101_0000
lrwxrwxrwx 1 fzeng g0620 130 2018-08-10 14:28 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_satcap.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50418780 2018-08-10 13:19 ../build/Linux/bin/GEOSldas.x*

CAP.rc: check the END_DATE, JOB_SGMT and NUM_SGMT. No change needed.

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 168 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &
    Add "exit" after L179 "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x"
  
> ./lenkf.j

It's running. Stopped it at 1981-01-01T04:00:00.

> mv lenkf.j.orig lenkf.j

> qsub lenkf.j

Restored catchmentCN.F90:

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> cp catchmentCN.F90.restore catchmentCN.F90

4. Completed Good Housekeeping 5 course.

5. Read the reviews of Joanna's MS from Remote Sensing of Environment.

==========
20180813:

1. The e0004s_trCO2 has finished simulating 20090101 through 20150401. Processed the output

~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_trCO2

For 20150401 onward:

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09
> cp e0004s_transientCO2/run/lenkf.j.template e0004s_trCO2/run/.
> cd e0004s_trCO2/run
> cat year_co2.txt 
        2015  [Correct!]
> nedit lenkf.j.template & [To comment out post processing and archive]

#${HOME}/Catchment/SMAP_M09/tile2grid_ease_daily e0004s_trCO2 YEARB MONTHB DAYB YEARPP MONTHPP DAYPP $USER

# check for full months to archive if post-process successful
# -----------------------------------------------------------
#if($status == 0) archive-smap 

Since Abhishek wants daily output starting 20150401, uncommented the above two lines in lenkf.j.template.

${HOME}/Catchment/SMAP_M09/tile2grid_ease_daily e0004s_trCO2 YEARB MONTHB DAYB YEARPP MONTHPP DAYPP $USER

# check for full months to archive if post-process successful
# -----------------------------------------------------------
if($status == 0) archive-smap 

> cp -p ~/Catchment/SMAP_M09/archive-smap ../.

2. Set up the experiment e0005s_trCO2: 

(1) Prepare files for experiment setup: 

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
> cp M09_e0004s_trCO2.bat M09_e0005s_trCO2.bat
> cp M09_CN_e0004s_trCO2.exe M09_CN_e0005s_trCO2.exe
> nedit M09_e0005s_trCO2.bat M09_CN_e0005s_trCO2.exe &
  M09_e0005s_trCO2.bat:
    job-name                = e0005s_trCO2
  M09_CN_e0005s_trCO2.exe:
    exp_id                  = e0005s_trCO2
    start_time              = 2015-04-01-00-00-00
    end_time                = 2016-02-01-00-00-00
    restart_path            = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/output/
    restart_domain          = SMAP_EASEv2_M09_GLOBAL
    restart_id              = e0004s_trCO2
    driver_inputs_file      = driver_inputs_M09_GLOBAL_HNL_daily.nml

(2) Run ldsetup: 

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0005s_trCO2/Linux/bin
> source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
> ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0005s_trCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0005s_trCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN    
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

(3) Check the executable, restart file and create year_co2.txt:

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s_trCO2 
> ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
    -rwxr-xr-x 1 fzeng g0620 69824336 2018-08-09 12:47 build/Linux/bin/LDASsaCN_mpi.x* [Correct!]
> ls -l input/restart/ (to make sure the restart file is the right one)
    lrwxrwxrwx 1 fzeng g0620 70 2018-08-13 12:58 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/output/

(4) Create year_co2.txt:
> cd run
> echo 2015 > year_co2.txt
> cat year_co2.txt (to make sure the year of CO2 is correct)
    2015 [Correct!]

(5) Copy and modify lenkf.j.template:

> cp -p ../../e0004s_trCO2/run/lenkf.j.template .
> nedit lenkf.j.template & 
#SBATCH --job-name=e0005s_trCO2
#SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/YYEARB/MMONTHB/e0005s_trCO2.ldas_log.YEARBMONTHBDAYB_0000.txt
#SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/YYEARB/MMONTHB/e0005s_trCO2.ldas_out.YEARBMONTHBDAYB_0000.txt
/discover/nobackup/UNAME/Catchment/SMAP_EASEv2_M09/e0005s_trCO2/build/Linux/bin/LDASsaCN_mpi.x \
-exp_id e0005s_trCO2 \
-restart_path ../input/restart/output \  [This line needs modification for the subsequent lenkf.j!]
-restart_id e0004s_trCO2 \               [This line needs modification for the subsequent lenkf.j!]

(6) Copy archive-smap: 

> cp -p ~/Catchment/SMAP_M09/archive-smap ../. 

3. Checked with David Mocko. He said there is no need to change my SMAP.ctl for the new SMAP data. See his email today.

4. process-smap

> cd ~/Catchment/SMAP_M09
> nedit process-smap &
set rst = 1
if ($rst == 1) then

  # set the desired end date for the purpose of saving restart files for specific months
  # F Zeng, Feb 2016
  # ------------------------------------------------------------------------------------
  set ymde = 20160131

endif

Also updated the experiment names.

Run it:

~/Catchment/SMAP_M09 > process-smap
 
data available for all three inputs through 20160131
ls: No match.
processing SMAP data from 20150331 to 20160131

Running in Batch mode
GrADS SMAP.gs: 2015-03-31 to 2016-01-31
 user name: fzeng
 processing SMAP data for:        2015           3          31
    1    1   89 1981    3   30**********************        NaN  0.0037653
    2    2   90 1981    3   31**********************        NaN -0.0000000
    3    3   91 1981    4    1**********************        NaN***********
  
    4    1   89 1982    3   30**********************        NaN -0.0000000
    5    2   90 1982    3   31**********************        NaN***********
    6    3   91 1982    4    1**********************        NaN***********
  
    7    1   89 1983    3   30**********************        NaN***********
    8    2   90 1983    3   31**********************        NaN  0.0000000
    9    3   91 1983    4    1**********************        NaN***********
    
This doesn't look right to me.

Check SMAP.gs and wet2_percentile_SMAP.f90 in ~/Catchment/SMAP_M09.

Added " convert='big_endian'," to wet2_percentile_SMAP.f90 when opening the daily rzmc data for read. Recompiled. Now the numbers make sense.

Ran process-smap again. Need to tune process-smap to make it work.         

5. The GEOSldas_CF90_satcap is complete. Analyzed the output using ~/matlab/plot_eint_evap.m

However, GEOSldas_CF90_satcap is only available for 1981. Re-do GEOSldas_CF90_satcap for the year 2001 to be more comparable to Justin's run starting 20010101.

> cd /discover/nobackup/fzeng/Catchment/M2n5P
> mv GEOSldas_CF90_satcap GEOSldas_CF90_satcap_1981

Set up the experiment: 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/run
> nedit CF0090x6C.exec &
EXP_ID                  : GEOSldas_CF90_satcap
BEG_DATE                : 20001231 210000
END_DATE                : 20020101 000000

RESTART                 : 1
RESTART_ID              : GEOSldas_CF90
RESTART_PATH            : /discover/nobackup/fzeng/Catchment/M2n5P/
RESTART_DOMAIN          : CF0090x6C

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/satcap/Linux/bin
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
> source $ESMADIR/src/g5_modules
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0090x6C.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/CF0090x6C.bat

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/run/

Check restart file:
> cat cap_restart 
  20001231 210000
> ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 155 2018-08-13 11:56 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/output/CF0090x6C/rs/ens0000/Y2000/M12/GEOSldas_CF90_satcap.catchcn_internal_rst.20001231_2100
lrwxrwxrwx 1 fzeng g0620 130 2018-08-13 11:56 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_satcap.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50418780 2018-08-10 13:19 ../build/Linux/bin/GEOSldas.x*

CAP.rc: check the END_DATE, JOB_SGMT and NUM_SGMT. No change needed.

HISTORY.rc: 

  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: added "IMPOSE_CT_CO2:                      1" to the end (L38).

Did an interactive run:

> interactive.py -A sp3 -n 168 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_satcap/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &
    Add "exit" after L179 "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x"
  
> ./lenkf.j

It's running. Stopped it at 2001-01-01T02:00:00.

> mv lenkf.j.orig lenkf.j

> qsub lenkf.j

==========
20180814:

1. Modified ~/Catchment/SMAP_M09/process-smap and the lenkf.j.template in the run directory of e0004s_trCO2 and e0005s_trCO2.

For lenkf.j.template: 
Added "mkdir -p ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/YYEARB/MMONTHB".

For process-smap: 
Updated some numbers to make it work as expected.

Then: 
> cd ~/Catchment/SMAP_M09
> process-smap
To set up and submit both e0004s_trCO2 and e0005s_trCO2.

2. The GEOSldas_CF90_satcap run finished simulating 2001. Analyzed the results using ~/matlab/plot_eint_evap.m

3. The GEOSldas_CF90 finished simulating 19810101 through 20171231.

4. Worked on cesm2:

> cd /discover/nobackup/mathomp4/ForFanwei/cesm/cime/config/cesm/machines 
> cp -p config_compilers.xml config_machines.xml /discover/nobackup/fzeng/clm_orig/cesm2/cime/config/cesm/machines/.
> nedit config_machines.xml & [Changed the directories.]

https://escomp.github.io/cesm/release-cesm2/quickstart.html 

> cd /discover/nobackup/fzeng/clm_orig/cesm2/cime/scripts 
> ./query_config --help
> ./query_config --compsets

I2000Clm50Cn         : 2000_DATM%GSWP3v1_CLM50%CN_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV
I2000Clm50BgcCru     : 2000_DATM%CRUv7_CLM50%BGC_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV

Description of I2000Clm50Cn can be found at http://www.cesm.ucar.edu/models/cesm2/config/compsets.html

Following '1.4 Quick start' on https://esmci.github.io/cime/users_guide/introduction-and-overview.html#prerequisites

> cd /discover/nobackup/fzeng/clm_orig/cesm2/cime/scripts
> ./create_newcase --case I2000Clm50Cn --compset I2000Clm50Cn --res f19_g17
> cd I2000Clm50Cn
> ./case.setup
> ./case.build
> ./case.submit

However, after running the './create_newcase ...', I got:

Compset longname is 2000_DATM%GSWP3v1_CLM50%CN_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV
Compset specification file is /gpfsm/dnb31/fzeng/clm_orig/cesm2/cime/../components/clm//cime_config/config_compsets.xml
Compset forcing is 1972-2004
ATM component is  Data driven ATM  GSWP3v1 data set 
LND component is clm5.0:CN: Carbon Nitrogen model
ICE component is Stub ice component
OCN component is Stub ocn component
ROF component is MOSART: MOdel for Scale Adaptive River Transport
GLC component is cism2 (default, higher-order, can run in parallel):cism ice evolution turned off (this is the standard configuration unless you're explicitly interested in ice evolution):
WAV component is Stub wave component
ESP component is 
Pes     specification file is /gpfsm/dnb31/fzeng/clm_orig/cesm2/cime/../components/clm//cime_config/config_pes.xml
Machine is discover
Pes setting: grid match    is l%1.9x2.5 
Pes setting: grid          is a%1.9x2.5_l%1.9x2.5_oi%null_r%r05_g%gland4_w%null_m%gx1v7 
Pes setting: compset       is 2000_DATM%GSWP3v1_CLM50%CN_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV 
Pes setting: tasks       is {'NTASKS_ATM': -2, 'NTASKS_ICE': -2, 'NTASKS_CPL': -2, 'NTASKS_LND': -2, 'NTASKS_WAV': -2, 'NTASKS_ROF': -2, 'NTASKS_OCN': -2, 'NTASKS_GLC': -2} 
Pes setting: threads     is {'NTHRDS_ICE': 1, 'NTHRDS_ATM': 1, 'NTHRDS_ROF': 1, 'NTHRDS_LND': 1, 'NTHRDS_WAV': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1, 'NTHRDS_GLC': 1} 
Pes setting: rootpe      is {'ROOTPE_OCN': 0, 'ROOTPE_LND': 0, 'ROOTPE_ATM': 0, 'ROOTPE_ICE': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0, 'ROOTPE_ROF': 0, 'ROOTPE_GLC': 0} 
Pes setting: pstrid      is {} 
Pes other settings: {}
Pes comments: none
 Compset is: 2000_DATM%GSWP3v1_CLM50%CN_SICE_SOCN_MOSART_CISM2%NOEVOLVE_SWAV 
 Grid is: a%1.9x2.5_l%1.9x2.5_oi%null_r%r05_g%gland4_w%null_m%gx1v7 
 Components in compset are: ['datm', 'clm', 'sice', 'socn', 'mosart', 'cism', 'swav', 'sesp', 'drv', 'dart'] 

*********************************************************************************************************************************
This compset and grid combination is not scientifically supported, however it is used in 2 tests.
*********************************************************************************************************************************

Using project from config_machines.xml: none
No charge_account info available, using value from PROJECT
Using project from config_machines.xml: none
cesm model version found: cesm2.0.0
Batch_system_type is slurm
job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None
ERROR: No queues found

Asked Justin. Need to add a block for Discover to machines/config_batch.xml. See his email.

==========
20180815:

1. Discover is down for maintenance all day.

2. Worked on PEP - performance review for the new quarter. 

3. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180816:

1. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

2. The SMAP simulations e0004s_trCO2 and e0005s_trCO2 (set up and submitted by process-smap) finished simulating 20150401 through 20160131. 

The post-processing and archive didn't work though. It's because of some bug in tile2grid_ease_daily.f90. Fixed it and recompiled. Then

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s_trCO2/run
> ${HOME}/Catchment/SMAP_M09/tile2grid_ease_daily e0005s_trCO2 2015 04 01 2016 01 31 $USER
> archive-smap

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/run
> ${HOME}/Catchment/SMAP_M09/tile2grid_ease_daily e0004s_trCO2 2015 04 01 2016 01 31 $USER

NOTE: for e0004s_trCO2, need to archive and remove any original output files before 20150401 from cat/ens_avg to make archive-smap work. Used archive-smap_prior2016 to do this. 

> qsub archive-smap_prior2016

After the archive-smap_prior2016 job is complete: 

> archive-smap

3. Found that in e0004s_trCO2/post, monthly grid-space output for 200901 through 201503 has a dimension of 720x360, while the monthly and daily grid-space output after that has a dimension of 720x287 to save disk space. 

Copied the 200901-201503 monthly output from archive. e.g.
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/output/SMAP_EASEv2_M09_GLOBAL/cat/ens_avg
> cp /archive/u/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/e0004s_trCO2_tile_monthly_2009.tgz .
> tar -xf e0004s_trCO2_tile_monthly_2009.tgz

Saved a copy of the previous 720x360 output files.
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/post
> mkdir old
> mv e0004s_trCO2_2009??.gdat old
> mv e0004s_trCO2_2010??.gdat old
> mv e0004s_trCO2_2011??.gdat old
> mv e0004s_trCO2_2012??.gdat old
> mv e0004s_trCO2_2013??.gdat old
> mv e0004s_trCO2_2014??.gdat old
> mv e0004s_trCO2_201501.gdat old
> mv e0004s_trCO2_201502.gdat old
> mv e0004s_trCO2_201503.gdat old

Modified ~/Catchment/SMAP_M09/tile2grid_ease_spinup_monthly.f90 for e0004s_trCO2 to write out 720x287 output files.  
 
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_trCO2

4. Compared rzmc, gpp etc between e0004s_trCO2 and e0005s_trCO2

5. Modified process-smap to submit the next batch for both simulations: first check the rzmc data!

> ~/Catchment/SMAP_M09 
> nedit process-smap &
L109:   set ymde = 20161130
> process-smap
It will first compute rzmc for e0005s_trCO2. This takes a couple of hours. It will then create and submit lenkf.j for both simulations.

==========
20180817:

1. For some reason e0004s_trCO2 did not run. e0005s_trCO2 is almost done by 9:50am. The lenkf.j file is the same between the two except exp_id and restart_id. Don't understand.

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/run

Did an interactive run:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_trCO2/run
> chmod 755 lenkf.j
> ./lenkf.j

It's running. Stopped it at date_time_new 20160201_040730z.

> qsub lenkf.j 

==========
20180820:

1. Analyzed the output of e0004s_trCO2 and e0005s_trCO2 using fpar_smap_3panel.gs, gpp_smap_3panel.gs, lhflux_smap_3panel.gs, nee_smap_3panel.gs, npp_smap_3panel.gs and rzmc_smap_3panel.gs in ~/Catchment/SMAP_M09.

2. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180821:

1. Prepared meeting slides.

2. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

3. Continued investigating the high LAI41 issue:

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > cp /archive/u/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_28.tgz .

The restart file is CN_restart, so my e0004s_28 used the m3-15 tag.

Can't plot LAI11 through LAI41 using ~/matlab/plot_CN_restart_pftvars.m.

==========
20180822:

1. Continued working on cesm2:

https://escomp.github.io/cesm/release-cesm2/quickstart.html 

Following '1.4 Quick start' on https://esmci.github.io/cime/users_guide/introduction-and-overview.html#prerequisites

Start from fresh:

> cd /discover/nobackup/fzeng/clm_orig/cesm2/cime/scripts
> rm -rf I2000Clm50Cn
> ./create_newcase --case I2000Clm50Cn --compset I2000Clm50Cn --res f19_g17
> cd I2000Clm50Cn
> mkdir -p /discover/nobackup/fzeng/clm_orig/cesm2/inputdata
> ./case.setup

ERROR: module command None purge  failed with message:
/bin/sh: None: command not found

Asked Matt and Justin. 

Modified cesm2/cime/config/cesm/machines/config_machines.xml following Justin's. 

Repeat the above steps again.

ERROR: Not safe to leave unresolved items in env var value: '$MKLPATH'

2. 

> cd ~/matlab/SMAP
> mv read_fields.m tile2globalSUM.m

3. Meeting with Randy and Eunjee

4. All hands + Jonathan's farewell party

==========
20180823:

1. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180824:

1. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180827:

1. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

2. Met with Randy and Eunjee

==========
20180828:

1. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

2. Met with Randy, Rolf and Jana on Jana's FPAR assimilation work

3. Met with Randy, Rolf and Qing on the SMAP project

==========
20180829:

1. Computed standard deviation of rzmc difference (= exp - ctrl) from the SMAP project, using ~/matlab/SMAP/calculate_std_diff.m

2. IDS meeting

==========
20180830:

1. Ran process-smap to extend the e0004s_trCO2 and e0005s_trCO2 runs through 20180604.

2. Extent the GEOSldas_CF90 run to 20180731 to examine the impact of the 2018 summer heat wave. 

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/run
> cat cap_restart 
20180101 000000
> ls -l ../input/restart/
total 0
lrwxrwxrwx 1 fzeng g0620 141 2018-08-14 11:10 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/Y2018/M01/GEOSldas_CF90.catchcn_internal_rst.20180101_0000
lrwxrwxrwx 1 fzeng g0620 116 2018-07-25 13:14 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90/output/CF0090x6C/rs/ens0000/GEOSldas_CF90.vegdyn_internal_rst
> nedit CAP.rc &
END_DATE: 20180801 000000
> qsub lenkf.j

3. For the new experiment to investigate the impact of Princeton vs. MERRA2 precipitation difference on GPP.

Since all the files in ~/Catchment/M2n5P_m0001/princetonVSmerra2 are matlab program files, moved the entire directory to ~/matlab:
~/Catchment/M2n5P_m0001 > mv princetonVSmerra2 ../../matlab/

Created and ran ~/Catchment/merra2_prcp_monthly_cli.f90 to compute MERRA2 monthly climatoloty for precipitation (sum of convective rainfall, large scale rainfall and snowfall).

4. Modified the model code for the new experiment to investigate the impact of Princeton vs. MERRA2 precipitation difference on GPP.

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> xdiff clsm_ensdrv_force_routines.F90 clsm_ensdrv_force_routines.F90.restore & [Identical. Good!]
> xdiff process_cn.F90 process_cn.F90.restore & [Identical. Good!]

> cp clsm_ensdrv_force_routines.F90 clsm_ensdrv_force_routines.F90.pPRCP
> nedit clsm_ensdrv_force_routines.F90.pPRCP & [Following modifications in clsm_ensdrv_force_routines.F90.pTair]

> cp clsm_ensdrv_force_routines.F90.pPRCP clsm_ensdrv_force_routines.F90
> setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
> source $ESMADIR/src/g5_modules 
> gmake install
> cd ../../Applications/LDAS_App/
> gmake install
> cd ../../..
> mkdir -p exec/clm4_pPRCP
> /bin/cp -pr Linux exec/clm4_pPRCP/.

Restored clsm_ensdrv_force_routines.F90:

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
> cp clsm_ensdrv_force_routines.F90.restore clsm_ensdrv_force_routines.F90

5. SIF unit?

6. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180831:

1. Met with Sarith on the pLAI work. 

2. Worked on Catchment-CN with CLM4.5. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. 

3. Verify the Princeton and MERRA2 precipitation climatology computed, using ~/Catchment/CLM4/verify_exp.gs

4. Set up and automate the clm4_pPRCP_DE720 spinup run:

(1) Create the template:

Modified D0.5_cn.exe and D0.5.bat in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run.

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4_pPRCP/Linux/bin
> source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
> ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/
> mv clm4_pPRCP_DE720 clm4_pPRCP_DE720_template
> cd clm4_pPRCP_DE720_template/run
> nedit lenkf.0.j & 
  changed "-restart_path ../input/restart/rst_clm4 \" to "-restart_path ../input/restart/output \" 
  -restart_id clm4_pPRCP_DE720 \ 
> cp -p ../../clm4_DE720_42/run/edit_lenkf.csh .
> cp -p ../../clm4_DE720_42/run/LSM_Calib.rc .
> mkdir ../lenkf_orig
> cp -p lenkf.*.j ../lenkf_orig/.
> ./edit_lenkf.csh
Change the last line to "/home/fzeng/Catchment/CLM4/submit_next_batch_DE720 clm4_pPRCP_DE720" in lenkf.4.j

(2) Set up the clm4_pPRCP_DE720_20 cycle following steps in /home/fzeng/Catchment/CLM4/submit_next_batch_DE720:

> set exp = clm4_pPRCP_DE720

> cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4
> /bin/cp -r ${exp}_template ${exp}
> cd ${exp}/input/restart
> ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output
> /bin/rm rst_clm4 

> cd ../../run
        
Check the executable:
> ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/build/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 70070449 2018-08-30 12:46 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/build/Linux/bin/LDASsaCN_mpi.x*

Check the restart file:
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/input/restart > ls -l
lrwxrwxrwx 1 fzeng g0620 76 2018-08-31 13:57 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output/

> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/run
> nedit lenkf.0.j & 
  change "-restart_id clm4_pPRCP_DE720 \" to "-restart_id clm4_DE720 \"

Did an interactive run to make sure it works:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pPRCP_DE720/run
> ./lenkf.0.j

It's running. Stopped it at date_time_new 19800101_040730z.

> /bin/rm ../output/global/rc_out/*
> qsub lenkf.0.j

5. August report.