==========
20170403:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170404:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170405:

1. March report.

2. Processed GMAO GEOS5 seasonal forecast for April.

3. Organized emails

4. Met with Eunjee about the manuscript

==========
20170406:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170407:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170410:

1. The e0004s_preindustrialCO2 finished the 17th cycle.

discover05:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01
total 7354336
-rw-r--r-- 1 fzeng g0620 7547615847 2017-04-08 02:18 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z
discover05:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12
total 526496
-rw-r--r-- 1 fzeng g0620 539112640 2017-04-08 02:18 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin

Process the output (covert from tile-space to grid-space):
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2

Check the output on GrADS. Looks good. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/
mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_17
cd e0004s_preindustrialCO2_17/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/
/bin/rm -rf Y200[1-9] Y201[0-5]
mkdir -p Y2001/M01
cd Y2001/M01
ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
Make sure that in M09_CN_e0004s_preindustrialCO2.exe: 
"restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_17/output/"
"restart_id   = e0004s_preindustrialCO2"

Run ldsetup to set up the next cycle:
cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit

Double-checked the executable and the restart file. They are both correct.

2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

3. Updated higgins rain data through Apr 8, 2017.

==========
20170411:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170412:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

2. Discussed with Randy about Eunjee's manuscript (the transient CO2 run part).

==========
20170413:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

2. Discussed with Eunjee about her manuscript (the GPP trend part).

==========
20170414:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170417:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170418:

1. The e0004s_preindustrialCO2 finished the 18th cycle.

discover05:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01
total 7354336
-rw-r--r-- 1 fzeng g0620 7547615847 2017-04-17 09:20 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z
discover05:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12
total 526496
-rw-r--r-- 1 fzeng g0620 539112640 2017-04-17 09:20 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin

Process the output (covert from tile-space to grid-space):
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2

Check the output on GrADS. Looks good. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/
mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_18
cd e0004s_preindustrialCO2_18/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/
/bin/rm -rf Y200[1-9] Y201[0-5]
mkdir -p Y2001/M01
cd Y2001/M01
ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
Make sure that in M09_CN_e0004s_preindustrialCO2.exe: 
"restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_18/output/"
"restart_id   = e0004s_preindustrialCO2"

Run ldsetup to set up the next cycle:
cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_preindustrialCO2/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_preindustrialCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_preindustrialCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit

Double-checked the executable and the restart file. They are both correct.

2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170418:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170419:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170420:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

==========
20170421:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.

2. Sarith showed me the FPAR work.

M36 (36km spatial resolution), US regional run
Saved daily output, averaged to 8-day to match MODIS FPAR temporal resolution

Compared FPAR anomaly between Catchment-CN and MODIS: 
(1) climatology: x -- 1:46 (46*8 = 368 days, about 1 year); y -- each 8-day FPAR anomaly averaged over 2003-2016; one plot for each PFT.
This is for the case if we want to use the clim of MODIS FPAR. 
(2) CDF (cumulative density function): x -- FPAR anomaly; y -- CDF; one plot for each of the 46 8-day periods of the year.
Why use anomaly? Sarith said this is to keep the IAV of Catchment-CN (not sure if I got this correct)? He said we can also use FPAR itself later.  

==========
20170424:

1. Updated higgins rain data through Apr 8, 2017.

2. Organized notes from meeting with Sarith on 20170421.

3. Checked the convergence of the e0004s_preindustrialCO2 run: 18 vs 17 cycles

4. Modified process_cn.F90 for the e0004s_transientCO2 run. Compiled and set up a regional test run.

(4.1) Prepare for the test run:

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run

(a) driver_inputs_M09_neUS.nml (same as test runs for e0004s_preindustrialCO2): 
37N-40N, 75W-78W (893 tiles total); 
3-hrly and monthly tile output.

(b) M09_CN_neUS.exe: 
exp_id = e0004s_transientCO2_neUS
start_time = 2001-01-01-00-00-00
end_time   = 2001-04-01-00-00-00    !! not so important
restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_18/output/

(c) M09_neUS.bat: 
walltime = 01:00:00                 !! not so important
job-name = e0004s_trCO2

Getting lenkf.0.j and lenkf.1.j right is important.

(4.2) Run ldsetup to set up a new experiment:
cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_neUS.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_neUS.bat --runmodel --monthsperjob 15 --landmodel catchCN

In the created /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_neUS/run/lenkf.0.j, added:
#SBATCh --qos=debug
qsub lenkf.1.j
lenkf.0.j: runs from 2001-01-01-00-00-00 to 2001-04-01-00-00-00 and writes out 3hrly and monthly output

Under the same directory /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_neUS/run, created lenkf.1.j:
cp ../../e0004s_preindustrialCO2_neUS/run/lenkf.1.j .
nedit lenkf.1.j &
lenkf.1.j: runs from 2001-04-01-00-00-00 to 2003-01-01-00-00-00 and writes out  monthly output

(4.3) Run the experiment:

qsub lenkf.0.j but it crashed because I forgot to provide year_co2.txt.

echo 1851 > year_co2.txt
cat year_co2.txt (to make sure the year of CO2 is correct)

Run interactively:

ssh -XY discover-sp3
interactive.py -A sp3 -n 84 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_neUS/run
/bin/rm ../output/SMAP_EASEv2_M09_neUS/rc_out/*
./lenkf.0.i

It runs! Stopped it and submit lenkf.0.j. 

The job lenkf.0.j finished successfully.

==========
20170425:

1. Checked the description of variables in the new offline restart file Sarith created /discover/nobackup/smahanam/catchcn_internal_clm45, and got back with suggestions of modifications. 

2. Plotted the 3hrly 20010101 to 20010331 CO2 output from the lenkf.0.j run. It looks correct (same as what we saw in the preindustrial CO2 test run).  

3. Checked the lenkf.1.j. It seems that it didn't run at all. No log file to track. Why?

I forgot to create driver_inputs_M09_neUS_mOUT.nml which lenkf.1.j needs. 
/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_neUS/run > cp ../../e0004s_preindustrialCO2_neUS/run/driver_inputs_M09_neUS_mOUT.nml .

Still doesn't run at all. Why?

Run it interactively to see why.

ssh -XY discover-sp3
interactive.py -A sp3 -n 84 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_neUS/run
/bin/rm ../output/SMAP_EASEv2_M09_neUS/rc_out/*
./lenkf.1.i

It runs. So it seems that I should have "/bin/rm ../output/SMAP_EASEv2_M09_neUS/rc_out/*" before submitting lenkf.1.j. 

Now, try:
/bin/rm ../output/SMAP_EASEv2_M09_neUS/rc_out/*
modify lenkf.1.j to make it a debug run
qsub lenkf.1.j
It runs!

Changed the time requested and submitted lenkf.1.j. It is running!

Plotted the monthly 20010101 tl 20021231 CO2 output from the lenkf.1.j run. It looks correct (CO2 increasing from year to year).  

4. Listed CO2 years and meteorology years for each segment in the 1851-2000 transient run. Use 2001-2015 meteorology for each cycle or longer? Will ask Randy. 

5. The e0004s_preindustrialCO2 finished the 19th cycle.

discover07:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2016/M01
total 7354336
-rw-r--r-- 1 fzeng g0620 7547615847 2017-04-25 11:11 e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z
discover07:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2015/M12
total 526496
-rw-r--r-- 1 fzeng g0620 539112640 2017-04-25 11:11 e0004s_preindustrialCO2.ens_avg.ldas_tile_monthly_out.201512.bin

Process the output (covert from tile-space to grid-space):
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_preindustrialCO2

Check the output on GrADS. Looks good. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/
mv e0004s_preindustrialCO2 e0004s_preindustrialCO2_19
cd e0004s_preindustrialCO2_19/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/
/bin/rm -rf Y200[1-9] Y201[0-5]
mkdir -p Y2001/M01
cd Y2001/M01
ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20010101_0000z

6. Set up the transient CO2 run, assuming that we use 2001-2015 meteorology for each cycle (same as the preindustrial CO2 run). Will re-do if a longer period of meteorology is desired. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
cp M09_CN_e0004s_preindustrialCO2.exe M09_CN_e0004s_transientCO2.exe
cp M09_e0004s_preindustrialCO2.bat M09_e0004s_transientCO2.bat
nedit M09_CN_e0004s_transientCO2.exe M09_e0004s_transientCO2.bat &

Run ldsetup to set up a new experiment:
cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below) 

Check the executable, restart file and create year_co2.txt:
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2
ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
-rwxr-xr-x 1 fzeng g0620 69819515 2017-04-24 14:43 build/Linux/bin/LDASsaCN_mpi.x*
ls -l input/restart/ (to make sure the restart file is the right one)
total 0
lrwxrwxrwx 1 fzeng g0620 84 2017-04-25 16:21 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_19/output/
cd run
echo 1851 > year_co2.txt
cat year_co2.txt (to make sure the year of CO2 is correct)

Modify the job scripts before submission:
Under /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run:
mkdir ../lenkf_orig      (Important: lenkf_orig has to be one level up. Otherwise the lenkf.*.j files in lenkf_orig will be counted by edit_lenkf.csh)
cp -p lenkf.*.j ../lenkf_orig/.
cp ../../../M2n5P/CF0180x6C/CF0180x6C_13/run/edit_lenkf.csh .
nedit edit_lenkf.csh &
./edit_lenkf.csh 
Check the lenkf.*.j files and they look correct. 
qsub lenkf.0.j

NOTE: for transientCO2 run,  

Need to use edit_lenkf.csh to add these lines below to the end of each job file lenkf.N.j (as above):
cat year_ts.txt >> year_ts_all.txt
cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt
cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt
qsub lenkf.N+1.j

Need to copy edit_lenkf.csh, year_co2.txt, year_ts_all.txt, EEAco2g_ts_all.txt and co2_scalar_ts_all.txt from previous cycle.

==========
20170426:

1. Checked the transient CO2 run. It's simulating July 2002. The .txt files written out look correct so far. 

Process the available output (covert from tile-space to grid-space):
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2

Verify the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2.gs

2. Prepared for meeting with Randy this afternoon.

3. Met with Randy.

Randy said it's better to use a longer period of meteorology (e.g. 1981-2016) in the transient run because it gives a better climatology. It doesn't matter that it's different from the period of meteorolgy (2001-2015) used in the preindustrial CO2 spin up run. 

So in the transient CO2 run:

CO2_year     Meteorology years
1851-1872    1995-2016
1873-1908    1981-2016
1909-1944    1981-2016   
1945-1980    1981-2016
1981-2016    1981-2016

4. Re-do the transient CO2 run. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09
mv e0004s_transientCO2 e0004s_transientCO2_2001-2015

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
nedit M09_CN_e0004s_transientCO2.exe &
start_time              = 1995-01-01-00-00-00
end_time                = 2017-01-01-00-00-00

Run ldsetup to set up a new experiment:
cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

Check the executable, restart file and create year_co2.txt:
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2
ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
-rwxr-xr-x 1 fzeng g0620 69819515 2017-04-24 14:43 build/Linux/bin/LDASsaCN_mpi.x*
ls -l input/restart/ (to make sure the restart file is the right one)
lrwxrwxrwx 1 fzeng g0620 84 2017-04-26 17:16 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_19/output/
cd run
echo 1851 > year_co2.txt
cat year_co2.txt (to make sure the year of CO2 is correct)

Modify the job scripts before submission:
Under /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run:
mkdir ../lenkf_orig      (Important: lenkf_orig has to be one level up. Otherwise the lenkf.*.j files in lenkf_orig will be counted by edit_lenkf.csh)
cp -p lenkf.*.j ../lenkf_orig/.
cp ../../e0004s_transientCO2_2001-2015/run/edit_lenkf.csh .
./edit_lenkf.csh 
Check the lenkf.*.j files and they look correct. 
qsub lenkf.0.j

NOTE: for transientCO2 run,  

(1) Need to use edit_lenkf.csh to add these lines below to the end of each job file lenkf.N.j (as above):
cat year_ts.txt >> year_ts_all.txt
cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt
cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt
qsub lenkf.N+1.j

(2) Need to copy edit_lenkf.csh, year_co2.txt, year_ts_all.txt, EEAco2g_ts_all.txt and co2_scalar_ts_all.txt from previous cycle.

(3) When the first cycle finishes, need to change the start_time in M09_CN_e0004s_transientCO2.exe before running ldsetup to set up the 2nd cycle.

The run crashed. 
forrtl: error (73): floating divide by zero
LDASsaCN_mpi.x     000000000077CEFD  catchment_cn_mode        1749  catchmentCN.F90

==========
20170427:

1. Found that the run printed out a lot of this line "Transient period:: forcing year, CO2 year, EEA global average CO2 (ppm), co2 scalar" because I forgot to add "if(master_proc)" before the print statement. 

Added it and recompiled. 
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/e0004s_transientCO2/.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > ls -l exec/e0004s_transientCO2/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 exec/e0004s_transientCO2/Linux/bin/LDASsaCN_mpi.x*

Found this in the log file:
could not find file: ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y1995/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.19950101_0000z

So the run crashed because I forgot to provide a restart file. Fixed this:
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_19/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/
mkdir -p Y1995/M01
cd Y1995/M01
ln -s ../../Y2016/M01/e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.20160101_0000z e0004s_preindustrialCO2.ens0000.catchcn_ldas_rst.19950101_0000z

Re-do the transient CO2 run. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09
rm -rf e0004s_transientCO2

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
nedit M09_CN_e0004s_transientCO2.exe &
start_time              = 1995-01-01-00-00-00
end_time                = 2017-01-01-00-00-00

Run ldsetup to set up a new experiment:
cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

Check the executable, restart file and create year_co2.txt:
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2
ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
-rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 build/Linux/bin/LDASsaCN_mpi.x*
ls -l input/restart/ (to make sure the restart file is the right one)
lrwxrwxrwx 1 fzeng g0620 84 2017-04-27 10:21 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_preindustrialCO2_19/output/

Create year_co2.txt:
cd run
echo 1851 > year_co2.txt
cat year_co2.txt (to make sure the year of CO2 is correct)

Modify the job scripts before submission:
Under /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run:
mkdir ../lenkf_orig      (Important: lenkf_orig has to be one level up. Otherwise the lenkf.*.j files in lenkf_orig will be counted by edit_lenkf.csh)
cp -p lenkf.*.j ../lenkf_orig/.
cp ../../e0004s_transientCO2_2001-2015/run/edit_lenkf.csh .
./edit_lenkf.csh 
Check the lenkf.*.j files and they look correct. 
qsub lenkf.0.j

NOTE: for transientCO2 run,  

(1) Need to use edit_lenkf.csh to add these lines below to the end of each job file lenkf.N.j (as above):
cat year_ts.txt >> year_ts_all.txt
cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt
cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt
qsub lenkf.N+1.j

(2) Need to copy edit_lenkf.csh, year_co2.txt, year_ts_all.txt, EEAco2g_ts_all.txt and co2_scalar_ts_all.txt from previous cycle.

(3) When the first cycle finishes, need to change the start_time in M09_CN_e0004s_transientCO2.exe before running ldsetup to set up the 2nd cycle.

2. Set up the CO2 experiment that uses CT 3hrly CO2 + 200ppm:

cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001
mkdir -p M2.n5P.3hco2+200ppm.glb/RUN/rs/ens0000/Y2001/M01
cd M2.n5P.3hco2+200ppm.glb
cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/CN_restart .
cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.3hco2+200ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+200ppm.glb > ls -l CN_restart 
-rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart
dali10:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+200ppm.glb > ls -l RUN/rs/ens0000/Y2001/M01/
-rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.3hco2+200ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

cp ../M2.n5P.3hco2.glb/lenkf.j .
nedit lenkf.j &

ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+200ppm.x
-rwxr-xr-x 1 fzeng g0620 49563596 2017-04-27 10:43 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+200ppm.x*

qsub lenkf.j

It's running.

Processed the available output:
~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.3hco2+200ppm.glb

Verified the output using ~/Catchment/M2n5P_m0001/verify_M2.n5P.3hco2+200ppm.glb.gs. Looks correct.

==========
20170428:

1. Further verified the transient CO2 run output. 

Process the available output (covert from tile-space to grid-space):
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2

Verify the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2.gs

2. Processed the available M2.n5P.3hco2+200ppm.glb output. 

3. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily.