==========
20190603:

1. May report.

2. Met with Randy and Eunjee to update my work.

3. Processed GMAO June forecast.

4. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

5. For hindcast data bias correction: download monthly and daily hindcast data

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng
> mv FORECASTS FORECASTS_old
> mkdir FORECASTS

> cd ~/geos5/FORECASTS_BCSD/Fanwei/download
> rm -rf Log_files_old
> mkdir Log_files_old
> mv *.err *.out Log_files_old/

(1) Modified scripts for downloading hindcast data.

> mkdir scripts_old

> cp -p Download_GEOS5V2_all.H.sh scripts_old/.
> nedit Download_GEOS5V2_all.H.sh &

> cp -p DNload_code_library/process_monthly_forecasts.scr scripts_old/.

Now the variables are from two collections: 'geosgcm_vis2d' and 'geosgcm_surf'.
Modified DNload_code_library/process_monthly_forecasts.scr for this.

==========
20190604:

1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

2. For hindcast data bias correction: download monthly and daily hindcast data

The modifications made to DNload_code_library/process_monthly_forecasts.scr have some error.

Kept modifying it until it works as expected.

Checked the output by comparing to the old files using "ncview" and "ncdiff".

ncdiff: Command not found.

module load other/nco-4.6.8-gcc-5.3-sp3 [so that ncdiff works, see NCCS email on 20181116]

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Monthly/1982/jan01/ens1
> ncdiff jan01.monthly.198207.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Monthly/1982/jan01/ens1/geosgcm_vis2d/jan01.geosgcm_vis2d.monthly.198207.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old monthly output, i.e. all zeros in NEWmOLD. Great!
> rm NEWmOLD

process_monthly_forecasts.scr is good now. It downloads the monthly hindcast data. 

Need to modify the one downloading the daily hindcast next.

> cp -p DNload_code_library/run_process_daily_forecasts.scr scripts_old/.
> cp -p DNload_code_library/process_daily_forecasts.scr scripts_old/.

Daily geos_surf data no long exist in NOBACKUP. Need to extract from ARCHIVE. 

More modifications needed. 

==========
20190605:

1. For hindcast data bias correction: download monthly and daily hindcast data

Continued modifying the scripts for downloading daily forecast data.

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily/1982/jan01/ens1 
> ncdiff jan01.19811212_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1982/jan01/ens1/geosgcm_vis2d/jan01.geosgcm_vis2d.19811212_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old daily output, i.e. all zeros in NEWmOLD. Great!
> rm NEWmOLD

/discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily/1982/jan01 > rm -rf *
/discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Monthly > rm -rf *

~/geos5/FORECASTS_BCSD/Fanwei/download > Download_GEOS5V2_all.H.sh

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Monthly/1982/jan01/ens1
> ncdiff jan01.monthly.198207.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Monthly/1982/jan01/ens1/geosgcm_vis2d/jan01.geosgcm_vis2d.monthly.198207.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old monthly output, i.e. all zeros in NEWmOLD. Great!
> rm NEWmOLD

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily/1982/jan01/ens1 
> ncdiff jan01.19811212_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1982/jan01/ens1/geosgcm_vis2d/jan01.geosgcm_vis2d.19811212_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old daily output, i.e. all zeros in NEWmOLD. Great!

There is something wrong with downloading the variables from the geosgcm_surf collection. It worked during the test. Don't know why. Investigating ...

In Download_GEOS5V2_all.H.sh:
#     sbatch $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
      sh $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
      
No other changes.

Run Download_GEOS5V2_all.H.sh interactively (instead of sbatch). This is working. Somehow the sbatch jobs encountered some issue which I don't understand. Just let it run interactively since it's working.

2. For hindcast data bias correction: download monthly MERRA-2 data

> cp -p Download_Monthly_MERRA2.scr scripts_old/.
> nedit Download_Monthly_MERRA2.scr &

==========
20190606:

1. Prepared slides for meeting this afternoon.

2. For hindcast data bias correction: download monthly MERRA-2 data

Continued modifying Download_Monthly_MERRA2.scr.

3. Met with Randy and Eunjee. 

4. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

==========
20190607:

1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

2. For hindcast data bias correction: download daily hindcast data

The daily hindcast data was downloaded interactively because using sbatch gave me some errors. However, the interactive downloading process was interrupted due to lost of connection to NCCS, when it was downloading 1992 hindcast daily data. 

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily
> rm -rf 1992

> cd ~/geos5/FORECASTS_BCSD/Fanwei/download 
> nedit Download_GEOS5V2_all.H.sh &
   FCST_SYR=1992  #1982
   FCST_EYR=1995  #2016  [tried for 4 years]
    for ((YEAR=$FCST_SYR; YEAR<=$FCST_EYR; YEAR++)); do
      sbatch $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
#     sh $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
    done
> Download_GEOS5V2_all.H.sh

Checked the 1991 data:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily/1991/jan01/ens4 
> ncdiff jan01.19910930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1991/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.19910930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old daily output, i.e. all zeros in NEWmOLD. Great!
> rm NEWmOLD 

The batch jobs are still in the queue.

Download these years interactively.

> cd ~/geos5/FORECASTS_BCSD/Fanwei/download 
> nedit Download_GEOS5V2_all.H.sh &
   FCST_SYR=1996  #1982
   FCST_EYR=2016  #2016  
    for ((YEAR=$FCST_SYR; YEAR<=$FCST_EYR; YEAR++)); do
#     sbatch $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
      sh $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
    done
> Download_GEOS5V2_all.H.sh

The interactive job is running.

The batch jobs failed again. Error message example:
tar: /archive/gmaofcst/GEOS_S2S/seasonal/Y1991/dec27/ens1/geosgcm_surf/*.nc4.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tool(90):ERROR:105: Unable to locate a modulefile for 'tool/tview-8.11.0.0'
cdo.exe selname: Open failed on >/discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2//Daily/1992/jan01/ens4/unarchive/*.nc4<
No such file or directory
tool(90):ERROR:105: Unable to locate a modulefile for 'tool/tview-8.11.0.0'
cdo.exe remapbil: Open failed on >/discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2//Daily/1992/jan01/ens4/junk_geosgcm_surf..nc4<
No such file or directory
No such file or directory
Location: file nccopy.c; line 1354
tool(90):ERROR:105: Unable to locate a modulefile for 'tool/tview-8.11.0.0'
cdo.exe merge (Abort): Output file name /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2//Daily/1992/jan01/ens4/jan01.geosgcm_vis2d.19920930_0900z.nc4 is equal to input file name on position 279!

Need to download daily hindcast data for 1992-1995 later. 
/discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily > rm -rf 1992 1993 1994 1995

Tried this:
> cd ~/geos5/FORECASTS_BCSD/Fanwei/download
> nedit DNload_code_library/run_process_daily_forecasts.scr &
Added "#SBATCH --ntasks=1"
> nedit Download_GEOS5V2_all.H.sh &
   FCST_SYR=1992  #1982
   FCST_EYR=1995  #2016

    for ((YEAR=$FCST_SYR; YEAR<=$FCST_EYR; YEAR++)); do
      sbatch $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
#     sh $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
    done

> Download_GEOS5V2_all.H.sh

3. For hindcast data bias correction: download monthly MERRA-2 data separately for different time periods

> nedit Download_Monthly_MERRA2.scr &
> Download_Monthly_MERRA2.scr 1982 1991
> Download_Monthly_MERRA2.scr 1992 2000
> Download_Monthly_MERRA2.scr 2001 2010
> Download_Monthly_MERRA2.scr 2011 2017

> module load other/nco-4.6.8-gcc-5.3-sp3 [so that ncdiff works, see NCCS email on 20181116]
> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/MERRA2/Monthly
> ncdiff all_forc_var_198201.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/MERRA2/Monthly/all_forc_var_198201.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWGAB", "PS", "SPEED" and "SWGDN" are identical between the new and old monthly output, i.e. all zeros in NEWmOLD. Great!
> rm NEWmOLD 

> ncdiff all_forc_var_201612.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/MERRA2/Monthly/all_forc_var_201612.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWGAB", "PS", "SPEED" and "SWGDN" are identical between the new and old monthly output, i.e. all zeros in NEWmOLD. Great!
> rm NEWmOLD 

4. For hindcast data bias correction: calculate hindcast and MERRA-2 monthly climatology

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2
> /bin/rm Log_files_old/*
> mv *.err *.out Log_files_old/
> mkdir scripts_old

> cp -p PART1_CLIM.sh scripts_old/.
> nedit PART1_CLIM.sh &

> cp -p code_library/Calc_and_Write_observational_climatology.py scripts_old/.
> cp -p code_library/Calc_and_Write_forecast_climatology.py scripts_old/.

> nedit code_library/Calc_and_Write_observational_climatology.py &  [no changes made]

> nedit code_library/Calc_and_Write_forecast_climatology.py &

> PART1_CLIM.sh

Checked the output:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS
> diff LWGAB_obs_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/LWGAB_obs_clim.nc 
Returned nothing, so they are identical. Great!
> diff PS_obs_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/PS_obs_clim.nc 
Returned nothing, so they are identical. Great!
> diff SPEED_obs_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/SPEED_obs_clim.nc 
Returned nothing, so they are identical. Great!
> diff SWGDN_obs_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/SWGDN_obs_clim.nc 
Returned nothing, so they are identical. Great!

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST
> diff LWS_fcst_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST/LWS_fcst_clim.nc
Returned nothing, so they are identical. Great!
> diff PRECTOT_fcst_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST/PRECTOT_fcst_clim.nc
Returned nothing, so they are identical. Great!
> diff PS_fcst_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST/PS_fcst_clim.nc
Returned nothing, so they are identical. Great!
> diff SLRSF_fcst_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST/SLRSF_fcst_clim.nc
Returned nothing, so they are identical. Great!
> diff SPEED_fcst_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST/SPEED_fcst_clim.nc
Returned nothing, so they are identical. Great!

5. For hindcast data bias correction: bias correct 2014 monthly hindcast data

> cp -p PART2_BCSD-Calc.H.sh scripts_old/.
> nedit PART2_BCSD-Calc.H.sh &
   FCST_SYR=2014
   FCST_EYR=2014

> cp -p code_library/run_BCSD_calc.scr scripts_old/.
> nedit code_library/run_BCSD_calc.scr & 
Changed the location of the log files.

> cp -p code_library/Bias_correction_module.py scripts_old/.
> nedit code_library/Bias_correction_module.py &

> PART2_BCSD-Calc.H.sh

==========
20190610:

1. This below I tried on 20190607 still did not work.

Tried on 20190607:
> cd ~/geos5/FORECASTS_BCSD/Fanwei/download
> nedit DNload_code_library/run_process_daily_forecasts.scr &
Added "#SBATCH --ntasks=1"
> nedit Download_GEOS5V2_all.H.sh &
   FCST_SYR=1992  #1982
   FCST_EYR=1995  #2016

    for ((YEAR=$FCST_SYR; YEAR<=$FCST_EYR; YEAR++)); do
      sbatch $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
#     sh $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
    done
> Download_GEOS5V2_all.H.sh

/discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily > rm -rf 1992 1993 1994 1995

Download the 1992-1995 daily hindcast data interactively.

> nedit Download_GEOS5V2_all.H.sh &
   FCST_SYR=1992  #1982
   FCST_EYR=1995  #2016
    for ((YEAR=$FCST_SYR; YEAR<=$FCST_EYR; YEAR++)); do
#     sbatch $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
      sh $SRCDIR/run_process_daily_forecasts.scr $YEAR $YEAR $iMon $SRCDIR $OUTDIR2 $FORCEDIR1 $MERRA2_GRIDDESC $CHIRPS_GRIDDESC ${ic[@]} $FORCEDIR2
    done
> Download_GEOS5V2_all.H.sh

2. Checked daily hindcast data:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Daily
> module load other/nco-4.6.8-gcc-5.3-sp3

(The 1992 through 1995 ones were actually checked on 20190611.)

> ncdiff 1992/jan01/ens4/jan01.19920930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1992/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.19920930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD 

> ncdiff 1993/jan01/ens4/jan01.19930930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1993/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.19930930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD

> ncdiff 1994/jan01/ens4/jan01.19940930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1994/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.19940930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD

> ncdiff 1995/jan01/ens4/jan01.19950930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1995/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.19950930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD

> ncdiff 1996/jan01/ens4/jan01.19960930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/1996/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.19960930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD 

> ncdiff 2004/jan01/ens4/jan01.20040930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/2004/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.20040930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD 

> ncdiff 2010/jan01/ens4/jan01.20100930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/2010/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.20100930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD

> ncdiff 2014/jan01/ens4/jan01.20140930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/2014/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.20140930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD

> ncdiff 2016/jan01/ens4/jan01.20160930_0900z.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Daily/2016/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.20160930_0900z.nc4 NEWmOLD
> ncview NEWmOLD &
> rm NEWmOLD  

For all of these, all the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old daily output, i.e. all zeros in NEWmOLD. Great! 

3. Checked 2014 monthly bias corrected hindcast data: 

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse 
> diff LWGAB.GEOS5v2.jan01_2014_2014.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/LWGAB.GEOS5v2.jan01_2014_2014.nc
Files LWGAB.GEOS5v2.jan01_2014_2014.nc and /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/LWGAB.GEOS5v2.jan01_2014_2014.nc differ

Why?

The climatology data were checked on 20190607 and they were identical between new and old. Double-checked:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST 
> diff LWS_fcst_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/FCST/LWS_fcst_clim.nc
Returned nothing => identical.
> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS 
> diff LWGAB_obs_clim.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/LWGAB_obs_clim.nc 
Returned nothing => identical.

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Monthly/2014/jan01/ens4
> ncdiff jan01.monthly.201409.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Monthly/2014/jan01/ens4/geosgcm_vis2d/jan01.geosgcm_vis2d.monthly.201409.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old files.

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/MERRA2/Monthly
> ncdiff all_forc_var_201410.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/MERRA2/Monthly/all_forc_var_201410.nc4 NEWmOLD
> ncview NEWmOLD &
All the fields in common "LWGAB", "PS", "SPEED", and "SWGDN" are identical between the new and old files. 

Checked my modifications to PART2_BCSD-Calc.H.sh, code_library/run_BCSD_calc.scr and code_library/Bias_correction_module.py but couldn't find anything wrong. 

Checked the output in a different way:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse 

> ncdiff LWGAB.GEOS5v2.jan01_2014_2014.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/LWGAB.GEOS5v2.jan01_2014_2014.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

> ncdiff PS.GEOS5v2.jan01_2014_2014.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/PS.GEOS5v2.jan01_2014_2014.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

> ncdiff SPEED.GEOS5v2.jan01_2014_2014.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/SPEED.GEOS5v2.jan01_2014_2014.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

> ncdiff SWGDN.GEOS5v2.jan01_2014_2014.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/SWGDN.GEOS5v2.jan01_2014_2014.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

4. For hindcast data bias correction: bias correct 2016 monthly hindcast data

> nedit PART2_BCSD-Calc.H.sh &
   FCST_SYR=2016
   FCST_EYR=2016

> PART2_BCSD-Calc.H.sh

5. For hindcast data bias correction: compute MERRA-2 6-hourly climatology

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> cp -p MERRA2_6hourly_CLIM.m scripts_old/.
> nedit MERRA2_6hourly_CLIM.m &

6. Plotted 1997-2016 climatology of Qair and Wind using ~/Catchment/CLM4.5/check_fire_bug_fix.gs for the Amazon point to see how they affect fire.

7. Met with Randy and Eunjee to update my work.

==========
20190611:

1. Checked 2016 monthly bias corrected hindcast data: 

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse 

> ncdiff LWGAB.GEOS5v2.jan01_2016_2016.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/LWGAB.GEOS5v2.jan01_2016_2016.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

> ncdiff PS.GEOS5v2.jan01_2016_2016.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/PS.GEOS5v2.jan01_2016_2016.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

> ncdiff SPEED.GEOS5v2.jan01_2016_2016.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/SPEED.GEOS5v2.jan01_2016_2016.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

> ncdiff SWGDN.GEOS5v2.jan01_2016_2016.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/Coarse/SWGDN.GEOS5v2.jan01_2016_2016.nc NEWmOLD
> ncview NEWmOLD & 
The min and max are 0 => the new and old files are identical. Great!
> rm NEWmOLD

2. For hindcast data bias correction: compute MERRA-2 6-hourly climatology (cont.)

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> nedit MERRA2_6hourly_CLIM.m & 

> module load tool/matlab-R2017a
> matlab -nodisplay < MERRA2_6hourly_CLIM.m 

Checked output:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/MERRA_2-6hrly
> module load other/comp/gcc-5.3-sp3  [Note: the old module "other/nco-4.6.8-gcc-5.3-sp3" no longer works!]

> ncdiff MERRA2_6hrly_CLIMO_001.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/MERRA_2-6hrly/MERRA2_6hrly_CLIMO_001.nc NEWmOLD
> ncview NEWmOLD & 
The variables in common "LWdown", "PARdffs", "PARdrct", "Psurf", "SWdown", "Snowf" and "Wind". Great!
> rm NEWmOLD

[Below is done on 20190614].
> ncdiff MERRA2_6hrly_CLIMO_365.nc /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/CLIM_DATA/OBS/MERRA_2-6hrly/MERRA2_6hrly_CLIMO_365.nc NEWmOLD
> ncview NEWmOLD & 
The variables in common "LWdown", "PARdffs", "PARdrct", "Psurf", "SWdown", "Snowf" and "Wind". Great!
> rm NEWmOLD

3. Checked and made sure that it's relative humidity, not humidity, that is used in the fire model in Catchment-CN4.5. 

4. For hindcast data bias correction: downscale 2014 bias corrected hindcast data from monthly to daily

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> cp -p PART3_TmpDisagg.H.sh scripts_old/.
> nedit PART3_TmpDisagg.H.sh &

> cp -p code_library/run_Temporal_disagg.scr scripts_old/.
> cp -p code_library/Daily_Temporal_disaggregation_module.py scripts_old/.
> cp -p code_library/Sub_Daily_Temporal_disaggregation_module.py scripts_old/.

==========
20190612:

1. For hindcast data bias correction: downscale 2014 bias corrected hindcast data from monthly to daily

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> nedit code_library/run_Temporal_disagg.scr & [didn't make any change]
> nedit code_library/Daily_Temporal_disaggregation_module.py &
> nedit code_library/Sub_Daily_Temporal_disaggregation_module.py

==========
20190613:

1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

2. Prepard meeting slides.

3. GMAO Theme meeting

4. Met with Randy and Eunjee

==========
20190614:

1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

2. Think about how to do the experiment for Huisheng

gs6101-land01:/land/fzeng> rm -rf merra2_old

On land01:

> cd /land/fzeng/projects/par_effect/point_driver_LDASsa_m3-15_2-CN

Need to confirm with Huisheng about which experiment(s).

3. Met with Randy and Eunjee 

4. For Eunjee's AGCM experiments:

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> cp -p GEOS_CatchCNGridComp.F90 GEOS_CatchCNGridComp.F90.restore
> nedit GEOS_CatchCNGridComp.F90 &
L245-251: no change
    call MAPL_GetResource ( MAPL, ATM_CO2,        Label="ATM_CO2:",DEFAULT=2, RC=STATUS)
    VERIFY_(STATUS)
    ! 0: uses a fix value defined by CO2
    ! 1: CT tracker monthly mean diurnal cycle
    ! 2: CT tracker monthly mean diurnal cycle scaled to match EEA global average CO2
    ! 3: spatially fixed interannually varyiing CMIP from getco2.F90 look up table (AGCM only)
    ! 4: import AGCM model CO2 (AGCM only)

==========
20190617:

1. GSFC Carbon workshop: 9-12:30

2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

3. Two experiments for Huisheng.

gs6101-land01:/land/fzeng/projects/par_effect/point_driver_LDASsa_m3-15_2-CN

(1)

> nedit process_cat.F90 &
> cp -p process_cat.F90 process_cat.F90.AODpt02.COD0
> gmake [to compile]
> mv clsm.x clsm_AODpt02_COD0.x
> chmod 755 clsm_AODpt02_COD0.x [it was -rwx------]
> gmake clean

> cp runit_AODpt05_COD0 runit_AODpt02_COD0
> nedit runit_AODpt02_COD0 &
> chmod 755 runit_AODpt02_COD0 [it was -rwx------]
> ./runit_AODpt02_COD0

Once done:
> cd /land/fzeng/projects/par_effect/point_driver_LDASsa_m3-15_2-CN/output
> nedit move &
> move

(2) 

> nedit process_cat.F90 &
> cp -p process_cat.F90 process_cat.F90.AODpt15.COD0
> gmake [to compile]
> mv clsm.x clsm_AODpt15_COD0.x
> chmod 755 clsm_AODpt15_COD0.x [it was -rwx------]
> gmake clean

> cp runit_AODpt05_COD0 runit_AODpt15_COD0
> nedit runit_AODpt15_COD0 &
> chmod 755 runit_AODpt15_COD0 [it was -rwx------]
> ./runit_AODpt15_COD0

Once done:
> cd /land/fzeng/projects/par_effect/point_driver_LDASsa_m3-15_2-CN/output
> nedit move &
> move

(3) Extract NPP data for Huisheng: 

> cd /land/fzeng/projects/par_effect/analysis
> cp point_AODpt05_COD0.ctl point_AODpt02_COD0.ctl
> nedit point_AODpt02_COD0.ctl &

> cp point_AODpt05_COD0.ctl point_AODpt15_COD0.ctl
> nedit point_AODpt15_COD0.ctl &

> nedit extract_var.gs &

==========
20190618:

1. Copied the experiment results from land01 to Discover for Huisheng.

discover06:/discover/nobackup/fzeng/Huisheng > scp -p gs6101-land01:/land/fzeng/projects/par_effect/analysis/npp_2000_2012_monthly_089694_4.txt .
Same for the other 4 files. 

Restored process_cat.F90:
gs6101-land01:/land/fzeng/projects/par_effect/point_driver_LDASsa_m3-15_2-CN> cp process_cat.F90.orig process_cat.F90 

2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

3. For Eunjee's AGCM experiments: (cont. from 20190614)

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> nedit GEOS_CatchCNGridComp.F90 &
L245-251: no change
    call MAPL_GetResource ( MAPL, ATM_CO2,        Label="ATM_CO2:",DEFAULT=2, RC=STATUS)
    VERIFY_(STATUS)
    ! 0: uses a fix value defined by CO2
    ! 1: CT tracker monthly mean diurnal cycle
    ! 2: CT tracker monthly mean diurnal cycle scaled to match EEA global average CO2
    ! 3: spatially fixed interannually varyiing CMIP from getco2.F90 look up table (AGCM only)
    ! 4: import AGCM model CO2 (AGCM only)
Added an option of ATM_CO2 == 5 to scale CT tracker monthly mean diurnal cycle to match global averaged CO2 defined by user. It doesn't affect the other options, so the updated GEOS_CatchCNGridComp.F90 can be used as the default file. Save a copy in case. 

> cp -p GEOS_CatchCNGridComp.F90 GEOS_CatchCNGridComp.F90.ATM_CO2_opt5

Build:

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN
> source $ESMADIR/src/g5_modules
> make -j 8 install

> cd ..
> mkdir -p exec/standard
> /bin/cp -pr Linux exec/standard/.

Set up:

> cd /discover/nobackup/fzeng/Catchment/M2n5P/run
> cp GEOSldas_CF90_test2.exec GEOSldas_CF90_2008CO2.exec
> cp GEOSldas_CF90_test2.bat GEOSldas_CF90_2008CO2.bat
> nedit GEOSldas_CF90_2008CO2.exec GEOSldas_CF90_2008CO2.bat &

GEOSldas_CF90_2008CO2.bat:

ntasks                  : 336

GEOSldas_CF90_2008CO2.exec:

EXP_ID                  : GEOSldas_CF90_2008CO2
EXP_DOMAIN              : CF0090x6C
NUM_ENSEMBLE            : 1
BEG_DATE                : 19810101 000000
END_DATE                : 20180101 000000

RESTART                  : 1
RESTART_ID              : e0004s_transientCO2_05
RESTART_PATH            : /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09
RESTART_DOMAIN          : SMAP_EASEv2_M09_GLOBAL

MET_TAG                 : M2COR_cross__precCPCUGPCP22clim_MERRA2_BMTXS
MET_PATH                : /discover/nobackup/projects/gmao/merra/iau/merra_land/MERRA2_land_forcing/
FORCE_DTSTEP            : 3600

BCS_PATH                : /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/

LSM_CHOICE                     : 2
DTCN                           : 5400
ATM_CO2: 5
SCALE_ALBFPAR: 0

JOB_SGMT                       : 00010000 000000
NUM_SGMT                       : 37
MONTHLY_OUTPUT                : 2
MET_HINTERP                    : 1

> cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/exec/standard/Linux/bin/
> ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_2008CO2.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_2008CO2.bat

37
13505
2017 12 23
13505
2054 12 14
creating dir structure
creating restart and bc

Correct the tile file if it is an old EASE tile format... 

cmd:   ./preprocess_ldas.x correctease  /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C/rc_out//CF0090x6C_DE0360xPE0180-Pfafstetter.til /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C/rc_out//MAPL_CF0090x6C_DE0360xPE0180-Pfafstetter.til
Creating f2g.txt....

cmd:   ./preprocess_ldas.x c_f2g /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C/rc_out//CF0090x6C_DE0360xPE0180-Pfafstetter.til LDAS_domain_def.nml /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/clsm/catchment.def GEOSldas_CF90_2008CO2 198101010000
 Creating domain..., reading white and black lists if there have ones...
 Finish domain setup...
 Writing catparam file : /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_
 2008CO2/output/CF0090x6C/rc_out///Y1981/M01/GEOSldas_CF90_2008CO2.ldas_catparam
 .19810101_0000z.bin
linking bcs...
Creating and lining restart...
cmd:   ./process_rst.csh  g0620 GEOSldas_CF90_2008CO2 /discover/nobackup/fzeng/Catchment/M2n5P  /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/ CF0090x6C_DE0360xPE0180-Pfafstetter.til 2  1 19810101  e0004s_transientCO2_05 SMAP_EASEv2_M09_GLOBAL /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ 1  0
Please hold on for a while until the restart file is created  .....
restart: 1
catcRstFile1: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y1981/M01/e0004s_transientCO2_05.catchcn_internal_rst.19810101_0000
catcRstFile: /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_2008CO2.catchcn_internal_rst.19810101_0000
Updating restart path...
creating RC Files
Optimizing... decomposition of processes.... 

cmd:   ./preprocess_ldas.x optimize /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/input/tile.data 336
 total tiles      475330

['/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc/GEOSldas_CAP.rc', '/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc/GEOSldas_ExtData.rc', '/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc/GEOSldas_HIST.rc', '/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc/GEOSldas_LDAS.rc']
CAP.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/run
ExtData.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/run
HIST.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/run
PE90x540-CF
LDAS.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/run
creating gcm style batch Run scripts lenkf.j

Experiment directory: /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2

creating batch Run scripts

> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/run
> nedit lenkf.j & [no change]

Check restart file:
> cat cap_restart 
19810101 000000
> ls -l ../input/restart/
total 0
lrwxrwxrwx 1 fzeng g0620 157 2019-06-18 12:39 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_2008CO2.catchcn_internal_rst.19810101_0000
lrwxrwxrwx 1 fzeng g0620 132 2019-06-18 12:39 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_2008CO2.vegdyn_internal_rst

Check executable:
> ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50813990 2019-06-18 13:08 ../build/Linux/bin/GEOSldas.x*

CAP.rc: 
check the END_DATE, JOB_SGMT and NUM_SGMT. No change. 
MAPL_ENABLE_TIMERS: NO

HISTORY.rc: 

  tavg1_2D_lfs_Nx.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,
  Added to L236: 'CNCO2'    , 'CATCHCN'  ,

LDAS.rc: 
added "ATM_CO2_frac:                       385.8387e-6" to the end (L49).

Did an interactive run:

> interactive.py -A sp3 -n 336 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_2008CO2/run

> cp lenkf.j lenkf.j.orig
> nedit lenkf.j &
    Add "exit" after L278 "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x"
  
> ./lenkf.j

It's running. Stopped it at 1981-01-01T07:00:00.

> cp lenkf.j.orig lenkf.j

> ./ldas_batchrun.j 

4. Met with Lesley and Eunjee

5. Randy's webinar

6. For hindcast data bias correction: downscale 2014 bias corrected hindcast data from monthly to daily, and from daily to 6-hourly

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> nedit code_library/Sub_Daily_Temporal_disaggregation_module.py & 
> PART3_TmpDisagg.H.sh 

==========
20190619:

1. For hindcast data bias correction: downscale 2014 bias corrected hindcast data from monthly to daily, and from daily to 6-hourly

There is a bug in PART3_TmpDisagg.H.sh so the jobs failed. "UNIT" should be "UNITS". Fixed the bug and rerun.

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> nedit PART3_TmpDisagg.H.sh & 
> PART3_TmpDisagg.H.sh

It's finished. 

Checked the output:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1
> module load other/nco-4.6.8-gcc-5.3-sp3  [Note: this old module works, while the new one "other/comp/gcc-5.3-sp3" doesn't work!] 

> ncdiff PARDF.201401.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1/PARDF.201401.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff PARDR.201401.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1/PARDR.201401.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff LWGAB.201401.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1/LWGAB.201401.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff SWGDN.201409.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1/SWGDN.201409.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff PS.201409.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1/PS.201409.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff SPEED.201409.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2014/ens1/SPEED.201409.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

2. The GEOSldas_CF90_2008CO2 run: 

The first job segment ran from 19810101 to 19830815 and stopped at 1983-08-15T14:00:00 due to time limit.
The second job segment started from 19830101. This is correct! Good.

Checked the CO2 using ~/Catchment/CLM4/check_GEOSldas_CF90_2008CO2.gs.

3. For hindcast data bias correction: downscale 2016 bias corrected hindcast data from monthly to daily, and from daily to 6-hourly

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> nedit PART3_TmpDisagg.H.sh & 
   FCST_SYR=2016            # SYR and EYR should be the same as 
   FCST_EYR=2016            # the monthly BCSD files with start and end years
> PART3_TmpDisagg.H.sh 

==========
20190620:

1. For hindcast data bias correction: downscale 2016 bias corrected hindcast data from monthly to daily, and from daily to 6-hourly (cont.)

It's finished. 

Checked the output:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1
> module load other/nco-4.6.8-gcc-5.3-sp3  [Note: this old module works, while the new one "other/comp/gcc-5.3-sp3" doesn't work!] 

> ncdiff PARDF.201601.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1/PARDF.201601.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff PARDR.201601.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1/PARDR.201601.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff LWGAB.201601.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1/LWGAB.201601.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff SWGDN.201609.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1/SWGDN.201609.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff PS.201609.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1/PS.201609.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> ncdiff SPEED.201609.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_DATA/6-hourly/2016/ens1/SPEED.201609.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference are both 0. => The data are identical between the two files. Great!
> rm NEWmOLD

2. For hindcast data bias correction: check the 6hrly BC hindcast TSOIL2 and TSOIL3

Created and used these scripts below to check the 6hrly BC hindcast TSOIL2 and TSOIL3:
check_BC_hindcast_6hrly_var.py  
check_BC_hindcast_daily_var.py  
check_MERRA2_6hrly_clim.py  
check_raw_hindcast_daily_var.py  
extract_MERRA2_var_hourly.py

Looks correct to me.

3. For hindcast data bias correction: combine the individual fields into one file for each month

> cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 
> nedit PART4_Combine.H.sh &

> cp -p code_library/run_Combining.scr scripts_old/.
> nedit code_library/run_Combining.scr &

> cp -p code_library/Combine_Sub_Daily_downscaled_forcings.py scripts_old/.
> nedit code_library/Combine_Sub_Daily_downscaled_forcings.py &

> PART4_Combine.H.sh

Checked the output:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/2014/jan01/ens1
> module load other/nco-4.6.8-gcc-5.3-sp3

> ncdiff GEOS5.201401.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/2014/jan01/ens1/GEOS5.201401.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference in LWGAB, PARDF, PARDR, PS, SPEED, SWGDN are all 0. => The data are identical between the two files. Great!
> rm NEWmOLD

> cd ../../../2016/jan01/ens4
> ncdiff GEOS5.201609.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/2016/jan01/ens4/GEOS5.201609.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference in LWGAB, PARDF, PARDR, PS, SPEED, SWGDN are all 0. => The data are identical between the two files. Great!
> rm NEWmOLD
> ncdiff GEOS5.201601.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/2016/jan01/ens4/GEOS5.201601.nc4 NEWmOLD
> ncview NEWmOLD &
Min and max of the difference in LWGAB, PARDF, PARDR, PS, SPEED, SWGDN are all 0. => The data are identical between the two files. Great!
> rm NEWmOLD

4. Met with Huisheng

5. Prepared slides for meeting tomorrow.

==========
20190621:

1. Read Fang Li's papers.

2. Met with Randy, Melanie and Eunjee.

3. Following discussion in the meeting, made time series plots using ~/Catchment/CLM4.5/check_fire_bug_fix.gs. 

Created ~/python/clm4-to-clm4.5/tile2grid_poros.py to extract tile-space porosity data and mapped to grid-space.

fuelc(c) = totlitc(c)+totvegc_col(c)-rootc_col(c)-fuelc_crop(c)*cropf_col(c) 

==========
20190624:

1. Met with Randy, Eunjee and Lesley.

2. Create a Fortran script for Huisheng: /home/fzeng/Catchment/tile2grid_ndep_CF90.f90

This script is not a working version since I don't know the required netCDF libraries to compile the code, but it shows how to map from tile to grid which Huisheng can refer to in her IDS script.

3. For IDS project:

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > ls e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2014/M01
e0004s_transientCO2.ens0000.catchcn_ldas_rst.20140101_0000z

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > ls e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2016/M01
e0004s_transientCO2.ens0000.catchcn_ldas_rst.20160101_0000z

We have restarts for both 20140101 and 20160101. Good.

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp

Modified clsm_ensdrv_force_routines.F90 to read the bias-corrected hindcast data.

==========
20190625:

1. For IDS project: modified the offline code for the hindcast experiments

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp

These files clsm_ensdrv_glob_param.F90, clsm_ensdrv_init_routines.F90, clsm_ensdrv_out_routines.F90, compute_rc.F90 and process_cn.F90 were modified for the e0004s_transientCO2_3hrlyOUT experiment. Checked and made sure they are already restored.

> mkdir e0004s_transientCO2_BChindcast
> cd e0004s_transientCO2
> cp * ../e0004s_transientCO2_BChindcast/.  [use the same modifications as for e0004s_transientCO2)

> cd ../e0004s_transientCO2_BChindcast
> cp -p ../clsm_ensdrv_force_routines.F90 .  [clsm_ensdrv_force_routines.F90 already modified to read BC hindcast data)
> cp clsm_ensdrv_force_routines.F90 ../.
> cp clsm_ensdrv_out_routines.F90 ../.
> cp process_cn.F90 ../.
> cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

> setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
> source $ESMADIR/src/g5_modules 

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp
> gmake install

> cd ../../../GEOSlana_GridComp/
> gmake install

> cd ../../Applications/LDAS_App/
> gmake install

> cd ../../..
> mkdir -p exec/e0004s_transientCO2_BChindcast
> /bin/cp -pr Linux exec/e0004s_transientCO2_BChindcast/.

2. For IDS project: set up the 8 hindcast experiments, take e0004s_transientCO2_BChindcast_2014Jan01_ens1 as an example.

(1) Prepare files and run directory for experiment setup: 

> cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
> xdiff  M09_CN_e0004s_transientCO2.exe ../e0004s_transientCO2_05/run/. & [Identical. Good!]
> xdiff M09_e0004s_transientCO2.bat ../e0004s_transientCO2_05/run/. & [Identical. Good!]
> cp M09_e0004s_transientCO2.bat M09_e0004s_transientCO2_BChindcast.bat
> cp M09_CN_e0004s_transientCO2.exe M09_CN_e0004s_transientCO2_BChindcast.exe
> nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat &

M09_CN_e0004s_transientCO2_BChindcast.exe:

exp_id                  = e0004s_transientCO2_BChindcast_2014Jan01_ens1
exp_domain              = SMAP_EASEv2_M09_GLOBAL
N_ens                   = 1
start_time              = 2014-01-01-00-00-00
end_time                = 2014-10-01-00-00-00

force_dtstep            = 21600
restart_path            = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/
restart_domain          = SMAP_EASEv2_M09_GLOBAL
restart_id              = e0004s_transientCO2
met_tag                 = HindcastBC_netcdf
met_path                = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/

M09_e0004s_transientCO2_BChindcast.bat:

job-name                = BChindcast_2014_ens1

> cp ../e0004s_transientCO2_05/run/driver_inputs_M09_GLOBAL.nml driver_inputs_M09_GLOBAL.nml
> nedit driver_inputs_M09_GLOBAL.nml & [No change made!] 

(2) Run ldsetup:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin
> ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN

(3) Check the executable, restart file and create year_co2.txt:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2014Jan01_ens1
> ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
-rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x*
> ls -l input/restart/ (to make sure the restart file is the right one)
lrwxrwxrwx 1 fzeng s1460 80 2019-06-25 11:20 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/

(4) Create year_co2.txt:

For 2014 ones:
> cd run
> echo 2014 > year_co2.txt
> cat year_co2.txt (to make sure the year of CO2 is correct)
    2014 [Correct!]

For 2016 ones:
> cd run    
> echo 2016 > year_co2.txt
> cat year_co2.txt (to make sure the year of CO2 is correct)
    2016 [Correct!]    

(5) Modify the restart_id in the first job script:

> nedit lenkf.0.j & [No change made!]
 -restart_path ../input/restart/output \
 -restart_domain SMAP_EASEv2_M09_GLOBAL \
 -restart_id e0004s_transientCO2 \

(6) Create LDAS.rc:

> nedit LDAS.rc &
INIT_MON:                           jan01
ENS_NUM:                            1
NOTE: The space is important, so just copy and paste from /discover/nobackup/fzeng/Catchment/BChindcast/2014Jan01_ens1/run/LDAS.rc.

(7) Did an interactive run to make sure it works:

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2014Jan01_ens1/run
> ./lenkf.0.j

It's running and using the right meteorology data. 

 get_forcing(): bias-corrected GEOS-5 hindcast forcing data set
 opening../input/met_forcing/6-hourly//2014/jan01/ens1/GEOS5.201401.nc4

Stopped it at date_time_new 20140101_060000z.

(8) Submit the job:

> qsub lenkf.0.j

3. Helped Zhuo Wang to plot 21600x43200 (dlat = 0.00833333 and dlon = 0.00833333) recharge data: created /home/fzeng/Catchment/plot_recharge_forZhuoWang.gs. 

I was able to plot the data on GrADS at its native resolution. It took a while but the plot did show up globally after a while. However, it failed to print the image to the *.gx file. I got these error messages.
 
Out of buffer space
 
Error status on print metafile is positive.
Cannot print current frame.
 
I think she needs to regrid the data to a coarser resolution before she can plot it.

4. Sent Lesley and Eunjee an email updating them about the IDS work I have done recently.

5. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.

==========
20190626:

1. For IDS project:

The simulations stopped at 20140930_180730z:

 date_time_new 20140930_180730z
 get_forcing(): bias-corrected GEOS-5 hindcast forcing data set
 opening../input/met_forcing/6-hourly//2014/jan01/ens1/GEOS5.201410.nc4
 LDAS ERROR (3000) from get_BC_GEOS_hindcast: error opening netcdf file

This is different from GEOSldas. 

Checked the old runs using Princeton forcing and found the same behavior, i.e. for the last 3 hours it reads the first 3 hours of next month/year probably for interpolation purpose. 

In /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App/cnlsm_ensdrv_main.F90:

L1489-1513:

     ! initialize forcing
     
     if (master_proc)  then
        
        ! allocate full domain structures           
        
        allocate(met_force_obs_tile_new_f(N_catf))
        
        ! get initial forcing (note input date_time "force_time_old")
        ! check whether SWnet is available if needed (alb_from_SWnet)

        call get_forcing( force_time_old, force_dtstep,                        &
             met_path, met_tag, N_catf, tile_coord_f, met_hinterp,             &
             met_force_obs_tile_new_f, move_met_force_obs_new_to_old,          &
             .true., alb_from_SWnet)
        
        ! NOTE: For NON-GEOS-5 forcing, the "flux" fields in "met_force_obs_tile_new_f"
        !        are from the forcing interval STARTING at "force_time_old". (convention)
        !       For GEOS-5 forcing,     the "flux" fields in "met_force_obs_tile_new_f"
        !        are from the forcing interval ENDING   at "force_time_old".
        !       This is addressed below when data are read for "force_time_new" within 
        !        the forcing loop.
        ! - reichle, 23 Feb 2016
        
     end if

L1821-1837:

        ! forcing
        
        if (datetime_eq_refdatetime( date_time_old, force_time_new)) then
           
           ! read "new" forcing at end of first forcing time step
           
           force_time_old         = force_time_new
           met_force_obs_tile_old = met_force_obs_tile_new
           
           call augment_date_time( force_dtstep, force_time_new )
           
           if (master_proc)  allocate(met_force_obs_tile_new_f(N_catf))
           
           if (master_proc)                                                        &
                call get_forcing( force_time_new, force_dtstep, met_path, met_tag, &
                N_catf, tile_coord_f, met_hinterp,                                 &
                met_force_obs_tile_new_f, move_met_force_obs_new_to_old )
     
Found that when I used GEOSldas_m4-17_6 to do the experiments, I also modified (with Weiyuan's help) GEOS_MetforceGridComp.F90 for the bias-corrected hindcast data so that it's not reading the forcing data of the next forcing dtstep. 

Try to make the same modifications to LDASsa's cnlsm_ensdrv_main.F90:

> cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App
> cp -p cnlsm_ensdrv_main.F90 cnlsm_ensdrv_main.F90.restore
> nedit cnlsm_ensdrv_main.F90 &

> setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
> source $ESMADIR/src/g5_modules 
> gmake install

> cd ../../..
> /bin/cp -pr Linux exec/e0004s_transientCO2_BChindcast/.

Check the runs:

> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2014Jan01_ens1
> ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
-rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x*
> ls -l input/restart/ (to make sure the restart file is the right one)
lrwxrwxrwx 1 fzeng s1460 80 2019-06-25 11:20 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/

> cd run
> cat year_co2.txt
2014  [correct!]
> cat LDAS.rc 
INIT_MON:                           jan01
ENS_NUM:                            1
[correct!]

> /bin/rm co2_scalar_ts.txt  EEAco2g_ts.txt lenkf_job_completed.txt year_ts.txt

> interactive.py -A sp3 -n 140 -a g0620 -X --debug
> cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2014Jan01_ens1/run
> ./lenkf.0.j

 date_time_new 20140131_180730z
 get_forcing(): bias-corrected GEOS-5 hindcast forcing data set
 opening../input/met_forcing/6-hourly//2014/jan01/ens1/GEOS5.201401.nc4

This is what I want. Good. Stopped the run.

> qsub lenkf.0.j

Repeat these steps except the interactive run steps for the other 7 simulations.

2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 

==========
20190627:

1. For IDS project:

The 8 simulations are finished. Created ~/Catchment/SMAP_M09/tile2grid_ease_BChindcast_monthly.f90 to process the output.

~/Catchment/SMAP_M09 > ifort -o tile2grid_ease_BChindcast_monthly tile2grid_ease_BChindcast_monthly.f90
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2014Jan01_ens1 2014
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2014Jan01_ens2 2014
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2014Jan01_ens3 2014
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2014Jan01_ens4 2014

> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2016Jan01_ens1 2016
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2016Jan01_ens2 2016
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2016Jan01_ens3 2016
> tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2016Jan01_ens4 2016

Created e0004s_transientCO2_BChindcast_monthly_2014.ctl and e0004s_transientCO2_BChindcast_monthly_2016.ctl and checked on GrADS.

Emailed Eunjee about the output files.

2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 

==========
20190628:

1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.