Using allinea:
--------------

Take the offline as an example. 

Need to get an interactive node first? No necessary. Can do this on one single dali/discover node as well. See below.

(1) Compile with BOPT=g:

If a routine in GEOSlana_GridComp is modified, under GEOSlana_GridComp and then Applications/LDAS_App:

gmake clean  [This is very important!!]
gmake install BOPT=g (NOTE: not BOPT=-g)

(2) If run ddt on one single dali/discover node: (NOTE, don't use more than 6 processors)

cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/m0001hdeg_clm4.5/run
/bin/rm ../output/global/rc_out/*
module load tool/allinea-tools-7.0.4
ddt /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/m0001hdeg_clm4.5/build/Linux/bin/LDASsaCN_mpi.x &
(Note: the order is important. If do "module load tool/allinea-tools-6.1.0" before "source ../build/Linux/bin/g5_modules", the latter will unload the module.)
Copy the part of lenkf.1.i from work_path to the end (no need to be in one line) to arguments.
Copy this part of lenkf.1.i to Environment variables:
~~~~~~~~~~~~~~~~~~~~~~
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
setenv MYNAME `finger $USER | cut -d':' -f3 | head -1`
~~~~~~~~~~~~~~~~~~~~~~
click "Run"
click the play button on the upper left to make it run until it stops.


(3) If need more processors, use SLURM: 

logging into dali with "-XY" (This is important!)
ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug (or interactive.py -A sp3 -n 96 -a g0620 -X -t 03:00)
cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/m0001hdeg_clm4.5/run
/bin/rm ../output/global/rc_out/*
limit stacksize unlimited
source ../build/Linux/bin/g5_modules
module load tool/allinea-tools-7.0.4
ddt /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/m0001hdeg_clm4.5/build/Linux/bin/LDASsaCN_mpi.x &
(Note: the order is important. If do "module load tool/allinea-tools-6.1.0" before "source ../build/Linux/bin/g5_modules", the latter will unload the module.)
Copy the part of lenkf.1.i from work_path to the end (no need to be in one line) to arguments.
Copy this part of lenkf.1.i to Environment variables:
~~~~~~~~~~~~~~~~~~~~~~
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
setenv MYNAME `finger $USER | cut -d':' -f3 | head -1`
~~~~~~~~~~~~~~~~~~~~~~
click "Run"
click the play button on the upper left to make it run until it stops.

Another example, debug mk_LDASsaRestarts:

interactive.py -A sp3 -n 28 -a g0620 -X --debug 
cd to the run directory (Catchment/SMAP_EASEv2_M09/restarts)
module load tool/allinea-tools-6.0.4
limit stacksize unlimited
ddt bin/mk_LDASsaRestarts &
Put "M09 50" in Argument



================================================================================================================================
Using totalview:
----------------

Debug conus_force_loop.F90 on totalview.

First, added "-g" to "GNUmakefile" and then compiled the point driver. 

gs6101-land01:~/point_driver_LDASsa/projects/dtcn/point_driver_LDASsa_m3-15_2-CN> totalview ./clsm.x & (totalview is already available on land01, so no need to load module)
In "Arguments", input "tileID 1.0 1.0"

Click "Step" to look into subroutines.
Highlight a variable, then click "View"->"Dive" to find the value of the variable.

More notes in "totalview" and "sarith" in /discover/nobackup/fzeng/notes.