========== 20171201: 1. Met with Randy and Eunjee, and continued working on the slides. 2. Finished and turned in Python HW3. ========== 20171204: 1. Updated Higgins rain data through Dec 03. 2. November report. 3. Continued working on the presentation slides. 4. Made plots for Eunjee's AGU poster. 5. ACMAP meeting (Huisheng's aerosol-cloud-vegetation study). ========== 20171205: 1. Processed December forecast. 2. Python class. 3. Commented on Eunjee's AGU poster. 4. Continued working on the presentation slides. ========== 20171206: 1. Practised the presentation. 2. IDS meeting. 3. Checked with Randy about the soil layers with different temperature in Catchment-CN and sent the info to Lesley and Lei. ========== 20171207: 1. Gave the presentation at the GMAO Science Theme Meeting. 2. Read these scripts below to understand the procedure for the SMAP EASEv2 M09 runs. ~/Catchment/SMAP_M09/process-smap ~/Catchment/SMAP.gs ~/Catchment/wet2_percentile_SMAP.f90 ========== 20171208: 1. For the SMAP EASEv2 M09 run: (1) Remember to change iyrb, iyre and ntsize in wet2_percentile_SMAP.f90 process-smap and SMAP.gs needs to be updated too. (2) Forcing: -corrected precip: /discover/nobackup/projects/gmao/merra/iau/merra_land/MERRA2_land_forcing/precip_corr_CPCUGPCP22clim_MERRA2_BMTXS/MERRA2_400/diag -other forcing: /discover/nobackup/projects/gmao/merra/iau/merra_land/MERRA2_land_forcing/MERRA2_400/diag (3) SMAP data: # check SMAP data # path changed due to SMAP data reprocessing in Oct 2015 # ---------------------------------------------------- # set sdat=`ls -1AR /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/V10002/gph/*/*/*/*.h5 | tail -1` # new path to SMAP data from 20150331 to 20151025 # make sure the path in /home/fzeng/Catchment/SMAP.ctl is the same # ---------------------------------------------------------------- #set sdat=`ls -1AR /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vb1010/gph/*/*/*/*.h5 | tail -1` # new path to SMAP data from 20151026 to 20160421 # make sure the path in /home/fzeng/Catchment/SMAP.ctl is the same # ---------------------------------------------------------------- #set sdat=`ls -1AR /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vb1012/gph/*/*/*/*.h5 | tail -1` # new path to SMAP data from 20160422 20160614 # make sure the path in /home/fzeng/Catchment/SMAP.ctl is the same # ---------------------------------------------------------------- set sdat=`ls -1AR /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vv2010/gph/*/*/*/*.h5 | tail -1` /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vv2010/gph: 20150331 to 20170714 /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vv3030/gph: 20150331 to 20171205 Which one should I use? /discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vv3030/gph? Emailed Qing. She said YES. (4) Greg said in his two emails back in June last year that "I shifted file names by one day, so that data written out at 00z 2 Jan 2001 would be renamed 1 Jan 2001 (since the data written out at 00z for day N was actually the soil moisture average for day N-1)." Don't quite understand. Asked Sarith whether "e0004s_wet2.ens_avg.ldas_tile_daily_out.19810102.bin" would be the daily mean of this date 19810102, instead of 19810101. He said YES. Maybe what Greg said is for the old driver LDASsa_m3-15_2-CN he used for setting up his SMAP runs? Extracted Greg's e0004s_27 from archive but the original output has been deleted and I can only see the monthly post-processed output such as e0004_200607.gdat. Extracted Greg's rzmc_2014.tar from archive. They are daily files from rzmc_20140101.gdat throught rzmc_20141231.gdat. They might be the shifted ones. I can't tell what the original daily files would look like. Don't worry about this, since I don't need to do so for the output from Sarith's new driver. But then Sarith replied: "I think output file names of SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3, LDASsa_m3-16_0_p2, and LDASsa_m3-15_2-CN. are all the same Greg must be correct (I just trust his words, he must have studied the codes), would you be able to dig and figure out (I'll have to dig myself otherwise, I can't say anything offhand) or ask Rolf to be sure" Asked to Rolf. Rolf helped me check his m3-16 code and he confirmed that the output with "19810101" in the filename is for the daily mean of 19810101. I still need to check the m3-16 code I use because Rolf is not sure if Sarith would have made any changes when he consolidated the code. Checked /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/clsm_ensdrv_out_routines.F90 subroutine output_calcs (L839-863) in clsm_ensdrv_out_routines.F90: case ('d') ! daily n_steps = real(86400/model_dtstep) if (out_choice%daily%tile) then out_tile = .true. file_tag = 'ldas_tile_daily_out' fname_tile = get_io_filename( work_path, exp_id, file_tag, & date_time=date_time, option=4, ens_id=ens_id_tmp ) end if if (out_choice%daily%grid) then out_grid = .true. file_tag = 'ldas_grid_daily_out' fname_grid = get_io_filename( work_path, exp_id, file_tag, & date_time=date_time, option=4, ens_id=ens_id_tmp ) end if function get_io_filename in clsm_ensdrv_functions.F90: L51-55: ! NOTE: option=1: return "io_path/dir_name/[ensXXXX]/*.ext" ! option=2: return "io_path/dir_name/[ensXXXX]/Yyyyy/Mmm/*.YYYYMM.ext" ! option=3: return "io_path/dir_name/[ensXXXX]/Yyyyy/Mmm/*.YYYYpPP.ext" ! option=4: return "io_path/dir_name/[ensXXXX]/Yyyyy/Mmm/*.YYYYMMDD.ext" ! option=5: return "io_path/dir_name/[ensXXXX]/Yyyyy/Mmm/*.YYYYMMDD_HHMMz.ext" L121-126: if (present(date_time)) then write (YYYY,'(i4.4)') date_time%year write (MMDD,'(i4.4)') date_time%month*100 + date_time%day write (HHMM,'(i4.4)') date_time%hour*100 + date_time%min write (PP, '(i2.2)') date_time%pentad L147-149: elseif (tmp_option==4) then date_time_string = '.' // YYYY // MMDD cnlsm_ensdrv_main.F90 (L2579-2589): if (out_time%daily .and. out_ensavg%daily%any) then ! write daily output call output_calcs( 'out', out_collection_ID, N_catl, & cat_progn_ensavg, cat_diagS_ensavg, cat_diagF_ensavg, & met_force_ensavg_ntp, veg_param_ntp, bal_diagn_ensavg, & cat_progn_daily, cat_diagS_daily, cat_diagF_daily, & met_force_daily, veg_param_daily, bal_diagn_daily, & cat_param, mwRTM_param, tile_coord_l, & out_ensavg, date_time_old, work_path, exp_id, 'd', model_dtstep, & out_tile, out_grid, fname_tile, fname_grid, tile_data_tavg_l, & cn_diagn_avg = cn_diagn_daily) They are the same as Rolf's, at least for the parts we just looked at together. So yes, e0004s_wet2.ens_avg.ldas_tile_daily_out.19810101.bin is the daily mean of 19810101, and e0004s_wet2.ens_avg.ldas_tile_daily_out.19810102.bin is the daily mean of 19810102, etc. 2. Recovered my merra2/m0008hdeg_20 and merra2/m0008hdeg_19. They are from 200701 to 201512. They used MERRA-2 forcing, see lenkf.j: /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_73q_e0007s_old_daily.x \ -met_tag M2COR_cross \ -met_path /discover/nobackup/projects/gmao/merra/iau/merra_land/MERRA2_land_forcing/ \ /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/merra2_m0008hdeg > ls -l process_cat.F90 -rwxr-xr-x 1 fzeng g0620 94946 2016-06-10 10:45 process_cat.F90* real, parameter :: co2v = 390.e-6 ! atmospheric carbon dioxide concentration So co2 is 390ppm. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/merra2_m0008hdeg > ls -l total 64 -rwxr-xr-x 1 fzeng g0620 60010 2016-06-21 10:05 CN_DriverMod.F90* ! sndz calculated from subroutine snowrt in StieglitzSnow.F90 ! is relative to the area covered by snow only. Multiply it by asnow (fraction ! of area covered by snow, calculated in subroutine StieglitzSnow_calc_asnow in ! process_cat.F90) here so that snowdp=sndz*asnow is representative of the entire catchment area. ! Fanwei Zeng and Randy Koster, June 2016 ! snowdp(n) = sndzn(nc) ! snow depth (m) snowdp(n) = sndzn(nc)*asnow(nc) ! snow depth (m) So snowdp is corrected? The executable no longer exists!! /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_20: -rw-r--r-- 1 fzeng g0620 71084848 2016-03-29 09:21 m0008_201512.gdat The m0008hdeg_20 should be before snowdp correction! The 21 cycle of m0008hdeg was used to investigate what time steps are good enough, from which I found the bug in snowdp and fixed it after that, so the "2016-06-21 10:05 CN_DriverMod.F90" was after m0008hdeg_20. ========== 20171211: 1. Continued to collect information about merra2/m0008hdeg_20 and p0007s_67 to see if they can be used to compare for understanding the impact of meteorology difference on GPP difference. /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp > ls -l CNDecompMod.F90 -rw-r--r-- 1 fzeng g0620 26362 2016-04-15 14:08 CNDecompMod.F90 Since the merra2/m0008hdeg_20 was done in March 2016, so it should be before the Eunjee's modification on HR. /discover/nobackup/fzeng/Catchment/princeton/p0007s_67 > ls -l -rw-r--r-- 1 fzeng g0620 3708984 2015-11-09 10:31 p0007_201212.gdat So p0007s_67 was before HRv2 and snowdp correction. 2. Created ~/Catchment/M2n5P_m0001/princetonVSmerra2/compare_princeton_merra2.m and the functions it calls to compute the difference between Princeton and MERRA2. Created ~/Catchment/matlab/m_map/plot_princeton_merra2_diff.m to make difference plots. ========== 20171212: 1. Made abs and diff maps to compare the forcing and carbon fluxes between p0007s_67 and merra2/m0008hdeg_20. 2. Created ~/Catchment/M2n5P_m0001/princetonVSmerra2/compare_ts.m to compute and plot annual GPP and NEE ts of p0007s_67 and merra2/m0008hdeg_20. 3. Jana needs a list of parameters for calibration so that modeled FPAR agrees with MODIS FPAR better. Randy said Greg once tuned the amount of carbon that does not display to improve FPAR. Searched Greg's notes in /home/gkwalker/notes to see what Greg tuned to improve FPAR earlier. It's about the amount of carbon that does not display. Greg's note below may be helpful to understand the high CLM4.5 FPAR in the high latitudes: 201403:got FPAR and FWET diagnostics working in ccsm4; we need to supply appropriate FWET to compute_rc- our FPAR for the winter boreal forest belt is too high, and it is because OMEGA (scattering coefficient) is too low; it's not taking snow on the canopy into account. 20121205: CN mods: fcur=0.5; half of new carbon goes to current display PSIWILT changed from -2 to WPWET (MPa) CRIT_ONSET_FDD changed from 7 to 15 to help prevent frequent shutdown changes to seasonal/moisture stress deciduous; function of latitude unsure if a leap year problems exists Is this what we are looking for? Check the code. This parameter "fcur" may be the one that Greg tuned to improve FPAR. Its definition is "allocation parameter: fraction of allocation that goes to currently displayed growth, remainder to storage". You can find it in subroutine CN_init in GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/CN_DriverMod.F90. The relevant lines of code are: pftcon%fcur = fcux ! allocation parameter: fraction of allocation that goes to currently displayed growth, remainder to storage !!data fcux / 0., 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0/ !! -- original values data fcux / 0., 1.0, 1.0, 0.5, 1.0, 1.0, 0.5, 0.5, 0.5, 1.0, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5/ !! -- values being used ========== 20171213: 1. For Jana's calibration work, checked catch_types.F90 for what the cn parameters are: type cn_param_type real, dimension (NUM_VEG) :: ITY ! vegetation_type real, dimension (NUM_VEG) :: FVG ! vegetation_fraction real :: TILE_ID ! catchment_tile_id real :: NDEP ! CLM_nitrogen_deposition [g m-2 s-1] real :: CLI_T2M ! Annual mean T [K](MERRA2 or princeton) real :: BGALBVR ! MODIS soil background albedo visible direct real :: BGALBVF ! MODIS soil background albedo visible diffused real :: BGALBNR ! MODIS soil background albedo near-infrared direct real :: BGALBNF ! MODIS soil background albedo near-infrared diffused end type cn_param_type BGALBVR, BGALBVF, BGALBNR and BGALBNF are used in process_cn.F90 to compute albedo. Need more research to understand if and how this would affect FPAR. CLI_T2M is an input argument in CN_Driver. It's used in CNPhenology to compute crit_onset_gdd. Subroutine CNAnnualUpdate in CNAnnualUpdateMod.F90 computes annual average 2m air temperature, but it's not called (commented out in CNEcosystemDyn). NDEP is an global input data set from CLM. It doesn't affect FPAR directly. It may affect FPAR throught GPP and LAI. 2. Continued to learn how to run the original CLM4.5. This is to investigate why CLM4.5 GPP is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171214: 1. Continued to learn how to run the original CLM4.5. This is to investigate why CLM4.5 GPP is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171215: 1. Continued to learn how to run the original CLM4.5. This is to investigate why CLM4.5 GPP is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171218: 1. Updated Higgins rain data through Dec 17. 2. Processed the e0004s_wet2 output. ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_wet2_daily e0004s_wet2 Checked the FPAR on GrADS. Looks correct. Emailed Sarith to let him know that the APAR and IPAR output is ready. 3. Modified Greg's rzmc.f90 to extract and map the e0004s_wet2 rzmc output back to the correct tile order: ~/Catchment/SMAP_M09 > cp /home/gkwalker/Catchment/rzmc.f90 . ~/Catchment/SMAP_M09 > nedit rzmc.f90 & 4. Continued learning how to run the original CLM4.5. This is to investigate why CLM4.5 GPP is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. 5. Met with Randy and Eunjee: forcing and carbon flux differences between Princeton and MERRA2 ========== 20171219: 1. Continued modifying ~/Catchment/SMAP_M09/rzmc.f90 and ran it. This program extracts the daily rzmc from e0004s_wet2 output and put it back to the original tile order for use in the experiment (e0005s) later. 2. The 20150301-20161231 period of the new e0004s run will be identical to that of e0004s_transientCO2_c05. Since now the CO2 year is the same as the meteorology year, modify the code used for e0004s_transientCO2_c05 to avoid using the external files year_co2.txt and *all.txt. It's easy to make mistakes whenever using these external files. This will benefit the e0005s run too. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/e0004s_transientCO2 cp -p clsm_ensdrv_out_routines.F90 compute_rc.F90 process_cn.F90 ../e0004s/. [These are the only 3 files in e0004s_transientCO2.] cd ../e0004s nedit process_cn.F90 & No change made to clsm_ensdrv_out_routines.F90 and compute_rc.F90. Compiled the code. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 mkdir -p exec/e0004s /bin/cp -pr Linux exec/e0004s/. 3. Set up the new e0004s run using the 20150301 restart file from e0004s_transientCO2_c05 and the same executable as e0004s_transientCO2_cNN, and run it through 20150331. The 20150401 restart file from this new e0004s run will be used to provide initial conditions to the new e0005s run which uses SMAP L4 soil moisture percentiles. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run cp M09_CN_e0004s_transientCO2.exe M09_CN_e0004s.exe cp M09_e0004s_transientCO2.bat M09_e0004s.bat nedit M09_CN_e0004s.exe M09_e0004s.bat & M09_CN_e0004s.exe: exp_id = e0004s start_time = 2015-03-01-00-00-00 end_time = 2015-04-01-00-00-00 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ M09_e0004s.bat: walltime = 2:00:00 job-name = e0004s Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s.bat --runmodel --monthsperjob 1 --landmodel catchCN Check the executable and restart file: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69813672 2017-12-19 15:41 build/Linux/bin/LDASsaCN_mpi.x* ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng g0620 80 2017-12-19 15:44 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run nedit lenkf.0.j & Looks correct. Did an interactive run to test: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run ./lenkf.0.i It used the right restart file: Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2015/M03/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20150301_0000z Writing restart (or incr) file ../output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2015/M03/e0004s.ens0000.catchcn_ldas_rst.20150301_0000z The year and CO2 data looks correct: ***LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3*** DTCN and FireFac : 5400 0.1000000 Year, EEA global average CO2 (ppm), co2 scalar: 2015 394.8710 1.012776 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9634510E-04 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9634510E-04 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9634510E-04 Stopped the interactive run and submit the full job: qsub lenkf.0.j 3. Read Stockli et al., JGR 2011 and see what the parameters in Table 1 are in Catchment-CN. They are: FPAR, LAI, T, L, W, and the min and max of these, also LAImax, FPARsat, and leaf growth and senescence rates. A lot of these do not exist in Catchment-CN though. Read compute_rc.F90 and see what parameters we can calibrate to improve FPAR. PFT-dependent parameters that affect FPAR: ------------------------------------------ fcur: fraction of allocation that goes to currently displayed growth, remainder to storage xl: leaf/stem orientation index rhol: leaf reflectance (visible) rhos: stem reflectance (visible) taul: leaf transmittance (visible) taus: stem transmittance (visible) Their values can be found in subroutine CN_init in CN_DriverMod.F90. Constants that affect FPAR: --------------------------- omegas = 0.80 ! two-stream parameter omega for snow (visible) betads = 0.50 ! two-stream parameter betad for snow betais = 0.50 ! two-stream parameter betai for snow albgrd = 0.10 ! ground albedo direct albgri = 0.10 ! ground albedo indirect They are in compute_rc.F90. Other variables (catchment-level, column-level or pft-level) that affect FPAR include: -------------------------------------------------------------------------------------- ityp: canopy vegetation index (PFT), aka vegetation types fveg: canopy vegetation fractions elai: one-sided leaf area index (not buried by snow) esai: one-sided stem area index (not buried by snow) coszen: cosine zenith angle fwet: fraction of canopy that is wet (0-1) Any parameters that affect LAI can affect FPAR indirectly, e.g. ndep (nitrogen deposition, catchment-level). ========== 20171220: 1. The e0004s run finished the simulation for 20150301-20150331. Compared the year and co2_scalar between e0004s and e0004s_transientCO2_05: e0004s: ------- ***LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3*** DTCN and FireFac : 5400 0.1000000 Year, EEA global average CO2 (ppm), co2 scalar: 2015 394.8710 1.012776 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9701560E-04 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9701560E-04 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9701560E-04 e0004s_transientCO2_05: ----------------------- ***LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3*** DTCN and FireFac : 5400 0.1000000 Transient period:: forcing year, CO2 year, EEA global average CO2 (ppm), co2 sc alar: 2015 2015 394.8710 1.012776 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9701560E-04 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9701560E-04 calculate CT co2 at 2015 3 1 0 7 30 mean co2v (fraction): 3.9701560E-04 They are identical. Good! Here the mean co2v in the log file of e0004s differs from that printed out on screen in the interactive run yesterday (see above). This should be because the number of processors used are different, 140 in the slurm run and 96 in the interactive run. Therefore, the number of tiles on the master processor is likely different. My part is ready to set up the e0005s run that uses SMAP L4 soil moisture percentiles. However, my e0004s run uses Sarith's new tile system, while the SMAP run is still using Sarith's old tile system. Need to talk to Randy about this. 2. Continued working on 3 on 20171219. 3. Justin said running GEOSldas with Catchment-CN on the debuggers crashed. Check out the code and take a look. Check Out: cvs co -r GEOSldas_m4-17_0 GEOSldas_m4 Build (go to src): setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0 source $ESMADIR/src/g5_modules make install BOPT=g The executable is Linux/bin/GEOSldas.x. Set up a test run: cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/src/Applications/LDAS_App ./ldas_setup sample --exeinp > GEOSldas_test.exec ./ldas_setup sample --batinp > GEOSldas_test.bat nedit GEOSldas_test.exec GEOSldas_test.bat & (following /discover/nobackup/fzeng/Catchment/M2n5P/m0003/run/) ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/m0003/ ./GEOSldas_test.exec ./GEOSldas_test.bat (ldas_setup can also be run from /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/Linux/bin) Got this error message when I used /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360: --------------------------------- cmd: ./preprocess_ldas.x c_f2g /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test2_DE720/output/global/rc_out/DE_00720x00360_PE_0720x0360.til LDAS_domain_def.nml /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test2_DE720/output/global/rc_out /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/DE_00720x00360_PE_0720x0360/clsm/catchment.def forrtl: severe (64): input conversion error, unit -5, file Internal List-Directed Read Image PC Routine Line Source preprocess_ldas.x 00000000015F2F23 Unknown Unknown Unknown preprocess_ldas.x 000000000162166A Unknown Unknown Unknown preprocess_ldas.x 000000000044D298 ldas_tilecoordrou 410 LDAS_TileCoordRoutines.F90 preprocess_ldas.x 000000000043EF8D Unknown Unknown Unknown preprocess_ldas.x 000000000043A7F0 Unknown Unknown Unknown preprocess_ldas.x 000000000043A57E Unknown Unknown Unknown libc-2.11.3.so 00002AAAAF30CC36 __libc_start_main Unknown Unknown preprocess_ldas.x 000000000043A415 Unknown Unknown Unknown Traceback (most recent call last): File "./ldas_setup", line 1598, in status = ld.createLnRstBc() File "./ldas_setup", line 616, in createLnRstBc with open('f2g.txt') as f2gfile : IOError: [Errno 2] No such file or directory: 'f2g.txt' --------------------------------- It seems that the current standard GEOSldas_m4-17_0 doesn't work for my DE_00720x00360_PE_0720x0360 bcs. Did I need to modify GEOSldas_m4-17_0 earlier to make it work for my DE_00720x00360_PE_0720x0360 bcs? Maybe. Will check my notes about this later. Changed the bcs_path to /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0144xPC0091_DE0360xPE0180/ in GEOSldas_test.exec and tried again. It works. ***************************DON'T USE THIS. SEE UPDATE PROCEDURES BELOW.*************************** cd /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DC0144/run nedit lenkf.j & #SBATCH --ntasks=28 [This is very important. This number can not be larger than the number requested which is 28 below.] setenv DEBUGGER 2 [or 1] interactive.py -A sp3 -n 96 -a g0620 -X --debug ./lenkf.j For totalview: in the "parallel" tab, choose "open MPI", 28 processes and 1 node For DDT: Check "MPI" for Open MPI, number of processes: 28; processes per node: 28 and uncheck "OpenMP" [This is important! -- But Justin said he didn't have to uncheck this.] **************************************************************************************************** Both totalview and DDT work. Can't repeat it on DDT: -------------------------------------------------------------------------- There are not enough slots available in the system to satisfy the 28 slots that were requested by the application: /gpfsm/dnb31/fzeng/offline_code/GEOSldas_m4-17_0/Linux/bin/GEOSldas.x Either request fewer slots for your application, or make more slots available for use. -------------------------------------------------------------------------- But can repeat it on totalview (in the "parallel" tab, choose "open MPI", 28 processes and 1 node): ---------------------------------------------------------------- In MAPL_Shmem: NumCores per Node varies from 4 to 24 NumNodes in use = 2 Total PEs = 28 In MAPL_InitializeShmem (NodeRootsComm): NumNodes in use = 2 ---------------------------------------------------------------- Should I use 14 processes per node for DDT? Try it. It works. Looks like it's better not to use all the processes in one node. Also, we can leave "OpenMP" checked. Totalview is smarter than DDT in this regard. Although I put in "1 node" in "parallel", it actually uses 2 nodes, so it doesn't have this problem. So the right procedures should be: **************************************************************************************************** cd /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DC0144/run nedit lenkf.j & #SBATCH --ntasks=28 [This is very important. This number can not be larger than the number requested which is 28 below.] setenv DEBUGGER 2 [or 1] interactive.py -A sp3 -n 96 -a g0620 -X --debug ./lenkf.j For totalview: in the "parallel" tab, choose "open MPI", 28 processes and 1 (or 2) node(s) For DDT: Check "MPI" for Open MPI, number of processes: 28; processes per node: 14 **************************************************************************************************** ========== 20171221: 1. Continued investigating why the GPP in the upgraded Catchment-CN is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171226: 1. Continued investigating why the GPP in the upgraded Catchment-CN is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171227: 1. Continued investigating why the GPP in the upgraded Catchment-CN is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171228: 1. Compare Catchment-CN to observations for 2011 and 2012 US droughts ~/Catchment/M2n5P_m0002/compare_catchCN_obs.m ~/Catchment/matlab/m_map/fz_catchCN_obs.m 2. Continued investigating why the GPP in the upgraded Catchment-CN is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20171229: 1. Continued investigating why the GPP in the upgraded Catchment-CN is so low. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.