20160301: 1. Searched for MODIS FPAR product. 2. Met with Randy: 1) Daily FPAR output: a. Greg saved daily FPAR at local noon before. What about daily mean? Check with Joanna what they want. DONE. b. Ask Joanna what spatial resolution they want. See if Sarith has tile file and the related bcs files for this resolution. See what resolution we can do first and reply to Joanna. DONE. 2) Our FPAR project: a. Spin up the MERRA2 run, then from there spin up the other experiment runs. Randy will think about this and get back to me on this. b. Jim's suggestion: using MODIS FPAR as a reference. (Currently only found 1km 8-day data, could be very noisy. Will see if there is monthly product.) c. SWdown issue (send Randy the plots). Randy suggests me to read subroutine SIBALB and see if SWnet or SWdown is used. d. Show Randy the slide comparing CLM4 to CLM4.5 in terms of absorbed PAR. 3) Hourly Princeton forcing (>144G) and daily FPAR output are big. Anywhere else I can put them? Only save sif, sifl, apar, ipar, psn, and lai that Joanna and Yasuko need? - check with them if they need anything else. For hourly Princeton forcing, ask Sarith and Qing what they usually do in such case. For the daily FPAR data, save the full output (may want to ignore some output variables that are unlikely used by Joanna) -> extract daily sif, sifl, apar, ipar, psn and lai to a separate file for Joanna and Yasuko -> archive the dull output and remove it from $NOBACKUP. 3. We can run the model at whatever spatial resolution. It can be coarser or finer than the spatial resolution of the forcing data. 4. Steps to run the model at 0.5x0.5 spatial resolution and save daily (both daily mean and 9:30am local time) SIF, APAR, IPAR, PSN, and LAI for 2007-2015. (1) Run merra/m0008s to get CN_restart for 20070101_00z. (2) Ask Sarith for 0.5x0.5 tile file. (3) Modify grid2tile_pft_ndep_cli_t2m_alb.f90 to create pft.dat, ndep.dat etc. for the new tile system. (4) Use convert_offline_cn_restart.f90 to convert the CN_restart from old tile to the new tile system. (5) For a land model restart, the initial run will use restart=.false. (refer to Greg's cn.txt "exercise: run offline on different tile coordinate system") (6) Modify the code (process_cat.F90 and else?) and the namelist (LDASsa_inputs_driver.nml) to save daily (both daily mean and 9:30am local time) output. In Greg's cn.txt: L466: /discover/nobackup/gkwalker/LDASsa_m2-09_p1_unified/src/Components/GEOSlana_GridComp/73n/process_cat.F90-FPAR: offline code that saved daily FPAR /discover/nobackup/gkwalker/LDASsa_m2-09_p1_unified/src/Components/GEOSlana_GridComp/73n/process_cat.F90.p0036g-lai_daily 5. Run merra/m0008s to 20070101 to obtain CN_restart. /discover/nobackup/fzeng/Catchment/merra > diff m0008s/CN_restart m0008s_33/. Returned nothing. So m0008s is the 34th cycle. dali09:/discover/nobackup/fzeng/Catchment/merra/m0008s > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1533248288 2015-10-30 02:17 CN_restart dali09:/discover/nobackup/fzeng/Catchment/merra/m0008s > ls -l RUN/rs/ens0000/Y2001/M01/ total 40576 -rw-r--r-- 1 fzeng g0620 41519520 2015-10-30 02:17 m0008.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/merra/m0008s > nedit lenkf.j & (to set STOPYEAR to 2007) /discover/nobackup/fzeng/Catchment/merra/m0008s > sbatch lenkf.j After the run finished, /discover/nobackup/fzeng/Catchment/merra > mv m0008s m0008s_34_end20070101 /discover/nobackup/fzeng/Catchment/merra/m0008s_34_end20070101 > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1533248288 2016-03-02 12:12 CN_restart /discover/nobackup/fzeng/Catchment/merra/m0008s_34_end20070101/RUN/rs/ens0000/Y2007/M01 > ls -l total 40576 -rw-r--r-- 1 fzeng g0620 41519520 2016-03-02 12:12 m0008.ens0000.catch_ldas_rst.20070101_0000z.bin /discover/nobackup/fzeng/Catchment/merra/m0008s_34_end20070101 > mv CN_restart CN_restart.20070101_0000z ============= 20160302: 1. Feb 2016 report. 2. Talked to Jim about MODIS FPAR. He told me that there is MODIS FPAR climate gridded product. 3. Eunjee's run using MERRA-2 forcing with SMAP Nature run 5 precipitation crashed at the end of 1991 which is the last year available in stream 100. The model run attamped to read the forcing data files from stream 200 while it's expected to read from stream 100. Helped her look at the issue. Wasn't able to figure out why that happened. She suspected that could be due to potential problem with symbolic links and she re-submitted the run to see if it can go through 1991 and go on this time. 4. Read clsm_ensdrv_force_routines.F90 to find out why the SWdown output from merra2/m0004s (using MERRA-2 forcing but SWdown is replaced by Princeton forcing) is not the same as the SWdown output from p0005s (using Princeton forcing): (call this issue SWdown issue hereafter) L2239-2246: ! Time convention for "met_force_new" as stated in get_forcing() does ! NOT apply to get_GEOS(), which reads forcing states (such as ! Tair) at date_time and "previous" (or "backward-looking") forcing fluxes ! (such as SWdown), rather than "subsequent" (or "forward-looking") ! forcing fluxes. ! ! Example: if date_time=3z, met_force_new will contain Tair at 3z ! and SWdown for average from 0z to 3z (as stored in 1:30z file) Could this be the reason? It seems to me that the difference caused by this small shift in time (< 3hours) should not be large enough to explain the SWdown issue. ============= 20160303: 1. Checked Unified_rc3f_matrix_calc.F90. This routine does not use SWdown. It use SWnet in the following subroutines: UNIFIED: SWNETF, SWNETS, SWNET0 FLUXES: SWNETF MATRIX_CALC: SWNET Randy said if SWnet is used instead of SWdown, that may be the reason for the differences in the SWdown output between m0004s and p0005s. 2. Checked if SWdown is modified after it's read in by clsm_ensdrv_force_routines.F90: (1) Under /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp, these routines have SWdown: catch_types.F90 ("SWdown" is in the instructions that are commented out.) clsm_ensdrv_drv_routines.F90: need to double check clsm_ensdrv_force_routines.F90: checked clsm_ensdrv_mpi.F90 (only one occurance of "SWdown" in a line defining SWdown commented out) clsm_ensdrv_out_routines.F90 (writes out the forcing and the diagnostics, no change made to SWdown.) clsm_ensdrv_pert_routines.F90 (for running the Ensemble Kalman Filter with the catchment model offline driver. We don't use this, right?) driver_types.F90 (defines the driver type and SWdown is one of the forcing states. Applies to both MERRA-2 forcing and Princeton forcing.) ens_driver_routines.F90: (no other modification to SWdown except calling repair_forcing) process_cat.F90: checked (a) Double checked clsm_ensdrv_force_routines.F90: Unplysical SWdown values are repaired by subroutine repair_forcing so that they are within 0-SWDN_MAX (=1360 W/m2 in the beginning of module clsm_ensdrv_force_routines), but this applies to both m0004s and p0005s and should not cause any difference in the SWdown output in these two runs. (b) process_cat.F90: L831-835: ! tentative albedo for snow-free and snow-covered portion ! (naively assumes that each SW radiation component contributes 0.25 to ! total SWdown -- these tentative albedos will NOT be used as such ! if SWnet is provided in forcing data) ! reichle+qliu, 9 Oct 2008 But SWdown is not modified. (2) Under /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp, only GEOS_CatchGridComp.F90 includes "SWDOWNLAND" and it's not used in the offline run. 3. Sarith created boundary conditions files for the 0.5-degree grid for me and I copied the files to my nobackup: /discover/nobackup/fzeng/bcs/Heracles-4_3/Heracles-4_3_MERRA-3 > cp -pr /gpfsm/dnb02/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/DE_00720x00360_PE_0720x0360 . ============= 20160304: 1. Met with Randy and Sarith to talk about the catchcn_internal_rst and SWdown issues. 2. Modified the 4 routines below to save 9:30 am local time sif, apar, ipar, psn and lai. These modified routines are in /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/m0008s_hdeg_daily: catch_types.F90 clsm_ensdrv_init_routines.F90 clsm_ensdrv_out_routines.F90 process_cat.F90 Compiled. The executable is /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_73q_e0007s_old_daily.x. Set up a run p0005s-daily that uses LDASsa_mpi_73q_e0007s_old_daily.x to run from 19480101 to 19480201 to save daily mean and 9:30 am local time output. Also run p0005s (the control run that uses LDASsa_mpi_73q_e0005s.x) to run for the same period. The results will not be identical because LDASsa_mpi_73q_e0007s_old_daily.x has fires on while LDASsa_mpi_73q_e0005s.x has fire off, but the mean over the month in p0005s-daily should be close to p0005s. dali17:/discover/nobackup/fzeng/Catchment/princeton/p0005s > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2015-10-21 07:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/princeton/p0005s > ls -l RUN/rs/ens0000/Y1948/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2015-10-21 07:09 p0005.ens0000.catch_ldas_rst.19480101_0000z.bin 3. Copied Greg's grid_restore_daily.f90. ~/Catchment > cp -p /home/gkwalker/Catchment/grid_restore_daily.f90 . Something is wrong about lail (the others look correct). Lail is -8885 which is impossible. Fixed some bug in process_cat.F90. Re-compiled and set up a new run. /discover/nobackup/fzeng/Catchment/princeton > mv p0005s-daily p0005s-daily-wrong /discover/nobackup/fzeng/Catchment/princeton > mkdir -p p0005s-daily/RUN/rs/ens0000/Y1948/M01 /discover/nobackup/fzeng/Catchment/princeton > cp -p p0005s_63/CN_restart p0005s-daily/. /discover/nobackup/fzeng/Catchment/princeton > cp -p p0005s_63/RUN/rs/ens0000/Y2013/M01/p0005.ens0000.catch_ldas_rst.20130101_0000z.bin p0005s-daily/RUN/rs/ens0000/Y1948/M01/p0005.ens0000.catch_ldas_rst.19480101_0000z.bin /discover/nobackup/fzeng/Catchment/princeton/p0005s-daily > cp -p ../p0005s-daily-wrong/lenkf.j . dali17:/discover/nobackup/fzeng/Catchment/princeton/p0005s-daily > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2015-10-21 07:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/princeton/p0005s-daily > ls -l RUN/rs/ens0000/Y1948/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2015-10-21 07:09 p0005.ens0000.catch_ldas_rst.19480101_0000z.bin /discover/nobackup/fzeng/Catchment/princeton/p0005s-daily > sbatch lenkf.j Compared lail vs. lai, sifl vs. sif, etc. for multiple days in Jan 1948 using ~/Catchment/verify_p0005s_daily.gs. The differences between lail and lai are small enough (<0.1 in most places). ============= 20160307: 1. Processed GMAO GEOS5 seasonal forecast for March. 2. Created ~/geos5/grid2tile_pft_ndep_cli-t2m_alb-2degTOhdeg.f90 (modified from grid2tile_pft_ndep_cli-t2m_alb-20160226.f90) to produce pft.dat, ndep.dat, cli_t2m.dat and MODIS background albedo files for the hdeg tile system. Note that here grid is same as tile, so there are fewer tiles for this hdeg tile system compared to /discover/nobackup/fzeng/bcs/Ganymed-4_1/Ganymed-4_1_Reynolds/testing/DC0144xPC0091_DE0360xPE0180/DC0144xPC0091_DE0360xPE0180-Pfafstetter.til which is at 2x2.5. 3. Created ~/geos5/convert_offline_cn_restart-2degTOhdeg.f90 to convert CN_restart from old tile system to the new hdeg tile system. Got this error message: 365000 125.55 7.70 2 19 0.8332 9790.3 8.1 370000 169.41 -46.24 1 18 0.9279 4515.7 2.4 forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source libintlc.so.5 00002AAAACBF2961 Unknown Unknown Unknown libintlc.so.5 00002AAAACBF10B7 Unknown Unknown Unknown libifcoremt.so.5 00002AAAAAFCDF22 Unknown Unknown Unknown libifcoremt.so.5 00002AAAAAFCDD76 Unknown Unknown Unknown libifcoremt.so.5 00002AAAAAF378B6 Unknown Unknown Unknown libifcoremt.so.5 00002AAAAAF48C78 Unknown Unknown Unknown libpthread.so.0 00002AAAACE51810 Unknown Unknown Unknown a.out 0000000000402B40 MAIN__ 350 convert_offline_cn_restart-2degTOhdeg.f90 a.out 000000000040134E Unknown Unknown Unknown libc.so.6 00002AAAAD281C36 Unknown Unknown Unknown a.out 0000000000401259 Unknown Unknown Unknown Adding "limit stacksize unlimited" doesn't help. Found one bug: L350 "if(ityp(n,nv)<=2 .or. ityp(n,nv)==4 .or. ityp_new(n,nv)==5 .or. ityp_new(n,nv)==9) then" should be "if(ityp(n,nv)<=2 .or. ityp(n,nv)==4 .or. ityp(n,nv)==5 .or. ityp(n,nv)==9) then" because "ityp_new" is not initialized until L490 "read(8,rec=n) ityp_new(n,:),fveg_new(n,:)". Fixed it. This time it went through that point but then another error message popped up: 70 0.0000000E+00 19.10216 71 0.0000000E+00 3.000000 72 9.9999998E-03 38.37931 73 0.0000000E+00 31.19254 74 5.0000001E-02 19.10216 forrtl: severe (24): end-of-file during read, unit 8, file /discover/nobackup/fzeng/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/DE_00720x00360_PE_0720x0360/DE_00720x00360_PE_0720x0360.til Image PC Routine Line Source libifcoremt.so.5 00002AAAAAF36DAA Unknown Unknown Unknown libifcoremt.so.5 00002AAAAAF76974 Unknown Unknown Unknown a.out 00000000004037EB MAIN__ 467 convert_offline_cn_restart-2degTOhdeg.f90 a.out 000000000040134E Unknown Unknown Unknown libc.so.6 00002AAAAD281C36 Unknown Unknown Unknown a.out 0000000000401259 Unknown Unknown Unknown Tried to use totalview to debug but wasn't able to open X display: dali11:~/geos5 > ifort -g -traceback -mcmodel=medium -openmp convert_offline_cn_restart-2degTOhdeg.f90 (to compile with "-g") dali11:~/geos5 > salloc --ntasks=1 --cpus-per-task=28 --constraint=hasw --time=01:00:00 --qos=debug borgo057:~/geos5 > setenv OMP_NUM_THREADS 28 borgo057:~/geos5 > module load tool/tview-8.15.0-15 borgo057:~/geos5 > totalview ./a.out Unable to open X display. Please check your $DISPLAY environment variable to ensure that it is defined correctly and that you are authorized to connect to this X server. (Logging in using "-XY" didn't help.) I think the problem is in this block (L472-L487): icnt = 0 ilwi = 100 do while (ilwi == 100) ! loop over land tiles read(8,*) ilwi,i1,xval,yval,ix,jx,f1,i2 if(ilwi == 100) then icnt = icnt + 1 if(icnt <= N_new) then xlon_new(icnt) = xval xlat_new(icnt) = yval endif endif end do close(8) print *, 'output:',icnt,' tiles' The new hdeg tile file only has land tiles (unlike the old tile files which usually have ocean or other non-land tiles that end the while loop), so ilwi is always 100 and the while loop never ends. Therefore, it keeps reading the tile file although it's already the end of the file. Changed "do while (ilwi == 100) ! loop over land tiles" to "do while (ilwi == 100 .and. icnt < N_new) ! loop over land tiles" (L474). This solved the problem. Got CN_restart created in /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15. In the part "verify restart", a lot of error messages came out: tile_id zone veg var_pft ityp_new fveg_new pft_new(tile_id,zone,veg,var_pft) pft error: 50893 1 1 28 12 0.5697000 65790.02 pft error: 50893 1 3 28 13 0.4303000 65790.02 ... pft error: 67704 3 1 28 13 0.6491000 86400.00 pft error: 67704 3 3 28 8 0.3509000 86400.00 The block of the code verifying the CN_restart created: do n = 1,N_new do nz = 1,nzone do nn = 1,var_col if(col_new(n,nz,nn)==nan .or. col_new(n,nz,nn)/=col_new(n,nz,nn) .or. & col_new(n,nz,nn)cmax(nn)) print *, 'col error:',n,nz,nn,col_new(n,nz,nn),cmax(nn),cmin(nn) end do do nv = 1,nveg if(ityp_new(n,nv)>0 .and. fveg_new(n,nv)>fmin) then do nn = 1,var_pft if(pft_new(n,nz,nv,nn)==nan .or. (pft_new(n,nz,nv,nn)/=pft_new(n,nz,nv,nn)) .or. & pft_new(n,nz,nv,nn)pmax(nn)) print *, 'pft error:',n,nz,nv,nn,ityp_new(n,nv),fveg_new(n,nv),pft_new(n,nz,nv,nn) end do endif end do end do end do var_pft min max 28 0.0000000E+00 65423.07 The value "65790.02" is greater than the max value (65423.07) for this var_pft. That's why I got this error message. The bug is in this block of code: ! recompute length of daylight for new tile to account for latitude shift from source tile ! ---------------------------------------------------------------------------------------- jday = 182 ! Julian day of year (1=Jan 1) gkw: set this to the date of CN_restart (ok to shift to 21z from 00z for GEOS5?) hour = 21 ! hour (0=00z) minu = 0 ! minute seco = 0 ! second This was for created 20150701 21z restart for GCM, and I forgot to change it back to Jan 1, 00z (my CN_restart is for 20070101 00z). Changed it to below and problem fixed: ! recompute length of daylight for new tile to account for latitude shift from source tile ! ---------------------------------------------------------------------------------------- jday = 1 ! Julian day of year (1=Jan 1) gkw: set this to the date of CN_restart (ok to shift to 21z from 00z for GEOS5?) hour = 0 ! hour (0=00z) minu = 0 ! minute seco = 0 ! second 4. Set up an interactive run to debug: In case anything wrong happens, save a copy of the CN_restart: /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp -p CN_restart CN_restart.DE_00720x00360_PE_0720x0360.20070101.00z /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp ../m0008s_33/lenkf.j . /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > mkdir RUN /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > nedit lenkf.j & (following the section "exercise: run offline ..." in ...) dali16:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp lenkf.j lenkf-test.j dali16:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > nedit lenkf-test.j & /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > chmod 755 lenkf-test.j /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > salloc --ntasks=28 --ntasks-per-core=1 --constraint=hasw --time=01:00:00 --qos=debug borgo075:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > lenkf-test.j Got this error message: read_til_file(): reading from/discover/nobackup/fzeng/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//DE_00720x00360_PE_0720x0360/DE_00720x00360_PE_0720x0360.til file contains coordinates for 67704 tiles LDAS ERROR (3000) from read_til_file: unknown tile definitions, filename=DE_007 20x00360_PE_0720x0360.til This is from src/Components/GEOSlana_GridComp/tile_coord.F90. Need to add a case for DE_00720x00360_PE_0720x0360.til in tile_coord.F90. Emailed Sarith and waiting for his reply. ============= 20160308: 1. Sarith confirmed that adding a new case (see below) to tile_coord.F90 as I said in my email should fix the issue. case('DE_00720x00360_PE_0720x0360.til') ! ?? half-degree grid date_line_on_center = .false. pole_on_center = .false. Added the above block to tile_coord.F90. Recompile. 2. Did the interactive run again: /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > /bin/rm *.dat DE_00720x00360_PE_0720x0360 lenkf_job_completed.txt /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > salloc --ntasks=28 --ntasks-per-core=1 --constraint=hasw --time=01:00:00 --qos=debug borgo075:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > lenkf-test.j (1) Got this error message: read_til_file(): reading from/discover/nobackup/fzeng/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//DE_00720x00360_PE_0720x0360/DE_00720x00360_PE_0720x0360.til file contains coordinates for 67704 tiles tile definition grid: date line on edge pole on edge tile_grid_g.gridtype = 'LatLon' ; tile_grid_g.ind_base = 1 ; tile_grid_g.i_dir = 1 ; tile_grid_g.j_dir = 1 ; tile_grid_g.N_lon = 720 ; tile_grid_g.N_lat = 360 ; tile_grid_g.i_offg = 0 ; tile_grid_g.j_offg = 0 ; tile_grid_g.ll_lon = -180.0000 ; tile_grid_g.ll_lat = -90.00000 ; tile_grid_g.ur_lon = 180.0000 ; tile_grid_g.ur_lat = 90.00000 ; tile_grid_g.dlon = 0.5000000 ; tile_grid_g.dlat = 0.5000000 ; forrtl: severe (64): input conversion error, unit 10, file /discover/nobackup/fzeng/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//DE_00720x00360_PE_0720x0360/DE_00720x00360_PE_0720x0360.til Image PC Routine Line Source libmpi_usempif08. 00002AAAAC1C4E57 Unknown Unknown Unknown LDASsa_mpi_73q_e0 0000000002B42811 Unknown Unknown Unknown LDASsa_mpi_73q_e0 00000000005AC2CC tile_coord_routin 482 tile_coord.F90 LDASsa_mpi_73q_e0 00000000005AA3EB tile_coord_routin 832 tile_coord.F90 LDASsa_mpi_73q_e0 000000000045DEAA MAIN__ 752 clsm_ensdrv_main.F90 LDASsa_mpi_73q_e0 00000000004376BE Unknown Unknown Unknown libc.so.6 00002AAAAB8F9C36 Unknown Unknown Unknown LDASsa_mpi_73q_e0 00000000004375C9 Unknown Unknown Unknown The error points to L482-L498 in tile_coord.F90: read (10,*) & tile_coord(i)%typ, & ! 1 tile_coord(i)%area, & ! 2 * tile_coord(i)%com_lon, & ! 3 tile_coord(i)%com_lat, & ! 4 tile_coord(i)%i_indg, & ! 5 tile_coord(i)%j_indg, & ! 6 tile_coord(i)%frac_cell, & ! 7 tmpint1, & ! 8 tile_coord(i)%pfaf, & ! 9 * tmpint2, & ! 10 tile_coord(i)%frac_pfaf, & ! 11 tmpint3 ! 12 * (previously "tile_id") ! change units of area to [km^2] - 23 Sep 2010: fixed units, reichle tile_coord(i)%area = tile_coord(i)%area*MAPL_RADIUS*MAPL_RADIUS/1000./1000. In the new hdeg tile file Sarith created for me, there are fewer columns and the unit of area seems not km^2. Changed L442-457 in tile_coord.F90 if (ease_grid)) then ! EASE grid til file has fewer columns ! (excludes "tile_id", "frac_pfaf", and "area") read (10,*) & tile_coord(i)%typ, & ! 1 tile_coord(i)%pfaf, & ! 2 tile_coord(i)%com_lon, & ! 3 tile_coord(i)%com_lat, & ! 4 tile_coord(i)%i_indg, & ! 5 tile_coord(i)%j_indg, & ! 6 tile_coord(i)%frac_cell ! 7 tile_coord(i)%frac_pfaf = nodata_generic tile_coord(i)%area = ease_cell_area*tile_coord(i)%frac_cell to if (ease_grid .or (index(tile_coord_file,'DE_00720x00360_PE_0720x0360')/=0)) then ! EASE grid til file has fewer columns ! (excludes "tile_id", "frac_pfaf", and "area") read (10,*) & tile_coord(i)%typ, & ! 1 tile_coord(i)%pfaf, & ! 2 tile_coord(i)%com_lon, & ! 3 tile_coord(i)%com_lat, & ! 4 tile_coord(i)%i_indg, & ! 5 tile_coord(i)%j_indg, & ! 6 tile_coord(i)%frac_cell ! 7 tile_coord(i)%frac_pfaf = nodata_generic tile_coord(i)%area = ease_cell_area*tile_coord(i)%frac_cell Recompiled and ran again. (2) Got this error message: Writing catparam file /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15/DE_00720x00360_PE_0720x0360//rc_out///Y2007/M01/m0008.ldas_catparam.20070101_0000z.bin done writing Writing mwRTMparam file /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15/DE_00720x00360_PE_0720x0360//rc_out///Y2007/M01/m0008.ldas_mwRTMparam.20070101_0000z.bin done writing Writing restart (or incr) file /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15/DE_00720x00360_PE_0720x0360//rs/ens0000//Y2007/M01/m0008.ens0000.catch_ldas_rst.20070101_0000z.bin reading number of data times for LAI, GRN, ALB Cannot find file /green_clim_.data in: /discover/nobackup/fzeng/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//DE_00720x00360_ PE_0720x0360/./ LDAS ERROR (3000) from open_land_param_file: ERROR opening file Added "case('DE_00720x00360_PE_0720x0360' ); res_ftag= '720x360_DE'" to L370 in clsm_ensdrv_init_routines.F90. Recompiled and ran again. Works!! Let it finish simulating 2007. Actually it was only able to finish simulating 5 months in 1 hour. 3. Created ~/Catchment/grid_restore_merra_hdeg_daily.f90, compiled and ran it: The information of "gridtype,ind_base,i_dir,j_dir,N_lon,N_lat,i_offg,j_offg, & ll_lon,ll_lat,ur_lon,ur_lat,dlon,dlat" in this order printed out is: LatLon 1 1 1 720 287 0 61 -180.0000 -59.50000 180.0000 84.00000 0.5000000 0.5000000 This information will be needed to create GrADS control files. character(40) :: gridtype ! type of grid, eg. "LatLon", "EASE_M36", "EASE_M09", ... integer :: ind_base ! 0=zero-based indices (EASE), 1=one-based indices (LatLon) integer :: i_dir ! direction of indices (+1=W-to-E, -1=E-to-W) integer :: j_dir ! direction of indices (+1=S-to-N, -1=N-to-S) integer :: N_lon ! number of longitude nodes integer :: N_lat ! number of latitude nodes integer :: i_offg ! minimum lon index (offset from *global* grid) integer :: j_offg ! minimum lat index (offset from *global* grid) ! ! "LatLon": i_offg -> westernmost longitude ! ! j_offg -> southernmost latitude ! ! "EASE_Mxx": i_offg -> westernmost longitude ! ! j_offg -> northernmost latitude real :: ll_lon ! lower left longitude of grid cell edge [deg] real :: ll_lat ! lower left latitude of grid cell edge [deg] real :: ur_lon ! upper right longitude of grid cell edge [deg] real :: ur_lat ! upper right latitude of grid cell edge [deg] real :: dlon ! longitude grid spacing [deg] real :: dlat ! latitude grid spacing [deg] Noticed that there are vertical bands in iparl (9:30 local time ipar). 4. Talked to Randy about the vertical bands. Randy suggests to run the model at 5min time step to see if the bands go away. /discover/nobackup/fzeng/Catchment/merra > mkdir -p m0008s_34_hdeg_07-15_5min/RUN/rs/ens0000/Y2007/M01 /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp -p CN_restart lenkf-test.j ../m0008s_34_hdeg_07-15_5min/. Found that I forgot to change the force_dtstep in the namelist from 10800 to 3600. Before doing the 5min time step run, corrected this and ran interactively again. /discover/nobackup/fzeng/Catchment/merra > mv m0008s_34_hdeg_07-15 m0008s_34_hdeg_07-15_old /discover/nobackup/fzeng/Catchment/merra > cp -pr m0008s_34_hdeg_07-15_5min m0008s_34_hdeg_07-15 /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp -p ../m0008s_34_hdeg_07-15_old/lenkf-test.j . /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > salloc --ntasks=28 --ntasks-per-core=1 --constraint=hasw --time=01:00:00 --qos=debug borgo075:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > lenkf-test.j The bands are still there. Try 5min run. /discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15_5min > sbatch lenkf-test.j Using 5min time step removed the bands. ============= 20160309: 1. Ran the model at 20 minutes time step and save monthly output to spin up the model as far as I can. The spinup is needed to get the land state and particularly the carbon state in equilibrium because any change (e.g. change in tile coordinates in this case) always alters the equilibrium. Will ask Joanna whether she wants us to run the model at 5min or 20min time step on Friday, and whether she really doesn't want us to spin up the model before we collect the daily output for 2007-2015. Then I will save daily output at the time step she perfers for 2007-2015. /discover/nobackup/fzeng/Catchment/merra > mkdir -p m0008hdeg/RUN/rs/ens0000/Y2007/M01 dali16:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp -p CN_restart lenkf.j ../m0008hdeg/. dali16:/discover/nobackup/fzeng/Catchment/merra/m0008s_34_hdeg_07-15 > cp -p RUN/rs/ens0000/Y2007/M01/m0008.ens0000.catch_ldas_rst.20070101_0000z.bin ../m0008hdeg/RUN/rs/ens0000/Y2007/M01/. dali16:/discover/nobackup/fzeng/Catchment/merra/m0008hdeg > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 280023744 2016-03-07 15:25 CN_restart dali16:/discover/nobackup/fzeng/Catchment/merra/m0008hdeg > ls -l RUN/rs/ens0000/Y2007/M01/ total 7424 -rw-r--r-- 1 fzeng g0620 7583072 2016-03-08 12:03 m0008.ens0000.catch_ldas_rst.20070101_0000z.bin The land state and carbon state do not match, but that's ok, we will spin up the model to get them in equilibrium and sync. /discover/nobackup/fzeng/Catchment/merra/m0008hdeg > nedit lenkf.j & /discover/nobackup/fzeng/Catchment/merra/m0008hdeg > sbatch lenkf.j Checked and made sure that the MARRA_Land forcing data is available through the end of 2015 (also available through 20160308 so far). /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing/d591_fpit/diag/Y2015/M12 Created ~/Catchment/submit_next_batch_merra_hdeg. Recorded this run in notes/experiments_log. Made sure the five routines catch_types.F90, clsm_ensdrv_init_routines.F90, clsm_ensdrv_out_routines.F90, process_cat.F90, tile_coord.F90 in m0008hdeg are updated. Restored them (i.e. putting the ones in p0005s back except tile_coord.F90 because changes in tile_coord.F90 do not affect any other model run). 2. process-smap. Noticed that in the e0005s run submitted on 20160219, which is supposed to run through 20160217 as for e0004s, an error occurred: forrtl: severe (29): file not found, unit 7593, file /gpfsm/dnb31/fzeng/Catchment/SMAP_EASEv2_M09/e0005s/rzmc/rzmc_20160128.gdat Image PC Routine Line Source libirc.so 00002AFB00CF0A1E Unknown Unknown Unknown libirc.so 00002AFB00CEF4B6 Unknown Unknown Unknown LDASsa_mpi_73q_e0 0000000002631732 Unknown Unknown Unknown LDASsa_mpi_73q_e0 00000000025D27AB Unknown Unknown Unknown LDASsa_mpi_73q_e0 00000000025D1D12 Unknown Unknown Unknown LDASsa_mpi_73q_e0 00000000025E36B8 Unknown Unknown Unknown LDASsa_mpi_73q_e0 00000000005455C9 process_cat_mp_pr 1495 process_cat.F90 LDASsa_mpi_73q_e0 0000000000513748 ens_driver_routin 294 ens_driver_routines.F90 LDASsa_mpi_73q_e0 00000000004482F9 MAIN__ 1758 clsm_ensdrv_main.F90 Found that rzmc_20160128.gdat was not produced for some reason: dali13:/discover/nobackup/fzeng/SMAP > ls -l *201601*.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:55 rzmc_20160101.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:55 rzmc_20160102.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:55 rzmc_20160103.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:56 rzmc_20160104.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:56 rzmc_20160105.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:56 rzmc_20160106.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:57 rzmc_20160107.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:18 rzmc_20160108.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:18 rzmc_20160109.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:19 rzmc_20160110.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:19 rzmc_20160111.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:19 rzmc_20160112.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160113.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160114.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160115.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160116.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:21 rzmc_20160117.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:21 rzmc_20160118.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:21 rzmc_20160119.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160120.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160121.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160122.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160123.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:23 rzmc_20160124.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:23 rzmc_20160125.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:21 rzmc_20160126.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:21 rzmc_20160127.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:21 rzmc_20160129.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:22 rzmc_20160130.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:22 rzmc_20160131.gdat Why did this happen? discover/nobackup/projects/gmao/smap/SMAP_L4/L4_SM/Vb1012/gph/Y2016/M01/D28 has data files that look normal. Read SMAP.gs and wet2_percentile_SMAP.f90 but still couldn't figure out why that happened. Tried to create rzmc_20160128.gdat manually: ~/Catchment > grads -lbc "run SMAP.gs 20160128 20160128 fzeng" dali13:/discover/nobackup/fzeng/SMAP > ls -l *201601*.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:55 rzmc_20160101.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:55 rzmc_20160102.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:55 rzmc_20160103.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:56 rzmc_20160104.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:56 rzmc_20160105.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:56 rzmc_20160106.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-11 13:57 rzmc_20160107.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:18 rzmc_20160108.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:18 rzmc_20160109.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:19 rzmc_20160110.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:19 rzmc_20160111.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:19 rzmc_20160112.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160113.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160114.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160115.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:20 rzmc_20160116.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:21 rzmc_20160117.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-22 12:21 rzmc_20160118.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:21 rzmc_20160119.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160120.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160121.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160122.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:22 rzmc_20160123.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:23 rzmc_20160124.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-01-28 17:23 rzmc_20160125.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:21 rzmc_20160126.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:21 rzmc_20160127.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-03-09 15:14 rzmc_20160128.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:21 rzmc_20160129.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:22 rzmc_20160130.gdat -rw-r--r-- 1 fzeng g0620 6612628 2016-02-19 13:22 rzmc_20160131.gdat dali17:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 6837457352 2016-02-19 15:50 CN_restart dali17:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > ls -l RUN/rs/ens0000/Y2016/M02 -rw-r--r-- 1 fzeng g0620 185153808 2016-02-19 15:50 e0004.ens0000.catch_ldas_rst.20160218_0000z.bin So it's correct to run e0004s starting from 20160218. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > nedit lenkf.j & (to make sure that the start and end dates are correct) dali17:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 6837457352 2016-01-28 17:47 CN_restart dali17:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s > ls -l RUN/rs/ens0000/Y2016/M01 -rw-r--r-- 1 fzeng g0620 185153808 2016-01-28 17:47 e0005.ens0000.catch_ldas_rst.20160126_0000z.bin So it's correct to run e0005s starting from 20160126. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s > nedit lenkf.j & (to make sure that the start and end dates are correct) Submitted the jobs to run both e0004s and e0005s through 20160229 (restart will be for 20160301 00z): /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > sbatch lenkf.j /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s > sbatch lenkf.j Need to check the experiment log files to make sure the runs finish successfully in the future. 3. Used single point driver to look into the SWdown issue. The input catchment parameters and CN input files located in the "data" directory in the single point driver are from three sources on /discover/nobackup: land01:~/clm4/point_driver_LDASsa_m3-15_2-CN_SWdown/data> ls -lt -rw-r--r--. 1 gkwalker users 11843872 2015-07-12 14:46 pft_FV_144x91_SMAP_TestBed-test_370121.dat -rw-r--r--. 1 gkwalker users 5921936 2014-11-18 13:15 alb_b_2_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r--. 1 gkwalker users 5921936 2014-11-18 13:14 alb_w_2_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r--. 1 gkwalker users 5921936 2014-11-18 13:13 alb_w_1_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r--. 1 gkwalker users 5921936 2014-11-18 13:08 alb_b_1_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r--. 1 gkwalker users 1480484 2014-11-18 11:43 t2m_cli_FV_144x91_SMAP_TestBed_370121.dat -rw-r--r--. 1 gkwalker users 1480484 2014-11-18 11:30 ndep_FV_144x91_SMAP_TestBed_370121.dat -rw-r--r--. 1 gkwalker users 26648725 2014-10-15 13:12 catchment.def -rw-r--r--. 1 gkwalker users 78139760 2014-09-30 12:19 DC0144xPC0091_DE0360xPE0180-Pfafstetter.til -rw-r--r--. 1 gkwalker users 73283958 2014-09-30 11:08 ar.new -rw-r--r--. 1 gkwalker users 24798107 2014-09-30 11:08 bf.dat -rw-r--r--. 1 gkwalker users 54407787 2014-09-30 11:08 soil_param.dat -rw-r--r--. 1 gkwalker users 30349922 2014-09-30 11:08 ts.dat -rw-r--r--. 1 gkwalker users 20727784 2014-09-30 09:00 green_clim_144x91_DC.data -rw-r--r--. 1 gkwalker users 21096897 2014-09-30 08:36 tau_param.dat -rw-r--r--. 1 gkwalker users 28499317 2014-09-30 08:30 CLM_veg_typs_fracs -rw-r--r--. 1 gkwalker users 15915203 2014-09-30 08:25 mosaic_veg_typs_fracs dali16:/discover/nobackup/fzeng/bcs/SMAP_TestBed/FV_144x91 > ls -lt -rwxr-xr-x 1 fzeng g0620 26648725 2014-10-15 13:12 catchment.def* -rwxr-xr-x 1 fzeng g0620 78139760 2014-09-30 12:19 DC0144xPC0091_DE0360xPE0180-Pfafstetter.til* -rwxr-xr-x 1 fzeng g0620 54407787 2014-09-30 11:08 soil_param.dat* -rwxr-xr-x 1 fzeng g0620 30349922 2014-09-30 11:08 ts.dat* -rwxr-xr-x 1 fzeng g0620 24798107 2014-09-30 11:08 bf.dat* -rwxr-xr-x 1 fzeng g0620 73283958 2014-09-30 11:08 ar.new* -rwxr-xr-x 1 fzeng g0620 20727784 2014-09-30 09:00 green_clim_144x91_DC.data* -rwxr-xr-x 1 fzeng g0620 21096897 2014-09-30 08:36 tau_param.dat* -rwxr-xr-x 1 fzeng g0620 28499317 2014-09-30 08:30 CLM_veg_typs_fracs* -rwxr-xr-x 1 fzeng g0620 15915203 2014-09-30 08:25 mosaic_veg_typs_fracs* dali16:/discover/nobackup/fzeng/Catchment > ls -lt -rw-r--r-- 1 fzeng g0620 11843872 2015-07-12 14:46 pft_FV_144x91_SMAP_TestBed-test_370121.dat -rw-r--r-- 1 fzeng g0620 1480484 2014-11-18 11:43 t2m_cli_FV_144x91_SMAP_TestBed_370121.dat -rw-r--r-- 1 fzeng g0620 1480484 2014-11-18 11:30 ndep_FV_144x91_SMAP_TestBed_370121.dat dali16:/discover/nobackup/fzeng/modis_soil_albedo_cmg_v3 > ls -lt -rw-r--r-- 1 fzeng g0620 5921936 2014-11-18 13:15 alb_b_2_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r-- 1 fzeng g0620 5921936 2014-11-18 13:14 alb_w_2_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r-- 1 fzeng g0620 5921936 2014-11-18 13:13 alb_w_1_FV_144x91_SMAP_TestBed_370121_tile.dat -rw-r--r-- 1 fzeng g0620 5921936 2014-11-18 13:08 alb_b_1_FV_144x91_SMAP_TestBed_370121_tile.dat ============= 20160310: 1. Used single point driver to look into the SWdown issue. Read ~/Catchment/extract_princeton_point_pexp_hires.f90. Need to modify extract_princeton_point_pexp_hires.f90 to extract MERRA-2 and princeton forcing for tiles of interest. Can refer to clsm_ensdrv_force_routines.F90. Read process_cat.F90 to see what's in the output. 2. m0008hdeg: Stopped because the precipiptation correction data is not available beyond 20150623: get_forcing(): assuming GEOS GCM forcing data set read SWGDN from /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing////d591_fpit/diag/Y2015/M06/d591_fpit.tavg1_2d_lfo_Nx.20150623_0630z.nc4 read SWLAND from /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing////d591_fpit/diag/Y2015/M06/d591_fpit.tavg1_2d_lfo_Nx.20150623_0630z.nc4 read LWGAB from /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing////d591_fpit/diag/Y2015/M06/d591_fpit.tavg1_2d_lfo_Nx.20150623_0630z.nc4 read PARDR from /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing////d591_fpit/diag/Y2015/M06/d591_fpit.tavg1_2d_lfo_Nx.20150623_0630z.nc4 read PARDF from /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing////d591_fpit/diag/Y2015/M06/d591_fpit.tavg1_2d_lfo_Nx.20150623_0630z.nc4 get_GEOS_netcdf_openfile(): Unsuccessfully tried to open files: /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing///precip_corr_CPCUG5RPFPITv2//d591_fpit/diag/Y2015/M06/d591_fpit.tavg1_2d_lfo_Nx_corr.20150623_0630z.nc4 /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing///precip_corr_CPCUG5RPFPITv2//d591_fpit/diag/Y2015/d591_fpit.tavg1_2d_lfo_Nx_corr.20150623_0630z.nc4 LDAS ERROR (3000) from get_GEOS: error opening file Asked Randy if I should use MERRA-2 forcing. Randy replied on 20160311: "Let's ask Rolf about this at our meeting this afternoon. MERRA-2 should be as good as MERRA-Land, except at high latitudes, and if Joanna is mostly interested in high latitudes, then we should probably just do 2007-2014 with MERRA-Land." dali17:/discover/nobackup/fzeng/Catchment/merra/m0008hdeg > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 280023744 2016-03-10 14:32 CN_restart dali17:/discover/nobackup/fzeng/Catchment/merra/m0008hdeg > ls -l RUN/rs/ens0000/Y2015/M01 total 7424 -rw-r--r-- 1 fzeng g0620 7583072 2016-03-10 14:32 m0008.ens0000.catch_ldas_rst.20150101_0000z.bin Can start from Y2015/M01 next time. ============= 20160311: 1. Read process_cat.F90 to understand why exactly we saw the bands in iparl for 20 min model time step. See my notes in notebook. 2. Read the slides from Joanna for our 1pm meeting today. 3. Read CN_DriverMod.F90: Vcmax25 is a constant for each vegetation type in the model code. The values are in CN_DriverMod.F90. 4. Use clsm_ensdrv_force_routine.F90 from the LDASsa_m3-16_0 tag for m0008hdeg so that we can use MERRA-2 forcing in m0008hdeg. Going forward, use this clsm_ensdrv_force_routine.F90 from the LDASsa_m3-16_0 as default. Recompiled and set up a run. /discover/nobackup/fzeng/Catchment/merra/m0008hdeg > /bin/rm *.dat DE_00720x00360_PE_0720x0360 lenkf_job_completed.txt /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/RUN > /bin/rm -r cat rc_out /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/RUN/rs/ens0000 > ls -l /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/CN_restart -rw-r--r-- 1 fzeng g0620 280023744 2016-03-10 14:32 /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/CN_restart /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/RUN/rs/ens0000 > ls -l Y2015/M01 total 7424 -rw-r--r-- 1 fzeng g0620 7583072 2016-03-10 14:32 m0008.ens0000.catch_ldas_rst.20150101_0000z.bin /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/RUN/rs/ens0000 > /bin/rm -rf Y200? Y201[0-4] /discover/nobackup/fzeng/Catchment/merra/m0008hdeg/RUN/rs/ens0000/Y2015 > /bin/rm -rf M0[2-6] /discover/nobackup/fzeng/Catchment/merra > mv m0008hdeg m0008hdeg_00 (Note: m0008hdeg_00 used MERRA_Land forcing) This new m0008hdeg will start from m0008hdeg_01 and will use MERRA-2 forcing. /discover/nobackup/fzeng/Catchment/merra > mkdir -p m0008hdeg/RUN/rs/ens0000/Y2007/M01 /discover/nobackup/fzeng/Catchment/merra/m0008hdeg_00 > cp -p CN_restart lenkf.j ../m0008hdeg/. /discover/nobackup/fzeng/Catchment/merra/m0008hdeg_00 > cp -p RUN/rs/ens0000/Y2015/M01/m0008.ens0000.catch_ldas_rst.20150101_0000z.bin ../m0008hdeg/RUN/rs/ens0000/Y2007/M01/m0008.ens0000.catch_ldas_rst.20070101_0000z.bin dali17:/discover/nobackup/fzeng/Catchment/merra/m0008hdeg > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 280023744 2016-03-10 14:32 CN_restart dali17:/discover/nobackup/fzeng/Catchment/merra/m0008hdeg > ls -l RUN/rs/ens0000/Y2007/M01/ total 6144 -rw-r--r-- 1 fzeng g0620 7583072 2016-03-10 14:32 m0008.ens0000.catch_ldas_rst.20070101_0000z.bin /discover/nobackup/fzeng/Catchment/merra/m0008hdeg > nedit lenkf.j & (to change the met_tag and met_path so that MERRA-2 forcing is used) Submitted a test run and worked. Submitted the full run. 5. clsm_ensdrv_force_routines.F90: ! define intervals that determine which MERRA-2 stream is used ! in integrations that "cross" multiple streams ! ! 1/1/1980 - 12/31/1991: MERRA2_100 (Stream 1) ! 1/1/1992 - 12/31/2000: MERRA2_200 (Stream 2) ! 1/1/2001 - 12/31/2010: MERRA2_300 (Stream 3) ! 1/1/2011 - present: MERRA2_400 (Stream 4) See if anything wrong will happen at the end of 12/31/2010 when it crosses from Stream 3 to Stream 4 (which happened to Eunjee when it crosses from Stream 1 to Stream 2). ============= 20160314: 1. Checked the output of m0008hdeg_01 through 04. Moved them to Catchment/merra2 since they use MERRA-2 forcing. Archived them. Checked the convergence of the latest two cycles. 2. Updated Higgins precipitation data through 20160313. 3. Process-smap to run from 20160301 to 20160311. 4. Used single point driver to look into the SWdown issue. ~/Catchment > cp extract_princeton_point_pexp_hires.f90 extract_merra2_princeton_point_pexp_hires.f90 ~/Catchment > nedit extract_merra2_princeton_point_pexp_hires.f90 & 5. Completed FY2016 Cybersecurity Training. ============= 20160315: 1. Checked the carbon convergence of m0008hdeg. Talked to Randy and we decide to let the run goes while I am away. The output of each cycle is about 8 GB in size. It can finish ~2 cycles a day, so it will probably finish ~30 cycles by the time I get back. The total size would be ~240 GB and it won't fill up my nobackup yet. Should be fine. 2. Prepared for the GMAO Carbon Cycle meeting. Checked the CO2 concentrations and forcing data I used for the p0008se_63_p1 through p3. ============= 20160329: 1. Stopped the m0008hdeg when it finished 20 cycles at 20min time step and saving monthly output, and continued it at 5min time step and made it save daily output. /discover/nobackup/fzeng/Catchment/merra2 > mkdir -p m0008hdeg_21_5min/RUN/rs/ens0000/Y2007/M01 /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_20 > cp -p CN_restart lenkf.j ../m0008hdeg_21_5min/. /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > cp -p ../m0008hdeg_20/RUN/rs/ens0000/Y2016/M01/m0008.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2007/M01/m0008.ens0000.catch_ldas_rst.20070101_0000z.bin dali09:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 280023744 2016-03-29 09:19 CN_restart dali09:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l RUN/rs/ens0000/Y2007/M01 total 7424 -rw-r--r-- 1 fzeng g0620 7583072 2016-03-29 09:19 m0008.ens0000.catch_ldas_rst.20070101_0000z.bin /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > nedit lenkf.j & (change the namelist to run at 5min time step and save daily output) /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > sbatch lenkf.j 2. Updated Higgins precipitation data through 20160328. 3. Contacted Demi Feng, curator of the webpage where I found MOD15CM (MODIS CMG Monthly LAI and FPAR Product), for where to download MOD15CM. ============= 20160330: 1. Got Demi Feng's reply. Fanwei, I don't think MODAPS is making any MOD15CM, and they never did. The page you have found was created long time ago, I will take it off. I can help you to find out if the product is on science team's plan or not and will keep you updated. Thanks Min 2. Processed merra2/m0008hdeg_21_5min output. ~/Catchment > grid_restore_merra_hdeg_daily 0008 hdeg_21_5min Verified the output using ~/Catchment/verify_m0008hdeg_21_5min.gs. /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > /bin/rm *.dat DE_00720x00360_PE_0720x0360 lenkf_job_completed.txt /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min/RUN > /bin/rm -rf cat rc_out /discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min/RUN/rs/ens0000 > /bin/rm -rf Y200? Y201[0-5] ============= 20160331: 1. Checked out the new Catchment (non-CatchmentCN) tag by Rolf (see his email on 20160325): cvs co -r reichle-LDASsa_m3-16_0_p2 LDASsa_m3 "All, I created a new patch for LDASsa (non-CatchmentCN): co -r reichle-LDASsa_m3-16_0_p2 LDASsa_m3 The key new feature is improved performance, due to restructuring of the I/O and MPI calls that read the forcing data. Depending on the type of experiment, wall-time savings should be between 8% (L4_SM) and 15% (single member model run). These changes are in clsm_ensdrv_force_routines.F90 and clsm_ensdrv_main.F90. You may want to adopt these changes in whatever version of LDASsa you are running. I also fixed a couple of bugs that affected reading the perturbations std-dev from a file, and made a couple more minor changes. See "README.LDASsa_history" for details. The new tag is zero-diff w.r.t. reichle-LDASsa_m3-16_0_p1 for the SMAP model and assimilation test cases. The g5_modules file also includes a minor upgrade of the compiler version. UMD collaborators will want to ignore this. Many thanks to Matt and Manuela for their contributions. Please let me know if you have any questions, Rolf" 2. Checked out the new offline tag (Catchment/Catchment-CN optional) by Sarith (see his email on 20160331): cvs co -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 LDASsa_m3 3. Verified the m0008hdeg_21_5min daily output with Randy (via email) and sent Joanna and Yasuko the data. 4. Modified tile_coord.F90 and clsm_ensdrv_init_routines.F90 in LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 so that it can run on the hdeg tile coordinate system that Sarith created for me to use in my m2_m0008hdeg run. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > cp -p /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/cleanup . Compiled. 5. Compared clsm_ensdrv_force_routines.F90 and clsm_ensdrv_main.F90 between Rolf's tag and Sarith's tag. No difference in clsm_ensdrv_force_routines.F90. The two clsm_ensdrv_main.F90 files differ. 5. Compared clsm_ensdrv_force_routines.F90 and clsm_ensdrv_main.F90 between Rolf's tag and LDASsa_m3-15_2-CN. 6. e0004_201602.gdat and e0005_201602.gdat were not produced, and the 9km output for 201602 was not archived. Looked into this issue. archive-smap did not work properly. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > archive-smap Got this error message: Y2016/M02 full month not found for e0004s 2016 02 28 29, not ready to archive dys remains 28 despite of the command lind below: "if($mo == 2 && ($yr == 2016 || $yr == 2020 || $yr == 2024)) set dys=29" Changed the above command line to "if($mo == 02 && ($yr == 2016 || $yr == 2020 || $yr == 2024)) set dys=29" and it worked. Then, /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > archive-smap /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s > archive-smap to archive the 9km output and create monthly mean for 201602 for both e0004s and e0005s.