Model restart

[NS-Nik]

Hi!

I am running the model for 30 years period and calculation has stopped due to some troubles on the server. The restart settings were set on generating restart file once a year.
I want to restart the model from the last restart point. I created a new run and set up the model, but after checking all the initial setttings the calculation freezes.

For the new run I use wind.ww3 and ice.ww3 files from the previous run.
I checked everything. All the dates in the shel.inp match the the last restart point created by the previous run.

What could be the problem?

Here is the shel.inp settings for the restart run:

$ -------------------------------------------------------------------- $
$ WAVEWATCH III shell input file                                       $
$ -------------------------------------------------------------------- $
$ Define input to be used with flag for use and flag for definition
$ as a homogeneous field; four input lines.
$ 1) Water levels
$ 2) Currents
$ 3) Winds
$ 4) Ice concentrations (cannot be homogeneous).
$ 5,6,7)  Assimilation data
   F F
   F F
   T F
   T
   F
   F
   F
$
$ Time frame of calculations ----------------------------------------- $
$ - Starting time in yyyymmdd hhmmss format.
$ - Ending time in yyyymmdd hhmmss format.
$
   20121226 000000
   20221231 230000
$
   1
$
$ Define output data ------------------------------------------------- $
$ Five output types are available (see below). All output types share
$ a similar format for the first input line:
$ - first time in yyyymmdd hhmmss format, output interval (s), and 
$   last time in yyyymmdd hhmmss format (all integers).
$ Output is disabled by setting the output interval to 0.
$
$          Fields of mean wave parameters
$          Standard line and line with flags to activate output fields
$          as defined in section 2.4 of the manual. The second line is
$          not supplied if no output is requested.
$                               The raw data file is out_grd.ww3, 
$                               see w3iogo.ftn for additional doc.
$
   20121226 000000  3600  20221231 230000
$
  N
  HS FP DP MXH MXHC T02 DIR SPR PHS PTP PDIR
$
$         Point output
$          Standard line and a number of lines identifying the 
$          longitude, latitude and name (C*10) of output points.
$          The list is closed by defining a point with the name
$          'STOPSTRING'. No point info read if no point output is
$          requested (i.e., no 'STOPSTRING' needed).
$                               The raw data file is out_pnt.ww3, 
$                               see w3iogo.ftn for additional doc.
$
   20121226 000000  3600  20221231 230000
$

143.5666667	52.71666667	"PAA"
143.4983333	52.93305556	"PAB"
143.6619444	51.41527778	"LUN"
143.6441667	51.52694444	"FB1"
143.6213889	52.81333333	"FB2"
142.0416667	47.06166667	"KHM"
143.33488	          51.74277778	"KGN"
142.8991667	46.61861111	"LNG"
142.9252778	46.57833333	"OET"
143.6333333	52.9075	          "PA-KP10"
143.6472222	52.81833333	"PA-KP20"
143.6513889	52.72888889	"PA-KP30"
143.6561111	52.63944444	"PA-KP40"
143.6108333	52.56472222	"PA-KP50"
143.4736111	52.53611111	"PA-KP60"
143.3312	          52.53361111	 "PA-KP70"
143.3111111	52.5141	           "PA-KP71"
143.5941667	51.41222222	"LUN-KP5"
143.5230556	51.40583333	"LUN-KP10"
143.496	          51.40152778	"LUN-KP14"
144.145182	51.356809	          "MTK"

    
     0.0   0.0  'STOPSTRING'
$
$ Output along  track.
   20121226 000000    0  20221231 230000
$ Restart files (no additional data required). The data file is restartN.ww3, see w3iors.ftn for additional doc.
   20121226 000000    31536000  20221231 230000
$ Boundary data (no additional data required). The data file is nestN.ww3, see w3iobp.ftn for additional doc.
   20121226 000000     0 20221231 230000
$ Separated wave field data (dummy for now)
   20121226 000000    0  20221231 230000
   'STP'
$ -------------------------------------------------------------------- $
$ End of input file                                                    $
$ -------------------------------------------------------------------- $

And output from the shel file:

shel.log

[aliabdolali]

Did you rename the last restart???.ww3 to restart.ww3?
Check the restart*.ww3 sizes and make sure the one you are using is not corrupted and size is ~ equal to the others.
Then if everything is ok, the log file should say the model started with initial values.
If it fails and everything is ok, it might be a memory issue. The model requires a larger memory to initialize the simulations.

[NS-Nik]

Thanks for the answer.

Sorry, I attached the wrong log file with the output from the shel.inp. I tried different options and one of them was the attempt to run the model from the cold start (I misleadingly attached the log file from this attempt).

I attached the correct file here.
The model reads the restart file but freezes on the stage of updating winds for the first calculation. So, I don't think it is a problem with the restart fie.

But I just tried to run the model from the very beginning of the calculation period and it works fine. Hence, it is not the memory issue either.

So, the bottom line is: I have a restart file, I have preprocessed forcings fo the 30 years of calculation ( wind.ww3 and ice.ww3).
But I cannot run the model from the last restart point (2012.12.26 00:00:00). It runs only from the beginning of the initial calculation period (from 1991.01.01 00:00:00).

Maybe it is an issue with the forcings?

shel.log

[aliabdolali]

what switch did you use to compile the code with?

[NS-Nik]

This one:

F90 NOGRB NC4 TRKNC DIST MPI PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT4 DB1 MLIM TR0 BS0 IC0 IS0 REF1 XX0 WNT2 WNX1 RWND CRT1 CRX1 TIDE O0 O1 O2 O2a O2b O2c O3 O4 O5 O6 O7

[aliabdolali]

is there any reason you use REF1, WNT2 and TIDE?

[NS-Nik]

Thats a copy from the Ifremer1 switch.
I wanted to use ST4 parameterization. Other than that, there is no reason.

[aliabdolali]

try REF0, WNT1 and remove TIDE and test it. You need to redo the simulations most probably

[NS-Nik]

Still the same problem(

[CarstenHansen]

Other than an issue with the forcing, there seems to be a corruption of the model date-time values. The warning in shel.log: "output dates out of run dates : Restart files second request deactivated" means that no restart files will be written, because either the model start is after the restart files 'to' date-time, or the model end is before the restarts files 'from' date-time. But this is clearly wrong, as you have made a correct specification of the dates. Maybe @aliabdolali could suggest further modifications to examine which switches may cause the error? I would also check that the model runs on the same hardware and with the same compiled program and libraries as the first run that was broken by the server troubles.

[CarstenHansen]

Sorry, I have done a mistake here. I was assuming you are running the version 6.07.1, but the warning message I have discussed above is about a 'second stream of restart files' which I see was introduced in the early stages of the develop branch. There is no good reason for letting the code print this warning to standard output, but it is not an indication of any conflict.