"floating invalid"

Hi again. 
Just to update: The error does not appear in initial runs, i.e. when not starting from restart files. So the problem seems to be related to restarting OpenIFS. 
I've tried switching Intel compiler versions and also switching to "-O1" instead of "-O3", but that does not help.

I've just managed to get around the problem, but in an ugly way. 
The restart files in year 190 were named "srf0693960000.0001" etc, but I renamed them to "srf0036520000.0001". I also changed the "rcf" file to have CSTEP="0087648" and CTIME   = "0036520000", and used "grib_set" to change the date in the INIT and INIUA files. 

Before, OpenIFS was reading dataDate=19500101 and then setting the date as 19500101 + 69396 days (=21400101), but now it instead reads dataDate=21300101 and sets the date to 21300101 + 3652 days (=21400101). 
And since I renamed the restart files, OpenIFS still reads the restarts from year 190, but now "thinks" it's at year 10. 

So it seems that my problem is related to the time step? Is there some limitation somewhere in OpenIFS that it can't handle more than 190 year long simulations? My time step number (69396 days * 24 hours = 1665504) is somewhere between 2^20 and 2^21. Could it be some integer overflow of NSTEP or some other internal counter? 

Curious to hear if anyone else has had this issue before. I'm also wondering if I'll run into the same problem again 190 simulated years down the road... 

Cheers
Joakim 

Hi Joakim,

Without further investigation this looks like of the problems IFS has with long integrations. Have you tried talking with the EC-Earth people?

I am planning on adding long run fixes from PRIMAVERA & EC-Earth to the next release of 43r3 which should overcomes these problems.

EC-Earth have also noted a problem where with model restarts where it appears the model does not correctly read a restart file. But I don't think that's the same problem as you have here as they reported differing results from two identical model runs.

Cheers,   Glenn

Hi Glenn

Sounds like a really good idea! I feel we've all done various "long run fixes" but all slightly differently... 

As far as I know, I'm the only one who has experienced this problem. 
It might be specific to 40r1 or the compiler (Intel 18.0.3) or the hardware (Intel Skylake). 
I spoke to Klaus W from SMHI about their "long run fixes" and it seems that my fixes are pretty close to the EC-Earth fixes, but not identical, so maybe that could explain the problem as well. 

I need to switch compiler versions soon anyways (for NEMO reasons), so maybe that could solve it. Or it may get solved with 43r3. 

I'll keep you posted how it goes. 

Cheers

Joakim

I also met this problem! I use 43r3 anyway,

forrtl: error (65): floating invalid
Image              PC                Routine            Line        Source             
master.exe         0000000001D9976E  Unknown               Unknown  Unknown
master.exe         00000000008B77AC  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B3A8F4785D0  Unknown               Unknown  Unknown
master.exe         0000000000AD07A6  tpm_pol_mp_ini_po          68  tpm_pol.F90
master.exe         0000000000ABD4B1  suleg_mod_mp_sule         223  suleg_mod.F90
master.exe         00000000008F682B  setup_trans_              405  setup_trans.F90
master.exe         00000000004D2A46  sutrans_                   87  sutrans.F90
master.exe         00000000004C703E  sugeometry_               142  sugeometry.F90
master.exe         00000000004C5434  su0yoma_                  248  su0yoma.F90
master.exe         00000000004189EC  cnt0_                     131  cnt0.F90
master.exe         0000000000412E57  MAIN__                     96  master.F90
master.exe         0000000000412DDE  Unknown               Unknown  Unknown
libc-2.17.so       00002B3A8FA8A3D5  __libc_start_main     Unknown  Unknown
master.exe         0000000000412CE9  Unknown               Unknown  Unknown
......


In my NODE.001_01:
......

---- Set up geometry, (namelist variables) for global model -----
---------------------------------- 
 
 --- Printings in SUGEM_NAML:
NSTTYP =      1 RMUCEN = 0.1000000E+01 RLOCEN = 0.0000000E+00 RSTRET = 0.1000000E+01
NHTYP  =      2
RNLGINC  = 0.1000000E+01
------ Set up transforms for this resolution --
---------------------------------- 
 LUSEFLT= F
 LUSERPNM= T  LKEEPRPNM= T
 LFFTW= F


And in my ifs.stat (only one line):

18:38:50 000000000 CNT3      -999     3.376    3.376    5.391      0:00      0:00 0.00000000000000E+00     160GB       0MB

Hi Anna,

The model has fallen over right at the start of the run. Was this a restart from a previous run? (I suspect not but just to check).

I don't have the code in front of me but from memory the routine it's fallen over in, tpm_pol.F90, is responsible for setting up the Legendre transform coefficients.   Could be a problem with the initial file (probably the ICMSH* file) that has bad data or the wrong data for the resolution you are trying to run.

Cheers.