ECMWF, GFS, NMM And UKMO: How Have The Models Been Performing?

[johnholmes]

well the next one could be interesting, T+240 is what most of the hot air is about this evening on the model thread. Anyway I've logged them both and we can see which is nearer the mark in 10 days.

[summer blizzard]

Next set of charts, GEM, ECWMF Vs GFS.

[Thundery wintry showers]

The latest charts show that GFS has caught up with UKMO in the Northern Hemisphere at both 5 and 6 days out, and both are only slightly behind ECMWF. In the Southern Hemisphere it is still a fair way behind though.

http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz5.html

http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz6.html

[johnholmes]

The 14th check is not an easy one to decide which is nearest. Neither were that good but ECMWF gave better guidance in my view.

See pdf below.

Score is now GFS=6

ECMWF=4

Neither=4

No 14 thur 7 for sun 17 oct 2010-T+240.pdf

Edited October 17, 2010 by johnholmes

[summer blizzard]

Next set of charts, GEM, ECWMF Vs GFS.

Actual Chart..

I call that one a win for the ECWMF, with GEM very close but losing out because ECWMF picked out the higher mid-Atlantic pressure, GFS was 24-48 hours too slow.

With this is mind, after five 240 hour checks, the table looks like this (2 points for a win, 1 for second and 0 for last)..

GEM: 7 points

ECWMF: 5 points

GFS: 3 points

GEM still leads due to remarkable consistency.

[BlueHedgehog074]

The models seemed to have done reasonably well, in general, modelling the positioning of some of those pressure systems, although they perhaps did underdo the deepness of the Low(s) to the North/North East of the UK (which I suppose is to be expected considering the changes and disturbances the models/forecasts pick out with every daily update).

Must admit, very surprised with the results. Not that I think GEM is a bad model or does not deserve its place, it's just I have been so used to the other models: GFS, ECMWF and Met Office that I forgot the fact their were other models out their which performed just as good, or if not, better than the ones I mentioned.

With what has been happening this, and some of last week, I can remember when GFS started to 'shape' a Northerly which had potential to bring down enough cold air from the North to bring sleet/snow - even to Southern areas of the UK. Although they kept upgrading and downgrading the strength of this Northerly as well as the dam levels (another factor for snow), GFS did seem consistent about when it expected to occur - around 18/19th October onwards. Yet, eventhough the ECMWF expected it to occur around the same time as GFS, ECMWF had the colder air pushed slightly further East and making less of an influence on most of its runs. Unfortunetly didn't look at the other models, despite being excited by the possibility of seeing something wintry this week. Seemed to turn out that due to the behaviour of the high pressure to the West, GFS had overcooked the length and strength of that Northerly with the ridging of that high to the West cutting of that cold North-Wind before it made much of an influence for the whole of the UK. (Though Northern areas did retain it for longer - and it would also be fair to say that the nights were still chilly/frosty, and still are now). So would personally say that with this instance, ECMWF were more on the spot.

Also would like to say thank you guys for updating the thread and going to all the trouble doing these models checks and model scorings - is really appreciated! Thank you. And will still try to help out/add my own views when I can.

[summer blizzard]

The models seemed to have done reasonably well, in general, modelling the positioning of some of those pressure systems, although they perhaps did underdo the deepness of the Low(s) to the North/North East of the UK (which I suppose is to be expected considering the changes and disturbances the models/forecasts pick out with every daily update).

Must admit, very surprised with the results. Not that I think GEM is a bad model or does not deserve its place, it's just I have been so used to the other models: GFS, ECMWF and Met Office that I forgot the fact their were other models out their which performed just as good, or if not, better than the ones I mentioned.

With what has been happening this, and some of last week, I can remember when GFS started to 'shape' a Northerly which had potential to bring down enough cold air from the North to bring sleet/snow - even to Southern areas of the UK. Although they kept upgrading and downgrading the strength of this Northerly as well as the dam levels (another factor for snow), GFS did seem consistent about when it expected to occur - around 18/19th October onwards. Yet, eventhough the ECMWF expected it to occur around the same time as GFS, ECMWF had the colder air pushed slightly further East and making less of an influence on most of its runs. Unfortunetly didn't look at the other models, despite being excited by the possibility of seeing something wintry this week. Seemed to turn out that due to the behaviour of the high pressure to the West, GFS had overcooked the length and strength of that Northerly with the ridging of that high to the West cutting of that cold North-Wind before it made much of an influence for the whole of the UK. (Though Northern areas did retain it for longer - and it would also be fair to say that the nights were still chilly/frosty, and still are now). So would personally say that with this instance, ECMWF were more on the spot.

Also would like to say thank you guys for updating the thread and going to all the trouble doing these models checks and model scorings - is really appreciated! Thank you. And will still try to help out/add my own views when I can.

Indeed, the majority of model discussion centres around the 144< timeframe in which ECWMF has a clear lead followed by GFS and UKMO, however at the 240 hour timeframe it appears that GEM is an excelent trend model, whereas the ECWMF tends to be very hit and miss (either bang on or way off).

Next charts..

Can the combined might of the GFS and ECWMF take on the mighty GEM, we will see.

[johnholmes]

its a small point SB but can be significant at times, 2 of the models are 00z outputs but the GFS is 06z?

See the difference in outputs from the 06 and 12z GFS to illustrate my point? There is a difference of 1020 to <995 mb at one point between the two runs.

Edited October 25, 2010 by johnholmes

[summer blizzard]

its a small point SB but can be significant at times, 2 of the models are 00z outputs but the GFS is 06z?

See the difference in outputs from the 06 and 12z GFS to illustrate my point? There is a difference of 1020 to <995 mb at one point between the two runs.

Fair point however when looking at model accuracy in general from ten days out, i cannot see it making much of a difference given that the ECWMF is made up of different data to the GFS. It would however be reelevent if for instance we were comparing the GFS6z against the GFS12z at the timeframe.

[johnholmes]

The 15th check shows GFS with a pretty good guidance, look at the 12z chart for today. It had the low well positioned, its depth a good approximation for 10 days out and even the 1010mb isobar over about the right place across the UK.

So its now got a score of 7, ECMWF has 4 and neither showing any guidance from either of them also have 4

[Isolated Frost]

It'll be interesting to see which gets closest in this LP situation coming up, my money's on ECM...

[Ladyofthestorm]

I still think the GFS... as it has it tracking in the right direction!

[DT-wxrisk]

It'll be interesting to see which gets closest in this LP situation coming up, my money's on ECM...

it always should be.

a few days ago the wretched GFS had this Low for next week hundreds of miles further N of where it is now showing this Low will track...

[summer blizzard]

Indeed, the majority of model discussion centres around the 144< timeframe in which ECWMF has a clear lead followed by GFS and UKMO, however at the 240 hour timeframe it appears that GEM is an excelent trend model, whereas the ECWMF tends to be very hit and miss (either bang on or way off).

Next charts..

Can the combined might of the GFS and ECWMF take on the mighty GEM, we will see.

Unfortunately, i was unable to be online on the 4th and can not access an archived chart until the end of the month (unless anybody knows another source), so this one will have to wait until then.

Will post a new set of charts tommorow.

[bluearmy]

Unfortunately, i was unable to be online on the 4th and can not access an archived chart until the end of the month (unless anybody knows another source), so this one will have to wait until then.

Will post a new set of charts tommorow.

http://www.meteociel.fr/modeles/gfse_cartes.php?jour=4&mois=11&annee=2010&heure=0&archive=1&mode=0&ech=6

gem definitely third with a narrow victory for ecm over gfs

[ZONE 51]

GFS and UKMO both has been and are my favourite models, ECM next altho a great exciter then a disapearing act, but GFS really i have noticed no more super storm effects, lets see how the upcoming storm is handled, very good on the last storm GFS/ECM/UKMO these especially. UKMO performance is always excellent, i never find any wild goose chase situations with this model, just changes and updates as needed.

Edited November 10, 2010 by nimbilus

[summer blizzard]

Should be interesting given the 3 opposing solutions, though my money is on the GEM, just because of the trend.

[summer blizzard]

Should be interesting given the 3 opposing solutions, though my money is on the GEM, just because of the trend.

GEM was marginally closet being the only model with an easterly flow, and GFS had the trough a little far north, ECWMF was last, though again not too far had the high being further north east.

Taking into account todays result and Bluearmy's post, the table stands at..

GEM: 9 points

ECWMF: 7 points

GFS: 5 points

Amazing table.

Edited November 20, 2010 by summer blizzard

[summer blizzard]

Last set for this year, for xmas day..

GFS vs GEM + ECWMF

[BlueHedgehog074]

He, hee... this should be interesting.

Although the cold spell is just starting for a large area of the UK, it's been quite astounding how the models have been playing about with the position of the Low pressure system for this upcoming cold spell. And how they are playing about with it now. Some models, such as the ECMWF, had the Low pressure too far West, while the GFS generally had the Low pressure area more to the East and, as such, giving a colder outlook.

Even now, for the weekend, the models are unsure how far South they expect the Low pressure system to be, which is undertandable as it is one of those trickier weather scenarios to predict, so some varying performance going on from the models. The Low pressure placement would have an effect on the areas who would see the snow on Saturday and Sunday.

Example from the GFS and ECMWF H500's Pressure charts for Saturday:

GFS chart for Saturday

ECMWF chart for Saturday

Here, the ECMWF and GFS are in slight disagreement with the Low pressure placement for around 00:00 to 12:00 on Saturday. It appears that ECMWF are keen to have it track slightly South-Westwards, while GFS have it on more a direct Southerly track(although not getting to much further South than where they are showing it on these examples.

However, looking at this objectively, I feel that the Low pressue system will end up far South/South-West as shown and that just the far Southern Counties will probably just see the snow, due to the Low pressure being strongest on its Southern end. Though obviously still some possible room for changes (such as the Low Pressue system eneding up being further North).

[summer blizzard]

Last set for this year, for xmas day..

GFS vs GEM + ECWMF

http://www.wetterzentrale.de/pics/archive/ra/2010/Rrea00120101225.gif

That was the actual.

ECWMF won, GEM was second and GFS third.

Final 2010 results were..

GEM: 10 points

ECWMF: 9 points

GFS: 5 points

Overall, the GEM model performed best at day 10 which was a suprise to say the least. ECWMF was very close and GFS was a poor third.

I will start the new model competition from March 1st.

[bluearmy]

http://www.wetterzentrale.de/pics/archive/ra/2010/Rrea00120101225.gif

That was the actual.

ECWMF won, GEM was second and GFS third.

Final 2010 results were..

GEM: 10 points

ECWMF: 9 points

GFS: 5 points

Overall, the GEM model performed best at day 10 which was a suprise to say the least. ECWMF was very close and GFS was a poor third.

I will start the new model competition from March 1st.

can i suggest comparing the gefs/ecm/gem ens means at T240? i think they may be a far better

tool at this range.

[summer blizzard]

can i suggest comparing the gefs/ecm/gem ens means at T240? i think they may be a far better

tool at this range.

If you can find the ensembles means for all three models at 240 hours out, then yes.

[bluearmy]

If you can find the ensembles means for all three models at 240 hours out, then yes.

the ecm ens and gefs are shown on wz. for all three in the same format, you can use meteociel.

[BlueHedgehog074]

An interesting post from Thundery wintry showers regarding the performance of the main models:

Edit: we do still get a northerly on Sunday but not much snow from it, as it topples very quickly.

Unless the pattern changes significantly this has to go down as an event where ECMWF and UKMO have come up trumps over GFS, after a number of events earlier in the season when ECMWF had some uncharacteristic wobbles. However, the perception of a relatively poor-performing ECM isn't unfounded, as the 6 day verification statistics for the previous month show ECM performing only marginally better than GFS and, unusually, a fair way behind the UKMO:

http://www.emc.ncep..../html/acz6.html

The days when we could reasonably talk of dodgy UKMO T+144 outputs appear to be well behind us now!

And kinda stays true to the fact of the ECMWF performing a little better than the GFS when looking at the latest scores on the thread. (ECMWF: 9 points. GFS: 5 points).

Does seem to be the case that the models are having a difficult time predicting this possible cold spell (which still looks very likely for far Northern parts this week) with Shortwaves and the toppling of the high much later on in the outputs making it tricky to predict what is going to happen for later on this week, the weekend and further beyond that.

Edited March 7, 2011 by Rainbow Snow