ECMWF, GFS, NMM And UKMO: How Have The Models Been Performing?

[BlueHedgehog074]

First of all, thanks to Paul for giving me the idea to create this topic (see "Welcome To The New Forcasting Model Discussion" thread).

So basically, this is the place for talking about how the main models have been/or are performing, plus their accuracy. You can include other models like GEFS Extra and NOGAPS, when discussing a model's performance it you wish.

Because of all the various weather models thier are along with their associated outcomes, you may find the way one model performs differs to that of another. The weather in the U.K, especially, can clearly be very unpredictable at times and, sometimes, the UK/Worldwide models may produce all sorts of outcomes and summaries with every update they show. Broadly speaking, when all (or most) models have shown similar outcomes for a good while, usually the predictions for the next few days will "come off". An example of this would be when all the three main models: ECMWF, GFS and UK Met Office, could be agreeing to a dry, sunny, warm spell for the next five days, which has a good chance of occuring of due to the outcomes not only being within the reliable timeframe, but the lack of disagreement helps to raise these chances.

It tends to be the case the the further into Fantasy Island - (term used for weather that is outside the reliable time frame in the models), the less likely the outlook will happen. The differences between the GFS, ECM, and UKMO, tends to become more apparent the further on into their predictions, with all the models trying to pick up various patterns and disturbances in the weather like how will the Jet Stream behave? and how will it have an effect on the settled or unsettled weather we have in the future? These are just some examples.

At times, one or two of the models may show an outlier outlook - even inside the reliable time frame, where they show a completely different setup compared to most other models/forcasts, or one model may not agree with another. I have seen one or two times, where the GFS, in particular, have shown a cool unsettled outlook with limited ridging of high pressure, only to show a completely different scenario on its next update. This was illustrated quite well a few days back and I remember some people, inlcuding me, talked how the 12Z GFS (on 23rd June: http://forum.netweat.../page__st__357) was acting as an outlier with the very sudden hot/very hot weather it showed in FI.

(Image example of when models show different outcomes):

Both the GFS and ECMWF, in this example, show different outcomes in the positioning of the pressure in the H500's charts at 192 hours - GFS have Low Pressure making more of an influence in North-Western areas, whereas ECMWF have the pressure making more of an influence in North-Eastern areas, so some disagreement going on here.

But while the models and the related ensembles may show outliers, sometimes it could be a case that the outliers may have detected new trends, and the other models may, or may not come into agreement with the others that showed the different setups. Although an outlier can be an indication of a models poor performance, this is not always the case. The NAOO (or was it NCEP? correct me if I'm wrong) model inspectors carry out checks on various models to see how they are performing.

So, what would your thoughts be on the models' performance, accuracy, weaknesses or strengths?

Edited June 28, 2010 by Rainbow Snow

[johnholmes]

hi RS

Good idea and best wishes for it being a popular place to show how the models are comparing.

I've started a T+240 check on how GFS and ECMWF do at that time scale, only 7 results GFS showing a better 'handle' on 5 with 1 no real winner so to speak. ECMWF has just got 1, the most recent which I've not really analysed yet but the link is below

post 301 in the Technical area on 16 June at 0927,

http://forum.netweather.tv/topic/58354-in-depth-technical-model-discussion/page__st__289

If you wish I'll drop my results in as they arrive in here?

[BlueHedgehog074]

hi RS

Good idea and best wishes for it being a popular place to show how the models are comparing.

I've started a T+240 check on how GFS and ECMWF do at that time scale, only 7 results GFS showing a better 'handle' on 5 with 1 no real winner so to speak. ECMWF has just got 1, the most recent which I've not really analysed yet but the link is below

post 301 in the Technical area on 16 June at 0927,

http://forum.netweat...n/page__st__289

Thanks.

If you wish I'll drop my results in as they arrive in here?

As longs as that's okay with you. Would be quite interesting.

Edit: Apon reading your document, I have to say I was somewhat surprised with the results, considering GFSes up and down performance. Does look like the GFS handled the modeling of the pressure really well over a long distance of time.

Edited June 28, 2010 by Rainbow Snow

[chionomaniac]

Hi RS.

I agree it is always good to have a comparison thread ( even if the ECM wins every time!)

If you really have trouble sleeping why not look at the detailed statistics on this site: http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

c

[johnholmes]

Hi RS.

I agree it is always good to have a comparison thread ( even if the ECM wins every time!)

c

my checks suggest EC is not doing that well at T+240 ch. It is of course very early days with only 7 checks but they are spread over 3 months.

[BlueHedgehog074]

Hi RS.

I agree it is always good to have a comparison thread ( even if the ECM wins every time!)

If you really have trouble sleeping why not look at the detailed statistics on this site: http://www.emc.ncep....gmb/STATS_vsdb/

c

Cool, thanks.

Having had a look, it is amazing how much detail goes into all the various analysis and all the options you can select like Height, U Wind, V Wind and temperature. That would come in useful, while also getting used to what some of the detailed statistics show (lol). But I'm assuming the more white the images show, the generally more accurate the model has been around that time period?

While I was surprised, though, about ECMWF's and GFS's performance from yesterday, I think it should be noted that during the Winter/Spring period, I remember how the GFS was particularly good at spotting a Northerly wind pattern before other models, even after a week or two away. But not to be mean about GFS or anything, but I have to agree with one or two of the members when they say that the GFS does goes a bit too much to the extremes at times. I found on some instances, that the GFS has over-eaggerated the rainfall amount for my area. Sometimes, when I have been scheduled for two or three minimeters of rain, I've ended up with a completely rain-free day, although to be honest, GFS did get the totals more spot-on for today (rained just after midnight - first time after a week or two without rain). :unsure:

Edited June 29, 2010 by Rainbow Snow

[johnholmes]

this link, its for the 6 day hemisphere verification, shows just how variable the models are, EC is usually the one with the best score but it does vary as you can see below

http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz6.html

In terms of model accuracy at longer distance then you may find work done by TWS on checking ECvGFS and my own T+168 GFS checks, both somewhere in the bowels of Net Wx files, of interest.

It will be interesting to see just how ECvGFS at T+240 shows after a 12 month period. I have to say its results over 3 months with just 7 checks has surprised me.

Another feature is that at T+144 UK Met had a poor reputation on here. Whether it was justified or not I am unable to comment as I never did any checks. Looking at its overall performance at 6 day (T+144) over recent months and its not far off ECMWF.

A shame that NOAA do not do a similar check for the T+240 (10 day period). I'm sure ECMWF must do a check but I've not been able to find it yet-anyone?

After all ECMWF was set up with the specific task of improving forecasts out to 10 days.

[Radiating Dendrite]

Has the GFS accuracy improved since they started using the parallel run as the main run in December or has it not made much difference? I remember there seemed to be quite high hopes for the parallel being better.

[johnholmes]

my impression is that there is little overall improvement but others may disagree.

[Thundery wintry showers]

I just posted a reference to that in the other thread- I don't think NOAA's stats show any improvement at all in the GFS since that December upgrade and if they are to catch up to the UKMO again their 27th July upgrade needs to be good.

When I did my 12Z model checks back in 2006/07, ECMWF came out with the lead overall, with UKMO and GFS sharing second place and not a lot between them. In general GFS was as likely to get the pattern right as ECMWF, but was often more wildly wrong when it didn't, whereas ECMWF often came up with something that wasn't too far off the correct outcome. I recall my results often cast doubt on UKMO's T+144 outputs though it is possible that the UKMO T+144 might have got a bad reputation primarily because that was as far as the run ever went.

The NOAA stats from around then had ECMWF in the lead and ECMWF/UKMO approximately level in second place but they didn't show up any discrepancy with the UKMO at T+144. Since then UKMO has moved ahead of GFS.

[BlueHedgehog074]

I hope I'm not treading on any toes I'm not meaning to but it does highlight how using something with little overall knowledge can lead to over expectation.

I've lost count of the number of times folk have been reminded of how, on most occasions, GFS under does temperature max. Remember also its NOT predicting max temps but those at the spot locations it shows for the time given.

So use it sensibly be it for ideas of depth of lows or highs. Amounts of precip, places affected, as well as when looking at temp predictions.

Watch for its trends, even in the short time scale, they can be a better guide than accepting a particular runs predictions.

I doubt very much that any forecaster amateur or professional predicted temps that GFS kept showing over the last 2-3 days.

What it DID get correct was the change in the weather, cloudier and wetter for most than the past 10 days or so.

(Post quoted above used from another model thread).

Have to agree with that. I noticed that the GFS, even a few days out, were expecting rain for my area for Monday (today). Although they were quite dynamic with the amounts when looking at some of the updates, the fact that it has rained is kind of an indication of how well they spotted this trend. I have also noticed myself that the GFS were expecting many areas to become unsettled from this week with a larger spread of lower pressure, and looking at some of the other forecasts, this seems to be the case.

Then again, the weather ahead could still quickly change, although with support from the ECMWF of lower pressure effecting all areas (with pressure still lowest in the North-West):

...think it is safe to say that the GFS have indeed nailed this unsettled spell ahead rather well, and I think shows good performance in this area of modelling.

Edited July 12, 2010 by Rainbow Snow

[Ben_Cambs]

http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz6.html

Appears that the GFS (the 00z run is used in these statistics) is ever so slightly ahead of the other main models to date.

[summer blizzard]

Given the model discrepency that we have in the sort term at the moment, i am going to post the charts from the models at 144 hours out and then see which model got it right.

Notes, all models are based on 0Z model runs at 144 hours aside from GFS which is based on 6z model run and the chart is at the 138 hours timeframe to compensate, GME only runs to 132 hours out, so 12 hour discrepency.

GFS

GME

ECWMF

GEM

NEOGAPS

UKMO

As you can see, it is GFS and GME against all the other models.

[J10]

Going through each individually.

GFS not bad general pattern right but trough to North of UK and pressure too low over southern UK by around 10mb. 6.5/10

GME large trough to east of UK completely incorrect, but general pattern not that far off. 6/10

ECM has High Pressure quite close to actual but overdid trough stopping linking Scandi High. 7/10

GEM slighlty overdid High over UK, but general pattern close to accurate. 7.5/10

NOGAPS had the pattern just about right but the trough over Scandi was wrong. 6/10

UKMO had the pattern very close in term of pressure over the UK, and even the ridge to Scandi but this was slighlty underplayed. 8/10

To be fair most models did a decent job, but I think UKMO edged it.

[summer blizzard]

Going through each individually.

GFS not bad general pattern right but trough to North of UK and pressure too low over southern UK by around 10mb. 6.5/10

GME large trough to east of UK completely incorrect, but general pattern not that far off. 6/10

ECM has High Pressure quite close to actual but overdid trough stopping linking Scandi High. 7/10

GEM slighlty overdid High over UK, but general pattern close to accurate. 7.5/10

NOGAPS had the pattern just about right but the trough over Scandi was wrong. 6/10

UKMO had the pattern very close in term of pressure over the UK, and even the ridge to Scandi but this was slighlty underplayed. 8/10

To be fair most models did a decent job, but I think UKMO edged it.

Thankyou for the analysis, i had almost forgot about my post.

I notice there that the GEM model did very well, is this usually the case or is it still a back water model. Do you have any resolution statistics for the smaller models, i know the ECWMF is higher than GFS, particulaly in the 180-240 hour timeframe.

[J10]

To be honest they all did pretty well, and an average of all of them would have been spot on.

I think the consensus is ECM first, but I find that it has huge swings past T+168, but at least this means that it drops bad ideas quickly. UKMO edges it over GFS, but the GFS has a habit of picking up a trend dropping it only to find itself right later. GEM diud quite well last winter,but it is not in the top 3.

For the GFS the latest upgrade hopefully will see it improve, after all it is the only one out 4 times a day, and provides the biggest amount of free data.

Exercises like this even though they are only a snapshot are very useful, so if you did it at others times of model divergence it would prove very informative.

[summer blizzard]

To be honest they all did pretty well, and an average of all of them would have been spot on.

I think the consensus is ECM first, but I find that it has huge swings past T+168, but at least this means that it drops bad ideas quickly. UKMO edges it over GFS, but the GFS has a habit of picking up a trend dropping it only to find itself right later. GEM diud quite well last winter,but it is not in the top 3.

For the GFS the latest upgrade hopefully will see it improve, after all it is the only one out 4 times a day, and provides the biggest amount of free data.

Exercises like this even though they are only a snapshot are very useful, so if you did it at others times of model divergence it would prove very informative.

Indeed, over recent years the ECWMF has gone quite far ahead of the GFS, i believe the point it took over GFS was probably summer 2006, since then it has maintained a lead over GFS, however i think that we do need to establish a pecking order especially in regards to the lower models. The GFS upgrade certainly should be interesting, especially in regards to what bias it will have in FI, because this has differed with almost each upgrade over the past few years (in 2005 it always progged northerlies in FI to be replaced by high pressure during the 2007 upgrade, a bias it still maintains today. I also agree that these discussions are extremely useful, especially given the recent divergance, which is probably amplified by the tropical activity.

My plan if the GFS maintains the trough east of the UK at 144 hours tommorow, is to post the charts from all models again, and also do a seperate check at 240 hours for the GFS, ECWMF and GEM.

[johnholmes]

We all have to ensure we are checking like with like-okay I know most know this but for those new to model watching its not a bad idea to give the varying time scales.

in the models GFS goes out to T+384 hours

ECMWF goes out to T+240

UK Met goes out to T+144

This means that the main 3 can only be assessed directly out to that time, useful that ECMWF is now available through the whole time scale and not starting at T+72 as it did until recently, for those of us on unpaid www that is; its always done T+00 upwards within WMO circles.

Remember also that we only see UK Met and ECMWF issues at 00z and 12z.

ECMWF and GFS can be compared directly out to T+240 at 00 and 12z.

GFS is on its own in the big 3 after that times.

A direct check on how the 3 are doing, in terms of the hemispherical 500mb output can be found at the NOAA check site for day 5 (T+120 hours) and 6 days (T+144 hours), see link below

http://www.emc.ncep....TATS/STATS.html

In spite of negative comments until fairly recently about UK Met at T+144 it is in fact better than GFS, usually, and not far off ECMWF in the long term statistics which NOAA run.

Of course sods law is in operation and GFS is, on the latest checks, ahead, just of UK and ECMWF, but the overall averages are as I wrote a moment ago.

In the T+120 then UK and ECMWF are regularly superior to GFS.

On a very small scale the direct checks I'm doing when I get time at T+240 of GFS to ECMWF are surprising at the moment, 5 to GFS, 1 to ECMWF, 1 with neither giving useful advice, both correct=0, very early days though with many more examples needed to give a view of their output usefulness in our small area of concern. Having said that its not just the UK I judge them by, over about 50W to 40E, pole to 20N, with the emphasis on the standard of guidance for the UK weather. but that is a relatively small segment compared to the hemisphere

GFS and ECMWF can be compared

Edited July 24, 2010 by johnholmes

[bluearmy]

JH, i dont suppose that next time you do one of these, you could include GEM 00z which also goes out to T240 and has, annecdotally been doing reasonably well recently in that it has never forecast the block becoming resident in our vicinity in the latter stages of its run.

[summer blizzard]

JH, i dont suppose that next time you do one of these, you could include GEM 00z which also goes out to T240 and has, annecdotally been doing reasonably well recently in that it has never forecast the block becoming resident in our vicinity in the latter stages of its run.

Indeed, if we were to do it to 192 hours instead, we could also include the JMA. GEM model looks pretty good to me, it tends to support the ECWMF as opposed to GFS, so i assume that it is a European model.

Infact, i will post the charts tommorow, when we have all four models available around dinner time.

[Thundery wintry showers]

Sadly little evidence of an improvement in GFS's accuracy at 6 days out since the upgrade but tentative signs of improvement at 5 days:

http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz5.html

http://www.emc.ncep.noaa.gov/gmb/STATS/html/acz6.html

Also UKMO has inched ahead of ECMWF at T+144 for the Northern Hemisphere- which would've been considered unthinkable a few years ago!

[J10]

Are there equivalent figures for longer than 6 days, as the ECM seems to be flipping forecasts quite a bit recently after T+144.

[summer blizzard]

As you can see, at 240 hours it is effectively the GFS and GEM Vs ECWMF, i was unable to get the JMA after 72 hours, there may be a discrepency because the GFS is 6z, but we will have a look at the 0z models on 22nd August.

[johnholmes]

JH, i dont suppose that next time you do one of these, you could include GEM 00z which also goes out to T240 and has, annecdotally been doing reasonably well recently in that it has never forecast the block becoming resident in our vicinity in the latter stages of its run.

I'm not a fan of looking at more than the 3 main models I'm afraid. In my view, and of those ex colleagues I still have contact with, other than the main 3, none of the other models have sufficient routine accuracy to be worth me looking at. Yes UK Met look at all but its the main 3 in terms of synoptic length which they generally take notice of.

Re Julian below

Yes out to T+240 Julian with ECMWF and GFS, at least on the ones I check, admittedly not very often but I will be putting them in more regularly in the forecast discussion thread I posted in a day or so ago. For the NOAA checks for all the main models then T+144, 6 days, is the furthest I've seen them routinely have checks for?

[The PIT]

The GFS 00 oz is leader at the moment over the last few days although it's now a tie.

Pity they don't go further out as that would be interesting. Also anyone know why the southern hemisphere gives the models more problems. Is it lack of data because I would have thought overall less land mass would make modelling easier.

Also the GFS upgrade doesn't seem to have much effect. I've noticed a few wonky temps like 28C for London at midnight once which sort suggests problems somewhere.