What Is Behind Confusing & Poor Model Performance in U.S. Data?

January 25th, 2013 at 12:11 am by under Chad's WLFI Weather Blog

Cliff Mass did a very interesting posting on issues with our models on his own blog…………..This is not my writing, it is directly from his blog & I am not here to endorse any opinion about any weather agency, I have just noticed poor U.S. model performance & bad flip-flopping.  There seems to be a fundamental issue with numerical prediction in the U.S.  I stumbled upon this article tonight & thought I would share it with you.  I was amazed at how many other articles from meteorologists, 2009 to 2012, speak of the same issues & thoughts.

I use a lot of analog data to forecast & often shy away from a complete model approach.  This is very helpful, but a numerical model backbone is needed for forecasting.

On a side note, a new string of supercomputers is being installed for U.S. models in August 2013 to help combat the increasingly lowering ranking of U.S. weather computer models.  In fact, the big, higher-res U.S. NAM model was ranked LAST of the 6 world models for accuracy in a survey!  The European, Canadian & British model beat it by a large margin.


The U.S. Has Fallen Behind in Numerical Weather Prediction: Part I

It’s a national embarrassment.  It has resulted in large unnecessary costs for the U.S. economy and needless endangerment of our citizens.  And it shouldn’t be occurring.

What am I talking about?   The third rate status of numerical weather prediction in the U.S.  It is a huge story, an important story, but one the media has not touched, probably from lack of familiarity with a highly technical subject.   And the truth has been buried or unavailable to those not intimately involved in the U.S. weather prediction enterprise. This is an issue I have mentioned briefly in previous blogs, and one many of you have asked to learn more about.  It’s time to discuss it.

Weather forecasting today is dependent on numerical weather prediction, the numerical solution of the equations that describe the atmosphere.  The technology of weather prediction has improved dramatically during the past decades as faster computers, better models, and much more data (mainly satellites) have become available.

Supercomputers are used for numerical weather prediction

U.S. numerical weather prediction has fallen to third or fourth place worldwide, with the clear leader in global numerical weather prediction (NWP) being the European Center for Medium Range Weather Forecasting (ECMWF).  And we have also fallen behind in ensembles (using many models to give probabilistic prediction) and high-resolution operational forecasting.  We used to be the world leader decades ago in numerical weather prediction:  NWP began and was perfected here in the U.S.  Ironically, we have the largest weather research community in the world and the largest collection of universities doing cutting-edge NWP research (like the University of Washington!).   Something is very, very wrong and I will talk about some of the issues here.  And our nation needs to fix it.

But to understand the problem, you have to understand the competition and the players.  And let me apologize upfront for the acronyms.

In the U.S., numerical weather prediction mainly takes place at the National Weather Service’s Environmental Modeling Center (EMC), a part of NCEP (National Centers for Environmental Prediction).   They run a global model (GFS) and regional models (e.g., NAM).

The Europeans banded together decades ago to form the European Center for Medium-Range Forecasting (ECMWF), which runs a very good global model.  Several European countries run regional models as well.

The United Kingdom Met Office (UKMET) runs an excellent global model and regional models.   So does the Canadian Meteorological Center (CMC).

There are other major global NWP centers such as the Japanese Meteorological Agency (JMA), the U.S. Navy (FNMOC), the Australian center, one in Beijing, among others.   All of these centers collect worldwide data and do global NWP.

The problem is that both objective and subjective comparisons indicate that the U.S. global model is number 3 or number 4 in quality, resulting in our forecasts being noticeably inferior to the competition.  Let me show you a rather technical graph (produced by the NWS) that illustrates this.  This figure shows the quality of the 500hPa forecast (about halfway up in the troposphere–approximately 18,000 ft) for the day 5 forecast.  The top graph is a measure of forecast skill (closer to 1 is better) from 1996 to 2012 for several models (U.S.–black, GFS; ECMWF-red, Canadian:  CMC-blue, UKMET: green, Navy: FNG, orange).  The bottom graph shows the difference between the U.S. and other nation’s model skill.

You first notice that forecasts are all getting better. That’s good.  But you will notice that the most skillful forecast (closest to one)  is clearly the red one…the European Center.  The second best is the UKMET office.  The U.S. (GFS model) is third…roughly tied with the Canadians.

1.   The U.S. has inadequate computer power available for numerical weather prediction.  The ECMWF is running models with substantially higher resolution than ours because they have more resources available for NWP.   This is simply ridiculous–the U.S. can afford the processors and disk space it would take.  We are talking about millions or tens of millions of dollars at most to have the hardware we need.   A part of the problem has been NWS procurement, that is not forward-leaning, using heavy metal IBM machines at very high costs.

2.  The U.S. has used inferior data assimilation.  A key aspect of NWP is to assimilate the observations to create a good description of the atmosphere.   The European Center, the UKMET Office, and the Canadians using 4DVAR, an advanced approach that requires lots of computer power.   We used an older, inferior approach (3DVAR).  The Europeans have been using 4DVAR for 20 years!   Right now, the U.S. is working on another advanced approach (ensemble-based data assimilation), but it is not operational yet.

3.  The NWS numerical weather prediction effort has been isolated and has not taken advantage of the research community.   NCEP’s Environmental Modeling Center (EMC) is well known for its isolation and “not invented here” attitude.  While the European Center has lots of visitors and workshops, such things are a rarity at EMC.  Interactions with the university community have been limited and EMC has been reluctant to use the models and approaches developed by the U.S. research community.  (True story:  some of the advances in probabilistic weather prediction at the UW has been adopted by the Canadians, while the NWS had little interest).  The National Weather Service has invested very little in extramural research and when their budget is under pressure, university research is the first thing they reduce.  And the U.S. NWP center has been housed in a decaying building outside of D.C.,one  too small for their needs as well.  (Good news… a new building should be available soon).

4.  The NWS approach to weather related research has been ineffective and divided.  The governmnent weather research is NOT in the NWS, but rather in NOAA.  Thus, the head of the NWS and his leadership team do not have authority over folks doing research in support of his mission.  This has been an extraordinarily ineffective and wasteful system, with the NOAA research teams doing work that often has a marginal benefit for the NWS.

5.  Lack of leadership.   This is the key issue.  The folks in NCEP, NWS, and NOAA leadership have been willing to accept third-class status, providing lots of excuses, but not making the fundamental changes in organization and priority that could deal with the problem.  Lack of resources for NWP is another issue…but that is a decision made by NOAA/NWS/Dept of Commerce leadership.

This note is getting long, so I will wait to talk about the other problems in the NWS weather modeling efforts, such as our very poor ensemble (probabilistic) prediction systems.  One could write a paper on this…and I may.

I should stress that I am not alone in saying these things.  A blue-ribbon panel did a review of NCEP in 2009 and came to similar conclusions (found here).  And these issues are frequently noted at conferences, workshops, and meetings.

Let me note that the above is about the modeling aspects of the NWS, NOT the many people in the local forecast offices.  This part of the NWS is first-rate.  They suffer from inferior U.S. guidance and fortunately have access to the ECMWF global forecasts.  And there are some very good people at NCEP that have lacked the resources required and suitable organization necessary to push forward effectively.

This problem at the National Weather Service is not a weather prediction problem alone, but an example of a deeper national malaise.  It is related to other U.S. issues, like our inferior K-12 education system.  Our nation, gaining world leadership in almost all areas, became smug, self-satisfied, and a bit lazy.   We lost the impetus to be the best.   We were satisfied to coast.    And this attitude must end…in weather prediction, education, and everything else… or we will see our nation sink into mediocrity.

The U.S. can reclaim leadership in weather prediction, but I am not hopeful that things will change quickly without pressure from outside of the NWS.  The various weather user communities and our congressional representatives must deliver a strong message to the NWS that enough is enough, that the time for accepting mediocrity is over.  And the Weather Service requires the resources to be first rate, something it does not have at this point.

Part II will discuss the problems with ensemble and high-resolution numerical weather prediction in the U.S.

9 Responses to “What Is Behind Confusing & Poor Model Performance in U.S. Data?”

  1. Chris in Tipton says:

    Looking forward to the next part of this installment. The models have let us down — and this isn’t the only time….or the only winter….. amazing in 2013…..that we are not up to speed….literally….

  2. indiana_man says:

    Ditto on Chris’ comment. This explains why I keep hearing of these different models, even from the national weather forecasters. It was especially evident in the days leading up to Hurricane Sandy. Fortunately (or unfortunately) the forecasters got that one right in general at about 3 days just before it hit. At 4 days prior to Sandy, there was no concern…

  3. Jim says:

    It’s snowing at least. Chad you aren’t wrong man you did your best. Still looking at 2 inches

  4. ML in Rochester says:

    I totally agree with the statement regarding our inferior K-12 educational system. As a teacher who demands excellence and
    Perfection of her students, I am in conflict with many of the parental and society’s view points of education. Heaven forbid if I require students to learn and study the content material.

  5. Bud says:

    Very interesting Chad. Thanks for taking the time to educate us. I’m looking forward to the next installment as well.


  6. Jim says:

    Snow stopping in laf

  7. Mary Anne Best says:

    Only a light dusting here in Remington. And of course it only remains on walks and roads :-( A good day to stay inside and make soup of some kind. Tomorrow we are to have lots of sunshine and then fzr Saturday night into Sunday. Wonderful :-(

    MA in REM

  8. Jim says:

    Mad man. Got maybe a half inch to a inch in laf. Still see grass you?

  9. renee says:

    As of 9:30am still coming down pretty good in Logansport

Leave a Reply