Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages

Go Home

System Dynamics Blog

SYSTEM DYNAMICS BLOG

Don’t Dismiss Goodness-of-Fit

Don’t Dismiss Goodness-of-Fit

by | Jul 21, 2021

Next year, the famous Limits to Growth (LtG) study will turn fifty. The coming anniversary is causing many to look back to discuss its merits, its message, and the impact it had or could have had. Once again, LtG (and its 20- and 30-year follow-up books) is making a stir.

Exhibit A is a recent article by Gaya Herrington, a director at the Big Four accounting firm KPMG, who has done a fit-to-history analysis of the World3-03 model presented in 2004’s Limits to Growth: The 30-Year Update. She computes goodness-of-fit statistics for 10 different output variables, and four different model scenarios, against historical data from 1990 to 2020.  She finds that the model—and especially its overshoot-and-collapse Business-as-Usual scenario—generally fits the data well. Running with the news, Vice magazine has proclaimed: “MIT predicted in 1972 that society will collapse this century. New research shows we’re on schedule.”

Our colleague Tom Fiddaman has taken issue (in an email forwarded to the MIT SD Group) with both the Herrington study and how it was reported by Vice magazine. He feels that the use of goodness-of-fit testing, and the sensationalistic way it is reported, tend to overemphasize point prediction and that they obscure the fact that LtG was primarily meant to communicate broad dynamic insights.  Tom is discouraged that “the message isn’t out there sufficiently”, and he wonders, “Perhaps it’s time for a new approach?”

Let’s think about this. First, is the Vice magazine piece in fact harmful?  Although the headline is admittedly misleading, the article itself does a rather good job on the facts. Far from trashing LtG, the author echoes Herrington’s plea for people to wake up to our precarious global situation and to support efforts for global sustainability. On balance, I would say that the article is a plus for the cause of LtG.

Second, is the goodness-of-fit analysis done by Herrington really off-track and should it be cause for discouragement?

  Setting aside the fact that Herrington is an avid supporter of the LtG message and a leading advocate for it within KPMG, did she do something wrong by emphasizing goodness-of-fit as a demonstration of model validity?

I don’t think so. I do not scold Herrington for her analysis—in fact, I applaud her. The reliability of system dynamics models rests on the twin pillars of structural realism and behavioral realism, and one useful way of judging behavioral realism is through goodness-of-fit (also known as behavior reproduction) testing.  It is far from a sufficient proof of validity, of course, but I do believe it is a necessary and important one, and I have found it indispensable in my own modeling going back decades.

Indeed, if there is any fault I can find in the LtG books (especially the later ones, when global data had become more plentiful), it is that they did not do goodness-of-fit testing.  I think such testing might have helped minimize the vicious backlash from economists like William Nordhaus and others concerned about the model’s apparent lack of quantitative support.  It might have smoothed the way for greater acceptance and changing more people’s mental models. I’m grateful that Gaya Herrington and others have done goodness-of-fit testing to demonstrate (even if in a small way) the behavioral solidity of World3.

So, is it time for a new approach in how we present World3 and other System Dynamics models, as Tom suggests? Maybe, but I do not think that our main problem has been poor communication skills. I think the bigger problem is that we have not wanted to play by the same rules as other policy modelers do. We have considered it beneath us to do goodness-of-fit testing because it is (so the story goes) a weak demonstration of behavioral validity. I disagree with this dismissive attitude and rather view such testing as an important tool of persuasion to help open people’s minds to systems insights.

by Jack Homer (VP of Professional Practice)

Recent Posts

Practitioner Profile: Jack Homer, Homer Consulting

Practitioner Profile: Jack Homer, Homer Consulting Welcome to Practitioner Profiles, a series of up-close blog-length interviews with experienced System Dynamics practitioners.  We have a standard set of 10 questions and let practitioners take the responses in any...

Data & Uncertainty in System Dynamics

Data & Uncertainty in System Dynamics Jay Forrester cautioned that “fitting curves to past system data can be misleading”. Certainly, that can be true, if the model is deficient. But we can have our cake and eat it too: a good model that passes traditional System...

Upcoming Events

Green Action Global Connection Zone

Green Action Global Connection Zone

The event website: https://climateaction.works/ Schedule and registration form to be announced. Roberto Pasqualino and the Transboundary Groundwater Resilience Network of Networks are confirmed speakers. En-ROADS will have multiple workshops. What: The Green Action...

Recent Business cases

RSC Uses System Dynamics to Increase HVACR Sales Against the Tide

RSC Uses System Dynamics to Increase HVACR Sales Against the Tide “Using the proven Strategy Dynamic process focused our limited resources on organizing strategic issues, identifying the critical resources, and developing the insight to more rapidly create intuitive...

Achieving a Polio-Free World Through System Dynamics Simulation

Achieving a Polio-Free World Through System Dynamics Simulation EXECUTIVE Summary This System Dynamics model underpinned a 192 country resolution to eradicate polio globally and led the Bill and Melinda Gates Foundation to give Rotary International $100 million to...

MasterCard Halts Market Share Slide With Scenario Planning

MasterCard Halts Market Share Slide With Scenario Planning The Problem MasterCard, a major credit card company, once held a dominant market position US Market Share with only one primary competitor, Visa. They woke up one day and realized that, after six years of...

Join us

Jack Homer
Jack Homer

Homer Consulting; System Dynamics Society VP Professional Practice
Dr. Jack Homer is a management consultant and expert in System Dynamics (SD) simulation modeling. His models assist organizations looking for a formal yet flexible approach to strategic planning and policy design.