Models, Maps, and Levels of Evidence
Anyone who has attended a System Dynamics conference in recent years, or has read past posts in the System Dynamics Society blog, is surely aware of differences of opinion on the value of qualitative maps as opposed to quantified simulations.
What you may not know is that this debate has been going on for a long time, stretching back to the early 1980s, when Eric Wolstenholme and others asserted that one might be able to infer dynamics from qualitative maps. In 1990, Peter Sengeâs book The Fifth Discipline upped the ante by suggesting that certain problems might be categorized according to System Archetypes, which some took to mean that one could go directly to solutions without first simulating. This became known as Systems Thinking.
In 2000, Geoff Coyle wrote a paper (SDR 2000, 16:3) that pointedly asked whether there might be situations so uncertain that quantified modeling cannot tell us more than qualitative mapping alone. Rogelio Oliva and I wrote a rejoinder (SDR 2001, 17:4) that challenged this idea and sought to reclaim Jay Forresterâs original view of simulation as the necessary test bed for any dynamic hypothesis that might conceivably lead to policy decisions.
Even now, 20 years later, the debate continues. We see in our conferences increasing use of Group Model Building that goes no further than a qualitative map, and from which the authors claim to have derived dynamic insights.
Some have been dismayed by this developmentâwhich appears to be the further expansion of Systems Thinkingâfearing it is diluting and dumbing down the fundamentals of our field. The modelers accuse the mappers of lacking rigor, while the mappers say that good group process has a rigor of its own.
What can we do about this long-brewing pot of trouble? Is there any way out of the impasse?
Iâd like to suggest a possibility. Several years ago, I published a paper (SDR 2014, 30:1-2) describing how a âlevels of evidenceâ approachâa standard for classifying work in the biomedical sciencesâmight be applied in SD. To achieve an âAâ level of evidence, one would need both strong structural and behavioral evidence and the ability to reliably test oneâs model. Structural evidence comes from conversations with subject matter experts and focused studies of cause and effect. Behavioral evidence comes from a comparison of model output with time series data and historical records.   Work with strong support for structure but not behavior, or behavior but not structure, would achieve at best a âBâ rating. Work with strong support for neither would get a âCâ rating.
In the biomedical sciences, works that have âBâ or âCâ ratings can still be presented at conferences and even appear in prestigious journals. A rating less than âAâ does not mean poor quality but rather lack of full, iron-clad reliability for drawing conclusions and making decisions; that is, something more exploratory or tentative. Its level of evidence is designated in the conference proceedings or at the top of the paper so that the audience knows what they are dealing withâa decisive work (one from which decisions can be made with confidence) or something less than that.
It seems to me we can apply the Levels of Evidence filter objectively across both simulation models and qualitative maps. Letâs start with the simple fact that a simulation model can be tested formally, while a qualitative map cannot. It follows that the best possible rating for a simulation model is âAâ, while the best possible for a qualitative map is âBâ.
If we can agree on this much, then we may be able to find a way for simulation models and qualitative maps to coexist. It would require acknowledgment from the advocates of qualitative maps that their work can never be considered decisive. And, it would require acknowledgment from the advocates of simulation that a model lacking sufficient evidence may be no more reliable than a well-developed qualitative map.
Recent Posts
Modeling for Impact in the Era of Big Data
Modeling for Impact in the Era of Big Data On the occasion of the fieldâs 50th anniversary, in 2007, Jay Forrester spoke to us (SDR 23:359-370) about prospects for the field of System Dynamicsâhis view of the situation summarized by the phrase âaimless plateauâ. He...
Models Donât Make Predictions, People Do!
Models Donât Make Predictions, People Do! Models and I go back a long way. I created relatively simple spreadsheet models by hand for 6 years at Arthur Andersen & Co. These were done on 14-column paper with a box of Ticonderoga #3 pencils and a Pink Pearl eraser....
Simulation During Times of Crisis
Simulation During Times of Crisis Scientists and researchers spend a long time crafting mathematical models that simulate multiple phenomena occurring in organizations and societies. Those models use varied methodologies based on either proven theories or envisioned...
Upcoming Events

SDS Economics SIG hosts Roberto Pasqualino who will present on Resources, Financial Risk and the Dynamics of Growth
The System Dynamics Society Economics SIG and WPI System Dynamics host Dr. Roberto Pasqualino who will present his work on his Resources, Financial Risk and the Dynamics of Growth book. The book presents a deep analysis on the World3 and provides a global updated...
Recent Business cases
General Motors OnStar
Name The General Motors OnStar Project Modelers Vince Barabba, Chet Huber, Fred Cooke, Nick Pudar, Jim Smith, Mark Paich Client General Motors Client Type Corporation The Official Website onstar.com is the official website in which you can become a member, get...
Pharmaceutical Product Branding Strategies
Pharmaceutical Product Branding Strategies Name Pharmaceutical Product Branding Strategies â Simulating Patient Flow and Portfolio Dynamics Modelers Mark Paich, Corey Peck, Jason Valant Contact Jason Valant or Corey Peck Client Numerous Pharmaceutical Companies Client...
Polio Eradication
Polio Eradication Name Polio Eradication Modelers Kimberly M. Thompson,and Radboud J. Duintjer Tebbens Client World Health Organization (WHO) Client Type NGO The Issue You Tackled Following successful eradication of smallpox and impressive progress in the elimination...
- Models, Maps, and Levels of Evidence - December 22, 2020
- Modeling for Impact in the Era of Big Data - October 28, 2020
- What scientific rigor can and cannot buy us - October 1, 2019
If one wants to compare a qualitative and quantitative work, the first difference is not about the methodology, but about the quality of the work. A quantitative work may be less credible than a qualitative one and reciprocally depending on their respective quality. Once verified that their level of quality is approximately the same, one can start to discuss about their methodology. About the A,B and C the idea is good, excepted that one cannot verify the outputs of a qualitative model, which makes the verification of the model itself more difficult. Hence this is more difficult for a quantitative model to get an A. But for a quantitative model, the outputs must be comparable to some kind of reality or judgment about it, otherwise the output will not have any utility nor its quantitative quality. For me what is important for a model, whether qualitative or quantitative is its capacity to help people to make good decisions and to effectively do it.
Besides that, it is true that one can find many qualitative studies that seem totally incredible especially for someone used to build quantitative models. Building system thinking models is easier than SD models FORMALLY only. But a model that is associated and helped with a quantitative mechanism of verification is finally easier to build than a qualitative one that has not this facility.
The reality is that system thinking attracts more people just because it is easier than SD and less costly for consultants or any practitioner who sells his work as an employee. Nobody will change this fact. It it true too that the SDS by accepting system thinking and SD reduces its specialization. It becomes more difficult to satisfy people who think more differently
Correction: this is more difficults for a ‘qualitative’ model to get an A. sorry for the mistake.