Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages

Go Home

How Did En-ROADS Get 755,000 users? Lessons on Modeling, Interface Design, and Facilitation

How Did En-ROADS Get 755,000 users? Lessons on Modeling, Interface Design, and Facilitation

Achieving widespread engagement is a significant challenge with a System Dynamics model. Yet, En-ROADS, the climate solutions simulator co-developed by Climate Interactive and MIT Sloan, has captivated hundreds of thousands of users globally. In this webinar led by Andrew Jones, John Sterman, and Florian Kapmeier, the team shared how their commitment to System Dynamics modeling principles, innovative interface design, and thoughtful facilitation created a global movement. The lessons learned are relevant for all System Dynamics modelers seeking to make a greater impact in their fields.

1. Modeling: Build Trust and Ensure Accuracy

“You have to have a rigorous, evidence-based model that is fully documented and tested every which way.” John Sterman

En-ROADS’ impact is built on a robust modeling foundation that adheres to rigorous standards of transparency and continuous testing. John Sterman underscores the importance of rigorous validation: “You have to have a rigorous, evidence-based model that is fully documented and tested every which way,” he states, highlighting the necessity for clear, accessible documentation that allows users to understand and trust the model’s operations.

The En-ROADS team enhances model reliability by calibrating past model behavior against historical data (by, e.g. Lazard, the IEA, etc.) and comparing future model behavior against the behavior of climate models used by the Intergovernmental Panel on Climate Change (IPCC), identifying discrepancies not only to prove correctness, but also to pinpoint areas needing improvement. Andrew Jones elaborates on this process: “We compare the model against historical data to understand where the discrepancies are and how we can improve. It’s not about proving the model’s correctness but about identifying areas for growth and improvement.”

For System Dynamics modelers looking to build confidence in the model, these practices are essential. Providing full transparency through comprehensive documentation and continuously validating models against industry benchmarks are crucial steps in establishing credibility. By adhering to these principles, modelers can ensure their models are technically robust and able to support decision-making.

2. Interface Design: Guide Users to Key Insights

“We designed the interface to lead a user to discover our modelers’ top insights without support.” Andrew Jones

En-ROADS’ interface is designed to ensure users reach key insights intuitively. Andrew emphasized the goal: “We designed the interface to lead a user to discover our modelers’ top insights without support.” The interface offers both a beginner mode for quick understanding and an advanced mode for users with more technical skills, providing detailed charts and additional assumptions. Dynamic visuals and animations transform static data into a narrative, making complex concepts accessible and engaging.

Within a few minutes exploring the En-ROADS interface, new users quickly grasp three critical insights:

  1.  “It’s still possible” – to bend the emissions curve and reduce average temperature increase.
  2. “There’s no silver bullet”, highlighting that no single solution can solve the climate crisis.
  3.  “Many solutions together”, emphasizing that combining multiple climate solutions is necessary to mitigate the climate crisis.

An interface designed around key insights helps users understand complexity. The ease of engaging with a new innovation—trialability—is crucial for initial adoption and fostering word of mouth. Yet, negative experiences can lead to adverse feedback, potentially stifling the adoption and diffusion of the model. Sterman emphasized: “if people try something and have a bad experience, they will generate unfavorable word of mouth that can squelch the adoption and diffusion of your model.”

John Sterman emphasizes the consequences of neglecting interface quality: “Suppose you have the world’s greatest model, but a terrible interface. Nobody learns anything, nothing will change, and you and everyone have wasted precious time and resources. That’s a failure mode.”  But he warns, “The opposite, though, is far more dangerous: a great interface with an underlying model that lacks integrity and hasn’t been carefully tested means you’re helping people learn things that are wrong and possibly downright harmful far more effectively than ever before.”

By focusing on intuitive design and rigorous validation, En-ROADS ensures that the simulation is insightful and reliable for understanding possible climate solutions. This strategy serves as a reference for those aiming at making their models both informative and influential. As a System Dynamics Modeler, which 2-3 key takeaways would you like users to learn from your model?

3. Facilitation: Create a Safe Space for Learning

Make sure that you create a safe room where participants can share and challenge their mental models to allow them to engage deeply and learn together.” – Florian Kapmeier

Facilitation is critical to En-ROADS’ global impact. Through experiences like the interactive En-ROADS Climate Workshop and the roleplaying game Climate Action Simulation Game, Climate Interactive effectively conveys the insights of the model by fostering environments that encourage deep learning and reflection. Florian Kapmeier emphasized the importance of these settings: “Make sure that you create a safe room where participants can share and challenge their mental models to allow them to engage deeply and learn together.” This approach allows participants to openly test their assumptions and understand the underlying dynamics of climate change.

Florian, referred to Andrew’s Top 10 Tips to Engage People with a System Dynamics Model,  highlighting one key aspect: By asking participants  to mentally the likely impact of a climate solution on the temperature before running a scenario in En-ROADS, participants reveal their existing mental models and learn more effectively by comparing their understanding with model results. “Learning happens when the theory of thinking is laid out and people have to make a choice.”

The commitment to creating a secure and open space for dialogue is foundational to the success of the engagements with En-ROADS. This facilitation strategy enhances the participants’ ability to understand complex concepts, and encourages them to apply these insights in practical and impactful ways. By ensuring that each session serves as a safe space for exploration and challenge, Climate Interactive fosters an environment where transformative learning and genuine understanding can occur.

Climate Action Simulation for IKEA, Australia

Community & Policy Engagement

Community building has been fundamental to extend En-ROADS’ reach globally. Thousands have completed the Mastering En-ROADS training program, which empowers facilitators to engage diverse audiences. The En-ROADS Climate Ambassador program further nurtures hundreds of facilitators who provide En-ROADS Workshops and Climate Action Simulations worldwide, ensuring a unified network of like-minded advocates. This well-structured support system ensures that these facilitators are equipped to spread their knowledge effectively, fostering a collaborative and impactful community.

In addition to training and support, Climate Interactive’s team emphasized the importance of engaging with policy makers, adapting communication to meet where they are in terms of climate change understanding. Presenting complex model data in an accessible manner is crucial for meaningful engagement. This strategy ensures that decision-makers receive information and are equipped to act on it.

The Power of System Dynamics

“System Dynamics is the most powerful way to engage other people in ways to improve system performance.” – John Sterman

The success of En-ROADS hinges on a commitment to the core principles of System Dynamics, paired with innovative interface design and effective facilitation strategies. This approach has educated a global audience on climate solutions and empowered them to act. John Sterman, reflecting on the broader implications, asserts, “System dynamics is the most powerful way to engage other people in ways to improve system performance”, underscoring the potential of System Dynamics to facilitate meaningful discussions and drive change across various domains.

Sterman reminded the audience that the discipline’s strength lies in its ability to convey complexity in a way that decision-makers can understand and act upon: “System Dynamics modelers need to invite people into the conversation, to join the collective effort and improve system performance together.” This collaborative spirit is reflected in the global network of facilitators, policymakers, and climate advocates brought together by En-ROADS.

As System Dynamics modelers, the challenge is to not only build robust models but also to design engaging interfaces, facilitate participatory workshops, and nurture a community of like-minded advocates. By doing so, modelers can extend their influence beyond their field and create meaningful change in the world.

Watch the recording below

Recent Posts

Society Governance Updates

Society Governance Updates Welcome, Allyson! New President Allyson Beall King joined the Policy Council as our 2024 President. Her primary role is as director of the Washington State University School of the Environment, which focuses on regional ecologies and our...

Call for Presenters: Seminar Series

Call for Presenters: Seminar Series We at the System Dynamics Society are continually seeking vibrant and knowledgeable presenters for our ongoing Seminar Series. As we unfold the calendar, there’s always a place for more insights, experiences, and expertise to enrich...

Upcoming Events

Recent Business cases

A Design Value Calculator: A System Dynamics Boardgame

A Design Value Calculator: A System Dynamics Boardgame EXECUTIVE Summary Product design is a specific form of complex innovation that touches all areas of an organization’s management. While entrepreneurs recognise the value of design, they often tend to focus...

Join us

Volunteer Sprint

Are you interested in volunteering for the Society? Meet other volunteers and network, while helping out with a Society project! We need help rating the models in our model repository. Volunteers with any range of System Dynamics knowledge are welcome. Towards the end of the event, we’ll have a chance to network and connect you with other Society volunteer opportunities you may be interested in.

If you haven’t yet filled out the Society Volunteer form, please complete it here and RSVP to this event.

We greatly appreciate our volunteers’ support for the Society’s initiatives. We look forward to connecting with you!

Green Action Global Connection Zone

The event website: https://climateaction.works/

Roberto Pasqualino and the Transboundary Groundwater Resilience Network of Networks are confirmed speakers. En-ROADS will have multiple workshops.

What: The Green Action Global Connection Zone (online) will connect NGOs, activists and institutions working on the ground in North America, South America, Africa, Asia and Oceania to protect our biodiversity. The online zone is a part of the CEPA Fair and a side event to the official COP15 Biodiversity conference taking place in Montreal from December 7-19.

The entire event will feature Indigenous ecological knowledge, stewardship and management, with Indigenous participation from around the world. The intention is to have broad land-based learnings shared with participants and attendees.

There will also be opportunities for interactive, hands-on activities, such as those offered by Biodiversity and Climate Fresks, to introduce a powerful and innovative way to learn more about the complex implications of biological diversity and climate change.

 

Our audience: action-oriented Indigenous groups and NGOs seeking to connect, strengthen ties, exchange knowledge and experience, to coalesce around concrete action.

We are seeking participants interested in networking with like-minded groups around the world, with the goal of working together to hold governments accountable to the objectives set at COP15, as well as pursuing our own biodiversity goals potentially set on this forum.

 

The format: workshops, meetings, presentations, and debates around major themes.

Online space will be available for exhibition booths, mainstage style featured guests and keynote speakers, small conference room discussions and presentations, and even a “rooftop” setting for casual networking.

 

When: Dec 11-12, 2022, continuously around the clock to accommodate all time zones

Programming on the main stage and in conference rooms will be periodic throughout the event.

 

Who: The event is co-organized by Climate Action Works/Peace Innovation Institute and the Global Greens. Associate co-organizers include MasseCritique, Ohneganos, the McGill Environment Student Society and other NGOs.

Where: The Bramble.Live platform

Applied System Dynamics for Students and Beginners

REGISTER HERE

Several guidelines on the System Dynamics modeling process are found in the literature. While these are rigorous, we have experienced challenges in the use of these existing guidelines in practice, especially where implementation based on the modeling results is critical. In this seminar, we will take you through an integrated system dynamics modeling process which is based on the customized process followed by Eskom SOC with specific elements from the Sustainable Development Programme at Stellenbosch University and Dynamic Strategies. We will also cover the do’s and don’ts when it comes to validation and calibration.

Visit the South African Chapter Website

 

Top 10 Tips to Engage People with a System Dynamics Model

Top 10 Tips to Engage People with a System Dynamics Model

Andrew P. Jones is Co-Founder and Co-Director of Climate Interactive and a Research Affiliate at MIT Sloan. Andrew is an expert on climate change and energy issues, a prominent System Dynamics modeler, and a keynote speaker.

In his webinar with System Dynamics Society, Jones presented his 10+1 tips to engage people with a System Dynamics model. Here’s a summary of the tips, but you can watch the recording below for all details!

All Seminar Series are free for members of the Society. Join us today and unlock all benefits!

1. Make it a challenging adventure on their terms in your virtual world
Make your model playful to entice curiosity about what it tests. This is a virtual world where people will experience something new. The challenging adventure or game is what we do mostly with the interface of the model. If you open the En-ROADS interface, you will see two main graphs, the main output, and several sliders. When playing with the sliders, your changes will be reflected in the graphs. Avoid telling people: “I’m going to show you how to do this”, instead, try saying: “On your terms, you’re going to have to figure out how to get that graph down to your target line”

“We do NOT model systems. We model problems.

2. Focus on mental models you want to improve
We do NOT model systems. We model problems. In System Dynamics, we define a problem as a behavior over time, a reference mode that we’re curious about why does it behave like that. We’re so curious about the problem that we develop a simulation to understand it. We build interfaces to improve the mental models of the stakeholders or clients. System Dynamics modelers know that mostly the mental models of decision-makers don’t match with the real world and how the actual system behaves. We don’t build interfaces to give access to policy levers in the system, but we are trying to leverage the necessary mental model improvements.

3. Make participants mentally simulate before running a scenario
If people don’t have a theory, they don’t notice the mental model they use to forecast what’s going to happen. Then they don’t get to learn and improve their understanding. Therefore, you must encourage people to anticipate, based on their own theories, what’s going to happen when running a scenario. No theory, no learning.” (Edward Deming)

“Tell people what to look at, let them guess what they are going to see, and then make it exciting like a movie”

4. Create a dynamic visual experience
Humans like to see change and it is important to employ this fact to get people to run and look at your graphs without losing their focus. Make people think and anticipate and then direct their eyes to what you want them to notice. If you do this well, you don’t even need to draw a Causal Loop Diagram because they are drawing it in their heads. Tell people what to look at, let them guess what they are going to see, and then make it exciting like a movie.

5. Set up participants to talk with each other
People learn socially and with hot-button issues like climate change, they need to know that their friends are not going to hate them if they change their minds. You need people to be processing the information collectively and socially with others. When presenting your model, stop frequently and advise: “Turn to the person next to you and discuss what you think of that conclusion”.

“Keep the discussion on improving system performance not on the tool you’re using”

6. Know along the way that you are playing at least seven roles

Try to present your model with the help of, at least, another person. Ideally, you need someone to facilitate the discussion while another runs the model. As a facilitator, you will be playing four roles including:

Coach – Helping participants to extract insights out of the model.
Professor – Teaching the audience theories and sharing factual information
Playwright – Creating an emotional journey of ups and downs.
Fellow Traveler – Being authentic, vulnerable, and just another person trying to solve a complex problem. If it is worth making a model about, it’s very important. If it is very important, you must deeply care about it.

You want the conversation to be around how to solve a complex problem together and what your audience is going to do about it. Keep the discussion on improving system performance not on “oh that’s a cool model how did you make that”. That’s all side information.

You want to minimize your roles in:
Tech – you must keep the conversation on improving the system performance, not the way or tool you have used to develop the model or create the graphs
Advocate – when people disagree with you, your job is not to fight them or disagree with them back, but to set up others to find the voice of your work.
Defender – Don’t get involved in the fight of “YOU HAVE A BAD MODEL!”. Avoid this fight as much as you can.

Read the article “Teamwork in Group Model Building” on the System Dynamics Review for more insights on strategies for efficient and effective model building in groups.

7. Build confidence and share testing as needed

You can share your tests and comparisons to other models and/or predictions to build stronger confidence. For instance, Climate Interactive and MIT Sloan built the En-ROADS with the best science available, using the data sources such as the Intergovernmental Panel on Climate Change (IPCC) and International Energy Agency. All the assumptions are available open-source in the 400-page reference guide. Many of the assumptions that someone might not believe in are changeable within the model. Several models do not have good literature of other scenarios to compare against, while En-ROADS can be compared with six integrated assessment models to build confidence.

8. Use loops and stock/flow diagrams only to illuminate
As you help people improve their intuition, sometimes you need to use loops and stock/flow diagrams, especially if you are presenting to a more technical audience but always connect it to the simulator. Avoid showing this to policymakers, they are usually not interested in the loops, instead, tell a story about reinforcing or balancing feedback.

9. Make space for feelings and processing

Deliver your presentation with excitement and intensity, but you need to slam on the brakes and let participants compromise with their feelings and do the necessary processing. You may create a scenario of success, and they get to create their vision, something that they would love to see. And it’s time to slam on the brakes and may invite them to 60 seconds of silence. Yes! It is weird but imagine 60 seconds of silence of people sitting with a scenario of success.

“Create the conditions where people are open to changing their minds”

10. Pay attention to three legs of the learning stool
Reflective Conversation – Create the conditions where people are open to changing their minds, surfacing and testing assumptions, and talking to their peers about improving their assumptions. You’ve got to make the space where people are open to being wrong and thinking differently.

Vision – Help people see a future that they fall in love with they just want it so badly that they see the gap between the vision they want and the reality that they feel and experience that tension in between. Orient towards what one genuinely cares about.

Systems Thinking – Explaining how a complex system works where time, cause, and effect are distant in time and space can be difficult, especially when you include stocks, flows, feedback loops, and other inner relationships. You can learn how to facilitate a training that uses systems thinking and System Dynamics for free with Climate Interactive learning resources

10 + 1. Give them the simulator

People need your help as a facilitator, but you must give them something to play with that naturally gives them the mental model improvements that you want and guides them towards committing themselves to action to improve system performance. As Buckminster Fuller states “If you want to teach people a new way of thinking, don’t bother trying to teach them. Instead, give them a tool, the use of which will lead to new ways of thinking.”

Watch the recording below

Whoops, this recording is available for members only. If you have a membership, please log in. If not, you can definitely get access! Purchase a membership here. If you're not a member but have purchased a ticket to this webinar, please contact us at office@systemdynamics.org

Recent Posts

Society Governance Updates

Society Governance Updates Welcome, Allyson! New President Allyson Beall King joined the Policy Council as our 2024 President. Her primary role is as director of the Washington State University School of the Environment, which focuses on regional ecologies and our...

Call for Presenters: Seminar Series

Call for Presenters: Seminar Series We at the System Dynamics Society are continually seeking vibrant and knowledgeable presenters for our ongoing Seminar Series. As we unfold the calendar, there’s always a place for more insights, experiences, and expertise to enrich...

Upcoming Events

Recent Business cases

A Design Value Calculator: A System Dynamics Boardgame

A Design Value Calculator: A System Dynamics Boardgame EXECUTIVE Summary Product design is a specific form of complex innovation that touches all areas of an organization’s management. While entrepreneurs recognise the value of design, they often tend to focus...

Join us

Modeling for Implementation: an Illusion of Control?

Modeling for Implementation: an Illusion of Control?

Every so often, I see someone in our field drawing attention to the fact that many client-based system dynamics modeling projects do not lead to effective implementation of their findings, and they wonder what can be done to fix this problem.  This is not a new idea, but rather one that has been expressed for more than 40 years.

The first important example was Ed Roberts’ 1977 paper, “Strategies for effective implementation of complex corporate models”, which advocated extensive client involvement, substantive real-world detail and metrics in the model, actionable findings, and supporting the client team into the implementation phase. 

Thirty years later, in 2007, came an even more assertive article from Andreas Grössler, “System dynamics projects that failed to make an impact” (System Dynamics Review 23:4). Grössler argues that System Dynamics projects should be viewed as nothing less than organizational interventions, involving both analysis and action planning. He asks for more research to be done on how modeling can be smoothly integrated into large-scale organizational change efforts.  

Jorgen Randers later amplified this message in his 2019 paper, “The great challenge for system dynamics on the path forward: implementation and real impact” (System Dynamics Review 35:1). Randers says that we need to focus much more on the question of implementation and to develop an effective implementation methodology to stand alongside our modeling methodology.

These are important thoughts on implementation, and I do not doubt the authors when they point to cases of successful implementation as evidence that we can do better.  I must confess, however, that I get uncomfortable when someone speaks of modeling as merely the lead-in to the broader task of organizational change.  I ask myself, “Have I not set my sights high enough?  Why can I not see beyond today’s modeling to the more profound organizational reverberations to follow?”  

But this is silly. Over more than 30 years of private and public sector consulting (often side by side with major-league consulting firms), I have never witnessed profound organizational change happening, not even with longtime clients. I have never expected that any of my models would lead to a cascading change of organizational structure or culture. I really only expect to complete a high-quality model that the client finds useful and clear.

Why would I expect more than that? What ends up happening after the model is completed is largely out of my hands.  I was not trained in organizational intervention, and even those who do claim expertise have a mixed track record.

I have decided that it is sufficient to do a good job of System Dynamics modeling for the client without any particular expectation of implementation afterward. Developing a useful model takes all of my attention. Neither the client nor I can know with certainty what the next step will be after the current modeling project is complete, nor in fact whether this work will continue to be supported by the organization.

So, to answer the question in this blog’s title—yes, I would say that a System Dynamics modeler who believes he or she knows how to maximize the chances of implementation is under an illusion of control. Changing an organization is an incredibly difficult undertaking, and it seems presumptuous to think that one’s modeling skills extend into social engineering or executive leadership. 

I plan to stick with my only real skill set, which is modeling for clients, and to keep trying to do a better job of it. The only way I “model for implementation” is by satisfying the client and putting them in a position to take a more confident next step, whatever it may be.

Recent Posts

Society Governance Updates

Society Governance Updates Welcome, Allyson! New President Allyson Beall King joined the Policy Council as our 2024 President. Her primary role is as director of the Washington State University School of the Environment, which focuses on regional ecologies and our...

Call for Presenters: Seminar Series

Call for Presenters: Seminar Series We at the System Dynamics Society are continually seeking vibrant and knowledgeable presenters for our ongoing Seminar Series. As we unfold the calendar, there’s always a place for more insights, experiences, and expertise to enrich...

Upcoming Events

Recent Business cases

A Design Value Calculator: A System Dynamics Boardgame

A Design Value Calculator: A System Dynamics Boardgame EXECUTIVE Summary Product design is a specific form of complex innovation that touches all areas of an organization’s management. While entrepreneurs recognise the value of design, they often tend to focus...

Join us

Webinar Q&A | Local Level COVID Models

Webinar Q&A | Local Level COVID Models

We had an insightful Webinar with the participation of The COVID 19 Localisation Modelling Group. Kim Warren and Maurice Glucksman and the formidable youth Farrah Farnejad, Quinn Kennedy, Harshita Magroria, and Brahmani Nutakki joined us for a great discussion about the positive impacts of using a localized System Dynamics model to understand COVID realities locally.

You can watch the webinar recording here and download the presentation here. And don’t forget to check the Understanding Your Local COVID-19 Outbreak FREE course that allows you to apply the model to your own locality!

Here are the answers to questions asked live during the Webinar. Please refer to the event page to get more information about the project.

Learn more about the Seminar Series.

Q&A Webinar Local Level COVID Models: Bringing Youth to the Table

Answers by Kim Warren and Maurice Glucksman

  1. Do any of the models consider bigger system impacts on the health of the community and the economy?

These are big important issues but beyond the scope of what we are doing. Our work on mass localizing is starting to address some of the macro issues but only as it relates to the pandemic. Our collaboration with the Emergent Alliance is aimed at wider societal impacts on this and Kim especially is currently engaged with the NHS on Mental Health issues as they relate to the wider impact of COVID-19.

  1. Can you repeat what equation you used to graph/visualize the map?

The data we use comes from publicly available sources that produce the maps.

  1. How can this model help decision-makers make better decisions?

Good question.  We deliberately focused on young people who are captains of their ship and uniquely in this pandemic their actions and decisions are very important for reducing the impact on others through asymptomatic transmission  — even though as a group they are less directly impacted because of severe illness and death in young people is a tiny fraction of older people. So if you define decision-makers as the largest group of people impacting others then it’s very big and it is starting to percolate through the schools and social networks. If you define decision-makers as the traditional healthcare policy organizations it could also be big but not yet.

  1. What data needs to be available to fit a localized model?

Simply 10-year age-group population numbers, and daily series for reported cases, deaths, and hospital cases. You also make estimates of some localization-specific differences. This is all documented in our free webinars and course.

  1. What do the models tell us about the longer-term outlook?

The models tell us radically different stories for every local area and the story is hugely uncertain because of reinfections and mutations. In some cases, the pandemic is over, as the slums of Mumbai, Here in London we are just getting started but vaccines might save us. There is no one story except that you can’t analyze or manage the pandemic well at a national or even state level.

  1. Can we see the model structures? What software did you use?

This is all freely available and documented in our webinars and course.

  1. Quinn — Did you update the model based on your learning to see if you could successfully predict what happened from Jul-Dec 2020 case? And what policies would you recommend going forward?

You need to ask Quinn – will pass this to him. We are constantly revising the model as new information comes in. The pace of revisions has slowed but continues.

  1. Could this be applied to other areas of concern say Climate Change or Food Security to demonstrate inequities?

We have used the principle of localization in Supply Chain, Operations, Marketing, Sales, Strategy so we know it is widely applicable. Issues like Climate change are global by their nature. However, our approach can certainly be used to tackle widely-found environmental challenges – a generic model of common issues, easily configurable to specific local conditions. Others in our field are doing great work on such issues!

  1. Does the definition of local ever change because of events?

Yes. If you look at historical examples of pandemics the maps show each pandemic is a dynamic self-organizing animal without a brain flowing like a swollen river breaking through its banks and spreading out across the landscape influenced by topography, demographics, infrastructure. Inevitably the definition of a local area changes as that happens. Our challenge as analysts is to select a local area that is stable enough to get insights about policies that will help.

  1. How are you defining youth? Including young adults?  What’s the age range?

See question 3.  Youth is our focus because they are important and have been disenfranchised in the pandemic. There is no barrier for anyone of any age to use the models.

  1. Do any of the models include vaccinations and their consequences?

Yes. You can see examples of this analysis in the Westminster presentation.

  1. Farrah — You did a fantastic job. When calibrating your model what data did you find most difficult to find and you missed the most?

You have to ask Farrah! I will pass the question to her but short of that in our Webinars, we go through the data quality issues in detail. The least reliable data is paradoxically the most widely reported: new cases. This is perhaps partly why unfortunately there are so many problems managing the pandemic — it’s like trying to drive your car by looking at google maps on your phone with bad reception and ignoring the view of the road outside. Opportunities for misinterpretation are constant. Death rates are mostly more reliable, but still under-reported to varying degrees, often substantial! The most reliable information is seroprevalence surveys – but these are very occasional [no daily time-series] These allow us to ‘back into’ what must be happening with the stock of susceptible and total stock of infected people. The WHO, CDC, etc will tell you these are the key high leverage drivers of strategy and management of the pandemic but rarely if ever reported.

  1. Have local public health authorities and policymakers who have an influence on policies that affect these communities been involved in the discussion of these results?

This has been limited so far partly because of our focus on young people who have to work hard to win the trust of authorities before their views are taken into account, but it has started to percolate through in surprising ways and we are optimistic that over time the Greta Thunberg effect may take hold in the pandemic.

  1. Many people think that a more effective vaccine is better for them. Is there any way to show that a more rapid roll-out of a less effective vaccine is of greater benefit to everyone?

Farrah is working on a one vs two-dose strategy for Westminster and other boroughs in London. Our working hypothesis is there is no one-size-fits-all strategy. To see why it’s useful to do a thought experiment imagining two districts right next to each other (like Mumbai) where one district has many susceptible people and the other hardly any. The best dosing strategy in these two districts is highly unlikely to be the same.

  1. Can this be broadened to be a ‘system of systems’ where local models are threaded together to look at the local effects, compounded together / interacting?

This is one objective of the ‘mass localization’ project we are working on now.   There will be ‘meta’ interactions between locations that we can’t see working from one location.  It’s the forest for the parable of the trees.

  1. Good to have such a worldwide community for modeling important material

The insights from every location are informing the others. We believe this is very powerful and we hope it is creating an enduring network between young people that would never have developed otherwise.

  1. This is impressive work by the young team led by Prof. Kim. The model has been validated or not?

The model has been validated in dozens of locations globally and provides good insights into radically different locations. Data quality is always a major issue.  Validation is ongoing updating parameters as new mutations emerge and regional differences in the proportion of asymptomatic infections as well as the varying impact of differing vaccines. Occasionally new structures emerge — a good example is a migration that occurred last summer in the Dharavi slum — we could not replicate the pandemic outbreak without allowing half the population to leave the slum. Our belief is no possibility of correctly validating any model of the pandemic that is not local and not developed in collaboration with people actually living in that location or at the very least with excellent access to people who are living there. 

  1. We see so much data on a daily basis, without it being connected in the way models such as this work. Do we feel there would be a benefit if models such as these were more widely shared, and what may those benefits be? 

Tough question… we don’t know. We are sharing our work as open-source with the hope that it will help. The evidence is it has helped our students. They have spoken in this presentation about how the modeling helps them not only understand the pandemic but it has spilled over into other areas in their educations and the way they think about the world. Whether that will have long-term benefits it’s hard to say. I think from our perspective, working on this project has helped us stay sane in what would otherwise be mind-numbingly boring lockdowns, and from a selfish perspective has been a great learning experience and a great way to mobilize our personal networks to have a positive impact on society.

  1. Maurice & Kim – great work! Any tips on how to engage youth effectively?

Thank you. Many tips but mainly motivation. Our key resource was paradoxically having Covid 19 as a research assistant — that motivated all of us to try to solve this and we learned a great deal from the virus about how to make a problem-solving effort viral. It may be possible to leverage other networks, e.g. teachers, but they have had other pressures from COVID.

  1. Wondering about the potential of applying this localization principle could be applied to Doughnut Economic metrics linking social foundation metrics with ecological ceiling metrics to demonstrate complex relationships.

Apparently, donughts are a big reason why diabetes is a major global issue — that’s a complex chain of interaction we can blame on runnin’ on Dunkin’ and Krispy Kreme 😉

  1. Could use it for predicting the new COVID variant found out in Amazonas, Brazil?

We are working with Guttenberg Ferreira Passos and Niraldo Nascimento right now. Suggest you reach out to them and get involved. Let us know if you need us to connect you.

  1. How does System Dynamics modeling compare to Agent-Based Modelling of infection?

We are not agent-based modeling experts but have an amateur understanding of how they work. Agent-based models are excellent for understanding the dynamics of disease transmission – especially for geospatial patterns – because the agents capture the infection pathways and can show emergent dynamics you might miss if you are working only with and SEIR Stock and flow. I believe they are complementary and both offer insight.

  1. How have you raised awareness and gained interest in System Dynamics (COVID) modeling across institutions and what’s next?

We have worked like hell to generate awareness and I would say its early days but the signs are positive. Our work with the NHS in the UK, MIT students in the MISTI program, engagement with Junior Achievement, and most recently the Emergent Alliance are good lead indicators but we can do much more.

  1. Lockdowns serve to restrict the movement between locales and thereby enhance the relevance of the model

Lockdowns, shielding of vulnerable, hygiene measures, testing and tracing policies, vaccines, hospital capacity are amongst the policy levers directly available.

  1. The final model appears to use Silico. What is this?

Silico is a relatively new System Dynamics Simulation software package. We use it because it is free for personal use as long as you are happy to share your model as an open-source resource, it is easy to teach and use and very intuitive… check it out in our webinars and course. Check their website.

Recent Posts

Society Governance Updates

Society Governance Updates Welcome, Allyson! New President Allyson Beall King joined the Policy Council as our 2024 President. Her primary role is as director of the Washington State University School of the Environment, which focuses on regional ecologies and our...

Call for Presenters: Seminar Series

Call for Presenters: Seminar Series We at the System Dynamics Society are continually seeking vibrant and knowledgeable presenters for our ongoing Seminar Series. As we unfold the calendar, there’s always a place for more insights, experiences, and expertise to enrich...

Upcoming Events

Recent Business cases

A Design Value Calculator: A System Dynamics Boardgame

A Design Value Calculator: A System Dynamics Boardgame EXECUTIVE Summary Product design is a specific form of complex innovation that touches all areas of an organization’s management. While entrepreneurs recognise the value of design, they often tend to focus...

Join us

Models, Maps, and Levels of Evidence

Models, Maps, and Levels of Evidence

Anyone who has attended a System Dynamics conference in recent years, or has read past posts in the System Dynamics Society blog, is surely aware of differences of opinion on the value of qualitative maps as opposed to quantified simulations.

What you may not know is that this debate has been going on for a long time, stretching back to the early 1980s, when Eric Wolstenholme and others asserted that one might be able to infer dynamics from qualitative maps.  In 1990, Peter Senge’s book The Fifth Discipline upped the ante by suggesting that certain problems might be categorized according to System Archetypes, which some took to mean that one could go directly to solutions without first simulating.  This became known as Systems Thinking.

In 2000, Geoff Coyle wrote a paper (SDR 2000, 16:3) that pointedly asked whether there might be situations so uncertain that quantified modeling cannot tell us more than qualitative mapping alone.  Rogelio Oliva and I wrote a rejoinder (SDR 2001, 17:4) that challenged this idea and sought to reclaim Jay Forrester’s original view of simulation as the necessary test bed for any dynamic hypothesis that might conceivably lead to policy decisions.

Even now, 20 years later, the debate continues.  We see in our conferences increasing use of Group Model Building that goes no further than a qualitative map, and from which the authors claim to have derived dynamic insights.

Some have been dismayed by this development—which appears to be the further expansion of Systems Thinking—fearing it is diluting and dumbing down the fundamentals of our field.  The modelers accuse the mappers of lacking rigor, while the mappers say that good group process has a rigor of its own.

What can we do about this long-brewing pot of trouble?

  Is there any way out of the impasse?

I’d like to suggest a possibility.  Several years ago, I published a paper (SDR 2014, 30:1-2) describing how a “levels of evidence” approach—a standard for classifying work in the biomedical sciences—might be applied in SD.  To achieve an “A” level of evidence, one would need both strong structural and behavioral evidence and the ability to reliably test one’s model.

  Structural evidence comes from conversations with subject matter experts and focused studies of cause and effect.  Behavioral evidence comes from a comparison of model output with time series data and historical records.   Work with strong support for structure but not behavior, or behavior but not structure, would achieve at best a “B” rating.  Work with strong support for neither would get a “C” rating.

In the biomedical sciences, works that have “B” or “C” ratings can still be presented at conferences and even appear in prestigious journals.  A rating less than “A” does not mean poor quality but rather lack of full, iron-clad reliability for drawing conclusions and making decisions; that is, something more exploratory or tentative.  Its level of evidence is designated in the conference proceedings or at the top of the paper so that the audience knows what they are dealing with—a decisive work (one from which decisions can be made with confidence) or something less than that.

It seems to me we can apply the Levels of Evidence filter objectively across both simulation models and qualitative maps.  Let’s start with the simple fact that a simulation model can be tested formally, while a qualitative map cannot.  It follows that the best possible rating for a simulation model is “A”, while the best possible for a qualitative map is “B”.

If we can agree on this much, then we may be able to find a way for simulation models and qualitative maps to coexist.  It would require acknowledgment from the advocates of qualitative maps that their work can never be considered decisive.  And, it would require acknowledgment from the advocates of simulation that a model lacking sufficient evidence may be no more reliable than a well-developed qualitative map.

Recent Posts

Society Governance Updates

Society Governance Updates Welcome, Allyson! New President Allyson Beall King joined the Policy Council as our 2024 President. Her primary role is as director of the Washington State University School of the Environment, which focuses on regional ecologies and our...

Call for Presenters: Seminar Series

Call for Presenters: Seminar Series We at the System Dynamics Society are continually seeking vibrant and knowledgeable presenters for our ongoing Seminar Series. As we unfold the calendar, there’s always a place for more insights, experiences, and expertise to enrich...

Upcoming Events

Recent Business cases

A Design Value Calculator: A System Dynamics Boardgame

A Design Value Calculator: A System Dynamics Boardgame EXECUTIVE Summary Product design is a specific form of complex innovation that touches all areas of an organization’s management. While entrepreneurs recognise the value of design, they often tend to focus...

Join us