Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Search in posts
Search in pages

Go Home

System Dynamics Blog

SYSTEM DYNAMICS BLOG

For Better Estimation

For Better Estimation

by | Aug 13, 2021

Nate Silver teaches us how signal and noise can be confounded in magical ways, and how we are trying our best to get the best signals out, even during the fall of 2016 and 2020. Data, and in particular big data, are always a treasure to help people establish better understanding of our life and the life of others in the world (if we care). Except
when they are not.

We cannot for a single second blame the error-prone yet painstaking efforts of data collectors: they risk their lives to keep account how many nasal swab samples are tested positive, how many second-dose vaccines are being injected, and what is the virus concentration of tap water specimens – they simply are the most respectable warriors. It is not their job to separate signal from noise, and in fact, sometimes
they really should not waste their time to do that – if not many of them are doing the same thing in the same way.

Data analysts and modelers as we are, who sit at the back-end on the field of the battle against, say, against COVID (in fact, any data-driven estimation task has an enemy – the unknown), are obliged to never squander the efforts of first-row warriors. It is us who must bring out the best cuisine with raw data, upon which life-saving policies can be made.

Cooking is not easy; and in many cases, something could be skewed. Unconsciously. In Chinese cuisine, a dish is evaluated in three dimensions: look, smell, and taste – it is very probable that a cook may fail to deliver the best taste, the core of a dish, because he’s focusing too much on look and smell. Such may happen to a data cook: one might claim to yield a very important estimate of the basic reproduction number of COVID by using a very complex while realistic model (even that the results have quite narrow confidence intervals), yet the estimation scheme he adopted might be left unchecked.

We believe that the choice of estimation scheme plays a non-trivial role in parameter estimation. Which mold are you using determines what you produce. And for the estimation of comprtment models, one failure mode makes things wrong: conforming to norms. That means adopting a least squares estimation (summing over the squares of the difference between data series and model series, and then minimizing the sum; that really seems right, isn’t it) is not only easy, but also safe.

As one shouldn’t stay long in the comfort zone, we try to take a step out; we find that, NO, least squares is not a reliable estimation scheme for noisy data. In this study, we test a panel of advanced estimation schemes, and discover that the performance of least squares can be improved by a substantial margin. This means that policy recommendations based on least squares estimation results may be less accurate than we hope they are.

What’s the alternative? Well, we don’t know the optimal scheme for sure, but some solutions might be helpful. The negative binomial likelihood performs well across a range of conditions, so does the technique of Kalman Filtering. If these two methods seem too complicated (as they sound), then even a simple scaling of residual’s variance could largely remedy least squares. These are better molds for an equipped modeler
in simulation studies.

But these techniques do not, still, guarantee the complete removal of bias in parameter estimates. For one thing, as there is no golden mold that is one-size-fits-all, there is no perfect estimation scheme that best suits all data conditions. For another, estimation techniques are always second-order: you’ve got to first have a good model. As all models are wrong, no model estimate will be perfectly right.

This is not a doomsday call, though; this is calling on us to keep going in bringing out the best from data. We should be as painstaking as the front-line workers, and play with our own model specimens. For sure, we are as error-prone as they are – and we may be making even more errors – but just keep trying.

 

Li, Rahmandad, and Sterman are authors of “Improving Parameter Estimation of Epidemic Models: Likelihood Functions and Kalman Filtering”, winner of the Dana Meadows Award at the 39th ISDC. If registered, you can see the presentation of their work until August 31st 2021 here.

Recent Posts

How to Sell System Dynamics (Or Anything Else)

How to Sell System Dynamics (Or Anything Else) If it’s so good – why is System Dynamics so hard to sell? The key to selling System Dynamics has nothing to do with what we’ve learned in grad school. Instead, the key is to clarify a fundamental confusion everyone seems...

How Food and System Dynamics Gave me A Career

How Food and System Dynamics Gave me A Career A discussion of two System Dynamics projects that had some real impact and then reflect on how this happened, and what needs to be in place for us system dynamicists to have an impact. Papers mentioned: Transforming Food...

Upcoming Events

Recent Business cases

Twinings Uses System Dynamics Games to Enhance HR Capability

Twinings Uses System Dynamics Games to Enhance HR Capability “Realistic simulation is a powerful approach to building capability. The business simulation developed [by Dashboard Simulations and Lane4] gave [Twinings staff] an experience that called for them to develop...

RSC Uses System Dynamics to Increase HVACR Sales Against the Tide

RSC Uses System Dynamics to Increase HVACR Sales Against the Tide “Using the proven Strategy Dynamic process focused our limited resources on organizing strategic issues, identifying the critical resources, and developing the insight to more rapidly create intuitive...

Achieving a Polio-Free World Through System Dynamics Simulation

Achieving a Polio-Free World Through System Dynamics Simulation EXECUTIVE Summary This System Dynamics model underpinned a 192 country resolution to eradicate polio globally and led the Bill and Melinda Gates Foundation to give Rotary International $100 million to...

Join us