For Better Estimation
Nate Silver teaches us how signal and noise can be confounded in magical ways, and how we are trying our best to get the best signals out, even during the fall of 2016 and 2020. Data, and in particular big data, are always a treasure to help people establish better understanding of our life and the life of others in the world (if we care). Except
when they are not.
We cannot for a single second blame the error-prone yet painstaking efforts of data collectors: they risk their lives to keep account how many nasal swab samples are tested positive, how many second-dose vaccines are being injected, and what is the virus concentration of tap water specimens – they simply are the most respectable warriors. It is not their job to separate signal from noise, and in fact, sometimes
they really should not waste their time to do that – if not many of them are doing the same thing in the same way.
Data analysts and modelers as we are, who sit at the back-end on the field of the battle against, say, against COVID (in fact, any data-driven estimation task has an enemy – the unknown), are obliged to never squander the efforts of first-row warriors. It is us who must bring out the best cuisine with raw data, upon which life-saving policies can be made.
Cooking is not easy; and in many cases, something could be skewed. Unconsciously. In Chinese cuisine, a dish is evaluated in three dimensions: look, smell, and taste – it is very probable that a cook may fail to deliver the best taste, the core of a dish, because he’s focusing too much on look and smell. Such may happen to a data cook: one might claim to yield a very important estimate of the basic reproduction number of COVID by using a very complex while realistic model (even that the results have quite narrow confidence intervals), yet the estimation scheme he adopted might be left unchecked.
We believe that the choice of estimation scheme plays a non-trivial role in parameter estimation. Which mold are you using determines what you produce. And for the estimation of comprtment models, one failure mode makes things wrong: conforming to norms. That means adopting a least squares estimation (summing over the squares of the difference between data series and model series, and then minimizing the sum; that really seems right, isn’t it) is not only easy, but also safe.
As one shouldn’t stay long in the comfort zone, we try to take a step out; we find that, NO, least squares is not a reliable estimation scheme for noisy data. In this study, we test a panel of advanced estimation schemes, and discover that the performance of least squares can be improved by a substantial margin. This means that policy recommendations based on least squares estimation results may be less accurate than we hope they are.
What’s the alternative? Well, we don’t know the optimal scheme for sure, but some solutions might be helpful. The negative binomial likelihood performs well across a range of conditions, so does the technique of Kalman Filtering. If these two methods seem too complicated (as they sound), then even a simple scaling of residual’s variance could largely remedy least squares. These are better molds for an equipped modeler
in simulation studies.
But these techniques do not, still, guarantee the complete removal of bias in parameter estimates. For one thing, as there is no golden mold that is one-size-fits-all, there is no perfect estimation scheme that best suits all data conditions. For another, estimation techniques are always second-order: you’ve got to first have a good model. As all models are wrong, no model estimate will be perfectly right.
This is not a doomsday call, though; this is calling on us to keep going in bringing out the best from data. We should be as painstaking as the front-line workers, and play with our own model specimens. For sure, we are as error-prone as they are – and we may be making even more errors – but just keep trying.
Li, Rahmandad, and Sterman are authors of “Improving Parameter Estimation of Epidemic Models: Likelihood Functions and Kalman Filtering”, winner of the Dana Meadows Award at the 39th ISDC. If registered, you can see the presentation of their work until August 31st 2021 here.
Book Signing & Book Swap at ISDC 2023
Book Signing & Book Swap at ISDC 2023 Learn about their books directly from the authors! Don’t miss out! Several book authors will join us at the Book Signing event at ISDC 2023. This event will be held live at the conference in Chicago, on Tuesday, July 25 at...
Webinar Highlights & Recording: Digital Twin Business Models for Strategy and Operational Management
Webinar Highlights & Recording: Digital Twin Business Models for Strategy and Operational Management We had the opportunity to delve into the world of business model simulations with industry experts Kim Warren, founder of Strategy Dynamics, and Martin Lycko,...
Webinar Highlights & Recording: System Dynamics Modeling with R.
Webinar Highlights & Recording: System Dynamics Modeling with R. System Dynamics is a powerful approach to understanding complex systems in various fields, such as public health, ecology, and economics. Prof. Jim Duggan, an expert in applying System Dynamics and...
6th Annual System Dynamics Competition by the South Africa System Dynamics Chapter
System Dynamics Competition CORRECT TIME: 2 PM CENTRAL EUROPE The System Dynamics Society’s South Africa Chapter is hosting a System Dynamics Competition that offers participants the opportunity to learn new modeling skills directly from experts, collaborate...
Recent Business cases
System Dynamics Helps Farmers Escape Poverty Trap in Guatemala
System Dynamics Helps Farmers Escape Poverty Trap in Guatemala EXECUTIVE Summary Guatemala holds the 4th highest global ranking for chronic malnutrition, and climate change is intensifying the challenges subsistence farmers face in providing food for their families....
Twinings Uses System Dynamics Games to Enhance HR Capability
Twinings Uses System Dynamics Games to Enhance HR Capability “Realistic simulation is a powerful approach to building capability. The business simulation developed [by Dashboard Simulations and Lane4] gave [Twinings staff] an experience that called for them to develop...
RSC Uses System Dynamics to Increase HVACR Sales Against the Tide
RSC Uses System Dynamics to Increase HVACR Sales Against the Tide “Using the proven Strategy Dynamic process focused our limited resources on organizing strategic issues, identifying the critical resources, and developing the insight to more rapidly create intuitive...