Generic selectors
Exact matches only
Search in title
Search in content
Search in posts
Search in pages
System Dynamics Blog


For Better Estimation

For Better Estimation

by | Aug 13, 2021

Nate Silver teaches us how signal and noise can be confounded in magical ways, and how we are trying our best to get the best signals out, even during the fall of 2016 and 2020. Data, and in particular big data, are always a treasure to help people establish better understanding of our life and the life of others in the world (if we care). Except
when they are not.

We cannot for a single second blame the error-prone yet painstaking efforts of data collectors: they risk their lives to keep account how many nasal swab samples are tested positive, how many second-dose vaccines are being injected, and what is the virus concentration of tap water specimens – they simply are the most respectable warriors. It is not their job to separate signal from noise, and in fact, sometimes
they really should not waste their time to do that – if not many of them are doing the same thing in the same way.

Data analysts and modelers as we are, who sit at the back-end on the field of the battle against, say, against COVID (in fact, any data-driven estimation task has an enemy – the unknown), are obliged to never squander the efforts of first-row warriors. It is us who must bring out the best cuisine with raw data, upon which life-saving policies can be made.

Cooking is not easy; and in many cases, something could be skewed. Unconsciously. In Chinese cuisine, a dish is evaluated in three dimensions: look, smell, and taste – it is very probable that a cook may fail to deliver the best taste, the core of a dish, because he’s focusing too much on look and smell. Such may happen to a data cook: one might claim to yield a very important estimate of the basic reproduction number of COVID by using a very complex while realistic model (even that the results have quite narrow confidence intervals), yet the estimation scheme he adopted might be left unchecked.

We believe that the choice of estimation scheme plays a non-trivial role in parameter estimation. Which mold are you using determines what you produce. And for the estimation of comprtment models, one failure mode makes things wrong: conforming to norms. That means adopting a least squares estimation (summing over the squares of the difference between data series and model series, and then minimizing the sum; that really seems right, isn’t it) is not only easy, but also safe.

As one shouldn’t stay long in the comfort zone, we try to take a step out; we find that, NO, least squares is not a reliable estimation scheme for noisy data. In this study, we test a panel of advanced estimation schemes, and discover that the performance of least squares can be improved by a substantial margin. This means that policy recommendations based on least squares estimation results may be less accurate than we hope they are.

What’s the alternative? Well, we don’t know the optimal scheme for sure, but some solutions might be helpful. The negative binomial likelihood performs well across a range of conditions, so does the technique of Kalman Filtering. If these two methods seem too complicated (as they sound), then even a simple scaling of residual’s variance could largely remedy least squares. These are better molds for an equipped modeler
in simulation studies.

But these techniques do not, still, guarantee the complete removal of bias in parameter estimates. For one thing, as there is no golden mold that is one-size-fits-all, there is no perfect estimation scheme that best suits all data conditions. For another, estimation techniques are always second-order: you’ve got to first have a good model. As all models are wrong, no model estimate will be perfectly right.

This is not a doomsday call, though; this is calling on us to keep going in bringing out the best from data. We should be as painstaking as the front-line workers, and play with our own model specimens. For sure, we are as error-prone as they are – and we may be making even more errors – but just keep trying.


Li, Rahmandad, and Sterman are authors of “Improving Parameter Estimation of Epidemic Models: Likelihood Functions and Kalman Filtering”, winner of the Dana Meadows Award at the 39th ISDC. If registered, you can see the presentation of their work until August 31st 2021 here.

Recent Posts

Using System Dynamics to Teach and Learn about COVID-19

Using System Dynamics to Teach and Learn about COVID-19 This Webinar is free due to the generous contribution of the University at Albany and California State University, Chico The COVID Pandemic constituted a dynamic and complex problem challenging leaders and...

Economics SIG News: Summer 2022

Economics SIG News: Summer 2022 Summer 2022 Events Tyrone Keynes on “The Impact on National Accounts from NPI’s: Economic Pandemic Model” May 12th, 2022 Noon – 1PM (New York) Demonstrating the effects on the national and regional accounts of...

New Horizons of Systems Science

New Horizons of Systems Science This Seminar was sponsored by the International Council on Systems Engineering (INCOSE) Systems theory is developing to include new perspectives with a focus on integrated and inclusive transdisciplinary system approaches. This panel...

Upcoming Events

WPI System Dynamics Peer Tutoring Session

WPI System Dynamics Peer Tutoring Session

In this Collective Learning Meeting (CLM), WPI System Dynamics will host a Peer Tutoring Session Peer Tutoring Sessions are open sessions where attendees network and discuss systems and simulation. Please come to this CLM if you have questions about: Modeling and...

Recent Business cases

General Motors OnStar

Name The General Motors OnStar Project Modelers Vince Barabba, Chet Huber, Fred Cooke, Nick Pudar, Jim Smith, Mark Paich Client General Motors Client Type Corporation The Official Website is the official website in which you can become a member, get...

Pharmaceutical Product Branding Strategies

Pharmaceutical Product Branding Strategies Name Pharmaceutical Product Branding Strategies — Simulating Patient Flow and Portfolio Dynamics Modelers Mark Paich, Corey Peck, Jason Valant Contact Jason Valant or Corey Peck Client Numerous Pharmaceutical Companies Client...

Join us

Tianyi Li
Tianyi Li