Nate Silver teaches us how signal and noise can be confounded in magical ways, and how we are trying our best to get the best signals out, even during the fall of 2016 and 2020. Data, and in particular big data, are always a treasure to help people establish better understanding of our life and the life of others in the world (if we care). Except
when they are not.

We cannot for a single second blame the error-prone yet painstaking efforts of data collectors: they risk their lives to keep account how many nasal swab samples are tested positive, how many second-dose vaccines are being injected, and what is the virus concentration of tap water specimens – they simply are the most respectable warriors. It is not their job to separate signal from noise, and in fact, sometimes
they really should not waste their time to do that – if not many of them are doing the same thing in the same way.

Data analysts and modelers as we are, who sit at the back-end on the field of the battle against, say, against COVID (in fact, any data-driven estimation task has an enemy – the unknown), are obliged to never squander the efforts of first-row warriors. It is us who must bring out the best cuisine with raw data, upon which life-saving policies can be made.

Cooking is not easy; and in many cases, something could be skewed. Unconsciously. In Chinese cuisine, a dish is evaluated in three dimensions: look, smell, and taste – it is very probable that a cook may fail to deliver the best taste, the core of a dish, because he’s focusing too much on look and smell. Such may happen to a data cook: one might claim to yield a very important estimate of the basic reproduction number of COVID by using a very complex while realistic model (even that the results have quite narrow confidence intervals), yet the estimation scheme he adopted might be left unchecked.

We believe that the choice of estimation scheme plays a non-trivial role in parameter estimation. Which mold are you using determines what you produce. And for the estimation of comprtment models, one failure mode makes things wrong: conforming to norms. That means adopting a least squares estimation (summing over the squares of the difference between data series and model series, and then minimizing the sum; that really seems right, isn’t it) is not only easy, but also safe.

As one shouldn’t stay long in the comfort zone, we try to take a step out; we find that, NO, least squares is not a reliable estimation scheme for noisy data. In this study, we test a panel of advanced estimation schemes, and discover that the performance of least squares can be improved by a substantial margin. This means that policy recommendations based on least squares estimation results may be less accurate than we hope they are.

What’s the alternative? Well, we don’t know the optimal scheme for sure, but some solutions might be helpful. The negative binomial likelihood performs well across a range of conditions, so does the technique of Kalman Filtering. If these two methods seem too complicated (as they sound), then even a simple scaling of residual’s variance could largely remedy least squares. These are better molds for an equipped modeler
in simulation studies.

But these techniques do not, still, guarantee the complete removal of bias in parameter estimates. For one thing, as there is no golden mold that is one-size-fits-all, there is no perfect estimation scheme that best suits all data conditions. For another, estimation techniques are always second-order: you’ve got to first have a good model. As all models are wrong, no model estimate will be perfectly right.

This is not a doomsday call, though; this is calling on us to keep going in bringing out the best from data. We should be as painstaking as the front-line workers, and play with our own model specimens. For sure, we are as error-prone as they are – and we may be making even more errors – but just keep trying.

Li, Rahmandad, and Sterman are authors of “Improving Parameter Estimation of Epidemic Models: Likelihood Functions and Kalman Filtering”, winner of the Dana Meadows Award at the 39th ISDC. If registered, you can see the presentation of their work until August 31st 2021 here.