Multi-model and multi-team ensemble forecasts have become widely used to generate reliable short-term forecasts. Whilst there is robust and consistent evidence that multi-team ensemble forecasts provide reliable and performant forecasts across domains they also have a range of downsides. The most significant is the difficulty in interpreting them. Here we develop a parsimonious forecast model based on observations of ensemble behaviour and evaluate its predictive performance.
Light the beacons - we have a new preprint
You can read the paper here. If interested in the code repository you can check it out here. I particularly recommend the commit messages as they give a fun window into the writing process. For the kind of people that just want model code check it out here.
Over the last few years we have produced a lot of forecasts. In particular, we have submitted weekly to US, European, and German/Poland Forecast Hubs using a range of models for an extended period of time2. An interesting aspect of submitting to these platforms is that your forecast becomes part of another product, the forecast ensemble. This is a good thing as ensembles are typically more robust than forecasts from single models and in many cases give better performing forecasts. However, they can be very hard to learn from, and though we have really tried to learn more about how to forecast it has been difficult.
In the summer of 2021, I was getting a little frustrated at the lack of progress improving our forecasting approaches. It felt like we needed to go back to the basics of the models we were using in order to learn things. I was also seeing a lot of discussion about the use of genomic data for surveillance and wanted to do some investigation. As great minds think alike, Johannes Bracher had recently developed a short-term forecasting model that included variant dynamics. I shamelessly stole this3 and reimplemented it in a modern PPL4 - creating the forecast.vocs R package. Everything was steaming ahead to do a large-scale evaluation of using sequences for short-term case forecasts5. Unfortunately, storm clouds were brewing.
In the winter of 2021 we started to see news of the Omicron variant with the first reports coming out of South Africa6. This looked like bad news. Several of the authors of this study helped pivot the forecast.vocs R package to start looking at the transmission advantage of this variant, as well as trying to estimate if it was changing over time (as an indicator of immune escape), and to produce short-term case forecasts7. We were able to do this as, unlike forecast ensembles, our model was very simple and easy to modify.
Sadly, I like many people then proceeded to catch COVID-198. This meant I was consigned to the house with really very little to do whilst cooling down from a very intense work period9. The natural thing to do was browse the various Forecast Hub websites and think about how the models were performing and trying to see if they were managing to deal with the rise of Omicron. After jotting down some notes about how I thought things were working I realised that the model we had been using for our Omicron work was perhaps a good test bench for seeing if my ideas for why the ensemble behaved the way it did were correct. Some intense hacking on the forecast.vocs model followed and this paper was born.
After this semi-delusional rampage some adults10 entered the picture and transformed this from a vague idea into a complete bit of work. It has been really great working with them all and I would thoroughly recommend it.
If interested in this you can use this to submit your forecasts to the European Forecast Hub: https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe-submissions↩︎
You can read more about some of these methods here: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010405↩︎
with some adaption perhaps↩︎
shots fired JAGs↩︎
We are now circling back to this and I think it should turn out to be pretty interesting. You can check out what is already in place and keep and eye out for more here: https://github.com/epiforecasts/evaluate-delta-for-forecasting↩︎
Great job SA!↩︎
See a version of the report here: https://github.com/epiforecasts/omicron-sgtf-forecast↩︎
On Christmas day from an extended family member - it was a little awkward.↩︎
Daily model updates and a several other related bits of work - tiring!↩︎
i.e. my lovely co-authors↩︎
If you see mistakes or want to suggest changes, please create an issue on the source repository.
For attribution, please cite this work as
Abbott (2022, Oct. 13). Sam Abbott: Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble. Retrieved from https://samabbott.co.uk/posts/2022-10-13-surrogate-ensemble-forecast/
BibTeX citation
@misc{abbott2022evaluating,
author = {Abbott, Sam},
title = {Sam Abbott: Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble},
url = {https://samabbott.co.uk/posts/2022-10-13-surrogate-ensemble-forecast/},
year = {2022}
}