With the inclusion of epi outcome as a covariate, @sbfnk raised the question if we should consider an interaction with model structure.
From @sbfnk :
the results don't quite square with what we report in Bosse et al., 2022 - where we found that humans were better at forecasting cases, but worse at forecasting deaths. This hints at a weakness in the structure of the GAMM in that it assumes models don't differ in their ability to forecast cases vs. deaths. That's justifiable with respect to model structure but I think given our previous findings it's not a good assumption once we include human judgement models. So I would suggest to add an interaction term s(Method, Epi_target, bs = "re") and then still plot cases/deaths separately (even if in a joint model). (I've tried it using your super-clear code and it does confirm the Bosse et al. finding whilst retaining everything else as it was).
We discussed several options:
- Keep as is.
- Include the interaction term
s(Method, Epi_target, bs = "re") .
This explicitly recognises the earlier human judgement finding. The explanation for human judgement outperforming computational models on cases was that humans were quicker to incorporate real-time information. This is very much in the spirit of the current paper in identifying a specific difference due to model type
- Include an interaction at the level of individual model, in the same way that we include an interaction between model and horizon.
This is more flexible for allowing any model to differ in performance between the two targets. Theoretically any computational model is an expression of human judgement, so in principle could reflect the hypothesised information difference / lagged effect. (Even if in practice modellers don't/can't update a computational model as rapidly as their own judgement.)
Plus another option:
- Reclassify the human judgement forecasts under an "Ensemble" model type.
I'm not hugely persuaded by classifying the human judgement submission as separate model type here. In these submissions, the sample was of modellers with some model input; and especially because the submissions here are legitimately an ensemble of 2-10 human judgement forecasts. To me this seems more analogous to an ensemble of models than an identifiably different model type. (If we did this, possibly we could then add back in the actual Hub ensemble, which we currently exclude.)
I think my preference is for (2) and (4) together, but I'm not strongly invested in (4) and open to views. @sbfnk would be glad to hear your thoughts.
While thinking this over, I had a few tangential notes.
- I'm generally a bit sceptical of interpreting the difference in human judgement vs model performance here, because it the original conclusion was based on a small biased sample of both humans and models, for a very small sample of targets over a short window.
Reading the paper again I'm struck by how similar the human judgement performance is to the Hub ensemble. From a quick look at the plots, coverage is very similar to Hub ensemble, but with much narrower dispersion for cases (not deaths). I wondered if this might be enough to explain the overall case/death differential.
- We could consider a different/later analysis which specifically unpicks why human judgement is better at one than another target. Not a developed idea, but noting the thought here anyway.
This could look at variables that might be informing the domain knowledge that explains why human judgement is more responsive to cases (e.g. a policy tracker, data revision history, ... ); adjusting for these to suggest what information makes the difference.
- This all makes me think we could take a more principled/structured approach to re-running something similar for e.g. winter flu to see if that result really holds.
With the inclusion of epi outcome as a covariate, @sbfnk raised the question if we should consider an interaction with model structure.
From @sbfnk :
We discussed several options:
s(Method, Epi_target, bs = "re").This explicitly recognises the earlier human judgement finding. The explanation for human judgement outperforming computational models on cases was that humans were quicker to incorporate real-time information. This is very much in the spirit of the current paper in identifying a specific difference due to model type
This is more flexible for allowing any model to differ in performance between the two targets. Theoretically any computational model is an expression of human judgement, so in principle could reflect the hypothesised information difference / lagged effect. (Even if in practice modellers don't/can't update a computational model as rapidly as their own judgement.)
Plus another option:
I'm not hugely persuaded by classifying the human judgement submission as separate model type here. In these submissions, the sample was of modellers with some model input; and especially because the submissions here are legitimately an ensemble of 2-10 human judgement forecasts. To me this seems more analogous to an ensemble of models than an identifiably different model type. (If we did this, possibly we could then add back in the actual Hub ensemble, which we currently exclude.)
I think my preference is for (2) and (4) together, but I'm not strongly invested in (4) and open to views. @sbfnk would be glad to hear your thoughts.
While thinking this over, I had a few tangential notes.
Reading the paper again I'm struck by how similar the human judgement performance is to the Hub ensemble. From a quick look at the plots, coverage is very similar to Hub ensemble, but with much narrower dispersion for cases (not deaths). I wondered if this might be enough to explain the overall case/death differential.
This could look at variables that might be informing the domain knowledge that explains why human judgement is more responsive to cases (e.g. a policy tracker, data revision history, ... ); adjusting for these to suggest what information makes the difference.