Chapter 5 Reassessing the evidence for universal school-age BCG vaccination in England and Wales

5.1 Introduction

Prior to the change in BCG vaccination policy in 2005 (see Chapter 2) several studies were carried out to assess the impact of any potential policy change. In this Chapter, I aim to update one of these studies.

I will recreate a previous approach for estimating the impact of ending the BCG schools scheme in England and Wales, updating the model with parameter uncertainty, and measurement error. I investigate scenarios considered by the Joint Committee on Vaccination and Immunisation (JCVI), and explore new approaches using notification data (see Chapter 4). I will estimate the number of vaccines needed to prevent a single notification, and the average annual additional notifications caused by ending the BCG schools’ scheme. This work was adapted from a preprint¹⁷ supervised by Hannah Christensen and Ellen Brooks-Pollock.

5.2 Background

The Bacillus Calmette–Guérin (BCG) vaccine remains the only licensed vaccine for use against Tuberculosis (TB), although its use globally is controversial due to evidence of variable effectiveness,[25] and waning protection 10-15 years after vaccination.[28] Global usage of the BCG varies between no vaccination, universal vaccination, and high-risk group vaccination and may target either neonates or school-aged children.[5,61] The World Health Organization (WHO) recommends vaccination for all neonates as early as possible after birth in high burden settings, with vaccination in low burden settings being dependent on the country specific epidemiology of TB.[39] This recommendation is based on the strong evidence that the BCG is highly protective in children,[23,24] whilst its effectiveness has been shown to vary with latitude when given later in life.[40]

In England and Wales, universal school-aged (at 13-14 years old) vaccination (hereafter referred to as the BCG schools scheme) was introduced after a MRC trial in the 1950s estimated BCG’s effectiveness at 78% in the ethnic White UK born population.[26] The policy remained in place until 2005, when England and Wales changed to targeted vaccination of high-risk neonates. The 2005 change in BCG vaccination policy was motivated by evidence of decreased transmission of TB, an increasing proportion of TB cases occurring in the non-UK born,[2] and modelling evidence that suggested stopping the BCG schools scheme would have minimal long term effects on incidence rates.[41] Due to the complex nature of both TB and the BCG vaccine, the ongoing impact of this change in policy is hard to directly estimate, with decision makers relying on expert opinion, evidence from surveillance data, and insight from modelling studies.

In 1987, an assessment of the school-age vaccination program was carried out in England and Wales.[41] This study was used, combined with a sensitivity analysis of notification rates, as supporting evidence by the Joint Committee on Vaccination and Immunisation (JCVI) BCG subgroup for the change in vaccination policy.[62,63] This chapter aims to re-evaluate this modelling, and re-estimate the predicted impact of stopping the schools scheme. Re-evaluating this work allows for the strength of the evidence used in decision making to be assessed and may highlight any issues with the approach used.

5.3 Methods

5.3.1 Modelling the impact of ending the BCG schools scheme

I implemented, and updated, Sutherland et al.’s model for estimating the impact of ending the BCG schools scheme, which is outlined briefly below.[41] This model was based on data from TB notification surveys conducted in 1973, 1978, and 1983.[64] These were used to estimate incidence rates, stratified by BCG vaccination status, in the ethnic White UK born population of England and Wales aged 15-19 years old, 20-24 years old and 25-29 years old. Future incidence rates were forecast by assuming an annual decrease in incidence rates, which was based on historic trends.[41,65] Primary impacts from ending the schools scheme, including the number of vaccines required to prevent a single notification, were estimated by calculating the difference in incidence rates between the vaccinated and unvaccinated populations. Additional notifications from TB transmission were then calculated using a transmission chain model and combined with the primary impact estimates, to calculate the number of annual additional notifications due to ending BCG vaccination. Based on data availability, the model used a 5-year time step.

5.3.1.1 Original estimation of notification rates

The effectiveness of the BCG vaccine was originally estimated by an MRC trial in 1953 at 78% in the United Kingdom.[26] As a follow up to this trial members of the MRC bio-statistics unit conducted a series of notification surveys attempting to ascertain any change in effectiveness, as well as acting as an estimate of notification rates across different demographics.[64] Surveys of those aged 15-24 years were carried out at 5-year intervals in 1973, 1978 and 1983 in England and Wales. For the 1983 survey records of BCG status, Tuberculin status and ethnicity were extracted from the records of notifying physicians and the records of the local health and education authorities. Total notifications across the study period were then aggregated for the following groups: Tuberculin negative and BCG vaccinated, Tuberculin negative and BCG unvaccinated, Tuberculin positive and not vaccinated and those who did not participate. These totals were then combined with the population estimates for each cohort at 13 years of age to estimate the ethnic make up of the population, and to construct notification rates for each category. Data were drawn from a range of sources including: Office of National Statistics data; annual local authority returns for total Tuberculin test results; BCG vaccinations in the schools scheme; and the Labour force survey (1983).

For 1983, there were 874 notifications recorded in 15-24 year old ethnic White UK born persons in England and Wales; survey participation was 80%. As the number of Tuberculin negative subjects not given BCG was unreported the number of notifications was estimated at 1.9% of those vaccinated with the BCG.[64] See [64] for full details of the survey and the additional assumptions used to give similar estimates for both the 1973 and 1978 surveys. The findings of these surveys were as follows: in the ethnic White population notification rates had fallen by an annual average of 9% and BCG efficacy had remained high.[41,65]

Evidence suggests that the BCG vaccine has a high efficacy for at least the first 15 years after vaccination, therefore Sutherland et al. extrapolated from the data on the 20-24 cohort to a theoretical 25-29 year old cohort. Data on the notifications in 25-29 year olds was available for the first 6 months of the 1983 survey and this was then scaled up to a yearly estimate using the ratio of notifications from this age group against the total number of notifications recorded in that year. Population estimates for the 25-29 year old cohort were based on data from the 20-24 cohort adjusted for all causes of mortality (0.34%). Migration was ignored. The Tuberculin positive cohort had a sharp decline in the previous two age cohorts, therefore it was assumed that this continued. Lastly, the efficacy was estimated as being that seen in the 20-24 cohort but with the same decline in protection seen between the last two cohorts. These assumptions allowed notification rates to be estimated for the 25-29 year old population, resulting in a complete cohort over the projected 15 years of BCG effectiveness.

5.3.1.2 Original construction of forward estimates

Based on these estimated notification rates, Sutherland et al. then sought to quantify the ongoing risk of developing notified TB, projected forward in time, for both the vaccinated and unvaccinated populations. To construct these estimates several key assumptions, based on the results seen in the previous surveys, were made. Firstly, it was assumed that efficacy was not degrading within the ethnic White population and therefore historic estimates would continue to apply into the future. Additionally, it was assumed that the annual decay of 9% in notification rates, across all ethnic White populations, would continue indefinitely.

These assumptions allowed the notification rates in both the BCG vaccinated and unvaccinated groups to be projected forward in time. By assuming that the schools scheme is responsible for the observed variation between vaccinated and unvaccinated rates the rate of prevented cases can then be estimated. By scaling this against a cohort of 100,000 13 year old individuals, the number of prevented cases over a 15-year period can be projected for each cohort. By dividing the total number in a given cohort by the number of prevented cases the estimated number of vaccines required to prevent a single case in the 15-year period can then be calculated.

To estimate the total number of prevented notifications, for each cohort, in England and Wales the total number receiving the BCG and the coverage of the schools scheme was required. The coverage of the BCG schools scheme was estimated from annual reports of the Department of Health and Social Security (DHSS) and was assumed to be 75% for all years. The number of BCG vaccines given each year was estimated from the DHSS returns for the years 1967 to 1981, it was then taken as 75% of the estimated ethnic White population aged 13 years from 1982-1996, for each 5-year period thereafter it was assumed to be 2.1 million.

Using the data on BCG coverage, the number of vaccines given each year, and the projected differences between vaccinated and unvaccinated notification rates allowed the number of prevented notifications, due to vaccination, for each age group to be found for each year. These estimates can then be used to give the total number of prevented notifications for those aged between 15-29 years. To understand these estimates, estimates of the projected yearly notifications if the scheme continues were required. These totals were derived from the vaccinated and unvaccinated rates supplemented with similar projections from the Tuberculin positive or otherwise ineligible sourced from the 1983 BCG survey.[64]

5.3.1.3 Original transmission chain model

Sutherland et al. defined their TB transmission model as follows:

1. The total expected number of secondary notifications (\(T\)) arising from any single primary notification was estimated as,

\[\begin{equation} T = (1-d)^z < 1 \end{equation}\]

Where \(d\) is the percentage annual decay in notification rates, and \(z\) is the average interval between the notification of any individual and the notification of the patient who infected them.

2. The total expected number of secondary notifications arising from any single primary notification (\(T\)) is related to the number of notifications in each generation using the relative generation size (\(x\)) with the following power series,

\[\begin{equation} T=x+x^2+x^3+ . . = \frac{x}{1-x} \end{equation}\]

3. The expected average interval between each primary notification and all secondary notifications (\(Z\)) is defined to be the sum of time to all notifications, weighted by the fraction in each generation, divided by the sum of all notifications.

\[\begin{equation} Z = \frac{xz + x^{2}2z + x^{3}3z + . .}{x+x^2+x^3+ . . }= \frac{z}{1-x} \end{equation}\]

Both 2. and 3. are only valid when \(x < 1\).

5.3.1.4 Updating the transmission chain model

If we assume a constant decay rate of \(d\)% per year and that the next generation of secondary cases are notified \(z\) years after the person who infected them, then 1 notification in year 0 will result in \((1-d)^z\) secondary notifications \(z\) years later. This is therefore the relative generation size (\(x\)), not the total number of secondary notifications (\(T\)), as (incorrectly) stated by Sutherland et al. Therefore, to correct this I revised 1. to the following,

\[\begin{equation} x = (1-d)^z < 1 \end{equation}\]

The Sutherland et al. model required several additional assumptions.[41] Firstly, as incidence rates for those ineligible for the BCG schools scheme are not published, I assumed that they were equal to those in the unvaccinated population. In addition, in order to reproduce the distribution of cases annually (rather than by generation or overall) reported in Sutherland et al. I introduced an additional model step and parameter; the proportion of secondary cases in the first generation that occurred in the first year (\(f\)). This modelling step is only required to reproduce the final table from [41] and does not impact estimates of the impact of ending the BCG schools scheme. It is included only for validation purposes.

The annual distribution of secondary notifications (\(N\)) was modelled by first estimating the number of secondary notifications that occurred in the current year (\(i\)) due to primary notifications in that year (\(N^{\text{Current}}\)) and then estimating how many secondary notifications occurred 5 years later (\(N^{\text{Projected}}\)). \(N^{\text{Current}}\) was estimated using the number of primary notifications (\(P\)) multiplied by the number of total expected number of secondary notifications per primary notification (\(T\)), the proportion of secondary cases in the first generation that occurred in the first year (\(f\)), and the relative size of the first generation (\(x\)). \(N^{\text{Projected}}\) was then estimated by assuming that it was equal to the number of secondary notifications, minus notifications occurring in the first year, that occurred \(Z\) (the expected average interval between each primary notification and all secondary notifications) years ago. As this used the overall number of notifications from the previous time step a decay of \((1 - d)^{5-Z}\) was applied. This approach can be summarised as follows,

\[\begin{equation} N^{\text{Current}} = P T f x \end{equation}\]

\[\begin{equation} N^{\text{Projected}} = (PT - N^{\text{Current}})(1 - d)^{5-Z} \end{equation}\]

\[\begin{equation} N_i = N^{\text{Current}}_i + N^{\text{Projected}}_{i-1} \end{equation}\]

I fitted the proportion of secondary cases in the first generation that occurred in the first year (\(f\)) using least squares to the original estimates of the total notifications due to ending the scheme under several scenarios, for several years. I validated the fitted model by comparing the results with those from the original implementation using the mean absolute percentage error, normalised by the original estimate, as the performance metric.

5.3.2 Updating model parameter estimates and incorporating parameter uncertainty

Incidence rates were included as point estimates in [41]; in the updated model I included uncertainty in these rates. I did this by first estimating notifications for 1973, 1978, and 1983, using published incidence rates and population estimates. Samples were then generated using a Poisson distribution.[41,64] These samples were then used to estimate a distribution of incidence rates to replace the point estimates used in the original analysis. Sutherland et al. assumed a serial interval (\(z\)) of 2 years between linked notifications. Using a newly available literature source I updated this assumption with an estimate of 1.44 (95% CI 1.29 to 1.63) years.[66]

I considered the original assumption of a 9% annual decrease in incidence rates as well as three scenarios based on those considered by the JCVI BCG subgroup:[62,63] these were a 3.9% decrease, a 1.9% decrease, and no change annual in incidence rates. Data on the annual decrease in incidence rates in the ethnic White UK population were not available so I used two proxy measures. The first proxy measure was the annual change in notifications in England and Wales, which was estimated using data from Public Health England (PHE). The standard deviation of this measure was then calculated using the prop.test function in R.[56] The second proxy used was the annual decrease in the UK born age-specific incidence rates in the English population. These were calculated using notification data from the Enhanced TB surveillance system (ETS) and the June Labour Force Survey.[2] Incidence rates were estimated using the epiR package.[60] Uncertainty was incorporated by sampling from a normal distribution for both proxy measures. Data collection for the ETS began in 2000 and prior to this notification data was only available in years with notifications surveys (1973, 1978, and 1983). I therefore estimated incidence rates between 1984 and 1999, and for the years between notifications surveys (1974-1977 and 1979-1982), using locally estimated scatter plot smoothing (LOESS) regression fitted to incidence rates published in [11] and the estimated incidence rates from 2000 on-wards. LOESS is a local regression method that combines multiple regression models in a k-nearest neighbours meta-model.[67] This approach allows nonlinear trends to be fitted using a series of linear models. For years prior to 1973 the annual decreases were assumed to be the mean of the annual decreases from the previous 3 years. For both proxy measures the annual decreases in incidence rates post 2016 were assumed to be the average of the estimates in 2013-2015.

5.3.3 Statistical analysis

For each scenario, I ran the model for 69 years (1969-2028) with 10,000 parameter samples. I tested the difference between scenarios using the Mann-Whitney test for the number of vaccines needed to prevent a single case in 15 years after vaccination for a cohort aged 13-14 years old at vaccination. As in [41] a 15-year time horizon was used with 5-year intervals. The year closest to the year of the change in vaccination policy (2005), which had model estimates, was used as the baseline. The code for this analysis is available online¹⁸.

5.4 Results

The model produced results that were comparable with those from [41] (Table 5.1; Table 5.2). When estimating the total notifications from ending the BCG schools scheme at different times in ethnic White UK born adults aged 15-29 years old in England and Wales the model had a median absolute error of 2.03% (2.5, 97.5% Q: 0.00%, 3.72%) and a maximum absolute error of 3.91% when compared to [41]. I found that the percentage of cases in the first year was \(f=0.764\) when fitted to the Sutherland et al. estimates using the least squares method.

Table 5.1: Comparison of results published by Sutherland et al. vs. the recreated model. This table shows the total notifications including primary and secondary effects from ending the BCG schools scheme at various times in ethnic White adults aged 15-29 years old in England and Wales.

	1988			1993			1998
Year of Ending Scheme	Original	Recreated	Difference	Original	Recreated	Difference	Original	Recreated	Difference
1986	288	296	8	226	226	0	208	205	-3
1991	288	296	8	165	166	1	130	131	1
1996	288	296	8	165	166	1	90	91	1

Table 5.2: Continued: Comparison of results published by Sutherland et al. vs. the recreated model. This table shows the total notifications including primary and secondary effects from ending the BCG schools scheme at various times in ethnic White adults aged 15-29 years old in England and Wales.

	2003			2008			2013
Year of Ending Scheme	Original	Recreated	Difference	Original	Recreated	Difference	Original	Recreated	Difference
1986	181	175	-6	128	123	-5	80	78	-2
1991	128	126	-2	115	111	-4	80	78	-2
1996	77	76	-1	80	80	0	72	70	-2

5.4.1 Annual change in TB incidence rates

I found that the assumption of a 9% annual decrease in incidence rates in the ethnic White UK born was not comparable to estimates using either notification data or age-specific incidence rates in the time period studied (Figure 5.1). The median annual decrease estimated using notifications was 3.06% (2.5, 97.5% Quantiles (Q): -8.32%, 11.45%), with a maximum of 15.14% (2.5, 97.5% Q: 14.22%, 16.03%) in 1987 and a minimum of -10.17% (2.5, 97.5% Q: -10.82%, -9.52%) in 2005. Using age-specific incidence rates I estimated the median annual decrease in incidence rates for 15-19 year olds was 1.65% (2.5, 97.5% Q: -40.49%, 39.97%), 3.16% (2.5, 97.5% Q: -33.95%, 38.30%) for 20-24 year olds, and 2.63% (2.5, 97.5% Q: -36.28%, 37.17%) for 25-29 year olds. There was substantial variation between years and a high degree of uncertainty.

Figure 5.1: Annual percentage change in ethnic White UK born incidence rates for those aged 15-19, 20-24, and 25-29 years old under different scenarios. For the notification and incidence rate scenarios each line represents the median of 10,000 parameter samples.

5.5 Vaccines required to prevent a single notification

I found that incorporating uncertainty, did not alter the number of vaccines required to prevent a single notification within 15 years in a cohort vaccinated at school-age, when the annual decrease in incidence rates was assumed to be 9% (Figure 5.2; Table 5.3). However, the updated estimate had a wide range (15000 (2.5, 97.5% Q: 12000, 19000) vaccines required in 2004). As the assumed annual decrease in incidence rates was reduced the number of vaccines required to prevent a single notification also reduced. Assuming an annual decrease of 1.9% (one of the scenarios evaluated by the JCVI) resulted in an estimate of 1600 (2.5, 97.5% Q: 1300, 2000) vaccines required to a prevent a single notification in 2004. This assumption was the most comparable, although not equivalent, to estimates derived using notifications (1400 (2.5, 97.5% Q: 1100, 1700), P: 0.077) and age-specific incidence rates (1500 (2.5, 97.5% Q: 460, 4900), P: 0.083). The estimate using incidence rates had a high degree of uncertainty (Figure 5.2; Table 5.3). The number of vaccines required increased slightly over time with 1800 (2.5, 97.5% Q: 1500, 2200) required in 2009, 2000 (2.5, 97.5% Q: 1600, 2500) required in 2014, and 2200 (2.5, 97.5% Q: 1800, 2700) required in 2019 when an annual decrease of 1.9% in incidence rates was assumed.

Figure 5.2: Vaccines required in a cohort of those vaccinated at school-age to prevent a single case of Tuberculosis within 15 years of vaccination in 2004, 2009, 2014, or 2019. The years presented were dictated by the 5-year timestep of the model. The percentage annual decrease scenarios considered were based on those considered by the JCVI BCG subgroup, with the addition of a scenario using aggregate notification data and a scenario using estimates of age-specific incidence rates in the UK born. Each boxplot summarises the output of 10,000 model simulations for each scenario.

Table 5.3: The median number (with the 2.5% and 97.5% quantiles) of vaccines required to prevent a single case of TB within 15 years in a ethnic White UK born adult vaccinated at 13 years old. The percentage annual decrease scenarios considered were based on those considered by the JCVI BCG subgroup, with the addition of a scenario using aggregate notification data and a scenario using estimates of age-specific incidence rates in the UK born.

Year of Vaccination	9% decrease (original parameters)	9% decrease	3.9% decrease	1.9% decrease	0% decrease	Notifications	Incidence Rates
1969	460	460 (390, 540)	460 (390, 540)	460 (390, 540)	460 (390, 540)	460 (390, 540)	460 (390, 540)
1974	940	940 (780, 1200)	880 (720, 1100)	850 (700, 1100)	830 (680, 1000)	860 (710, 1100)	860 (640, 1100)
1979	1400	1400 (1100, 1800)	1100 (910, 1400)	1000 (820, 1300)	900 (740, 1100)	1000 (860, 1300)	1100 (680, 1700)
1984	2200	2200 (1800, 2900)	1300 (1100, 1700)	1100 (900, 1400)	900 (740, 1100)	1600 (1300, 2000)	1400 (730, 2900)
1989	3600	3600 (2900, 4600)	1600 (1300, 2100)	1200 (990, 1500)	900 (740, 1100)	1800 (1500, 2200)	1700 (760, 3800)
1994	5800	5800 (4700, 7300)	2000 (1600, 2500)	1300 (1100, 1700)	900 (740, 1100)	1700 (1400, 2200)	1600 (640, 4200)
1999	9300	9200 (7600, 12000)	2500 (2000, 3100)	1500 (1200, 1900)	900 (740, 1100)	1600 (1300, 2000)	1500 (510, 4200)
2004	15000	15000 (12000, 19000)	3000 (2400, 3800)	1600 (1300, 2000)	900 (740, 1100)	1400 (1100, 1700)	1500 (460, 4900)
2009	24000	24000 (19000, 30000)	3600 (3000, 4600)	1800 (1500, 2200)	900 (740, 1100)	1200 (960, 1500)	1200 (350, 4300)
2014	38000	38000 (31000, 48000)	4500 (3600, 5600)	2000 (1600, 2500)	900 (740, 1100)	1500 (1200, 1900)	1500 (390, 6000)
2019	61000	61000 (50000, 78000)	5400 (4400, 6900)	2200 (1800, 2700)	900 (740, 1100)	2100 (1800, 2700)	2300 (470, 11000)
2024	98000	98000 (80000, 120000)	6600 (5400, 8400)	2400 (1900, 3000)	900 (740, 1100)	3200 (2600, 4100)	3300 (550, 18000)

5.5.1 Average annual additional cases from ending the BCG schools scheme at various dates

I found that updating parameter values, and incorporating uncertainty, did not alter the average annual primary additional notifications from stopping the BCG schools scheme when the annual decrease was assumed to be 9%. However, when these changes were combined with the updated transmission model we found that the impact of ending BCG vaccination was greater than previously reported with an increase in the number of estimated cases due to onwards transmission (Figure 5.3; Table 5.4). These estimates were uncertain with 94 (2.5, 97.5% Q: 72, 119) additional annual notifications if vaccination was stopped in 2001. As the assumed annual decrease in incidence rates was reduced the annual number of additional notifications increased with 6099 (2.5, 97.5% Q: 4691, 7719) notifications when the annual decrease was assumed to be 1.9% and vaccination stopped in 2001. The number of annual notifications reduced with time: 5314 (2.5, 97.5% Q: 4082, 6725) from ending vaccination in 2006; 4327 (2.5, 97.5% Q: 3315, 5496) from ending vaccination in 2011, and 2852 (2.5, 97.5% Q: 2074, 3741) from ending vaccination in 2016 (Figure 5.3; Table 5.4).

Figure 5.3: Annual additional (stratified into primary and secondary) notifications in 15-29 year olds from stopping the BCG schools scheme in 2006, and 2011 until 2028. The years presented were dictated by the 5-year timestep of the model. The percentage annual decrease scenarios considered were based on those considered by the JCVI BCG subgroup. Data based scenarios and the JCVI 0% decrease scenario were not presented here as the updated transmission model could not support these scenarios. Each boxplot summarises the output of 10,000 model simulations for each scenario. Secondary notifications are reported assuming they occurred in the same year as the primary notifications that caused them.

Table 5.4: The median number (with the 2.5% and 97.5% quantiles) of additional annual notifications due to ending the BCG schools scheme in selected years. The percentage annual decrease scenarios considered were based on those considered by the JCVI BCG subgroup. Data based scenarios and the JCVI 0% decrease scenario were not presented here as the updated transmission model could not support these scenarios.

Year Ending Scheme	9% decrease (original parameters)	9% decrease	3.9% decrease	1.9% decrease
1971	111	111 (88, 135)	204 (161, 249)	286 (226, 350)
1976	90	90 (71, 110)	187 (148, 229)	276 (219, 337)
1981	55	55 (44, 68)	152 (120, 185)	244 (193, 298)
1986	35	35 (28, 43)	125 (99, 152)	218 (172, 266)
1991	25	25 (19, 30)	109 (86, 133)	203 (161, 247)
1996	17	17 (13, 21)	93 (74, 114)	185 (147, 225)
2001	12	12 (9, 15)	79 (62, 96)	166 (132, 203)
2006	8	8 (6, 10)	65 (51, 79)	145 (115, 177)
2011	6	5 (4, 7)	50 (39, 62)	118 (92, 145)
2016	3	3 (2, 4)	32 (23, 40)	78 (58, 99)
2021	1	1 (1, 2)	15 (9, 21)	39 (24, 54)

5.6 Discussion

The existing method for estimating the impact of the BCG schools scheme produced uncertain estimates of the impact of ending the scheme in all years evaluated when parameter uncertainty was included. The approach used to estimate additional notifications due to transmission was found to be incorrect. Once corrected, the transmission model produced much higher estimates of additional notifications caused by ending BCG vaccination than previously reported. Further updating the model with the annual decrease in TB notifications based on both notifications and using age-specific incidence rates resulted in a decrease in the number of vaccines needed to prevent a single case in all years considered. A scenario with a 1.9% annual decrease in incidence rates was most comparable to the results based on notifications. Using this scenario, I found that the number of TB notifications arising from ending school age BCG vaccination was much greater than originally estimated using the scenario considered in Sutherland et al.

In addition to identifying that the public health impact of the change in vaccine policy was likely much larger than originally estimated, my results also provide new insights into the uncertainty of the previously published model predictions by including parameter uncertainty and measurement error and updates these predictions using newly available data. As historical data on incidence rates in the ethnic White UK born in England and Wales were not available, I considered two approaches to proxy them and investigated multiple scenarios based on those explored by the JCVI BCG subgroup. The simulation approach used here, although updated where possible, is not the most accurate method for estimating the impact of ending the BCG schools scheme as it relies on numerous assumptions based on the available knowledge in 1987 and does not account for the role of non-White and non-UK born cases. However, the strength of this work is that the estimates are based on the framework used to inform policy making. This allowed the strength of the model used in the decision-making process to be assessed once parameter uncertainty had been incorporated and for flaws in the model to be identified. This would not have been possible if the impact had been assessed using only the observed data or with an alternative model. It also allowed estimates based on updated data to be compared to historic estimates within the same framework. This would also not have been possible if a different framework had been used. As mentioned, a weakness of the model used in this study is that it did not include the whole population or age groups outside those directly affected by vaccination. Heterogeneous mixing between these groups is also likely to be an important consideration. The exclusion of these factors means that my results are likely to underestimate the impact of ending the BCG schools scheme. A final limitation is that this study only considers the impact of ending the BCG schools scheme and not the impact of the introduction of the targeted neonatal vaccination program. This should be considered when evaluating the change in policy as a whole.

Little work has been done to assess the impact of the 2005 change in BCG vaccination policy or to assess the quantitative evidence used in decision making. However, multiple studies have evaluated the cost effectiveness of various BCG programs and the impact of switching between them. A cluster-randomised trial in Brazil found that BCG vaccination of those at school-age was cheaper than treatment and would prevent one TB case per 381 vaccinations even with a vaccine effectiveness of only 34% (8-53%).[68] This is substantially fewer than my estimate of 2000 (2.5, 97.5% Q: 1600, 2500) (the most comparable year from my results). However, the same trial found that for regions close to the equator BCG effectiveness was low in school-age children but unchanged in neonates,[69] highlighting the importance of considering the BCG vaccines reduced effectiveness near the equator when determining vaccination policy.[70] There is also some research which supports universal re-vaccination of those at school-age, in countries with high incidence and universal vaccination of neonates, as it may be cost effective when BCG effectiveness is moderate to high.[69,71] There is some evidence that targeted vaccination of high risk neonates maybe more cost effective than universal vaccination of neonates.[72,73] However, a study in Sweden found that incidence rates in Swedish-born children increased slightly after universal vaccination of neonates was discontinued in favour of targeted vaccination.[74] In France, which switched from universal vaccination of neonates to targeted vaccination in 2007, it has also been shown that targeted vaccination reduced coverage in those most at risk.[75] Targeted vaccination may not be more cost effective than universal vaccination when possible reductions in transmission are considered. This chapter indicated that a substantial number of cases due to transmission may be preventable if universal school-age BCG vaccination was still in place. This result is dependent on the effectiveness of BCG vaccination when given later in life, for which there is good evidence in the ethnic White UK born.[26] I did not consider neonatal vaccination which would be less impacted by BCG’s effectiveness reducing when given later in life, but may also be less likely to result in the same reductions in ongoing transmission.

This chapter indicates that some of the evidence used to justify the 2005 change in BCG vaccination policy may have depended on a methodologically flawed model, resulting in the impact of ending BCG vaccination being underestimated. Modelling evidence can often be complex and difficult to reproduce, it is important that policy makers, or those who work with them, have the skills to assess its quality. This study also highlights the importance of including both parameter and measurement error, as excluding these sources of variation may lead to spuriously precise results. Spurious precision is problematic for policy makers as the worst-case scenario often needs to be considered when making policy decisions. In addition, my exploration of the assumptions used to estimate the annual change in TB incidence rates in the ethnic White UK born illustrates the structural impact of assuming an annual decrease in TB incidence rates. More realistic estimates of the annual decrease in incidence rates resulted in a greatly increased impact of ending the BCG schools scheme. Policy makers should consider these updated estimates when assessing the role of BCG vaccination in those at school-age. However, decisions regarding vaccine policy in the UK require economic evaluation, which discounts costs and benefits in the future; discounting has not been applied in this study which estimates the epidemiological impact of vaccination only.

This chapter has reassessed some of the evidence previously used in decision making, correcting the transmission model used, and updating the approach with new data. However, as 15 years of detailed surveillance data have been collected since the ending of the BCG schools scheme it is now possible to use regression-based approaches to estimate the direct impact on incidence rates of ending the BCG schools scheme (see Chapter 7).[76] These approaches could also be used to estimate the impact of vaccinating high-risk neonates, which may outweigh any negative impacts of ending the BCG schools scheme. In addition, the development (see Chapter 8), and use, of a transmission dynamic model would allow the more accurate estimation of indirect effects and the forecasting of long-term impacts (see Chapter 10).

5.7 Summary

• This chapter reevaluates a key piece of the quantitative evidence used to motivate the change in BCG vaccination policy in 2005, correcting a methodological flaw in the original model.

• The inclusion of parameter uncertainty, and measurement error, allowed the uncertainty in the final estimates to be presented. Previously published estimates may have been spuriously precise.

• As this study used a historical approach the model used is not the most accurate method for assessing the impact of ending the BCG schools scheme. However, it provides an estimate that is based on the available data and on the framework used to inform policy making. This allowed the strength of some the quantitative evidence used in the decision-making process to be assessed.

• The impact of ending the BCG schools scheme was found to be greater than previously thought when notification data were used. These results highlight the importance of independent evaluations of modelling evidence, including uncertainty, and evaluating multiple scenarios when forecasting the impact of changes in vaccination policy.

• The code for the analysis contained in this chapter can be found at: doi.org/10.5281/zenodo.2583056¹⁹

References

2 Public Health England. Tuberculosis in England 2017 report ( presenting data to end of 2016 ) About Public Health England. 2017.

5 Zwerling A, Behr MA, Verma A et al. The BCG world atlas: A database of global BCG vaccination policies and practices. PLoS medicine 2011;8:e1001012.

23 Rodrigues LC, Diwan VK, Wheeler JG. Protective effect of BCG against tuberculous meningitis and miliary tuberculosis: a meta-analysis. International journal of epidemiology 1993;22:1154–8.

24 Colditz GA, Brewer TF, Berkey CS et al. Efficacy of BCG Vaccine in the Prevention of Tuberculosis. JAMA 1994;271:698.

25 Mangtani P, Abubakar I, Ariti C et al. Protection by BCG Vaccine Against Tuberculosis: A Systematic Review of Randomized Controlled Trials. Clinical infectious diseases : an official publication of the Infectious Diseases Society of America 2014;58:470–80.

26 Hart PDA, Sutherland IAN. BCG and vole bacillus vaccines in the prevention of tuberculosis in adolescence and early adult life. The American Statistician 1972;46:371–85.

28 Abubakar I, Pimpin L, Ariti C et al. Systematic review and meta-analysis of the current evidence on the duration of protection by bacillus Calmette-Guérin vaccination against tuberculosis. Health technology assessment 2013;17:1–372, v–vi.

39 World Health Organization. Global Tuberculosis Report. 2017.

40 Mangtani P, Abubakar I, Ariti C et al. Protection by BCG vaccine against tuberculosis: A systematic review of randomized controlled trials. Clinical Infectious Diseases 2014;58:470–80.

41 Sutherland I, Springett VH. The effects of the scheme for BCG vaccination of schoolchildren in England and Wales and the consequences of discontinuing the scheme at various dates. Journal of epidemiology and community health 1989;43:15–24.

56 R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: 2016.

60 Stevenson M, Nunes T, Heuer C et al. epiR: Tools for the Analysis of Epidemiological Data. 2017.

61 Pilger D, Nguipdop-Djomo P, Abubakar I et al. BCG vaccination in England since 2005: A survey of policy and practice. BMJ Open 2012;2:e001303.

62 Joint Committee on Vaccination, panel IB. Minutes. 2002.

63 Joint Committee on Vaccination, panel IB. Minutes. 2003.

64 Sutherland I. Effectiveness of BCG vaccination in England and Wales in 1983. Tubercle 1987;68:81–92.

65 Springett VH, Darbyshire JH, Nunn AJ et al. Changes in tuberculosis notification rates in the white ethnic group in England and Wales between 1953 and 1983. Journal of epidemiology and community health 1988;42:370–6.

66 Borgdorff MW, Hof S van den, Kalisvaart N et al. Influence of sampling on clustering and associations with risk factors in the molecular epidemiology of tuberculosis. Am J Epidemiol 2011;174:243–51.

67 Fox J, Weisberg S. An R companion to applied regression. Third. Thousand Oaks CA:: Sage 2019. https://socialsciences.mcmaster.ca/jfox/Books/Companion/

68 Pereira SM, Barreto ML, Pilger D et al. Effectiveness and cost-effectiveness of first BCG vaccination against tuberculosis in school-age children without previous tuberculin test (BCG-REVAC trial): A cluster-randomised trial. Lancet Infectious Diseases 2012;12:300–6.

69 Barreto ML, Pilger D, Pereira SM et al. Causes of variation in BCG vaccine efficacy: Examining evidence from the BCG REVAC cluster randomized trial to explore the masking and the blocking hypotheses. Vaccine 2014;32:3759–64.

70 Fine PEM. Variation in protection by BCG: implications of and for heterologous immunity. Lancet 1995;346:1339–45.

71 Dye C. Making wider use of the world’s most widely used vaccine: Bacille Calmette-Guerin revaccination reconsidered. Journal of The Royal Society Interface 2013;10:20130365–5.

72 Usher C, Adams R, Schmitz S et al. Evaluating the neonatal BCG vaccination programme in Ireland. Archives of Public Health 2016;74:1–12.

73 Hersh AL, Tala-Heikkila M, Tala E et al. A cost-effectiveness analysis of universal versus selective immunization with Mycobacterium bovis bacille Calmette-Guérin in Finland. Int J Tuberc Lung Dis 2003;7:22–9.

74 Romanus V, Svensson Å, Hallander HO. The impact of changing BCG coverage on tuberculosis incidence in Swedish-born children between 1969 and 1989. Tubercle and Lung Disease 1992;73:150–61.

75 Guthmann JP, Antoine D, Fonteneau L et al. Assessing BCG vaccination coverage and incidence of paediatric tuberculosis following two major changes in BCG vaccination policy in France. 2011;1–6.

76 Abbott S, Christensen H, Welton NJ et al. Estimating the effect of the 2005 change in bcg policy in england: A retrospective cohort study, 2000 to 2015. Eurosurveillance 2019;24:1900220. doi:10.2807/1560-7917.ES.2019.24.49.1900220

---

17. Preprint: https://doi.org/10.1101/624916↩

18. Code: https://github.com/seabbs/AssessBCGPolicyChange↩

19. Alternatively available from: https://github.com/seabbs/AssessBCGPolicyChange↩