Ch 2. The Greatest Fraud Fluoridation: Errors and Omissions in Experimental Trials - 1b THE EVANSTON STUDY

 

Ch 2. The Greatest Fraud Fluoridation: Errors and Omissions in Experimental Trials - 1b THE EVANSTON STUDY

Errors and Omissions in Experimental Trials - 1b

THE EVANSTON STUDY

The United Kingdom Mission (1953), after having observed the Evanston study, described it as "one of the most elaborate investigations." Hill et al. (1950) considered that they had planned the study so "as to measure every variable that might exert an influence and obscure the findings." It is the only trial in which bite-wing examinations were made for all subjects examined.

The importance of X-ray examinations. Blayney and Greco (1952) reported that in this trial "the X-ray disclosed 53.84 per cent of the total number of carious lesions observed by both clinical and X-ray methods". That said: "We believe it extremely important to employ both clinical and X-ray techniques in any study program which is directed toward the determination of the prevalence or the control and reduction in the rate of caries attack." This result must throw considerable doubt on the accuracy of the caries attack rates which were reported from the test

and control areas in the other studies considered; for in these, X-ray examinations were incomplete or absent.

The ideal control community. The authors of the study stated that "It seemed logical to think of Oak Park, Illinois, as the ideal control community because of its close similarity to the study area" (Blayney and Tucker, 1948). The manner in which that city resembled Evanston was not stated. The United Kingdom Mission (1953) made the important observation that in Evanston the economic level was high, and "dental care was outstandingly good."

Lower caries rates in control community. It soon became apparent that Oak Park could not be called "the ideal control community", for Hill et al. (1951) stated that "Comparison of the caries rates of all children in the study area (Evanston, Ill.) and the control area (Oak Park, Ill.) prior to the addition of sodium fluoride to the communal water supply of the study area indicated a lower caries rate for school children of the control area."

Different rates in student groups. The authors continued:

 

In an effort to find the source of these differences in caries prevalence, it was found to be due largely to differences in the make-up of the student groups examined in the two areas. While in the study area 22.2 per cent of the children examined were attending parochial schools, no such children were included in the control area: and while 5.6 per cent of the children in the study area were Negro children, only 0.1 per cent of the children in the control area were Negro. Statistically significant differences were found to exist between the caries rates of Negro and parochial school children on one hand and public white school children on the other hand. Generally the caries rates of parochial school children were found to be higher and those of Negro children lower than those of white children in public schools.

Exclusion of data. Hill et al. (1951) continued:

Therefore, comparisons of caries rates for the study group and the control group are based on the caries experience of public white school children only, while such comparisons involving children in only the study area are based on the caries experience of all children in total. The caries rates for the Evanston white school children in the 1946 survey and the Oak Park white school children in the 1947 survey were very similar.

Six lines later, it was stated: "In further comparing the rates for Oak Park (control) and Evanston (study area) it is apparent that the baseline figures are very similar."

The only comparisons that can be made from the paper which has just been mentioned are the figures for the children aged twelve, thirteen and fourteen years. Negro and parochial school children constituted 27.8 per cent of the Evanston children. By excluding this part of the data the rates in that city were then considerably lower than those in the control city, the rates (Table IV) being 707.51, 946.17 and 1133.33 in every 100 of the Evanston public school white children for the ages twelve, thirteen and fourteen years; those in Oak Park being 774.29, 970.00 and 1194.64 for the same three ages.

An altered explanation. A different, but, at first sight, a reasonable explanation for the exclusion of the data of Negro and parochial school children, when making comparisons with data from Oak Park, was given in the XV Report (Hill et al., 1957a): "As the control area (Oak Park) examinations included only public school white children it was necessary to evaluate the Evanston data on the basis of school groups, public white, parochial, and Foster (Negro) to make comparisons of like groups." It can be seen that in that paper the exclusion of data was attributed, not to the fact that this process was undertaken because there was "a lower caries rate for school children of the control area" (Hill et al., 1951), but to the different racial composition of, and type of school attended by the children in the two cities. Hill et al. (1950) mentioned that one of their seven "other objectives" was "to compare the dental caries experience of white with that of Negro school children." No reference was made to the possibility of a difference being found between the rates of white public and parochial school children. However, the original statement (Hill et al., 1951) makes it clear that the different school groups were taken into account only after the unsatisfactory results of the first Oak Park examination became apparent.: "In an effort to find the source of these differences in caries prevalence." In assessing the accuracy of the second (1957a) explanation, it should be realized that in the younger age group "comparisons of like groups", or even the dissection of the data into the three school groups, were not published in the reports dealing with that age group, namely the 1950, 1952, 1954, 1956 and 1957b papers, or even in the XV Report (Hill et al., 1957a) which dealt with both age ranges, but showed this dissection for the children of the older age group only. Furthermore, when, after a delay of more than ten years, the 1947 Oak Park rates for the younger children were published for the first time by Hill et al. In 1958, no "comparisons of like groups" were made by them. The reader is prevented from making this comparison by the fact that, even now, the dissection of this age range into the three school groups has not been published, despite the statement by Hill et al in 1951 that the rates for "school children" were significantly different in each type of school.

"Correction" of data. When making comparisons with the control city, the authors excluded from the three groups of data obtained in the test city the two which diverged most from the rates of the children in the control city (Hill et al., 1951). This process should be considered in connection with the following statement (Hill et al., 1950):

In order to be able to generalize from our findings, we must be certain that any such variables as effect caries experience are represented in our study to the same extent as in the population. Before drawing any ultimate conclusions, we will, therefore, correct our data in such a manner as to include only those groups of children which are representative of the population, with respect to dental caries experience. We feel that this precaution is necessary to allow the ultimate findings to be considered valid and reliable.

However, the process which they described - the arbitrary selection of a section of the data, which is then termed "representative" - instead of making "the ultimate findings to be considered valid and reliable", would render a report based on this selected data unfit for serious consideration.

"Population" sampled. It is not clear what the authors meant by the term "the population." If the population referred to was that of Evanston, the sample of children examined in this study - if properly drawn - provided an unbiased estimate of the dental condition of the population of that city; if only some of the data are included, the results will be biased. If this term "population" was intended to refer to the general population of the U.S.A., it should be realized that the results from Evanston can represent only a stratum of the country as a whole, varying as to climate and racial composition, to mention only two variables.

It will be recalled that the caries rates were said to be significantly different, even between children attending the different types of school in Evanston; and also that the rates in that city were considerably different from those in Oak Park, which was at first stated to be "the ideal control community" for Evanston (Blayney and Tucker, 1948). These differences emphasize the fact that caution should be exercised when applying results obtained in a test city to a wider population, of which the test city may not be representative.

Altered methods in latest report. In the latest report (Hill et al., 1958) which shows the findings for the permanent teeth of children in the control city of Oak Park, the authors have published in the same tables as the results of the control groups, the DMF rates, not of the public school white children, but of the total sample of Evanston children. This is strange in view of their statement that "comparisons of caries rates for the study group and the control group are based on the caries experience of public white school children only" (Hill et al., 1951). It would appear that they no longer held the opinion which they stated the previous year (Hill et al., 1957a) that it is necessary "to make comparisons of like groups."

As a result of this change in procedure the differences between initial caries rates in Evanston and Oak Park are diminished. In children aged twelve to fourteen years, the pre-fluoridation rates reported for the 1,226 public school white children in Evanston were far closer to the values found in the Oak Park children than were either the rates of the 96 Negro, or of the 379 parochial school children (Hill et al., 19571, 1958). However, the rates of the Negro children were lower, and the rates of the parochial school students were considerably higher than those of the public school white children. By adopting the authors' latest (1958) method, which is to add the results of the three groups, it is found that the pre-fluoridation rates of the twelve and fourteen-year-old children are considerably less divergent from those of the initial examinations in Oak Park - and those of the thirteen white children of those ages. Whether this situation arises with regard to the six-, seven- and eight-year old children cannot be determined, for no dissection into the rates prevalent in the three school groups has been published.

Late examination in the control city. The United Kingdom Mission (1953) stated: "Before fluoridation started a dental survey was made of 4,375 children in the selected groups in Evanston and of 2,493 children in Oak Park. Further examinations have been carried out each year since 1947 and will continue until 1962." However, the examinations in Oak Park were not commenced until after the fluoridation of the Evanston water supply on 11 February 1947, for Blayney and Tucker (1948) stated: "The study in Oak Park was instituted on Feb. 26, 1947". Also, at the time of the United Kingdom Mission Report (1953), no further examinations had been conducted in Oak Park; even in Evanston only one age group was examined during each year, as can be seen by inspecting the "schema for study" published by Blayney and Tucker in 1948, and reproduced in several subsequent reports.

Only two examinations in the control city. This "schema" indicates that the design of the trial provided for only two examinations - eleven years apart - to be made in the control city. It would appear that the authors did not anticipate changes in the caries rates of the control, such as were reported in Muskegon (Arnold et al., 1953), and, as will be seen later, in Sarnia (Brown et al., 1954b), and in Kingston (Ast, Finn and Chase, 1951). The first examination was made in 1947, and the second, although not scheduled until 1958, was commmenced in 1956 when it became apparent that the water supply of Oak Park would be fluoridated (Hill et al., 1956). This examination was completed on 14 November 1956, soon after the fluoridation of the Oak Park water on 1 August (Hill et al., 1958).

A ten-year delay in the publication of data. Caries attack rates for the six-, seven- and eight-year-old children which were obtained in Oak Park in 1947 (Blayney and Tucker, 1948) have only recently been published by Hill et al. (1958). This great delay is inexplicable and is particularly unfortunate, because it is in regard to these younger children that the major claims are made for reduction of dental caries as a result of fluoridation. No explanation was offered for this delay, and the members of the United Kingdom Mission (1953) did not comment on this strange omission, merely saying that "The incidence of caries among the children aged 6-8 years is compared with the baseline data of Evanston itself while caries experience of children aged 12-14 years is compared with that of Oak Park."

Gross differences in initial caries rates. The latest report (Hill et al., 1958) reveals that in the younger children there were gross differences between the initial caries attack rates in Evanston and Oak Park. The rates were: 46.85, 26.89 for age six years; 153.49, 102.63 for age seven years; and 249.93, 222.44 for age eight years in Evanston and Oak Park respectively.

In regard to the great difference between the pre-fluoridation rate for the six-year-old children in Evanston and the initial one for children of that age in Oak Park, 46.85 and 26.89 respectively, a footnote to Table I (Hill et al., 1958), referring to the former rate, stated: "This figure results from the very high DMF rate of 87.91 found in one school in 1946." However, as the children were drawn "from 24 schools in the study area" (Blayney and Greco, 1952), it is probable that the rates for six-year-old children in most schools approached the figure of 46.85, unless the school with the high DMF rate also happened to provide a disproportionately large number of six-year-old children.

It should be noted that no comment on the magnitude of this rate of 46.85 was made in any of the four reports in which it had been shown previously (Hill et al., 1950, 1952, 1956, 1957a); all of which were published before the rate of 26.89 for Oak Park was released, and therefore before a comparison with it could be made. The rate of 46.85 was used in all those papers - and even in their latest report (1958) - in calculating the "% reduction", and in computing the "Probability of difference due to chance."

Much unpublished data. The members of the Evanston Dental Caries Study devoted most of the years 1947 and 1956 to the collection of data from children in Oak Park (Blayney and Tucker, 1948; Hill et al., 1958). Despite this fact, the major part of each of the two tables shown in the XVIII Report (Hill et al., 1958) was devoted to a re-presentation of data obtained in Evanston, although this report was said to have as its purpose the comparison of the permanent teeth dental caries experience rates in children examined in Oak Park in 1947 and 1956. The Oak Park data were restricted to four lines of figures showing the DMF rates in permanent teeth. No report was made of other findings such as those which had been shown in reports on Evanston children. For instance, in the XV Report (Hill et al., 1957a), no fewer than eight tables relating to the twelve-, thirteen and fourteen year-old children only were devoted to these other findings. This very incomplete presentation of the data obtained in Oak Park is unaccountable.

Figure 3. Gross differences in initial caries rates in Evanston and its control city of Oak Park. The Oak Park rates remained unpublished for over ten years.

Disagreements between results. In their XVIII Report, Hill et al. (1958) stated: "The DMF rates and percentage reduction from year to year for the Evanston children of all age groups shown in Tables I and 2 have been published in previous reports. However, four of the figures for the year 1955, shown in Table I of the 1958 Report, are different from "the rates and percentage reduction" given, for the same year, in Table I and the text of the XVI Report (Hill et al., 1956). The DMF rates at age seven years were only slightly different (40.95 and 40.92, in the XVI and the XVIII Reports respectively), but at age eight years the two rates were 114.04 and 120.32. It is very improbable that these different rates are due to typographical errors, for they were confirmed by the "per cent reduction from 1946", which was given in the summary and in Table I of the respective reports as 73.32 and 73.34 for children aged seven years, and as 54.37 and 51.85 for those that were eight years of age. This "reduction" was shown in the XVIII Report as 85.96 for the six-year-old children, but in the XVI Report it was given as "80 per cent" in the findings and as "85.96 per cent" in the summary.

Disagreement between tables. The DMF rate in terms of tooth surfaces was given only twice in this study (Hill et al., 1955, Table X and 1957a, Table XII). In both papers the "DMF rate per 100 surfaces" for children aged fourteen years was 14.82 in 1949 and 13.94 in 1952. However, in the former report this rate was given as 15.09 in 1946, but in the latter one, for children of the same age in the same year, the figure shown was 15.92. As a result of this change, the "% differences from 1946" were altered from 1.78 to 6.85 (1949) and from 7.62 to 12.44 (1952). By using these new rates it can be said that "all 3 methods, namely; per hundred children, per hundred teeth, and per hundred surfaces all express approximately the same proportion of percentage differences in rates" (Hill et al., 1957a). This result is a good illustration of the comment made on the method most commonly used in these studies to express changes in caries experience, that "relatively small variations in the baseline values will produce substantial alterations in the percentage reduction obtained" (Part One, p. 137).

It may be mentioned that the "total tooth surfaces considered" for thirteen-year-old children in 1954 (Table X11, Hill et al., 1957a) should be 58,325 not 58,352; and that for fourteen-year-old children in 1949, in the column of that table giving the "% differences from 1946", the figures shown should be 6.91 not 6.85. In their XI Report (Table IX) and their XV Report (Table XI), Hill et al. (1955, 1957a) showed different figures for children aged twelve years examined in 1952. Although both tables show the same total number of teeth considered, in the former table children were shown as examined, with a "DMF rate per 100 teeth" of 25.76, and a difference from 1946 of 19.50 per cent. In the latter table, the figures were 516, 25.60 and 20.00 per cent respectively. In 1953 Hill et al. published the figure of 19.50 per cent.

No data for deciduous teeth. The authors have not published any data regarding the deciduous teeth of children in the control city, either for the first (1947) or the second (1956) examination. The most important omission, the def rates, could have been shown by adding only two lines to Table I in Hill et al. (1958). This omission is particularly unfortunate in view of the fact that in the deciduous teeth in Evanston during the first four years of fluoridation the def rate of the six to eight years group was considerably higher than the initial one (Hill et al., 1952). It was not until nine years after the commencement of the study that a significant reduction in this rate was reported.

In 1950, Hill et al. stated that the caries rate for deciduous teeth in these children "does not indicate any trend", despite the fact that in Table I of that report the initial rise in this rate during the first two years of fluoridation was shown by them to be statistically significant (P = 0.005). Two years later these authors altered their opinion of the significance of this rise. In 1952 they re-published the same data for children aged six, seven and eight years in 1946 and 1948, but computed different rates for the combined age group six to eight years. The rise in the def rate was then said to be not statistically significant.

Variations in caries rates in control. The meagre data regarding caries attack rates in Oak Park which have been published are included in Tables I and 2 of Hill et al. (1958). Of the six age groups shown, between the years 1947 and 1956 the authors reported a significant increase in the DMF rate of children aged seven years, and non-significant upward trends in the rates of those aged eight and thirteen years, and downward ones in the caries attack rates in children aged six, twelve and fourteen years. (The question of "significant" changes in the rates in control cities will be considered later.) The authors said: "The children 12, 13 and 14 years of age, Table 2, have only minute differences between the 1947 and 1956 rates. These are not considered to be significant." The footnote to that table is more definite, in each comparison stating: "Difference is not statistically significant." Although these differences of 61.20, 34.96 and 58.87 DMF teeth, for children aged twelve, thirteen and fourteen years respectively, were termed "minute differences", those seen in the rates of the twelve and fourteen-year-old children are approximately a third the size of the absolute drop in the rates recorded for the same age groups in Evanston since the inception of fluoridation. It cannot be assumed that the fluctuations in the rates during the intervening period of nine years, when no examinations were made, did not exceed the differences between the initial and final rates. It will be recalled that considerable variations occurred in Muskegon (see Figs 1 and 2).

Inadequacy of the control. Blayney and Tucker (1948) realized that "A study of this nature must have an adequate control." Therefore, it is strange that in the "schema" which they published there was provision for only two examinations, eleven years apart, to be made in the control area. It should have been obvious that the usefulness of data gathered in such a manner would be, at most, very limited. The explanation given by the authors for their failure to examine the children in the control city "every year" (instead of only twice) was the strange one that "It was not necessary to do so in as much as Evanston and Oak Park are subjected to the same advertising campaigns, have a similar economic level, participate in comparable educational programmes, and so forth" (Hill et al., 1958). It is extraordinary that the authors advanced this explanation and that they adhered to such a plan, despite the marked dispanity in canes rates disclosed in the first examinations in Evanston and Oak Park (Hill et al., 1958), which makes it obvious that the latter city was a poor choice in seeking an "adequate control" for the former one.

Differences between school groups. Hill et al. (195 1) stated that "statistically significant differences were found to exist [in 1946] between the caries rates of Negro and parochial school children on one hand, and public white school children on the other hand." However, they made a further statement that "the caries rates of parochial school children were found to be higher and those of Negro children lower than those of white children in public schools" (Hill et al., 195 1). These two statements are inconsistent. The first appears to mean that the comparisons between Negro children and white children in public schools, and that the comparison between white children attending parochial schools and those attending public schools, were both statistically significant in 1946.

"Nearly comparable" or significantly different? The XV Report Hill et al., 1957a) stated that "In 1946 and 1954 the public school white children and the Foster School (Negro) children maintained nearly comparable DMF rates". The actual rates" (per 100 children) in 1946 for twelve, thirteen and fourteen-year-old white children attending public schools were 707.51, 946.17, and 1133.33; for the Negro children of the same ages they were 658.82, 861.76 and 1035.71. (The rates of each school group of younger children were not published.)

It is not understood how the same authors could on one occasion (Hill et al., 195 1) state that there were "statistically significant differences" between the two series of rates, and later (Hill et al., 1957a) describe them as "nearly comparable DMF rates" It may be thought that the word "maintained" referred to a comparison between the DMF rates of the white children in public schools, and of the children in the Negro school, between 1946 and 1954. However, this cannot be the case, for the authors claimed for these twelve, thirteen and fourteen-year-old children "a reduction of approximately 21.96 per cent in dental caries-experience rates of the permanent teeth" (Hill et al., 1957a). (In this study, percentages were frequently shown "approximately" to two decimal places.) Table IV of that paper shows that both the Negro and the public school children participated in the reductions reported.

Decline in eruption rate. An observation of considerable interest is obtainable from Tables V and VI of the X Report (Hill, et al., 1952). The former table shows the rates per 100 six, seven and eight-year-old children that had occlusal surface pit and fissure caries or fillings in their first permanent molars; the latter one, the number of these teeth which were free from those defects. The mean number of erupted first permanent molars per 100 children may be obtained, in each age group, by adding these two rates to that showing the extracted and congenitally missing permanent molars. It is probable that the number of congenitally missing teeth was negligible and that the number of permanent molars which had been extracted in these young children was small, particularly in the six years age group (five and a half to six and a half years). Therefore, it would be expected that, in each age group, the mean number of erupted molars per 100 children would be similar at the time of each examination. This was the case in children aged eight years; the figures for the examinations made in 1946 (pre-fluolidation), 1948, 1950 and 1951 being (to the nearest whole number) 387, 387, 384 and 386 respectively. At age seven years the numbers erupted were 330, 336, 320 and 315; but in the six-year-old children, the number of erupted molars showed a marked and progressive decline 189, 156, 140 and 132 during the period covered by those four examinations.

The question naturally arises whether the eruption rate of these teeth had decreased; a possibility of extreme importance in interpreting the results of a fluoridation trial. However, further consideration of this matter is prevented by the authors' failure to publish this type of data when they reported the results of the two later examinations (conducted in 1953 and 1955) which were made of children of these ages; and the "schema for study" indicates that children aged six to eight years will not be examined again until 1960.

This failure to publish this type of data for the 1953 and 1955 examinations is extraordinary, especially in view of the fact that the authors continued to show similar data for the permanent molars of the older age group (Hill et al., 1955, 1957a); the latter report, the only one showing results for both age groups, gave the prevalence of occlusal pit and fissure caries and fillings in the molars of the older, but not of the younger age group.

In considering the eruption of teeth, the odd method of assessment used in this study must be taken into account. Hill et al. (1955) said: "Only teeth which were 50 per cent or more erupted were considered. A carious or filled tooth was, of course, considered regardless of its stage of eruption."

Figure 4. Suggestion of a progressive decline in the number of erupted first permanent molar teeth in six-year-old children in Evanston. The results obtained in the examinations conducted in 1953 and 1955 were omitted from the published reports.

Strange superiority of artificial fluoridation. The authors of this study compared the Evanston DMF rates per child with those of children in Aurora, Illinois (Dean et al., 1950) in the expectation that after sufficient time had elapsed for all the erupted teeth to have been formed since fluoridation commenced "the Evanston rate will closely approach the Aurora rate" (Hill et al., 1957a). It is surprising that this parity between the rates of Aurora and Evanston was expected, because in the Aurora survey only clinical methods of examination were used, but in the Evanston examinations X-ray surveys were used routinely. Hill et al. (1951) stated: "We find our baseline figures for caries experience in Evanston and Oak Park approximately 32 per cent higher than those of Dean and his co-workers for Evanston and Oak Park in 1941. We assume this may be explained partially by differences in the techniques of examination, particularly in the use of X-ray in the current investigation." The United Kingdom Mission (1953) stated that in this study "the minutest radiolucency was taken as indicating caries."

In view of these findings, it is even more strange that Hill et al. (1957a) were able to report: "The Evanston 6 and 7-year-olds of 1953 have a lower dental caries experience rate after 71 to 82 months of fluoridation than the Aurora 6 and 7-year-olds of 1945-1946 with lifetime exposure to water naturally fluoridated to 1.2 ppm." That this difference was not only slightly below the 1945-1946 Aurora rate for children of the same age" (Hill et al., 1957a) can be seen by comparing the actual rates reported. In Evanston and Aurora respectively, the rates were 14.73, 28.0 at age six years and 53.35, 70.5 at age seven years (Hill et al., 1957a; Arnold et al., 1953). It should be noted that in Evanston two years previously (195 1), after a shorter period of fluoridation, the rate for the six-year-old children was even lower, 12.36 (Hill et al., 1952) and was less than half the Aurora rate; in 1955 (Hill et al., t956) it had become 6.58, less than a quarter of the Aurora rate. Blayney and Greco (1952) found that in children in the Evanston study, with regard to proximal caries "the 6-year-olds have the highest percentage (83.90) disclosed by X-ray findings only. In the 7-year-old group 79.04 per cent of proximal lesions were demonstrated by X-ray findings only". Therefore, if clinical methods of examination only had been used in Evanston, as was the case in Aurora, what may be thought to be a strange superiority of artificially over naturally fluoridated water as a means of reducing dental caries attack rates would have appeared to have been even more marked.

"Weighting" of results. The method of combining the results of the six, seven and eight-year-old children into one category introduces an important source of error when comparisons are made between the results obtained in the control city and in the test one, or between those found on different occasions in Evanston. Owing to the great differences in caries attack rates which are observed between children of these ages (the baseline DMF rates for these three ages in Evanston were 46.85, 153.49, and 249.93, according to Hill et al., 1950), the results may inadvertently be "weighted" by including a preponderance of young or of old children in the age group six to eight years. If this occurs, the average value will be lower or higher than it would have been if the three ages had been equally represented in the sample. In comparing the results of the control and the test cities, "weighting" of this nature could make it appear that large differences were present, when, in fact, they were either slight or absent, or the presence of actual differences could be hidden.

An example of "weighting". The results of the pre-fluoridation, and of the first post-fluoridation survey at Evanston (Hill et al., 1950), clearly demonstrate the process of "weighting" and show that its occurrence is not merely a theoretical possibility. On these two occasions, the number of children in each of the age groups six, seven and eight years that were examined in 1946 was 461, 759 and 771 respectively; the corresponding numbers seen in 1948 were 756, 838 and 440. On both occasions the results of the three ages were combined, and a caries rate was computed for the age range six to eight years.

Significant tests and "weighting ". Despite the rather obvious "weighting" in the examples which have just been cited, tests were applied to determine the significance of the difference between the caries attack rates found during the two examinations in the combined age range six to eight years. In regard to the permanent teeth, it was stated that "The probability of this difference being due to chance is 0,0000" (Hill et al., 1950). Curiously, in those teeth a decrease in the caries rate was reported, contrasting with the statement of a significant rise in the rate of the deciduous ones.

Random variation ignored. Hill et al. (1950) stated: "It is to be expected that the rate of caries in all teeth varies from year to year due to chance. A significant reduction of caries prevalence can therefore be assumed to exist only when the statistical analysis of the data provides almost absolute certainty that the observed differences are not due to chance." However, in a subsequent paper (Hill et al., 1956) these authors ignored the variations in the intervening years, even when these were as marked as those in Table 5 of that report, and stated: "Difference between 1946 and 1955 rates is statistically significant."

Original results altered. In the X Report (Hill et al., 1952), and in all the later ones, alterations were made to the rates shown for the years 1946 and 1948 in children of the combined age group six to eight years, which were published by Hill et al. in 1950 (Tables I to VI). The original rates were replaced by values which are the means of the mean rates for the children of each of the three ages six, seven and eight years (Hill et al., 1952, Tables 11 to IX).

System of computation changed. The change in the system of computation was explained by Hill et al. (1952) in these terms: "The group averages, shown in previous reports, represents weighted averages of the individual mean caries rates. Inasmuch as the composition of the groups of children with respect to the number of 6, 7 and 8-year-olds varies from year to year, it was felt that unweighted group averages form a more sound basis for comparison of group caries rates between years."

The new method of computation. In 1952 Hill et al. stated that "The new averages were obtained by taking a simple arithmetical mean of the individual caries rates of the 6, 7 and 8-year-old children." This description of the new method is apt to cause some confusion, for it is considered to describe accurately the old method. It was used by these authors in 1950, and then abandoned by them in favour of the new one. The results for 1950 and 1951 in Table IV of Hill et al. (1952), and those for 1953 in Table I of Hill et al. (I 957a), and for 1955 in Table I of Hill et al. (1956) make it clear that in this new method of calculation, the rate per 100 children aged six to eight for each examination was obtained by taking a simple arithmetical mean of the mean rate for each of the three ages six, seven and eight years.

Errors in amended rates. The amended rates published by the authors (Hill et al., 1952) for the age group six to eight years need further amendment, and the difference between them is even less than that stated. The mean of the three values shown for 1948 in their Table IV, 23.54, 103.58 and 194.09, is found to be 107.07, not 92.07 as stated; also, the mean of the three values for 1946 - 46.85, 153.49 and 249.93 is 150,09 not 149.76. These errors were repeated in the XV and the XVI Reports (Hill et al., 1957a, 1956).

The figure 149.76 was shown also in the XIV Report (Hill et al., 1954). In that report the rate for age six to eight years was said to be "65.82 in 1953." However, in Table I of the XVI Report (Hill et al., 1956) the rate for 1953 for age six to eight years was given as 63.52. The latter figure is the mean of the three mean rates shown for the six, the seven and the eight year-old children.

The XIV Report (Hill et al., 1954) stated: "The combined 6 to 8-year-old children had a permanent tooth DMF rate of 149.76 per 100 children in 1946 and 65.82 in 1953. This is a difference of 60.38 per cent." In fact, by using their standard method of calculation, the "difference" is 56.05 per cent.

A confusing calculation. The situation is made even more confusing by the figures shown in Table 6 of the XVI Report (Hill et al., 1956). If the method commonly used in these trials is employed, when the difference between the DMF rates for 1946 and 1955, which is 95.90 (the rates being 149.76 and 53.86), is expressed as a percentage of 149.76, the "per cent difference" is 64.04, not 64.11 as shown. However, if the correct figure of 150.09 (which does not appear to have been mentioned in these reports) is substituted for 149.76, the "per cent difference" becomes 64. 11 as shown in their Table 6.

Was sampling used? The six, seven, eight and twelve, thirteen, fourteen year age groups were chosen for study (Blayney and Tucker, 1948), but it was not stated whether all children of these ages (the ages were taken to the nearest birthday) were examined, or whether a sampling method was used. The VII Report of Hill et al. (1951) said that "0. 1 per cent of the children in the control area were Negro." However, in the XV Report (Hill et al., 1957a) it was stated that "the control area (Oak Park) examinations included only public school white children". It is not clear whether the Negro children in that city were excluded from the examination by design, or by the chance of a sampling method. The former alternative is suggested by the statement of Hill et al. (1955) that "In the control village of Oak Park, only public school children were studied".

Were children "continuous residents"? It is not clear whether all the children included in the early reports (Blayney and Tucker, 1948; Hill et al., 1950, 1951, 1952, 1953, 1954) were "continuous residents". Although the questionnaires recorded the residence record of each child, it was not until the X1 Report (Hill et al., 1955) that the statement was made that "The data given in this report are limited to those children whose entire lives have been on Lake Michigan water." The United Kingdom Mission (1953) stated that "The study includes only white children attending public schools in the city who have lived in the area continuously from birth." However, as the first part of that statement presents an incomplete description of the authors" method, doubt is raised as to the accuracy of the statement made in regard to continuous residence.

Disturbing disagreements. In the following paragraphs are cited some disturbing disagreements between the statements made regarding the number of children examined. No suggestion has been found that more than one series of examinations was conducted in Evanston in each year from 1946 onwards, and in Oak Park in 1947 and 1956. Therefore, although the situation is uncertain regarding sampling and continuous residence. it would be expected that all the reports would agree with regard to the number of subjects of each age that were examined in each individual year. The exception is the XVII Report (Hill et al., 1957b), which compares the caries rates of white with those of Negro children; for it was stated that "in this report no attempt has been made to limit the examinations to continuous resident children." Therefore, it would be expected that the sample sizes shown in this report may be larger than those published in other reports.

Gross discrepancies between sample sizes. The numbers of children of each of the ages twelve, thirteen and fourteen years that were examined in 1946, 1949, 1952 and 1954 were given in the second column of Tables XI and XII of the XV Report (Hill et al., 1957a), the same figures appearing in both tables. It is to be noted that in eleven out of the twelve cases, the sample sizes given there are different from those shown in Tables 111, V, VI, VII, VIII, IX and X of the same report. In six cases the samples were larger in Tables XI and XII than in the other tables mentioned, and in five cases they were smaller. The largest discrepancy was between the number of children aged twelve years that were examined in 1949. Tables XI and XII showed this figure as 627, and the other tables gave 522 as the sample size. Similar discrepancies (for 1946, 1949 and 1952) are present between the sample sizes shown in Tables IX and X of the 1955 paper of these authors, and Tables 1, 111, IV, V, VI, VII and VIII of that report. The authors (Hill et al., 1957a) stated: "The number of teeth and surfaces associated with the DMF rates from 1946 through 1954 are shown in Tables XI and XII." In other tables mentioned in that report the "Rate per hundred children" was employed, but there appears to be no reason why the number of children examined should not be the same for both of these comparisons. No explanation for the different sample sizes was advanced by the authors.

Disparities in Negro sample sizes. Marked disparities are seen between the sample sizes shown for Negro children, for, judging from Table 10 of the XVII Report (Hill et al., 1957b), data >from only about half of the Negro children aged twelve to fourteen years who were examined in 1946, and of less than a third of those examined in 1954, were included in the XV Report (Hill et al., 1957a). The number studied is given in Table IV of the latter paper as 96 in 1946, and as 79 in 1954. However, the XVII Report (Hill et al., 1957b, Table 10), shows that 188 Negro children of those ages were examined in 1946, and 250 in 1954.

The XI Report (Hill et al., 1955) also shows that 96 Negro children were examined in 1946. The VII and XVIII Reports (Hill et al., 1951, 1958), although they do not state the number of Negro children, indicate the same sample size, 1,701 children, as the XI and XV Reports (Hill et al-, 1955, 1957a). In the last mentioned report, referring to the 1954 results, the authors said: "It is admitted that the Foster (Negro) school sample (79) was limited." Why, then, were so few of the 250 Negro children aged twelve to fourteen years that were examined in that year included in the report? Were less than a third of these children continuous residents?

The situation with regard to children aged six to eight years cannot be investigated, because the XVII Report is the only one in which the data of the younger age group of Negro children are shown separately from those of the white children.

Further unexplained differences. The position revealed in the last paragraph is further confused by the presence of large variations between the number of white children, aged twelve to fourteen years, whose data were shown in earlier reports, and the number given in Report XVIL In the former reports (Hill et al., 1955, Table 11; 1957a, Table IV) the number of these children examined in 1946 (public plus parochial schools) is stated to be 1,605, but, according to the XVII Report (Hill et al., 1957b, Table 10) the number seen in that year was 1,368. In 1954 the examinations of white children totalled 1,247 (Hill et al., 1957a, Table IV), but the figure of 1,905 is shown in the XVII Report (Hill et al., 1957b).

In the younger children, as no dissection of the data into school groups has been published, only the total number inspected can be considered. The XVII Report (Table 10) states that 1,754 children were examined in 1946 and 2,952 in 1955; but Table I of the XVI Report (Hill et al., 1956) shows 1,991 and 1,376 examinations respectively. The two statements of sample sizes (XVII Report figures minus the XVI Report ones) therefore differ by -237 and + 1,576 children.

It is possible that the larger sample sizes shown in the XVII Report for the examinations in 1954 and 1955 were due, despite the sizes of the increases (171 Negro and 658 white children aged twelve to fourteen years, and 1,576 children aged six to eight years), to the inclusion of all subjects, and not only those who were "continuous resident children". If, at the time of commencement of the study in 1946, children who had not lived in Evanston "continuously" since birth were excluded from the main study, an explanation can be found for the larger number of Negro children included for that year in the XVII Report. However, it is strange that that report, which included children who were not "continuous residents" (Hill et al., 1957b), in 1946 should be based on 237 fewer white children aged twelve to fourteen years and on 237 fewer white plus Negro children aged six to eight years than were included for that year in the other reports mentioned.

Incompatible statements. The authors made incompatible statements regarding the total number of children examined during the initial examinations in Evanston and Oak Park. In Report II (Blayney and Tucker, 1948) it was stated that the "baseline observations were made on 4,375 North Shore" (study area) "children and 2,493 Oak Park children." These figures were repeated in 1950 by Hill et al. However, Tables I to VI of the latter paper show that 1,991 children aged six to eight years were examined in Evanston in 1946; Tables 1, 11 and III of Hill et al. (195 1) indicate that 1,701 children aged twelve to fourteen years were examined in that year, that is, a total of 3,692 children. One or both of these figures (1,991 and 1,701) were repeated by the authors (or may be obtained by adding figures for individual yearly age groups) in 1952, 1955, 1956, 1957a and 1958.

Figure 5. Incompatible statements regarding the number of children inspected during the initial examinations in Evanston and its control city of Oak Park. Evanston statement A is from Blayney and Tucker (1948) and Hill et al. (1950). Statement B is from Hill et al. (1950, 1951, 1952, 1955, 1956, 1957a and 1958). Statement C is from Hill et al. (1957b). Oak Park statement D is from Blayney and Tucker (1948) and Hill et al. (1950), and statement E from Hill et al. (1958). See p. 211.

The third total sample size for Evanston in 1946 is shown in the XVII Report (Hill et al., 1957b). By totalling the figures in Table 10, it appears that 1,754 children aged six to eight years, and 1,556 aged twelve to fourteen years, were examined, a total of 3,310 subjects. From Tables I and 2 of Hill et al. (1958) it is deduced that a total of >2,051 children were examined in Oak Park in 1947 (see figure 5, p. 167).

Therefore, three very different sample sizes were given for the 1946 examination in Evanston: 4,375, 3,692 and 3,310; and two total sample sizes of 2,493 and 2,051 subjects examined in Oak Park in 1947. The smallest sample size for Evanston (3,3 10) was given in the XVII Report, despite the statement of the authors (Hill et al., 1957b) that "in this report no attempt has been made to limit the examinations to continuous resident children."

Remarkable changes in assessment of statistical significance. In the footnote to Table II in Hill et al. (1952) it was stated: "It should be noted that the caries rates per 100 children for the 6-8 year olds as a group shown in this report, vary slightly from those shown in previous reports." Although these were said to be slight variations, the remarkable fact emerges that, although based on the same data, the difference between the 1946 and the 1948 caries attack rates for the deciduous teeth of children of that age range, which was said to be statistically significant (the probability being given as 0.005) in the 1950 Report, was stated by the same authors, in 1952, to be "not statistically significant."

On reading the X Report (Hill et al., 1952), it appears that even more extraordinary changes of opinion with regard to the significance of results based on the same data occur in five comparisons between the rates of permanent teeth; significant differences (probability "0.0000") being altered to "not statistically significant." However, a correction (J. dent. Res., 31, 597) stated that the footnotes to Tables IV, V, VI, VII and VIII were incorrect, and that the statements: "Differences are not statistically significant" should have read "Differences are statistically significant". It is considered likely that the correction is incomplete, and that in the footnote to Table IX of that paper, the word "not" should be deleted. If this alteration is not made, that footnote indicates that the difference between the rates for 1946 and 1948 is "not statistically significant", although two years earlier, the difference computed from the same data was stated in the footnote to Table VI of Hill et al. (1950) to be significant (probability "0.0000") .

At first sight, the employment of statistical terminology in the presentation of this study engenders confidence in the results reported, but the few examples which have been cited clearly indicate their unreliability.

Cover | Introduction | 1a | 1b | 1c | 1d | 2a | 2b | 2c | 2d | 3

---