Errors
and Omissions in Experimental Trials - 1b
THE
EVANSTON STUDY
The United Kingdom Mission (1953), after having
observed the Evanston study, described it as "one of the most elaborate
investigations." Hill et al.
(1950) considered that they had planned the study so "as to measure every
variable that might exert an influence and obscure the findings." It is
the only trial in which bite-wing examinations were made for all subjects
examined.
The
importance of X-ray examinations.
Blayney and Greco (1952) reported that in this trial "the X-ray disclosed
53.84 per cent of the total number of carious lesions observed by both clinical
and X-ray methods". That said: "We believe it extremely important to
employ both clinical and X-ray techniques in any study program which is directed
toward the determination of the prevalence or the control and reduction in the
rate of caries attack." This result must throw considerable doubt on the
accuracy of the caries attack rates which were reported from the test and
control areas in the other studies considered; for in these, X-ray examinations
were incomplete or absent.
The ideal
control community. The authors of
the study stated that "It seemed logical to think of Oak Park, Illinois,
as the ideal control community because of its close similarity to the study
area" (Blayney and Tucker, 1948). The manner in which that city resembled
Evanston was not stated. The United Kingdom Mission (1953) made the important
observation that in Evanston the economic level was high, and "dental care
was outstandingly good."
Lower
caries rates in control community. It
soon became apparent that Oak Park could not be called "the ideal control
community", for Hill et al.
(1951) stated that "Comparison of the caries rates of all children in the
study area (Evanston, Ill.) and the control area (Oak Park, Ill.) prior to the
addition of sodium fluoride to the communal water supply of the study area
indicated a lower caries rate for school children of the control area."
Different
rates in student groups. The authors continued:
In an effort to find the source of
these differences in caries prevalence, it was found to be due largely to
differences in the make-up of the student groups examined in the two areas.
While in the study area 22.2 per cent of the children examined were attending
parochial schools, no such children were included in the control area: and while
5.6 per cent of the children in the study area were Negro children, only 0.1
per cent of the children in the control area were Negro. Statistically
significant differences were found to exist between the caries rates of Negro
and parochial school children on one hand and public white school children on
the other hand. Generally the caries rates of parochial school children were
found to be higher and those of Negro children lower than those of white
children in public schools.
Exclusion
of data. Hill et al. (1951) continued:
Therefore, comparisons of caries rates
for the study group and the control group are based on the caries experience of
public white school children only, while such comparisons involving children in
only the study area are based on the caries experience of all children in
total. The caries rates for the Evanston white school children in the 1946
survey and the Oak Park white school children in the 1947 survey were very
similar.
Six lines later, it was stated:
"In further comparing the rates for Oak Park (control) and Evanston (study
area) it is apparent that the baseline figures are very similar."
The only comparisons that can be made
from the paper which has just been mentioned are the figures for the children
aged twelve, thirteen and fourteen years. Negro and parochial school children
constituted 27.8 per cent of the Evanston children. By excluding this part of
the data the rates in that city were then considerably lower than those in the
control city, the rates (Table IV) being 707.51, 946.17 and 1133.33 in every
100 of the Evanston public school white children for the ages twelve, thirteen
and fourteen years; those in Oak Park being 774.29, 970.00 and 1194.64 for the
same three ages.
An
altered explanation.
A different, but, at first sight, a reasonable explanation for the exclusion of
the data of Negro and parochial school children, when making comparisons with
data from Oak Park, was given in the XV Report (Hill et al., 1957a): "As the control area (Oak Park) examinations
included only public school white children it was necessary to evaluate the
Evanston data on the basis of school groups, public white, parochial, and
Foster (Negro) to make comparisons of like groups." It can be seen that in
that paper the exclusion of data was attributed, not to the fact that this
process was undertaken because there was "a lower caries rate for school
children of the control area" (Hill et
al., 1951), but to the different racial composition of, and type of school
attended by the children in the two cities. Hill et al. (1950) mentioned that one of their seven "other
objectives" was "to compare the dental caries experience of white
with that of Negro school children." No reference was made to the
possibility of a difference being found between the rates of white public and
parochial school children. However, the original statement (Hill et al., 1951) makes it clear that the
different school groups were taken into account only after the unsatisfactory
results of the first Oak Park examination became apparent.: "In an effort
to find the source of these differences in caries prevalence." In
assessing the accuracy of the second (1957a) explanation, it should be realized
that in the younger age group "comparisons of like groups", or even the
dissection of the data into the three school groups, were not published in the
reports dealing with that age group, namely the 1950, 1952, 1954, 1956 and
1957b papers, or even in the XV Report (Hill et al., 1957a) which dealt with both age ranges, but showed this
dissection for the children of the older age group only. Furthermore, when,
after a delay of more than ten years, the 1947 Oak Park rates for the younger
children were published for the first time by Hill et al. In 1958, no "comparisons of like groups" were made
by them. The reader is prevented from making this comparison by the fact that,
even now, the dissection of this age range into the three school groups has not
been published, despite the statement by Hill et al in 1951 that the rates for "school children" were
significantly different in each type of school.
"Correction"
of data. When making comparisons with the
control city, the authors excluded from the three groups of data obtained in
the test city the two which diverged most from the rates of the children in the
control city (Hill et al., 1951).
This process should be considered in connection with the following statement
(Hill et al., 1950):
In order to be able to generalize from
our findings, we must be certain that any such variables as effect caries
experience are represented in our study to the same extent as in the
population. Before drawing any ultimate conclusions, we will, therefore,
correct our data in such a manner as to include only those groups of children
which are representative of the population, with respect to dental caries
experience. We feel that this precaution is necessary to allow the ultimate
findings to be considered valid and reliable.
However, the process which they
described - the arbitrary selection of a section of the data, which is then
termed "representative" - instead of making "the ultimate
findings to be considered valid and reliable", would render a report based
on this selected data unfit for serious consideration.
"Population"
sampled.
It is not clear what the authors meant by the term "the
population." If the population referred to was that of Evanston, the
sample of children examined in this study - if properly drawn - provided an
unbiased estimate of the dental condition of the population of that city; if only
some of the data are included, the results will be biased. If this term
"population" was intended to refer to the general population of the
U.S.A., it should be realized that the results from Evanston can represent only
a stratum of the country as a whole, varying as to climate and racial
composition, to mention only two variables.
It will be recalled that the caries
rates were said to be significantly different, even between children attending
the different types of school in Evanston; and also that the rates in that city
were considerably different from those in Oak Park, which was at first stated
to be "the ideal control community" for Evanston (Blayney and Tucker,
1948). These differences emphasize the fact that caution should be exercised
when applying results obtained in a test city to a wider population, of which
the test city may not be representative.
Altered
methods in latest report.
In the latest report (Hill et al., 1958) which shows the findings for the permanent teeth of
children in the control city of Oak Park, the authors have published in the
same tables as the results of the control groups, the DMF rates, not of the
public school white children, but of the total sample of Evanston children.
This is strange in view of their statement that "comparisons of caries
rates for the study group and the control group are based on the caries
experience of public white school children only" (Hill et al., 1951). It would appear that they
no longer held the opinion which they stated the previous year (Hill et al., 1957a) that it is necessary
"to make comparisons of like groups."
As a result of this change in procedure
the differences between initial caries rates in Evanston and Oak Park are
diminished. In children aged twelve to fourteen years, the pre-fluoridation
rates reported for the 1,226 public school white children in Evanston were far
closer to the values found in the Oak Park children than were either the rates
of the 96 Negro, or of the 379 parochial school children (Hill et al., 19571, 1958). However, the rates
of the Negro children were lower, and the rates of the parochial school
students were considerably higher than those of the public school white
children. By adopting the authors' latest (1958) method, which is to add the
results of the three groups, it is found that the pre-fluoridation rates of the
twelve and fourteen-year-old children are considerably less divergent from
those of the initial examinations in Oak Park - and those of the thirteen white
children of those ages. Whether this situation arises with regard to the six-,
seven- and eight-year old children cannot be determined, for no dissection into
the rates prevalent in the three school groups has been published.
Late
examination in the control city. The United Kingdom
Mission (1953) stated: "Before fluoridation started a dental survey was
made of 4,375 children in the selected groups in Evanston and of 2,493 children
in Oak Park. Further examinations have been carried out each year since 1947
and will continue until 1962." However, the examinations in Oak Park were
not commenced until after the fluoridation of the Evanston water supply on 11
February 1947, for Blayney and Tucker (1948) stated: "The study in Oak
Park was instituted on Feb. 26, 1947". Also, at the time of the United
Kingdom Mission Report (1953), no further examinations had been conducted in
Oak Park; even in Evanston only one age group was examined during each year, as
can be seen by inspecting the "schema for study" published by Blayney
and Tucker in 1948, and reproduced in several subsequent reports.
Only
two examinations in the control city. This "schema" indicates that
the design of the trial provided for only two examinations - eleven years apart
- to be made in the control city. It would appear that the authors did not
anticipate changes in the caries rates of the control, such as were reported in
Muskegon (Arnold et al., 1953), and,
as will be seen later, in Sarnia (Brown et
al., 1954b), and in Kingston (Ast, Finn and Chase, 1951). The first
examination was made in 1947, and the second, although not scheduled until
1958, was commmenced in 1956 when it became apparent that the water supply of
Oak Park would be fluoridated (Hill et al.,
1956). This examination was completed on 14 November 1956, soon after the fluoridation
of the Oak Park water on 1 August (Hill et
al., 1958).
A
ten-year delay in the publication of data. Caries
attack rates for the six-, seven- and eight-year-old children which were
obtained in Oak Park in 1947 (Blayney and Tucker, 1948) have only recently been
published by Hill et al. (1958). This
great delay is inexplicable and is particularly unfortunate, because it is in
regard to these younger children that the major claims are made for reduction
of dental caries as a result of fluoridation. No explanation was offered for
this delay, and the members of the United Kingdom Mission (1953) did not
comment on this strange omission, merely saying that "The incidence of
caries among the children aged 6-8 years is compared with the baseline data of
Evanston itself while caries experience of children aged 12-14 years is
compared with that of Oak Park."
Gross
differences in initial caries rates. The latest report (Hill et al., 1958) reveals that in the
younger children there were gross differences between the initial caries attack
rates in Evanston and Oak Park. The rates were: 46.85, 26.89 for age six years;
153.49, 102.63 for age seven years; and 249.93, 222.44 for age eight years in
Evanston and Oak Park respectively.
In regard to the great difference between
the pre-fluoridation rate for the six-year-old children in Evanston and the
initial one for children of that age in Oak Park, 46.85 and 26.89 respectively,
a footnote to Table I (Hill et al.,
1958), referring to the former rate, stated: "This figure results from the
very high DMF rate of 87.91 found in one school in 1946." However, as the
children were drawn "from 24 schools in the study area" (Blayney and
Greco, 1952), it is probable that the rates for six-year-old children in most
schools approached the figure of 46.85, unless the school with the high DMF
rate also happened to provide a disproportionately large number of six-year-old
children.
It should be noted that no comment on
the magnitude of this rate of 46.85 was made in any of the four reports in
which it had been shown previously (Hill et
al., 1950, 1952, 1956, 1957a); all of which were published before the rate
of 26.89 for Oak Park was released, and therefore before a comparison with it
could be made. The rate of 46.85 was used in all those papers - and even in
their latest report (1958) - in calculating the "% reduction", and in
computing the "Probability of difference due to chance."
Much
unpublished data. The members of the Evanston Dental
Caries Study devoted most of the years 1947 and 1956 to the collection of data
from children in Oak Park (Blayney and Tucker, 1948; Hill et al., 1958). Despite this fact, the major part of each of the two
tables shown in the XVIII Report (Hill et
al., 1958) was devoted to a re-presentation of data obtained in Evanston,
although this report was said to have as its purpose the comparison of the
permanent teeth dental caries experience rates in children examined in Oak Park
in 1947 and 1956. The Oak Park data were restricted to four lines of figures
showing the DMF rates in permanent teeth. No report was made of other findings
such as those which had been shown in reports on Evanston children. For
instance, in the XV Report (Hill et al.,
1957a), no fewer than eight tables relating to the twelve-, thirteen and
fourteen year-old children only were devoted to these other findings. This very
incomplete presentation of the data obtained in Oak Park is unaccountable.
 |
Figure
3. Gross differences in initial caries rates in Evanston and
its control city of Oak Park. The Oak Park rates remained unpublished for
over ten years.
|
Disagreements
between results. In their XVIII Report, Hill et al. (1958) stated: "The DMF
rates and percentage reduction from year to year for the Evanston children of
all age groups shown in Tables I and 2 have been published in previous reports.
However, four of the figures for the year 1955, shown in Table I of the 1958
Report, are different from "the rates and percentage reduction"
given, for the same year, in Table I and the text of the XVI Report (Hill et al., 1956). The DMF rates at age
seven years were only slightly different (40.95 and 40.92, in the XVI and the
XVIII Reports respectively), but at age eight years the two rates were 114.04
and 120.32. It is very improbable that these different rates are due to typographical
errors, for they were confirmed by the "per cent reduction from
1946", which was given in the summary and in Table I of the respective
reports as 73.32 and 73.34 for children aged seven years, and as 54.37 and
51.85 for those that were eight years of age. This "reduction" was
shown in the XVIII Report as 85.96 for the six-year-old children, but in the
XVI Report it was given as "80 per cent" in the findings and as
"85.96 per cent" in the summary.
Disagreement
between tables. The DMF rate in terms of tooth surfaces
was given only twice in this study (Hill et
al., 1955, Table X and 1957a, Table XII). In both papers the "DMF rate
per 100 surfaces" for children aged fourteen years was 14.82 in 1949 and
13.94 in 1952. However, in the former report this rate was given as 15.09 in
1946, but in the latter one, for children of the same age in the same year, the
figure shown was 15.92. As a result of this change, the "% differences
from 1946" were altered from 1.78 to 6.85 (1949) and from 7.62 to 12.44
(1952). By using these new rates it can be said that "all 3 methods,
namely; per hundred children, per hundred teeth, and per hundred surfaces all
express approximately the same proportion of percentage differences in
rates" (Hill et al., 1957a).
This result is a good illustration of the comment made on the method most
commonly used in these studies to express changes in caries experience, that
"relatively small variations in the baseline values will produce
substantial alterations in the percentage reduction obtained" (Part One,
p. 137).
It may be mentioned that the
"total tooth surfaces considered" for thirteen-year-old children in
1954 (Table X11, Hill et al., 1957a)
should be 58,325 not 58,352; and that for fourteen-year-old children in 1949,
in the column of that table giving the "% differences from 1946", the
figures shown should be 6.91 not 6.85. In their XI Report (Table IX) and their
XV Report (Table XI), Hill et al.
(1955, 1957a) showed different figures for children aged twelve years examined
in 1952. Although both tables show the same total number of teeth considered,
in the former table children were shown as examined, with a "DMF rate per
100 teeth" of 25.76, and a difference from 1946 of 19.50 per cent. In the
latter table, the figures were 516, 25.60 and 20.00 per cent respectively. In
1953 Hill et al. published the figure
of 19.50 per cent.
No
data for deciduous teeth. The authors have not published
any data regarding the deciduous teeth of children in the control city, either
for the first (1947) or the second (1956) examination. The most important
omission, the def rates, could have been shown by adding only two lines to
Table I in Hill et al. (1958). This
omission is particularly unfortunate in view of the fact that in the deciduous
teeth in Evanston during the first four years of fluoridation the def rate of
the six to eight years group was considerably higher than the initial one (Hill
et al., 1952). It was not until nine
years after the commencement of the study that a significant reduction in this rate
was reported.
In 1950, Hill et al. stated that the caries rate for deciduous teeth in these
children "does not indicate any trend", despite the fact that in
Table I of that report the initial rise in this rate during the first two years
of fluoridation was shown by them to be statistically significant (P = 0.005).
Two years later these authors altered their opinion of the significance of this
rise. In 1952 they re-published the same data for children aged six, seven and
eight years in 1946 and 1948, but computed different rates for the combined age
group six to eight years. The rise in the def rate was then said to be not
statistically significant.
Variations
in caries rates in control. The meagre data regarding
caries attack rates in Oak Park which have been published are included in
Tables I and 2 of Hill et al. (1958).
Of the six age groups shown, between the years 1947 and 1956 the authors
reported a significant increase in the DMF rate of children aged seven years,
and non-significant upward trends in the rates of those aged eight and thirteen
years, and downward ones in the caries attack rates in children aged six,
twelve and fourteen years. (The question of "significant" changes in
the rates in control cities will be considered later.) The authors said:
"The children 12, 13 and 14 years of age, Table 2, have only minute
differences between the 1947 and 1956 rates. These are not considered to be
significant." The footnote to that table is more definite, in each
comparison stating: "Difference is not statistically significant."
Although these differences of 61.20, 34.96 and 58.87 DMF teeth, for children
aged twelve, thirteen and fourteen years respectively, were termed "minute
differences", those seen in the rates of the twelve and fourteen-year-old
children are approximately a third the size of the absolute drop in the rates
recorded for the same age groups in Evanston since the inception of
fluoridation. It cannot be assumed that the fluctuations in the rates during
the intervening period of nine years, when no examinations were made, did not
exceed the differences between the initial and final rates. It will be recalled
that considerable variations occurred in Muskegon (see Figs 1 and 2).
Inadequacy
of the control.
Blayney and Tucker (1948) realized that "A study of
this nature must have an adequate control." Therefore, it is strange that
in the "schema" which they published there was provision for only two
examinations, eleven years apart, to be made in the control area. It should
have been obvious that the usefulness of data gathered in such a manner would
be, at most, very limited. The explanation given by the authors for their
failure to examine the children in the control city "every year"
(instead of only twice) was the strange one that "It was not necessary to
do so in as much as Evanston and Oak Park are subjected to the same advertising
campaigns, have a similar economic level, participate in comparable educational
programmes, and so forth" (Hill et
al., 1958). It is extraordinary that the authors advanced this explanation
and that they adhered to such a plan, despite the marked dispanity in canes
rates disclosed in the first examinations in Evanston and Oak Park (Hill et al., 1958), which makes it obvious
that the latter city was a poor choice in seeking an "adequate
control" for the former one.
Differences
between school groups. Hill et al. (195 1) stated that "statistically significant
differences were found to exist [in 1946] between the caries rates of Negro and
parochial school children on one hand, and public white school children on the
other hand." However, they made a further statement that "the caries
rates of parochial school children were found to be higher and those of Negro
children lower than those of white children in public schools" (Hill et al., 195 1). These two statements are
inconsistent. The first appears to mean that the comparisons between Negro
children and white children in public schools, and that the comparison between
white children attending parochial schools and those attending public schools,
were both statistically significant in 1946.
"Nearly
comparable" or significantly different? The XV Report Hill et al., 1957a) stated that "In 1946 and 1954 the public school
white children and the Foster School (Negro) children maintained nearly
comparable DMF rates". The actual rates" (per 100 children) in 1946
for twelve, thirteen and fourteen-year-old white children attending public
schools were 707.51, 946.17, and 1133.33; for the Negro children of the same
ages they were 658.82, 861.76 and 1035.71. (The rates of each school group of
younger children were not published.)
It is not understood how the same
authors could on one occasion (Hill et al.,
195 1) state that there were "statistically significant differences"
between the two series of rates, and later (Hill et al., 1957a) describe them as "nearly comparable DMF
rates" It may be thought that the word "maintained" referred to
a comparison between the DMF rates of the white children in public schools, and
of the children in the Negro school, between 1946 and 1954. However, this
cannot be the case, for the authors claimed for these twelve, thirteen and
fourteen-year-old children "a reduction of approximately 21.96 per cent in
dental caries-experience rates of the permanent teeth" (Hill et al., 1957a). (In this study,
percentages were frequently shown "approximately" to two decimal
places.) Table IV of that paper shows that both the Negro and the public school
children participated in the reductions reported.
Decline
in eruption rate. An observation of considerable interest
is obtainable from Tables V and VI of the X Report (Hill, et al., 1952). The former table shows the rates per 100 six, seven
and eight-year-old children that had occlusal surface pit and fissure caries or
fillings in their first permanent molars; the latter one, the number of these
teeth which were free from those defects. The mean number of erupted first
permanent molars per 100 children may be obtained, in each age group, by adding
these two rates to that showing the extracted and congenitally missing
permanent molars. It is probable that the number of congenitally missing teeth
was negligible and that the number of permanent molars which had been extracted
in these young children was small, particularly in the six years age group
(five and a half to six and a half years). Therefore, it would be expected
that, in each age group, the mean number of erupted molars per 100 children
would be similar at the time of each examination. This was the case in children
aged eight years; the figures for the examinations made in 1946
(pre-fluolidation), 1948, 1950 and 1951 being (to the nearest whole number)
387, 387, 384 and 386 respectively. At age seven years the numbers erupted were
330, 336, 320 and 315; but in the six-year-old children, the number of erupted
molars showed a marked and progressive decline 189, 156, 140 and 132 during the
period covered by those four examinations.
The question naturally arises whether the eruption rate of these teeth
had decreased; a possibility of extreme importance in interpreting the results
of a fluoridation trial. However, further consideration of this matter is
prevented by the authors' failure to publish this type of data when they
reported the results of the two later examinations (conducted in 1953 and 1955)
which were made of children of these ages; and the "schema for study"
indicates that children aged six to eight years will not be examined again
until 1960.
This failure to publish this type of data
for the 1953 and 1955 examinations is extraordinary, especially in view of the
fact that the authors continued to show similar data for the permanent molars
of the older age group (Hill et al.,
1955, 1957a); the latter report, the only one showing results for both age
groups, gave the prevalence of occlusal pit and fissure caries and fillings in
the molars of the older, but not of the younger age group.
In considering the eruption of teeth,
the odd method of assessment used in this study must be taken into account.
Hill et al. (1955) said: "Only
teeth which were 50 per cent or more erupted were considered. A carious or
filled tooth was, of course, considered regardless of its stage of
eruption."
 |
Figure
4. Suggestion of a progressive decline in the number of
erupted first permanent molar teeth in six-year-old children in Evanston. The
results obtained in the examinations conducted in 1953 and 1955 were omitted
from the published reports.
|
Strange
superiority of artificial fluoridation. The authors of
this study compared the Evanston DMF rates per child with those of children in
Aurora, Illinois (Dean et al., 1950)
in the expectation that after sufficient time had elapsed for all the erupted
teeth to have been formed since fluoridation commenced "the Evanston rate
will closely approach the Aurora rate" (Hill et al., 1957a). It is surprising that this parity between the rates
of Aurora and Evanston was expected, because in the Aurora survey only clinical
methods of examination were used, but in the Evanston examinations X-ray
surveys were used routinely. Hill et al.
(1951) stated: "We find our baseline figures for caries experience in
Evanston and Oak Park approximately 32 per cent higher than those of Dean and
his co-workers for Evanston and Oak Park in 1941. We assume this may be
explained partially by differences in the techniques of examination,
particularly in the use of X-ray in the current investigation." The United
Kingdom Mission (1953) stated that in this study "the minutest
radiolucency was taken as indicating caries."
In view of these findings, it is even
more strange that Hill et al. (1957a)
were able to report: "The Evanston 6 and 7-year-olds of 1953 have a lower
dental caries experience rate after 71 to 82 months of fluoridation than the
Aurora 6 and 7-year-olds of 1945-1946 with lifetime exposure to water naturally
fluoridated to 1.2 ppm." That this difference was not only slightly below
the 1945-1946 Aurora rate for children of the same age" (Hill et al., 1957a) can be seen by comparing
the actual rates reported. In Evanston and Aurora respectively, the rates were
14.73, 28.0 at age six years and 53.35, 70.5 at age seven years (Hill et al., 1957a; Arnold et al., 1953). It should be noted that
in Evanston two years previously (195 1), after a shorter period of
fluoridation, the rate for the six-year-old children was even lower, 12.36
(Hill et al., 1952) and was less than
half the Aurora rate; in 1955 (Hill et al.,
t956) it had become 6.58, less than a quarter of the Aurora rate. Blayney and
Greco (1952) found that in children in the Evanston study, with regard to
proximal caries "the 6-year-olds have the highest percentage (83.90)
disclosed by X-ray findings only. In the 7-year-old group 79.04 per cent of
proximal lesions were demonstrated by X-ray findings only". Therefore, if clinical
methods of examination only had been used in Evanston, as was the case in
Aurora, what may be thought to be a strange superiority of artificially over
naturally fluoridated water as a means of reducing dental caries attack rates
would have appeared to have been even more marked.
"Weighting"
of results. The method of combining the results of
the six, seven and eight-year-old children into one category introduces an
important source of error when comparisons are made between the results
obtained in the control city and in the test one, or between those found on
different occasions in Evanston. Owing to the great differences in caries
attack rates which are observed between children of these ages (the baseline
DMF rates for these three ages in Evanston were 46.85, 153.49, and 249.93,
according to Hill et al., 1950), the
results may inadvertently be "weighted" by including a preponderance
of young or of old children in the age group six to eight years. If this
occurs, the average value will be lower or higher than it would have been if
the three ages had been equally represented in the sample. In comparing the
results of the control and the test cities, "weighting" of this
nature could make it appear that large differences were present, when, in fact,
they were either slight or absent, or the presence of actual differences could
be hidden.
An
example of "weighting". The results of the
pre-fluoridation, and of the first post-fluoridation survey at Evanston (Hill et al., 1950), clearly demonstrate the
process of "weighting" and show that its occurrence is not merely a
theoretical possibility. On these two occasions, the number of children in each
of the age groups six, seven and eight years that were examined in 1946 was
461, 759 and 771 respectively; the corresponding numbers seen in 1948 were 756,
838 and 440. On both occasions the results of the three ages were combined, and
a caries rate was computed for the age range six to eight years.
Significant
tests and "weighting ". Despite the rather
obvious "weighting" in the examples which have just been cited, tests
were applied to determine the significance of the difference between the caries
attack rates found during the two examinations in the combined age range six to
eight years. In regard to the permanent teeth, it was stated that "The
probability of this difference being due to chance is 0,0000" (Hill et al., 1950). Curiously, in those teeth
a decrease in the caries rate was reported, contrasting with the statement of a
significant rise in the rate of the deciduous ones.
Random
variation ignored.
Hill et al. (1950)
stated: "It is to be expected that the rate of caries in all teeth varies
from year to year due to chance. A significant reduction of caries prevalence
can therefore be assumed to exist only when the statistical analysis of the
data provides almost absolute certainty that the observed differences are not
due to chance." However, in a subsequent paper (Hill et al., 1956) these authors ignored the variations in the
intervening years, even when these were as marked as those in Table 5 of that
report, and stated: "Difference between 1946 and 1955 rates is
statistically significant."
Original
results altered. In the X Report (Hill et al., 1952), and in all the later
ones, alterations were made to the rates shown for the years 1946 and 1948 in
children of the combined age group six to eight years, which were published by
Hill et al. in 1950 (Tables I to VI).
The original rates were replaced by values which are the means of the mean
rates for the children of each of the three ages six, seven and eight years
(Hill et al., 1952, Tables 11 to IX).
System
of computation changed. The change in the system of
computation was explained by Hill et al.
(1952) in these terms: "The group averages, shown in previous reports,
represents weighted averages of the individual mean caries rates. Inasmuch as
the composition of the groups of children with respect to the number of 6, 7
and 8-year-olds varies from year to year, it was felt that unweighted group
averages form a more sound basis for comparison of group caries rates between
years."
The
new method of computation. In 1952 Hill et al. stated that "The new
averages were obtained by taking a simple arithmetical mean of the individual
caries rates of the 6, 7 and 8-year-old children." This description of the
new method is apt to cause some confusion, for it is considered to describe
accurately the old method. It was used by these authors in 1950, and then
abandoned by them in favour of the new one. The results for 1950 and 1951 in Table
IV of Hill et al. (1952), and those
for 1953 in Table I of Hill et al. (I
957a), and for 1955 in Table I of Hill et
al. (1956) make it clear that in this new method of calculation, the rate
per 100 children aged six to eight for each examination was obtained by taking
a simple arithmetical mean of the mean rate for each of the three ages six,
seven and eight years.
Errors
in amended rates. The amended rates published by the
authors (Hill et al., 1952) for the
age group six to eight years need further amendment, and the difference between
them is even less than that stated. The mean of the three values shown for 1948
in their Table IV, 23.54, 103.58 and 194.09, is found to be 107.07, not 92.07
as stated; also, the mean of the three values for 1946 - 46.85, 153.49 and
249.93 is 150,09 not 149.76. These errors were repeated in the XV and the XVI
Reports (Hill et al., 1957a, 1956).
The figure 149.76 was shown also in the
XIV Report (Hill et al., 1954). In
that report the rate for age six to eight years was said to be "65.82 in
1953." However, in Table I of the XVI Report (Hill et al., 1956) the rate for 1953 for age six to eight years was
given as 63.52. The latter figure is the mean of the three mean rates shown for
the six, the seven and the eight year-old children.
The XIV Report (Hill et al., 1954) stated: "The combined
6 to 8-year-old children had a permanent tooth DMF rate of 149.76 per 100
children in 1946 and 65.82 in 1953. This is a difference of 60.38 per
cent." In fact, by using their standard method of calculation, the
"difference" is 56.05 per cent.
A
confusing calculation. The situation is made even more
confusing by the figures shown in Table 6 of the XVI Report (Hill et al., 1956). If the method commonly
used in these trials is employed, when the difference between the DMF rates for
1946 and 1955, which is 95.90 (the rates being 149.76 and 53.86), is expressed
as a percentage of 149.76, the "per cent difference" is 64.04, not
64.11 as shown. However, if the correct figure of 150.09 (which does not appear
to have been mentioned in these reports) is substituted for 149.76, the
"per cent difference" becomes 64. 11 as shown in their Table 6.
Was
sampling used? The six, seven, eight and twelve,
thirteen, fourteen year age groups were chosen for study (Blayney and Tucker,
1948), but it was not stated whether all children of these ages (the ages were
taken to the nearest birthday) were examined, or whether a sampling method was
used. The VII Report of Hill et al.
(1951) said that "0. 1 per cent of the children in the control area were
Negro." However, in the XV Report (Hill et al., 1957a) it was stated that "the control area (Oak Park)
examinations included only public school white children". It is not clear
whether the Negro children in that city were excluded from the examination by
design, or by the chance of a sampling method. The former alternative is
suggested by the statement of Hill et al.
(1955) that "In the control village of Oak Park, only public school
children were studied".
Were
children "continuous residents"? It is not
clear whether all the children included in the early reports (Blayney and
Tucker, 1948; Hill et al., 1950,
1951, 1952, 1953, 1954) were "continuous residents". Although the
questionnaires recorded the residence record of each child, it was not until
the X1 Report (Hill et al., 1955)
that the statement was made that "The data given in this report are
limited to those children whose entire lives have been on Lake Michigan
water." The United Kingdom Mission (1953) stated that "The study
includes only white children attending public schools in the city who have
lived in the area continuously from birth." However, as the first part of
that statement presents an incomplete description of the authors" method,
doubt is raised as to the accuracy of the statement made in regard to
continuous residence.
Disturbing
disagreements.
In the following paragraphs are cited some disturbing
disagreements between the statements made regarding the number of children
examined. No suggestion has been found that more than one series of
examinations was conducted in Evanston in each year from 1946 onwards, and in
Oak Park in 1947 and 1956. Therefore, although the situation is uncertain
regarding sampling and continuous residence. it would be expected that all the
reports would agree with regard to the number of subjects of each age that were
examined in each individual year. The exception is the XVII Report (Hill et al., 1957b), which compares the
caries rates of white with those of Negro children; for it was stated that
"in this report no attempt has been made to limit the examinations to
continuous resident children." Therefore, it would be expected that the
sample sizes shown in this report may be larger than those published in other
reports.
Gross
discrepancies between sample sizes. The numbers of
children of each of the ages twelve, thirteen and fourteen years that were
examined in 1946, 1949, 1952 and 1954 were given in the second column of Tables
XI and XII of the XV Report (Hill et al.,
1957a), the same figures appearing in both tables. It is to be noted that in
eleven out of the twelve cases, the sample sizes given there are different from
those shown in Tables 111, V, VI, VII, VIII, IX and X of the same report. In
six cases the samples were larger in Tables XI and XII than in the other tables
mentioned, and in five cases they were smaller. The largest discrepancy was
between the number of children aged twelve years that were examined in 1949.
Tables XI and XII showed this figure as 627, and the other tables gave 522 as
the sample size. Similar discrepancies (for 1946, 1949 and 1952) are present
between the sample sizes shown in Tables IX and X of the 1955 paper of these
authors, and Tables 1, 111, IV, V, VI, VII and VIII of that report. The authors
(Hill et al., 1957a) stated:
"The number of teeth and surfaces associated with the DMF rates from 1946
through 1954 are shown in Tables XI and XII." In other tables mentioned in
that report the "Rate per hundred children" was employed, but there
appears to be no reason why the number of children examined should not be the
same for both of these comparisons. No explanation for the different sample
sizes was advanced by the authors.
Disparities
in Negro sample sizes. Marked disparities are seen between
the sample sizes shown for Negro children, for, judging from Table 10 of the
XVII Report (Hill et al., 1957b),
data >from only about half of the Negro children aged twelve to fourteen
years who were examined in 1946, and of less than a third of those examined in
1954, were included in the XV Report (Hill et
al., 1957a). The number studied is given in Table IV of the latter paper as
96 in 1946, and as 79 in 1954. However, the XVII Report (Hill et al., 1957b, Table 10), shows that 188
Negro children of those ages were examined in 1946, and 250 in 1954.
The XI Report (Hill et al., 1955) also shows that 96 Negro
children were examined in 1946. The VII and XVIII Reports (Hill et al., 1951, 1958), although they do
not state the number of Negro children, indicate the same sample size, 1,701
children, as the XI and XV Reports (Hill et
al-, 1955, 1957a). In the last mentioned report, referring to the 1954
results, the authors said: "It is admitted that the Foster (Negro) school
sample (79) was limited." Why, then, were so few of the 250 Negro children
aged twelve to fourteen years that were examined in that year included in the
report? Were less than a third of these children continuous residents?
The situation with regard to children
aged six to eight years cannot be investigated, because the XVII Report is the
only one in which the data of the younger age group of Negro children are shown
separately from those of the white children.
Further
unexplained differences. The position revealed in the
last paragraph is further confused by the presence of large variations between
the number of white children, aged twelve to fourteen years, whose data were
shown in earlier reports, and the number given in Report XVIL In the former reports
(Hill et al., 1955, Table 11; 1957a,
Table IV) the number of these children examined in 1946 (public plus parochial
schools) is stated to be 1,605, but, according to the XVII Report (Hill et al., 1957b, Table 10) the number seen
in that year was 1,368. In 1954 the examinations of white children totalled
1,247 (Hill et al., 1957a, Table IV),
but the figure of 1,905 is shown in the XVII Report (Hill et al., 1957b).
In the younger children, as no
dissection of the data into school groups has been published, only the total
number inspected can be considered. The XVII Report (Table 10) states that
1,754 children were examined in 1946 and 2,952 in 1955; but Table I of the XVI
Report (Hill et al., 1956) shows
1,991 and 1,376 examinations respectively. The two statements of sample sizes
(XVII Report figures minus the XVI Report ones) therefore differ by -237 and +
1,576 children.
It is possible that the larger sample
sizes shown in the XVII Report for the examinations in 1954 and 1955 were due,
despite the sizes of the increases (171 Negro and 658 white children aged
twelve to fourteen years, and 1,576 children aged six to eight years), to the
inclusion of all subjects, and not only those who were "continuous
resident children". If, at the time of commencement of the study in 1946,
children who had not lived in Evanston "continuously" since birth
were excluded from the main study, an explanation can be found for the larger
number of Negro children included for that year in the XVII Report. However, it
is strange that that report, which included children who were not
"continuous residents" (Hill et
al., 1957b), in 1946 should be based on 237 fewer white children aged
twelve to fourteen years and on 237 fewer white plus Negro children aged six to
eight years than were included for that year in the other reports mentioned.
Incompatible
statements.
The authors made incompatible statements regarding the total
number of children examined during the initial examinations in Evanston and Oak
Park. In Report II (Blayney and Tucker, 1948) it was stated that the
"baseline observations were made on 4,375 North Shore" (study area)
"children and 2,493 Oak Park children." These figures were repeated
in 1950 by Hill et al. However,
Tables I to VI of the latter paper show that 1,991 children aged six to eight
years were examined in Evanston in 1946; Tables 1, 11 and III of Hill et al. (195 1) indicate that 1,701
children aged twelve to fourteen years were examined in that year, that is, a
total of 3,692 children. One or both of these figures (1,991 and 1,701) were
repeated by the authors (or may be obtained by adding figures for individual
yearly age groups) in 1952, 1955, 1956, 1957a and 1958.
 |
Figure
5. Incompatible statements regarding the number of children
inspected during the initial examinations in Evanston and its control city of
Oak Park. Evanston statement A is from Blayney and Tucker (1948) and Hill et al. (1950). Statement B is from
Hill et al. (1950, 1951, 1952,
1955, 1956, 1957a and 1958). Statement C is from Hill et al. (1957b). Oak Park statement D is from Blayney and Tucker
(1948) and Hill et al. (1950), and
statement E from Hill et al.
(1958). See p. 211.
|
The third total sample size for Evanston in 1946 is
shown in the XVII Report (Hill et al.,
1957b). By totalling the figures in Table 10, it appears that 1,754 children
aged six to eight years, and 1,556 aged twelve to fourteen years, were
examined, a total of 3,310 subjects. From Tables I and 2 of Hill et al. (1958) it is deduced that a total
of >2,051 children were examined in Oak Park in 1947 (see figure 5, p. 167).
Therefore, three very different sample sizes were
given for the 1946 examination in Evanston: 4,375, 3,692 and 3,310; and two
total sample sizes of 2,493 and 2,051 subjects examined in Oak Park in 1947.
The smallest sample size for Evanston (3,3 10) was given in the XVII Report,
despite the statement of the authors (Hill et
al., 1957b) that "in this report no attempt has been made to limit the
examinations to continuous resident children."
Remarkable
changes in assessment of statistical significance. In the footnote to Table II in Hill et al. (1952) it was stated: "It
should be noted that the caries rates per 100 children for the 6-8 year olds as
a group shown in this report, vary slightly from those shown in previous
reports." Although these were said to be slight variations, the remarkable
fact emerges that, although based on the same data, the difference between the
1946 and the 1948 caries attack rates for the deciduous teeth of children of
that age range, which was said to be statistically significant (the probability
being given as 0.005) in the 1950 Report, was stated by the same authors, in
1952, to be "not statistically significant."
On reading the X Report (Hill et al., 1952), it appears that even more extraordinary changes of
opinion with regard to the significance of results based on the same data occur
in five comparisons between the rates of permanent teeth; significant
differences (probability "0.0000") being altered to "not
statistically significant." However, a correction (J. dent. Res., 31, 597)
stated that the footnotes to Tables IV, V, VI, VII and VIII were incorrect, and
that the statements: "Differences are not statistically significant"
should have read "Differences are statistically significant". It is
considered likely that the correction is incomplete, and that in the footnote
to Table IX of that paper, the word "not" should be deleted. If this
alteration is not made, that footnote indicates that the difference between the
rates for 1946 and 1948 is "not statistically significant", although
two years earlier, the difference computed from the same data was stated in the
footnote to Table VI of Hill et al.
(1950) to be significant (probability "0.0000") .
At first sight, the employment of statistical
terminology in the presentation of this study engenders confidence in the
results reported, but the few examples which have been cited clearly indicate
their unreliability.
No comments:
Post a Comment