Errors
and Omissions in Experimental Trials - 1b
THE EVANSTON STUDY
The United Kingdom Mission (1953),
after having observed the Evanston study, described
it as "one of the most elaborate investigations."
Hill et al. (1950) considered that they had
planned the study so "as to measure every variable
that might exert an influence and obscure the
findings." It is the only trial in which bite-wing
examinations were made for all subjects examined.
The importance of X-ray
examinations. Blayney and Greco (1952) reported
that in this trial "the X-ray disclosed 53.84 per
cent of the total number of carious lesions observed
by both clinical and X-ray methods". That said: "We
believe it extremely important to employ both
clinical and X-ray techniques in any study program
which is directed toward the determination of the
prevalence or the control and reduction in the rate
of caries attack." This result must throw
considerable doubt on the accuracy of the caries
attack rates which were reported from the test and
control areas in the other studies considered; for
in these, X-ray examinations were incomplete or
absent.
The ideal control community.
The authors of the study stated that "It seemed
logical to think of Oak Park, Illinois, as the ideal
control community because of its close similarity to
the study area" (Blayney and Tucker, 1948). The
manner in which that city resembled Evanston was not
stated. The United Kingdom Mission (1953) made the
important observation that in Evanston the economic
level was high, and "dental care was outstandingly
good."
Lower caries rates in control
community. It soon became apparent that Oak Park
could not be called "the ideal control community",
for Hill et al. (1951) stated that
"Comparison of the caries rates of all children in
the study area (Evanston, Ill.) and the control area
(Oak Park, Ill.) prior to the addition of sodium
fluoride to the communal water supply of the study
area indicated a lower caries rate for school
children of the control area."
Different rates in student
groups. The authors continued:
In an effort to find the source of
these differences in caries prevalence, it was found
to be due largely to differences in the make-up of
the student groups examined in the two areas. While
in the study area 22.2 per cent of the children
examined were attending parochial schools, no such
children were included in the control area: and
while 5.6 per cent of the children in the study area
were Negro children, only 0.1 per cent of the
children in the control area were Negro.
Statistically significant differences were found to
exist between the caries rates of Negro and
parochial school children on one hand and public
white school children on the other hand. Generally
the caries rates of parochial school children were
found to be higher and those of Negro children lower
than those of white children in public schools.
Exclusion of data. Hill et
al. (1951) continued:
Therefore, comparisons of caries
rates for the study group and the control group are
based on the caries experience of public white
school children only, while such comparisons
involving children in only the study area are based
on the caries experience of all children in total.
The caries rates for the Evanston white school
children in the 1946 survey and the Oak Park white
school children in the 1947 survey were very
similar.
Six lines later, it was stated: "In
further comparing the rates for Oak Park (control)
and Evanston (study area) it is apparent that the
baseline figures are very similar."
The only comparisons that can be
made from the paper which has just been mentioned
are the figures for the children aged twelve,
thirteen and fourteen years. Negro and parochial
school children constituted 27.8 per cent of the
Evanston children. By excluding this part of the
data the rates in that city were then considerably
lower than those in the control city, the rates
(Table IV) being 707.51, 946.17 and 1133.33 in every
100 of the Evanston public school white children for
the ages twelve, thirteen and fourteen years; those
in Oak Park being 774.29, 970.00 and 1194.64 for the
same three ages.
An altered explanation.
A different, but, at first sight, a reasonable
explanation for the exclusion of the data of Negro
and parochial school children, when making
comparisons with data from Oak Park, was given in
the XV Report (Hill et al., 1957a): "As the
control area (Oak Park) examinations included only
public school white children it was necessary to
evaluate the Evanston data on the basis of school
groups, public white, parochial, and Foster (Negro)
to make comparisons of like groups." It can be seen
that in that paper the exclusion of data was
attributed, not to the fact that this process was
undertaken because there was "a lower caries rate
for school children of the control area" (Hill et
al., 1951), but to the different racial
composition of, and type of school attended by the
children in the two cities. Hill et al.
(1950) mentioned that one of their seven "other
objectives" was "to compare the dental caries
experience of white with that of Negro school
children." No reference was made to the possibility
of a difference being found between the rates of
white public and parochial school children. However,
the original statement (Hill et al., 1951)
makes it clear that the different school groups were
taken into account only after the unsatisfactory
results of the first Oak Park examination became
apparent.: "In an effort to find the source of these
differences in caries prevalence." In assessing the
accuracy of the second (1957a) explanation, it
should be realized that in the younger age group
"comparisons of like groups", or even the dissection
of the data into the three school groups, were not
published in the reports dealing with that age
group, namely the 1950, 1952, 1954, 1956 and 1957b
papers, or even in the XV Report (Hill et al.,
1957a) which dealt with both age ranges, but showed
this dissection for the children of the older age
group only. Furthermore, when, after a delay of more
than ten years, the 1947 Oak Park rates for the
younger children were published for the first time
by Hill et al. In 1958, no "comparisons of
like groups" were made by them. The reader is
prevented from making this comparison by the fact
that, even now, the dissection of this age range
into the three school groups has not been published,
despite the statement by Hill et al in 1951
that the rates for "school children" were
significantly different in each type of school.
"Correction" of data.
When
making comparisons with the control city, the
authors excluded from the three groups of data
obtained in the test city the two which diverged
most from the rates of the children in the control
city (Hill et al., 1951). This process should
be considered in connection with the following
statement (Hill et al., 1950):
In order to be able to generalize
from our findings, we must be certain that any such
variables as effect caries experience are
represented in our study to the same extent as in
the population. Before drawing any ultimate
conclusions, we will, therefore, correct our data in
such a manner as to include only those groups of
children which are representative of the population,
with respect to dental caries experience. We feel
that this precaution is necessary to allow the
ultimate findings to be considered valid and
reliable.
However, the process which they
described - the arbitrary selection of a section of
the data, which is then termed "representative" -
instead of making "the ultimate findings to be
considered valid and reliable", would render a
report based on this selected data unfit for serious
consideration.
"Population" sampled.
It is not clear what the authors meant by the term
"the population." If the population referred to was
that of Evanston, the sample of children examined in
this study - if properly drawn - provided an
unbiased estimate of the dental condition of the
population of that city; if only some of the data
are included, the results will be biased. If this
term "population" was intended to refer to the
general population of the U.S.A., it should be
realized that the results from Evanston can
represent only a stratum of the country as a whole,
varying as to climate and racial composition, to
mention only two variables.
It will be recalled that the caries
rates were said to be significantly different, even
between children attending the different types of
school in Evanston; and also that the rates in that
city were considerably different from those in Oak
Park, which was at first stated to be "the ideal
control community" for Evanston (Blayney and Tucker,
1948). These differences emphasize the fact that
caution should be exercised when applying results
obtained in a test city to a wider population, of
which the test city may not be representative.
Altered methods in latest report.
In the latest report (Hill et al., 1958)
which shows the findings for the permanent teeth of
children in the control city of Oak Park, the
authors have published in the same tables as the
results of the control groups, the DMF rates, not of
the public school white children, but of the total
sample of Evanston children. This is strange in view
of their statement that "comparisons of caries rates
for the study group and the control group are based
on the caries experience of public white school
children only" (Hill et al., 1951). It would
appear that they no longer held the opinion which
they stated the previous year (Hill et al.,
1957a) that it is necessary "to make comparisons of
like groups."
As a result of this change in
procedure the differences between initial caries
rates in Evanston and Oak Park are diminished. In
children aged twelve to fourteen years, the
pre-fluoridation rates reported for the 1,226 public
school white children in Evanston were far closer to
the values found in the Oak Park children than were
either the rates of the 96 Negro, or of the 379
parochial school children (Hill et al.,
19571, 1958). However, the rates of the Negro
children were lower, and the rates of the parochial
school students were considerably higher than those
of the public school white children. By adopting the
authors' latest (1958) method, which is to add the
results of the three groups, it is found that the
pre-fluoridation rates of the twelve and
fourteen-year-old children are considerably less
divergent from those of the initial examinations in
Oak Park - and those of the thirteen white children
of those ages. Whether this situation arises with
regard to the six-, seven- and eight-year old
children cannot be determined, for no dissection
into the rates prevalent in the three school groups
has been published.
Late examination in the control
city. The United Kingdom Mission (1953) stated:
"Before fluoridation started a dental survey was
made of 4,375 children in the selected groups in
Evanston and of 2,493 children in Oak Park. Further
examinations have been carried out each year since
1947 and will continue until 1962." However, the
examinations in Oak Park were not commenced until
after the fluoridation of the Evanston water supply
on 11 February 1947, for Blayney and Tucker (1948)
stated: "The study in Oak Park was instituted on
Feb. 26, 1947". Also, at the time of the United
Kingdom Mission Report (1953), no further
examinations had been conducted in Oak Park; even in
Evanston only one age group was examined during each
year, as can be seen by inspecting the "schema for
study" published by Blayney and Tucker in 1948, and
reproduced in several subsequent reports.
Only two examinations in the
control city. This "schema" indicates
that the design of the trial provided for only two
examinations - eleven years apart - to be made in
the control city. It would appear that the authors
did not anticipate changes in the caries rates of
the control, such as were reported in Muskegon
(Arnold et al., 1953), and, as will be seen
later, in Sarnia (Brown et al., 1954b), and
in Kingston (Ast, Finn and Chase, 1951). The first
examination was made in 1947, and the second,
although not scheduled until 1958, was commmenced in
1956 when it became apparent that the water supply
of Oak Park would be fluoridated (Hill et al.,
1956). This examination was completed on 14 November
1956, soon after the fluoridation of the Oak Park
water on 1 August (Hill et al., 1958).
A ten-year delay in the
publication of data. Caries attack rates for the
six-, seven- and eight-year-old children which were
obtained in Oak Park in 1947 (Blayney and Tucker,
1948) have only recently been published by Hill
et al. (1958). This great delay is inexplicable
and is particularly unfortunate, because it is in
regard to these younger children that the major
claims are made for reduction of dental caries as a
result of fluoridation. No explanation was offered
for this delay, and the members of the United
Kingdom Mission (1953) did not comment on this
strange omission, merely saying that "The incidence
of caries among the children aged 6-8 years is
compared with the baseline data of Evanston itself
while caries experience of children aged 12-14 years
is compared with that of Oak Park."
Gross differences in initial
caries rates. The latest report (Hill
et al., 1958) reveals that in the younger
children there were gross differences between the
initial caries attack rates in Evanston and Oak
Park. The rates were: 46.85, 26.89 for age six
years; 153.49, 102.63 for age seven years; and
249.93, 222.44 for age eight years in Evanston and
Oak Park respectively.
In regard to the great difference
between the pre-fluoridation rate for the
six-year-old children in Evanston and the initial
one for children of that age in Oak Park, 46.85 and
26.89 respectively, a footnote to Table I (Hill
et al., 1958), referring to the former rate,
stated: "This figure results from the very high DMF
rate of 87.91 found in one school in 1946." However,
as the children were drawn "from 24 schools in the
study area" (Blayney and Greco, 1952), it is
probable that the rates for six-year-old children in
most schools approached the figure of 46.85, unless
the school with the high DMF rate also happened to
provide a disproportionately large number of
six-year-old children.
It should be noted that no comment
on the magnitude of this rate of 46.85 was made in
any of the four reports in which it had been shown
previously (Hill et al., 1950, 1952, 1956,
1957a); all of which were published before the rate
of 26.89 for Oak Park was released, and therefore
before a comparison with it could be made. The rate
of 46.85 was used in all those papers - and even in
their latest report (1958) - in calculating the "%
reduction", and in computing the "Probability of
difference due to chance."
Much unpublished data. The
members of the Evanston Dental Caries Study devoted
most of the years 1947 and 1956 to the collection of
data from children in Oak Park (Blayney and Tucker,
1948; Hill et al., 1958). Despite this fact,
the major part of each of the two tables shown in
the XVIII Report (Hill et al., 1958) was
devoted to a re-presentation of data obtained in
Evanston, although this report was said to have as
its purpose the comparison of the permanent teeth
dental caries experience rates in children examined
in Oak Park in 1947 and 1956. The Oak Park data were
restricted to four lines of figures showing the DMF
rates in permanent teeth. No report was made of
other findings such as those which had been shown in
reports on Evanston children. For instance, in the
XV Report (Hill et al., 1957a), no fewer than
eight tables relating to the twelve-, thirteen and
fourteen year-old children only were devoted to
these other findings. This very incomplete
presentation of the data obtained in Oak Park is
unaccountable.
 |
Figure 3.
Gross differences in
initial caries rates in Evanston and its control
city of Oak Park. The Oak Park rates remained
unpublished for over ten years.
|
Disagreements between results.
In their XVIII Report, Hill et al. (1958)
stated: "The DMF rates and percentage reduction from
year to year for the Evanston children of all age
groups shown in Tables I and 2 have been published
in previous reports. However, four of the figures
for the year 1955, shown in Table I of the 1958
Report, are different from "the rates and percentage
reduction" given, for the same year, in Table I and
the text of the XVI Report (Hill et al.,
1956). The DMF rates at age seven years were only
slightly different (40.95 and 40.92, in the XVI and
the XVIII Reports respectively), but at age eight
years the two rates were 114.04 and 120.32. It is
very improbable that these different rates are due
to typographical errors, for they were confirmed by
the "per cent reduction from 1946", which was given
in the summary and in Table I of the respective
reports as 73.32 and 73.34 for children aged seven
years, and as 54.37 and 51.85 for those that were
eight years of age. This "reduction" was shown in
the XVIII Report as 85.96 for the six-year-old
children, but in the XVI Report it was given as "80
per cent" in the findings and as "85.96 per cent" in
the summary.
Disagreement between tables.
The DMF rate in terms of tooth surfaces was given
only twice in this study (Hill et al., 1955,
Table X and 1957a, Table XII). In both papers the "DMF
rate per 100 surfaces" for children aged fourteen
years was 14.82 in 1949 and 13.94 in 1952. However,
in the former report this rate was given as 15.09 in
1946, but in the latter one, for children of the
same age in the same year, the figure shown was
15.92. As a result of this change, the "%
differences from 1946" were altered from 1.78 to
6.85 (1949) and from 7.62 to 12.44 (1952). By using
these new rates it can be said that "all 3 methods,
namely; per hundred children, per hundred teeth, and
per hundred surfaces all express approximately the
same proportion of percentage differences in rates"
(Hill et al., 1957a). This result is a good
illustration of the comment made on the method most
commonly used in these studies to express changes in
caries experience, that "relatively small variations
in the baseline values will produce substantial
alterations in the percentage reduction obtained"
(Part One, p. 137).
It may be mentioned that the "total
tooth surfaces considered" for thirteen-year-old
children in 1954 (Table X11, Hill et al.,
1957a) should be 58,325 not 58,352; and that for
fourteen-year-old children in 1949, in the column of
that table giving the "% differences from 1946", the
figures shown should be 6.91 not 6.85. In their XI
Report (Table IX) and their XV Report (Table XI),
Hill et al. (1955, 1957a) showed different
figures for children aged twelve years examined in
1952. Although both tables show the same total
number of teeth considered, in the former table
children were shown as examined, with a "DMF rate
per 100 teeth" of 25.76, and a difference from 1946
of 19.50 per cent. In the latter table, the figures
were 516, 25.60 and 20.00 per cent respectively. In
1953 Hill et al. published the figure of
19.50 per cent.
No data for deciduous teeth.
The authors have not published any data regarding
the deciduous teeth of children in the control city,
either for the first (1947) or the second (1956)
examination. The most important omission, the def
rates, could have been shown by adding only two
lines to Table I in Hill et al. (1958). This
omission is particularly unfortunate in view of the
fact that in the deciduous teeth in Evanston during
the first four years of fluoridation the def rate of
the six to eight years group was considerably higher
than the initial one (Hill et al., 1952). It
was not until nine years after the commencement of
the study that a significant reduction in this rate
was reported.
In 1950, Hill et al. stated
that the caries rate for deciduous teeth in these
children "does not indicate any trend", despite the
fact that in Table I of that report the initial rise
in this rate during the first two years of
fluoridation was shown by them to be statistically
significant (P = 0.005). Two years later these
authors altered their opinion of the significance of
this rise. In 1952 they re-published the same data
for children aged six, seven and eight years in 1946
and 1948, but computed different rates for the
combined age group six to eight years. The rise in
the def rate was then said to be not statistically
significant.
Variations in caries rates in
control. The meagre data regarding caries attack
rates in Oak Park which have been published are
included in Tables I and 2 of Hill et al.
(1958). Of the six age groups shown, between the
years 1947 and 1956 the authors reported a
significant increase in the DMF rate of children
aged seven years, and non-significant upward trends
in the rates of those aged eight and thirteen years,
and downward ones in the caries attack rates in
children aged six, twelve and fourteen years. (The
question of "significant" changes in the rates in
control cities will be considered later.) The
authors said: "The children 12, 13 and 14 years of
age, Table 2, have only minute differences between
the 1947 and 1956 rates. These are not considered to
be significant." The footnote to that table is more
definite, in each comparison stating: "Difference is
not statistically significant." Although these
differences of 61.20, 34.96 and 58.87 DMF teeth, for
children aged twelve, thirteen and fourteen years
respectively, were termed "minute differences",
those seen in the rates of the twelve and
fourteen-year-old children are approximately a third
the size of the absolute drop in the rates recorded
for the same age groups in Evanston since the
inception of fluoridation. It cannot be assumed that
the fluctuations in the rates during the intervening
period of nine years, when no examinations were
made, did not exceed the differences between the
initial and final rates. It will be recalled that
considerable variations occurred in Muskegon (see
Figs 1 and 2).
Inadequacy of the control.
Blayney and Tucker (1948) realized that "A study
of this nature must have an adequate control."
Therefore, it is strange that in the "schema" which
they published there was provision for only two
examinations, eleven years apart, to be made in the
control area. It should have been obvious that the
usefulness of data gathered in such a manner would
be, at most, very limited. The explanation given by
the authors for their failure to examine the
children in the control city "every year" (instead
of only twice) was the strange one that "It was not
necessary to do so in as much as Evanston and Oak
Park are subjected to the same advertising
campaigns, have a similar economic level,
participate in comparable educational programmes,
and so forth" (Hill et al., 1958). It is
extraordinary that the authors advanced this
explanation and that they adhered to such a plan,
despite the marked dispanity in canes rates
disclosed in the first examinations in Evanston and
Oak Park (Hill et al., 1958), which makes it
obvious that the latter city was a poor choice in
seeking an "adequate control" for the former one.
Differences between school
groups. Hill et al. (195 1) stated that
"statistically significant differences were found to
exist [in 1946] between the caries rates of Negro
and parochial school children on one hand, and
public white school children on the other hand."
However, they made a further statement that "the
caries rates of parochial school children were found
to be higher and those of Negro children lower than
those of white children in public schools" (Hill
et al., 195 1). These two statements are
inconsistent. The first appears to mean that the
comparisons between Negro children and white
children in public schools, and that the comparison
between white children attending parochial schools
and those attending public schools, were both
statistically significant in 1946.
"Nearly comparable" or
significantly different? The XV Report
Hill et al., 1957a) stated that "In 1946 and
1954 the public school white children and the Foster
School (Negro) children maintained nearly comparable
DMF rates". The actual rates" (per 100 children) in
1946 for twelve, thirteen and fourteen-year-old
white children attending public schools were 707.51,
946.17, and 1133.33; for the Negro children of the
same ages they were 658.82, 861.76 and 1035.71. (The
rates of each school group of younger children were
not published.)
It is not understood how the same
authors could on one occasion (Hill et al.,
195 1) state that there were "statistically
significant differences" between the two series of
rates, and later (Hill et al., 1957a)
describe them as "nearly comparable DMF rates" It
may be thought that the word "maintained" referred
to a comparison between the DMF rates of the white
children in public schools, and of the children in
the Negro school, between 1946 and 1954. However,
this cannot be the case, for the authors claimed for
these twelve, thirteen and fourteen-year-old
children "a reduction of approximately 21.96 per
cent in dental caries-experience rates of the
permanent teeth" (Hill et al., 1957a). (In
this study, percentages were frequently shown
"approximately" to two decimal places.) Table IV of
that paper shows that both the Negro and the public
school children participated in the reductions
reported.
Decline in eruption rate.
An
observation of considerable interest is obtainable
from Tables V and VI of the X Report (Hill, et al.,
1952). The former table shows the rates per 100 six,
seven and eight-year-old children that had occlusal
surface pit and fissure caries or fillings in their
first permanent molars; the latter one, the number
of these teeth which were free from those defects.
The mean number of erupted first permanent molars
per 100 children may be obtained, in each age group,
by adding these two rates to that showing the
extracted and congenitally missing permanent molars.
It is probable that the number of congenitally
missing teeth was negligible and that the number of
permanent molars which had been extracted in these
young children was small, particularly in the six
years age group (five and a half to six and a half
years). Therefore, it would be expected that, in
each age group, the mean number of erupted molars
per 100 children would be similar at the time of
each examination. This was the case in children aged
eight years; the figures for the examinations made
in 1946 (pre-fluolidation), 1948, 1950 and 1951
being (to the nearest whole number) 387, 387, 384
and 386 respectively. At age seven years the numbers
erupted were 330, 336, 320 and 315; but in the
six-year-old children, the number of erupted molars
showed a marked and progressive decline 189, 156,
140 and 132 during the period covered by those four
examinations.
The question naturally arises
whether the eruption rate of these teeth had
decreased; a possibility of extreme importance in
interpreting the results of a fluoridation trial.
However, further consideration of this matter is
prevented by the authors' failure to publish this
type of data when they reported the results of the
two later examinations (conducted in 1953 and 1955)
which were made of children of these ages; and the
"schema for study" indicates that children aged six
to eight years will not be examined again until
1960.
This failure to publish this type of
data for the 1953 and 1955 examinations is
extraordinary, especially in view of the fact that
the authors continued to show similar data for the
permanent molars of the older age group (Hill et
al., 1955, 1957a); the latter report, the only
one showing results for both age groups, gave the
prevalence of occlusal pit and fissure caries and
fillings in the molars of the older, but not of the
younger age group.
In considering the eruption of
teeth, the odd method of assessment used in this
study must be taken into account. Hill et al.
(1955) said: "Only teeth which were 50 per cent or
more erupted were considered. A carious or filled
tooth was, of course, considered regardless of its
stage of eruption."
 |
Figure 4.
Suggestion of a
progressive decline in the number of erupted
first permanent molar teeth in six-year-old
children in Evanston. The results obtained in
the examinations conducted in 1953 and 1955 were
omitted from the published reports.
|
Strange superiority of artificial
fluoridation. The authors of this study compared
the Evanston DMF rates per child with those of
children in Aurora, Illinois (Dean et al.,
1950) in the expectation that after sufficient time
had elapsed for all the erupted teeth to have been
formed since fluoridation commenced "the Evanston
rate will closely approach the Aurora rate" (Hill
et al., 1957a). It is surprising that this
parity between the rates of Aurora and Evanston was
expected, because in the Aurora survey only clinical
methods of examination were used, but in the
Evanston examinations X-ray surveys were used
routinely. Hill et al. (1951) stated: "We
find our baseline figures for caries experience in
Evanston and Oak Park approximately 32 per cent
higher than those of Dean and his co-workers for
Evanston and Oak Park in 1941. We assume this may be
explained partially by differences in the techniques
of examination, particularly in the use of X-ray in
the current investigation." The United Kingdom
Mission (1953) stated that in this study "the
minutest radiolucency was taken as indicating
caries."
In view of these findings, it is
even more strange that Hill et al. (1957a)
were able to report: "The Evanston 6 and 7-year-olds
of 1953 have a lower dental caries experience rate
after 71 to 82 months of fluoridation than the
Aurora 6 and 7-year-olds of 1945-1946 with lifetime
exposure to water naturally fluoridated to 1.2 ppm."
That this difference was not only slightly below the
1945-1946 Aurora rate for children of the same age"
(Hill et al., 1957a) can be seen by comparing
the actual rates reported. In Evanston and Aurora
respectively, the rates were 14.73, 28.0 at age six
years and 53.35, 70.5 at age seven years (Hill et
al., 1957a; Arnold et al., 1953). It
should be noted that in Evanston two years
previously (195 1), after a shorter period of
fluoridation, the rate for the six-year-old children
was even lower, 12.36 (Hill et al., 1952) and
was less than half the Aurora rate; in 1955 (Hill
et al., t956) it had become 6.58, less than a
quarter of the Aurora rate. Blayney and Greco (1952)
found that in children in the Evanston study, with
regard to proximal caries "the 6-year-olds have the
highest percentage (83.90) disclosed by X-ray
findings only. In the 7-year-old group 79.04 per
cent of proximal lesions were demonstrated by X-ray
findings only". Therefore, if clinical methods of
examination only had been used in Evanston, as was
the case in Aurora, what may be thought to be a
strange superiority of artificially over naturally
fluoridated water as a means of reducing dental
caries attack rates would have appeared to have been
even more marked.
"Weighting" of results.
The
method of combining the results of the six, seven
and eight-year-old children into one category
introduces an important source of error when
comparisons are made between the results obtained in
the control city and in the test one, or between
those found on different occasions in Evanston.
Owing to the great differences in caries attack
rates which are observed between children of these
ages (the baseline DMF rates for these three ages in
Evanston were 46.85, 153.49, and 249.93, according
to Hill et al., 1950), the results may
inadvertently be "weighted" by including a
preponderance of young or of old children in the age
group six to eight years. If this occurs, the
average value will be lower or higher than it would
have been if the three ages had been equally
represented in the sample. In comparing the results
of the control and the test cities, "weighting" of
this nature could make it appear that large
differences were present, when, in fact, they were
either slight or absent, or the presence of actual
differences could be hidden.
An example of "weighting".
The results of the pre-fluoridation, and of the
first post-fluoridation survey at Evanston (Hill
et al., 1950), clearly demonstrate the process
of "weighting" and show that its occurrence is not
merely a theoretical possibility. On these two
occasions, the number of children in each of the age
groups six, seven and eight years that were examined
in 1946 was 461, 759 and 771 respectively; the
corresponding numbers seen in 1948 were 756, 838 and
440. On both occasions the results of the three ages
were combined, and a caries rate was computed for
the age range six to eight years.
Significant tests and "weighting
". Despite the rather obvious "weighting" in the
examples which have just been cited, tests were
applied to determine the significance of the
difference between the caries attack rates found
during the two examinations in the combined age
range six to eight years. In regard to the permanent
teeth, it was stated that "The probability of this
difference being due to chance is 0,0000" (Hill
et al., 1950). Curiously, in those teeth a
decrease in the caries rate was reported,
contrasting with the statement of a significant rise
in the rate of the deciduous ones.
Random variation ignored.
Hill et al. (1950) stated: "It is to be
expected that the rate of caries in all teeth varies
from year to year due to chance. A significant
reduction of caries prevalence can therefore be
assumed to exist only when the statistical analysis
of the data provides almost absolute certainty that
the observed differences are not due to chance."
However, in a subsequent paper (Hill et al.,
1956) these authors ignored the variations in the
intervening years, even when these were as marked as
those in Table 5 of that report, and stated:
"Difference between 1946 and 1955 rates is
statistically significant."
Original results altered. In
the X Report (Hill et al., 1952), and in all
the later ones, alterations were made to the rates
shown for the years 1946 and 1948 in children of the
combined age group six to eight years, which were
published by Hill et al. in 1950 (Tables I to
VI). The original rates were replaced by values
which are the means of the mean rates for the
children of each of the three ages six, seven and
eight years (Hill et al., 1952, Tables 11 to
IX).
System of computation changed.
The change in the system of computation was
explained by Hill et al. (1952) in these
terms: "The group averages, shown in previous
reports, represents weighted averages of the
individual mean caries rates. Inasmuch as the
composition of the groups of children with respect
to the number of 6, 7 and 8-year-olds varies from
year to year, it was felt that unweighted group
averages form a more sound basis for comparison of
group caries rates between years."
The new method of computation.
In 1952 Hill et al. stated that "The new
averages were obtained by taking a simple
arithmetical mean of the individual caries rates of
the 6, 7 and 8-year-old children." This description
of the new method is apt to cause some confusion,
for it is considered to describe accurately the old
method. It was used by these authors in 1950, and
then abandoned by them in favour of the new one. The
results for 1950 and 1951 in Table IV of Hill et
al. (1952), and those for 1953 in Table I of
Hill et al. (I 957a), and for 1955 in Table I
of Hill et al. (1956) make it clear that in
this new method of calculation, the rate per 100
children aged six to eight for each examination was
obtained by taking a simple arithmetical mean of the
mean rate for each of the three ages six, seven and
eight years.
Errors in amended rates.
The
amended rates published by the authors (Hill et
al., 1952) for the age group six to eight years
need further amendment, and the difference between
them is even less than that stated. The mean of the
three values shown for 1948 in their Table IV,
23.54, 103.58 and 194.09, is found to be 107.07, not
92.07 as stated; also, the mean of the three values
for 1946 - 46.85, 153.49 and 249.93 is 150,09 not
149.76. These errors were repeated in the XV and the
XVI Reports (Hill et al., 1957a, 1956).
The figure 149.76 was shown also in
the XIV Report (Hill et al., 1954). In that
report the rate for age six to eight years was said
to be "65.82 in 1953." However, in Table I of the
XVI Report (Hill et al., 1956) the rate for
1953 for age six to eight years was given as 63.52.
The latter figure is the mean of the three mean
rates shown for the six, the seven and the eight
year-old children.
The XIV Report (Hill et al.,
1954) stated: "The combined 6 to 8-year-old children
had a permanent tooth DMF rate of 149.76 per 100
children in 1946 and 65.82 in 1953. This is a
difference of 60.38 per cent." In fact, by using
their standard method of calculation, the
"difference" is 56.05 per cent.
A confusing calculation.
The
situation is made even more confusing by the figures
shown in Table 6 of the XVI Report (Hill et al.,
1956). If the method commonly used in these trials
is employed, when the difference between the DMF
rates for 1946 and 1955, which is 95.90 (the rates
being 149.76 and 53.86), is expressed as a
percentage of 149.76, the "per cent difference" is
64.04, not 64.11 as shown. However, if the correct
figure of 150.09 (which does not appear to have been
mentioned in these reports) is substituted for
149.76, the "per cent difference" becomes 64. 11 as
shown in their Table 6.
Was sampling used? The six,
seven, eight and twelve, thirteen, fourteen year age
groups were chosen for study (Blayney and Tucker,
1948), but it was not stated whether all children of
these ages (the ages were taken to the nearest
birthday) were examined, or whether a sampling
method was used. The VII Report of Hill et al.
(1951) said that "0. 1 per cent of the children in
the control area were Negro." However, in the XV
Report (Hill et al., 1957a) it was stated
that "the control area (Oak Park) examinations
included only public school white children". It is
not clear whether the Negro children in that city
were excluded from the examination by design, or by
the chance of a sampling method. The former
alternative is suggested by the statement of Hill
et al. (1955) that "In the control village of
Oak Park, only public school children were studied".
Were children "continuous
residents"? It is not clear whether all the
children included in the early reports (Blayney and
Tucker, 1948; Hill et al., 1950, 1951, 1952,
1953, 1954) were "continuous residents". Although
the questionnaires recorded the residence record of
each child, it was not until the X1 Report (Hill
et al., 1955) that the statement was made that
"The data given in this report are limited to those
children whose entire lives have been on Lake
Michigan water." The United Kingdom Mission (1953)
stated that "The study includes only white children
attending public schools in the city who have lived
in the area continuously from birth." However, as
the first part of that statement presents an
incomplete description of the authors" method, doubt
is raised as to the accuracy of the statement made
in regard to continuous residence.
Disturbing disagreements.
In the following paragraphs are cited some
disturbing disagreements between the statements made
regarding the number of children examined. No
suggestion has been found that more than one series
of examinations was conducted in Evanston in each
year from 1946 onwards, and in Oak Park in 1947 and
1956. Therefore, although the situation is uncertain
regarding sampling and continuous residence. it
would be expected that all the reports would agree
with regard to the number of subjects of each age
that were examined in each individual year. The
exception is the XVII Report (Hill et al.,
1957b), which compares the caries rates of white
with those of Negro children; for it was stated that
"in this report no attempt has been made to limit
the examinations to continuous resident children."
Therefore, it would be expected that the sample
sizes shown in this report may be larger than those
published in other reports.
Gross discrepancies between
sample sizes. The numbers of children of each of
the ages twelve, thirteen and fourteen years that
were examined in 1946, 1949, 1952 and 1954 were
given in the second column of Tables XI and XII of
the XV Report (Hill et al., 1957a), the same
figures appearing in both tables. It is to be noted
that in eleven out of the twelve cases, the sample
sizes given there are different from those shown in
Tables 111, V, VI, VII, VIII, IX and X of the same
report. In six cases the samples were larger in
Tables XI and XII than in the other tables
mentioned, and in five cases they were smaller. The
largest discrepancy was between the number of
children aged twelve years that were examined in
1949. Tables XI and XII showed this figure as 627,
and the other tables gave 522 as the sample size.
Similar discrepancies (for 1946, 1949 and 1952) are
present between the sample sizes shown in Tables IX
and X of the 1955 paper of these authors, and Tables
1, 111, IV, V, VI, VII and VIII of that report. The
authors (Hill et al., 1957a) stated: "The
number of teeth and surfaces associated with the DMF
rates from 1946 through 1954 are shown in Tables XI
and XII." In other tables mentioned in that report
the "Rate per hundred children" was employed, but
there appears to be no reason why the number of
children examined should not be the same for both of
these comparisons. No explanation for the different
sample sizes was advanced by the authors.
Disparities in Negro sample
sizes. Marked disparities are seen between the
sample sizes shown for Negro children, for, judging
from Table 10 of the XVII Report (Hill et al.,
1957b), data >from only about half of the Negro
children aged twelve to fourteen years who were
examined in 1946, and of less than a third of those
examined in 1954, were included in the XV Report
(Hill et al., 1957a). The number studied is
given in Table IV of the latter paper as 96 in 1946,
and as 79 in 1954. However, the XVII Report (Hill
et al., 1957b, Table 10), shows that 188 Negro
children of those ages were examined in 1946, and
250 in 1954.
The XI Report (Hill et al.,
1955) also shows that 96 Negro children were
examined in 1946. The VII and XVIII Reports (Hill
et al., 1951, 1958), although they do not state
the number of Negro children, indicate the same
sample size, 1,701 children, as the XI and XV
Reports (Hill et al-, 1955, 1957a). In the
last mentioned report, referring to the 1954
results, the authors said: "It is admitted that the
Foster (Negro) school sample (79) was limited." Why,
then, were so few of the 250 Negro children aged
twelve to fourteen years that were examined in that
year included in the report? Were less than a third
of these children continuous residents?
The situation with regard to
children aged six to eight years cannot be
investigated, because the XVII Report is the only
one in which the data of the younger age group of
Negro children are shown separately from those of
the white children.
Further unexplained differences.
The position revealed in the last paragraph is
further confused by the presence of large variations
between the number of white children, aged twelve to
fourteen years, whose data were shown in earlier
reports, and the number given in Report XVIL In the
former reports (Hill et al., 1955, Table 11;
1957a, Table IV) the number of these children
examined in 1946 (public plus parochial schools) is
stated to be 1,605, but, according to the XVII
Report (Hill et al., 1957b, Table 10) the
number seen in that year was 1,368. In 1954 the
examinations of white children totalled 1,247 (Hill
et al., 1957a, Table IV), but the figure of
1,905 is shown in the XVII Report (Hill et al.,
1957b).
In the younger children, as no
dissection of the data into school groups has been
published, only the total number inspected can be
considered. The XVII Report (Table 10) states that
1,754 children were examined in 1946 and 2,952 in
1955; but Table I of the XVI Report (Hill et al.,
1956) shows 1,991 and 1,376 examinations
respectively. The two statements of sample sizes
(XVII Report figures minus the XVI Report ones)
therefore differ by -237 and + 1,576 children.
It is possible that the larger
sample sizes shown in the XVII Report for the
examinations in 1954 and 1955 were due, despite the
sizes of the increases (171 Negro and 658 white
children aged twelve to fourteen years, and 1,576
children aged six to eight years), to the inclusion
of all subjects, and not only those who were
"continuous resident children". If, at the time of
commencement of the study in 1946, children who had
not lived in Evanston "continuously" since birth
were excluded from the main study, an explanation
can be found for the larger number of Negro children
included for that year in the XVII Report. However,
it is strange that that report, which included
children who were not "continuous residents" (Hill
et al., 1957b), in 1946 should be based on
237 fewer white children aged twelve to fourteen
years and on 237 fewer white plus Negro children
aged six to eight years than were included for that
year in the other reports mentioned.
Incompatible statements.
The authors made incompatible statements
regarding the total number of children examined
during the initial examinations in Evanston and Oak
Park. In Report II (Blayney and Tucker, 1948) it was
stated that the "baseline observations were made on
4,375 North Shore" (study area) "children and 2,493
Oak Park children." These figures were repeated in
1950 by Hill et al. However, Tables I to VI
of the latter paper show that 1,991 children aged
six to eight years were examined in Evanston in
1946; Tables 1, 11 and III of Hill et al.
(195 1) indicate that 1,701 children aged twelve to
fourteen years were examined in that year, that is,
a total of 3,692 children. One or both of these
figures (1,991 and 1,701) were repeated by the
authors (or may be obtained by adding figures for
individual yearly age groups) in 1952, 1955, 1956,
1957a and 1958.
 |
|
Figure 5.
Incompatible
statements regarding the number of children
inspected during the initial examinations in
Evanston and its control city of Oak Park.
Evanston statement A is from Blayney and Tucker
(1948) and Hill et al. (1950). Statement
B is from Hill et al. (1950, 1951, 1952,
1955, 1956, 1957a and 1958). Statement C is from
Hill et al. (1957b). Oak Park statement D
is from Blayney and Tucker (1948) and Hill et
al. (1950), and statement E from Hill et
al. (1958). See p. 211.
|
The third total sample size for
Evanston in 1946 is shown in the XVII Report (Hill
et al., 1957b). By totalling the figures in
Table 10, it appears that 1,754 children aged six to
eight years, and 1,556 aged twelve to fourteen
years, were examined, a total of 3,310 subjects.
From Tables I and 2 of Hill et al. (1958) it
is deduced that a total of >2,051 children were
examined in Oak Park in 1947 (see figure 5, p. 167).
Therefore, three very different
sample sizes were given for the 1946 examination in
Evanston: 4,375, 3,692 and 3,310; and two total
sample sizes of 2,493 and 2,051 subjects examined in
Oak Park in 1947. The smallest sample size for
Evanston (3,3 10) was given in the XVII Report,
despite the statement of the authors (Hill et al.,
1957b) that "in this report no attempt has been made
to limit the examinations to continuous resident
children."
Remarkable changes in assessment
of statistical significance. In the
footnote to Table II in Hill et al. (1952) it
was stated: "It should be noted that the caries
rates per 100 children for the 6-8 year olds as a
group shown in this report, vary slightly from those
shown in previous reports." Although these were said
to be slight variations, the remarkable fact emerges
that, although based on the same data, the
difference between the 1946 and the 1948 caries
attack rates for the deciduous teeth of children of
that age range, which was said to be statistically
significant (the probability being given as 0.005)
in the 1950 Report, was stated by the same authors,
in 1952, to be "not statistically significant."
On reading the X Report (Hill et
al., 1952), it appears that even more
extraordinary changes of opinion with regard to the
significance of results based on the same data occur
in five comparisons between the rates of permanent
teeth; significant differences (probability
"0.0000") being altered to "not statistically
significant." However, a correction (J. dent. Res.,
31, 597) stated that the footnotes to Tables IV, V,
VI, VII and VIII were incorrect, and that the
statements: "Differences are not statistically
significant" should have read "Differences are
statistically significant". It is considered likely
that the correction is incomplete, and that in the
footnote to Table IX of that paper, the word "not"
should be deleted. If this alteration is not made,
that footnote indicates that the difference between
the rates for 1946 and 1948 is "not statistically
significant", although two years earlier, the
difference computed from the same data was stated in
the footnote to Table VI of Hill et al.
(1950) to be significant (probability "0.0000") .
At first sight, the employment of
statistical terminology in the presentation of this
study engenders confidence in the results reported,
but the few examples which have been cited clearly
indicate their unreliability.
No comments:
Post a Comment