Errors and
Omissions in Experimental Trials - 2b
CRITICISMS
AND COMMENTS [cont.]
DR R. M.
GRAINGER
The
third review of this book in the Australian Dental Journal, by DR R. M.
GRAINGER, Division of Dental Research, Faculty of Dentistry University of
Toronto, was as follows:
Those
whose work has been so unfairly criticized might well ask P. R. N. Sutton
if he feels his own work is proof that the unimpeachable study can be
done(26), or if he would welcome similar scrutiny of his publications.
While
we do not claim to be able to answer every question to P R. N. Sutton's
satisfaction (or even our own), in order to help set the record as
straight as possible(27) the following are specific comments on points
raised by P. R. N. Sutton in his discussion of the Brantford Study. No
attempt is made to rationalize why specific workers directed or
restricted their research efforts or discussion in any areas other than
to comment that they no doubt accomplished as much as they could under
the circumstances in which they had to work.
Item
1: Reference to Hutton et al.
(1951). The numbers of children of the same age examined in the years
1944 and 1945 were very similar with the exception of the nine-year age
group. From the unpublished data released at annual meetings in Brantford
the number of children examined in 1944 was 239, and in 1945, 319; making
a total of 558 (not 608). The data in Table I (Hutton et al.) were apparently combined
by pooling the two years' results not by averaging the averages. However,
the point is rather academic(28).
Item
2 : Reference to Ontario Health Department report. The date of
water-fluoridation in Tables I and 11 was given as 1946 through a
typographical error but was twice correctly stated to be June, 1945, in
the text referring to the Table. The small error in percentage reduction
for seven-year-olds was also conceded. These points do not seriously
underline the usefulness of the work(29).
Item
3: Re late commencement of National Health and Welfare study and
detection of caries protection for young individuals born prior to
commencement of fluoridation. Despite the fact that the Department of
National Health and Welfare began its control study nearly three years
after fluoridation began, much worthwhile information was obtained and
the effect of late commencement, if any, was to result in underestimation
of the fluoride protection(30).
Item
4: Reason for selection of control cities. It seems clear that Brown
(1951) gave adequate reasons for selecting Sarnia and Brantford(31).
Item
5: Re superior dental care in Brantford. The difference in level of
dental care between the cities is factual as recorded by Brown (1952).
This variation of numbers of teeth classified as F. rather than D. or M.
does not fundamentally influence the DMF rate(32).
Item
6: Comparability of rates. As stated under item 5, the dental condition
of the children in Sarnia and Brantford differed in 1948 because a lower
level of dental treatment in Sarnia resulted in higher tooth mortality.
The tooth mortality rates thus differed but it does not follow that the
DMF rates differed(33). Brown's statement (1951) "by 1948 the
Brantford data were not greatly different from those in Sarnia" is
obviously referring to DMF rates and hence quite valid(34). The
differences in oral hygiene are also only remotely related to the DMF
rates under discussion(35).
Item
7: Concentration of fluorides. The fluoride content of the Brantford
water supply was raised to approximately 1 ppm in June, 1945, and raised
to 1.2 in February, 1949(36) The Stratford water fluoride content is
believed to have been in the order of 1.3 to 1.6 ppm since 1917 when the
wells were drilled. Naturally no analysis for fluoride was available
prior to the beginning of the interest in fluoridation and early
techniques for analysis were not as reliable as present methods(37). These
facts have been recorded in the writings of the primary workers (Hutton et al., 1951; and Brown et al., 1951, 1952, 1953, 1954,
1956) and the differences in amounts from other writers night seem less
"strange" if they were merely acknowledged to be minor
misquotations(38).
Item
8: Re statement by Brown et al.
(1956). The statement is substantially correct with the exception that
Brown's observational period did not begin until 1948 hence is less than
10 years(39). The decrease in mean df rates for the 9-11 years group in
Sarnia between years 1948 and 1954 (Brown, 1956), did not continue into
1955(40). There was a highly significant decrease over the period 1948 to
1955 (2.37 to 1.93) in Brantford and no significant decrease in Sarnia
(2.50 to 2.31)(41). In the same periods the mean df rates for this age
[in] Stratford remained nearly equal (1.66 and 1.65) and increased for
other ages (42).
Item
9: Re Table II: Ontario Department of Health Report. The printing of
dashes rather than percentages for the control cities was to avoid
confusing the table With "negative reductions" and in the case
of the 9 to 11 df figure to avoid emphasising what was considered to be a
spurious decrease(43). This judgment was borne out by the 1955
figures(44).
Item
10: Differences in reported rates between examiners. Different examiners
give characteristically higher or lower rates upon examining the same
individuals due to differences in skill, training the physical condition.
Thus the differences quoted are no reflection on the design of the
experiment or the care taken in the work. The strength of the double
examinations comes through corroboration of caries trends in Brantford
over the years and not through interchangeability of data(45).
Item
11: Significant fluctuations in controls. The important point is that for
the controls the inter-year changes were upward trends or mere
fluctuations (even though in some cases calculated to be beyond change),
whereas in Brantford the change took the form of a highly significant
continual downward trend(46).
Item
12: Larger percentage changes in control. There is no definite
explanation as to why rates increased in Stratford and also in Sarnia
over the ten years, but this may be a reflection of a general post-war
increase in dental caries which has been seen in other areas. However, it
is significant that in the various fluoridation experiments, e.g. in
Brantford, Newburgh, Grand Rapids, etc., the shift has always been
significantly downward in the fluoridated cities whereas the control city
rates have remained about the same or in the case of Stratford,
increased(47).
Item
13: Smaller percentage decrease after longer fluoridation. As pointed out
by Sutton himself on page 168 (middle paragraph) the fluoride protection
for permanent teeth of the children aged six to seven seemed to occur
within two or three years after fluoridation began. Thereafter the yearly
DMF rates were subject to random fluctuation and the differences in
percentage decrease of 69 per cent and 51 per cent are most likely a
reflection of this inter-year variation(48).
Item
14: The quotation from the Ontario Government Report is taken out of
context from a series of summary statements. The previous statement was
to the effect that in Brantford there had been a significant decrease of
about 60 per cent in DMF rates. In the statement following, as picked out
by Sutton, it was stated that "no change" occurred in Stratford
and Sarnia(49). It should be clear enough from the context that the words
"no similar downward change" were inferred(50).
Item
15: Possible weighting effect. The critic is referring to a possible
shift in age distribution within the group, e.g. a possible sampling
shift within the six to eight-year age group so that certain years had
disproportionately higher or lower numbers of eight-year-olds and hence
higher or lower average caries scores. This is rather remote in that
selection methods used by Dr Brown were the same each year, moreover very
large shifts in age distribution would be needed to produce the significant
differences to which P N. R. Sutton refers(51).
Commentary on the Review by Dr
R. M. Grainger
(26)
Dr R. M. Grainger raises the question as to whether "the
unimpeachable study can be done". This is, of course, unlikely. It
is precisely for this reason that all papers (and these include my own)
which set out to present new knowledge should be examined, in order to
reduce the chance that findings which are not soundly based will be
accepted at their face value. This is particularly necessary in those
studies which may involve the health of the public.
(27)
The result of Dr Grainger's attempt "to help set the record as
straight as possible" will be judged after considering his other
remarks.
(28)
Item 1, Reference to Hutton et al.
(1951); The numbers of children of the same age examined in the years
1944 and 1945 were very similar with the exception of the nine-year age
group. From the unpublished data released at annual meetings in Brantford
the number of children examined in 1944 was 239, and in 1945, 319; making
a total of 558 (not 608). The data in Table I (Hutton et al.) were apparently combined
by pooling the two years' results not by averaging the averages. However,
the point is rather academic.
Comment.
The phrase "making total of 558 (not 608)" suggests that the
figure 608 was an error in this monograph. This is not the case, in fact
this figure was not mentioned. It was published by the authors of this
study, Hutton et al., in 1951
(Table 1, column 2). Dr Grainger, therefore, is suggesting that the total
558 children (derived from the unpublished figures of 239 and 319) is
correct, and that the figure of 608 children examined, published by the
authors of the study, is incorrect. It should be noted that, five years
after this figure of 608 was first published, in Table II, column 2, of
their final report Hutton et al.
(1956) again published their figure of 608. In both the tables in which
it appears it has been used in computing the def and the DMF rates. Also,
if one accepted Dr Grainger's figure of 558 as the correct number of
nine-year-old children examined in these two years, the impossible
situation would also have to be accepted in which the number of these
children with decayed, missing or filled teeth, which Hutton et al. (1951) gave as 595, would
exceed the number of children examined.
(29)
Item 2: Reference to Ontario Health Department report. The date of
water-fluoridation in Tables I and II was given as 1946 through a
typographical error but was twice correctly stated to be June, 1945, in
the text referring to the Table. The small error in percentage reduction
for seven-year-olds was also conceded. These points do not seriously
undermine the usefulness of the work.
Comment.
The "small" error in percentage reduction, which, Dr Grainger
said "was also conceded", was the showing of 51 per cent
instead of 66 per cent (p. 167). Dr Grainger does not mention here the
substitution of dashes for figures in the two cases of reduction in the
caries rate in the control cities (pp. 4, 37, 44). Several types of
errors are present in Tables I and 11: (a) two omissions, which Dr
Grainger implied-Item 9 of this review(43)-were made deliberately; (b)
two typographical errors; (c) two arithmetical errors (Table I, age 7,
"% Reduction Since 1944-45" in the caries attack rates should
be 66, not 51, and in Table II, age 10, the "% Reduction Since
1944" in the caries attack rates should be 18, not 61); and if, as
appears likely, the figures given by Dr Grainger in Item 1, of this
review(28), are incorrect and were used, (d) four incorrect mean figures.
Dr
Grainger contends that the points which he mentioned "do not
seriously undermine the usefulness of the work"; but the occurrence,
on one page alone, of all the errors and omissions which have just been
mentioned certainly undermines confidence in the care taken in the
preparation of this official report by the anonymous
"statisticians" of the Division of Medical Statistics, Ontario
Department of Health.
(30)
Item 3: Re late commencement of National Health and Welfare Study and
detection of caries protection for young individuals born prior to
commencement of fluoridation. Despite the fact that the Department of
National Health and Welfare began its control study nearly three years
after fluoridation began, much worthwhile information was obtained and
the effect of late commencement, if any, was to result in underestimation
of the fluoride protection.
Comment.
Dr Grainger does not state the nature of this "worthwhile
information" but, whatever it was, it could not compensate for the
lack of a pre-fluoridation caries assessment in this study. Its late
commencement could be justified only if it was known that the caries
rates in Brantford had not been affected by the ingestion of fluorides
prior to the baseline examination (p. 168). However, the results from the
City Health Department study, if taken at their face value, indicated
that there had been marked and erratic changes: at first a considerable
rise in the DMF rates after about one year of fluoridation, followed by a
marked fall during the second year. It is surprising, therefore, that,
out of all the cities in Canada, Brantford was chosen as the location of
two long-term studies, for it should have been obvious that the value of
the second study would be severely limited by the fact that the very
important data showing the pre-fluoridation caries rates could never be
obtained.
(31)
Item 4: Reason for selection of control cities. It seems clear that Brown
(1951) gave adequate reasons for selecting Sarnia and Brantford.
Comment.
As Dr Grainger notes, this paragraph refers to the selection of the
control cities, which were Sarnia and Stratford-not "Sarnia and
Brantford". The sole reference to the selection of control cities
which Brown (1951) gave is as follows: "The Ontario Dental Division,
under Dr Frank Kohli, volunteered assistance, as did Dr G. L. Anderson,
Medical Officer of Health for Sarnia, and Dr H. B. Kenner, Medical
Officer of Health for Stratford, and both these cities entered the study
as controls. (The water of Sarnia is fluorine-free, and that of Stratford
contains 1.3 ppm. of fluorine from a natural source.)" Dr Grainger
considers that "Brown (1951) gave adequate reasons" for the
selection of the control cities-but few would agree with him.
(32)
Item 5: Re superior dental care in Brantford. The difference in level of
dental care between the cities is factual as recorded by Brown (1952).
This variation of numbers of teeth classified as F. rather than D. or M.
does not fundamentally influence the DMF rate.
Comment.
Increased dental care usually includes some prophylactic treatments and,
as noted by Doctors Blayney and Hill (18). even regular examinations may
be accompanied by "more emphasis on the teaching of oral health."
This statement by Dr Grainger implies that he considers that such
increased dental care has no influence on the total DMF rate.
(33)
Item 6: Comparability of rates. As stated under Item 5, the dental
condition of the children in Sarnia and Brantford differed in 1948
because a lower level of dental treatment in Sarnia resulted in higher
tooth mortality. The tooth mortality rates thus differed but it does not
follow that the DMF rates differed.
Comment.
It does not state "that the DMF rates differed; it points out, as
its title states, the "Doubtful comparability of rates" owing
to the delay in setting up this study. Dr Grainger's comments suggest
either that he has not understood the meaning of the first sentence of
the paragraph, or that he is seeking to distract attention from the
presence of this important deficiency in the study-its late commencement.
(34)
Brown's statement (1951) "by 1948 the Brantford data were not
greatly different from those in Sarnia" is obviously referring to
DMF rates and hence quite valid.
Comment.
This quotation does not appear in Brown (1951) but a similar statement
was made by Brown et al. in
1953 and 1954 (b) and is given on page 169. The fact that it was
"obviously referring to DMF rates" was not questioned. The
implications of this remark were discussed.
(35)
The differences in oral hygiene are only remotely related to the DMF
rates under discussion.
Comment.
Brown et al. (1954b) said that
"marked differences in oral hygiene as between the test and control
groups might conceivably affect the findings". Such "marked
differences" were reported-but were disregarded.
(36)
Item 7: Concentration of fluorides. The fluoride content of the Brantford
water supply was raised to approximately 1 ppm in June, 1945, and raised
to 1.2 in February, 1949
Comment.
This statement of Dr Grainger is welcome because it provides the answer
to the question: Which of the statements regarding the concentrations of
fluorides in the Brantford water, which were reported on page forty-two,
are accurate and which ones are not?
(37)
The Stratford water fluoride content is believed to have been in the
order of 1.3 to 1.6 ppm since 1917 when the wells were drilled. Naturally
no analysis for fluoride was available prior to the beginning of the
interest in fluoridation and early techniques for analysis were not as
reliable as present methods.
Comment.
This statement is most revealing for it indicates that the wells at
Stratford have been analyzed to determine their fluoride content only
since "the beginning of the interest in fluoridation". If this
is the case, the statements of Brown et
al, (1953, 1956), concerning the "continuous" use of water
containing fluorides in concentrations of 1.3 ppm or 1.6 ppm since 1917
are not founded on data and are, therefore, merely different guesses.
(38)
These facts have been recorded in the writings of the primary workers
(Hutton et al., 1951; and Brown
et al., 1951, 1952, 1953, 1954,
1956) and the differences in amounts from other writers might seem less
"strange" if they were merely acknowledged to be minor
misquotations.
Comment.
It was pointed out that the "facts" regarding fluoride
concentrations were stated differently in these papers. In regard to the
concentration in the Stratford supply, a comparison of the statements
made by Brown et al., in 1953
and 1956 suggests that the concentration of fluorides in this supply may
have increased from 1.3 to 1.6 in this three year period. The important
admission that the fluoride concentration in Stratford was obtained only
relatively recently, is not contained in any of the six "writings of
the primary workers" mentioned by Dr Grainger. Therefore his
statement is not correct.
As
Dr Grainger suggests, it is not unlikely that the statements regarding
fluoride concentration of the "other writers", the New Zealand
Commission of Inquiry (1957) and the Ontario Department of Health (1956),
were "minor misquotations".
(39)
Item 8 Re statement by Brown et al
(1956). The statement is substantially correct with the exception that
Brown's observational period did not begin until 1948, hence is less than
10 years.
Comment.
Dr Grainger suggests that the phrase "more than ten years" is
incorrect. It was inserted into the quotation of a statement by Brown et al. (1956), but enclosed in
square brackets to indicate that it was not a part of the quotation.
However, in the sentence which immediately precedes that quotation Brown et al. (1956) said:
"Brantford has had more than 10 years of experience with 1 part per
million of fluoride in its water supply. During that time... "It is
clear that they were not referring to "Brown's observational
period" of about seven years, but to the period of fluoridation in
Brantford which commenced in June 1945 (Hutton et al., 1951; p. 173) and was, therefore, "more than 10
years".
(40)
The decrease in mean df rates for the 9-11 years group in Sarnia between
the years 1948 and 1954 (Brown, 1954), did not continue into 1955.
Comment. The
1955 rate of 2.31 df was still below the 1948 and the 1951 figures of
2.50 and 2.41 respectively.
(41)
There was a highly significant decrease over the period 1948 to 1955
(2.37 to 1.93) in Brantford and no significant decrease in Sarnia (2.50
to 2.31).
Comment. The
decrease mentioned by Dr Grainger (2.37 to 1.93) was reported in
Brantford between 1948 and 1954 (Brown et al., 1954b) not "over the period 1948 to 1955".
In 1955 this rate rose to 1.99 (Brown et
al., 1956), and the difference between 1948 and 1955 was no longer
said to be "highly significant" (Brown, 1955)
The
rates quoted by Dr Grainger for Brantford are for the years 1948 and 1954
(see 63). It should be noted that it was between these two years that the
maximum "decrease" was reported in the rates in that test city
(2.37 to 1.93). Furthermore, in mentioning Sarnia, instead of giving the
figures for the same period (1948-54), 2.50 to 2.11, he cited the figures
2.50 to 2.31, which cover a different period (1948-55) and do not reveal
(Brown, 1955) the significant "decrease", in the rate in this
control city, which was shown in the previous report (Brown et al., 1954b). By the use of
these figures, the reviewer exaggerates the contrast between the test
city and this control. Thus, this statement by Dr Grainger is both
inaccurate and misleading.
(42)
In the same periods the mean df rates for this age [in] Stratford
remained nearly equal (1.66 and 1.65) and increased for other ages.
Comment.
The rates for the four examinations were: 1.66, 1.76, 1.58, 1.65 (Brown,
1955). (Throughout this monograph caries rates have been given in the
form in which they appear in the original papers although it is
recognized that, in cases such as these, the practice of showing caries
rates with two places of decimals is, probably, not warranted.) Dr
Grainger mentions the least variable of the ten caries rates in the
control cities-that for the deciduous teeth of children aged nine to
eleven years in Stratford. He omits to mention the DMF rates which show
the remarkable situation, in this control city, in which each of the
inter-year changes occurring in this age group, and in five out of the
six inter-year changes in the rates of the "other ages", were
said by Brown (1955) to be statistically significant.
(43)
Item 9: Re Table 11, Ontario Department of Health Report. The printing of
dashes rather than percentages for the control cities was to avoid
confusing the table with "negative reductions" and in the case
of the 9 to 11 df figure to avoid emphasizing what was considered to be a
spurious decrease.
Comment.
This astonishing explanation, for the printing of dashes in this table,
implies that these omissions were made deliberately because the results
did not conform to those expected. Why should a decrease of 0.44 df (18
per cent) in the test city be accepted and published, but a very similar
one of 0.39 df (16 per cent) in a caries rate in the control city of
Sarnia be considered "spurious" and not published-a dash being
shown in the appropriate position in the table? By printing these dashes,
the "statisticians" of the Ontario Department of Health could
have misled their Minister into thinking that there were no changes in
these caries rates in Sarnia and Stratford (particularly as the Summary
of the report said so) but that there had been a decrease of eighteen per
cent in the corresponding rate in Brantford due to fluoridation.
Talk
of "negative reductions" cannot disguise the fact that nothing
is more calculated to confuse a table than (as Dr Grainger implies) the
deliberate omission by "statisticians" of figures giving the
percentage changes in rates (one of which was said to be significant).
(44)
This judgement was borne out by the 1955 figures.
Comment.
Dr Grainger tries to justify this "judgment" by implying that
these so-called "spurious" decreases were not seen in the 1955
figures. However, small "decreases" were still shown in that
year, the "reduction" in Sarnia being 7.6 per cent.
In
any case, this so-called "judgment", regarding the omission of
data, has no place in the preparation of an unbiased report.
(45)
Item 10: Differences in reported rates between examiners. Different
examiners give characteristically higher or lower rates upon examining
the same individuals due to differences in skill, training the physical
condition [sic]. Thus the differences quoted are no reflection on the
design of the experiment or the care taken in the work. The strength of
the double examinations comes through corroboration of caries trends in
Brantford over the years and not through interchangeability of data.
Comment.
Dr Grainger refers to the differences between examiners in the assessment
of caries rates. This important matter has already been considered. The
aim of the paragraph mentioned was to show that, as the rates obtained by
the examiners in the two Brantford studies were different, data from the
City Health Department study could not be used to decrease "the
deficiency in the data of the National Health and Welfare study, owing to
its late commencement"
The
admission, which is implicit in Dr Grainger's remark, that
"interchangeability of data" was not permissible between the
two studies in Brantford confirms the point made. The degree of reliance
which can be placed on the "corroboration of caries trends in
Brantford over the years" must be considered in the light of the
widely divergent results obtained in these studies, which were discussed
in the second paragraph of page three.
(46)
Item 11: Significant fluctuations in controls. The important point is
that for the controls the inter-year changes were upward trends or mere
fluctuations (even though in some cases calculated to be beyond change)
[sic], whereas in Brantford the change took the form of a highly
significant continual downward trend.
Comment.
It was pointed out that in the control city of Stratford five out of the
six comparisons made between the permanent teeth rates of successive
years were said by Brown et al.
(1954b) to be significant changes (four of them being at the three
standard error level). In the permanent teeth rates in the other control
city, Sarnia, there were four highly significant and one significant
change in the nine comparisons made. Brown et al. (1953, 1954b) and Brown (1955) said that the odds
relating to the occurrence by chance or sampling variation of a
difference of the magnitude of three standard errors (such as were
reported in eight of these changes) "are 369 to I against".
Therefore, when Dr Grainger terms these unexplained changes "mere
fluctuations" he is rejecting that remark of Brown et al. and denying the meaning of
statistical significance.
Dr
Grainger neglected to mention that the "highly significant continual
downward trend" in the caries rates in Brantford occurred only in
children who were aged twelve to fourteen years. In the two other age
groups, in both the deciduous and the permanent dentitions, there was an
upward trend in the caries rates in the fluoridated city during the last
year of the study, the rise from 0.44 DMF to 0.69 DMF, in the youngest
age group, being said to be a highly significant rise (Brown, 1955).
Therefore this statement by Dr Grainger, that there was a "continual
downward trend" in Brantford, is incorrect and is misleading.
(47)
Item 12. Larger percentage changes in control. There is no definite
explanation as to why rates increased in Stratford and also in Sarnia
over the ten years but this may be a reflection of a general post-war
increase in dental caries which has been seen in other areas. However, it
is significant that in the various fluoridation experiments e.g. in
Brantford, Newburgh, Grand Rapids, etc., the shift has always been
significantly downward in the fluoridated cities whereas the control city
rates have remained about the same or in the case of Stratford,
increased.
Comment.
Dr Grainger's statement, that the "rates increased in Stratford and
also in Sarnia over the ten years", is inaccurate for, as he pointed
out in his Item eight (39), "Brown's observational period did not
begin until 1948, hence is less than 10 years." Sarnia was first
examined in March 1948 and Stratford in October of that year (Brown,
1952). Therefore the caries rates of the children in both those towns
were known for a period of about seven years, not one of ten years.
Furthermore, in the deciduous teeth in Sarnia, the younger age group
showed a higher rate in 1955 than in 1948, but in the older children the
final rate was lower than the initial one. In this city the DMF rate rose
between 1948 and 1953 but between that year and 1955 there was a decrease
in this rate in each of the three age groups.
The
suggestion that there has been "a general post-war increase in
dental caries" is not supported by these studies, for such a rise it
was not seen in any of the unfluoridated control cities considered. In
Muskegon and Oak Park there was no definite trend. At the time when (as a
result of their being fluoridated) these cities ceased to serve as
controls, the rates for the children of some ages were higher, and for
other ages they were lower, than during the initial examination. The
trend in Kingston cannot be investigated owing to the method of
presenting the data in the Newburgh study. The contention that "the
shift has always been significantly downward in the fluoridated
cities" can be accepted only if the many deficiencies pointed out in
this monograph are ignored and the figures from these trials accepted at
their face value.
(48)
Item 13: Smaller percentage decrease after longer fluoridation. As
pointed out by Sutton himself on page 168 (middle paragraph) the fluoride
protection for permanent teeth of the children aged six to seven seemed
to occur within two or three years after fluoridation began. Thereafter
the yearly DMF rates were subject to random fluctuation and the
differences in percentage decrease of 69 per cent and 51 per cent are
most likely a reflection of this inter-year variation.
Comment.
No specific mention was made of "the children aged six to
seven" when pointing out the reductions in the DMF rates which were
reported to have occurred in the early years of the City Health
Department study (Hutton et al.,
1951). Some implications of this reported early decrease in caries rates
were discussed on pages two and thirty-eight.
When
he makes the remark that "the differences in percentage decrease of
69 per cent and 51 per cent are most likely a reflection of this inter-year
variation", Dr Grainger shows that he could not have noted that the
increase in the DMF rate in these children, in the fluoridated city, from
0.44 in 1954 to 0.69 in 1955 (so that the "percentage decrease"
dropped from 69 per cent to 51 per cent), was shown by Brown (1955) to
be, statistically speaking, a highly significant (three standard error
level) rise in the rate in the test city during the final year of the
study. Therefore, when he makes this suggestion, he is expressing an
opinion which contradicts the notation given by Brown (1955), in his
Table IV, which indicates that the rise in rates which produced this
percentage "decrease" is statistically significant at the three
standard error level.
Dr
Grainger may be right-but if he is, Brown's (1955) indication of
statistical significance in this case is incorrect, and the methods used
in the National Health and Welfare study for calculating statistical
significance must be questioned. In consequence, all statements made in
the study regarding significant changes in the caries rates, both in the
test city and its controls, become doubtful.
It
is of interest to note that the "1955 Report" from this study
was released in two different publications. The first, a booklet, was
dated November 1955, and was "prepared by H. K. Brown . . . with the
assistance of H. R. McLaren... G. H. Josie... and Barbara J.
Stewart". The second publication is a paper by Brown, McLaren, Josie
and Stewart published in 1956 in the Canadian Journal of Public Health,
no reference being made to the previous publication. The ten tables and
the two figures in the body of these reports are the same and the text of
both is practically identical. However, there is one important
difference: that part of the discussion dealing with the unexplained rise
(shown as significant) in the DMF rate of children aged six to eight
years in Brantford (and also in the control city with the
"optimum" concentration of fluorides in its water), was omitted
from the later report (Brown et al.,
1956).
The questions should be asked: Why was this very important small section
of the original report omitted when it was published in the Journal? Why,
in both these 1955 reports, was the smaller percentage
"decrease" between the initial and the final caries rates of
the youngest age group in Brantford not published?
(49-50)
Item 14 The quotation from the Ontario Government Report is taken out of
context from a series of summary statements. The previous statement was
to the effect that in Brantford there had been a significant decrease of
about 60 per cent in DMF rates. In the statement following, as picked out
by Sutton, it was stated that "no change" occurred in Stratford
and Sarnia(49). It should be clear enough from the context at the words
"no similar downward change" were inferred(50)
Comment.
The charge that the quotation mentioned was "taken out of
context" is meaningless unless it implies that the sense of the
original statement has been altered. So that the reader can judge this
matter, the whole of paragraph three (the "previous statement"
referred to by Dr Grainger) and paragraph four are reproduced. The
quotation given on page 176, which was the one criticized, is shown here
in italic type to distinguish it from its context.
The
evidence produced by the investigators of the Brantford City Health
Department and of the Department of National Health and Welfare,
independently show that since the introduction of the fluorine in the
water there has been a significant decrease, amounting to approximately
60 per cent in the number of decayed, missing and filled teeth suffered
by Brantford school children.
At
the same time, it has been
established that there has been no change in the already low dental
caries attack rates in Stratford which has 1.2 ppm of natural fluoride in
its domestic water, or in the relatively high rates for Sarnia which
has had virtually no fluoride in its water."
Dr
Grainger suggests that the writers of this report, when they used the
expression "no change", really meant "no similar downward
change". However, as they were members of the Division of Medical
Statistics and, presumably, were trained in the very precise science, of
statistics, it is unlikely that they would use such an inexact
expression. Also, the term "downward change" is a rather clumsy
substitute for the word "decrease" which was used in the
previous paragraph (para. 3 above). Both these points suggest that Dr
Grainger's interpretation is incorrect.
(51)
Item 15 Possible weighting effect. The critic is referring to a possible
shift in age distribution within the group, e.g. a possible sampling
shift within the six to eight-year age group so that certain years had
disproportionately higher or lower numbers of eight-year-olds and hence
higher or lower average caries scores. This is rather remote in that
selection methods used by Dr Brown were the same each year; moreover very
large shifts in age distribution would be needed to produce the
significant differences to which P N. R. Sutton refers.
Comment.
After reading Dr Grainger's remarks it may be supposed that it was said
that the inter-year significant changes in caries rates, reported from
the control cities, were due to a "weighting" effect. However,
if the original paragraph is consulted, it will be found that it is headed
"Unexplained Significant changes in controls." A
"weighting" effect was mentioned as a possible explanation for
these unexplained changes. This suggestion was made following the failure
of the authors (Brown, 1951, 1952; Brown et al., 1953, 1954b, 1956) to advance even a suggestion why
these changes occurred. Dr Grainger's comment leaves these significant
changes in the caries rates of the control cities as the authors of this
study left them - unexplained.
This
reviewer has made it clear that the statements regarding the
"continuous" use of water containing fluorides in
concentrations of 1.3 or 1.6 ppm since 1917 in Stratford are not based on
data (comment 37); and that the misleading omission of figures from the
Ontario Department of Health Report (1956) was made deliberately (comment
43). However it has been seen that, although Dr Grainger said (27) that
his aim was "to help set the record as straight as possible",
most of his comments, if they had been accepted at their face value,
would have had the reverse effect.
MR J. FERRIS
FULLER
Apart
from the reviews already quoted above, the only published criticism known
to the author is that contained in the Book Reviews section of the
January 1960 issue of the New Zealand Dental Journal. This was
contributed by MR J. FERRIS FULLER, a member of the Dental Research
Committee of the New Zealand Medical Research Council and a member of the
Fluoridation Committee of the Department of Health, whose submissions to
the New Zealand Commission of Inquiry (1957) are mentioned in over twenty
paragraphs of its report.
Mr Fuller's
review stated:
"Everyone
is out of step except our Albert," or so the author would have us
conclude. Altogether an extraordinary book; clever but unfortunate;
skilfully contrived and yet-stripped of its finery-rather slender. It
could be ignored if the matter rested within the Sciences; but since by
the very nature of the subject it takes us into the public forum, some of
the errors must be stated..
Part
I of Fluoridation: Errors and Omissions in Experimental Trials (Chapter
19 here) is a reprint of a paper by Sutton and Amies (see footnote on
page 136) that appeared recently in the Medical Journal of Australia
criticising the Brantford-Sarnia-Stratford study in Canada(52). But the
authors have omitted to read the literature(53), and their criticisms
therefore are not based on the known facts. This is a serious matter
especially when the comments come from two critics who exalt themselves
above fellow scientists of at least equivalent status in other parts of
the world. They accuse the Canadian workers of failing to devise a
randomisation procedure that would eliminate bias(54), of deliberately
omitting vital information in some of the tables(55), and finally of
displaying bias in the presentation of results(56). Their comments are
based on a report of the Ontario Department of Health (1956) to the
Ontario Minister of Health, a report obviously written in simple
abbreviated terms for public consumption(57). Sutton and Amies failed to
read two official publications readily available(58), namely, a 51-page
booklet "A suggested methodology for fluoridation surveys in
Canada" and the 35-page detailed report of the Department of Health
and Welfare, of November, 1955 These two booklets together show that great
care was taken to introduce a well-designed randomisation procedure(59),
that examiner variability was eliminated as far as humanly possible by
the employment of one examiner only throughout the whole period of the
study(60), and that the information alleged to have been omitted is in
fact shown in detail in the tables in the 1955 report(61), together with
the standard error for each of the indices used. In short, the more
important criticisms that appear so damaging are in fact without
foundation. Thus, when the authors say that "what must be eventually
a statistical study does not appear to have been designed as such"
and "no attempt at statistical evaluation has been considered"
their comments are absurd and, indeed, irresponsible(62). The full
official report on the Brantford study was available in New Zealand,
incidentally, when the Commission of Inquiry held its hearings(63), and
three of its tables are included in the published report of the
Commission.
In
Part 2 of the book Sutton continues in the same vein. He complains that
misleading comments are made in some reports, yet his own book contains
many misleading statements. For example, he claims that a proper
evaluation of examination errors at Grand Rapids has not been carried out(64),
and he doubts the accuracy of caries attack rates in test and control
areas because X-ray examinations were incomplete or absent(65). It is
significant that he omits to refer to a report by Hayes, McAuley, and
Arnold published in the U.S. Public Health Report in December, 1956,
which is a key reference in this subject(66). This report met the
specific point that "some observers have suggested that X-rays are
essential to determine the efficacy of caries control measures" and
an investigation was undertaken "to determine whether or not
supplementing direct observation with X-ray examinations would affect the
conclusions based on direct observation alone." The conclusion was
that supplementary X-ray examinations supported the clinical findings and
did not change the basic observation that substantial decreases in dental
caries occurred during the test period. The very standard errors that
Sutton demands for a proper statistical evaluation were available in this
report(67). He quotes a subsequent (1957) paper by McAuley that suits his
book and, in the light of his criticisms and allegations, this makes the
omission of any reference to the 1956 report more damaging(68). To borrow
his own phrase, omissions of this nature render his work "open to
doubt." Sutton criticises his overseas colleagues for their
inability to examine children in control towns prior to fluoridation(69).
With personal experience of a study of this nature he would appreciate
that where on the one hand the interests of a large number of people and
their local bodies and institutions are concerned as compared with only
one or two examining personnel on the other, it is almost impossible to
operate a plan to the exactitude dreamed of at the statistician's desk.
In any event, the criticism is rather meaningless as far as the Grand
Rapids study(70) is concerned when we realise that the baseline
examination in the control city of Muskegon showed that caries prevalence
in that city is of the same order as in Grand Rapids.
In
attacking the Evanston-Oak Park study, Sutton bemoans the lack of
information about the design of the study and phrases such as "It is
not clear...", "It is not understood...", (It) was not
stated..." give the lead to questions and speculations that follow.
But why not adopt the simple expedient of writing to the workers
concerned and so finding out instead of speculating? This attitude is
typical of the book(71). And typical also is the quibbling over details
that do not detract one iota from the part that fluoridation has played
in these areas in reducing dental decay(72). "The total tooth
surfaces considered... should be 58,325, not 58,352" says the
author, and also... the mean of these values for 1946... is 150.09, not
149.76"(73). Dear me, Dr Sutton, how dreadful.
And
then we come to the Newburgh-Kingston study. Prominence is given to the
different composition of the waters at Newburgh as compared with the
control city of Kingston(74), and this is cited as the reason why the
latter is unacceptable as a control. But once again Sutton omits any
reference to a key report, that by Dean, Arnold, and Elvove of August,
1942, listing caries prevalence rates in communities where the variables
in the domestic water mentioned by Sutton varied to a greater degree than
between Newburgh and Kingston without caries prevalence being markedly
affected(75).
The
author complains of bias in the manner in which some results are
presented but, as can be seen, he displays bias himself in the choice of
articles he quotes(76) and in his omission to read others. It is not
surprising, therefore, to see him fall into the familiar pattern of the
anti-fluoridationist. Those who question fluoridation are given the
familiar title of "eminent authorities," a distinction not
afforded anyone else(77). It is surprising, however, to see him serve his
ends by quoting Feltman's study on the use of fluoride tablets. This
study lacks the very control that one would expect Sutton to consider
essential(78).
As
one would expect, there are no bouquets for the New Zealand Commission of
Inquiry, one complaint being that "no mention was made of the
employment of a statistician to assist its members in evaluating
numerical data." Had the author inquired, he would have been told
that the Professor of Biochemistry on the Commission was well versed in
biometrics, and that scientific witnesses quickly discovered that tables
were unacceptable unless they contained complete details including
standard errors, so that he could evaluate data statistically for himself
and the Commission(79).
Finally,
a warning to those reading this book, lest they be misled by the polemics
and the array of figures. Please note that Sutton's conclusions in part 2
(which forms the greater part of the book) are confined to variations in
the prevalence of dental decay in control cities and not to the cities
where fluoride has been added(80). What of the places where fluoridation
has been adopted? Sutton does not dispute the fact that the prevalence of
dental decay has been substantially reduced in the fluoridation cities of
Grand Rapids, Newburgh, Brantford, and Evanston(81), nor does he mention
that these good results have been confirmed by several independent
studies in the U.S.A., and also in Tasmania, Brazil, Japan, Germany,
Sweden, and at Hastings in New Zealand(82). The validity of the results
from Hastings, incidentally, has been checked by the Applied Mathematics
Laboratory of the New Zealand Department of Scientific and Industrial
Research(83).
The
anti-fluoridationists will rejoice with fresh ammunition to replenish
their stocks; but it is unlikely that this work will serve any useful
purpose in scientific circles despite the author's rather pretentious
hopes. The performance is almost as old as Time: "The mountains are
in labour, there will be born a ridiculous mouse," said the ancient
poet.
|
No comments:
Post a Comment