Paul Maxim's "Renorming Ron Hoeflin's Mega Test"
Published with permission from the author
References:
1. Ron Hoeflin's "Sixth
Norming of the Mega Test."
2. Fred Vaughan's "Intelligence
Filters." (appears in the online version of Gift of
Fire)
3. Kevin Langdon's Reply to Paul Maxim. (Gift of Fire no.
81)
4. Darryl Miyaguchi's Comments
on Paul Maxim's renorming.
5. Darryl Miyaguchi's Generic I.Q.
Chart
6. SAT
percentile ranks table
RENORMING RON HOEFLIN'S MEGA TEST
Copyright (C) 1996 by PAUL MAXIM
Reference Data Used:
1. Data sheets provided by Dr. Hoeflin, showing Mega test scores for 531
testees, plus prior test scores they reported, including the following: 220 on
SAT, 106 on GRE, 76 on LAIT, 80 on Cattell, 75 on CTMM, 46 on Stanford-Binet,
34 on WAIS, 28 on AGCT, and 28 on MAT.
2. Score breakdowns for 5,157,642 SAT testees during the period 1984-1989.
3. GRE norming data from ETS.
4. The IQ scale chart ("Selectivity by IQ") ascribed to Kjeld Hvatum.
5. A description by Dr. Hoeflin of his Mega test norming process, as con-
tained in his letter to Mr. Maxim of December 19, 1995. My renorming does not
attempt to theoretically analyze that norming process, but rather replicates
it, using the same data Dr. Hoeflin used; hence, it constitutes an audit of
his results.
NOTE: This article has not been reviewed by, or approved by, Dr.
Hoeflin.
Description. The method Dr. Hoeflin used to norm his Mega test, based on 693
reports he received of prior test scores, can be illustrated in terms of his
largest prior score sample, consisting of 220 SAT scores. (All statements
herein pertain to SAT before its recent recentering.)
a) In order to norm the Mega test at the "4-sigma" level (IQ 164), it is
first necessary to determine which score level on SAT corresponds to "4-sigma"
in the general population, equivalent to the top .00333% of all individuals.
b) Since SAT testees are slightly more intelligent than the population as
a whole, by a factor of about 4/3, their "4-sigma" percentile is raised to
.00444%, representing the top 1/22,500th of all SAT testees.
c) This percentage is then applied to total number of scores for the period
tabulated, indicating that 229 scores were at the "4-sigma" level or above.
d) Beginning with the top SAT score level of 1600 on Math plus Verbal (a
"perfect" score), one can then count downward 229 scores until the "4-sigma"
level is reached for this data aggregate.
e) Dr. Hoeflin's norming assumed that 1550 on SAT represented its "4-sigma"
level. However, his own data reveals the following numbers:
From this data, it can be interpolated that a score of 1565 1591-1600 - 35
on SAT represents the "4-sigma" level; that is, all scores 1581-1590 - 8
at 1565 or above number .00444% of the total. 1571-1580 -149
1561-1570 - 71
f) It is now necessary, from inspection of the prior SAT
scores reported by Mega testees, to determine how many of them were at 1565 or
above. Dr. Hoeflin presumed that his top ten SAT scorers were in the "4-sigma"
category; their scores were as follows: 1595, 1586, 1582, 1580, 1570, 1565,
1560, 1560, 1556, and 1555. However, only six of these scores are at or above
1565, and so it is they which represent the true "4-sigma" group.
g) Once it is determined that the top six SAT scores represent the "4-sigma"
category on this test, the next step is to pick out, among those 220 testees
reporting SAT scores, their top six Mega scores: these are 44, 44, 43, 43, 40,
and 39. In other words, an implicit assumption is made that the SAT-reporting
group will attain the same number of "4-sigma" scores on the Mega test as they
did on SAT; hence, the lowest of those six top Mega scores represents the as-
sumed "4-sigma" threshold on Mega, for the SAT-reporting group. This score was
39.
h) This putative "4-sigma" level on Mega will now have to be combined and
correlated with the "4-sigma" level derived from a similar procedure applied
to testees reporting GRE scores, to testees reporting CTMM scores, etc. Then,
when all applicable "4-sigma" levels have been determined, a weighted average
can integrate them in terms of sample size, so as to arrive at one "4-sigma"
score level for the Mega test as a whole.
GRE Sample. Dr. Hoeflin did not report to me the GRE (combined Math plus Ver-
bal) score level he equated with "4-sigma," but data emanating from ETS indi-
cates that this should be "1620." The problem in attempting to norm a high-
IQ test on GRE scores at the 4-sigma level is that, around the late 1970's,
ETS reduced the ceiling on each GRE from 900 to 800, making it impossible for
any testee, from that point on, to record a "4-sigma" score.
A number of Dr.
Hoeflin's Mega testees apparently took the GRE before its ceiling restrictions
were imposed, since three of them reported scores of 1620 or above. However,
since the Mega test was normed around 1986, it is possible that one or more of
the 106 testees reporting GRE scores was prevented, by the ceiling, from at-
taining a "4-sigma" score. In order to "adjust" for this possibility, I have
increased the number of GRE scores at the "4-sigma" level from three to four.
The next step (as in the case of the SAT sample) is to count off the top four
Mega test scores attained by testees in the "GRE group"; these were as fol-
lows: 43, 43, 41, and 40. This indicates that a raw score of 40 on the Mega
test corresponds to "4-sigma" (1620) in the GRE sample.
LAIT Sample. This represented the next most numerous group, accounting for
11% of all prior score reports used by Dr. Hoeflin. In his letter to me
dated December 19, 1995, Dr. Hoeflin wrote: "Of 77 (Mega testees) who reported
LAIT scores, 16 scored IQ 164 or higher on LAIT. For this same sample of 77
people, the top 16 scored 33 or higher on my Mega test. This puts the four
sigma level (for the Mega test at) 33..."
But there is a problem here, that
Dr. Hoeflin has failed to examine critically. To begin with, the incidence of
putative "4-sigma" scores in the LAIT sample (16 of 77) is far higher than
that of any other sample, and in percentage terms (20.78%) is over six thou-
sand times higher than the incidence of "4-sigma" in the general population
(.00333%). This stems directly from the fact that LAIT is an inflationary
test, and generated numerous invalid "4-sigma" scores.
This distortion may
also be noted in the fact that, in order for 20.78% of any sample to score at
the "4-sigma" level or above, the entire sample would require a mean IQ over
3.2 sigma -- that is, over IQ 152, which is about ten points higher than the
general profile for MEGA testees, and for LAIT testees as well.
Dr. Hoeflin's Mega test data indicates that LAIT scores were, on average, 4
IQ points higher than Mega scores, which in turn were about one point higher
than CTMM scores, even though the CTMM was taken, on average, about a decade
earlier than Mega. This means that LAIT scores were at least five points
higher, on average, than scores emanating from a professional test such as
CTMM, which was normed on about 60,000 members of the general population, not
on prior, anecdotally-reported test scores. Furthermore, Dr. Hoeflin's data
indicates that 17 LAIT scores at the 4-sigma level and above were, on average,
eight IQ points higher than Mega test scores attained by the same testees,
with a mean time interval between tests of about six years.
Now, when scores from an inflated
test (LAIT) are used uncritically to norm a subsequent test (Mega), the second
test tends to acquire some of the inflationary characteristics of its predeces-
sor, producing what might be called the "house of cards" syndrome -- that is,
one inaccuracy piled atop another. To compensate for this factor, I deducted
four points from each reported LAIT IQ score used by Dr. Hoeflin, which re-
duced the number of putative "4-sigma" LAIT scores from 16 to six. (This is
probably still too high, since six 4-sigma scores out of 77 is still 2300 times
greater than the incidence of 4-sigma in the population as a whole.)
Follow-
ing this adjustment, the six highest Mega scores attained by the LAIT testees
were as follows: 44, 43, 42, 41, 40, and 39, which accords very well with the
"4-sigma" level on the Mega test suggested by the SAT and GRE samples. Hence,
the 4-sigma level for the LAIT sample on the Mega test is pegged at "39."
CTTM Sample. Dr. Hoeflin's data notes only two Mega testees who reported
prior CTMM scores over "4-sigma," which on this test is 164 IQ; these scores
were 179 and 174. Similarly, among the 75 Mega testees who reported CTMM
scores, the two highest Mega scores were both "43," indicating that this score
should be presumed to represent the "4-sigma" level on Mega. But for some rea-
son unknown to me, Dr. Hoeflin did not use this figure verbatim; he instead
"adjusted" it downward to "40," which seems to have no statistical justifica-
tion. Hence, I have used the original CTMM-generated Mega score of 43 in my
renorming of Dr. Hoeflin's test.
Stanford-Binet Sample. 46 Mega testees reported prior scores on the Stanford-
Binet, including a hefty score of "230" attained by Marilyn vos Savant, who
also scored "46" on Mega. For some reason, Dr. Hoeflin did not use Marilyn's
score in his Mega test norming -- perhaps because it is identifiably a youth-
ful score, but I have included it in my renorming. With Marilyn's score in-
cluded, the S-B testees manifested a mean IQ of 146.9, about 7 IQ points high-
er than the CTMM testees, which suggests that there may have been other youth-
ful scores (aside from Marilyn's) included in this sample.
There are five
Stanford-Binet scores over 4-sigma, and the corresponding top five Mega test
scores (for the S-B group) are 46, 40, 34, 32, and 29. This points toward 29
as the raw score on Mega which best corresponds to "4-sigma" in the norming
sample, but is ten points below the other indications.
WAIS Sample. 34 Mega testees reported prior WAIS scores, representing 4.9%
of the total scores used by Dr. Hoeflin. Two of these scores were "4-sigma"
(164 and 162, since the standard deviation on WAIS is 15), and the two top
Mega scores attained by the WAIS testees were 34 and 33.
Cattell, AGCT, and MAT Samples. Since there is no indication that any of the
testees in these samples reported a "4-sigma" score, they cannot be used for
norming the Mega test at the 4-sigma level.
Combining Sample Results. The various "4-sigma" levels (on the Mega test) gen-
erated by each prior test sample cited above may now be combined into one, by
means of a "weighting" technique, so that the effect of each sample will be
proportional to its size; this is done by computing a weighted average, as fol-
lows:
Prior Test Per Cent of the 4-Sigma Level on Mega
Utilized Norming Sample Test for this Sample Product
SAT 31.7% 39 1,236.3
GRE 15.3 40 612.0
LAIT 11.0 39 429.0
CTMM 10.8 43 464.0
Stanford-B. 6.7 29 194.3
WAIS 4.9 33 161.7
80.4% 3,098.7
The next step is to divide the sum of products (3,098.7) by the sum of weights
(80.4), so as to arrive at a quotient of 38.54, representing the putative "4-
sigma" level on the Mega test for this group of 531 testees; another group
might yield slightly different results. It would also be helpful to repeat the
above norming process at the 3-sigma and 2-sigma levels, and then draw a smooth
curve through all points; however, it should be fairly obvious that it is the
norming at the 4-sigma level which most directly affects societies such as Pro-
metheus and Mega.
At this point, before equating "38.54" raw points on Mega
with 164 IQ on Stanford-Binet, a slight adjustment must be made to account for
"regression to the mean." Since we are dealing with the very top scores, and
since a decade or more elapsed, on average, between taking the Mega test and
its predecessors, this group of testees should expect to score slightly lower
on the Mega than on their prior tests, all other factors being equal. This
means that the "4-sigma" level we computed for the Mega test corresponds to
slightly less than "164 IQ" on the Stanford-Binet, and the simplest way to in-
corporate this adjustment is to set the "4-sigma" level on the Mega test at 39.
If the "4-sigma" level on Mega is moved upwards by three raw points, from Ron
Hoeflin's value of "36" to "39," this also implies that the raw score corres-
ponding to the 4.75 sigma level must be moved up by a similar amount, from its
former setting of "43," to a new value of "46."
Another important factor to be considered in appraising the accuracy of
the Mega test norming is whether or not it is generating too many IQ's at the
4.75 sigma level. In 1992, the Wall Street Journal reported that 12 such
scores had resulted from 4,000 Mega test administrations, for a percentage in-
cidence of .3% -- but this is 3,000 times greater than the incidence of 4.75
sigma in the general population.
It can be shown that, in order for .3% of the sample to score at a mean (not a
threshold) IQ of "176," the entire sample would need a mean IQ of about 155.
However, Hoeflin's norming sample of 531 testees (82% of which consisted of
OMNI readers, and 18% of high-IQ society members) had a mean IQ somewhere in
the low 140's, while his overall sample of 4,000 (over 90% of whom were OMNI
readers) probably had a mean IQ below 140. At a 141 IQ level (5 in 1,000),
five thousand tests would have to be administered to yield the expectation of
only one "176 IQ" score; this again leads me to believe that only one, or pos-
sibly two, valid "Mega-level" IQ's have ever been identified through the Mega
testing program.
Here is a tabular summary showing (by sample type) the incidence of "4-sigma"
scores used in norming -- and in renorming -- the Mega test:
Name of Size of Norm- ---Incidence of "4-Sigma" Scores---
Prior Test ing Sample As Used by Hoeflin -As Corrected--
No. Per Cent No. Per Cent
SAT 220 10 4.5% 6 2.7%
GRE 106 ? ? 4 3.8
LAIT 77 16 20.8 6 7.8
CTMM 75 2 2.7 2 2.7
Stanford-B. 46 4 8.7 5 10.8
WAIS 34 2 5.9 2 5.9
Cattell )
AGCT )
MAT ) These tests were not used for norming at the 4-sigma
Miscellaneous) level, since no "4-sigma" scores were reported.
In the above Table, the one glaring anomaly which stands out is the extremely
high incidence of "4-sigma" scores (20.8%) attributed to the LAIT norming sample.
But in his letter to me of December 19, 1995, Dr. Hoeflin stated as follows:
"Of 25,000 (testees) who took the LAIT, let's say 625 scored at or above the
four sigma level. Of 4,000 who took my Mega test, almost exactly 100 scored
at or above the four sigma level. If we compare these ratios, we find that
they are identical, namely: 625/25,000 = 100/4,000 = 1/40." (This equals 2.5%)
Herein lies another conundrum: If Dr. Hoeflin believes it reasonable for 2.5% of
all LAIT testees to score "4-sigma," why did he uncritically use (for purposes
of norming the Mega test) a LAIT sample whose incidence of "4-sigma" scores was
eight times greater, and did he not realize that this would inflate his Mega
test?
In the March 1996 issue of Dr. Hoeflin's magazine, The Puzzler (P. 16),
Mr. Langdon noted that only one out of three of the LAIT testees he originally
"qualified" for admission into the Four Sigma Society had true 164 IQ's. In ad-
dition, Dr. Hoeflin noted, "Langdon's second norming actually lowered everyone's
IQ at the upper levels by 5 IQ points compared with his first norming." In
other words, both Langdon and Hoeflin were well aware of the inflationary ef-
fects of LAIT on testee IQ assessments.
Going back even further in time (1986),
Dr. Hoeflin stated, in Gift of Fire: "I do not trust the norming of the Langdon
test, and would prefer that we (i.e., the Prometheus Society) adopt a more strin-
gent norming procedure...Inflated IQ standards that are not in harmony with real-
world facts strike me as dishonest...The Mega Society likewise has far more mem-
bers than its purported one-in-a-million standard warrants." But if Dr. Hoeflin
advocated "more stringent norming procedures" for the LAIT in 1986, one wonders
why he didn't use "more stringent procedures" to norm his own Mega test around
the same time, particularly as regards his casual acceptance of anomalous "4-
sigma" levels in his LAIT-derived sample?
To answer my own question, the only
reason I can see why Dr. Hoeflin did not utilize the "more stringent" norming
standards he recommended to Mr. Langdon is because this would have reduced the
membership in Dr. Hoeflin's Mega Society to two persons. In other words, Dr.
Hoeflin's 1986 statement in Gift of Fire was completely correct -- too bad he
didn't stick to it!
Return to the Uncommonly Difficult I.Q. Tests page.