Skip to main content

Hattie’s Effect Size: A pseudoscience or critics just being critics?


Hattie’s meta-mata analysis that culminated in the publication of his most influential work of Visible Learning (2009), and later updated to include more studies, has been hailed as the “holy grail” for educators and education leaders around the world. In particular, his effect size of instructional practice interventions has had the lion’s share of his work. Hattie considered that if schools set the effect size at 0 then “virtually everything works, and so we need to shift the question from “ what works in education” to “what works best in education”. Hattie’s meta-meta analysis of more than 800  meta-analyses studies comprising 50,000 studies (later included more 1500 meta-analyses) revealed that the baseline of the effect size that schools should start from is not 0 but 0.4, termed as the “hinge point”. In other words, for medium to large effect sizes on student achievement, the effect size of an instructional practice should be o.4 and above. This does not mean that we need to discard all positive effect sizes below o.4. It means that what works best is o.4 and above. Hattie also published a list of 150 influences on student achievements based on their effect sizes, now being collective teacher efficacy as the number one influencer on students achievement, teacher feedback, student and teacher expectations, student-teacher relationships, and  feedback for teachers among medium to large effect sizes.
It is worth noting  also that when considering an effect size, other factors need to be considered as well, like resources, effort, and time put into an instructional strategy. For example, if a school opted from smaller classes, which has a medium effect size, they need to consider the resources, money, and time they need to accommodate to have such a a medium effect size, which by comparison to teacher feedback (done well) the latter makes much more sense as it yields higher effect size with much less effort and resources.
However, Hattie’s effect size validity has been under numerous attacks lately, accusing it of being skewed. I was aware of such criticism before, but it has become more salient lately through some publications and discussions on Twitter.






This Twitter conversations were due to Dr. Robert Salvin’s (currently Director of the Center for Research and Reform in Education at Johns Hopkins University and Chairman of the Success for All Foundation) post John Hattie is Wrong  where he accuses Hattie of bias study findings by accepting “ the results of the underlying meta-analyses without question” as “most meta-analyses accept all sorts of individual studies of widely varying standards of quality”. He adds that
To create information that is fair and meaningful, meta-analysts cannot include studies of unknown and mostly low quality. Instead, they need to apply consistent standards of quality for each study, to look carefully at each one and judge its freedom from bias and major methodological flaws, as well as its relevance to practice. A meta-analysis cannot be any better than the studies that go into it. Hattie’s claims are deeply misleading because they are based on meta-analyses that themselves accepted studies of all levels of quality.

Towards the end of this post, Dr. Slavin refers to his work on Evidence Syntheses (www.bestevidence.org) as reliable and unbiased. Was he trying to promote his work? Does he feel Hattie’s work and commercialization of his work have overshadowed his work on reliable meta-analysis?

Hoever, Dr. Salvin’s post was not the only criticism, earlier in 2017 an academic article by Jerome Bergeron, published in McGill Journal of Education, and titled  HOW TO ENGAGE IN PSEUDOSCIENCE WITH REAL DATA: A CRITICISM OF JOHN HATTIE’S ARGUMENTS IN VISIBLE LEARNING FROM THE PERSPECTIVE OF A STATISTICIAN  likened Hattie’s research to “a fragile house house of cards that quickly falls apart”. He claims that Cohen’s d (Hattie’s measure of effect size) simply cannot be used as a universal measure of impact. He even accused Hattie, his works, and his followers and believers as blind and it falls into the promotion of pseudoscience (reminds me of NLP).
To believe Hattie is to have a blind spot in one’s critical thinking when assessing scientific rigor. To promote his work is to unfortunately fall into the promotion of pseudoscience. Finally, to persist in defending Hattie after becoming aware of the serious critique of his methodology constitutes willful blindness.

He says that Hattie, though not afraid of numbers, has created many errors, but the most two salient ones are:
  1. Miscalculation in Meta-analyses
  2. Inappropriate baseline comparisons
He says that Hattie does not use statistically sophisticated formulas, that he summarizes by calculating averages and standard deviations, the latter he did not use, that he uses bar graphs not histograms and uses a formula that converts a correlation to Cohen’s d without being heedful of the validity of his conversion.

He insists that effect size, despite the absence of a unit, is a relative measure that provides a comparison to a set, group, or baseline population, even if it may be implicit. He says that to compare two independent groups is not the same as comparing grades before and after an intervention implemented with the same group. Hattie’s comparisons are arbitrary and he is completely unaware of it.
He also points to an error of arbitrariness of Hattie’s barometer. He says that in addition to mixing multiple and incompatible dimensions, Hattie confounds two distinct populations: 1) factors that influence academic success and 2) studies conducted on these factors.
He concludes his article with a call for researchers in education and education departments to consult keen statisticians instead of doing it themselves.

Another fierce criticism on Hattie’s work (and Hattie himself) was termed the “cult of Hattie” in another academic article by Scott Eacott in 2017, titled School leadership and the cult of the guru: the neo-Taylorism of Hattie. In this article, Eacott contends that Hatti’s word in Visible Learning “has become not only the latest fad or fashion, almost to the point of saturation, but reached a level where it can now be labelled the ‘Cult of Hattie’. In which claims that Hattie has deliberately discarded context, a very powerful influence on student achievement. Eacott’s attack, unlike Bergeron’s, does not attack the statistical errors in Hattie’s work but claims that Hattie did three things to build is Visible Learning empire (He is localizing Hattie’s influence in Australia):
1- Hattie uses specific temporal conditions that enabled the rise of Hattie’s work across the nation (Australia)
2- Unlike past attempts at pedagogical reform, Hattie’s work has provided school leaders with data that appeal to their administrative pursuits>
3- Hattie articulated a new image of school leadership (him being their savior)

What do you think? Is Hattie’s “Holy Grail” the wrong one? a pseudoscience? or are these critics, among many others, are just being well, Critics?

Comments

Popular posts from this blog

Why some kids can’t spell and why spelling tests won’t help

  Misty Adoniou , University of Canberra A couple of years ago, early one morning, I received an SMS advising “resadents to stay indoors because of a nearby insadent”. I was shocked by the spelling, as much as the message. Surely, I thought, if it was a real message then the spelling would be correct. Spelling matters. In a text message from a friend teeing up a night out “c u at 8” is fine - but in an emergency warning text from a government agency, I expect the spelling to be standard. But why is it that some people struggle with standard spelling? Spelling remains the most relentlessly tested of all the literacy skills, but it is the least taught. Sending a list of words home on Monday to be tested on Friday is not teaching. Nor is getting children to write their spelling words out 10 times, even if they have to do it in rainbow colours. Looking, covering, writing and checking does not teach spelling. Looking for little words inside other words, and doing word searches ar...

Don’t let the rise of Europe steal world history

The centre of a map tells you much, as does the choice where to begin a story, or a history. Arab geographers used to place the Caspian Sea at the centre of world maps. On a medieval Turkish map, one that transfixed me long ago, we find the city of Balasaghun at the heart of the world. How to teach world history today is a question that is going to grow only more and more important. Last summer in the United States, a debate flared when the influential testing agency Advanced Placement (AP) announced a change to its attendant courses, a change in which ‘world history’ would begin in 1450. In practice, beginning world history in 1450 becomes a story about how Europeans came to dominate not one but all the continents, and excludes the origins of alphabets, agriculture, cities and civilisation. Before the 1400s, it was others who did the empire-building, drove sciences, medicine and philosophy, and sought to capitalise on and extend the trading networks that facilitated the flow and exc...

Here’s How to Create a Purpose-Driven Organization

This is not, yet another, post on the importance of an organization's purpose. Everyone seems to be writing about it. In fact, it’s the most discussed topic in transformational leadership since every organization should have a purpose of “being”. Talk to any employee and he can either recite his organizational purpose and values statements or remember its key words. In my school, some  of our teaching faculty know it by heart, most memorized its buzz words: Well-rounded, Interdependent, Social skills, Robust educational program. WISE they call it. We even have a mission week where our purpose statement is  displayed everywhere, football matches are played, and  prizes are won. Some banks even force their employees to memorize their vision and mission statements.  But in order for an organization to get employees to bring their brainpower and vigor to work, it needs to be purpose-driven not purpose-ostensible. Robert Quinn and Anjan Thakor, both prominent researchers ...