Meta-analysis in Health and Medicine
DeLMAR hosted a well-attended workshop on Meta-analysis in Health and Medicine on Thursday 9 March at Burwood Corporate Centre. A unifying focus of DeLMAR's work is the development and application of meta-analytic methods to research questions and problems in the social and natural sciences.
Tom Stanley, Professor of Meta-Analysis in the Faculty of Business and Law, and Emily Kothe, School of Psychology, discussed how meta-analytic methods are applied in a health context.
DeLMAR ANZMAC Meta-Analysis Workshop
DeLMAR was a bronze sponsor of ANZMAC 2017, held at RMIT, and hosted a highly successful Meta-Analysis workshop on Thursday 7 December, focussing on the motivation and scoping of meta-analytic studies, search terms and search locations, and hands-on exercises on extracting and coding data and calculating effect sizes and weights. Presenters also reviewed standards for Systematic Reviews, the PRISMA and MAER-Net guidelines, and how meta-analysis helps advance knowledge.
The presenters were (from left to right) Professor Tom Stanley, Dr Josh Newton, Professor Chris Dubelaar, and Professor Stephen Holden (Macquarie University)
An Introduction to Meta-Analytic techniques in Marketing and Management, Thursday 26 October 2017
Have you ever thought about running a meta-analysis but weren’t sure how to do one? Then this is the workshop for you! Using datasets from published papers and taught by people who have published meta-analyses in prestigious journals, this hands-on half-day workshop will introduce you to the basics of how to:
- Plan, conduct, and document a systematic search of the literature
- Extract the relevant data from the papers you find
- Convert the data you extract into the metrics required to conduct a meta-analysis
- Conduct a basic meta-analysis
- Test your data for basic problems of bias and heterogeneity
- Report your meta-analytic findings
DeLMAR Launch, 9th October 2017
DeLMAR was launched on Monday 9 October 2017 with a Keynote Address by Professor John P.A. Ioannidis, from Stanford University, and speakers Professor Mike Ewing, and the centre co-directors Professor Chris Dubelaar and Professor Chris Doucouliagos.
Professor Ioannidis' keynote presentation was titled Meta-analyses and meta-research: evidence synthesis and bird's eye views of science.
Workshop on the Meta-Analysis and Research Synthesis, 14th February 2017
Presenters: T.D Stanley (Hendrix College) and Chris Doucouliagos (Deakin)
In the first half of the workshop, Professor Stanley discussed meta-analysis methods and the synthesis of empirical research. The methods were illustrated with an example of the marriage premium and the baby penalty; do men gain from marriage and women lose from having children? In the second half of the workshop Chris Doucouliagos provided hands-on research synthesis training using data on the impact of private and public health expenditure on life expectancy and mortality.
Professor Chris Dubelaar's presents on the meta-analysis of the effect on both consumers and restaurateurs of printing calories on menus.
Stanley, T.D., Carter, E.C. and Doucouliagos C. (2018). What meta-analyses reveal about the replicability of psychological research. Forthcoming in Psychological Bulletin.
"What meta-analyses reveal about the replicability of psychological research" reports a survey of 12,065 estimated effects from nearly 8,000 research articles and finds that the average statistical power in psychology is 36% and only 8% of studies have adequate power. Typical heterogeneity is nearly three times larger than reported sampling error. Heterogeneity this large easily explains recent highly-publicized to replicate in psychology failures (e.g., Open Science Collaboration 2015, Science). In most cases, we find little evidence that publication bias is a major factor.
Ioannidis, J.P.A., Stanley, T.D. and Doucouliagos, C. (2017). “The power of bias in economics research,” The Economic Journal, 127: F236-265.
John Ioannidis (Stanford University, DeLMAR), Chris Dougcouliagos (Chair of Economics and DeLMAR) and Tom Stanley (DeLMAR) have published a paper, “The power of bias in economics research” as part of a The Economic Journal’s Feature: “The Confidence Crisis in Science.”
“The power of bias in economics research” is the first large-scale study of the bias, statistical power, and hence the scientific credibility of economics.
A survey of 64,076 economic estimates from 159 areas of research and 6,700 empirical studies finds that the median statistical power is 18%, or less. That is, the probability that an empirical economic investigation is able to identify what it seeks is usually 18%, or less. Impotence begets bias. This survey also identifies widespread bias. Typically, reported economic effects are inflated by 100% with one-third inflated by a factor of four or more.
In other words, over half of economic research results are reported to be twice as large as they actually are, and one-third are exaggerated to be four times too big. Lastly, 90% of economics findings are under-powered, relative to the widely accepted convention that 80% defines ‘adequate’ power, for half of these areas of economics research.
This DeLMAR research collaboration has generated much discussion in the popular press:
Stanley, T.D, Doucouliagos, C. and Ioannidis, J.P.A., (2017). Finding the power to reduce publication bias, Statistics in Medicine, 10: 1580- 1598
In this paper, the DeLMAR research team of John Ioannidis (Stanford University, DeLMAR), Chris Doucouliagos (Chair of Economics and DeLMAR) and Tom Stanley (DeLMAR) introduce a new meta-analysis summary estimator, the weighted average of the adequately powered (WAAP), that they show to be as good as and often better than conventional meta-analytic summary statistics, notably random-effects (RE). Their simulations show that WAAP will have consistently smaller bias than RE when there is selective reporting (or publication bias) at no loss to its other statistical properties. When there is no selective reporting, the difference between RE's and WAAP's statistical properties is practically negligible.
From what this DeLMAR team learned by conducting these two research projects, we now recommend that all meta-analyses report both the WAAP and the median power for their area of research. This short 10 minute video shows how to calculate median power and WAAP.
Meta-Analysis of Economics Research (MAER) Network
MAER-Net is an international network of scholars committed to improving economic science through meta-analysis. The purpose of our network is to serve as a clearinghouse for research in meta-analysis and economics and to improve the communication among scholars in this rapidly growing field.
Attendees at the 2017 MAER-Net colloquim Zeppelin University, Friedrichshafen, Germany, 12-14 October, with Professor Chris Doucouliagos, Alfred Deakin Professor, Chair in Economics and DeLMAR Director and Professor Tom Stanley.
Professor Tom Stanley gave a keynote lecture, 'Recent developments: should we WAAP Economics' low power?'.
MAER-Net attendees 2018
Tom Stanley gave an invited keynote lecture, “Meta-Analysis in the Social Science: An Introduction,’ at the “Frontiers of Transition Economics” conference at Kyoto University in Kyoto, Japan. The workshop was sponsored by the Kyoto Institute of Economic Research.
Tom Stanley gave a meta-analysis seminar at the Institute of Economic Research at Hitotsubashi University in Tokyo in 2016.
Emily Kothe gave a seminar on ‘Conducting Meta-Analysis in R: A Practical Introduction’ on 30 May 2017 at Deakin University
Databases, publications, methods and working papers
The primary analytical tool used by DeLMAR is meta-regression analysis, the statistical analysis of all relevant and comparable research on a specific scientific issue or, in other words, it is quantitative research synthesis. We seek to estimate genuine empirical effects, to understand research heterogeneity and to correct specification and selection biases endemic in economics and the social sciences.