Wow, good question and points!!!
Based on a PsyArXiv preprint with the admittedly slightly provocative title “Why most experiments in psychology failed: sample sizes required for randomization to generate equivalent groups as a partial solution to the replication crisis” a modest debate erupted on Facebook (see here; you need to be in the PsychMAP group to access the link, though) and Twitter (see here, here, and here) regarding randomization.
John Myles White was nice enough to produce a blog post with an example of why Covariate-Based Diagnostics for Randomized Experiments are Often Misleading (check out his blog; he has other nice entries, e.g. about why you should always report confidence intervals over point estimates).
I completely agree with the example he provides (except that where he says ‘large, finite population of N people’ I assume he means ‘large, finite sample of N people drawn from an infinite population’). This is what puzzled me about the whole discussion. I agreed with (almost all) arguments provided; but only a minority of the arguments seemed to concern the paper. So either I’m still missing something, or, as Matt Moehr ventured, we’re talking about different things.
So, hoping to get to the bottom of this, I’ll also provide an example. It probably won’t be as fancy as John’s example, but I have to work with what I have 🙂
In statistics, one of the first distributions that one learns about is usually the normal distribution. Not only because it’s pretty, also because it’s ubiquitous.
In addition, the normal distribution is often the reference that is used when discussion other distributions: right skewed is skewed to the right compared to the normal distribution; when looking at kurtosis, a leptokurtic distribution is relatively spiky compared to the normal distribution: and unimodality is considered the norm, too.
There exist quantitative representations of skewness, kurtosis, and modality (the dip test), and each of these can be tested against a null hypothesis, where the null hypothesis is (almost) always that the skewness, kurtosis, or dip test value of the distribution is equal to that of a normal distribution.
In addition, some statistical tests require that the sampling distribution of the relevant statistic is approximately normal (e.g. the t-test), and some require an even more elusive assumption called multivariate normality.
Perhaps all these bit of knowledge mesh together in people’s minds, or perhaps there’s another explanation: but for some reason, many researchers and almost all students operate on the assumption that their data have to be normally distributed. If they are not, they often resort to, for example, converting their data into categorical variables or transforming the data.
[ primary audience: behavior change intervention developers ]
Threatening communication is a popular behavior change method used tobacco packaging, to promote seatbelt use and discourage substance use. However, much research also suggests that it is not the best weapon of choice when the goal is to really change behavior, or even when the goal is to raise awareness or educate people.
How is that paradox possible? This blog post will answer that question.
This post is a response to a post by Daniel Lakens, “One-sided tests: Efficient and Underused“, whom I greatly respect and, apparently up until now, always vehemently agreed with. So this post is partly an opportunity for him and others to explain where I’m wrong, so dear reader, if you would take this time to point that out, I would be most grateful. Alternatively, telling me I’m right is also very much appreciated of course 🙂 In any case, if you haven’t done so yet, please read Daniel’s post first (also, see below this post for an update with more links and the origin of this discussion).
In deze korte post (korte link om te delen: https://pakjessigaretten.nl) wil ik uitleggen wat je moet doen op pakjes sigaretten. Ik leg kort uit waarom ik fel tegen angstaanjagende afbeeldingen en teksten ben; waarom ze zo populair zijn; en wat ik vind dat je wel op pakjes sigaretten moet zetten. (Haast? Ga gelijk naar de bottom line.)
[ This is a Dutch post, as it concerns a “study” by a Dutch TV channel, BNN ]
Op dinsdag 22 september 2015 kwamen er verontrustende berichten de wereld in:
Schokkend feitje nummer 1: 35% van die jongeren zegt meer drugs te gebruiken door de ophoging van de alcoholgrens! Damn.
[citatie van Spuiten en Slikken]
Bijna een derde van de jongeren gebruikt elke week drugs, en een derde doet maandelijks aan drugsgebruik.
[citatie van nu.nl]
Dit lijken ernstige signalen. Gelukkig blijkt bij nadere inspectie dat het onderzoek waar deze conclusies op gebaseerd worden, ongeschikt is om dit soort conclusies te trekken. Er zijn zes serieuze problemen met dit onderzoek: Continue reading “Een niet-representatieve steekproef zegt ja tegen MDMA”
Earlier (ok, in the only previous, first post on this blog) I discussed the recent study of Zachary Horne et al. (2015), where they concluded that threatening communication may be an effective approach to counter anti-vaccination attitudes. One of the problems with this study was that the manipulation was not valid: the conditions differed on many variables, any of which may explain the results they found.
After I deliberated for a while whether to inform the authors of the blog post, I decided to do so in the spirit of academic debate, transparency, and learning from each other. He swiftly replied, and one of the things he dis was correct my assumption that they did not share their data. They did actually share their data! I think that’s very commendable – I strongly believe that all researchers should Fully Disclose. Zachary posted it at the excellent (and free) Open Science Framework repository, specifically at http://osf.io/nx364. After having downloaded the data, I decided to write a brief follow-up post about matching of conditions and validity of manipulations. Continue reading “The importance of matching: a case study”
Recently, a number of media outlets enthusiastically reported that “Scare tactics may be the surest way to get parents to vaccinate their children“, suggesting to “Scare the crap out of [anti-vaccine parents]“, and happily claiming that “There’s a surprisingly simple way to convince vaccine skeptics to reconsider“.
Unfortunately, the study that these bold statements are based on, “Countering antivaccination attitudes” by Horne, Powell, Hummel and Holyoak and published in PNAS, suffers from a number of serious flaws. Continue reading “Countering antivaccination attitudes: don’t twiddle the dials before examining the engine”