This is a draft as a contribution to a discussion to a response to a discussion in the Facebook Page Psychological Methods Discussion Group.
The reason regression analyses aren’t a useful tool to determine the relative relevance of each behavioral determinant has three components.
[ Note: this is a first draft, a preprint of a blog post so to speak 🙂 ]
A recent 72-author preprint proposed to recalibrate when we award the qualitative label ‘significant’ in research in psychology (and other fields) such that more evidence is required before that label is used. In other words, the paper proposes that researchers have to be a bit more certain of their case before proclaiming that they have found a new effect.
The paper met with resistance, and although any proposal for change usually is, what’s interesting is that in this case, the resistance came in part from researchers involved in Open Science (the umbrella term for the movement to mature science through openness, collaboration and accountability). Since these researchers often fight for improved research practices ‘at all costs’ this resistance seems odd.
Thus ensued the Alpha Wars.
[Image by Silver Blue, https://flickr.com/photos/cblue98/]
[These are some thoughts that I’ll eventually work into a paper, so it may be a bit rough/drafty]
Psychology is characterized by an interesting paradox. On the one hand, it’s a very popular topic. After all, everybody’s a person, and the most important influences in most people’s worlds are other people. Who doesn’t love learning about oneself, one’s loved ones, one’s boss, and the leaders of one’s country? People are endlessly complex, so psychology and psychological research provide a veritable fount of knowledge.
On the other hand, that complexity of the human psychology is tenaciously denied. It is almost as if that complexity is seen rather like a spiritual entity, safe to invoke whenever it’s convenient to stare in wonder at the awesome quirks of nature and never-ending weirdness of people, but blissfully disregarded whenever it it threatening or gets in the way of day-to-day activities.
[ UPDATE: a commentary based on this blog post has now been published in the Journal of Informetrics at http://www.sciencedirect.com/science/article/pii/S1751157717302365 ]
Recently a preprint was posted at ArXiv to explore the question “Can the Journal Impact Factor Be Used as a Criterion for the Selection of Junior Researchers?“. The abstract concludes as follows:
The results of the study indicate that the JIF (in its normalized variant) is able to discriminate between researchers who published papers later on with a citation impact above or below average in a field and publication year – not only in the short term, but also in the long term. However, the low to medium effect sizes of the results also indicate that the JIF (in its normalized variant) should not be used as the sole criterion for identifying later success: other criteria, such as the novelty and significance of the specific research, academic distinctions, and the reputation of previous institutions, should also be considered.
In this post, I aim to explain why this is wrong (and more, how following this recommendation may retard scientific progress) and I have a go at establishing a common sense framework for researcher selection that might work.
In deze korte post (korte link om te delen: https://pakjessigaretten.nl) wil ik uitleggen wat je moet doen op pakjes sigaretten. Ik leg kort uit waarom ik fel tegen angstaanjagende afbeeldingen en teksten ben; waarom ze zo populair zijn; en wat ik vind dat je wel op pakjes sigaretten moet zetten. (Haast? Ga gelijk naar de bottom line.)
[ This is a Dutch post, as it concerns a “study” by a Dutch TV channel, BNN ]
Op dinsdag 22 september 2015 kwamen er verontrustende berichten de wereld in:
Schokkend feitje nummer 1: 35% van die jongeren zegt meer drugs te gebruiken door de ophoging van de alcoholgrens! Damn.
[citatie van Spuiten en Slikken]
Bijna een derde van de jongeren gebruikt elke week drugs, en een derde doet maandelijks aan drugsgebruik.
[citatie van nu.nl]
Dit lijken ernstige signalen. Gelukkig blijkt bij nadere inspectie dat het onderzoek waar deze conclusies op gebaseerd worden, ongeschikt is om dit soort conclusies te trekken. Er zijn zes serieuze problemen met dit onderzoek: Continue reading “Een niet-representatieve steekproef zegt ja tegen MDMA”
Earlier (ok, in the only previous, first post on this blog) I discussed the recent study of Zachary Horne et al. (2015), where they concluded that threatening communication may be an effective approach to counter anti-vaccination attitudes. One of the problems with this study was that the manipulation was not valid: the conditions differed on many variables, any of which may explain the results they found.
After I deliberated for a while whether to inform the authors of the blog post, I decided to do so in the spirit of academic debate, transparency, and learning from each other. He swiftly replied, and one of the things he dis was correct my assumption that they did not share their data. They did actually share their data! I think that’s very commendable – I strongly believe that all researchers should Fully Disclose. Zachary posted it at the excellent (and free) Open Science Framework repository, specifically at http://osf.io/nx364. After having downloaded the data, I decided to write a brief follow-up post about matching of conditions and validity of manipulations. Continue reading “The importance of matching: a case study”
Recently, a number of media outlets enthusiastically reported that “Scare tactics may be the surest way to get parents to vaccinate their children“, suggesting to “Scare the crap out of [anti-vaccine parents]“, and happily claiming that “There’s a surprisingly simple way to convince vaccine skeptics to reconsider“.
Unfortunately, the study that these bold statements are based on, “Countering antivaccination attitudes” by Horne, Powell, Hummel and Holyoak and published in PNAS, suffers from a number of serious flaws. Continue reading “Countering antivaccination attitudes: don’t twiddle the dials before examining the engine”