How to select junior (or other) researchers, and why not to use Impact Factors

[ UPDATE: a commentary based on this blog post has now been published in the Journal of Informetrics at ]

Recently a preprint was posted at ArXiv to explore the question “Can the Journal Impact Factor Be Used as a Criterion for the Selection of Junior Researchers?“. The abstract concludes as follows:

The results of the study indicate that the JIF (in its normalized variant) is able to discriminate between researchers who published papers later on with a citation impact above or below average in a field and publication year – not only in the short term, but also in the long term. However, the low to medium effect sizes of the results also indicate that the JIF (in its normalized variant) should not be used as the sole criterion for identifying later success: other criteria, such as the novelty and significance of the specific research, academic distinctions, and the reputation of previous institutions, should also be considered.

In this post, I aim to explain why this is wrong (and more, how following this recommendation may retard scientific progress) and I have a go at establishing a common sense framework for researcher selection that might work.

Continue reading “How to select junior (or other) researchers, and why not to use Impact Factors”

Gezondheidscommunicatie op tabaksverpakking: angst is een slechte raadgever

In deze korte post (korte link om te delen: wil ik uitleggen wat je moet doen op pakjes sigaretten. Ik leg kort uit waarom ik fel tegen angstaanjagende afbeeldingen en teksten ben; waarom ze zo populair zijn; en wat ik vind dat je wel op pakjes sigaretten moet zetten. (Haast? Ga gelijk naar de bottom line.)

Continue reading “Gezondheidscommunicatie op tabaksverpakking: angst is een slechte raadgever”

Een niet-representatieve steekproef zegt ja tegen MDMA

[ This is a Dutch post, as it concerns a “study” by a Dutch TV channel, BNN ]

Op dinsdag 22 september 2015 kwamen er verontrustende berichten de wereld in:

Schokkend feitje nummer 1: 35% van die jongeren zegt meer drugs te gebruiken door de ophoging van de alcoholgrens! Damn.

[citatie van Spuiten en Slikken]

Bijna een derde van de jongeren gebruikt elke week drugs, en een derde doet maandelijks aan drugsgebruik.

[citatie van]

Dit lijken ernstige signalen. Gelukkig blijkt bij nadere inspectie dat het onderzoek waar deze conclusies op gebaseerd worden, ongeschikt is om dit soort conclusies te trekken. Er zijn zes serieuze problemen met dit onderzoek: Continue reading “Een niet-representatieve steekproef zegt ja tegen MDMA”

The importance of matching: a case study

Earlier (ok, in the only previous, first post on this blog) I discussed the recent study of Zachary Horne et al. (2015), where they concluded that threatening communication may be an effective approach to counter anti-vaccination attitudes. One of the problems with this study was that the manipulation was not valid: the conditions differed on many variables, any of which may explain the results they found.

After I deliberated for a while whether to inform the authors of the blog post, I decided to do so in the spirit of academic debate, transparency, and learning from each other. He swiftly replied, and one of the things he dis was correct my assumption that they did not share their data. They did actually share their data! I think that’s very commendable – I strongly believe that all researchers should Fully Disclose. Zachary posted it at the excellent (and free) Open Science Framework repository, specifically at After having downloaded the data, I decided to write a brief follow-up post about matching of conditions and validity of manipulations. Continue reading “The importance of matching: a case study”

Countering antivaccination attitudes: don’t twiddle the dials before examining the engine

Recently, a number of media outlets enthusiastically reported that “Scare tactics may be the surest way to get parents to vaccinate their children“, suggesting to “Scare the crap out of [anti-vaccine parents]“, and happily claiming that “There’s a surprisingly simple way to convince vaccine skeptics to reconsider“.

Unfortunately, the study that these bold statements are based on, “Countering antivaccination attitudes” by Horne, Powell, Hummel and Holyoak and published in PNAS, suffers from a number of serious flaws. Continue reading “Countering antivaccination attitudes: don’t twiddle the dials before examining the engine”