Over the past year, Helen Pluckrose, James Lindsay & Peter Boghossian published a series of hoax papers in humanities journals oriented towards the “critical study” of gender and sexuality. Their plot was discovered midway through, forcing them to go public prematurely.
In a journal-length essay explaining their methods, findings, and intentions, “Academic Grievance Studies and the Corruption of Scholarship,” the authors hold up their publications as proof that research on race, gender, sexuality – along with the discipline of sociology – are defined by groupthink, confirmation bias and motivated reasoning, which allow bogus work to regularly be published in these (sub)fields.
Dubbed “Sokal Squared,” or the “Grievance Studies hoax,” the incident has become something of a Rorschach Test within academic and media spaces: Those already disposed towards skepticism of “critical” scholarship on race, gender or sexuality view the incident as damning proof of a deep rot within these subfields, and perhaps with social research or academia more broadly. Those more sympathetic to “critical studies” instead see a deeply flawed and limited experiment — and accuse the authors of overstating their findings, speaking beyond their data and, intentionally or not, feeding into the agenda of right-wing reactionaries. There have been relatively few unexpected validators or critics.
The purpose of this essay is not to adjudicate between these rival perspectives, but to provide some critical context on the background of these types of experiments and this particular study, explore the relevant ethical considerations, and provide a concise description and analysis of what the trio found.
Background
NYU physicist Alan Sokal was disturbed and annoyed by postmodern social critiques and, especially, the (mis)appropriation of concepts from the “hard sciences” into social research. And so, in 1996, he crafted a paper which he viewed as not just weak, but essentially nonsensical: “Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity,” and managed to get it published Social Text – a journal which, while not peer-reviewed, regularly published essays by social criticism luminaries of the time. Shortly after it was published, Sokal penned an essay for the American literary magazine Lingua Franca exposing his original paper as a hoax; he then co-authored a book with fellow physicist Jean Bricmont, Impostures Intellectulles, expanding his critique of postmodernism, the misuse of scientific and mathematical concepts in social research, and the lack of intellectual rigor in “critical studies.”
This was a significant departure from the standard operating procedure:
Within the social sciences, there was already a well-established tradition of sending problematic papers up for peer review, with false attribution of authorship, in order to better understand how editors exercise discretion in selecting and evaluating work. This line of research dates back (at least) to Abramowitz, Gomes & Abramowitz’s 1975 paper in the Journal of Applied Social Psychology, “Publish or Politic: Referee Bias in Manuscript Review.”
Most of these have included a control group, tested a fairly large number of papers (typically submitted to journals in the authors’ own fields), and – because they were merely testing whether editors would accept or reject the works – researchers did not allow accepted fake papers to proceed to publication, so as to avoid polluting the knowledge environment. Scholars have also tended to be fairly circumspect in detailing the findings and implications of their studies.
In contrast, Sokal wrote a single paper, submitted to a single journal. He was an outsider to the field he was testing. The paper was allowed to proceed all the way to publication, forcing the journal to retract it when the hoax was exposed. Sokal argued that the fact his piece was published suggests a deep problem with many “critical studies” fields:
” The results of my little experiment demonstrate, at the very least, that some fashionable sectors of the American academic Left have been getting intellectually lazy… . The targets of my critique have by now become a self-perpetuating academic subculture that typically ignores (or disdains) reasoned criticism from the outside. In such a situation, a more direct demonstration of the subculture’s intellectual standards was required. But how can one show that the emperor has no clothes? Satire is by far the best weapon; and the blow that can’t be brushed off is the one that’s self-inflicted. I offered the Social Text editors an opportunity to demonstrate their intellectual rigor. Did they meet the test? I don’t think so.”
As a result of this novelty, and the significance of the journal involved, the incident became a major story in mainstream media outlets in France, the U.S. and beyond. This, in turn, prompted robust criticism of the hoax from Jacques Derrida, Bruno Latour, Gabriel Stolzenberg and others – who argued that the conclusions Sokal was drawing were far too wide given he only published one single (non-peer reviewed) essay; critics also argued that Sokal seemed to be misunderstanding, misrepresenting, and/or uncharitably reading most of the works he (and Bricmont) singled out.
A year after the hoax, reflecting on these criticisms, and with the perspective distance provides, and seeing how other had been using the hoax to make even stronger claims than he was comfortable with, Sokal published an essay striking a more moderate tone:
“From the mere fact of publication of my parody I think that not much can be deduced. It doesn’t prove that the whole field of cultural studies, or cultural studies of science — much less sociology of science — is nonsense. Nor does it prove that the intellectual standards in these fields are generally lax. (This might be the case, but it would have to be established on other grounds.) It proves only that the editors of _one_ rather marginal journal were derelict in their intellectual duty, by publishing an article on quantum physics that they admit they could not understand, without bothering to get an opinion from anyone knowledgeable in quantum physics, solely because it came from a ‘conveniently credentialed ally’ (as Social Text co-editor Bruce Robbins later candidly admitted), flattered the editors’ ideological preconceptions, and attacked their ‘enemies’.”
Ten years later, Sokal wrote a book, Beyond the Hoax: Science, Philosophy and Culture, drawing out themes from his essays on these topics in the intervening years, reflecting on the “social life” of his experiment, and trying to evaluate its significance in retrospect. Here, he again emphasized that his initial criticisms may have been overly dismissive, and perhaps even insufficiently informed. He had come to believe that much of the work he and Bricmont were criticizing at the time was not outright wrong or nonsensical – instead, the dispute turned on deep (even reasonable) disagreements about the nature of scientific knowledge.
However, in that same book, Sokal highlighted at length the ways in which bad actors — from industry executives to mass media outlets to cynical politicians — have re-appropriated tenets of postmodernism in order to render the population more vulnerable to propaganda and advertisements, and less likely to believe expert criticism of the state or corporations. As a result, in many respects, critical theory may be accomplishing the opposite of its stated purpose (a point previously emphasized by Sokal’s critic, Bruno Latour, in his 2004 Critical Inquiry essay “Why Has Critique Run Out of Steam? From Matters of Fact to Matters of Concern”).
Notwithstanding Sokal’s own evolution on these issues, his 1996 hoax reenergized criticism, skepticism and resistance towards critical theory – particularly within the field of philosophy. In 1997, the journal Philosophy and Literature initiated a “Bad Writing Contest,” highlighting the most obtuse or meaningless selections of the year – generally targeting critical theorists. The contest achieved peak notoriety in 1999 when Judith Butler took the top prize (she was less than amused).
In his 1998 book, Achieving Our Country: Leftist Thought in Twentieth-Century America, pragmatist Richard Rorty urged social scientists and activists to kick their philosophy habit (hat tip to Conor Friedersdorf):
“Recent attempts to subvert social institutions by problematizing concepts have produced a few very good books. They have also produced many thousands of books which represent scholastic philosophizing at its worst. The authors of these purportedly ‘subversive’ books honestly believe that they are serving human liberty. But it is almost impossible to clamber back down from their books to a level of abstraction on which one might discuss the merits of a law, a treaty, a candidate, or a political strategy.
Even though what these authors ‘theorize’ is often something very concrete and near at hand—a current TV show, a media celebrity, a recent scandal—they offer the most abstract and barren explanations imaginable. These futile attempts to philosophize one’s way into political relevance are a symptom of what happens when a Left retreats from activism and adopts a spectatorial approach to the problems of its country” (pp. 93-4).
Incidentally, in that same work, twenty years before Plukrose, Lindsay & Boghossian (or John McWhorter before them), Rorty also noted the apparent religiosity of critical theorists:
“The Ubiquity of Foucaldian power is reminiscent of the ubiquity of Satan, and thus of the ubiquity of original sin—that diabolical stain on every human soul… in committing itself to what it calls ‘theory,’ this Left has gotten something which is entirely too much like religion. For the cultural Left has come to believe that we must place our country within a theoretical frame of reference, situate it within a vast quasi-cosmological perspective.
What stories about blue-eyed devils are to Black Muslims, stories about hegemony and power are to many cultural Leftists… To step into the intellectual world which some of these Leftists inhabit is to move out of a world in which the citizens of a democracy can join forces to resist sadism and selfishness into a Gothic world in which democratic politics has become a farce… in which all the daylight cheerfulness of Whitmanesque hyper-secularism has been lost, and ‘liberalism’ and ‘humanism’ are synonyms for naiveté—for an inability to grasp the full horror of our situation” (pp. 95-6).
There were many more attempts at Sokal-style hoaxes attempting to delegitimize certain lines of scholarship within the discipline. Perhaps the most prominent of these occurred right around the 20-year anniversary of the Sokal Affair: philosophers Phillipe Huneman & Anouk Barberousse pseudonymously published what they perceived to be a nonsensical article in the Journal of Badiou Studies: “Ontology, Neutrality and the Strive for (non-) Being-Queer.” The authors claimed they perpetrated the hoax in order to help dispel the cult around prominent philosopher Alain Badiou (for whom the journal was named). In their view, many credit him a genius because they like his politics, but understand or engage little with his metaphysics – the primary subject of most of his work – which the hoaxers viewed as subpar. Philosopher Justin E.H. Smith (University of Paris 7 – Denis Diderot) wrote in response:
“But what sort of victory is it? Here my laughter on first reading Huneman and Barberousse’s text quickly gave way to two concerns. One is that the joke is not so much on the abstruse theory-heads, as had surely been the case in the Sokal incident. The joke isn’t on anyone who is committed to any particular ideology or style of thinking at all. The joke is, rather, on the folks running these pop-up online journals with their ludicrously low editorial standards.
Remarkably, the editors of the Journal of Badiou Studies even admitted as much when they complained of Huneman and Barberousse’s ‘dated’ method of attack ‘in an age when the pressures on independent Open Access publishing include underfunding and time-pressured staff.’ In other words, the editors effectively confess that they do not have the resources to produce a decent journal on their own, and so must rely on the good will of the contributors to not send them crap.
But many people who submit to journals are not in a position to know of their own work whether it is crap or not, and for this reason alone a journal that does not have the resources to weed out crap would be doing scholarship a far greater service by simply not existing. The problem is compounded in the context of continental European publishing in English, where often, at every stage of production, from writing to proofreading to publishing, all of the people involved speak English, at best, as a second language. What slipped through at the Journal of Badiou Studies does not in fact look so different from what slips through on a regular basis at Springer or Brill.”
Notwithstanding these concerns, the next year Portland State philosopher Peter Boghossian and mathematician James Lindsay pseudonymously published “The Conceptual Penis as a Social Construct” in the journal Cogent Social Sciences; they revealed their hoax in Skeptic magazine alongside a lengthy critique of gender studies.
While many who were already critical of gender studies embraced the hoax as further evidence of what they already believed, the duo were also met with widespread criticism (including from Sokal himself!) – most trenchantly for their decision to target an obscure pay-to-play journal that lacked specialists from the field they were targeting. In other words, they could not confidently assert that their work was actually representative of the literature they were trying to critique – and as a result, the strong claims they made about gender & sexuality studies on the basis of the hoax seemed to be ill-substantiated.
Humanities scholar Helen Pluckrose rose to the defense of Boghossian & Lindsay in the online magazine she oversees (Areo). Even prior to the “Conceptual Penis” controversy, James Lindsay was a semi-regular contributor to this publication — and ultimately served as the bridge between Pluckrose and his Cogent Studies co-conspirator, Boghossian. After some discussion, the three of them decided to work together to replicate the hoax in a way that would render many of the criticisms of the “Conceptual Penis” paper moot:
They chose to target well-regarded peer-review journals specializing in the critical study of gender, sexuality… and race. Of course, a single published paper — even if placed in a high-quality specialized journal – could still be written off as non-representative: an artifact of a single set of bad reviewers. So the trio attempted to compose and place dozens of papers at well-respected journals that focused on gender, sexuality and/or race.
Yet, ironically, their hoax was prematurely exposed by others who were also highly critical of the “critical studies” approach to race, gender and sexuality. It began when Robby Soave published an article in Reason arguing that their essay on canine rape culture (“Human reactions to rape culture and queer performativity at urban dog parks in Portland, Oregon”) was so absurd he hoped it was a Sokal-style hoax – although it didn’t seem to be one.
This drew the attention of Toni Ariaksinen, who tried to identify and contact others at the Portland Ungendering Research Institute, which the author claimed to be affiliated with, as background for a Campus Reform story on the paper. She could not find any other scholars affiliated with the “institute,” and the website seemed to be thrown up just prior to the paper’s publication. She reached out to the handful of programs offering a PhD in feminist studies (which the author claimed to hold) — and none could confirm “Helen Wilson” had ever been a student there. And so, in July 2018 Ariaksinen published her article — arguing that the “dog rape culture” essay was either a hoax, or its author had lied about her credentials and affiliations. The essay was subsequently flagged by the journal it was published in (Gender, Place and Culture), pending an investigation into its authorship.
This piqued the interest of Wall Street Journal reporter Jillian Kay Melchior, who contacted “Helen Wilson” and conveyed that she planned to write a major story about the incident, and the possibility the paper was a hoax. The trio realized the jig was up and “came clean to the Wall Street Journal at the beginning of August and began preparing a summary as quickly as possible” despite the fact they “still had several papers progressing encouragingly through the review process.” In turn, WSJ apparently agreed to hold off on publishing the story, in order to allow the trio time to get their summary together.
Ethics
In principle, the “right” way to demonstrate flaws within a given literature would be to single out several prominent examples (works that were published in major journals and/or are highly-cited, especially from prominent scholars within that subfield) and show that they are deeply flawed. Ideally, the errors would seem to point in the same general direction, suggesting a systemic issue with published research in the subfield. These criticisms would, themselves, be subjected to the peer-review process and published in a scholarly journal – ideally, a journal within the subfield under criticism (rendering it more likely that the critique will reach the people who need to hear it most, and helping to ensure that the criticisms of the subfield are well-informed and fair-minded).
There seemed to be several compelling reasons why the authors did not go about criticizing “grievance studies” in the “right way.” Among them:
- The authors seemed to believe that many of the claims, and much of the evidence, presented in “critical studies” literature is so vague, esoteric or abstract that it cannot even be said to be “wrong.” Consequently, it becomes difficult to point out errors – not because the arguments are correct, but because there is so little “there” to refute.
- The authors seemed frustrated by the fact that, often, when outsiders do try to refute “critical studies” scholarship, these critiques are largely waved away on the grounds that their interlocutors just don’t “get it.”
This kind of move becomes much more difficult when the critics have published multiple studies in respected peer-reviewed journals in the subfield, been asked to serve as peer-reviewers for other journals in the subfield on the basis of their contributions / demonstrated expertise, and even won awards for their research in the subfield (the “dog rape culture” essay won a prize from the journal it was published in). Indeed, it only makes the problem worse if the authors truly don’t “get it” but still managed to achieve these feats. Hence the utility behind allowing the fake papers to proceed all the way through publication. - The process of identifying the ideal journal, submitting a paper, then having it reviewed, edited and published, typically takes months (sometimes even years). Given that their plot was discovered midway through the experiment, and they were about to be “scooped” on their own story (and lose control over the narrative) – they could not wait to publish their report in a peer-reviewed journal.
As an aside, given that most published academic research are never cited and barely read, it is likely that their work would have reached a far smaller audience/ made much less impact had it been published in a peer-reviewed academic journal rather than Areo. - Plurkrose studies literature and philosophy (epistemology); Boghossian is a philosopher; Lindsay is a mathematician trained in physics (similar to Sokal). Hence, one reason they seemed to lack the background or inclination to follow the same protocols as social scientists may be that… they are not, in fact, social scientists.
However, even granting these caveats, a number of other significant ethical issues remain.
For instance, the authors misrepresented their identities, credentials and affiliations, put forward bad-faith arguments and (perhaps more troubling) phony data – and allowed these to actually be published, and to stand for some time. If their plot had not been exposed prematurely, many other researchers could have been led (even further) astray through other works that cited the trio’s research, believing it to be reliable (in part because it had been published in these respected journals – and therefore, presumably vetted by authorities in the subfield).
It would also be very bad if this kind of hoaxing became a widespread practice (for instance, if the often well-credentialed contributors to Areo, Quillette and other sympathetic outlets decided to also flood social research journals with fake studies, allow accepted papers to proceed all the way through publication, and expose them as fakes, making sweeping claims against the relevant fields in the process) – this would greatly undermine the fidelity of peer review or citation, and public trust in social research. In Kantian terms, the trio’s actions could not be successfully scaled into a categorical imperative.
There are also financial considerations. Coming up with, researching for, writing, editing and placing such a large number of academic articles in such a short time became something like a full-time job over the course of the experiment. The trio received significant financial support from an unknown benefactor, allowing them to waive many of their other commitments and dedicate themselves fully to this project. Typically, research funding is disclosed — both at the time of article submission and (especially) at publication. This is important because, even outside of direct pressure or oversight from donors, a desire to “stay on their good side” or “demonstrate value” can influence how data are collected, interpreted and presented in a host of ways, both subtle and obvious (intentionally or not). Disclosure helps editors and readers better understand and account for these possible sources of bias — and therefore, more effectively evaluate research.
Pluckrose, Lindsay & Boghossian have asserted that their donors wished to remain anonymous out of fear they would be smeared as racist, sexist, etc. for being associated with the project — and face social and financial sanctions as a result. This seems like a legitimate concern given how the authors themselves have been depicted in many corners of academia and the media for carrying out the hoax. As the Wall Street Journal reports:
Mr. Boghossian doesn’t have tenure and expects the university will fire or otherwise punish him. Ms. Pluckrose predicts she’ll have a hard time getting accepted to a doctoral program. Mr. Lindsay said he expects to become ‘an academic pariah,’ barred from professorships or publications.”
And to be clear: the “dark money” associated with the project provides no reason to write it off out of hand; after all, funding the authors received could not plausibly have had any significant or direct influence on what they were most keen to measure: reviewers’ evaluation of their presented arguments and evidence (although, depending on the donor, the journal editor might have been more inclined to ultimately reject certain essays due to apparent conflicts-of-interest were the funding disclosed). Yet, significant anonymous funding does open the door to legitimate scrutiny about the motivations of the donors and authors, and how those might have informed the targets they chose, and the ways they opted to interpret and present their results (again, it does not in any way establish that there actually were ulterior motives or biases on the part of the authors, it merely elevates that possibility). And to her credit, Pluckrose recognizes this reality:
I do understand being concerned about funding to some extent but ultimately backers have the right to privacy, there is a need for some trust & there is no way this could have influenced journals’ acceptance, the existing scholarship we cited or the reviewer comments.
— Helen Pluckrose (@HPluckrose) October 11, 2018
Another concern is that the “Sokal Squared” hoax, much like its “Conceptual Penis” predecessor, provides plenty of “red meat” for reactionaries who want to discredit the left — or work on gender, race or sexuality – or even social research more broadly. This was a consequence the authors anticipated in advance (as their essay and interviews allude to); yet some have argued they did not do enough to hedge against this possibility (in their view the trio framed their study and described its findings in an irresponsible manner), or to push back against misuses. As a result, good research on gender, sexuality, race or inequality could find itself discredited or dismissed alongside the bad (to which the authors would likely reply, “this is already happening irrespective of the hoax – all the more reason for social research fields to better identify and purge bad scholarship”).
Some have alleged that the trio seem to be “punching down” by targeting lines of scholarship that are already marginal within social research fields – and which also happen to be oriented towards empowering historically disadvantaged and marginalized groups (in principle).
The authors have addressed this charge at length, arguing:
- “Critical studies” of race, gender or sexuality have had a huge impact – both within the academy and in the broader culture – particularly within elite academic and media circles, and among left-leaning activists.
- The activist-orientation of “critical studies” may provide a principled reason to be suspicious of their intellectual rigor. Attempting to promote a particular outcome, narrative or ideology seems, prima facie, incompatible with following the truth wherever it leads. Hence, there is good reason to suspect that these subfields may be more prone to errors than others, which warrants their being singled-out.
- The authors claim they are not out to discredit research on gender, sexuality, or race. On the contrary: precisely because these issues are so contentious and important, it is essential for research on these topics to be as objective, rigorous and precise as possible. Put another way: the more important these lines of research are held to be, the more necessary it becomes for someone to push back on bad scholarship within them.
One could add that it is not unheard of for “critical studies” scholars to pull this kind of hoax on one-another. For instance (as reported by The Guardian), pseudonymous authors published a “study” in the journal Totalitarianism and Democracy “proving” that the dogs East German border guards had used to patrol the Berlin Wall were direct descendants of those Nazis relied on in two of their concentration camps – thereby maintaining a “tradition of violence” among these animals and their descendants. The authors claimed to have perpetrated the hoax in order to discredit the “animal turn” in critical scholarship.
Hence, Pluckrose, Lindsay & Boghossian seem to be on solid ground for many of the moral objections people have raised with regards to their study – for others, apparently less so. Yet, as Bret Weinstein argued, whatever ethical concerns people may have over how the experiment was carried out and presented – the moral “rightness” of the authors is a separate question from “what does the experiment reveal?”
It is to this question we now turn.
Findings
By the time the experiment was called to a halt, the trio produced 20 papers overall. 7 were accepted for publication (4 already published online) in well-regarded, peer-reviewed, academic journals — specializing in the critical study of gender and sexuality. 4 other papers received a R&R ruling, but were unable to see the revisions completed and/or approved before they had to pull the plug. 1 paper was still under review for the first time when they had to go public.
Many of the non-hoax papers published in these journals seem to be of no higher quality than the accepted papers Pluckrose, Lindsay & Boghossian produced as satire; the principal difference seems to be the intent of the authors. Yet, on their account, an author’s motivations are immaterial to the soundness of their arguments or the objective reality researchers should be trying to describe. Within these subfields, if the only things distinguishing garbage essays from “legitimate” scholarship are “irrelevant” considerations like an author’s intent or identity (given peer-review is supposed to be blind, with works accepted or rejected on their merits alone) – then there is no meaningful distinction to be drawn between the “garbage” essays and the “legitimate” ones.
At least, that’s the “armchair” argument. However, the reality of the situation seems to be more complex:
In clips from the documentary on their project (included in the Areo write-up), the authors reveal that the originally tried to publish lazy, meaningless junk that merely evoked key buzzwords and seemed oriented towards “social justice.” All of these initial attempts were rejected out of hand. In order to produce successful papers they had to really immerse themselves in the literature, gain a deeper understanding of the arguments being made and the conversations being had, develop fluency in the language – and for many of their articles, conduct extensive research on the subject-matter in question. As Daniel Drezner put it:
“Reading this paper reminded me of a short story I read as a kid about a group of high-schoolers who thought they could program a computer to do a lot of their homework. When they were found out, the principal didn’t punish them, because he noted that to pull this stunt off, the students had to master the subject well enough to program the computer. Which meant that they were doing just as much work as the students who simply did the homework. When the authors of this paper acknowledged it took them several months of failure to learn how to craft a paper that would merit being sent out for peer review, I wondered if they had read that story and recognized the plot.”
Indeed, even after they had gained enough skill to virtually eliminate desk rejections, a majority of the trio’s submissions were still ultimately turned away or offered to be reconsidered after revisions.
Moreover, their purported empirical studies (with fake data) were more than twice as likely to be accepted for publication as their non-empirical papers. Of course, many of the experimental designs and reported results were obviously flawed, yet under-scrutinized — nonetheless, there was clearly a preference for claims that were purportedly backed up by observation and data over those justified on purely rhetorical grounds or by appeals to other research.
This suggests that if they had produced a set of “control” papers (in this case, essays drawing on the same frameworks, but with arguments the authors believed to be “true” + actual, accurate data), it is likely that the null hypothesis would fail. There would be significant differences between the treatment and control – particularly: as they increased the quality of the work they produced, their essays would grow increasingly likely to be published (especially in more prestigious and/or selective journals). Of course, this is precisely the way things should be.
Hence, the process may be troubled in these fields, but it is not totally broken. It does not seem to be the case that anything can be published “so long as it falls within the moral orthodoxy and demonstrates understanding of the existing literature.” Indeed, many (most) submitted papers to respected journals within these subfields likely
a) draw on the “critical studies” frameworks
b) cite relevant literature from the subfield
c) affirm the editors’ preferred narratives
Yet are declined on the basis of their perceived (lack of) quality or contribution to the field.
Update: In responding to questions on Twitter, James Lindsay has now provided information on the total number of attempts they made.
All said, the trio made 48 attempts (in 29 journals). Therefore, their “hit rate” was just under 1 acceptance or R&R for every four paper submissions (23%), and they ended up placing something (or getting an R&R) for 1 out of every 3 journals they selected (34%).
Yet the flip side to these stats is that — even in these interdisciplinary, ideological and advocacy-oriented subfields — the process actually did work to filter out junk more than three-quarters of the time (in fact, it shoots up even higher if we view their 4 R&Rs as signals of falling below the journal standards in their initial attempt).
One could argue that it is still a major problem if junk can get through peer review in these journals more than 1 out of every 5 attempts. Indeed, most (58%) of the papers they produced were eventually accepted somewhere, or were on track to be accepted (R&R) at the time the experiment was called to a halt.
Yet, the claim by the authors — and many other critics of these subfields — is that “just about anything can be made to work, so long as it falls within the moral orthodoxy and demonstrates understanding of the existing literature” (emphasis theirs). This does not seem to be borne out in their results: 77% of their submissions were rebuffed, they failed to publish even one single essay in 66% of the journals they submitted to, and 42% of the papers they produced were never granted so much as an R&R, often despite multiple publication attempts.
II.
The authors repeatedly claim that their study “proves” endemic flaws in research on gender and sexuality plus race and sociology – as a result, so have many in the media. Yet, of these, the trio only managed to successfully place papers on gender and sexuality – all seven of their accepted papers were in these subfields. And, as noted in the New York Times, “Most were interdisciplinary journals in highly niche fields, where there is less agreement about acceptable methodologies and the standards of peer review.”
None of their papers on race were actually accepted for publication by the time the experiment ended (and actually, none of their essays were centrally “about” race in the same way they were “about” gender and/or sexuality); nothing was accepted in race studies journals, nor did they publish a single essay in a peer-reviewed sociology journal. As UCLA sociologist Gabriel Rossman put it:
More specifically, sociology journals consistently rejected the hoax papers (6/6) but gender/cultural studies accepted or gave R&R to most of them (11/15). This isn’t an academia problem, or even a social science problem, it’s a problem with fields that end in “studies.”
— Rogue Works Progress Administration (@GabrielRossman) October 3, 2018
It’s certainly a possibility that the problem is not as severe in sociology, or that it may be less severe in race scholarship than gender scholarship – as the authors’ relative “hit rates” might suggest.
However, it also seems plausible that the authors were far more familiar with the gender & sexuality landscape than they were with contemporary work on race or in sociology — and the differential success rates may be primarily due to the fact that the trio didn’t quite “nail” how to authoritatively speak on race, or (especially) as sociologists – NOT because sociological or race research is actually more rigorous on average than work on gender and/or sexuality.
After all, Pluckrose studies literature by and about women; Boghossian & Lindsay’s “Conceptual Penis” hoax targeted scholarship on gender and sexuality. “Critical studies” basically sits at the intersection between the chosen fields of Pluckrose (literature) and Boghossian (philosophy). The genre tends to demand far less empirical evidence – and evaluates/ utilizes said evidence in a different manner than mainstream contemporary sociology (especially in the United States).
None of them seem to have any significant background on race scholarship or sociological research. Perhaps they believed they could easily expand into sociology because they understood it to be the discipline of Michel Foucault or Jean Baudrillard, whose work both heavily informed and was informed by “critical studies.” In reality, the field strongly favors quantitative work – or at best, mixed methods (this is especially true for the top journals in the field). Indeed, even contemporary ethnographic research tends to deploy quantitative logical and rhetorical devices.
As Andrew Gelman, Nassim Nicholas Taleb and many others (e.g. here, here, here) have pointed out, a good deal of work in fields like sociology, economics or political science is also riddled with ideologically-driven and erroneous research – the main difference being that *we* dress up our B.S. in statistical modeling and analysis rather than the kind of esoteric language deployed in journals like Hypatia (giving it an air of relative respectability and objectivity). Indeed, it has been a consistent theme of both my academic and public-facing work to highlight abuses of statistics promoting ideological narratives and/or political agendas – from my first journal essay (“Syria Contextualized: The Numbers Game”) through my recent paper exploring the burgeoning field of “Trump studies.”
In short: the authors were largely “barking up the wrong tree” insofar as they attempted to publish non-quantitative (and especially non-empirical) junk research in sociology journals. It is not clear what would have happened if their work had better mirrored disciplinary norms within the field. Nonetheless, Pluckrose, Lindsay & Boghossian are speaking beyond the data insofar as they implicate race studies or (especially) sociology on the basis of their successful hoaxes. They did not successfully publish any work in these (sub)fields. Yet the fact that their assertions about sociology or race scholarship are insufficiently justified by the evidence they marshal does not mean their assertions are actually wrong.
Maybe the hoax does suggest extraordinary systemic problems within gender and sexuality studies; if so, maybe work on race — or activism-oriented disciplines like sociology — would be susceptible to similar levels of error. Then again, perhaps the researchers are overstating their case against the gender and sexuality subfields — just as they seemed to have done with regards to work on race, or the discipline of sociology. Or perhaps there is a substantive difference between these (sub)fields, and it just doesn’t make sense to lump race and sociological research in with work on gender and sexuality under the pejorative title “grievance studies.” On the other hand, perhaps work in more “respectable” fields like political science or economics would fare little better if subjected to a comparable level of uncharitable scrutiny as research focused on race, gender and sexuality.
‘Hard science’ fields may not fare much better either, for that matter. A 2013 study published in Science, journalist John Bohannon submitted junk papers to open access STEM journals and found that, of the papers that had received a verdict by the time the experiment ran its course, more than 61% were accepted for publication despite blatant errors and red flags. Subsequent audits of medical science journals have found that papers that make literally no empirical, methodological or theoretical contribution at all are often published in top journals if published by prestigious scholars (who can drive citations and improve the impact factor of the journals).
It may be that what the hoax doesn’t indicate much of anything about ‘studies’ fields, but rather reflects that peer review across the board is not great at catching errors and identifying high-quality work, regardless of field.
Across the board, the “grievance studies” hoax raises far more questions than it effectively answers. But these are important questions for social researchers to wrestle with in earnest — rather than dismissing them out of hand, or reflexively adopting a position in line with one’s “tribe” (as many have seemed inclined to do).
1/7/2019 Update: The authors’ concerns about possible retaliation seem to be well-founded. Dr. Boghossian is currently under investigation by his university’s institutional review board for having carried out experiments on humans (journal editors) without IRB approval, and for falsifying data. For the record, here is my own view on the matter, as conveyed to Reason:
“Peter Boghossian is a philosopher, not a social scientist. There is a long tradition within the field of philosophy in carrying out hoaxes like these. They virtually never involve IRB approval.
None of Boghossian’s collaborators were social scientists either. Ms. Pluckrose is a humanities scholar gearing up for grad school, and Mr. Lindsay is a mathematician and physicist by training. It is likely that none of them are well-versed in IRB requirements and protocols. In short, to the extent that normal social science protocols were not met, it was likely an honest mistake rather than a willful violation. Nor is likely that there was serious harm inflicted upon any of the ‘human subjects’ involved. Hence, I can see no reason why Dr. Boghossian’s career should be menaced for taking part in the hoax.
And I’ll add in closing, given the bias in various parts of the standard research process (which is what their experiment was about!) it seems highly plausible that *had* they followed standard protocol, the IRB board would have rejected their proposal for political/ ideological reasons.”
3/21/2021 Update: For a great breakdown of the ‘Grievance Studies’ hoax paper that purported to be ‘Mein Kampf’ just tweaked to replace Nazi stuff with feminist stuff, see this new essay in Haaretz. In a nutshell, it demonstrates that the trio consistently and systematically misrepresented this paper in public discussions about the hoax. In fact, it does not closely track Mein Kampf with just a few substitutions. Instead, they modified the text to a degree that it would not even be recognizable as Mein Kampf, even to a Hitler scholar (the essay is written by an historian who specializes on Hitler, Nazis and their social movement). Nor did they choose a selection of the text that contained Hitler’s more provocative ideas. Indeed, they actively avoided those, and chose a section of the work that was “one of the least ideological and racist parts” of the book, focused primarily on logistical questions of how the “newly refounded NSDAP should “rebuild as a party and amplify its program.”
Readers of the preceding summary here will not be surprised to know that the trio seem to have misrepresented or overstated their claims on this topic. But it is fruitful to see this laid out in a concrete case by someone who is an expert on the Nazis.