The Rationalism Video I'll Never Make
So, my $10+ patrons voted on this month's topic: "videos I'll never make and why." SO my plan was to record a casual ramble covering a few different subjects. And then I realized it would take me literally ninety minutes to only talk about one of them. ;;;
For LITERALLY YEARS I've had notes for a video about LessWrong, Eliezer Yudkowsky, effective altruism, and the rationalist community, in terms of critiques about why I feel as though their epistemology doesn't hold up, why their predictions have been wrong, and why in my view the whole thing functions as a pipeline to politics I find troubling.
This isn't that video, exactly. This is me talking through my notes with you without fancy visuals or algorithm pushing it to strangers. I'm hoping we can have a conversation about a project I've been carrying for years and why at this point, I'm pretty confident the "real" version can't exist.
We'll cover: Harry Potter fanfiction as recruitment material, why Bayesian reasoning isn't what they think it is, what actually happened with AI predictions, and why the discourse around this community has become impossible to navigate responsibly. I hope you enjoy!
Comments (34)
I loved the stats paper you cited critiquing Bayesian statistics as data falsification. I have to admit that while I dislike a lot of what Yudkowsky says, I did once think Bayesian reasoning generally made sense. Regarding the Harry Potter fanfiction of it all, something I find interesting about HP fanfics is how often they rewrite Harry into a completely different character. Ron, Hermione, and pretty much everyone else might be made better or worse characters for the sake of a given story but they still generally maintain the same basic characterization. The specific thing I find interesting is that a lot of stories that pair off Harry and Hermione make him a stronger, smarter, and just plain better person in the process.
This is a weird position to come from, but I had never heard of Aella until I saw her on [your friend’s podcast] (I don’t want to put that person on blast, but I’m assuming you know who). It was genuinely one of the most interesting conversations I’ve ever seen in my life; I thought she was fascinating and generally really thoughtful. The rationalism stuff (which she only mentioned briefly, from what I remember) seemed wrong to me. But on a process level, the way she and your friend interacted was FASCINATING and incredibly engaging. And maybe it was more engaging from a process level than a content level; I don’t remember well enough to know. But I remember I was quite enamored with how her mind worked from the conversation. I know I didn’t come away believing in any tenets of rationalism or being more open to any of the problematic ideologies you discuss here. Honestly, I’ll have to watch it again knowing more about her public approach and positions. I do like to try to understand more about people whose beliefs I radically disagree with whenever possible, but it’s also a very interesting disconnect between your takes on her (and my own takes on the statements/beliefs of hers you shared here), and my experience of actually watching her talk.
I totally understand and respect not making this a main channel video but this is really good! I've listened to it twice and I probably will again sometimes when I'm playing video games.
Saw some comments glazing certain EA personalities here so I'd like to say that from what I've read of Kelsey Piper, she's a typical ivory tower techno centrist lib who's been hanging out with some of the worst people in the EA sphere and supporting racist and misogynist shit for a long time. She's also doing evangelist work for AI now. Re the 2024 election, she wrote" Most republicans were (rightly) furious at the Biden presidency + DEI + immigration + inflation and think the US government needs aggressive immediate change, but they are not sociopaths and the scocipaths are learning that no, mass death still isnb't cool and fun". Like both extremely wrong, extremely cruel, and extremely helpful for fascist and right-wingers. You cut a centrist and a fascist bleed. Every time. We're in January 2026 now. With what's going on around the world, is this the kind of people we wanna glaze?
Considering Kelsey Piper wrote pro eugenics dnd fanfic with Yud, I think it's pretty wild that people are trying to glaze her.
Introducing Scott Alexander by highlighting his racism is the only way to do it, kudos. We've gained nothing by pretending that there's more to him than a machine for popularizing racism and misogyny so I'm glad that's all you had to say about him.
while "rigged" works as a generic word for "not fair", the equivalent to "loaded die" is usually "weighted coin"
Man, did I pick a good month to become a patron! I have to ask, have you (or anyone else on here) read Neoreaction a Basilisk by Elizabeth Sandifer? I think it’s a must-read, and pretty unique, critique of the rationalist community that covers some of the same ground in this video, albeit often from a bit of a different perspective.
Regarding Antropic as someone who isn't familiar with it or Claude but is curious, what is the main separator between it and OpenAI / ChatGPT?
Neither company is ethically flawless, but IMO there are some meaningful differences: Anthropic was founded by people who left OpenAI because they were worried about safety being deprioritized for commercial speed. In 2024, OpenAI disbanded its Superalignment team after both leaders quit; one was hired by Anthropic. & structurally, Anthropic was set up as a public benefit corporation from day one, which legally requires balancing profit with public good. On the product side, Anthropic developed Constitutional AI, which means Claude is trained against an explicit set of ethical principles that are publicly available, rather than just vibes and human feedback. They recently published an 80-page document explaining Claude's values and reasoning, which in my view is unusual transparency for the industry. It's a pretty reasonable document in comparison to the ChatGPT stuff. Which is not to like, glaze them as a company - e.g they've taken on military contracts. My point is rather to contrast them with OpenAI which I think of as not just evil but like, *uniquely* evil, even within an already ethically fraught industy. Like, obviously there's the 4o stuff in the video which just like jesus christ. Or the Scarlett Johansson voice situation where they released a voice that was like, eerily similar to hers right after she declined to participate, the fuckin.. nonprofit restructuring drama thing that triggered attorney general investigations, the safety team exodus. Or there's the lawsuits which allege OpenAI knew 4o was "dangerously sycophantic and psychologically manipulative" but released it anyway to beat Google's Gemini to market. Them overriding concerns from expert testers who said the model "felt slightly off" because broader user metrics looked positive. That's where I would say 1) the harms of ChatGPT are the most obvious (it's been linked to murder-suicides!) and 2) where other companies, but especially Anthropic, have been doing a lot better. Claude is specifically designed to not do the thing that killed those people. It has a real-time classifier that detects when someone might be in crisis and surfaces actual mental health resources, and perhaps more importantly it's explicitly trained against sycophancy. Anthropic publishes actual numbers on this, and they've open-sourced their evaluation tools so you can check their work. When Anthropic and OpenAI did joint safety testing, they found GPT-4o and related models were "much more willing than Claude models to cooperate with simulated human misuse, often providing detailed assistance with clearly harmful requests." Both companies' models have some sycophancy problems, but the scale of consequences has been dramatically different. Again though, point here is not really to glaze Anthropic so much as to point out just how like, uniquely, absurdly evil and dangerous OpenAI has been with their product, and that, as evidenced by other companies, even within the realms of "create an LLM chatbot" it is not inevitable to create a Magic 8 Ball That Tells You To Kill Yourself.
"Gityhub flavoured Reaganism" goes hard as a term
Having now watched the video, I think you make an incredibly good point about why this is not a video for youtube. I am reminded of the point that Doubleca5t made once, about why she stopped making When Posting Goes Wrong videos - with something like this, especially with a big enough online presence, it's very easy to become part of the story oneself. It really sucks because your research about why rationalism is incorrect about a lot of things is fascinating, but even if you left out all of the Zizian stuff, you would almost certainly be asked to comment on it. Thank you for sharing your perspective on here!
This was so satisfying, I've had all of these critiques of the rat community for so, so long. I remember going to Scott Alexander's reddit community and being amazed at the outright racism being perpetuated. Also the insanity of how they treat AGI as a real existential risk and the reasoning is 'because I think it could happen'. Just so lovely to hear someone much more articulate point out all of the flaws in this community. I wish I'd had it to hand when I had to deal with my coworker explain to me why Bayesian reasoning explained why women don't like sports.
I've been tangentially connected to rationalism for a while. Some of my friends are REALLY into it. I am pretty seceptible to the "special smart boy" pitches, but I also just do not have the "vulnerable to cults" trait that some people seem to have. I've seen people ruin their lives with this stuff.
I'm not sure if I actually want to read any more Harry Potter fanfiction given the whole JKR stuff but I'm curious if there's something that's similar enough to HPMOR but not as problematic
If you make this as a video the title has to be "Critique Of Pure Rationalism"
I've been sort-of connected to the EA community for over a decade now, and this was a mix of articulations of things I'd vaguely thought for a while, things I wasn't aware of, and things I don't entirely agree with / I think there's more nuance to. EA was originally sold to me not via rationalism (which I never got into, despite reading around half of HPMOR back in the day) but more as a complementary way of approaching the same issues as the trade justice movement (this was when it was all about mosquito nets and deworming, and before the existential threat stuff was significant). The argument went something like: many people in the world have short life expectancy and easily-treatable/preventable medical conditions that we in the west don't (and this disparity is in large part due to past and continuing actions of European powers, the US and international institutions). Obviously things like debt forgiveness and changing policy is the long term solution, but we (comparatively rich people in the west) can relieve suffering *right now* by donating appropriately. I can't say I've ever bought into it thoroughly (I have some misgivings about utilitarianism generally, and the places it can lead), but as a way of answering the question "what do I do with my money to try to make the world a better place?", I find those original strands of it compelling as at least a partial answer. I might have to revisit my reliance on GiveWell, though, given your comments about their approach to deworming (which I wasn't aware of). The whole appeal to me has always been the idea of giving to things backed strongly by research, and the whole low-probability/potentially-high-impact stuff has always struck me as far too reliant on numbers that at best have large margins of error and at worst are just made up based on vibes. I'll also note that Tim Harford's Cautionary Tales podcast had a discussion back in July about Sam Bankman-Fried with Michael Lewis (who spent some time with him) with the rough takeaway of "there's something in this EA stuff, but it can also lead you to some very suspect conclusions if you're not careful".
oh, yeah, the reason i think that podcast episode is interesting is because Harford and Lewis don't portray the whole FTX thing as a personal failure of SBF, but as a failure mode of the sort of EA he bought into - they describe him as driven to do the most good possible in the world, but he had (and was encouraged by those around him to have) the 80000 hours ends-justify-the-means mindset ("what's a little fraud if i'm saving the world?") and (at least towards the end of his career) the long-termism viewpoint that led to him funnelling money into some highly suspect "interventions"
you are so right about hunter x hunter. other things too
A rigged coin...a coin of whimsy, perhaps
Omg if you aren't already you gotta be listening to media club plus while you watch hunter x hunter it's the best way to do it
I think Bayesian inference is often misused by rationalists to justify their beliefs in the way you mention, but Bayes’s theorem and Bayesian inference is a legitimate mathematic and statistical tool. I wasn’t sure from that section if you meant it as a general critique of Bayesian methods or not. Bayesian methods have strengths and weaknesses, but in many instances they are more appropriate than frequentist methods. No method is correct for every application, and part of being a good mathematician or statistician is knowing what methods are best for the question or problem you’re working on. Since frequentist methods are usually learned first, learning baye’s afterward and understanding its strengths can feel like you’re finally learning the “true” way to do things. There’s also been tons of work on developing minimally-biased priors. But most people just want to call their beliefs “priors” because it makes them sound smarter and aren’t really doing Bayesian reasoning at all.
YES totally true! I should have been clearer about this in the video (and, like, if it was a scripted video for. YouTube I would have lol). My critique isn't of Bayesian methods as a mathematical and statistical tool. Bayes' theorem is legitimate and genuinely useful in the right applications! My issue is specifically with the way rationalists deploy "Bayesian reasoning" as an epistemological framework for everyday belief formation - where, as you say, people call their beliefs "priors" because it sounds smarter but aren't actually doing Bayesian inference in any rigorous sense. The math ends up obscuring where the subjectivity is rather than eliminating it. You put it better than I did: no method is correct for every application, and part of being good at this is knowing what tools fit what problems. The rationalist move is to treat Bayesian reasoning as the universal solvent for all epistemological questions, which it isn't.
G’day Sarah, really loved hearing a critique of the rat community that’s deeper than surface level and isn’t all sneering. I wonder if there’s a way to post it without talking about the Zizians, as I think you have a lot of value to add to the discourse! I’ve come at this stuff more from the EA perspective. I run an EA global health charity now, although many of my charity-founder friends and I are pretty reticent to identify with the term. I would do the whole ‘not-all-EAs’ routine here, but generally, the community has been awful at learning from _many_ past mistakes and repeatedly refuses to disown rationalism, various cults (haven’t even gotten into Leverage Research), racists, FTX (or crypto, which has obvious other harms), etc. etc. There are just not enough of us pushing for a bigger divorce for it to succeed, but there just aren’t a lot of places for young people who wanna get their hands dirty with global health or animal welfare to go. I think it will come with time, as more and more people in those cause areas come from outside the rationalist recruitment pipeline and AI takes over more and more of ‘mainstream’ EA. (Come to think of it, maybe you could make essentially the video you wanted to make by discussing Leverage instead of the Zizians—there’s a good briefing here https://forum.effectivealtruism.org/posts/qYbqX3jX4JnTtHA5f/leverage-research-reviewing-the-basic-facts)
Putting these aside, I think the biggest steelperson here is probably that most EA work in global health comes out of a specific individualist, randomista approach to global development spruiked by mainstream economists like Kremer, Glennerster, Banerjee, etc., where you take studies that work with high confidence and apply them at scale. I’ve seen many charities get to high scale and then just… continue to raise and spend money. There’s no real sense of institution-building or government handover. This _has_ started to change, especially in 2025. The USAID cuts represent a real big fucking shift in the way this all works in global health. Funders no longer want to keep funding you and are increasingly looking for off-ramps to government; one of the top-billed talks at this year’s EA Global summit in London effectively said as much. So I suspect this will change for the better.
I appreciate the amount of research that went to this video -- reading the sequences twice and finding those research papers about bias was impressive. I think this could actually be a good public video if you focus on the object-level complaints about the rational methods ("the math obscures where the subjectivity is", "research shows awareness of bias is not enough", etc); as a rat I found those a lot more convincing and thought-provoking than all the specific rat drama stories (HPMOR, EY, Ziz), though perhaps it's because I've already been exposed to that drama multiple times so it's not new to me. There's some substansive stuff there, but admittedly it would not be as exciting as your usual hobby drama videos. Re: "rationalist framework has no real theory of institutional change" -- I heavily recommend reading some of the blog posts of Kelsey Piper (e.g. in The Argument), she is a big rat and she talks a lot about e.g. building more housing, improving education, giving money to people directly. And of course the entire EA movement (strongly tied to rationalists) is all about building projects to help people, and I think the biggest complaint within it is variations of "political activism/lobbying is a more effective way of helping the world, why are we so against it". I do think you're right that a lot of the political beliefs there are "Reagan-era prosperity gospel", this is practically what The Argument is writing about now (= Abundance movement), I just think you're giving a very negative spin on it by focusing only on your (sometimes legitimate) complaints about the movement. Still, I'm curious to know where the rat/EA revolutionary liberal groups are -- I'm definitely interested in such ideas and would probably consider join them if they had any coherent community.
I guess to steelperson why the drama is valuable: The core message Sarah identified in HPMOR is ‘if you’re a precocious kid who’s convinced they’re smarter than everyone else, you’re actually right’. This has built up a recruitment pipeline of already socially isolated kids and young adults, primarily looking for a combination of intellectual stimulation and belonging, which for a certain type of predator is the equivalent of a bear finding the end of a salmon cannon next to a waterfall. The reason the community keeps producing new kinds of cults is because it repeatedly creates the conditions for them. This isn’t just a side effect of rationalism, it’s kind of the core thing that the movement does, given that its biggest counterfactual successes always seem to be on its margins (ex. some EAs, or accidentally making advanced AI come about sooner).
Thank you so much for making this video. I had a whole comment typed out about why hearing your critiques of Rationalism gave me a new perspective on a past relationship, but I actually don't think unloading a bunch of info about my personal life is the right vibe here. Suffice to say, this video has inadvertantly helped me understand and move on from a difficult time, so thank you! And I also just found it very interesting! I was only vaguely familiar with Rationalism and had no idea exactly how dumb it was.
probably better for u if you don't. but, you never know. maybe your story is the little piece that breaks the case for someone else. hopefully the kind of person you want to help and not like. an internet supervillain
I gotta finish watching the video, but: my greatest tea nerd friends with a similarly fancy kettle also microwave their tea when it's cold, so it's defintiely not just you
Thank you for this video after Tor Parsons fucking travesty of a video about the RATS
Oh no. Oh no no no. Not the Methods of Rationality. Is this why the Miss Scribe stuff keeps popping up in my algorithm lately? Are you summoning fandoms that should eternal lie again?
most of the works we have from Aristotle boil down to his lecture notes and outlines. the barbarians will do everything to keep a free mind down. and yet, here we are
Oh my god, this really was the best month to become a patron. We're eating good tonight, boys!