www-programmablemutter-com-8wp6z1.txt (23165B)
1 [1][https] 2 3 [2]Programmable Mutter 4 5 Subscribe 6 Sign in 7 Share this post 8 [https] 9 10 What OpenAI shares with Scientology 11 12 www.programmablemutter.com 13 Copy link 14 Facebook 15 Email 16 Note 17 Other 18 19 What OpenAI shares with Scientology 20 21 Strange beliefs, fights over money and bad science fiction 22 23 [12][https] 24 [13]Henry Farrell 25 Nov 20, 2023 26 73 27 Share this post 28 [https] 29 30 What OpenAI shares with Scientology 31 32 www.programmablemutter.com 33 Copy link 34 Facebook 35 Email 36 Note 37 Other 38 [20] 39 17 40 [21] 41 Share 42 [22] 43 [https] 44 45 When Sam Altman was ousted as CEO of OpenAI, some hinted that lurid depravities 46 lay behind his downfall. Surely, OpenAI’s board wouldn’t have toppled him if 47 there weren’t some sordid story about to hit the headlines? But the [23] 48 reporting all seems to be saying that it was God, not Sex, that lay behind 49 Altman’s downfall. And Money, that third great driver of human behavior, seems 50 to have driven his attempted return and his [24]new job at Microsoft, which is 51 OpenAI’s biggest investor by far. 52 53 As the NYT describes the people who pushed Altman out: 54 55 Thanks for reading Programmable Mutter! Subscribe for free to receive new 56 posts. And if you want to support my work, [25]buy my and Abe Newman’s new 57 book, [26]Underground Empire, and sing its praises (so long as you actually 58 liked it), on Amazon, Goodreads, social media and everywhere else that people 59 find out about good books. 60 61 [35][ ] 62 Subscribe 63 Ms. McCauley and Ms. Toner [HF - two board members] have ties to the 64 Rationalist and Effective Altruist movements, a community that is deeply 65 concerned that A.I. could one day destroy humanity. Today’s A.I. technology 66 cannot destroy humanity. But this community believes that as the technology 67 grows increasingly powerful, these dangers will arise. 68 69 McCauley and Toner reportedly worried that Altman was pushing too hard, too 70 quickly for new and potentially dangerous forms of AI (similar fears led some 71 OpenAI people to bail out and found a competitor, Anthropic, a couple of years 72 ago). The FT’s reporting [37]confirms that the fight was over how quickly to 73 commercialize AI 74 75 The back-story to all of this is actually much weirder than the average sex 76 scandal. The field of AI (in particular, its debates around Large Language 77 Models (LLMs) like OpenAI’s GPT-4) is profoundly shaped by cultish debates 78 among people with some very strange beliefs. 79 80 As LLMs have become increasingly powerful, theological arguments have begun to 81 mix it up with the profit motive. That explains why OpenAI has such an unusual 82 corporate form - it is a non-profit, with a for-profit structure retrofitted on 83 top, sweatily entangled with a profit-maximizing corporation (Microsoft). It 84 also plausibly explains why these tensions have exploded into the open. 85 86 ******** 87 88 I joked on Bluesky that the OpenAI saga was as if “the 1990s browser wars were 89 being waged by rival factions of Dianetics striving to control the future.” 90 Dianetics - for those who don’t obsess on the underbelly of American 91 intellectual history - was the 1.0 version of L. Ron Hubbard’s Scientology. 92 Hubbard [38]hatched it in collaboration with the science fiction editor John W. 93 Campbell (who had a major science fiction award named after him until 2019, 94 when his racism finally caught up with his reputation). 95 96 The AI safety debate too is an unintended consequence of genre fiction. In 97 1987, multiple-Hugo award winning science-fiction critic Dave Langford [39] 98 began a discussion of the “newish” genre of cyberpunk with a complaint about an 99 older genre of story on information technology, in which “the ultimate computer 100 is turned on and asked the ultimate question, and replies `Yes, now there is a 101 God!' 102 103 However, the cliche didn’t go away. Instead, it cross-bred with cyberpunk to 104 produce some quite surprising progeny. The midwife was the writer Vernor Vinge, 105 who proposed a revised meaning for “singularity.” This was a term already 106 familiar to science fiction readers as the place inside a black hole where the 107 ordinary predictions of physics broke down. Vinge suggested that we would soon 108 likely create true AI, which would be far better at thinking than baseline 109 humans, and would change the world in an accelerating process, creating a 110 historical [40]singularity, after which the future of the human species would 111 be radically unpredictable. 112 113 These ideas were turned into novels by Vinge himself, including A Fire Upon the 114 Deep (fun!) and Rainbow’s End (weak!). Other SF writers like Charles Stross 115 wrote novels about humans doing their best to co-exist with “weakly godlike” 116 machine intelligence (also fun!). Others who had no notable talent for writing, 117 like the futurist Ray Kurzweil, tried to turn the Singularity into the 118 foundation stone of a new account of human progress. I still possess a 119 mostly-unread copy of Kurzweil’s mostly-unreadable magnum opus, The Singularity 120 is Near, which was distributed en masse to bloggers like meself in an early 121 2000s marketing campaign. If I dug hard enough in my archives, I might even be 122 able to find the message from a publicity flack expressing disappointment that 123 I hadn’t written about the book after they sent it. All this speculation had a 124 strong flavor of end-of-days. As the Scots science fiction writer, Ken MacLeod 125 memorably put it, the Singularity was the “Rapture of the Nerds.” Ken, being 126 the [41]offspring of a Free Presbyterian preacher, knows a millenarian religion 127 when he sees it: Kurzweil’s doorstopper should really have been titled The 128 Singularity is Nigh. 129 130 Science fiction was the gateway drug, but it can’t really be blamed for 131 everything that happened later. Faith in the Singularity has roughly the same 132 relationship to SF as UFO-cultism. A small minority of SF writers are true 133 believers; most are hearty skeptics, but recognize that superhuman machine 134 intelligences are (a) possible) and (b) an extremely handy engine of plot. But 135 the combination of cultish Singularity beliefs and science fiction has 136 influenced a lot of external readers, who don’t distinguish sharply between the 137 religious and fictive elements, but mix and meld them to come up with strange 138 new hybrids. 139 140 Just such a syncretic religion provides the final part of the back-story to the 141 OpenAI crisis. In the 2010s, ideas about the Singularity cross-fertilized with 142 notions about Bayesian reasoning and some really terrible fanfic to create the 143 online “rationalist” movement mentioned in the NYT. 144 145 I’ve never read a text on rationalism, whether by true believers, by 146 hangers-on, or by bitter enemies (often erstwhile true believers), that really 147 gets the totality of what you see if you dive into its core texts and 148 apocrypha. And I won’t even try to provide one here. It is some Very Weird Shit 149 and there is really great religious sociology to be written about it. The 150 fights around [42]Roko’s Basilisk are perhaps the best known example of 151 rationalism in action outside the community, and give you some flavor of the 152 style of debate. But the very short version is that [43]Eliezer Yudkowsky, and 153 his multitudes of online fans embarked on a massive collective intellectual 154 project, which can reasonably be described as resurrecting David Langford’s 155 hoary 1980s SF cliche, and treating it as the most urgent dilemma facing human 156 beings today. We are about to create God. What comes next? Add Bayes’ Theorem 157 to Vinge’s core ideas, sez rationalism, and you’ll likely find the answer. 158 159 The consequences are what you might expect when a crowd of bright but rather 160 naive (and occasionally creepy) computer science and adjacent people try to 161 re-invent theology from first principles, to model what human-created gods 162 might do, and how they ought be constrained. They include the following, 163 non-comprehensive list: all sorts of strange mental exercises, postulated 164 superhuman entities benign and malign and how to think about them; the jumbling 165 of parts from fan-fiction, computer science, home-brewed philosophy and ARGs to 166 create grotesque and interesting intellectual chimeras; Nick Bostrom, and a 167 crew of very well funded philosophers; Effective Altruism, whose fancier 168 adherents often prefer not to acknowledge the approach’s somewhat disreputable 169 origins. 170 171 All this would be sociologically fascinating, but of little real world 172 consequence, if it hadn’t profoundly influenced the founders of the 173 organizations pushing AI forward. These luminaries think about the technologies 174 that they were creating in terms that they have borrowed wholesale from the 175 Yudkowsky extended universe. The risks and rewards of AI are seen as largely 176 commensurate with the risks and rewards of creating superhuman intelligences, 177 modeling how they might behave, and ensuring that we end up in a Good 178 Singularity, where AIs do not destroy or enslave humanity as a species, rather 179 than a bad one. 180 181 Even if rationalism’s answers are uncompelling, it asks interesting questions 182 that might have real human importance. However, it is at best unclear that 183 theoretical debates about immantenizing the eschaton tell us very much about 184 actually-existing “AI,” a family of important and sometimes very powerful 185 statistical techniques, which are being applied today, with emphatically 186 non-theoretical risks and benefits. 187 188 Ah, well, nevertheless. The rationalist agenda has demonstrably shaped the 189 questions around which the big AI ‘debates’ regularly revolve, as [44] 190 demonstrated by the Rishi Sunak/Sam Altman/Elon Musk love-fest “AI Summit” in 191 London a few weeks ago. 192 193 We are on a very strange timeline. My laboured Dianetics/Scientology joke can 194 be turned into an interesting hypothetical. It actually turns out (I only 195 stumbled across this recently) that Claude Shannon, the creator of information 196 theory (and, by extension, the computer revolution) was an [45]L. Ron Hubbard 197 fan in later life. In our continuum, this didn’t affect his theories: he had 198 already done his major work. Imagine, however, a parallel universe, where 199 Shannon’s science and standom had become intertwined and wildly influential, so 200 that debates in information science obsessed over whether we could eliminate 201 the noise of our [46]engrams, and isolate the signal of our True Selves, 202 allowing us all to become [47]Operating Thetans. Then reflect on how your 203 imagination doesn’t have to work nearly as hard as it ought to. A similarly 204 noxious blend of garbage ideas and actual science is the foundation stone of 205 the Grand AI Risk Debates that are happening today. 206 207 To be clear - not everyone working on existential AI risk (or ‘x risk’ as it is 208 usually summarized) is a true believer in Strong Eliezer Rationalism. Most, 209 very probably, are not. But you don’t need all that many true believers to keep 210 the machine running. At least, that is how I interpret this [48]Shazeda Ahmed 211 essay, which describes how some core precepts of a very strange set of beliefs 212 have become normalized as the background assumptions for thinking about the 213 promise and problems of AI. Even if you, as an AI risk person, don’t buy the 214 full intellectual package, you find yourself looking for work in a field where 215 the funding, the incentives, and the organizational structures mostly point in 216 a single direction (NB - this is my jaundiced interpretation, not hers). 217 218 ******** 219 220 There are two crucial differences between today’s AI cult and golden age 221 Scientology. The first was already mentioned in passing. Machine learning 222 works, and has some very important real life uses. [49]E-meters don’t work and 223 are useless for any purpose other than fleecing punters. 224 225 The second (which is closely related) is that Scientology’s ideology and 226 money-hustle reinforce each other. The more that you buy into stories about the 227 evils of mainstream psychology, the baggage of engrams that is preventing you 228 from reaching your true potential and so on and so on, the more you want to 229 spend on Scientology counselling. In AI, in contrast, God and Money have a 230 rather more tentative relationship. If you are profoundly worried about the 231 risks of AI, should you be unleashing it on the world for profit? That tension 232 helps explain the fight that has just broken out into the open. 233 234 235 It’s easy to forget that OpenAI was founded as an explicitly non-commercial 236 entity, the better to balance the rewards and the risks of these new 237 technologies. To quote from its [50]initial manifesto: 238 239 It’s hard to fathom how much human-level AI could benefit society, and it’s 240 equally hard to imagine how much it could damage society if built or 241 used incorrectly. Because of AI’s surprising history, it’s hard to predict 242 when human-level AI might come within reach. When it does, it’ll be 243 important to have a leading research institution which can prioritize a 244 good outcome for all over its own self-interest. 245 246 We’re hoping to grow OpenAI into such an institution. As a non-profit, our 247 aim is to build value for everyone rather than shareholders. Researchers 248 will be strongly encouraged to publish their work, whether as papers, blog 249 posts, or code, and our patents (if any) will be shared with the world. 250 We’ll freely collaborate with others across many institutions and expect to 251 work with companies to research and deploy new technologies. 252 253 That … isn’t quite how it worked out. The Sam Altman justification for 254 deviation from this vision, laid out in various interviews, is that it turned 255 out to just be too damned expensive to train the models as they grew bigger, 256 and bigger and bigger. This necessitated the creation of an add-on structure, 257 which would sidle into profitable activity. It also required massive cash 258 infusions from Microsoft (reportedly in [51]the range of $13 billion), which 259 also has an exclusive license to OpenAI’s most recent LLM, GPT-4. Microsoft, it 260 should be noted, is not in the business of prioritizing “a good outcome for all 261 over its own self-interest.” It looks instead, to invest its resources along 262 the very best Friedmanite principles, so as to create whopping returns for 263 shareholders. And $13 billion is a lot of invested resources. 264 265 This, very plausibly explains the current crisis. OpenAI’s governance 266 arrangements are shaped by the fact that it was a non-profit until relatively 267 recently. The board is a non-profit board. The two members already mentioned, 268 McCauley and Toner, are not the kind of people you would expect to see making 269 the big decisions for a major commercial entity. They plausibly represent the 270 older rationalist vision of what OpenAI was supposed to do, and the risks that 271 it was supposed to avert. 272 273 But as OpenAI’s ambitions have grown, that vision has been watered down in 274 favor of making money. I’ve heard that there were a lot of people in the AI 275 community who were really unhappy with OpenAI’s initial decision to let GPT 276 rip. That spurred the race for commercial domination of AI which has shaped 277 pretty well everything that has happened since, leading to model after model 278 being launched, and to hell with the consequences. People like Altman still 279 talk about the dangers of AGI. But their organizations and businesses keep 280 releasing more, and more powerful systems, which can be, and are being, used in 281 all sorts of unanticipated ways, for good and for ill. 282 283 It would perhaps be too cynical to say that AGI existential risk rhetoric has 284 become a cynical hustle, intended to redirect the attentions of regulators 285 toward possibly imaginary future risks in the future, and away from problematic 286 but profitable activities that are happening right now. Human beings have an 287 enormous capacity to fervently believe in things that it is in their 288 self-interest to believe, and to update those beliefs as the interests change 289 or become clearer. I wouldn’t be surprised at all if Altman sincerely thinks 290 that he is still acting for the good of humankind (there are certainly enough 291 people assuring him that he is). But it isn’t surprising either that the true 292 believers are revolting, as Altman stretches their ideology ever further and 293 thinner to facilitate raking in the benjamins. 294 295 The OpenAI saga is a fight between God and Money; between a quite peculiar 296 quasi-religious movement, and a quite ordinary desire to make cold hard cash. 297 You should probably be putting your bets on Money prevailing in whatever 298 strange arrangement of forces is happening as Altman is beamed up into the 299 Microsoft mothership. But we might not be all that better off in this 300 particular case if the forces of God were to prevail, and the rationalists who 301 toppled Altman were to win a surprising victory. They want to slow down AI, 302 which is good, but for all sorts of weird reasons, which are unlikely to 303 provide good solutions for the actual problems that AI generates. The important 304 questions about AI are the ones that neither God or [52]Mammon has particularly 305 good answers for - but that’s a topic for future posts. 306 307 Thanks for reading Programmable Mutter! Subscribe for free to receive new 308 posts. And if you want to support my work, [53]buy my and Abe Newman’s new 309 book, [54]Underground Empire, and sing its praises (as long as you actually 310 liked it) on Amazon, Goodreads, social media and everywhere else that people 311 find out about good books. 312 313 [63][ ] 314 Subscribe 315 73 316 Share this post 317 [https] 318 319 What OpenAI shares with Scientology 320 321 www.programmablemutter.com 322 Copy link 323 Facebook 324 Email 325 Note 326 Other 327 [71] 328 17 329 [72] 330 Share 331 17 Comments 332 [https] 333 [ ] 334 Share this discussion 335 [https] 336 337 What OpenAI shares with Scientology 338 339 www.programmablemutter.com 340 Copy link 341 Facebook 342 Email 343 Note 344 Other 345 346 [81] 347 Tarik Najeddine 348 [82]Writes Factual Dispatch 349 [83]Nov 20, 2023 350 351 [80] ChatGPT is just Zapp Brannigan or a McKinsey consultant. A veneer of 352 [https] confidence and a person to blame when the executive "needs" to make a 353 hard decision. You previously blamed the Bain consultants when you 354 offshored a factory, now you blame AI. 355 356 Expand full comment 357 Reply 358 Share 359 360 [86] 361 Gerben Wierda 362 [87]Nov 21, 2023·edited Nov 21, 2023 363 364 Came here via Dave Karpf's link. Beautiful stuff, and "The Singularity 365 is Nigh" made me laugh out loud. 366 367 The psychological and sociological/cultural side of the current 368 GPT-fever is indeed far more important and telling than the technical 369 reality. Short summary: quantity has its own certain quality, but the 370 systems may be impressive, we humans are impressionable. 371 372 Recently, Sam Altman received a Hawking Fellowship for the OpenAI Team 373 [85] and he spoke for a few minutes followed by a Q&A (available on 374 [https] YouTube). In that session he was asked what are important qualities for 375 'founders' of these innovative tech firms. He answered that founders 376 should have ‘deeply held convictions’ that are stable without a lot of 377 ‘positive external reinforcement’, ‘obsession’ with a problem, and a 378 ‘super powerful internal drive’. They needed to be an 'evangelist'. The 379 link with religion shows here too. ([88]https:// 380 erikjlarson.substack.com/p/gerben-wierda-on-chatgpt-altman-and). TED 381 just released Ilya Sutskever’s talk and you see it there too. We have 382 strong believers turned evangelists and we have a world of disciples 383 and followers. It is indeed a very good analogy. 384 385 Expand full comment 386 Reply 387 Share 388 389 [90]15 more comments... 390 Top 391 New 392 Community 393 394 No posts 395 396 Ready for more? 397 398 [99][ ] 399 Subscribe 400 © 2024 Henry Farrell 401 [101]Privacy ∙ [102]Terms ∙ [103]Collection notice 402 [104] Start Writing[105]Get the app 403 [106]Substack is the home for great writing 404 This site requires JavaScript to run correctly. Please [107]turn on JavaScript 405 or unblock scripts 406 407 References: 408 409 [1] https://www.programmablemutter.com/ 410 [2] https://www.programmablemutter.com/ 411 [12] https://substack.com/profile/557668-henry-farrell 412 [13] https://substack.com/@henryfarrell 413 [20] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments 414 [21] javascript:void(0) 415 [22] https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F555fe47f-ac07-4614-b78b-5d269fde7539_1024x1024.webp 416 [23] https://www.nytimes.com/2023/11/18/technology/open-ai-sam-altman-what-happened.html 417 [24] https://www.ft.com/content/54e36c93-08e5-4a9e-bda6-af673c3e9bb5 418 [25] https://amzn.to/3PbIyqX 419 [26] https://amzn.to/3PbIyqX 420 [37] https://www.ft.com/content/54e36c93-08e5-4a9e-bda6-af673c3e9bb5 421 [38] https://longreads.com/2017/02/01/xenus-paradox-the-fiction-of-l-ron-hubbard/ 422 [39] https://ansible.uk/ai/pcwplus/pcwp1987.html 423 [40] https://edoras.sdsu.edu/~vinge/misc/singularity.html 424 [41] https://www.heraldscotland.com/life_style/arts_ents/14479010.science-fiction-writer-ken-macleod-free-presbyterian-childhood-time-communist-party-member-future-humanity/ 425 [42] https://www.lesswrong.com/tag/rokos-basilisk 426 [43] https://en.wikipedia.org/wiki/Eliezer_Yudkowsky 427 [44] https://www.politico.eu/article/rishi-sunak-artificial-intelligence-pivot-safety-summit-united-kingdom-silicon-valley-effective-altruism/ 428 [45] https://longreads.com/2018/10/23/the-dawn-of-dianetics-l-ron-hubbard-john-w-campbell-and-the-origins-of-scientology/ 429 [46] https://en.wikipedia.org/wiki/Engram_(Dianetics) 430 [47] https://en.wikipedia.org/wiki/Operating_Thetan 431 [48] https://crookedtimber.org/2023/11/16/from-algorithmic-monoculture-to-epistemic-monoculture-understanding-the-rise-of-ai-safety/ 432 [49] https://en.wikipedia.org/wiki/E-meter 433 [50] https://openai.com/blog/introducing-openai 434 [51] https://www.bloomberg.com/news/newsletters/2023-06-15/how-chatgpt-openai-made-microsoft-an-ai-tech-giant-big-take 435 [52] https://www.newadvent.org/cathen/09580b.htm 436 [53] https://amzn.to/3PbIyqX 437 [54] https://amzn.to/3PbIyqX 438 [71] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments 439 [72] javascript:void(0) 440 [80] https://substack.com/profile/1263175-tarik-najeddine 441 [81] https://substack.com/profile/1263175-tarik-najeddine 442 [82] https://factualdispatch.substack.com/?utm_source=substack&utm_medium=web&utm_content=comment_metadata 443 [83] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comment/43988738 444 [85] https://substack.com/profile/23165546-gerben-wierda 445 [86] https://substack.com/profile/23165546-gerben-wierda 446 [87] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comment/44033603 447 [88] https://erikjlarson.substack.com/p/gerben-wierda-on-chatgpt-altman-and 448 [90] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments 449 [101] https://substack.com/privacy 450 [102] https://substack.com/tos 451 [103] https://substack.com/ccpa#personal-data-collected 452 [104] https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer 453 [105] https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button 454 [106] https://substack.com/ 455 [107] https://enable-javascript.com/