dce/davideisinger.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Use w3m for archiving

Browse Source
This commit is contained in:
David Eisinger
2024-01-17 12:04:56 -05:00
parent c5f0c6161a
commit ae64f3eb0a
80 changed files with 28830 additions and 29811 deletions
Download Patch File Download Diff File Expand all files Collapse all files

827
static/archive/www-programmablemutter-com-8wp6z1.txt
View File

@@ -1,496 +1,455 @@
#[1]Programmable Mutter
[1][https]
[2][https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimage
s%2F6676303a-e6a9-4e7d-b10e-8662cfcfb435_1024x1024.png]
[2]Programmable Mutter
[3]Programmable Mutter
(BUTTON)
Subscribe
(BUTTON) Sign in
(BUTTON)
Share this post
Subscribe
Sign in
Share this post
[https]
What OpenAI shares with Scientology
www.programmablemutter.com
(BUTTON)
Copy link
(BUTTON)
Facebook
(BUTTON)
Email
(BUTTON)
Note
(BUTTON)
Other
www.programmablemutter.com
Copy link
Facebook
Email
Note
Other
What OpenAI shares with Scientology
Strange beliefs, fights over money and bad science fiction
[4]Henry Farrell
Nov 20, 2023
73
(BUTTON)
Share this post
[12][https]
[13]Henry Farrell
Nov 20, 2023
73
Share this post
[https]
What OpenAI shares with Scientology
www.programmablemutter.com
(BUTTON)
Copy link
(BUTTON)
Facebook
(BUTTON)
Email
(BUTTON)
Note
(BUTTON)
Other
17
www.programmablemutter.com
Copy link
Facebook
Email
Note
Other
[20]
17
[21]
Share
[22]
[https]
When Sam Altman was ousted as CEO of OpenAI, some hinted that lurid
depravities lay behind his downfall. Surely, OpenAI’s board wouldn’t
have toppled him if there weren’t some sordid story about to hit the
headlines? But the [5]reporting all seems to be saying that it was God,
not Sex, that lay behind Altman’s downfall. And Money, that third great
driver of human behavior, seems to have driven his attempted return and
his [6]new job at Microsoft, which is OpenAI’s biggest investor by far.
When Sam Altman was ousted as CEO of OpenAI, some hinted that lurid depravities
lay behind his downfall. Surely, OpenAI’s board wouldn’t have toppled him if
there weren’t some sordid story about to hit the headlines? But the [23]
reporting all seems to be saying that it was God, not Sex, that lay behind
Altman’s downfall. And Money, that third great driver of human behavior, seems
to have driven his attempted return and his [24]new job at Microsoft, which is
OpenAI’s biggest investor by far.
As the NYT describes the people who pushed Altman out:
As the NYT describes the people who pushed Altman out:
Thanks for reading Programmable Mutter! Subscribe for free to receive
new posts. And if you want to support my work, [7]buy my and Abe
Newman’s new book, [8]Underground Empire, and sing its praises (so long
as you actually liked it), on Amazon, Goodreads, social media and
everywhere else that people find out about good books.
____________________
(BUTTON) Subscribe
Thanks for reading Programmable Mutter! Subscribe for free to receive new
posts. And if you want to support my work, [25]buy my and Abe Newman’s new
book, [26]Underground Empire, and sing its praises (so long as you actually
liked it), on Amazon, Goodreads, social media and everywhere else that people
find out about good books.
Ms. McCauley and Ms. Toner [HF - two board members] have ties to the
Rationalist and Effective Altruist movements, a community that is
deeply concerned that A.I. could one day destroy humanity. Today’s
A.I. technology cannot destroy humanity. But this community believes
that as the technology grows increasingly powerful, these dangers
will arise.
[35][ ]
Subscribe
Ms. McCauley and Ms. Toner [HF - two board members] have ties to the
Rationalist and Effective Altruist movements, a community that is deeply
concerned that A.I. could one day destroy humanity. Today’s A.I. technology
cannot destroy humanity. But this community believes that as the technology
grows increasingly powerful, these dangers will arise.
McCauley and Toner reportedly worried that Altman was pushing too hard,
too quickly for new and potentially dangerous forms of AI (similar
fears led some OpenAI people to bail out and found a competitor,
Anthropic, a couple of years ago). The FT’s reporting [9]confirms that
the fight was over how quickly to commercialize AI
McCauley and Toner reportedly worried that Altman was pushing too hard, too
quickly for new and potentially dangerous forms of AI (similar fears led some
OpenAI people to bail out and found a competitor, Anthropic, a couple of years
ago). The FT’s reporting [37]confirms that the fight was over how quickly to
commercialize AI
The back-story to all of this is actually much weirder than the average
sex scandal. The field of AI (in particular, its debates around Large
Language Models (LLMs) like OpenAI’s GPT-4) is profoundly shaped by
cultish debates among people with some very strange beliefs.
The back-story to all of this is actually much weirder than the average sex
scandal. The field of AI (in particular, its debates around Large Language
Models (LLMs) like OpenAI’s GPT-4) is profoundly shaped by cultish debates
among people with some very strange beliefs.
As LLMs have become increasingly powerful, theological arguments have
begun to mix it up with the profit motive. That explains why OpenAI has
such an unusual corporate form - it is a non-profit, with a for-profit
structure retrofitted on top, sweatily entangled with a
profit-maximizing corporation (Microsoft). It also plausibly explains
why these tensions have exploded into the open.
As LLMs have become increasingly powerful, theological arguments have begun to
mix it up with the profit motive. That explains why OpenAI has such an unusual
corporate form - it is a non-profit, with a for-profit structure retrofitted on
top, sweatily entangled with a profit-maximizing corporation (Microsoft). It
also plausibly explains why these tensions have exploded into the open.
********
********
I joked on Bluesky that the OpenAI saga was as if “the 1990s browser
wars were being waged by rival factions of Dianetics striving to
control the future.” Dianetics - for those who don’t obsess on the
underbelly of American intellectual history - was the 1.0 version of L.
Ron Hubbard’s Scientology. Hubbard [10]hatched it in collaboration with
the science fiction editor John W. Campbell (who had a major science
fiction award named after him until 2019, when his racism finally
caught up with his reputation).
I joked on Bluesky that the OpenAI saga was as if “the 1990s browser wars were
being waged by rival factions of Dianetics striving to control the future.”
Dianetics - for those who don’t obsess on the underbelly of American
intellectual history - was the 1.0 version of L. Ron Hubbard’s Scientology.
Hubbard [38]hatched it in collaboration with the science fiction editor John W.
Campbell (who had a major science fiction award named after him until 2019,
when his racism finally caught up with his reputation).
The AI safety debate too is an unintended consequence of genre fiction.
In 1987, multiple-Hugo award winning science-fiction critic Dave
Langford [11]began a discussion of the “newish” genre of cyberpunk with
a complaint about an older genre of story on information technology, in
which “the ultimate computer is turned on and asked the ultimate
question, and replies `Yes, now there is a God!'
The AI safety debate too is an unintended consequence of genre fiction. In
1987, multiple-Hugo award winning science-fiction critic Dave Langford [39]
began a discussion of the “newish” genre of cyberpunk with a complaint about an
older genre of story on information technology, in which “the ultimate computer
is turned on and asked the ultimate question, and replies `Yes, now there is a
God!'
However, the cliche didn’t go away. Instead, it cross-bred with
cyberpunk to produce some quite surprising progeny. The midwife was the
writer Vernor Vinge, who proposed a revised meaning for “singularity.”
This was a term already familiar to science fiction readers as the
place inside a black hole where the ordinary predictions of physics
broke down. Vinge suggested that we would soon likely create true AI,
which would be far better at thinking than baseline humans, and would
change the world in an accelerating process, creating a historical
[12]singularity, after which the future of the human species would be
radically unpredictable.
However, the cliche didn’t go away. Instead, it cross-bred with cyberpunk to
produce some quite surprising progeny. The midwife was the writer Vernor Vinge,
who proposed a revised meaning for “singularity.” This was a term already
familiar to science fiction readers as the place inside a black hole where the
ordinary predictions of physics broke down. Vinge suggested that we would soon
likely create true AI, which would be far better at thinking than baseline
humans, and would change the world in an accelerating process, creating a
historical [40]singularity, after which the future of the human species would
be radically unpredictable.
These ideas were turned into novels by Vinge himself, including A Fire
Upon the Deep (fun!) and Rainbow’s End (weak!). Other SF writers like
Charles Stross wrote novels about humans doing their best to co-exist
with “weakly godlike” machine intelligence (also fun!). Others who had
no notable talent for writing, like the futurist Ray Kurzweil, tried to
turn the Singularity into the foundation stone of a new account of
human progress. I still possess a mostly-unread copy of Kurzweil’s
mostly-unreadable magnum opus, The Singularity is Near, which was
distributed en masse to bloggers like meself in an early 2000s
marketing campaign. If I dug hard enough in my archives, I might even
be able to find the message from a publicity flack expressing
disappointment that I hadn’t written about the book after they sent it.
All this speculation had a strong flavor of end-of-days. As the Scots
science fiction writer, Ken MacLeod memorably put it, the Singularity
was the “Rapture of the Nerds.” Ken, being the [13]offspring of a Free
Presbyterian preacher, knows a millenarian religion when he sees it:
Kurzweil’s doorstopper should really have been titled The Singularity
is Nigh.
These ideas were turned into novels by Vinge himself, including A Fire Upon the
Deep (fun!) and Rainbow’s End (weak!). Other SF writers like Charles Stross
wrote novels about humans doing their best to co-exist with “weakly godlike”
machine intelligence (also fun!). Others who had no notable talent for writing,
like the futurist Ray Kurzweil, tried to turn the Singularity into the
foundation stone of a new account of human progress. I still possess a
mostly-unread copy of Kurzweil’s mostly-unreadable magnum opus, The Singularity
is Near, which was distributed en masse to bloggers like meself in an early
2000s marketing campaign. If I dug hard enough in my archives, I might even be
able to find the message from a publicity flack expressing disappointment that
I hadn’t written about the book after they sent it. All this speculation had a
strong flavor of end-of-days. As the Scots science fiction writer, Ken MacLeod
memorably put it, the Singularity was the “Rapture of the Nerds.” Ken, being
the [41]offspring of a Free Presbyterian preacher, knows a millenarian religion
when he sees it: Kurzweil’s doorstopper should really have been titled The
Singularity is Nigh.
Science fiction was the gateway drug, but it can’t really be blamed for
everything that happened later. Faith in the Singularity has roughly
the same relationship to SF as UFO-cultism. A small minority of SF
writers are true believers; most are hearty skeptics, but recognize
that superhuman machine intelligences are (a) possible) and (b) an
extremely handy engine of plot. But the combination of cultish
Singularity beliefs and science fiction has influenced a lot of
external readers, who don’t distinguish sharply between the religious
and fictive elements, but mix and meld them to come up with strange new
hybrids.
Science fiction was the gateway drug, but it can’t really be blamed for
everything that happened later. Faith in the Singularity has roughly the same
relationship to SF as UFO-cultism. A small minority of SF writers are true
believers; most are hearty skeptics, but recognize that superhuman machine
intelligences are (a) possible) and (b) an extremely handy engine of plot. But
the combination of cultish Singularity beliefs and science fiction has
influenced a lot of external readers, who don’t distinguish sharply between the
religious and fictive elements, but mix and meld them to come up with strange
new hybrids.
Just such a syncretic religion provides the final part of the
back-story to the OpenAI crisis. In the 2010s, ideas about the
Singularity cross-fertilized with notions about Bayesian reasoning and
some really terrible fanfic to create the online “rationalist” movement
mentioned in the NYT.
Just such a syncretic religion provides the final part of the back-story to the
OpenAI crisis. In the 2010s, ideas about the Singularity cross-fertilized with
notions about Bayesian reasoning and some really terrible fanfic to create the
online “rationalist” movement mentioned in the NYT.
I’ve never read a text on rationalism, whether by true believers, by
hangers-on, or by bitter enemies (often erstwhile true believers), that
really gets the totality of what you see if you dive into its core
texts and apocrypha. And I won’t even try to provide one here. It is
some Very Weird Shit and there is really great religious sociology to
be written about it. The fights around [14]Roko’s Basilisk are perhaps
the best known example of rationalism in action outside the community,
and give you some flavor of the style of debate. But the very short
version is that [15]Eliezer Yudkowsky, and his multitudes of online
fans embarked on a massive collective intellectual project, which can
reasonably be described as resurrecting David Langford’s hoary 1980s SF
cliche, and treating it as the most urgent dilemma facing human beings
today. We are about to create God. What comes next? Add Bayes’ Theorem
to Vinge’s core ideas, sez rationalism, and you’ll likely find the
answer.
I’ve never read a text on rationalism, whether by true believers, by
hangers-on, or by bitter enemies (often erstwhile true believers), that really
gets the totality of what you see if you dive into its core texts and
apocrypha. And I won’t even try to provide one here. It is some Very Weird Shit
and there is really great religious sociology to be written about it. The
fights around [42]Roko’s Basilisk are perhaps the best known example of
rationalism in action outside the community, and give you some flavor of the
style of debate. But the very short version is that [43]Eliezer Yudkowsky, and
his multitudes of online fans embarked on a massive collective intellectual
project, which can reasonably be described as resurrecting David Langford’s
hoary 1980s SF cliche, and treating it as the most urgent dilemma facing human
beings today. We are about to create God. What comes next? Add Bayes’ Theorem
to Vinge’s core ideas, sez rationalism, and you’ll likely find the answer.
The consequences are what you might expect when a crowd of bright but
rather naive (and occasionally creepy) computer science and adjacent
people try to re-invent theology from first principles, to model what
human-created gods might do, and how they ought be constrained. They
include the following, non-comprehensive list: all sorts of strange
mental exercises, postulated superhuman entities benign and malign and
how to think about them; the jumbling of parts from fan-fiction,
computer science, home-brewed philosophy and ARGs to create grotesque
and interesting intellectual chimeras; Nick Bostrom, and a crew of very
well funded philosophers; Effective Altruism, whose fancier adherents
often prefer not to acknowledge the approach’s somewhat disreputable
origins.
The consequences are what you might expect when a crowd of bright but rather
naive (and occasionally creepy) computer science and adjacent people try to
re-invent theology from first principles, to model what human-created gods
might do, and how they ought be constrained. They include the following,
non-comprehensive list: all sorts of strange mental exercises, postulated
superhuman entities benign and malign and how to think about them; the jumbling
of parts from fan-fiction, computer science, home-brewed philosophy and ARGs to
create grotesque and interesting intellectual chimeras; Nick Bostrom, and a
crew of very well funded philosophers; Effective Altruism, whose fancier
adherents often prefer not to acknowledge the approach’s somewhat disreputable
origins.
All this would be sociologically fascinating, but of little real world
consequence, if it hadn’t profoundly influenced the founders of the
organizations pushing AI forward. These luminaries think about the
technologies that they were creating in terms that they have borrowed
wholesale from the Yudkowsky extended universe. The risks and rewards
of AI are seen as largely commensurate with the risks and rewards of
creating superhuman intelligences, modeling how they might behave, and
ensuring that we end up in a Good Singularity, where AIs do not destroy
or enslave humanity as a species, rather than a bad one.
All this would be sociologically fascinating, but of little real world
consequence, if it hadn’t profoundly influenced the founders of the
organizations pushing AI forward. These luminaries think about the technologies
that they were creating in terms that they have borrowed wholesale from the
Yudkowsky extended universe. The risks and rewards of AI are seen as largely
commensurate with the risks and rewards of creating superhuman intelligences,
modeling how they might behave, and ensuring that we end up in a Good
Singularity, where AIs do not destroy or enslave humanity as a species, rather
than a bad one.
Even if rationalism’s answers are uncompelling, it asks interesting
questions that might have real human importance. However, it is at best
unclear that theoretical debates about immantenizing the eschaton tell
us very much about actually-existing “AI,” a family of important and
sometimes very powerful statistical techniques, which are being applied
today, with emphatically non-theoretical risks and benefits.
Even if rationalism’s answers are uncompelling, it asks interesting questions
that might have real human importance. However, it is at best unclear that
theoretical debates about immantenizing the eschaton tell us very much about
actually-existing “AI,” a family of important and sometimes very powerful
statistical techniques, which are being applied today, with emphatically
non-theoretical risks and benefits.
Ah, well, nevertheless. The rationalist agenda has demonstrably shaped
the questions around which the big AI ‘debates’ regularly revolve, as
[16]demonstrated by the Rishi Sunak/Sam Altman/Elon Musk love-fest “AI
Summit” in London a few weeks ago.
Ah, well, nevertheless. The rationalist agenda has demonstrably shaped the
questions around which the big AI ‘debates’ regularly revolve, as [44]
demonstrated by the Rishi Sunak/Sam Altman/Elon Musk love-fest “AI Summit” in
London a few weeks ago.
We are on a very strange timeline. My laboured Dianetics/Scientology
joke can be turned into an interesting hypothetical. It actually turns
out (I only stumbled across this recently) that Claude Shannon, the
creator of information theory (and, by extension, the computer
revolution) was an [17]L. Ron Hubbard fan in later life. In our
continuum, this didn’t affect his theories: he had already done his
major work. Imagine, however, a parallel universe, where Shannon’s
science and standom had become intertwined and wildly influential, so
that debates in information science obsessed over whether we could
eliminate the noise of our [18]engrams, and isolate the signal of our
True Selves, allowing us all to become [19]Operating Thetans. Then
reflect on how your imagination doesn’t have to work nearly as hard as
it ought to. A similarly noxious blend of garbage ideas and actual
science is the foundation stone of the Grand AI Risk Debates that are
happening today.
We are on a very strange timeline. My laboured Dianetics/Scientology joke can
be turned into an interesting hypothetical. It actually turns out (I only
stumbled across this recently) that Claude Shannon, the creator of information
theory (and, by extension, the computer revolution) was an [45]L. Ron Hubbard
fan in later life. In our continuum, this didn’t affect his theories: he had
already done his major work. Imagine, however, a parallel universe, where
Shannon’s science and standom had become intertwined and wildly influential, so
that debates in information science obsessed over whether we could eliminate
the noise of our [46]engrams, and isolate the signal of our True Selves,
allowing us all to become [47]Operating Thetans. Then reflect on how your
imagination doesn’t have to work nearly as hard as it ought to. A similarly
noxious blend of garbage ideas and actual science is the foundation stone of
the Grand AI Risk Debates that are happening today.
To be clear - not everyone working on existential AI risk (or ‘x risk’
as it is usually summarized) is a true believer in Strong Eliezer
Rationalism. Most, very probably, are not. But you don’t need all that
many true believers to keep the machine running. At least, that is how
I interpret this [20]Shazeda Ahmed essay, which describes how some core
precepts of a very strange set of beliefs have become normalized as the
background assumptions for thinking about the promise and problems of
AI. Even if you, as an AI risk person, don’t buy the full intellectual
package, you find yourself looking for work in a field where the
funding, the incentives, and the organizational structures mostly point
in a single direction (NB - this is my jaundiced interpretation, not
hers).
To be clear - not everyone working on existential AI risk (or ‘x risk’ as it is
usually summarized) is a true believer in Strong Eliezer Rationalism. Most,
very probably, are not. But you don’t need all that many true believers to keep
the machine running. At least, that is how I interpret this [48]Shazeda Ahmed
essay, which describes how some core precepts of a very strange set of beliefs
have become normalized as the background assumptions for thinking about the
promise and problems of AI. Even if you, as an AI risk person, don’t buy the
full intellectual package, you find yourself looking for work in a field where
the funding, the incentives, and the organizational structures mostly point in
a single direction (NB - this is my jaundiced interpretation, not hers).
********
********
There are two crucial differences between today’s AI cult and golden
age Scientology. The first was already mentioned in passing. Machine
learning works, and has some very important real life uses.
[21]E-meters don’t work and are useless for any purpose other than
fleecing punters.
There are two crucial differences between today’s AI cult and golden age
Scientology. The first was already mentioned in passing. Machine learning
works, and has some very important real life uses. [49]E-meters don’t work and
are useless for any purpose other than fleecing punters.
The second (which is closely related) is that Scientology’s ideology
and money-hustle reinforce each other. The more that you buy into
stories about the evils of mainstream psychology, the baggage of
engrams that is preventing you from reaching your true potential and so
on and so on, the more you want to spend on Scientology counselling. In
AI, in contrast, God and Money have a rather more tentative
relationship. If you are profoundly worried about the risks of AI,
should you be unleashing it on the world for profit? That tension helps
explain the fight that has just broken out into the open.
The second (which is closely related) is that Scientology’s ideology and
money-hustle reinforce each other. The more that you buy into stories about the
evils of mainstream psychology, the baggage of engrams that is preventing you
from reaching your true potential and so on and so on, the more you want to
spend on Scientology counselling. In AI, in contrast, God and Money have a
rather more tentative relationship. If you are profoundly worried about the
risks of AI, should you be unleashing it on the world for profit? That tension
helps explain the fight that has just broken out into the open.
It’s easy to forget that OpenAI was founded as an explicitly
non-commercial entity, the better to balance the rewards and the risks
of these new technologies. To quote from its [22]initial manifesto:
It’s hard to fathom how much human-level AI could benefit society,
and it’s equally hard to imagine how much it could damage society if
built or used incorrectly. Because of AI’s surprising history, it’s
hard to predict when human-level AI might come within reach. When it
does, it’ll be important to have a leading research institution
which can prioritize a good outcome for all over its
own self-interest.
It’s easy to forget that OpenAI was founded as an explicitly non-commercial
entity, the better to balance the rewards and the risks of these new
technologies. To quote from its [50]initial manifesto:
We’re hoping to grow OpenAI into such an institution. As a
non-profit, our aim is to build value for everyone rather than
shareholders. Researchers will be strongly encouraged to publish
their work, whether as papers, blog posts, or code, and our patents
(if any) will be shared with the world. We’ll freely collaborate
with others across many institutions and expect to work with
companies to research and deploy new technologies.
It’s hard to fathom how much human-level AI could benefit society, and it’s
equally hard to imagine how much it could damage society if built or
used incorrectly. Because of AI’s surprising history, it’s hard to predict
when human-level AI might come within reach. When it does, it’ll be
important to have a leading research institution which can prioritize a
good outcome for all over its own self-interest.
That … isn’t quite how it worked out. The Sam Altman justification for
deviation from this vision, laid out in various interviews, is that it
turned out to just be too damned expensive to train the models as they
grew bigger, and bigger and bigger. This necessitated the creation of
an add-on structure, which would sidle into profitable activity. It
also required massive cash infusions from Microsoft (reportedly in
[23]the range of $13 billion), which also has an exclusive license to
OpenAI’s most recent LLM, GPT-4. Microsoft, it should be noted, is not
in the business of prioritizing “a good outcome for all over its own
self-interest.” It looks instead, to invest its resources along the
very best Friedmanite principles, so as to create whopping returns for
shareholders. And $13 billion is a lot of invested resources.
We’re hoping to grow OpenAI into such an institution. As a non-profit, our
aim is to build value for everyone rather than shareholders. Researchers
will be strongly encouraged to publish their work, whether as papers, blog
posts, or code, and our patents (if any) will be shared with the world.
We’ll freely collaborate with others across many institutions and expect to
work with companies to research and deploy new technologies.
This, very plausibly explains the current crisis. OpenAI’s governance
arrangements are shaped by the fact that it was a non-profit until
relatively recently. The board is a non-profit board. The two members
already mentioned, McCauley and Toner, are not the kind of people you
would expect to see making the big decisions for a major commercial
entity. They plausibly represent the older rationalist vision of what
OpenAI was supposed to do, and the risks that it was supposed to avert.
That … isn’t quite how it worked out. The Sam Altman justification for
deviation from this vision, laid out in various interviews, is that it turned
out to just be too damned expensive to train the models as they grew bigger,
and bigger and bigger. This necessitated the creation of an add-on structure,
which would sidle into profitable activity. It also required massive cash
infusions from Microsoft (reportedly in [51]the range of $13 billion), which
also has an exclusive license to OpenAI’s most recent LLM, GPT-4. Microsoft, it
should be noted, is not in the business of prioritizing “a good outcome for all
over its own self-interest.” It looks instead, to invest its resources along
the very best Friedmanite principles, so as to create whopping returns for
shareholders. And $13 billion is a lot of invested resources.
But as OpenAI’s ambitions have grown, that vision has been watered down
in favor of making money. I’ve heard that there were a lot of people in
the AI community who were really unhappy with OpenAI’s initial decision
to let GPT rip. That spurred the race for commercial domination of AI
which has shaped pretty well everything that has happened since,
leading to model after model being launched, and to hell with the
consequences. People like Altman still talk about the dangers of AGI.
But their organizations and businesses keep releasing more, and more
powerful systems, which can be, and are being, used in all sorts of
unanticipated ways, for good and for ill.
This, very plausibly explains the current crisis. OpenAI’s governance
arrangements are shaped by the fact that it was a non-profit until relatively
recently. The board is a non-profit board. The two members already mentioned,
McCauley and Toner, are not the kind of people you would expect to see making
the big decisions for a major commercial entity. They plausibly represent the
older rationalist vision of what OpenAI was supposed to do, and the risks that
it was supposed to avert.
It would perhaps be too cynical to say that AGI existential risk
rhetoric has become a cynical hustle, intended to redirect the
attentions of regulators toward possibly imaginary future risks in the
future, and away from problematic but profitable activities that are
happening right now. Human beings have an enormous capacity to
fervently believe in things that it is in their self-interest to
believe, and to update those beliefs as the interests change or become
clearer. I wouldn’t be surprised at all if Altman sincerely thinks that
he is still acting for the good of humankind (there are certainly
enough people assuring him that he is). But it isn’t surprising either
that the true believers are revolting, as Altman stretches their
ideology ever further and thinner to facilitate raking in the
benjamins.
But as OpenAI’s ambitions have grown, that vision has been watered down in
favor of making money. I’ve heard that there were a lot of people in the AI
community who were really unhappy with OpenAI’s initial decision to let GPT
rip. That spurred the race for commercial domination of AI which has shaped
pretty well everything that has happened since, leading to model after model
being launched, and to hell with the consequences. People like Altman still
talk about the dangers of AGI. But their organizations and businesses keep
releasing more, and more powerful systems, which can be, and are being, used in
all sorts of unanticipated ways, for good and for ill.
The OpenAI saga is a fight between God and Money; between a quite
peculiar quasi-religious movement, and a quite ordinary desire to make
cold hard cash. You should probably be putting your bets on Money
prevailing in whatever strange arrangement of forces is happening as
Altman is beamed up into the Microsoft mothership. But we might not be
all that better off in this particular case if the forces of God were
to prevail, and the rationalists who toppled Altman were to win a
surprising victory. They want to slow down AI, which is good, but for
all sorts of weird reasons, which are unlikely to provide good
solutions for the actual problems that AI generates. The important
questions about AI are the ones that neither God or [24]Mammon has
particularly good answers for - but that’s a topic for future posts.
It would perhaps be too cynical to say that AGI existential risk rhetoric has
become a cynical hustle, intended to redirect the attentions of regulators
toward possibly imaginary future risks in the future, and away from problematic
but profitable activities that are happening right now. Human beings have an
enormous capacity to fervently believe in things that it is in their
self-interest to believe, and to update those beliefs as the interests change
or become clearer. I wouldn’t be surprised at all if Altman sincerely thinks
that he is still acting for the good of humankind (there are certainly enough
people assuring him that he is). But it isn’t surprising either that the true
believers are revolting, as Altman stretches their ideology ever further and
thinner to facilitate raking in the benjamins.
Thanks for reading Programmable Mutter! Subscribe for free to receive
new posts. And if you want to support my work, [25]buy my and Abe
Newman’s new book, [26]Underground Empire, and sing its praises (as
long as you actually liked it) on Amazon, Goodreads, social media and
everywhere else that people find out about good books.
____________________
(BUTTON) Subscribe
73
(BUTTON)
Share this post
The OpenAI saga is a fight between God and Money; between a quite peculiar
quasi-religious movement, and a quite ordinary desire to make cold hard cash.
You should probably be putting your bets on Money prevailing in whatever
strange arrangement of forces is happening as Altman is beamed up into the
Microsoft mothership. But we might not be all that better off in this
particular case if the forces of God were to prevail, and the rationalists who
toppled Altman were to win a surprising victory. They want to slow down AI,
which is good, but for all sorts of weird reasons, which are unlikely to
provide good solutions for the actual problems that AI generates. The important
questions about AI are the ones that neither God or [52]Mammon has particularly
good answers for - but that’s a topic for future posts.
Thanks for reading Programmable Mutter! Subscribe for free to receive new
posts. And if you want to support my work, [53]buy my and Abe Newman’s new
book, [54]Underground Empire, and sing its praises (as long as you actually
liked it) on Amazon, Goodreads, social media and everywhere else that people
find out about good books.
[63][ ]
Subscribe
73
Share this post
[https]
What OpenAI shares with Scientology
www.programmablemutter.com
(BUTTON)
Copy link
(BUTTON)
Facebook
(BUTTON)
Email
(BUTTON)
Note
(BUTTON)
Other
17
Share
17 Comments
____________________________________________________________
____________________________________________________________
____________________________________________________________
____________________________________________________________
(BUTTON)
Share this discussion
www.programmablemutter.com
Copy link
Facebook
Email
Note
Other
[71]
17
[72]
Share
17 Comments
[https]
[ ]
Share this discussion
[https]
What OpenAI shares with Scientology
www.programmablemutter.com
(BUTTON)
Copy link
(BUTTON)
Facebook
(BUTTON)
Email
(BUTTON)
Note
(BUTTON)
Other
Tarik Najeddine
[27]Writes Factual Dispatch
[28]Nov 20, 2023
www.programmablemutter.com
Copy link
Facebook
Email
Note
Other
ChatGPT is just Zapp Brannigan or a McKinsey consultant. A veneer of
confidence and a person to blame when the executive "needs" to make a
hard decision. You previously blamed the Bain consultants when you
offshored a factory, now you blame AI.
Expand full comment
Reply
Share
(BUTTON)
Gerben Wierda
[29]Nov 21, 2023·edited Nov 21, 2023
[81]
Tarik Najeddine
[82]Writes Factual Dispatch
[83]Nov 20, 2023
Came here via Dave Karpf's link. Beautiful stuff, and "The Singularity
is Nigh" made me laugh out loud.
[80] ChatGPT is just Zapp Brannigan or a McKinsey consultant. A veneer of
[https] confidence and a person to blame when the executive "needs" to make a
hard decision. You previously blamed the Bain consultants when you
offshored a factory, now you blame AI.
The psychological and sociological/cultural side of the current
GPT-fever is indeed far more important and telling than the technical
reality. Short summary: quantity has its own certain quality, but the
systems may be impressive, we humans are impressionable.
Expand full comment
Reply
Share
Recently, Sam Altman received a Hawking Fellowship for the OpenAI Team
and he spoke for a few minutes followed by a Q&A (available on
YouTube). In that session he was asked what are important qualities for
'founders' of these innovative tech firms. He answered that founders
should have ‘deeply held convictions’ that are stable without a lot of
‘positive external reinforcement’, ‘obsession’ with a problem, and a
‘super powerful internal drive’. They needed to be an 'evangelist'. The
link with religion shows here too.
([30]https://erikjlarson.substack.com/p/gerben-wierda-on-chatgpt-altman
-and). TED just released Ilya Sutskever’s talk and you see it there
too. We have strong believers turned evangelists and we have a world of
disciples and followers. It is indeed a very good analogy.
Expand full comment
Reply
Share
(BUTTON)
[31]15 more comments...
Top
New
Community
[86]
Gerben Wierda
[87]Nov 21, 2023·edited Nov 21, 2023
No posts
Came here via Dave Karpf's link. Beautiful stuff, and "The Singularity
is Nigh" made me laugh out loud.
Ready for more?
____________________
(BUTTON) Subscribe
© 2024 Henry Farrell
[32]Privacy ∙ [33]Terms ∙ [34]Collection notice
Start Writing[35]Get the app
[36]Substack is the home for great writing
The psychological and sociological/cultural side of the current
GPT-fever is indeed far more important and telling than the technical
reality. Short summary: quantity has its own certain quality, but the
systems may be impressive, we humans are impressionable.
This site requires JavaScript to run correctly. Please [37]turn on
JavaScript or unblock scripts
Recently, Sam Altman received a Hawking Fellowship for the OpenAI Team
[85] and he spoke for a few minutes followed by a Q&A (available on
[https] YouTube). In that session he was asked what are important qualities for
'founders' of these innovative tech firms. He answered that founders
should have ‘deeply held convictions’ that are stable without a lot of
‘positive external reinforcement’, ‘obsession’ with a problem, and a
‘super powerful internal drive’. They needed to be an 'evangelist'. The
link with religion shows here too. ([88]https://
erikjlarson.substack.com/p/gerben-wierda-on-chatgpt-altman-and). TED
just released Ilya Sutskever’s talk and you see it there too. We have
strong believers turned evangelists and we have a world of disciples
and followers. It is indeed a very good analogy.
References
Expand full comment
Reply
Share
Visible links:
1. https://www.programmablemutter.com/feed
2. https://www.programmablemutter.com/
3. https://www.programmablemutter.com/
4. https://substack.com/@henryfarrell
5. https://www.nytimes.com/2023/11/18/technology/open-ai-sam-altman-what-happened.html
6. https://www.ft.com/content/54e36c93-08e5-4a9e-bda6-af673c3e9bb5
7. https://amzn.to/3PbIyqX
8. https://amzn.to/3PbIyqX
9. https://www.ft.com/content/54e36c93-08e5-4a9e-bda6-af673c3e9bb5
10. https://longreads.com/2017/02/01/xenus-paradox-the-fiction-of-l-ron-hubbard/
11. https://ansible.uk/ai/pcwplus/pcwp1987.html
12. https://edoras.sdsu.edu/~vinge/misc/singularity.html
13. https://www.heraldscotland.com/life_style/arts_ents/14479010.science-fiction-writer-ken-macleod-free-presbyterian-childhood-time-communist-party-member-future-humanity/
14. https://www.lesswrong.com/tag/rokos-basilisk
15. https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
16. https://www.politico.eu/article/rishi-sunak-artificial-intelligence-pivot-safety-summit-united-kingdom-silicon-valley-effective-altruism/
17. https://longreads.com/2018/10/23/the-dawn-of-dianetics-l-ron-hubbard-john-w-campbell-and-the-origins-of-scientology/
18. https://en.wikipedia.org/wiki/Engram_(Dianetics)
19. https://en.wikipedia.org/wiki/Operating_Thetan
20. https://crookedtimber.org/2023/11/16/from-algorithmic-monoculture-to-epistemic-monoculture-understanding-the-rise-of-ai-safety/
21. https://en.wikipedia.org/wiki/E-meter
22. https://openai.com/blog/introducing-openai
23. https://www.bloomberg.com/news/newsletters/2023-06-15/how-chatgpt-openai-made-microsoft-an-ai-tech-giant-big-take
24. https://www.newadvent.org/cathen/09580b.htm
25. https://amzn.to/3PbIyqX
26. https://amzn.to/3PbIyqX
27. https://factualdispatch.substack.com/?utm_source=substack&utm_medium=web&utm_content=comment_metadata
28. https://www.programmablemutter.com/p/look-at-scientology-to-understand/comment/43988738
29. https://www.programmablemutter.com/p/look-at-scientology-to-understand/comment/44033603
30. https://erikjlarson.substack.com/p/gerben-wierda-on-chatgpt-altman-and
31. https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments
32. https://substack.com/privacy
33. https://substack.com/tos
34. https://substack.com/ccpa#personal-data-collected
35. https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button
36. https://substack.com/
37. https://enable-javascript.com/
[90]15 more comments...
Top
New
Community
Hidden links:
39. https://substack.com/profile/557668-henry-farrell
40. https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments
41. javascript:void(0)
42. https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F555fe47f-ac07-4614-b78b-5d269fde7539_1024x1024.webp
43. https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments
44. javascript:void(0)
45. https://substack.com/profile/1263175-tarik-najeddine
46. https://substack.com/profile/1263175-tarik-najeddine
47. https://substack.com/profile/23165546-gerben-wierda
48. https://substack.com/profile/23165546-gerben-wierda
49. https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer
No posts
Ready for more?
[99][ ]
Subscribe
© 2024 Henry Farrell
[101]Privacy ∙ [102]Terms ∙ [103]Collection notice
[104] Start Writing[105]Get the app
[106]Substack is the home for great writing
This site requires JavaScript to run correctly. Please [107]turn on JavaScript
or unblock scripts
References:
[1] https://www.programmablemutter.com/
[2] https://www.programmablemutter.com/
[12] https://substack.com/profile/557668-henry-farrell
[13] https://substack.com/@henryfarrell
[20] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments
[21] javascript:void(0)
[22] https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F555fe47f-ac07-4614-b78b-5d269fde7539_1024x1024.webp
[23] https://www.nytimes.com/2023/11/18/technology/open-ai-sam-altman-what-happened.html
[24] https://www.ft.com/content/54e36c93-08e5-4a9e-bda6-af673c3e9bb5
[25] https://amzn.to/3PbIyqX
[26] https://amzn.to/3PbIyqX
[37] https://www.ft.com/content/54e36c93-08e5-4a9e-bda6-af673c3e9bb5
[38] https://longreads.com/2017/02/01/xenus-paradox-the-fiction-of-l-ron-hubbard/
[39] https://ansible.uk/ai/pcwplus/pcwp1987.html
[40] https://edoras.sdsu.edu/~vinge/misc/singularity.html
[41] https://www.heraldscotland.com/life_style/arts_ents/14479010.science-fiction-writer-ken-macleod-free-presbyterian-childhood-time-communist-party-member-future-humanity/
[42] https://www.lesswrong.com/tag/rokos-basilisk
[43] https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
[44] https://www.politico.eu/article/rishi-sunak-artificial-intelligence-pivot-safety-summit-united-kingdom-silicon-valley-effective-altruism/
[45] https://longreads.com/2018/10/23/the-dawn-of-dianetics-l-ron-hubbard-john-w-campbell-and-the-origins-of-scientology/
[46] https://en.wikipedia.org/wiki/Engram_(Dianetics)
[47] https://en.wikipedia.org/wiki/Operating_Thetan
[48] https://crookedtimber.org/2023/11/16/from-algorithmic-monoculture-to-epistemic-monoculture-understanding-the-rise-of-ai-safety/
[49] https://en.wikipedia.org/wiki/E-meter
[50] https://openai.com/blog/introducing-openai
[51] https://www.bloomberg.com/news/newsletters/2023-06-15/how-chatgpt-openai-made-microsoft-an-ai-tech-giant-big-take
[52] https://www.newadvent.org/cathen/09580b.htm
[53] https://amzn.to/3PbIyqX
[54] https://amzn.to/3PbIyqX
[71] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments
[72] javascript:void(0)
[80] https://substack.com/profile/1263175-tarik-najeddine
[81] https://substack.com/profile/1263175-tarik-najeddine
[82] https://factualdispatch.substack.com/?utm_source=substack&utm_medium=web&utm_content=comment_metadata
[83] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comment/43988738
[85] https://substack.com/profile/23165546-gerben-wierda
[86] https://substack.com/profile/23165546-gerben-wierda
[87] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comment/44033603
[88] https://erikjlarson.substack.com/p/gerben-wierda-on-chatgpt-altman-and
[90] https://www.programmablemutter.com/p/look-at-scientology-to-understand/comments
[101] https://substack.com/privacy
[102] https://substack.com/tos
[103] https://substack.com/ccpa#personal-data-collected
[104] https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer
[105] https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button
[106] https://substack.com/
[107] https://enable-javascript.com/