davideisinger.com/static/archive/www-wrecka-ge-rdi1xr.txt

[1]wreckage/salvage
[2][ ] 

  • [3]about
  • [4]posts

  • [5]sign up (free or paid)
  • [6]sign in
  • Search

Bad shape
21 days ago by [7]Erin Kissane — 11 min read

Bad shape

The idea I keep coming back to is that the big platforms, like Dickens' Marley,
were dead to begin with, and are now something particularly bad, which is dead
on their feet.

In the game of Go, bad shape is the term for configurations of stones on the
game board that are inefficient in achieving their offensive goal (territory
capture) and unlikely to achieve their defensive goal (the state of "[8]life").
You can extend a bad shape in a fruitless attempt to make it better, but you'll
generally be wasting your time.

The idea I keep coming back to is that the big platforms, [9]like Marley, were
dead to begin with, and are now something particularly bad, which is dead on
their feet. Not because they’re been abandoned by users (yet) but because
they’re structurally incapable of governing the systems they made, and most of
the things they try to do about it introduce more and weirder problems.

While they were still gobbling hundreds of millions of new users each year—and
while the old political machines were still catching up—platforms could outrun
the problem. Now, though, the number of remaining uncaptured humans dwindles,
the politicians and propagandists have adapted to exploit the mass-scale
machinery the platforms provide, and the positions platform companies have
contorted themselves into trying to shoehorn governance into ungovernable
structures are increasingly hard to maintain.

Facebook especially is likely to zombie-shamble along for some time, held
upright by its deep weave into the coordination of offline life and by [10]the
communities for whom it serves as a sole accessible connection to the internet,
but the whole apparatus looks increasingly precarious.

(These are very simple points, but it remains a wince-inducing faux pas to say
them in a lot of tech-thinking spaces, so I will keep pushing on the obvious.)

The evidence of the past decade and a half argues strongly that platform
corporations are structurally incapable of good governance, primarily because
most of their central aims (continuous growth, market dominance, profit via
extraction) conflict with many basic human and societal needs.

As entities, large social platforms continuously undergo rapid mutations in
service of their need to maximize profit and expansion while minimizing the
kinds of societal and individual harm that can plausibly cause them regulatory
trouble or user disengagement. (The set of things that can cause trouble is
also always shifting, as political and cultural spheres influence and are
influenced by the platforms.) But platform mutations emerge only within a
narrow range of possibilities delineated by the set of decisions considered
valid in, roughly speaking, [11]Milton Friedman's model of corporate purpose.

Within this circumscribed mutation zone, certain goals are able to be named and
earnestly pursued ("stop spam" or "eliminate the distribution of CSAM"), even
if they're never achieved. Other goals (anything to do with hate speech,
incitement to violence, or misinformation, for example) can be named and
pursued, but only in ways that don't hinder the workings of the
profit-extraction machinery—which mostly means that they come in on the margins
and after the fact, as in "[12]after the fact of a genocide that Facebook had
years of explicit advance warnings about." Working on the margins and after the
fact still matters—less damage is better than more damage—but it means "trust
and safety" is kept well clear of the core.

Again, this is all simple and obvious. A tractor structurally can't spare a
thought for the lives of the fieldmice; shouting at the tractor when it
destroys their nests is a category error. Business does business. The
production line doesn't stop just because a few people lose fingers or lives.
And what is a modern corporation but a legal spell for turning reasoning beings
into temporarily vacant machines? We know this, which is why we have OSHA and
the FAA and the FTC, for now.

It's no surprise that when prodded by entities with cultural or regulatory
power, platforms build more semi-effective AI classifiers, hire more underpaid
contract moderators, and temporarily stiffen their unevenly enforced community
rules, but then immediately slump back toward their natural form, which appears
to be a cartoonishly overgrown early-2000s web forum crammed with soft targets
and overrun by trolling, spam, and worse.

It’s possible to make the argument that sufficiently strong leadership could
make even a tech corporation appear to be capable of holding an ethical line,
and maybe even capable of accepting slightly smaller profits in service of
socially beneficial goals—and that, conversely, the awful people in charge are
the main source of the problems. It’s not a very good argument, though, even
when I make it myself.

Yes, X is currently controlled by a bizarrely gibbering billionaire with
obvious symptoms of late-stage Mad King disease. Yes, Facebook and
Instagram—which control vastly more territory than X—are controlled by a
feckless, Tulip-craze-mainlining billionaire with a long history of grudgingly
up-regulating governance efforts when under public or governmental pressure and
then immediately axeing them when the spotlight moves on. But would these
platforms inflict less damage if they were led by people who valued the well
being of others? Probably yes, to a degree. Twitter/X has offered a lurid
natural experiment, and the changes in X after it moved from Jack Dorsey’s
spacey techno-libertarian leadership to Elon Musk’s desperately needy
quasi-fascist circus act have been obviously bad. A version of Meta founded and
led by someone with a reasonably sharp ethical grounding clearly wouldn’t look
much like the real Meta at all.

On the other hand, TikTok’s social function is reasonably close to Meta and
X’s, and the fact that [13]its CEO, Shou Zi Chew, seems like a relatively
normal person, doesn’t seem to have correlated with dramatically better
performance in eliminating [14]Nazi organizing, [15]genocidal and
violence-inciting content, [16]CSAM distribution ([17]archive link), or the
kind of [18]semi-pro disinformation that makes it harder for people
experiencing natural disasters to understand what’s happening.

Crucially, more reasonable CEO behavior doesn’t seem to prevent the lower-level
and potentially even more destructive social effects of platforms that [19]
Henry Farrell persuasively explains from a social theory perspective, or that
Renée DiResta memorably calls a “Cambrian explosion of subjective, bespoke
realities” in [20]Invisible Rulers.* (I'll do a separate post collecting
thoughts on this angle, because it's too important to breeze by.)

The realities of our moment also work against arguments for the potential of
heroic leadership: even apparently level-headed tech executives now appear to
understand that the next Trump Administration intends to rule unreasonably and
vengefully, and that failure to [21]perform obeisance and [22]make tribute may
result in federal interference that could plausibly unmake their companies.

Those are not risks any global corporation can take, but our oddball lineup of
big platform companies is in a special bind. No matter how desperately they
want to be seen as neutral utilities, they have functioned, for good and ill,
like social and political wrecking balls—and real or feigned misapprehensions
about algorithms and censorship notwithstanding, real-world governments
understand this. The second coming of Trump makes the situation especially
stark, but the underlying dynamics are neither new nor temporary.

Given that every large platform posing as a public square has put itself into
the genuinely untenable situation of acting as a global corporate arbiter of
politically hot speech, they will all always be in the gunsights of the world’s
least reasonable governments. This was bad enough for the platforms when the
least reasonable governments were Putin’s or Erdoğan’s or Modi’s—a truly
unreasonable government in control of their home jurisdiction is an existential
threat.

And again, in reality, the corporations are configured to try to address the
least political kinds of abuse—CSAM, spam, scams, and a few other forms of
inauthentic behavior—and very little else. As a result, they can’t govern more
subtle or politicized speech for much longer than I can roll a quarter down a
piece of string.

So what would it take for a corporation to become capable of good governance of
things like political speech, incitement to violence and genocide, hate speech,
most forms of inauthentic behavior, and platform manipulation? Two things, at
least:

  • The ability and willingness to take and hold ethical stances that will be
    sharply unpopular with large swathes of the people mostly likely to
    effectively target them with legislation and abuses of power, and
  • the ability and willingness to devote something approaching the majority of
    their company’s time, money, and attention to building and running [23]
    devolved or [24]federated systems for doing high-performance high-context
    local governance according to those unpopular ethical stances.

Can you bring yourself to imagine—concretely and in detail—these conditions
occurring in the leadership of a global corporation?

And again, achieving a mode of governance that can appropriately handle those
most obvious elements—the hate speech, the network abuse, the inauthentic
behavior, all of it—is necessary but not sufficient for reaching something like
a healthy equilibrium. The elements of big social platforms that make them
attractive and fun and profitable are the same elements that, as currently
implemented, turn low-level human behavior patterns around status, belief,
conformity, and predation into a high-speed mass-scale mess of fractured
publics and realities.

Two points of clarification: First, I’m not saying “Can’t fix people problems
with technology,” which is exactly as true and useful as “Guns don’t kill
people, people kill people.” (I used the former in what I thought was a very
obviously sarcastic way, but apparently the intent was insufficiently clear.)
If a technological system makes human problems worse, you have to fix the
system or break it and build a better one.

Second, none of what I’m trying to get at here is about the intent of people
who work on big platforms. Corporate platform trust and safety staff routinely
work themselves to the brink of individual illness or collapse to handle what
they’re permitted and resourced to handle—which is itself a tiny fraction of
what would be necessary to handle to make platforms good. Corporate platform
governance by technology companies whose success requires growth and
attention-extraction, though, is a bankrupt idea.

If we briefly isolate the reality of our technological present, it’s hard to
find it anything but absurd to expect a corporation to govern global or even
local speech for any humanist value of “well.” And no one chose it, exactly, it
just happened when the fantasies of the internet as an Apollonian zone of
libertarian splendor met the reality of globally connected primate brains under
late capitalism. I explicitly blame the connected-computer dream of
technologically mediated liberation as cartoonishly exemplified in [25]JP
Barlow’s Declaration, which centered on keeping the bad old world of human
governance, which it equated with censorship, out of the internet:

    You claim there are problems among us that you need to solve. You use this
    claim as an excuse to invade our precincts. Many of these problems don't
    exist. Where there are real conflicts, where there are wrongs, we will
    identify them and address them by our means. We are forming our own Social
    Contract. This governance will arise according to the conditions of our
    world, not yours. Our world is different.

The governance arose, all right, once the money got real.

The computer dream’s rapidly evaporating and over-salinated shallows are still
keeping the tech industry’s dumbest boats afloat, but the platforms have been
scraping bottom for years while their owners slap on layers and layers of
patches and bilge-pumps and bucket brigades manned by people from former
colonies. The problem isn't (just) turning fact-checking on or off or
deactivating a swarm of halfassed AI classifers or ceasing to pretend to act on
most reports of misconduct, it's bad shape.

All of which is to say that yes, Zuckerberg is a terrible chump and Musk is a
grotesque quasi-Rasputin, and that does matter, but the boards they stand on
have been rotten the whole time. Centralized corporate governance of global
mega-platforms was always a goofy idea, and we should have given up on it years
ago.

This is where I get into awkward situations with lovely people, including
several I count as friends, because they’re determined "not to let platforms
off the hook.” I feel this, deeply, along with things like send the Sacklers to
the guillotine. But keeping the fucked-up mutant fish on the hook will not
magically transform it into an entity capable of governing.

—

Earlier this week at Platformer, [26]Casey Newton reported some insider views
on what Meta’s most recent roll-back of content moderation and fact-checking
means. The post is worth reading, and after the Myanmar research I did in 2023,
and for what it’s worth, I don’t think Casey’s sources overstate the dangers
inherent in what Meta’s doing: more real human beings are going to suffer and
lose children and be killed because of this. But I want to look at something in
the cursory background section of the newsletter, about the work that Meta put
in after 2016, when Facebook got criticized for hosting election-interference
ops in the US:

    Chastened by the criticism, Meta set out to shore up its defenses. It hired
    40,000 content moderators around the world, invested heavily in building
    new technology to analyze content for potential harms and flag it for
    review, and became the world’s leading funder of third-party fact-checking
    organizations. It spent $280 million to create an independent Oversight
    Board to adjudicate the most difficult questions about online speech. It
    disrupted dozens of networks of state-sponsored trolls who sought to use
    Facebook, Instagram, and WhatsApp to spread propaganda and attack
    dissenters.

[27]According to Financial Times reporting ([28]archive link), Meta currently
employs or has contracted with about 40,000 people to work on “safety and
security,” of which just 15,000 are content moderators, for a user base of
roughly four billion users, which works out to more than a quarter of a million
users per moderator. This chimes with [29]New York Times reporting ([30]archive
link) suggesting that in 2021, Accenture was billing Facebook for about 5,800
full-time contract moderators. (For what it’s worth, in 2017, Meta promised to
add all of [31]3,000 trust and safety staff.) Nor are Meta’s moderation
resources allocated evenly: About 90% of Facebook users are outside the US and
Canada; that overwhelming majority gets [32]approximately 13% of the company’s
moderation time ([33]archive link).

And while we’re here, in 2020—the year Oversight Board started hiring—Meta
cleared about $91 billion in profit. The Oversight Board trust got $280 million
from Meta, or just over 0.3% of the company’s annual profits. The Oversight
Board itself, though inclined to deliver thoughtful if glacially slow
recommendations, appears to have accomplished [34]remarkably little.

Again: The work tens of thousands of people around the world put in to try to
make platforms less terrible is real and essential work, and it’s often done at
a terrible cost. It’s also the barest gesture at serious governance, and much
of it is pure Potemkin Village.

That’s only a couple of things pulled from one paragraph that happened to hit
my inbox while I was drafting this post, but I did [35]a whole lot of that kind
of close reading in 2023, and came out believing that platform intensifications
of governance in response to periodic governmental pressure are best understood
as a little bit of real (though deeply inadequate) change and a whole lot of 
[36]flopping. Then, when the pressure comes off, the platforms re-orient like
compass needles tossed into in an MRI machine.

—

[37]Barlow’s Declaration—which is excruciating and which I’ve been making
myself reread annually for years as penance for participating in tech
culture—ends like this:

    We will create a civilization of the Mind in Cyberspace. May it be more
    humane and fair than the world your governments have made before.

What we got instead was a handful of global-scale company towns that continue
to prove their comprehensive unfitness to govern and their absolute
vulnerability to the offline governments the free internet was meant to work
around.

So sure: [38]Protocols over platforms. Then we have to actually do the
inelegant, un-heroic, expensive work of rebuilding the essential structures of
human civilization on top of the protocols, because it turns out we just have
the one world, online or off, no way out.

Thank you

This post, like the others on this site, exists [39]because people have signed
on to support the work. If you find it useful, and your situation allows for it
with ease, please consider signing up! Enormous thanks to those of you who
have. And a note to members: I've wrestled down the Ghost commenting problem
and the first real discussion post for project members goes up tomorrow, so if
you've signed up for a paid membership, look for that in your inbox soon.

Notes

* Renée’s book is very good and I recommend it for its lucid explanations and
commitment to drawing on previous eras of mass communication without doing too
deeply into either theoretical or historical rabbitholes (which I love, but
which don’t make for popular reading). I don’t 100% agree with her conclusions,
but they’re clearly stated and cleanly argued, which allows for productive
disagreement—and I value that more than full alignment.

A common response to the things I've been posting is "Okay, but what will work,
then?" I think there are hints at answers in the very chunky [40]fediverse
governance research I worked on last year, in online and offline cooperatives,
in [41]Rudy Fraser's Blacksky, and in the kinds of projects Nathan Schneider
assesses in [42]Governable Spaces. I'll continue to explore what I think might
be good shapes for governance here in ways that—I hope—will be more pragmatic
than quixotic.

The featured diagram for this post is International Marine Engineering's 1912
depiction of the profile and deck of the Titanic ([43]v. 17, p. 199).

[44] [45] [46] The link has been copied!
[47][https://www.wrecka.g]
[48] How we'll do discussions here
Newer post

[49]How we'll do discussions here

Older post

[50]What people in the global majority need from networks

[51] What people in the global majority need from networks

Subscribe to new posts.

[52][                    ] Subscribe
Processing your application Great! Check your inbox and confirm your
subscription There was an error sending the email
choose your type:

  • [54]( ) fancy (default)
  • [55]( ) simpler

[56]wreckage/salvage

Making and mending networks for humans.

wreckage/salvage © Erin Kissane 2025  
You’ve successfully subscribed to wreckage/salvage
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Success! Your email is updated.
Your link has expired
Success! Check your email for magic link to sign-in.

References:

[1] https://www.wrecka.ge/
[3] https://www.wrecka.ge/about/
[4] https://www.wrecka.ge/tag/posts/
[5] https://www.wrecka.ge/#/portal
[6] https://www.wrecka.ge/signin/
[7] https://www.wrecka.ge/author/erin/
[8] https://senseis.xmp.net/?LifeAndDeath
[9] https://www.gutenberg.org/files/46/46-h/46-h.htm
[10] https://www.theengineroom.org/library/new-report-exploring-a-transition-to-alternative-social-media-platforms-for-social-justice-organizations-in-the-majority-world/
[11] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctrine-the-social-responsibility-of-business-is-to.html
[12] https://erinkissane.com/meta-in-myanmar-part-i-the-setup
[13] https://www.wired.com/story/shou-zi-chew-tik-tok-big-interview/
[14] https://www.isdglobal.org/digital_dispatches/naztok-an-organized-neo-nazi-tiktok-network-is-getting-millions-of-views/
[15] https://www.isdglobal.org/wp-content/uploads/2021/08/HateScape_v5.pdf
[16] https://www.ft.com/content/d5549d48-8f02-464c-9c7d-17404a5b6d02
[17] https://archive.ph/tvn8K
[18] https://newrepublic.com/article/186928/misinformation-new-normal-disaster-response
[19] https://www.programmablemutter.com/p/were-getting-the-social-media-crisis
[20] https://www.hachettebookgroup.com/titles/renee-diresta/invisible-rulers/9781541703377/
[21] https://www.newsweek.com/tech-ceos-donations-donald-trump-joe-biden-inaugurations-compared-2010457
[22] https://qz.com/google-youtube-trump-inauguration-meta-amazon-apple-1851736124
[23] https://en.wikipedia.org/wiki/Devolution
[24] https://en.wikipedia.org/wiki/Federalism
[25] https://www.eff.org/cyberspace-independence
[26] https://www.platformer.news/meta-fact-checking-free-speech-surrender/?ref=platformer-newsletter
[27] https://www.ft.com/content/afeb56f2-9ba5-4103-890d-91291aea4caa
[28] https://archive.ph/UxgFy
[29] https://www.nytimes.com/2021/08/31/technology/facebook-accenture-content-moderation.html
[30] https://archive.ph/SEZpr
[31] https://www.nytimes.com/2017/05/03/technology/facebook-moderators-q1-earnings.html
[32] https://www.wsj.com/articles/facebook-drug-cartels-human-traffickers-response-is-weak-documents-11631812953
[33] https://archive.ph/https://www.wsj.com/articles/facebook-drug-cartels-human-traffickers-response-is-weak-documents-11631812953
[34] https://www.lawfaremedia.org/article/meta-s-oversight-board-and-the-need-for-a-new-theory-of-online-speech
[35] https://erinkissane.com/meta-in-myanmar-part-iii-the-inside-view
[36] https://www.youtube.com/watch?v=Ml8kf3UIpN0
[37] https://www.eff.org/cyberspace-independence
[38] https://knightcolumbia.org/content/protocols-not-platforms-a-technological-approach-to-free-speech
[39] https://www.wrecka.ge/#/portal
[40] https://fediverse-governance.github.io/
[41] https://www.wired.com/story/blacksky-is-nothing-like-black-twitter/
[42] https://www.ucpress.edu/books/governable-spaces/epub-pdf
[43] https://archive.org/details/internationalma171912newy/page/198/mode/2up
[44] https://tootpick.org/#text=%22Bad%20shape%22%20https://www.wrecka.ge/bad-shape/
[45] https://bsky.app/intent/compose?text=%22Bad%20shape%22+https://www.wrecka.ge/bad-shape/
[46] javascript:
[48] https://www.wrecka.ge/how-we-do-it-here/
[49] https://www.wrecka.ge/how-we-do-it-here/
[50] https://www.wrecka.ge/what-people-in-the-global-majority-need-from-networks/
[51] https://www.wrecka.ge/what-people-in-the-global-majority-need-from-networks/
[56] https://www.wrecka.ge/