davideisinger.com/static/archive/softwarecrisis-dev-7c7z9g.txt

[1]Out of the Software Crisis Bird flying logo [2]Newsletter [3]Book [4]AI Book
[5]Archive [6]Author

Modern software quality, or why I think using language models for programming
is a bad idea

By Baldur Bjarnason,
May 30th, 2023

This essay is based on a talk I gave at [7]Hakkavélin, a hackerspace in
Reykjavík. I had a wonderful time presenting to a lovely crowd, full of
inquisitive and critically-minded people. Their questions and the discussion
afterwards led to a number of improvements and clarifications as I turned my
notes into this letter. This resulted in a substantial expansion of this essay.
Many of the expanded points, such as the ones surrounding language model
security, come directly from these discussions.

Many thanks to all of those who attended. The references for the presentation
are also the references for this essay, which you can find all the way down in
the footnotes section.

The best way to support this newsletter or my blog is to buy one of my books, 
[8]The Intelligence Illusion: a practical guide to the business risks of
Generative AI or [9]Out of the Software Crisis. Or, you can buy them both [10]
as a bundle.

The software industry is very bad at software

Here’s a true story. Names withheld to protect the innocent.

A chain of stores here in Iceland recently upgraded their point-of-sale
terminals to use new software.

Disaster, obviously, ensued. The barcode scanner stopped working properly,
leading customer to be either overcharged or undercharged. Everything was
extremely slow. The terminals started to lock up regularly. The new invoice
printer sucked. A process that had been working smoothly was now harder and
took more time.

The store, where my “informant” is a manager, deals with a lot of businesses,
many of them stores. When they explain to their customers why everything is
taking so long, their answer is generally the same:

“Ah, software upgrade. The same happened to us when we upgraded our terminals.”

This is the norm.

The new software is worse in every way than what it’s replacing. Despite having
a more cluttered UI, it seems to have omitted a bunch of important features.
Despite being new and “optimised”, it’s considerably slower than what it’s
replacing.

This is also the norm.

Switching costs are, more often than not, massive for business software, and
purchases are not decided by anybody who actually uses it. The quality of the
software disconnects from sales performance very quickly in a growing software
company. The company ends up “owning” the customer and no longer has any
incentive to improve the software. In fact, because adding features is a key
marketing and sales tactic, the software development cycle becomes an act of
intentional, controlled deterioration.

Enormous engineering resources go into finding new ways to minimise the
deterioration—witness Microsoft’s “ribbon menu”, a widget invented entirely to
manage the feature escalation mandated by marketing.

This is the norm.

This has always been the norm, from the early days of software.

The software industry is bad at software. Great at shipping features and
selling software. Bad at the software itself.

Why I started researching “AI” for programming

In most sectors of the software industry, sales performance and product quality
are disconnected.

By its nature software has enormous margins which further cushion it from the
effect of delivering bad products.

The objective impact of poor software quality on the bottom lines of companies
like Microsoft, Google, Apple, Facebook, or the retail side of Amazon is a
rounding error. The rest only need to deliver usable early versions, but once
you have an established customer base and an experienced sales team, you can
coast for a long, long time without improving your product in any meaningful
way.

You only need to show change. Improvements don’t sell, it’s freshness that
moves product. It’s like store tomatoes. Needs to look good and be fresh.
They’re only going to taste it after they’ve paid, so who cares about the
actual quality.

Uptime reliability is the only quality measurement with a real impact on ad
revenue or the success of enterprise contracts, so that’s the only quality
measurement that ultimately matters to them.

Bugs, shoddy UX, poor accessibility—even when accessibility is required by
law—are non-factors in modern software management, especially at larger
software companies.

The rest of us in the industry then copy their practices, and we mostly get
away with it. Our margins may not be as enormous as Google’s, but they are
still quite good compared to non-software industries.

We have an industry that’s largely disconnected from the consequences of making
bad products, which means that we have a lot of successful but bad products.

The software crisis

Research bears this out. I pointed out in my 2021 essay [11]Software Crisis 2.0
that very few non-trivial software projects are successful, even when your
benchmarks are fundamentally conservative and short term.

For example, the following table is from [12]a 2015 report by the Standish
Group on their long term study in software project success:

         SUCCESSFUL CHALLENGED FAILED TOTAL
 Grand   6%         51%        43%    100%
 Large   11%        59%        30%    100%
 Medium  12%        62%        26%    100%
Moderate 24%        64%        12%    100%
 Small   61%        32%        7%     100%

The Chaos Report 2015 resolution by project size

This is based on data that’s collected and anonymised from a number of
organisations in a variety of industries. You’ll note that very few projects
outright succeed. Most of them go over budget or don’t deliver the
functionality they were supposed to. A frightening number of large projects
outright fail to ship anything usable.

In my book [13]Out of the Software Crisis, I expanded on this by pointing out
that there are many classes and types of bugs and defects that we don’t measure
at all, many of them catastrophic, which means that these estimates are
conservative. Software project failure is substantially higher than commonly
estimated, and success if much rarer than the numbers would indicate.

The true percentage of large software projects that are genuinely successful in
the long term—that don’t have any catastrophic bugs, don’t suffer from UX
deterioration, don’t end up having core issues that degrade their business
value—is probably closer to 1–3%.

The management crisis

We also have a management crisis.

The methods of top-down-control taught to managers are counterproductive for
software development.

  • Managers think design is about decoration when it’s the key to making
    software that generates value.
  • Trying to prevent projects that are likely to fail is harmful for your
    career, even if the potential failure is wide-ranging and potentially
    catastrophic.
  • When projects fail, it’s the critics who tried to prevent disaster who are
    blamed, not the people who ran it into the ground.
  • Supporting a project that is guaranteed to fail is likely to benefit your
    career, establish you as a “team player”, and protects you from harmful
    consequences when the project crashes.
  • Teams and staff management in the software industry commonly ignores every
    innovation and discovery in organisational psychology, management, and
    systems-thinking since the early sixties and operate mostly on management
    ideas that Henry Ford considered outdated in the 1920s.

We are a mismanaged industry that habitually fails to deliver usable software
that actually solves the problems it’s supposed to.

Thus, [14]Weinberg’s Law:

    If builders built buildings the way programmers wrote programs, then the
    first woodpecker that came along would destroy civilization.

It’s into this environment that “AI” software development tools appear.

The punditry presented it as a revolutionary improvement in how we make
software. It’s supposed to fix everything.

—This time the silver bullet will work!

Because, of course, we have had such a great track record with [15]silver
bullets.

So, I had to dive into it, research it, and figure out how it really worked. I
needed to understand how generative AI works, as a system. I haven’t researched
any single topic to this degree since I finished my PhD in 2006.

This research led me to write my book [16]The Intelligence Illusion: a
practical guide to the business risks of Generative AI. In it, I take a broader
view and go over the risks I discovered that come with business use of
generative AI.

But, ultimately, all that work was to answer the one question that I was
ultimately interested in:

Is generative AI good or bad for software development?

To even have a hope of answering this, we first need to define our terms,
because the conclusion is likely to vary a lot depending on how you define “AI”
or even "software development.

A theory of software development as an inclusive system

Software development is the entire system of creating, delivering, and using a
software project, from idea to end-user.

That includes the entire process on the development side—the idea, planning,
management, design, collaboration, programming, testing, prototyping—as well as
the value created by the system when it has been shipped and is being used.

My model is that of [17]theory-building. From my essay on theory-building,
which itself is an excerpt from [18]Out of the Software Crisis:

    Beyond that, software is a theory. It’s a theory about a particular
    solution to a problem. Like the proverbial garden, it is composed of a
    microscopic ecosystem of artefacts, each of whom has to be treated like a
    living thing. The gardener develops a sense of how the parts connect and
    affect each other, what makes them thrive, what kills them off, and how you
    prompt them to grow. The software project and its programmers are an
    indivisible and organic entity that our industry treats like a toy model
    made of easily replaceable lego blocks. They believe a software project and
    its developers can be broken apart and reassembled without dying.

    What keeps the software alive are the programmers who have an accurate
    mental model (theory) of how it is built and works. That mental model can
    only be learned by having worked on the project while it grew or by working
    alongside somebody who did, who can help you absorb the theory. Replace
    enough of the programmers, and their mental models become disconnected from
    the reality of the code, and the code dies. That dead code can only be
    replaced by new code that has been ‘grown’ by the current programmers.

Design and user research is an integral part of the mental model the programmer
needs to build, because none of the software components ultimately make sense
without the end-user.

But, design is also vital because it is, to reuse Donald G. Reinertsen’s
definition from Managing the Design Factory (p. 11), design is economically
useful information that generally only becomes useful information through
validation of some sort. Otherwise it’s just a guess.

The economic part usually comes from the end-user in some way.

This systemic view is inclusive by design as you can’t accurately measure the
productivity or quality of a software project unless you look at it end to end,
from idea to end-user.

  • If it doesn’t work for the end-user, then it’s a failure.
  • If the management is dysfunctional, then the entire system is
    dysfunctional.
  • If you keep starting projects based on unworkable ideas, then your
    programmer productivity doesn’t matter.

Lines of code isn’t software development. Working software, productively used,
understood by the developers, is software development.

A high-level crash course in language models

Language models, small or large, are today either used as autocomplete copilots
or as chatbots. Some of these language model tools would be used by the
developer, some by the manager or other staff.

I’m treating generative media and image models as a separate topic, even when
they’re used by people in the software industry to generate icons, graphics, or
even UIs. They matter as well, but don’t have the same direct impact on
software quality.

To understand the role these systems could play in software development, we
need a little bit more detail on what language models are, how they are made,
and how they work.

Most modern machine learning models are layered networks of parameters, each
representing its connection to its neighbouring parameters. In a modern
transformer-based language model most of these parameters are floating point
numbers—weights—that describe the connection. Positive numbers are an
excitatory connection. Negative numbers are inhibitory.

These models are built by feeding data through a tokeniser that breaks text
into tokens—often one word per token—that are ultimately fed into an algorithm.
That algorithm constructs the network, node by node, layer by layer, based on
the relationships it calculates between the tokens/words. This is done in
several runs and, usually, the developer of the model will evaluate after each
run that the model is progressing in the right direction, with some doing more
thorough evaluation at specific checkpoints.

The network is, in a very fundamental way, a mathematical derivation of the
language in the data.

A language model is constructed from the data. The transformer code regulates
and guides the process, but the distributions within the data set are what
defines the network.

This process takes time—both collecting and managing the data set and the build
process itself—which inevitably introduces a cut-off point for the data set.
For OpenAI and Anthropic, that cut-off point is in 2021. For Google’s PaLM2
it’s early 2023.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Aside: not a brain

This is very, very different from how a biological neural network interacts
with data. A biological brain is modified by input and data—its environment—but
its construction is derived from nutrition, its chemical environment, and
genetics.

The data set, conversely, is a deep and fundamental part of the language model.
The algorithm’s code provides the process while the weights themselves are
derived from the data, and the model itself is dead and static during input and
output.

The construction process of a neural network is called “training”, which is yet
another incredibly inaccurate term used by the industry.

  • A pregnant mother isn’t “training” the fetus.
  • A language model isn’t “trained” from the data, but constructed.

This is nonsense.

But this is the term that the AI industry uses, so we’re stuck with it.

A language model is a mathematical model built as a derivation of its training
data. There is no actual training, only construction.

This is also why it’s inaccurate to say that these systems are inspired by
their training data. Even though genes and nutrition make an artist’s mind they
are not in what any reasonable person would call “their inspiration”. Even when
they are sought out for study and genuine inspiration, it’s our representations
of our understanding of the genes that are the true source of inspiration.
Nobody sticks their hand in a gelatinous puddle of DNA and spontaneously gets
inspired by the data it encodes.

Training data are construction materials for a language models. A language
model can never be inspired. It is itself a cultural artefact derived from
other cultural artefacts.

The machine learning process is loosely based on decades-old grossly simplified
models of how brains work.

A biological neuron is a complex system in its own right—one of the more
complex cells in an animal’s body. In a living brain, a biological neuron will
use electricity, multiple different classes of neurotransmitters, and timing to
accomplish its function in ways that we still don’t fully understand. It even
has its own [19]built-in engine for chemical energy.

The brain as a whole is composed of not just a massive neural network, but also
layers of hormonal chemical networks that dynamically modify its function, both
granularly and as a whole.

The digital neuron—a single signed floating point number—is to a biological
neuron what a flat-head screwdriver is to a Tesla.

They both contain metal and that’s about the extent of their similarity.

The human brain contains roughly 100 billion neuron cells, a layered chemical
network, and a cerebrovascular system that all integrate as a whole to create a
functioning, self-aware system capable of general reasoning and autonomous
behaviour. This system is multiple orders of magnitude more complex than even
the largest language model to date, both in terms of individual neuron
structure, and taken as a whole.

It’s important to remember this so that we don’t fall for marketing claims that
constantly imply that these tools are fully functioning assistants.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The prompt

After all of this, we have a data set which can be used to generate text in
response to prompts.

Prompts such as:

    Who was the first man on the moon?

The input phrase, or prompt, has no structure beyond the linguistic. It’s just
a blob of text. You can’t give the model commands or parameters separately from
other input. Because of this, if your model lets a third party enter text, an
attacker will always be able to bypass whatever restrictions you put on it.
Control prompts or prefixes will be discovered and countermanded. Delimiters
don’t work. Fine-tuning the model only limits the harm, but doesn’t prevent it.

This is called a prompt injection and what it means is that model input can’t
be secured. You have to assume that anybody that can send text to the model has
full access to it.

Language models need to be treated like an unsecured client and only very
carefully integrated into other systems.

The response

What you’re likely to get back from that prompt would be something like:

    On July 20, 1969, Neil Armstrong became the first human to step on the
    moon.

This is NASA’s own phrasing. Most answers on the web are likely to be
variations on this, so the answer from a language model is likely to be so too.

  • The moon landing happens to be a fact, but the language model only knows it
    as a text.

The prompt we provided is strongly associated in the training data set with
other sentences that are all variations of NASA’s phrasing of the answer. The
model won’t answer with just “Neil Armstrong” because it isn’t actually
answering the question, it’s responding with the text that correlates with the
question. It doesn’t “know” anything.

  • The language model is fabricating a mathematically plausible response,
    based on word distributions in the training data.
  • There are no facts in a language model or its output. Only memorised text.

It only fabricates. It’s all “hallucinations” all the way down.

Occasionally those fabrications correlate with facts, but that is a
mathematical quirk resulting from the fact that, on average, what people write
roughly correlates with their understanding of a factual reality, which in turn
roughly correlates with a factual reality.

A knowledge system?

To be able to answer that question and pass as a knowledge system, the model
needs to memorise the answer, or at least parts of the phrase.

Because “AI” vendors are performing a sleight-of-hand here and presenting
statistical language synthesis engines as knowledge retrieval systems, their
focus in training and testing is on “facts” and minimising “falsehoods”. The
model has no notion of either, as it’s entirely a language model, so the only
way to square this circle is for the model to memorise it all.

  • To be able to answer a question factually, not “hallucinate”, and pass as a
    knowledge system, the model needs to memorise the answer.
  • The model doesn’t know facts, only text.
  • If you want a fact from it, the model will need to memorise text that
    correlates with that fact.

“Dr. AI”?

Vendors then compound this by using human exams as benchmarks for reasoning
performance. The problem is that bar exams, medical exams, and diagnosis tests
are specifically designed to mostly test rote memorisation. That’s what they’re
for.

The human brain is bad at rote memorisation and generally it only happens with
intensive work and practice. If you want to design a test that’s specifically
intended to verify that somebody has spent a large amount of time studying a
subject, you test for rote memorisation.

Many other benchmarks they use, such as those related to programming languages
also require memorisation, otherwise the systems would just constantly make up
APIs.

  • Vendors use human exams as benchmarks.
  • These are specifically designed to test rote memorisation, because that’s
    hard for humans.
  • Programming benchmarks also require memorisation. Otherwise, you’d only get
    pseudocode.

Between the tailoring of these systems for knowledge retrieval, and the use of
rote memorisation exams and code generation as benchmarks, the tech industry
has created systems where memorisation is a core part of how they function. In
all research to date, memorisation has been key to language model performance
in a range of benchmarks.^[20][1]

If you’re familiar with storytelling devices, this here would be a [21]
Chekhov’s gun. Observe! The gun is above the mantelpiece:

    👉🏻👉🏻 memorisation!

Make a note of it, because those finger guns are going to be fired later.

Biases

Beyond question and answer, these systems are great at generating the averagely
plausible text for a given prompt. In prose, current system output smells
vaguely of sweaty-but-quiet LinkedIn desperation and over-enthusiastic social
media. The general style will vary, but it’s always going to be the most
plausible style and response based on the training data.

One consequence of how these systems are made is that they are constantly
backwards-facing. Where brains are focused on the present, often to their
detriment, “AI” models are built using historical data.

The training data encompasses thousands of diverse voices, styles, structures,
and tones, but some word distributions will be more common in the set than
others and those will end up dominating the output. As a result, language
models tend to lean towards the “racist grandpa who has learned to speak fluent
LinkedIn” end of the spectrum.^[22][2]

This has implications for a whole host of use cases:

  • Generated text is going to skew conservative in content and marketing copy
    in structure and vocabulary. (Bigoted, prejudiced, but polite and
    inoffensively phrased.)
  • Even when the cut-off date for the data set is recent, it’s still going to
    skew historical because what’s new is also comparatively smaller than the
    old.
  • Language models will always skew towards the more common, middling,
    mediocre, and predictable.
  • Because most of these models are trained on the web, much of which is
    unhinged, violent, pornographic, and abusive, some of that language will be
    represented in the output.

Modify, summarise, and “reason”

The superpower that these systems provide is conversion or modification. They
can, generally, take text and convert it to another style or structure. Take
this note and turn it into a formal prose, and it will! That’s amazing. I don’t
think that’s a trillion-dollar industry, but it’s a neat feature that will
definitely be useful.

They can summarise text too, but that’s much less reliable than you’d expect.
It unsurprisingly works best with text that already provides its own summary,
such as a newspaper article (first paragraphs always summarise the story),
academic paper (the abstract), or corporate writing (executive summary).
Anything that’s a mix of styles, voices, or has an unusual structure won’t work
as well.

What little reasoning they do is entirely based on finding through correlation
and re-enacting prior textual descriptions of reasoning. They fail utterly when
confronted with adversarial or novel examples. They also fail if you rephrase
the question so that it no longer correlates with the phrasing in the data set.
^[23][3]

So, not actual reasoning. “Reasoning”, if you will. In other “AI” model genres
these correlations are often called “shortcuts”, which feels apt.

To summarise:

  • Language models are a mathematical expression of the training data set.
  • Have very little in common with human brains.
  • Rely on inputs that can’t be secured.
  • Lie. Everything they output is a fabrication.
  • Memorise heavily.
  • Great for modifying text. No sarcasm. Genuinely good at this.
  • Occasionally useful for summarisation if you don’t mind being lied to
    regularly.
  • Don’t actually reason.

Why I believe “AI” for programming is a bad idea

If you recall from the start of this essay, I began my research into machine
learning and language models because I was curious to see if they could help
fix or improve the mess that is modern software development.

There was reason to be hopeful. Programming languages are more uniform and
structured than prose, so it’s not too unreasonable to expect that they might
lend themselves to language models. Programming language output can often be
tested directly, which might help with the evaluation of each training run.

Training a language model on code also seems to benefit the model. Models that
include substantial code in their data set tend to be better at correlative
“reasoning” (to a point, still not actual reasoning), which makes sense since
code is all about representing structured logic in text.

But, there is an inherent [24]Catch 22 to any attempt at fixing software
industry dysfunction with more software. The structure of the industry depends
entirely on variables that everybody pretends are proxies for end user value,
but generally aren’t. This will always tend to sabotage our efforts at
industrial self-improvement.

The more I studied language models as a technology the more flaws I found until
it became clear to me that odds are that the overall effect on software
development will be harmful. The problem starts with the models themselves.

1. Language models can’t be secured

This first issue has less to do with the use of language models for software
development and more to do with their use in software products, which is likely
to be a priority for many software companies over the next few years.

Prompt injections are not a solved problem. OpenAI has come up with a few
“solutions” in the past, but none of them actually worked. Everybody expects
this to be fixed, but nobody has a clue how.

Language models are fundamentally based on the idea that you give it text as
input and get text as output. It’s entirely possible that the only way to
completely fix this is to invent a completely new kind of language model and
spend a few years training it from scratch.

  • A language model needs to be treated like an unsecured client. It’s about
    as secure as a web page form. It’s vulnerable to a new generation of
    injection vulnerabilities, both direct and indirect, that we still don’t
    quite understand.^[25][4]

The training data set itself is also a security hazard. I’ve gone into this in
more detail elsewhere^[26][5], but the short version is that training data set
is vulnerable to keyword manipulation, both in terms of altering sentiment and
censorship.

Again, fully defending against this kind of attack would seem to require
inventing a completely new kind of language model.

Neither of these issues affect the use of language models for software
development, but it does affect our work because we’re the ones who will be
expected to integrate these systems into existing websites and products.

2. It encourages the worst of our management and development practices

A language model will never question, push back, doubt, hesitate, or waver.

Your managers are going to use it to flesh out and describe unworkable ideas,
and it won’t complain. The resulting spec won’t have any bearing with reality.

People on your team will do “user research” by asking a language model, which
it will do even though the resulting research will be fiction and entirely
useless.

It’ll let you implement the worst ideas ever in your code without protest. Ask
a copilot “how can I roll my own cryptography?” and it’ll regurgitate a
half-baked expression of sha1 in PHP for you.

Think of all the times you’ve had an idea for an approach, looked up how to do
it on the web, and found out that, no, this was a really bad idea? I have a
couple of those every week when I’m in the middle of a project.

Language models don’t deliver productivity improvements. They increase the
volume, unchecked by reason.

A core aspect of the theory-building model of software development is code that
developers don’t understand is a liability. It means your mental model of the
software is inaccurate which will lead you to create bugs as you modify it or
add other components that interact with pieces you don’t understand.

Language model tools for software development are specifically designed to
create large volumes of code that the programmer doesn’t understand. They are
liability engines for all but the most experienced developer. You can’t solve
this problem by having the “AI” understand the codebase and how its various
components interact with each other because a language model isn’t a mind. It
can’t have a mental model of anything. It only works through correlation.

These tools will indeed make you go faster, but it’s going to be accelerating
in the wrong direction. That is objectively worse than just standing still.

3. Its User Interfaces do not work, and we haven’t found interfaces that do
work

Human factors studies, the field responsible for designing cockpits and the
like, discovered that humans suffer from an automation bias.

What it means is that when you have cognitive automation—something that helps
you think less—you inevitably think less. That means that you are less critical
of the output than if you were doing it yourself. That’s potentially
catastrophic when the output is code, especially since the quality of the
generated code is, understandably considering how the system works, broadly on
the level of a novice developer.^[27][6]

Copilots and chatbots—exacerbated by anthropomorphism—seem to trigger our
automation biases.

Microsoft themselves have said that 40% of GitHub Copilot’s output is committed
unchanged.^[28][7]

Let’s not get into the question of how we, as an industry, put ourselves in the
position where Microsoft can follow a line of code from their language model,
through your text editor, and into your supposedly decentralised version
control system.

People overwhelmingly seem to trust the output of a language model.

If it runs without errors, it must be fine.

But that’s never the case. We all know this. We’ve all seen running code turn
out to be buggy as hell. But something in our mind switches off when we use
tools for cognitive automation.

4. It’s biased towards the stale and popular

The biases inherent in these language models are bad enough when it comes to
prose, but they become a functional problem in code.

  • Its JS code will lean towards React and node, most of it several versions
    old, and away from the less popular corners of the JS ecosystem.
  • The code is, inevitably, more likely to be built around CommonJS modules
    instead of the modern ESM modules.
  • It won’t know much about Deno or Cloudflare Workers.
  • It’ll always prefer older APIs over new. Most of these models won’t know
    about any API or module released after 2021. This is going to be an issue
    for languages such as Swift.
  • New platforms and languages don’t exist to it.
  • Existing data will outweigh deprecations and security issues.
  • Popular but obsolete or outdated open source projects will always win out
    over the up-to-date equivalent.

These systems live in the popular past, like the middle-aged man who doesn’t
realise he isn’t the popular kid at school any more. Everything he thinks is
cool is actually very much not cool. More the other thing.

This is an issue for software because our industry is entirely structured
around constant change. Software security hinges on it. All of our practices
are based on constant march towards the new and fancy. We go from framework to
framework to try and find the magic solution that will solve everything. In
some cases language models might help push back against that, but it’ll also
push back against all the very many changes that are necessary because the old
stuff turned out to be broken.

  • The software industry is built on change.
  • Language models are built on a static past.

5. No matter how the lawsuits go, this threatens the existence of free and open
source software

Many AI vendors are mired in lawsuits.^[29][8]

These lawsuits all concentrate on the relationship between the training data
set and the model and they do so from a variety of angles. Some are based on
contract and licensing law. Others are claiming that the models violate fair
use. It’s hard to predict how they will go. They might not all go the same way,
as laws will vary across industries and jurisdictions.

No matter the result, we’re likely to be facing a major decline in the free and
open source ecosystem.

 1. All of these models are trained on open source code without payment or even
    acknowledgement, which is a major disincentive for contributors and
    maintainers. That large corporations might benefit from your code is a
    fixture of open source, but they do occasionally give back to the
    community.
 2. Language models—built on open source code—commonly replace that code.
    Instead of importing a module to do a thing, you prompt your Copilot. The
    code generated is almost certainly based on the open source module, at
    least partially, but it has been laundered through the language model,
    disconnecting the programmer from the community, recognition, and what
    little reward there was.

Language models demotivate maintainers and drain away both resources and users.
What you’re likely to be left with are those who are building core
infrastructure or end-user software out of principle. The “free software” side
of the community is more likely to survive than the rest. The Linux kernel,
Gnome, KDE—that sort of thing.

The “open source” ecosystem, especially that surrounding the web and node, is
likely to be hit the hardest. The more driven the open source project was by
its proximity to either an employed contributor or actively dependent business,
the bigger the impact from a shift to language models will be.

This is a serious problem for the software industry as arguably much of the
economic value the industry has provided over the past decade comes from
strip-mining open source and free software.

6. Licence contamination

Microsoft and Google don’t train their language models on their own code.
GitHub’s Copilot isn’t trained on code from Microsoft’s office suite, even
though many of its products are likely to be some of the largest React Native
projects in existence. There aren’t many C++ code bases as big as Windows.
Google’s repository is probably one of the biggest collection of python and
java code you can find.

They don’t seem to use it for training, but instead train on collections of
open source code that contain both permissive and copyleft licences.

Copyleft licences, if used, force you to release your own project under their
licence. Many of them, even non-copyleft, have patent clauses, which is poison
for quite a few employers. Even permissive licences require attribution, and
you can absolutely get sued if you’re caught copying open source code without
attribution.

Remember our Chekhov’s gun?

    👉🏻👉🏻 memorisation!

Well, 👉🏻👉🏻 pewpew!!!

Turns out blindly copying open source code is problematic. Whodathunkit?

These models all memorise a lot, and they tend to copy what they memorise into
their output. [30]GitHub’s own numbers peg verbatim copies of code that’s at
least 150 characters at 1%^[31][9], which is roughly the same, in terms of
verbatim copying, as what you seem to get in other language models.

For context, that means that if you use a language model for development, a
copilot or chatbot, three or four times a day, you’re going to get a verbatim
copy of open source code injected into your project about once a month. If
every team member uses one, then multiply that by the size of the team.

GitHub’s Copilot has a feature that lets you block verbatim copies. This
obviously requires both a check, which slows the result down, and it will throw
out a bunch of useful results, making the language model less useful. It’s
already not as useful as it’s made out to be and pretty darn slow so many
people are going to turn off the “please don’t plagiarise” checkbox.

But even GitHub’s checks are insufficient. The keyword there is “verbatim”,
because language models have a tendency to rephrase their output. If GitHub
Copilot copies a GPLed implementation of an algorithm into your project but
changes all the variable names, Copilot won’t detect it, it’ll still be
plagiarism and the copied code is still under the GPL. This isn’t unlikely as
this is how language models work. Memorisation and then copying with light
rephrasing is what they do.

Training the system only on permissively licensed code doesn’t solve the
problem. It won’t force your project to adopt an MIT licence or anything like
that, but you can still be sued if it’s discovered.

This would seem to give Microsoft and GitHub a good reason not to train on the
Office code base, for example. If they did, there’s a good chance that a prompt
to generate DOCX parsing code might “generate” a verbatim copy of the DOCX
parsing code from Microsoft Word.

And they can’t have that, can they? This would both undercut their own
strategic advantage, and it would break the illusion that these systems are
generating novel code from scratch.

This should make it clear that what they’re actually doing is strip-mine the
free and open source software ecosystem.

How much of a problem is this?

—It won’t matter. I won’t get caught.

You personally won’t get caught, but your employer might, and Intellectual
Property scans or similar code audits tend to come up at the absolute worst
moments in the history of any given organisation:

  • During due diligence for an acquisition. Could cost the company and
    managers a fortune.
  • In discovery for an unrelated lawsuit. Again, could cost the company a
    fortune.
  • During hacks and other security incidents. Could. Cost. A. Fortune.

“AI” vendors won’t take any responsibility for this risk. I doubt your business
insurance covers “automated language model plagiarism” lawsuits.

Language models for software development are a lawsuit waiting to happen.

Unless they are completely reinvented from scratch, language model code
generators are, in my opinion, unsuitable for anything except for prototypes
and throwaway projects.

So, obviously, everybody’s going to use them

  • All the potentially bad stuff happens later. Unlikely to affect your
    bonuses or employment.
  • It’ll be years before the first licence contamination lawsuits happen.
  • Most employees will be long gone before anybody realises just how much of a
    bad idea it was.
  • But you’ll still get that nice “AI” bump in the stock market.

What all of these problems have in common is that their impact is delayed and
most of them will only appear in the form of increased frequency of bugs and
other defects and general project chaos.

The biggest issue, licence contamination, will likely take years before it
starts to hit the industry, and is likely to be mitigated by virtue of the fact
that many of the heaviest users of “AI”-generated code will have folded due to
general mismanagement long before anybody cares enough to check their code.

If you were ever wondering if we, as an industry, were capable of coming up
with a systemic issue to rival the Y2K bug in scale and stupidity? Well, here
you go.

You can start using a language model, get the stock market bump, present the
short term increase in volume as productivity, and be long gone before anybody
connects the dots between language model use and the jump in defects.

Even if you purposefully tried to come up with a technology that played
directly into and magnified the software industry’s dysfunctions you wouldn’t
be able to come up with anything as perfectly imperfect as these language
models.

It’s nonsense without consequence.

Counterproductive novelty that you can indulge in without harming your career.

It might even do your career some good. Show that you’re embracing the future.

But…

The best is yet to come

In a few years’ time, once the effects of the “AI” bubble finally dissipates…

Somebody’s going to get paid to fix the crap it left behind.

The best way to support this newsletter or my blog is to buy one of my books, 
[32]The Intelligence Illusion: a practical guide to the business risks of
Generative AI or [33]Out of the Software Crisis. Or, you can buy them both [34]
as a bundle.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

 1. There’s quite a bit of papers that either highlight the tendency to
    memorise or demonstrate a strong relationship between that tendency and
    eventual performance.

      □ [35]An Empirical Study of Memorization in NLP (Zheng & Jiang, ACL 2022)
      □ [36]Does learning require memorization? a short tale about a long tail.
        (Feldman, 2020)
      □ [37]When is memorization of irrelevant training data necessary for
        high-accuracy learning? (Brown et al. 2021)
      □ [38]What Neural Networks Memorize and Why: Discovering the Long Tail
        via Influence Estimation (Feldman & Zhang, 2020)
      □ [39]Question and Answer Test-Train Overlap in Open-Domain Question
        Answering Datasets (Lewis et al., EACL 2021)
      □ [40]Quantifying Memorization Across Neural Language Models (Carlini et
        al. 2022)
      □ [41]On Training Sample Memorization: Lessons from Benchmarking
        Generative Modeling with a Large-scale Competition (Bai et al. 2021)
    [42]↩︎
 2. See the [43]Bias & Safety card at [44]needtoknow.fyi for references. [45]↩︎

 3. See the [46]Shortcut “Reasoning” card at [47]needtoknow.fyi for references.
    [48]↩︎

 4. Simon Willison has been covering this issue [49]in a series of blog posts. 
    [50]↩︎

 5.
      □ [51]The poisoning of ChatGPT
      □ [52]Google Bard is a glorious reinvention of black-hat SEO spam and
        keyword-stuffing
    [53]↩︎
 6. See, for example:

      □ [54]Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s
        Code Contributions (Hammond Pearce et al., December 2021)
      □ [55]Do Users Write More Insecure Code with AI Assistants? (Neil Perry
        et al., December 2022)
    [56]↩︎
 7. This came out [57]during an investor event and was presented as evidence of
    the high quality of Copilot’s output. [58]↩︎

 8.
      □ [59]Getty Images v. Stability AI - Complaint
      □ [60]Getty Images is suing the creators of AI art tool Stable Diffusion
        for scraping its content
      □ [61]The Wave of AI Lawsuits Have Begun
      □ [62]Copyright lawsuits pose a serious threat to generative AI
      □ [63]GitHub Copilot litigation
      □ [64]Stable Diffusion litigation
    [65]↩︎
 9. Archived link of the [66]GitHub Copilot feature page. [67]↩︎

Join the Newsletter

Subscribe to the Out of the Software Crisis newsletter to get my weekly (at
least) essays on how to avoid or get out of software development crises.

Join now and get a free PDF of three bonus essays from Out of the Software
Crisis.

[68][                    ]
Subscribe

We respect your privacy.

Unsubscribe at any time.

[70]Mastodon [71]Twitter [72]GitHub [73]Feed

References:

[1] https://softwarecrisis.dev/
[2] https://softwarecrisis.dev/
[3] https://softwarecrisis.baldurbjarnason.com/
[4] https://illusion.baldurbjarnason.com/
[5] https://softwarecrisis.dev/archive/
[6] https://softwarecrisis.dev/author/
[7] https://www.hakkavelin.is/
[8] https://illusion.baldurbjarnason.com/
[9] https://softwarecrisis.baldurbjarnason.com/
[10] https://baldurbjarnason.lemonsqueezy.com/checkout/buy/cfc2f2c6-34af-436f-91c1-cb2e47283c40
[11] https://www.baldurbjarnason.com/2021/software-crisis-2/
[12] https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
[13] https://softwarecrisis.baldurbjarnason.com/
[14] https://quoteinvestigator.com/2019/09/19/woodpecker/
[15] http://worrydream.com/refs/Brooks-NoSilverBullet.pdf
[16] https://illusion.baldurbjarnason.com/
[17] https://www.baldurbjarnason.com/2022/theory-building/
[18] https://softwarecrisis.baldurbjarnason.com/
[19] https://en.wikipedia.org/wiki/Mitochondrion
[20] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn1
[21] https://en.wikipedia.org/wiki/Chekhov's_gun
[22] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn2
[23] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn3
[24] https://en.wikipedia.org/wiki/Catch-22_(logic)
[25] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn4
[26] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn5
[27] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn6
[28] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn7
[29] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn8
[30] https://archive.ph/2023.01.11-224507/https://github.com/features/copilot#selection-19063.298-19063.462:~:text=Our%20latest%20internal%20research%20shows%20that%20about%201%25%20of%20the%20time%2C%20a%20suggestion%20may%20contain%20some%20code%20snippets%20longer%20than%20~150%20characters%20that%20matches%20the%20training%20set.
[31] https://softwarecrisis.dev/letters/ai-and-software-quality/#fn9
[32] https://illusion.baldurbjarnason.com/
[33] https://softwarecrisis.baldurbjarnason.com/
[34] https://baldurbjarnason.lemonsqueezy.com/checkout/buy/cfc2f2c6-34af-436f-91c1-cb2e47283c40
[35] https://aclanthology.org/2022.acl-long.434
[36] https://doi.org/10.1145/3357713.3384290
[37] https://doi.org/10.1145/3406325.3451131
[38] https://papers.nips.cc/paper/2020/hash/1e14bfe2714193e7af5abc64ecbd6b46-Abstract.html
[39] https://aclanthology.org/2021.eacl-main.86
[40] https://arxiv.org/abs/2202.07646
[41] https://dl.acm.org/doi/10.1145/3447548.3467198
[42] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref1
[43] https://needtoknow.fyi/card/bias/
[44] https://needtoknow.fyi/
[45] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref2
[46] https://needtoknow.fyi/card/shortcut-reasoning/
[47] https://needtoknow.fyi/
[48] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref3
[49] https://simonwillison.net/series/prompt-injection/
[50] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref4
[51] https://softwarecrisis.dev/letters/the-poisoning-of-chatgpt/
[52] https://softwarecrisis.dev/letters/google-bard-seo/
[53] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref5
[54] https://doi.org/10.48550/arXiv.2108.09293
[55] https://doi.org/10.48550/arXiv.2211.03622
[56] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref6
[57] https://www.microsoft.com/en-us/Investor/events/FY-2023/Morgan-Stanley-TMT-Conference#:~:text=Scott%20Guthrie%3A%20I%20think%20you%27re,is%20now%20AI%2Dgenerated%20and%20unmodified
[58] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref7
[59] https://copyrightlately.com/pdfviewer/getty-images-v-stability-ai-complaint/?auto_viewer=true#page=&zoom=auto&pagemode=none
[60] https://www.theverge.com/2023/1/17/23558516/ai-art-copyright-stable-diffusion-getty-images-lawsuit
[61] https://www.plagiarismtoday.com/2023/01/17/the-wave-of-ai-lawsuits-have-begun/
[62] https://www.understandingai.org/p/copyright-lawsuits-pose-a-serious
[63] https://githubcopilotlitigation.com/
[64] https://stablediffusionlitigation.com/
[65] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref8
[66] https://archive.ph/2023.01.11-224507/https://github.com/features/copilot#selection-19063.298-19063.462:~:text=Our%20latest%20internal%20research%20shows%20that%20about%201%25%20of%20the%20time%2C%20a%20suggestion%20may%20contain%20some%20code%20snippets%20longer%20than%20~150%20characters%20that%20matches%20the%20training%20set.
[67] https://softwarecrisis.dev/letters/ai-and-software-quality/#fnref9
[70] https://toot.cafe/@baldur
[71] https://twitter.com/fakebaldur
[72] https://github.com/baldurbjarnason
[73] https://softwarecrisis.dev/feed.xml