   #[1]Blog Posts RSS

   IFRAME: [2]https://www.googletagmanager.com/ns.html?id=GTM-55JC288

   [3]Dan Stroot
   [4]Home[5]About[6]Archive[7]Snippets[8]Uses

   (BUTTON) Toggle Menu

Making Software Last Forever

   Hero image for Making Software Last Forever
   27 min read
   Dan Stroot
   Dan Stroot
   May 25, 2023

   How many of us have bought a new home because our prior home was not
   quite meeting our needs? Maybe we needed an extra bedroom, or wanted a
   bigger backyard? Now, as a thought experiment, assume you couldn't sell
   your existing home. If you bought a new home, you'd have to "retire" or
   "decommission" your prior home (and your investment in it). Does that
   change your thinking?

   Further, imagine you had a team of five people maintaining your prior
   home, improving it, and keeping it updated, for the last ten years.
   You'd have a cumulative investment of 50 person/years in your existing
   home (5 people x 10 years) just in maintenance, on top of the initial
   investment. If each person was paid the equivalent of a software
   developer (we'll use $200k to include benefits, office space,
   leadership, etc.) you'd have an investment just in labor of $10 million
   dollars (50 person/years x $200,000). Would you walk away from that
   investment?

   When companies decide to re-write or replace an existing software
   application, they are making a similar decision. Existing software is
   "retired" or "decommissioned" (along with its cumulative investment).
   Yet the belief that new code is always better than old is patently
   absurd. Old code has weathered and withstood the test of time. It has
   been battle-tested. You know it's failure modes. Bugs have been found,
   and more importantly, fixed.

   Joel Spolsky (of Fog Creek Software and Stack Overflow) describes
   system re-writes in "[9]Things You Should Never Do, Part I" as “the
   single worst strategic mistake that any software company can make.”

   Continuing our home analogy, recent price increases for construction
   materials like lumber, drywall, and wiring (and frankly everything
   else) should, according to Economics 101, cause us to treat our current
   homes more dearly. Similarly, price increases for quality software
   engineers should force companies to treat existing software more
   dearly.

   Lots of current software started out as C software from the 1980s.
   Engineers don't often write software with portability as a goal at the
   beginning, but once something is relatively portable, it tends to stay
   that way. Code that was well designed and written often migrated from
   mini-computers to i386, from i386 to amd64, and now ARM and arch64,
   with a minimum of redesign or effort. You can take large, complicated
   programs from the 1980s written in C, and compile/run them on a modern
   Linux computer - even when the modern computer is running architectures
   which hadn't even been dreamt of when the software was originally
   written.

   Why can't software last forever? It's not made of wood, concrete, or
   steel. It doesn't "wear out", rot, weather, or rust. A working
   algorithm is a working algorithm. Technology doesn’t need to be
   beautiful, or impress other people, to be effective. Aren't
   technologists ultimately in the business of producing cost effective
   technology?

   I am going to attempt to convince you that maintaining your existing
   systems is one the most cost-effective technology investments you can
   make.

The World's Oldest Software Systems

   In 1958, the United States Department of Defense launched a new
   computer-based contract management system called "Mechanization of
   Contract Administration Services", or MOCAS (pronounced “MOH-cass”). In
   2015, [10]MIT Technology Review stated that MOCAS was the oldest
   computer program in continuous use they could verify. At that time
   MOCAS managed about $1.3 trillion in government obligations and 340,000
   contracts.

   According to the [11]Guinness Book of World Records, the oldest
   software system in use today is either the [12]SABRE Airline
   Reservation System (introduced in 1960), or the IRS Individual Master
   File (IMF) and Business Master File (BMF) systems introduced in
   1962–63.

   SABRE went online in 1960. It had cost $40 million to develop and
   install (about $400 million in 2022 dollars). The system took over all
   American Airlines booking functions in 1964, and the system was
   expanded to provide access to external travel agents in 1976.

   What is the secret to the long lifespan of these systems? Shouldn't
   companies with long-lived products (annuities, life insurance, etc.)
   study these examples? After all, they need systems to support products
   that last most of a human lifespan. However, shouldn't all companies
   want to their investments in software to last as long as possible?

Maintenance is About Making Something Last

   We spoke of SABRE above, and we know that airlines recognize the value
   of maintenance. Commercial aircraft are inspected at least once every
   two days. Engines, hydraulics, environmental, and electrical systems
   all have additional maintenance schedules. A "heavy" maintenance
   inspection occurs once every few years. This process maintains the
   aircraft's service life over decades.

   On average, an aircraft is operable for about 30 years before it must
   be retired. A Boeing 747 can endure 35,000 pressurization cycles —
   roughly 135,000 to 165,000 flight hours — before metal fatigue sets in.
   However, most older airframes are retired for fuel-efficiency reasons,
   not because they're worn out.

   Even stuctures made of grass can last indefinitely. [13]Inca rope
   bridges were simple suspension bridges constructed by the Inca Empire.
   The bridges were an integral part of the Inca road system were
   constructed using ichu grass.

   Inca Rope Bridge

   Even though they were made of grass, these bridges were maintained with
   such regularity and attention they lasted centuries. The bridge's
   strength and reliability came from the fact that each cable was
   replaced every June.

   The goal of maintenance is catching problems before they happen. That’s
   the difference between maintenance and repair. Repair is about fixing
   something that’s already broken. Maintenance is about making something
   last.

Unfortunately, Maintenance is Chronically Undervalued

   Maintenance is one of the easiest things to cut when budgets get tight.
   Some legacy software systems have decades of underinvestment in
   maintenance. This leads up to the inevitable "we have to replace it"
   discussion - which somehow always sounds more persuasive (even though
   it’s more expensive and riskier) than arguing to invest in system
   rehabilitation and deferred system maintenance.

   Executives generally can't refuse "repair" work because the system is
   broken and must be fixed. However, maintenance is a tougher sell. It’s
   not strictly necessary — or at least it doesn’t seem to be until things
   start falling apart. It is so easy to divert maintenance budget into a
   halo project that gets an executive noticed (and possibly promoted)
   before the long-term effects of underinvestment in maintenance become
   visible. Even worse, the executive is also admired for reducing the
   costs of maintenance and switching costs from "run" to "grow" - while
   they are torpedoing the company under the waterline.

   The other challenge is conflating enhancement work with maintenance
   work. Imagine you have $1,000 and you want to add a sunroof to your
   car, but you also need new tires (which coincidentally also cost
   $1,000). You have to replace the tires every so often, but a sunroof is
   "forever" right? If you spend the money on the sunroof the tires could
   get replaced next month, or maybe the month after - they'll last a
   couple more months, won't they?

   With software, users can't see "the bald tires" - they only thing they
   see, or experience (and value), are new features and capabilities.
   Pressure is always present to cut costs and to add new features. The
   result is budget always swings away from maintenance work towards
   enhancements.

   Finally, maintenance work is typically an operational cost, yet
   building a new system, or a significant new feature, can often be
   capitalized - making the future costs someone else's problem.

Risks of Replacing Software Systems

   It's usually not the design or the age of a system that causes it to
   fail but rather neglect. People fail to maintain software systems
   because they are not given the time, incentives, or resources to
   maintain them.

     "Most of the systems I work on rescuing are not badly built. They
     are badly maintained."
     — Marianne Bellotti, Kill it With Fire

   Once a system degrades it is an enormous challenge to fund deferred
   maintenance (or "technical debt"). No one plans for it, no one wants to
   pay for it, and no engineer wants to do it. Initiatives to restore
   operational excellence, much the way one would fix up an old house,
   tend to have few volunteers among engineering teams. No one gets
   noticed doing maintenance. No one ever gets promoted because of
   maintenance.

   It should be clear why engineers prefer to re-write a system rather
   than maintain it. They get to "write a new story" rather than edit
   someone else's. They will attempt to convince a senior executive to
   fund a project to replace a problematic system by describing all the
   new features and capabilities that could be added as well as how "bad"
   the existing, unmaintained, system has become. Further, they will get
   to use modern technology that makes them much more valuable in the
   market.

   Incentives aside, engineering teams tend to gravitate toward system
   rewrites because they incorrectly think of old systems as specs. They
   assume that since an old system works, the functional risks have been
   eliminated. They can focus on adding more features to the new system or
   make changes to the underlying architecture without worry. Either they
   do not perceive the ambiguity these changes introduce, or they see such
   ambiguity positively, imagining only gains in performance and the
   potential for innovation.

   Why not authorize that multimillion-dollar replacement if the engineers
   convince management the existing system is doomed? Eventually a
   "replacement" project will be funded (typically at a much higher
   expenditure than rehabilitating the existing system). Even if the
   executives are not listening to the engineers, they will be listening
   to external consultants telling them they are falling behind.

   What do you do with the old system while you’re building the new one?
   Most organizations put the old system on “life support” and give it
   only the resources for patches and fixes necessary to keep it running.
   This reduces maintenance even further and becomes a self-fulfilling
   prophecy that the existing system will eventually fail.

   Who gets to work on the new system, and who takes on the maintenance
   tasks of the old system? If the old system is written in older
   technology that the company is actively abandoning, the team
   maintaining the old system is essentially sitting around waiting to be
   fired. And don’t kid yourself, they know it. If the people maintaining
   the old system are not participating in the creation of the new system,
   you should expect that they are also looking for new jobs. If they
   leave before your new system is operational, you lose both their
   expertise and their institutional knowledge.

   If the new project falls behind schedule (and it almost certainly
   will), the existing system continues to degrade, and knowledge
   continues to walk out the door. If the new project fails and is
   subsequently canceled, the gap between the legacy system and
   operational excellence has widened significantly in the meantime.

   This explains why executives are loathe to cancel system replacement
   projects even when they are obviously years behind schedule and failing
   to live up to expectations. Stopping the replacement project seems
   impossible because the legacy system is now so degraded that restoring
   it to operational excellence seems impossible. Plus, politically
   canceling a marquee project can be career suicide for the sponsoring
   executive(s). Much better to do "deep dives" and "assessments" on why
   the project is failing and soldier on than cancel it.

   The interim state is not pretty. The company now has two systems to
   operate, much higher costs and new risks.
     * The new system will have high costs, limited functionality, new and
       unique errors/issues, and lower volumes (so the "per unit cost" of
       the new system will be quite high).
     * The older system will still be running most of the business, and
       usually all of the complex business, while having lost its best
       engineers and subject matter experts. Its maintenance budget will
       have been whittled down to nothing to redirect spending to
       implement (save?) the new system. This system will be in grave
       danger to significant system failure (which proponents of the new
       system will use to justify the investment in the new system, not
       admitting to a self-fulfilling prophecy).

   Neither system will exhibit operational excellence, and both put the
   organization at significant risk in addition to the higher costs and
   complexity of running two systems.

Maintaining Software to Last Forever

   As I discussed in [14]How Software Learns, software adapts over time -
   as it is continually refined and reshaped by maintenance and
   enhancements. Maintenance is crucial to software's lifespan and
   business relevance/value. When software systems are first developed,
   they are based on a prediction of the future - a prediction of the
   future that we know is wrong even as we make it. No set of requirements
   have ever been perfect. However, all new systems become "less wrong" as
   time, experience, and knowledge are continually added (e.g.,
   maintenance).

   Futureproofing means constantly rethinking and iterating on the
   existing system. We know from both research and experience that
   iterating and maintaining existing solutions is a much more likely, and
   less expensive, way to improve software's lifespan and functionality.

   Before choosing to replace a system that needs deferred maintenance
   remember it’s the lack of maintenance that create the impression that
   failure is inevitable, and pushes otherwise rational engineers and
   executives toward rewrites or replacements. What mechanisms will
   prevent lack of maintenance from eventually dooming the brand-new
   system? Has the true root problem been addressed?

   Robust maintenance practices could preserve software for decades, but
   first maintenance must be valued, funded, and applied. To maintain
   software properly we have to consider:
    1. How do you measure the overall health of a system?
    2. How do you define and manage maintenance work?
    3. How do you define a reasonable maintenance budget? How can you
       protect that budget?
    4. How do you motivate engineers to perform maintenance?

1. How do you measure the overall health of a system?

Objective measures

    1. Maintenance Backlog — If you added up all the open work requests,
       including work the software engineers deem necessary to eliminate
       technical debt, what is the total amount of effort? Now, divide
       that by the team capacity. For example, imagine you have a total
       amount of work of 560 days, and you have one person assigned to
       support the system - they work approximately 200 days annually. The
       backlog in days in 560, but in time it is 2.8 years (560 days / 200
       days/year = 2.8 years). What is a reasonable amount of backlog
       time?
    2. System Reliability/Downtime — If you added up all the time the
       system is down in a given period, what is the total amount? What is
       the user or customer impact of that downtime? Conversely, what
       would reducing that downtime be worth? What is the relationship of
       maintenance and downtime? In other words, does the system need to
       be taken down to maintain it (planned maintenance)? Does planned
       maintenance reduce unplanned downtime?
    3. Capacity/Performance Constraints — Is the existing hitting capacity
       constraints that will prevent future growth of the business? How
       unpredictable are the system capacity demands? What is the customer
       experience when the system capacity is breached? What is
       relationship between hardware and software that constrains the
       system? Is the software performant? Can hardware solve the problem?

Subjective measures

    1. User Satisfaction: User satisfaction includes both how happy your
       employees are with the applications and/or how well those
       applications meet your customer's needs. Many times I have found
       the technology team and the business users arguing over "bug" vs.
       "enhancement". It is a way of assigning blame. "Bug" means its
       engineering's fault, "enhancement" means it was a missed
       requirement. When emotions run hot it means that the maintenance
       budget is insufficient. I always tell everyone they are both just
       maintenance and the only important decision is which to prioritize
       and fix first.
    2. “Shadow IT” — If you used applications in the past that didn’t meet
       employees’ needs, and didn’t have a good governance plan to address
       problems, you may have noticed employees found other solutions on
       their own. This is an indication of underfunded maintenance.
    3. Adaptable Architecture — "The cloud", API-based integration, and
       unlocking your data are no longer “nice to haves.” Your
       architecture needs to adapt. If these are challenges, then the
       architecture must be addressed.
    4. Governance — Healthy application architecture isn’t just about
       technology—it’s also about having well-documented and
       well-understood governance documents that guide technology
       investments for your organization. Good governance helps create
       adaptable architecture and avoid “shadow IT” applications.

2. How do you define maintenance work?

   There are four general types of software maintenance. The first two
   types take up the majority of most organizations' maintenance budget,
   and may not even be considered maintenance - however, all four types
   must be funded adequately for software to remain healthy. If you can't
   fully address types three and four your maintenance budget is
   inadequate.

1. Corrective Software Maintenance (more accurately called "repair")

   Corrective software maintenance is necessary when something goes wrong
   in a piece of software including faults and errors. These can have a
   widespread impact on the functionality of the software in general and
   therefore must be addressed as quickly as possible. However, it is
   important to consider repair work separate from the other types of
   maintenance because repair work must get done. Note: this is generally
   the only type of work that happens when a system is put on "life
   support".

2. Perfective Software Maintenance (more accurately called "enhancements")

   Once software is released and is being used new issues and ideas come
   to the surface. Users will think up new features or requirements that
   they would like to see. Perfective software maintenance aims to adjust
   software by adding new features as necessary (and removing features
   that are irrelevant or not effective). This process keeps software
   relevant as the market, and user needs, evolve. It there is funding
   beyond "life support" it usually is spent here.

3. Preventative Software Maintenance (true maintenance is catching problems
before they happen.)

   Preventative software maintenance is looking into the future so that
   your software can keep working as desired for as long as possible. This
   includes making necessary changes, upgrades, and adaptations.
   Preventative software maintenance may address small issues which at the
   given time may lack significance but may turn into larger problems in
   the future. These are called latent faults which need to be detected
   and corrected to make sure that they won’t turn into effective faults.
   This type of maintenance is generally underfunded.

4. Adaptive Software Maintenance (true maintenance adapts to changes)

   Adaptive software maintenance is responding to the changing technology
   landscape, as well as new company policies and rules regarding your
   software. These include operating system changes, using cloud
   technology, security policies, hardware changes, etc. When these
   changes are performed, your software (and possibly architecture) must
   adapt to properly meet new requirements and meet current security and
   other policies.

3. How do you define a reasonable maintenance budget? How can you protect
that budget?

   In the case of the Inca rope bridges what was the cost of maintenance
   annually? Let's assume some of the build work was site preparation and
   building the stone anchors on each side, but most of the work was
   constructing the bridge itself. Since the bridge was entirely replaced
   each year, the maintenance costs could be as much as 80% of the initial
   build effort, every year.

   Comparing to "software as a service" (SaaS) vendors is difficult
   because they have shifted to a subscription model that bundles
   infrastructure, enhancements, and ongoing maintenance. Prior to SaaS
   subscription-based pricing one would typically buy a perpetual license
   plus maintenance at ~20-30% annual cost of the license to obtain
   support and updates.

   Side note: Now that the SaaS annual costs are commingled, some
   enterprises fall into the trap that “building it is cheaper because we
   pay up front but then it will cost less in the long run” assuming the
   "long run" almost always underprices infrastructure and assumes near
   zero maintenance cost. In the case of a brand-new, internally designed
   and developed software system - one that is well architected, well
   designed, well built, and meets all reliability, scalability, and
   performance needs (i.e., fantasy software) it's conceivable that there
   is no maintenance necessary for some period of time - but very
   unlikely.

   So, maintenance costs can have a very wide range. A general rule of
   thumb is 20-30% of the initial build cost will be required for ongoing
   maintenance work annually. However, maintenance costs usually start off
   lower and increase over time. They are also unpredictable costs that
   are hard to budget.

   The challenges should be obvious. First, budgets in large organizations
   tend be last year's budget plus 2-3%. If you start with a maintenance
   budget of zero on a new system, how do you ever get to the point of a
   healthy maintenance budget in the future? Second, maintenance costs are
   unpredictable, and organizations hate unpredictable costs. It's
   impossible to say when the next new hardware, or storage, or
   programming construct will occur, or when the existing system will hit
   a performance or scalability inflection point.

   This is like buying a brand-new car. The maintenance costs are
   negligible in the first couple years, until they start to creep up.
   Then things start to need maintenance, replacement, or repair. As the
   car ages the maintenance costs continue to increase until at some point
   it makes economic sense to buy another new car. Except none of us wait
   that long. Most of us buy new cars before our old one is completely
   worn out. As a counter-example, in Cuba some cars have been maintained
   meticulously for 30-40 years and run better than new.

Protecting your maintenance budget - creating a "maintenance fund"

   We know that maintenance cost increase over time, and the costs of
   proper maintenance are unpredictable. In addition, there is some amount
   of management discretion that can be applied. When your house needs a
   new roof it's reasonable to defer it through summer, but it probably
   needs to be done before winter.

   Since business require predictability of costs, unpredictable
   maintenance costs are easy to defer. "We didn't budget for that; we'll
   have to put it in next year's budget." Except of course in the budget
   process it will compete with other projects and enhancement work, where
   it's again likely to be deprioritized.

   What's the solution? Could it be possible to create some type of
   maintenance fund where a predictable amount is budgeted each year, and
   then spent "unpredictably" when/as needed? Could this also be a
   solution to preventing executives from diverting maintenance budget
   into pet projects by protecting this maintenance fund in some fashion?

4. How do you motivate software engineers to perform maintenance?

   There is a Chinese proverb about a discussion between a king and a
   famous doctor. The well-known doctor explains to the king that his
   brother (who is also a doctor) is superior at medicine, but he is
   unknown because he always successfully treats small illnesses,
   preventing them from evolving into more serious or terminal ones. So,
   people say "Oh he is a fine doctor, but he only treats minor
   illnesses". It's true: [15]Nobody Ever Gets Credit for Fixing Problems
   that Never Happened.

   To most software engineers, legacy systems seem like torturous dead-end
   work, but the reality is systems that are not important get turned off.
   Working on "estate" systems means working on some of the most critical
   systems that exist — computers that govern millions of people’s lives
   in enumerable ways. This is not the work of technical janitors, but
   battlefield surgeons.

   Engineering loves new technology. It gains the engineers attention and
   industry marketability. [16]Boring technology on the other hand is
   great for the company. The engineering cost is lower, and the skills
   are easier to obtain and keep, because these engineers are not being
   pulled out of your organization for double their salary by Amazon or
   Google.

   Well-designed, high-functioning software that is easy to understand
   usually blends in. Simple solutions do not do much to enhance one’s
   personal brand. Therefore, when an organization provides limited
   pathways to promotion for software engineers, they tend to make
   technical decisions that emphasize their individual contribution and
   technical prowess. You have to be very careful to reward what you want
   from your engineering team.

   What earns them the acknowledgment of their peers? What gets people
   seen is what they will ultimately prioritize, even if those behaviors
   are in open conflict with the official direction they receive from
   management. In most organizations shipping new code gets attention,
   while technical debt accrues silently in the background.

   The specific form of acknowledgment also matters a lot. Positive
   reinforcement in the form of social recognition tends to be a more
   effective motivator than the traditional incentive structure of
   promotions, raises, and bonuses. Behavioral economist Dan Ariely
   attributes this to the difference between social markets and
   traditional monetary-based markets. Social markets are governed by
   social norms (read: peer pressure and social capital), and they often
   inspire people to work harder and longer than much more expensive
   incentives that represent the traditional work-for-pay exchange. In
   other words, people will work really hard for positive reinforcement
   from their peers.

Legacy System Modernization

   Unmaintained software will certainly die at some point. Due to factors
   discussed above, software does not always receive the proper amount of
   maintenance to remain healthy. Eventually a larger modernization effort
   may become necessary to restore a system to operational and functional
   excellence.

   Legacy modernization projects start off feeling easy. The organization
   once had a reliable working system and kept it running for years. All
   the modernizing team should need to do is simply reshape it using
   better technology, better architecture, the benefit of hindsight, and
   improved tooling. It should be simple. But, because people do not see
   the hidden technical challenges they are about to uncover, they also
   assume the work will be boring. There’s little glory to be had
   re-implementing a solved problem.

   Modernization projects are also typically the ones organizations just
   want to get out of the way, so they launch into them unprepared for the
   time and resource commitments they require. Modernization projects take
   months, if not years of work. Keeping a team of engineers focused,
   inspired, and motivated from beginning to end is difficult. Keeping
   their senior leadership prepared to invest in what is, in effect,
   something they already have is a huge challenge. Creating momentum and
   sustaining it are where most modernization projects fail.

   The hard part about legacy modernization is the "system around the
   system". The organization, its communication structures, its politics,
   and its incentives are all intertwined with the technical product in
   such a way that to improve the product, you must do it by turning the
   gears of this other, complex, undocumented system. Pay attention to
   politics and culture. Technology is at most only 50% of the legacy
   problem, ways of working, organization structure and
   leadership/sponsorship are just as important to success.

   To do this, you need to overcome people’s natural skepticism and get
   them to buy in. The important word in the phrase "proof of concept" is
   proof. You need to prove to people that success is possible and worth
   doing. It can't be just an MVP, because [17]MVPs are dangerous.. A red
   flag is raised when companies talk about the phases of their
   modernization plans in terms of which technologies they are going to
   use rather than what value they will add.

   For all that people talk about COBOL dying off, it is good at certain
   tasks. The problem with most old COBOL systems is that they were
   designed at a time when COBOL was the only option. Start by sorting
   which parts of the system are in COBOL because COBOL is good at
   performing that task, and which parts are in COBOL because there were
   no other technologies available. Once you have that mapping, start by
   pulling the latter off into separate services that are written and
   designed using the technology we would choose for that task today.

   Going through the exercise of understanding what functionality is fit
   for use for specific languages/technologies not only gives engineers a
   way to keep building their skillsets but also is an opportunity to pair
   with other engineers who have different/complimentary skills. This
   exchange also has the benefit of diffusing the understanding of the
   system to a broader group of people without needing to solely rely on
   documentation (which never exists).

   Counterintuitively, SLAs/SLOs are valuable because they provide a
   "failure budget". When organizations stop aiming for perfection and
   accept that all systems will occasionally fail, they stop letting their
   technology rot for fear of change. In most cases, mean time to recovery
   (MTTR) is a more useful statistic to push than reliability. MTTR tracks
   how long it takes the organization to recover from failure. Resilience
   in engineering is all about recovering stronger from failure. That
   means better monitoring, better documentation, and better processes for
   restoring services, but you can’t improve any of that if you don’t
   occasionally fail.

   Although a system that constantly breaks, or that breaks in unexpected
   ways without warning, will lose its users’ trust, the reverse isn’t
   necessarily true. A system that never breaks doesn’t necessarily
   inspire high degrees of trust - and its maintenance budget is even
   easier to cut.

   People take systems that are too reliable for granted. Italian
   researchers Cristiano Castelfranchi and Rino Falcone have been
   advancing a general model of trust that postulates trust naturally
   degrades over time, regardless of whether any action has been taken to
   violate that trust. Under Castelfranchi and Falcone’s model,
   maintaining trust doesn’t mean establishing a perfect record; it means
   continuing to rack up observations of resilience. If a piece of
   technology is so reliable it has been completely forgotten, it is not
   creating those regular observations. Through no fault of the
   technology, the user’s trust in it slowly deteriorates.

   When both observability and testing are lacking on your legacy system,
   observability comes first. Tests tell you only what shouldn’t fail;
   monitoring tells you what is failing. Don’t forget: a perfect record
   will always be broken, but resilience is an accomplishment that lasts.
   Modern engineering teams use stats like service level objectives, error
   budgets, and mean time to recovery to move the emphasis away from
   avoiding failure and toward recovering quickly.

Summary

   Maintenance mostly happens out of sight, mysteriously. If we notice it,
   it’s a nuisance. When road crews block off sections of highway to fix
   cracks or potholes, we treat it as an obstruction, not a vital and
   necessary process. This is especially true in the public sector: it’s
   almost impossible to get governmental action on, or voter interest in,
   spending on preventive maintenance, yet governments make seemly
   unlimited funds available once we have a disaster. We are okay spending
   a massive amount of money to fix a problem, but consistently resist
   spending a much smaller amount of money to prevent it; as a business
   strategy this makes no sense.

   The [18]Open Mainframe Project estimates that there about 250 billion
   lines of COBOL code running today in the world economy, and nearly all
   COBOL code contains critical business logic. Companies should maintain
   that software and make it last as long as possible.

References

     * [19]Things You Should Never Do, Part I
     * [20]Patterns of Legacy Displacement
     * [21]Kill It with Fire: Manage Aging Computer Systems (and Future
       Proof Modern Ones)
     * [22]Building software to last forever
     * [23]The Disappearing Art Of Maintenance
     * [24]Inca rope bridge
     * [25]How Often Do Commercial Airplanes Need Maintenance?
     * [26]Nobody Ever Gets Credit for Fixing Problems that Never Happened
     * [27]Boring Technology Club
     * [28]Open Mainframe Project 2021 Annual Report
     * [29]How Popular is COBOL?
     __________________________________________________________________

   Image Credit: Bill Gates, CEO of Microsoft, holds Windows 1.0 floppy
   discs.

   (Photo by Deborah Feingold/Corbis via Getty Images) This was the
   release of Windows 1.0. The beginning. Computers evolve. The underlying
   hardware, CPU, memory, and storage evolves. The operating system
   evolves. Of course, the software we use must evolve as well.

Sharing is Caring

   (BUTTON) (BUTTON) (BUTTON)

   [30]Edit this page

   Dan Stroot · Blog
   I love building things. Made in California. Family man, technologist
   and Hacker News aficionado. Eternally curious.
   [31]Join me on Twitter.[32]Join me on LinkedIn.[33]Join me on GitHub.
   Crafted with ♥️ in California. © 2023, [34]Dan Stroot

References

   Visible links:
   1. https://www.danstroot.com/feed.xml
   2. https://www.googletagmanager.com/ns.html?id=GTM-55JC288
   3. file:///
   4. file:///
   5. file:///about
   6. file:///archive
   7. file:///snippets
   8. file:///uses
   9. https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
  10. https://www.technologyreview.com/2015/08/06/166822/what-is-the-oldest-computer-program-still-in-use/
  11. https://www.guinnessworldrecords.com/world-records/636196-oldest-software-system-in-continuous-use
  12. https://en.wikipedia.org/wiki/Sabre_(travel_reservation_system)
  13. https://en.wikipedia.org/wiki/Inca_rope_bridge
  14. file:///posts/2022-06-05-how-software-learns
  15. https://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_.pdf
  16. https://engineering.atspotify.com/2013/02/in-praise-of-boring-technology/
  17. file:///posts/2021-12-27-dangerous-mvps
  18. https://www.openmainframeproject.org/
  19. https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
  20. https://martinfowler.com/articles/patterns-legacy-displacement/
  21. https://www.amazon.com/Kill-Fire-Manage-Computer-Systems/dp/1718501188
  22. https://herman.bearblog.dev/building-software-to-last-forever/
  23. https://www.noemamag.com/the-disappearing-art-of-maintenance/
  24. https://en.wikipedia.org/wiki/Inca_rope_bridge
  25. https://monroeaerospace.com/blog/how-often-do-commercial-airplanes-need-maintenance/#:~:text=Commercial airplanes require frequent maintenance,inspection once every few years.
  26. https://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_.pdf
  27. https://boringtechnology.club/
  28. https://www.openmainframeproject.org/wp-content/uploads/sites/11/2022/04/OMP_Annual_Report_2021_040622.pdf
  29. https://news.ycombinator.com/item?id=33999718
  30. https://github.com/dstroot/blog-next-13/blob/master/content/posts/2023-05-25-making_software_last_forever.mdx
  31. https://twitter.com/danstroot
  32. https://www.linkedin.com/in/danstroot
  33. https://github.com/dstroot/blog-next
  34. file:///analytics

   Hidden links:
  36. file://localhost/search
  37. file://localhost/about
