The Cost of “Good Enough” Data: Why the Fundamentals Still Matter

My latest book, The Cost of “Good Enough” Data: Why Modern Architectures Fail at Scale, examines a problem that has been building quietly across enterprise IT for years: we have become increasingly willing to accept data that is merely “good enough.” And that approach is becoming increasingly expensive.

Organizations today are investing heavily in AI, analytics, cloud computing, data lakes, data lakehouses, streaming platforms, and increasingly complicated data pipelines. Yet underneath all of this technology lies something much more fundamental: the data itself. If that data is incomplete, inconsistent, duplicated, poorly governed, stale, or inadequately understood, no amount of sophisticated technology layered on top of it can completely compensate. Indeed, AI has changed the economics of data quality because errors that once affected a report or dashboard can now propagate through automated recommendations, decisions, and business processes.

This is where the mainframe perspective becomes particularly interesting.

Those of us who have spent our careers working with Db2 for z/OS and other mainframe technologies grew up in an environment where data management was taken very seriously. Mission-critical systems could not routinely tolerate questionable integrity, undocumented changes, unreliable recovery, uncontrolled access, or sloppy transaction processing. The mainframe disciplines surrounding ACID transactions, database administration, security, recovery, change management, performance engineering, and operational control evolved because the workloads demanded them.

Of course, mainframe environments are not perfect. Bad designs, poor SQL, inadequate governance, and questionable data can exist anywhere. But there is something important to learn from the engineering culture that developed around systems of record.

Accuracy matters. Consistency matters. Recovery matters. Lineage matters. Governance matters. And somebody has to be accountable for the data.

As enterprises distributed their data across warehouses, lakes, cloud platforms, SaaS applications, analytical databases, streaming systems, and countless pipelines, some of that discipline was weakened or lost. Copies multiplied. Ownership became less clear. Transformations accumulated. Metadata became inconsistent. And organizations accumulated what I refer to as data debt — the growing cost and risk created by compromises that seemed reasonable when they were originally made. The book examines how that debt can undermine AI initiatives, increase operational costs, introduce compliance risk, and impair business decision-making.

AI is now exposing that debt.

An AI model does not magically know which customer record is authoritative, whether replicated Db2 data is current, whether a transformation changed the meaning of a field, or whether a piece of information satisfies the organization's governance requirements. It consumes what the data infrastructure provides.

And this brings us back to the mainframe.

For many large enterprises, some of their most valuable, accurate, and carefully controlled business data still resides in Db2 for z/OS and other IBM Z systems. That should not be viewed as an impediment to modernization. Quite the opposite. Trusted systems of record can become an enormously valuable foundation for analytics and AI — provided organizations build architectures that preserve the integrity, context, security, and governance of that data as it moves into newer environments.

That is really one of the central themes of The Cost of “Good Enough” Data. The book is not an argument against the cloud, or any of the newer distributed architectures. It is an argument against complacency. The technology has changed, the scale has changed, and the pace has certainly changed, but the fundamental principles of sound data management have not.

Modernization should not mean abandoning the disciplines we learned building reliable enterprise systems. It should mean applying those disciplines to modern architectures.

Because whether your data resides in Db2 for z/OS, PostgreSQL, a cloud database, a data lakehouse, or all of them at once, the same basic truth applies:

If the data cannot be trusted, neither can anything you build with it.

And in the age of AI, settling for “good enough” data may be more costly than ever.

Why "Cloud-Only" Strategies Fail for Mission-Critical Workloads

 

The Future Isn't Cloud or Mainframe—It's Cloud and Mainframe

For more than a decade, many organizations have pursued what sounded like an irresistible vision: move everything to the cloud. The promise was compelling—lower costs, greater agility, virtually unlimited scalability, and freedom from maintaining expensive on-premises infrastructure. Industry analysts proclaimed that the data center was dead, and vendors eagerly promoted "cloud-first" and, eventually, "cloud-only" strategies as the inevitable future of enterprise computing.

Today, reality is beginning to catch up with the marketing. An increasing number of organizations are discovering that a cloud-only strategy works quite well for some workloads, but not for all of them. In particular, mission-critical operational systems continue to demand characteristics that are difficult, expensive, or simply impractical to reproduce entirely in the cloud.

The issue isn't whether the cloud is valuable. It certainly is. The real question is whether it should become the exclusive home for every enterprise workload. For many organizations, the answer is no.

Mission-Critical Means Different Things

One of the biggest mistakes I see is treating every application as though it has identical requirements. Of course, they don't. A marketing website has very different availability requirements than a bank's transaction processing system. A departmental reporting application is not held to the same standards as an airline reservation system. An internal collaboration tool does not carry the same consequences as a healthcare claims processing application.

Mission-critical systems typically require:

• Continuous availability
• Extremely high transaction throughput
• Consistent response times
• ACID transaction integrity
• Comprehensive security
• Extensive auditability
• Predictable operational behavior
• Recovery measured in seconds—not hours

These characteristics have defined enterprise computing for decades.

They're also the reason IBM Z, Db2, IMS, and CICS continue to run some of the world's largest businesses.

Reliability Is an Architectural Property

One misconception is that reliability can simply be purchased. Experienced users know that it can't. Reliability emerges from architecture, operational discipline, software maturity, governance, and decades of engineering refinement.

Mainframe systems have evolved specifically to deliver extraordinary levels of availability while processing millions of business transactions every day. And that is no accident! It is the result of engineering decisions that have been refined over generations of hardware and software.

Simply moving an application to the cloud does not automatically transfer those architectural qualities.

Latency Matters More Than People Think

Cloud advocates often focus on compute capacity. But mission-critical applications care just as much about latency. It is not uncommon for financial transactions, for example, to involve multiple application servers, databases, authentication services, security components, messaging systems, and external interfaces. 

Adding unnecessary network hops introduces delay. Sometimes that delay is insignificant, but sometimes it isn't.

When thousands of transactions occur every second, small increases in latency accumulate into measurable business costs.

The Economics Are Often Misunderstood

Cloud providers have become extraordinarily efficient. But that doesn't mean every workload becomes less expensive after migration. Consider:

• Operational databases generate enormous volumes of I/O.
• Large enterprises execute billions of SQL statements.
• Data is replicated.
• Backups are created.
• Logs are archived.
• Storage grows.
• Network traffic increases.

High availability configurations multiply infrastructure requirements. And organizations often discover that the monthly operating expense exceeds what they originally projected.

This is one reason cloud repatriation has become an increasingly common topic. Some workloads simply cost less to operate where they already run efficiently.

Data Gravity Is Real

One of my recurring themes is data gravity. Applications can move relatively easily but it is not always that easy for data. Operational databases become deeply integrated with hundreds or even thousands of applications. Moving those databases affects:

• Batch processing
• Replication
• CDC pipelines
• Reporting
• AI platforms
• Security controls
• Regulatory compliance
• Disaster recovery
• Data governance

In other words, the database is rarely an isolated component. It sits at the center of an enterprise ecosystem.

Ignoring that reality leads to expensive modernization projects that deliver far less value than expected.

AI Is Changing the Conversation

Ironically, artificial intelligence has strengthened the business case for retaining mission-critical operational systems. Why is that so? Well, it is because AI depends upon trusted data.

Large language models are becoming commodities... trusted enterprise information is not. Organizations increasingly recognize that their operational databases represent their most valuable information assets.

Db2 databases running on IBM Z often contain decades of highly governed, audited, transactionally consistent business information. The same can be said for IMS databases, which yes, still exist (though are less common).

The trusted data stored in Db2 and IMS is exactly the kind of data AI needs.

Moving it unnecessarily introduces complexity, synchronization challenges, governance concerns, and additional cost.

Modernization Does Not Mean Migration

Perhaps the most important lesson is this: modernization is not synonymous with migration.

Modernization can include:

• API enablement
• Event streaming
• Change Data Capture
• Hybrid cloud integration
• AI enablement
• REST interfaces
• Data virtualization
• Kubernetes integration

Notice what isn't required. Moving the operational database.

Many organizations are discovering that exposing trusted operational data through modern interfaces produces better business outcomes than relocating the underlying systems.

The Hybrid Enterprise Has Already Won

The enterprise architectures I encounter today rarely resemble the all-or-nothing visions presented a decade ago. Most commonly, they are hybrid. Operational systems remain on platforms where they perform best. Cloud platforms provide elasticity for analytics, AI experimentation, web applications, and new digital services. Streaming technologies connect everything together. APIs expose business capabilities. Data virtualization reduces unnecessary duplication. AI consumes trusted enterprise information regardless of where that information physically resides.

That isn't a compromise... it is a good architecture.

Final Thoughts

Technology decisions should always begin with business requirements not industry trends and hype. The cloud is an extraordinary platform for many workloads. But so is the mainframe!

The objective should never be to force every application into a single architectural model. It should be to place each workload where it delivers the greatest business value while minimizing risk, cost, and operational complexity.

After more than forty years working with enterprise databases, I've learned that the most successful architectures rarely chase the latest trend. Instead, they build upon proven strengths while selectively adopting new technologies where those technologies genuinely improve the business.

For mission-critical workloads, that often means embracing a hybrid strategy that combines the resilience and transactional integrity of IBM Z and Db2 with the flexibility and innovation of cloud services.

That may not be as catchy as "cloud-only," but it is far more likely to succeed.

Don't Forget About IBM Redbooks!

Many of my readers are probably aware of IBM redbooks, but for those who are not, here is a quick introduction to a valuable free technical resource. 

If you've been working with any IBM technology for any period of time you've probably become familiar with the IBM redbook. These are supplemental manuals that IBM employees and customers write and distribute free-of-charge over the Internet at http://www.redbooks.ibm.com/. There are redbooks on just about any technical topic that you might imagine - (and for the many Db2 zealots who follow my blog, 

IBM sponsor redbooks and guides the projects that are put together to write them. Some are written by IBMers but many are written by IBM's customers who are actual technology practitioners. Redbooks are written via residencies. A residency is an intensive, multi-week work effort where small teams explore and document product implementation, integration and operations. Each team is directed by an IBMer from their technical publications group. The team will consist of professionals from IBM field and development, from IBM Business Partners, from customers, and from the marketplace in general. So, you too, can research and write an IBM redbook! Of course, you have to make sure that you have the expertise, willingness, and time to work on the project. 

But more importantly, anyone can enjoy the results. IBM publishes all of its redbooks on the web in Adobe PDF format. And anyone can download any redbook for free! So all you need is an Internet connection and the free Adobe Reader software to start reading redbooks. 

Some of my favorite redbooks over the years have addressed topics like stored procedure development, IDAA, improving availability with parallel sysplex, Db2 13 for z/OS performance (there's usually one of these for each new Db2 version), and even AI and watsonx! 

And don't worry if your shop is not a big blue shop, you still might benefit from perusing the redbooks that are available. Yes, there are many redbooks on Db2, IBM Z, watsonx, WebSphere, CICS, and other IBM software products, but there are also redbooks on Linux, Windows, and other non-IBM software. There are even several SQL Server and Oracle redbooks. And these books are always very high quality - I've never read one that is sub-par... and the price is right. 

So, if you haven't checked out the IBM redbook library before - or even if you have, but not lately - take a moment to click over to the redbook site and search for a topic of your interest. You might be surprised at what is available there... 

Db2 Locking Explained in 90 Seconds

Think your enterprise application is running slow because of "aging hardware" or "insufficient CPU"? Think again. In my experience, I’ve found that the vast majority of performance problems blamed on slow infrastructure are actually caused by a silent, internal killer: locking. Or more precisely, inefficient coding built without an understanding of locking!

Let’s talk about Db2 locking, which is one of the most misunderstood performance topics in enterprise systems.

When Db2 locks a resource, it isn't doing it to annoy you; it’s protecting data integrity. It ensures that concurrent transactions don't overwrite each other’s work or read uncommitted data (preventing dirty reads, non-repeatable reads, and phantoms). But when locks escalate or linger unnecessarily, your entire system can grind to a screeching halt.

If you only have 90 seconds, here is the fundamental loop of a locking problem:

1. A transaction requests a lock: It could be a row lock, a page lock, or a table space lock depending on your bind parameters and data modifications.

2. The lock is held too long: If the application does too much processing before committing, or if the access path is inefficient, the transaction hoards that lock.

3. Other transactions must wait: Subsequent tasks trying to touch (read/update) that same resource enter a lock-wait state.

4. The Bottleneck: Accumulate enough waiting tasks, and you get an enterprise-level bottleneck. To the end-user, it looks like the database has frozen.

Why More Hardware Won't Fix It

The classic knee-jerk reaction to this kind of slowdown is throwing money at the problem: adding more CPU, faster storage, or upgrading the mainframe/server capacity.

But if Transaction A is waiting for Transaction B to release a lock, giving Transaction A a faster processor just means it will arrive at the brick wall a microsecond quicker. It still has to wait. The fix isn't more CPU... it’s changing how your data and code interact.

The True Fixes for Lock Contention

If you want to eliminate locking bottlenecks, focus your tuning efforts on these three pillars:

• Better Indexing: If Db2 has to perform a table space scan because an index is missing, it will lock vastly more data (potentially the entire table space) than if it could precisely target a single row via an index. Good indexes keep your locks small and surgical.

• Shorter Units of Work: Keep your transactions tight. Don't fetch data, perform complex business logic, call external APIs, and then commit. Do your non-database work outside the transaction bounds so the lock is held for milliseconds, not seconds.

• Proper Commit Frequency: If you are running batch processes or massive update loops without regular COMMIT statements, you are building an operational dam. Frequent, structured commits flush the log buffers and release held locks, keeping the data flowing smoothly for concurrent users.

The Bottom Line

If your team isn't actively monitoring lock contention, lock timeouts, and deadlocks using Db2 statistics and traces, you are likely tuning the wrong problem.

The next time someone complains that Db2 is "running slow," step away from the infrastructure metrics. Open your monitoring tools, look at the lock-wait times, and find out who is holding the key to the castle while everyone else is standing outside in the cold.

And if you are looking for more details on locking, I wrote a 17-part series on this blog awhile ago that is still pertinent. You can access it here (this is a link to part 17 that has links to the previous 16 parts, too).

What are your favorite techniques for tracking down elusive lock-waits in Db2? Drop a comment below and let’s discuss!

The Most Overlooked Db2 Performance Metric

 If I had to identify a single most overlooked Db2 performance metric, it would be GETPAGEs.

Many organizations focus on CPU utilization, elapsed time, or buffer pool hit ratios because those metrics are easy to understand and frequently appear on dashboards. But getpages often reveal performance problems long before those other metrics become alarming.

In Db2 for z/OS, a GETPAGE represents a request by Db2 to access a page of data or an index page. If the page is already in the buffer pool, no physical I/O is required. If not, Db2 must read it from disk. Either way, every GETPAGE consumes CPU resources.

Why GETPAGEs Matter

Because every SQL statement generates GETPAGE requests, the number of GETPAGEs is a fundamental measure of how much work Db2 performs to execute a workload. The more pages Db2 must examine to satisfy a query, the more work it performs. Reducing unnecessary GETPAGEs often leads directly to lower CPU consumption and better application performance.

High GETPAGE counts usually indicate one or more of the following:

• Inefficient index usage
• Poor clustering
• Table scans
• Low filter factors
• Redundant index probes
• Accessing more columns or rows than necessary

Even when all pages are found in the buffer pool, millions of unnecessary GETPAGEs still consume CPU.

The Hidden CPU Consumer

I've worked with organizations that upgraded processors because CPU costs continued to rise. After analyzing the workload, we discovered that many critical SQL statements were generating ten or even one hundred times more GETPAGEs than necessary.

A simple index redesign or SQL rewrite reduced GETPAGE counts dramatically. And CPU usage dropped without any hardware changes.

That's why I often tell clients: "Every unnecessary GETPAGE results in costly unneeded work."

Don't Just Count GETPAGEs

The raw number of GETPAGEs by itself does not tell the whole story. Instead, monitor:

• GETPAGEs per transaction
• GETPAGEs per SQL statement
• GETPAGEs per row returned
• Trends over time
• The highest GETPAGE-consuming applications

A workload processing twice as many transactions should naturally generate more GETPAGEs. The real warning sign is when GETPAGEs per transaction steadily increase. That usually means SQL or access paths have degraded.

GETPAGEs Point to Root Causes

Unlike CPU utilization, which tells you that work is occurring, GETPAGEs often explain why.

For example:

Symptom	What High GETPAGEs May Indicate
High CPU	Excessive index or table page accesses
Long elapsed time	Inefficient access path
Buffer pool pressure	Poor locality of reference
Lock contention	Long-running scans
Increased zIIP usage	More work being offloaded, but still excessive overall processing

My Rule of Thumb 👍

After more than four decades tuning Db2 systems, one principle has remained remarkably consistent:

Watch the work, not just the time.

Elapsed time can fluctuate because of concurrency. CPU depends on hardware generation. I/O depends on storage technology.

But GETPAGEs measure the amount of work Db2 is performing. If you reduce the work, you almost always improve performance.

Other Frequently Overlooked Metrics

Although GETPAGEs are my top choice, several other metrics deserve more attention:

• Synchronous read percentage — Indicates how often Db2 must wait for I/O instead of benefiting from prefetch.
• Pages read per GETPAGE — Helps evaluate buffer pool effectiveness.
• Lock suspension time — Often more meaningful than simply counting lock waits.
• Class 3 suspension time — Shows where Db2 is waiting (I/O, locks, logging, etc.).
• RID pool failures — Can reveal access path problems that are otherwise difficult to diagnose.
• Sort overflows — Indicate insufficient sort memory or inefficient SQL.
• Index leaf page split rates — A useful indicator of index maintenance issues and clustering degradation.

If I were teaching a new Db2 performance analyst, I'd spend less time looking at CPU graphs and more time asking, "Why is Db2 doing so much work?" In many cases, the answer begins with GETPAGEs.