Understanding Lock Escalation: Managing Resource Contention

Ensuring efficient data access while maintaining data integrity is critical to both performance and stability. One of the mechanisms Db2 employs to manage this balance is lock escalation. Though this feature is essential when managing large numbers of locks, improper handling can lead to performance bottlenecks. Understanding lock escalation and how it impacts your Db2 environment is crucial for database administrators (DBAs) seeking to optimize operations.

What Is Lock Escalation?

Lock escalation is Db2’s method of reducing the overhead associated with managing numerous individual row or page locks. Instead of holding thousands of fine-grained locks, Db2 “escalates” these to coarser-grained table or table space locks. This happens automatically when a session’s lock usage exceeds a predefined threshold.

The primary goal of lock escalation is to reduce the system resources spent on tracking and maintaining a large number of locks. Without escalation, too many locks could overwhelm system memory or negatively impact performance due to the lock management overhead. Escalating to a table (space) lock allows Db2 to control resource consumption and avoid these issues.

When Does Lock Escalation Occur?

There are two limits to be aware of. The first is NUMLKTS, which specifies the maximum nunber of locks a process can hold on a single table space. This is the default and it can be overridden in the DDL of a tablespace using the LOCKMAX clause. When NUMLKTS (or LOCKMAX) is exceeded, Db2 will perform lock escalation.

The second is NUMLKUS, which specifies the maximum number of locks a process can hold across all table spaces. When a single user exceeds the page lock limit set by the Db2 subsystem (as defined in DSNZPARMs), the program receives a -904 SQLCODE notification. The program can respond by issuing a ROLLBACK and generating a message suggesting that the program be altered to COMMIT more frequently (or use alternate approaches like executing a LOCK TABLE statement).

Lock escalation may also occur due to the lock list or lock table approaching its capacity. In such cases, Db2 may escalate locks to prevent the system from running out of resources.

Additionally, keep in mind that as of Db2 12 for z/OS FL507, there are two new built-in global variables that can be set by application programs to control the granularity of locking limits.

The first is SYSIBMADM.MAX_LOCKS_PER_TABLESPACE and it is similar to the NUMLKTS parameter. It can be set to an integer value for the maximum number of page, row, or LOB locks that the application can hold simultaneously in a table space. If the application exceeds the maximum number of locks in a single table space, lock escalation occurs.

The second is SYSIBMADM.MAX_LOCKS_PER_USER and it is similar to the NUMLKUS parameter. You can set it to an integer value that specifies the maximum number of page, row, or LOB locks that a single application can concurrently hold for all table spaces. The limit applies to all table spaces that are defined with the LOCKSIZE PAGE, LOCKSIZE ROW, or LOCKSIZE ANY options. 

These new FL507 options should be used sparingly and only under the review and control of the DBA team.

The Impact of Lock Escalation

While lock escalation conserves system resources, it can also lead to resource contention. By escalating locks from rows or pages to a table-level lock, Db2 potentially increases the chances of lock contention, where multiple transactions compete for the same locked resource. This can have a few side effects:

• Blocking: When an entire table is locked, other transactions that need access to that table must wait until the lock is released, even if they only need access to a small portion of the data.
• Deadlocks: With more coarse-grained locks, the likelihood of deadlocks can increase, especially if different applications are accessing overlapping resources.
• Performance degradation: While escalating locks reduces the overhead of managing many fine-grained locks, the side effect can be a performance hit due to increased contention. For systems with high concurrency, this can result in significant delays.

Managing Lock Escalation

A savvy DBA can take steps to minimize the negative impacts of lock escalation. Here are some strategies to consider:

1. Monitor Lock Usage: Db2 provides tools like DISPLAY DATABASE and EXPLAIN to track locking behavior. Regularly monitor your system to understand when lock escalation occurs and which applications or tables are most affected.

2. Adjust Lock Thresholds: If escalation is happening too frequently, consider adjusting your LOCKMAX parameter. A higher threshold might reduce the need for escalation, though be mindful of the system’s lock resource limits. Additionally, consider the FL507 built-in global variables for difficult to control situations. 

3. Optimize Application Design: Poorly optimized queries and transactions that scan large amounts of data are more prone to trigger lock escalation. Review your applications to ensure they are using indexes efficiently, and minimize the number of locks held by long-running transactions.

4. Partitioning: Partitioning larger tables can help mitigate the effects of lock escalation by distributing locks across partitions.

5. Use of Commit Statements: Frequent commits help release locks, lowering the risk of escalation. Ensure that programs are committing frequently enough to avoid building up large numbers of locks. A good tactic to employ is parameter-based commit processing, wherein a parameter is set and read by the program to control how frequently commits are issued. This way, you can change commit frequency without modifying the program code.

Conclusion

Lock escalation is a necessary mechanism in Db2, balancing the need for data integrity with resource efficiency. However, it can introduce performance issues if not properly managed. By understanding when and why escalation occurs, and taking proactive steps to optimize your environment, you can minimize its negative impact while maintaining a stable, efficient database system.

As with many aspects of Db2, the key lies in careful monitoring, tuning, and optimization. A well-managed lock escalation strategy ensures that your system remains responsive, even under heavy workloads, while preserving the data integrity that Db2 is known for.

Db2 for z/OS: The Performance and Management Champion!

Usually, posts I write for this blog focus on technical details, tips, and techniques for better using and optimizing your experience with Db2. Today, I want to do something a little different. You see, I am a big fan of Db2 for z/OS, and I do not see it getting the press, or the accolades that I think it is due. So I am going to use my platform to shout out the performance benefits of Db2 for z/OS.

When it comes to performance, nothing beats Db2 for z/OS. This mainframe database has been setting the standard for decades, delivering unmatched speed and efficiency for mission-critical applications. Let's explore some of the reasons why Db2 for z/OS is the performance champion.

Hardware Acceleration

• z/Architecture: Db2 for z/OS takes full advantage of the powerful z/Architecture, which includes specialized hardware for database operations. This hardware acceleration provides a significant performance boost for tasks like query processing and data loading.
• Storage Subsystem: The mainframe's storage subsystem is designed for high performance and reliability. With features like z/Hyperlink, data compression, and flash storage, Db2 for z/OS can access data quickly and efficiently.
• IDAA: IBM Db2 Analytics Accelerator is a high-performance, in-memory database appliance designed to accelerate analytic workloads. It's optimized for large-scale data analysis tasks, providing significant speedups compared to traditional disk-based databases. By leveraging solid-state drives (SSDs) and advanced hardware architecture, IDAA can handle complex queries and data manipulations with exceptional efficiency. This makes it ideal for applications requiring real-time analytics, data warehousing, and big data processing.

Database Optimization

• Query Optimization: Db2 for z/OS has a sophisticated query optimizer that can automatically select the most efficient execution plan for your queries. This ensures that your applications run as fast as possible.
• Data Compression: Db2 for z/OS supports data compression, which can reduce storage requirements and improve performance. By compressing data, Db2 can reduce the amount of data that needs to be read and processed.
• Parallel Processing: Db2 for z/OS can take advantage of multiple processors to perform tasks in parallel. This can significantly improve performance for large workloads.
• AI: IBM Db2 AI for z/OS integrates autonomics to simplify database management efforts. Using machine learning and AI, it can help improve operational performance and maintain Db2 for z/OS efficiency and health while enhancing Db2 for z/OS performance, reliability and cost effectiveness–even under the most demanding circumstances.

Workload Management

• Resource Allocation: Db2 for z/OS provides powerful tools for managing resources and ensuring that your database applications get the resources they need to perform optimally.
• Workload Balancing: Db2 can automatically balance workloads across multiple systems to ensure that resources are used efficiently.
• WLM: Workload Manager is an integrated, critical component of z/OS that is used for optimizing the performance and resource utilization of Db2 for z/OS. It provides a comprehensive framework for managing workloads across the mainframe environment, ensuring that Db2 applications receive the resources they need to perform optimally.

Data Sharing and Parallel Sysplex

Finally, Data Sharing using IBM Z Parallel Sysplex confers a significant advantage onto Db2 for z/OS in that it can enhanced availability by providing inherent redundancy, as multiple subsystems can access the same data. This helps to mitigate the impact of hardware failures or system outages. And in case of a disaster, data sharing can facilitate rapid recovery by allowing applications to access data from a different subsystem.

Furthermore, Data Sharing enhances scalability by enabling workloads to be distributed across multiple subsystems, improving scalability and preventing bottlenecks. It facilitates simpler growth: as data volumes and application demands increase, data sharing can help to accommodate growth without requiring significant hardware investments.

And Data Sharing can improve performance. By allowing multiple Db2 subsystems to access the same data without requiring individual copies, data sharing significantly reduces I/O operations, leading to improved performance. And with data readily available to multiple subsystems, queries can be executed more quickly, resulting in faster response times for applications.

So, IBM Z data sharing on Db2 offers a range of benefits, including improved performance, enhanced availability, increased scalability, reduced costs, and simplified management. These benefits make it a valuable feature for organizations that require high-performance, reliable, and scalable database solutions.

Real-World Results

Organizations around the world rely on Db2 for z/OS to power their most critical applications. From financial services to healthcare, Db2 has proven its ability to deliver the performance and reliability that businesses need to succeed.

So, if you're looking for a database that can handle your most demanding workloads and deliver exceptional performance, Db2 for z/OS is the way to go.

Highlights of the 2024 NA IDUG Db2 Tech Conference

Just a quick blog post today to let my readers know that I have written an overview of the 2024 IDUG Db2 Tech Conference in Chartlotte, NC this past June. 

The overview was written for the SHARE'd Intelligence blog, which is the official publication of SHARE. It offers news and education on enterprise solutions, and you would be wise to bookmark the site to keep up with the content shared there. 

The post I wrote is titled Riding the Waves of Knowledge at the IDUG Db2 Tech Conference. I hope you'll check it out, read my perspective, and share your thoughts on it here... and make plans to attend next year's IDUG event in Atlanta!

Coding Db2 Applications for Performance - Expert Videos Series

Today's blog post is to share with my readers that I have partnered with Interskill Learning and produced a series of videos in the Expert Video Series on how to code Db2 applications for performance.

My regular readers know that application performance is a passion of mine. You may also have read my recent book on the topic, A Guide to Db2 Performance for Application Developers. But if you are looking for videos to guide you through the process optimizing your application development for Db2, look no further than the six-part series I recorded for Interskill Learning, Coding Db2 Applications for Performance.

You do not need in-depth pre-existing knowledge of Db2 to gain insight from these video lessons. The outline of the six courses are as follows:

 Db2 Coding – Defining Database Performance

• Providing a Definition
• The Four Components
• Diving a Little Deeper

Db2 Coding – Coding Relationally

• What is Relational?
• Relational vs. Traditional Thinking
• What Does It Mean to Code Relationally?
• Unlearning Past Coding Practices

Db2 Coding – General SQL and Indexing Guidelines

• Types of SQL
• SQL Coding Best Practices
• Indexes and Performance
• Stages and Clustering

Db2 Coding – Coding for Concurrent Access

• Introduction to Concurrency
• Locking
• Locking Duration and Binding
• Locking Issues and Strategies
• Query Parallelism

Db2 Coding – Understanding and Reviewing Db2 Access Paths

• Single Table Access Paths
• Multi-table Access Paths
• Filter Factors
• Access Paths and EXPLAIN

Db2 Coding – SQL Coding Tips and Techniques

• Avoid Writing Code
• Reusable Db2 Code
• Dynamic and Static SQL
• SQL Guidelines
• Set Operations

So if you are looking for an introduction to Db2 performance or want to brush up on the fundamentals of coding for performance, look no further. Check out this series of videos on Coding Db2 Applications for Performance from Interskill Learning (featuring yours truly)!

Note that Interskill Learning also offers other categories of training in their Expert Video series including systems programming, quantum computing, and pervasive encryption. 

The Basics of Coding Db2 SQL for Performance

When it comes to assuring optimal performance of Db2 applications, coding properly formulated SQL is an imperative. Most experts agree that poorly coded SQL and application code is the cause of most performance problems – perhaps as high as 80% of poor relational performance is caused by “bad” SQL and application code.

But writing efficient SQL statements can be a tricky proposition. This is especially so for programmers and developers new to a relational database environment. So, before we delve into the specifics of coding SQL for performance, it is best that we take a few moments to review SQL basics.

SQL, an acronym for Structured Query Language, is a powerful tool for manipulating data. It is the de facto standard query language for relational database management systems and is used not just by Db2, but also by the other leading RDBMS products such as Oracle, Sybase, and Microsoft SQL Server.

SQL is a high-level language that provides a greater degree of abstraction than do procedural languages. Most programming languages require that the programmer navigate data structures. This means that program logic needs to be coded to proceed record-by-record through data elements in an order determined by the application programmer or systems analyst. This information is encoded in the program logic and is difficult to change after it has been programmed.

SQL, on the other hand, is fashioned so that the programmer can specify what data is needed, and not how to retrieve it. SQL is coded without embedded data-navigational instructions. Db2 analyzes the SQL and formulates data-navigational instructions "behind the scenes." These data-navigational instructions are called access paths. By having the DBMS determine the optimal access path to the data, a heavy burden is removed from the programmer. In addition, the database can have a better understanding of the state of the data it stores, and thereby can produce a more efficient and dynamic access path to the data. The result is that SQL, used properly, can provide for quicker application development.

Another feature of SQL is that it is not merely a query language. The same language used to query data is used also to define data structures, control access to the data, and insert, modify, and delete occurrences of the data. This consolidation of functions into a single language eases communication between different types of users. DBAs, systems programmers, application programmers, systems analysts, and end users all speak a common language: SQL. When all the participants in a project are speaking the same language, a synergy is created that can reduce overall system-development time.

Arguably, though, the single most important feature of SQL that has solidified its success is its capability to retrieve data easily using English-like syntax. It is much easier to understand the following than it is to understand pages and pages of program source code.

    SELECT  LASTNAME

    FROM    EMP

    WHERE   EMPNO = '000010';

Think about it; when accessing data from a file the programmer would have to code instructions to open the file, start a loop, read a record, check to see if the EMPNO field equals the proper value, check for end of file, go back to the beginning of the loop, and so on.

SQL is, by nature, quite flexible. It uses a free-form structure that gives the user the ability to develop SQL statements in a way best suited to the given user. Each SQL request is parsed by the DBMS before execution to check for proper syntax and to optimize the request. Therefore, SQL statements do not need to start in any given column and can be strung together on one line or broken apart on several lines. For example, the following SQL statement is equivalent to the previously listed SQL statement:

    SELECT LASTNAME FROM EMP WHERE EMPNO = '000010';

Another flexible feature of SQL is that a single request can be formulated in a number of different and functionally equivalent ways. One example of this SQL capability is that it can join tables or nest queries. A nested query always can be converted to an equivalent join. Other examples of this flexibility can be seen in the vast array of functions and predicates. Examples of features with equivalent functionality are:

·       BETWEEN versus <= / >=

·       IN versus a series of predicates tied together with AND

·       INNER JOIN versus tables strung together in the FROM clause separated by commas

·       OUTER JOIN versus a simple SELECT, with a UNION, and a correlated subselect

·       CASE expressions versus UNION ALL statements

This flexibility exhibited by SQL is not always desirable as different but equivalent SQL formulations can result in extremely differing performance. The ramifications of this flexibility are discussed later in this paper with guidelines for developing efficient SQL.

As mentioned, SQL specifies what data to retrieve or manipulate, but does not specify how you accomplish these tasks. This keeps SQL intrinsically simple. If you can remember the set-at-a-time orientation of a relational database, you will begin to grasp the essence and nature of SQL. A single SQL statement can act upon multiple rows. The capability to act on a set of data coupled with the lack of need for establishing how to retrieve and manipulate data defines SQL as a non-procedural language.

Because SQL is a non-procedural language a single statement can take the place of a series of procedures. Again, this is possible because SQL uses set-level processing and DB2 optimizes the query to determine the data-navigation logic. Sometimes one or two SQL statements can accomplish tasks that otherwise would require entire procedural programs to do.

High-Level SQL Coding Guidelines

When you are writing your SQL statements to access Db2 data be sure to follow the subsequent guidelines for coding SQL for performance. These are certain very simple, yet important rules to follow when writing your SQL statements. Of course, SQL performance is a complex topic and to understand every nuance of how SQL performs can take a lifetime. That said, adhering to the following simple rules puts you on the right track to achieving high-performing Db2 applications.

1)     The first rule is to always provide only the exact columns that you need to retrieve in the SELECT-list of each SQL SELECT statement. Another way of stating this is “do not use SELECT *”. The shorthand SELECT * means retrieve all columns from the table(s) being accessed. This is fine for quick and dirty queries but is bad practice for inclusion in application programs because:

·       Db2 tables may need to be changed in the future to include additional columns. SELECT * will retrieve those new columns, too, and your program may not be capable of handling the additional data without requiring time-consuming changes.

·       Db2 will consume additional resources for every column that requested to be returned. If the program does not need the data, it should not ask for it. Even if the program needs every column, it is better to explicitly ask for each column by name in the SQL statement for clarity and to avoid the previous pitfall.

2)     Do not ask for what you already know. This may sound simplistic, but most programmers violate this rule at one time or another. For a typical example, consider what is wrong with the following SQL statement:

    SELECT  EMPNO, LASTNAME, SALARY

    FROM    EMP

    WHERE   EMPNO = '000010';

 

Give up? The problem is that EMPNO is included in the SELECT-list. You already know that EMPNO will be equal to the value '000010' because that is what the WHERE clause tells DB2 to do. But with EMPNO listed in the WHERE clause Db2 will dutifully retrieve that column too. This causes additional overhead to be incurred thereby degrading performance.

3)     Use the WHERE clause to filter data in the SQL instead of bringing it all into your program to filter. This too is a common rookie mistake. It is much better for Db2 to filter the data before returning it to your program. This is so because Db2 uses additional I/O and CPU resources to obtain each row of data. The fewer rows passed to your program, the more efficient your SQL will be. So, the following SQL

    SELECT  EMPNO, LASTNAME, SALARY

    FROM    EMP

    WHERE   SALARY > 50000.00;

Is better than simply reading all of the data without the WHERE clause and then checking each row to see if the SALARY is greater than 50000.00 in your program.

These rules, though, are not the be-all, end-all of SQL performance tuning – not by a long shot. Additional, in-depth tuning may be required. But following the above rules will ensure that you are not making “rookie” mistakes that can kill application performance. 

In Closing

This short blog post is just the very beginning of SQL performance for Db2 programmers. Indeed, I wrote a book on the topic called A Guide to Db2 Performance for Application Developers, so check that out if this post has whetted your appetite for more Db2 performance tips... and if you are a more visual learner, I have also partnered with Interskill Learning for a series of videos in their Expert Video series on the topic of Coding Db2 Applications for Performance. So, why wait, dig in to a book, some videos, or both, to help improve the performance of your Db2 applications!