Monday, May 14, 2018

Improving Performance by Caching Select Statement Results

A significant trend in the database world these days is moving more data management and processing into memory. If you can bypass disk I/O you can achieve tremendous performance gains. There are many reasons for this, but of course, the most important reason is that disk access is much slower than memory access.

It is orders of magnitude more efficient to access data from computer memory than it is to read it from disk. Memory access is usually measured in microseconds, whereas disk access is measured in milliseconds (1 millisecond equals 1000 microseconds).

So, it makes sense to process as much data as we can in memory. This is the reason that most popular RDBMS use buffer pools to cache data in memory. But buffer pools are not always sufficient for caching everything that is needed. Just take a look at IBM Db2 12 for z/OS and the index FTB feature, which caches unique index structures in memory – not in any buffer pool.

FTBs, or Fast Traverse Blocks, provide in-memory index optimization. FTBs are stored in a separate area outside of your buffer pools. You must allocate additional real memory for it in Db2 12. This memory is an optimized structure for fast index lookups that provide random index access. The FTB contains the non-leaf structure of the index.

QuickSelect for Db2

Of course, FTBs are just one technique to optimize your memory usage. Another technique you might consider is to deploy QuickSelect for Db2, an in-memory optimization product from Log-On Software (an Israeli company that has been providing tools and services to the mainframe market for decades). QuickSelect for Db2 caches query results in memory. This is a different concept than buffer pools which cache table and index space pages or in-memory tables which cache entire tables.

By caching the results of frequently run queries in memory with QuickSelect for Db2, you can optimize the SQL statements that recur frequently in your environment. QuickSelect runs as a started task and it automatically caches repetitive query results in self-managed memory above the bar (64-bit).

QuickSelect returns the same data DB2 would – but from its cache. QuickSelect will identify and cache only frequently used data via a threshold parameter. This saves both CPU and I/O, and therefore can reduce elapsed time and improve application response times. All other requests are satisfied as usual by Db2.

Which leads to the question: which queries should be cached? QuickSelect helps you to identify ideal queries with its Survey mode which automatically finds SQL queries suitable for caching.

The results cached by QuickSelect are the actual data retrieved by Db2 queries. So the next time that same query runs (with the same host variable values) the results are quickly returned from the QuickSelect cache instead of invoking process-heavy disk I/O operations. Think of all the repetitive SQL that you are running in your shop every day – and all of the redundant I/O that is used to read that data over and over again.

But, you may be asking, what happens if the data changes? Well, QuickSelect is aware of all changes that are made, whether by a Db2 utility like Load, Reorg and Recover, or by your applications issuing Insert, Update, Delete, and Truncate statements. Data changes on the tables that are cached are detected by QuickSelect in real-time using various techniques. QuickSelect invalidates the cached data for the updated tables automatically, thus data integrity is preserved. Further queries will cause QuickSelect to cache the updated data, thereby continuing to save. QuickSelect runs in a multiple LPAR environment and supports update sensitivity by using XCF functions.

In other words, QuickSelect returns the same answer as Db2 does… only faster. And, of course, that is the most important thing.

If you were paying attention as you read through this so far you will have noticed that there are no application changes required whatsoever to enable QuickSelect. And you do not have to Rebind your programs to take advantage of QuickSelect, either!

What happens if you shut down QuickSelect? There should be no impact on your application programs; processing reverts entirely back to Db2. The data will no longer be returned from the QuickSelect cache, instead it will again be retrieved by Db2 just like before you used QuickSelect.

QuickSelect: The Latest Features

Log-On Software is not resting on its laurels; new features are added to QuickSelect on a regular basis. For example, in Febrauary 2018 auto-configuration was added to QuickSelect, thereby easing configuration in a Data Sharing environment.

With auto-configuration the QuickSelect group configuration is dynamically determined. It is no longer required to pre-configure QuickSelect groups. QuickSelect now automatically detects the active members in the Db2 group, if any, and automatically establishes a connection between all QuickSelect servers that correlate to the current Db2 configuration. This is much simpler than the previous requirement of hard-coding QuickSelect grouping using pre-defined parameters.  

The Bottom Line

QuickSelect for Db2 offers a different approach to in-memory caching by focusing on query results. QuickSelect for Db2 customers have been able to significantly reduce CPU during peak times – the times set by the rolling four-hour average that impact your monthly IBM software bill.

For example, one customer, a large European commercial Bank was able to turn off an entire CPU after installing QuickSelect in production due to the CPU savings they achieved. In this case, the customer was saving more than 10 billion SQL statements during a typical 12-hour window.

The bottom line is that in-memory processing can save CPU, improve performance and save money. Wise organizations will look into multiple ways of exploiting memory to achieve results, including potentially looking at novel software solutions like QuickSelect for Db2.

Wednesday, April 04, 2018

Catch You in Philly for the IDUG Db2 Tech Conference 2018

Well, this year’s IDUG North American DB2 Tech Conference is almost upon us and I hope you have made plans to be there. If not, it is still not too late, though. The conference is being held in Philadelphia, PA this year -- The Cradle of Liberty -- from April 29 - May 3, 2018.

I don’t know about you, but every year I mark my calendar ahead of time and then look forward to the week of IDUG because it always provides an educational and enjoyable time. If you’ve ever attended an IDUG conference before then you know all about the fantastic educational and entertainment opportunities that IDUG offers. Of course, there will be many informative technical sessions on all of the latest and greatest Db2 technologies and features. The 2018 event offers more educational opportunities and training than ever before, including:
  • Five days of educational sessions
  • Half and full-day workshops
  • More than 100 one-hour technical sessions
  • Two expert panels on z/OS and LUW

There are also two great keynote sessions scheduled this year. The first, from Michele Goetz, an analyst with Forrester Research, whose session is titled “Your Business Is Only As Fast As Your Data.” She will discuss the importance of data governance programs and how to establish sustainable governance.

And then there is the one I am most looking forward to, a visit from Andrew Flip Filipowski, IDUG co-founder, Executive Chairman and CEO SilkRoad Equity, founder and former CEO of PLATINUM technology, inc. and current co-CEO of Fluree, PBC, provider of the world’s only ACID compliant blockchain graph database system. Flip is always educational and entertaining, so I expect a great keynote session from him as he discusses a wide range of topics such as bitcoin, crypto assets, blockchain, FinTech, venture capital, entrepreneurship, and more.

I’m also looking forward to the spotlight session on “Db2 for z/OS…and Beyond” from Jeff Josten (chief architect of Db2 for z/OS) and Maureen Townsend (Db2 for z/OS Development Director).

As usual, I will be busy at this year’s IDUG. I will be delivering two sessions this year. The first is on Wednesday at 3:40 PM (Session 5028) titled Db2 Application Development for Performance: Be Early and Be Informed. This session is aimed at application developers. The general idea is to give an overview of the things that you can do as you design and code your Db2 programs with performance in mind. All too often performance is an afterthought – and that can be quite expensive. Nail down the basics by attending this session!

My second presentation, on Thursday at 9:20 AM, is titled Database Trends 2018. This session is for everybody as I run down the important trends that are impacting the jobs of modern data professionals. I’ll touch on Big Data, analytics, NoSQL, cloud, digital transformation, in-memory computing, DevOps, and more. And conclude with how it all impacts DBAs. Don’t miss this session!

Let's not forget the exhibit hall (aka Solutions Center) where vendors present and demo their products that can help you manage Db2 more effectively. It is a good place to learn about new technology solutions for Db2, but also to hang out and meet with IBMers, consultants, and your peers.

This year I'll be spending some time in the CorreLog booth (#209) in the Solutions Center. Be sure to stop by and say hello, take a look at CorreLog's great solutions for SIEM and auditing Db2 for z/OS, and register to win one of 4 of my DBA books that will be raffled off.

That is a lot for one week, but there is more. You can go to full-day education sessions on Sunday April 29th (at an additional cost), plan for Db2 12 for z/OS and continuous delivery (half day session on May 3rd), attend Special Interest Groups (where you can discuss the latest industry trends and topics with other interested technicians), attend Hands-On Labs (with hands-on training led by IBM), and even take complementary IBM certification exams

And be sure to download the mobile app for the conference to help you navigate all the opportunities available to you!

The IDUG Db2 TechConference is the place to be to learn all about Db2 from IBMers, gold consultants, IBM champions, end users, ISVs, and more. With all of this great stuff going on this year in Philadelphia, why wouldn't you want to be there!?!?

Thursday, March 29, 2018

The Db2 12 for z/OS Blog Series - Part 21: New Global Variables for Continuous Delivery

One of the most important new "features" of Db2 12 for z/OS is continuous delivery. With continuous delivery more functionality will be made available more quickly than ever before. Instead of waiting for big version migrations new function levels can be applied rapidly, thereby delivering desired functionality more quickly and agilely.

Of course, this impacts DBAs and systems programmers who manage the  version of Db2 more than it impacts developers. That said, developers always need to be aware of which version and now, level, of Db2 that they are using. This is important because it dictates the features that are available to use.

As part of the continuous delivery of Db2 functionality, Db2 12 adds several built-in global variables to help. In actuality, these new variables can be read by any application in Db2 11 NFM and Db2 12 (as long as the Db2 11 subsystem has applied the Db2 12 migration SPE and executed CATMAINT).

The first global variable we will discuss is PRODUCTID_EXT, which stores the extended product identifier of the database manager that was used to invoke the function. The value is VARCHAR(30) and it is maintained by the system. The schema is SYSIBM. 

The format of the extended product identifier values is pppvvrrmmm, defined as follows: 

  • ppp is a three-letter product code (such as, DSN for Db2)
  • vv is the version
  • rr is the release
  • mmm is the modification level (such as, 100, 500, 501)

For example, DSN1201501 identifies Db2 12 after the activation of Db2 12 new function level 501. Function level 500 is the first Db2 12 function level so any level 500 or greater indicates Db2 12 new functionality is availabile. 

An application accessing PRODUCTID_EXT from a coexistent Db2 11 member of a data sharing group would see a value of DSN1101500. 

The second new global variable for continuous delivery is the CATALOG_LEVEL. Appropriately enough, this global variable contains the current catalog level. Again, the data type is VARCHAR(30) and it is maintained by the system with a schema of SYSIBM. 

The format of the catalog level values is VvvRrMmmm, defined as follows:

  • vv is the version
  • r is the release
  • mmm is the modification level (such as 100, 500, 501)

For example, V12R1M500 identifies Db2 12 after the activation of Db2 12 and the initial CATMAINT run for Db2 12 runs. An application accessing CATALOG_LEVEL from a coexistent Db2 11 member of a data sharing group would see a value of V12R1M500 after the initial CATMAINT run for Db2 12 runs on a Db2 12 member.

The third and final new global session variable for continuous delivery is the DEFAULT_SQLLEVEL, which stores the default value of the SQLLEVEL SQL processing option (DECPSQLL). As with the others, the data type is VARCHAR(30) and it is maintained by the system with a schema of SYSIBM. 

The format of the catalog level values is V10R1, V11R1, or VvvRrMmmm, , defined as follows:

  • vv is the version
  • is the release
  • mmm is the modification level (such as 100, 500, 501)

For example, V12R1M501 identifies Db2 Version 12 Release 1 Function Level 501.

Keep these global variables in mind and use them as appropriate in your programs to ensure that the functionality you need is actually available to your program when it runs.

Friday, March 09, 2018

On Leaky Pipes and DBAs

How is a plumber like a DBA? 

I was working in my backyard a couple of weekends ago, trying to find the cause of a leak. You see, there was a big puddle of water in the backyard, it hadn’t rained lately, and it was larger than the sprinklers could’ve made. Down here in Texas almost everyone has a sprinkler system -- you kind of have to or your grass and plants will die and your house might come down with foundation problems. So everything has to stay somewhat "moist."

Anyway, it was obvious I had a problem. So I was turning water on and off and running the sprinkler and the hose and then I stopped. And started to think like a DBA. 

What does that mean? 

Well, one of my cardinal rules whenever confronted with a database problem is to do one thing at a time -- otherwise, if you’ve made multiple changes how do you know which helped; and which might’ve hurt? You don’t. 

I had violated this rule in my backyard. So I went back to doing one thing at a time. I figured that would give me the best chance to actually find the problem. This approach might take longer but at least it has a chance of success.

Slowly I stepped through turning on and off valves. And then I realized unless there was a huge flow of water leaking somewhere that I might not even notice the leak. It made more sense to turn everything off and let the standing water evaporate. And that takes even more time.

And then I decided it was time to call in an expert. Let them worry about my pipes and leaks -- and I’ll worry about the database!

Note: The leak was caused by an underground pipe that the experts found quickly using equipment I didn't have (and didn't even know existed). Another good reason to leave things to the experts!

Thursday, February 01, 2018

Db2 Application Development for Performance: Be Early and Be Informed

The title of this blog post is also the title of one of my two IDUG Db2 Tech Conference presentations in Philadelphia this year. I chose this topic because I am in the process of writing a book on Db2 performance from a developer perspective.

There are a lot of performance-focused presentations but most of them are from the perspective of monitoring and tuning. In other words, they are about looking for and fixing problems in code that already exists. Now there is nothing wrong with this. In fact, it is necessary. Even if everything is written correctly from the get-go (right, sure it is) over time things will change and performance will decline. So we need tools to ferret out what is going wrong and help us to fix it. This is mostly the domain of the DBA to do.

From the programmer's point-of-view, though, it seems that things are lacking. I frequently see SQL and program code that seems to have been written by someone with no understanding of relational basics. This has to change.

That is the reason for my IDUG presentation... and also for the book. The idea is to give advice and direction to Db2 application developers and programmers on writing efficient, well-performing programs. If you write code and access data in a Db2 database, then this book and presentation should be helpful to you.

The material is written for all Db2 professionals. It will be useful for users of Db2 for Linux, Unix and Windows as well as for users of Db2 for z/OS. When there are pertinent differences between the two I point it out. Also, much of the material will apply to any relational database system, not just Db2; nevertheless, the book is written and intended for Db2 users.

I don't try to teach basic programming skills, other than, of course, some guidance on SQL. And even that is not basic. I assume you can figure out a basic Select, Insert, Update or Delete... and even some more complex stuff like joins. The focus of the book is on programming, coding and developing applications.

If you are a DBA, most of the material will still be of interest to you because DBAs are responsible for overall Db2 performance. Therefore, it makes sense to understand the programming aspect of performance. That said, I will not be covering DBA and system administration level performance. But if you think the material is worthy, knowing it is there to recommend to your developers (new and old) can be worthwhile!

The book and presentation focus on guidance regarding application development procedures, techniques, and philosophies. The goal is to educate developers on how to write good application code that lends itself to optimal performance. Of course, this does not mean that every program you write will perform at top speed. But it should enable you to write code that does not require remedial after-the-fact modifications by performance analysts. If you follow the guidelines I outline in this book and presentation, I can say with confidence that your DBAs and performance analysts will love you!

So if you are going to IDUG in Philadelphia this year, be sure to attend my presentation. Be sure to say "Howdy!" and that this blog post guided you to the presentation... and also, keep an eye on my web site, Mullins Consulting, Inc., for information about the book when it gets published, hopefully later in 2018.