Les cookies nous permettent de vous proposer nos services plus facilement. En utilisant nos services, vous nous donnez expressément votre accord pour exploiter ces cookies.En savoir plus OK

High Availability On the AS/400 System A System Manager’s Guide

High Availability On the AS/400 System A System Manager’s Guide Revenir à l'accueil

 

Au format "texte" :

Redpaper ibm.com/redbooks High Availability On the AS/400 System A System Manager’s Guide Susan Powers Nick Harris Ellen Dreyer Andersen Sue Baker David Mee Tools and solutions to improve your highly available AS/400 system Components for a successful high availability system Hardware options for high availability Front cover International Technical Support Organization High Availability On the AS/400 System: A System Manager’s Guide June 2001 © Copyright International Business Machines Corporation 2001. All rights reserved. Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp. First Edition (June 2001) This edition applies to Version 4, Release Number 5 of OS/400 product number 5769-SS1. Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. JLU Building 107-2 3605 Highway 52N Rochester, Minnesota 55901-7829 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. Before using this information and the product it supports, be sure to read the general information in Appendix H, “Special notices” on page 183. Take Note! iii Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Comments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Part 1. What is high availability? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Chapter 1. Background. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 1.1 When to consider a high availability solution . . . . . . . . . . . . . . . . . . . . . . . .3 1.1.1 What a high availability solution is. . . . . . . . . . . . . . . . . . . . . . . . . . . .3 1.2 What high availability is. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 1.2.1 Levels of availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 1.3 Determining your availability requirements . . . . . . . . . . . . . . . . . . . . . . . . .7 1.4 Determining how high you need to go . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 1.5 Estimating the value of availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 1.6 iSeries factors for maximum availability . . . . . . . . . . . . . . . . . . . . . . . . . . .9 1.7 Scheduled versus unscheduled outage . . . . . . . . . . . . . . . . . . . . . . . . . . .10 1.7.1 Scheduled outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 1.7.2 Unscheduled outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 1.8 Comparison to planned preventive maintenance (PPM) . . . . . . . . . . . . . .11 1.9 Other availability definition considerations . . . . . . . . . . . . . . . . . . . . . . . .12 Chapter 2. Developing an availability plan . . . . . . . . . . . . . . . . . . . . . . . . .15 2.1 The business plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 2.1.1 Project scope and goal definition. . . . . . . . . . . . . . . . . . . . . . . . . . . .16 2.2 Human resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 2.2.1 Project organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 2.3 Communication and sponsorship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 2.4 Service level agreements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 2.5 Third party contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 2.5.1 Application providers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 2.5.2 Operating system provider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 2.5.3 Hardware providers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 2.5.4 Peripheral equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 2.5.5 Facilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 2.6 Verifying the implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 2.6.1 Documenting the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 2.7 Rollout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 Chapter 3. High availability example solutions . . . . . . . . . . . . . . . . . . . . . .25 3.1 A high availability customer: Scenario 1 . . . . . . . . . . . . . . . . . . . . . . . . . .25 3.2 A large financial institution: Scenario 2 . . . . . . . . . . . . . . . . . . . . . . . . . . .26 3.2.1 Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 3.3 A large retail company: Scenario 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 3.4 A small manufacturing company: Scenario 4. . . . . . . . . . . . . . . . . . . . . . .29 3.5 A distribution company: Scenario 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 Part 2. AS/400 high availability functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 Chapter 4. Hardware support for single system high availability . . . . . . .35 4.1 Protecting your data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 iv High Availability on the AS/400 System: A System Manager’s Guide 4.2 Disk protection tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Disk mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3.1 Standard mirrored protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.3.2 Mirrored protection: Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.3.3 Mirrored protection: Costs and limitations . . . . . . . . . . . . . . . . . . . . 39 4.3.4 Determining the level of mirrored protection. . . . . . . . . . . . . . . . . . . 40 4.3.5 Determining the hardware required for mirroring . . . . . . . . . . . . . . . 44 4.3.6 Mirroring and performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3.7 Determining the extra hardware required for performance . . . . . . . . 46 4.4 Remote DASD mirroring support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.4.1 Remote load source mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.4.2 Enabling remote load source mirroring. . . . . . . . . . . . . . . . . . . . . . . 47 4.4.3 Using remote load source mirroring with local DASD . . . . . . . . . . . . 47 4.5 Planning your mirroring installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.5.1 Comparing DASD management with standard and remote mirroring 49 4.6 Device parity protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.6.1 How device parity protection affects performance . . . . . . . . . . . . . . 51 4.6.2 Using both device parity protection and mirrored protection . . . . . . . 52 4.7 Comparing the disk protection options . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.8 Concurrent maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.9 Redundancy and hot spare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.10 OptiConnect: Extending a single system . . . . . . . . . . . . . . . . . . . . . . . . 55 4.11 Cluster support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.12 LPAR hardware perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.12.1 Clustering with LPAR support . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.13 UPS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.14 Battery backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.15 Continuously powered main storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.16 Tape devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.16.1 Alternate installation device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Chapter 5. Auxiliary storage pools (ASPs). . . . . . . . . . . . . . . . . . . . . . . . . 63 5.1 Deciding which ASPs to protect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.1.1 Determining the disk units needed . . . . . . . . . . . . . . . . . . . . . . . . . . 64 5.2 Assigning disk units to ASPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.3 Using ASPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.3.1 Using ASPs for availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.3.2 Using ASPs to dedicate resources or improve performance. . . . . . . 66 5.3.3 Using ASPs with document library objects . . . . . . . . . . . . . . . . . . . . 67 5.3.4 Using ASPs with extensive journaling . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.5 Using ASPs with access path journaling . . . . . . . . . . . . . . . . . . . . . 68 5.3.6 Creating a new ASP on an active system. . . . . . . . . . . . . . . . . . . . . 68 5.3.7 Making sure that your system has enough working space . . . . . . . . 69 5.3.8 Auxiliary storage pools: Example uses. . . . . . . . . . . . . . . . . . . . . . . 69 5.3.9 Auxiliary storage pools: Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.3.10 Auxiliary storage pools: Costs and limitations . . . . . . . . . . . . . . . . 70 5.4 System ASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.4.1 Capacity of the system ASP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.4.2 Protecting your system ASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.5 User ASPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.5.1 Library user ASPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 v Chapter 6. Networking and high availability . . . . . . . . . . . . . . . . . . . . . . . .75 6.1 Network management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75 6.2 Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 6.3 Network components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 6.4 Testing and single point of failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78 6.5 Hardware switchover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80 6.6 Network capacity and performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81 6.7 HSA management considerations with networking . . . . . . . . . . . . . . . . . .81 6.7.1 Network support and considerations with a HAV application . . . . . . .82 6.8 Bus level interconnection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82 6.8.1 Bus level interconnection and a high availability solution. . . . . . . . . .84 6.8.2 TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84 Chapter 7. OS/400: Built-in availability functions . . . . . . . . . . . . . . . . . . . .87 7.1 Basic OS/400 functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 7.1.1 Journaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 7.1.2 Journal receivers with a high availability business partner solution . .88 7.2 Commitment control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 7.2.1 Save-while-active with commitment control . . . . . . . . . . . . . . . . . . . .90 7.3 System Managed Access Path Protection (SMAPP) . . . . . . . . . . . . . . . . .90 7.4 Journal management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 7.4.1 Journal management: Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92 7.4.2 Journal management: Costs and limitations . . . . . . . . . . . . . . . . . . .92 7.5 Logical Partition (LPAR) support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93 7.6 Cluster support and OS/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 Chapter 8. Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95 8.1 Foundations for good performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95 8.1.1 Symmetric multiprocessing (SMP). . . . . . . . . . . . . . . . . . . . . . . . . . .95 8.1.2 Interactive jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .96 8.1.3 Batch jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .96 8.1.4 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .96 8.2 Journaling: Adaptive bundling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97 8.2.1 Setting up the optimal hardware environment for journaling . . . . . . .98 8.2.2 Setting up your journals and journal receivers . . . . . . . . . . . . . . . . . .98 8.2.3 Application considerations and techniques of journaling . . . . . . . . . .99 8.3 Estimating the impact of journaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100 8.3.1 Additional disk activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100 8.3.2 Additional CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .100 8.3.3 Size of your journal auxiliary storage pool (ASP). . . . . . . . . . . . . . .100 8.4 Switchover and failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101 Part 3. AS/400 high availability solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103 Chapter 9. High availability solutions from IBM . . . . . . . . . . . . . . . . . . . .105 9.1 IBM DataPropogator/400. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105 9.1.1 DataPropagator/400 description . . . . . . . . . . . . . . . . . . . . . . . . . . .106 9.1.2 DataPropagator/400 configuration. . . . . . . . . . . . . . . . . . . . . . . . . .107 9.1.3 Data replication process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .107 9.1.4 OptiConnect and DataPropagator/400. . . . . . . . . . . . . . . . . . . . . . .108 9.1.5 Remote journals and DataPropagator/400. . . . . . . . . . . . . . . . . . . .109 9.1.6 DataPropagator/400 implementation . . . . . . . . . . . . . . . . . . . . . . . .109 9.1.7 More information about DataPropagator/400 . . . . . . . . . . . . . . . . . .109 vi High Availability on the AS/400 System: A System Manager’s Guide Chapter 10. High availability business partner solutions . . . . . . . . . . . . 111 10.1 DataMirror Corporation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 10.1.1 DataMirror HA Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 10.1.2 ObjectMirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 10.1.3 SwitchOver System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 10.1.4 OptiConnect and DataMirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 10.1.5 Remote journals and DataMirror . . . . . . . . . . . . . . . . . . . . . . . . . 114 10.1.6 More information about DataMirror. . . . . . . . . . . . . . . . . . . . . . . . 114 10.2 Lakeview Technology solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 10.2.1 MIMIX/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 10.2.2 MIMIX/Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 10.2.3 MIMIX/Switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 10.2.4 MIMIX/Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 10.2.5 MIMIX/Promoter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 10.2.6 OptiConnect and MIMIX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 10.2.7 More Information About Lakeview Technology . . . . . . . . . . . . . . . 119 10.3 Vision Solutions: About the company. . . . . . . . . . . . . . . . . . . . . . . . . . 119 10.3.1 Vision Solutions HAV solutions . . . . . . . . . . . . . . . . . . . . . . . . . . 119 10.3.2 Vision Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 10.3.3 OMS/400: Object Mirroring System . . . . . . . . . . . . . . . . . . . . . . . 123 10.3.4 ODS/400: Object Distribution System . . . . . . . . . . . . . . . . . . . . . 124 10.3.5 SAM/400: System Availability Monitor . . . . . . . . . . . . . . . . . . . . . 124 10.3.6 High Availability Services/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 10.3.7 More information about Vision Solutions, Inc. . . . . . . . . . . . . . . . 126 Chapter 11. Application design and considerations . . . . . . . . . . . . . . . . 127 11.1 Application coding for commitment control. . . . . . . . . . . . . . . . . . . . . . 127 11.2 Application checkpointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 11.3 Application checkpoint techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 11.3.1 Historical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 11.4 Application scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 11.4.1 Single application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 11.4.2 CL program example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 Chapter 12. Basic CL program model . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 12.1 Determining a job step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 12.1.1 Summary of the basic program architecture . . . . . . . . . . . . . . . . . 133 12.2 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 12.2.1 Distributed relational database. . . . . . . . . . . . . . . . . . . . . . . . . . . 133 12.2.2 Distributed database and DDM . . . . . . . . . . . . . . . . . . . . . . . . . . 134 12.3 Interactive jobs and user recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 12.4 Batch jobs and user recovery and special considerations . . . . . . . . . . 135 12.5 Server jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 12.6 Client Server jobs and user recovery . . . . . . . . . . . . . . . . . . . . . . . . . . 137 12.7 Print job recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Part 4. High availability checkpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Appendix A. How your system manages auxiliary storage . . . . . . . . . . . .141 A.1 How disks are configured. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .141 A.2 Full protection: Single ASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142 A.3 Full protection: Multiple ASPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143 A.4 Partial protection: Multiple ASPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144 vii Appendix B. Planning for device parity protection. . . . . . . . . . . . . . . . . . . 147 B.1 Mirrored protection and device parity protection to protect the system ASP. 147 B.2 Mirrored protection in the system ASP and device parity protection in the user ASPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 B.2.1 Mirrored protection and device parity protection in all ASPs. . . . . . . . . 149 B.2.2 Disk controller and the write-assist device . . . . . . . . . . . . . . . . . . . . . . 150 B.2.3 Mirrored protection: How it works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Appendix C. Batch Journal Caching for AS/400 boosts performance. . . 153 C.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 C.2 Benefits of the Batch Journal Caching PRPQ. . . . . . . . . . . . . . . . . . . . . . . . 153 C.2.1 Optimal journal performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 C.3 Installation considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 C.3.1 Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 C.3.2 Limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 C.3.3 For more information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Appendix D. Sample program to calculate journal size requirement . . . 155 D.1 ESTJRNSIZ CL program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 D.2 NJPFILS RPGLE program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 D.3 Externally described printer file: PFILRPT . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Appendix E. Comparing availability options . . . . . . . . . . . . . . . . . . . . . . . . 167 E.1 Journaling, mirroring, and device parity protection . . . . . . . . . . . . . . . . . . . . 167 E.2 Availability options by time to recover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Appendix F. Cost components of a business case. . . . . . . . . . . . . . . . . . . 169 F.1 Costs of availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 F.1.1 Hardware costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 F.1.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 F.2 Value of availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 F.2.1 Lost business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 F.3 Image and publicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 F.4 Fines and penalties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 F.5 Staff costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 F.6 Impact on business decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 F.7 Source of information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 F.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 Appendix G. End-to-end checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 G.1 Business plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 G.1.1 Business operating hours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 G.2 High availability project planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 G.3 Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 G.4 Facilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 G.4.1 Power supply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 G.4.2 Machine rooms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 G.4.3 Office building. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 G.4.4 Multiple sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 G.5 Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 G.6 Systems in current use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 G.6.1 Hardware inventory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 G.6.2 Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 G.6.3 LPAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 viii High Availability on the AS/400 System: A System Manager’s Guide G.6.4 Backup strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180 G.6.5 Operating systems version by system in use . . . . . . . . . . . . . . . . . . . .180 G.6.6 Operating system maintenance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180 G.6.7 Printers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180 G.7 Applications in current use. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .181 G.7.1 Application operational hours current . . . . . . . . . . . . . . . . . . . . . . . . . .181 Appendix H. Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183 © Copyright IBM Corp. 2001 ix Preface Availability and disaster recovery represents a billion dollar industry in the United States alone. Professional associations and institutes, such as the Association of Contingency Planners, Business Resumption Planning Association, Contingency Planning and Recovery Institute, and their associated journals and magazines are devoted to keeping an information system (and, therefore, the business) available to both internal and external business users. The growth of e-business further emphasizes the need to maintain system availability. Implementing a high availability solution is a complex task that requires diligent effort and a clear view of the objectives to be accomplished. The key to the process is planning and project management. This includes planning for an event, such as an outage, that may never occur, and project management with the discipline to dogmatically prepare, test, and perform for business resumption. Planning is paramount to the health of a highly available business. This Redpaper is intended to help organize the tasks and simplify the decisions involved in planning and implementing a high availability solution. While some of the most relevent items are covered, this Redpaper cannot cover all cases because every situation is unique. To assist IT managers with understanding the most important facts when planning to implement a high availability solution, detailed information is provided. This information can help business partners and IBMers to discuss high availability considerations with customers. In addition, this Redpaper provides examples of highly available solutions, the hardware involved in AS/400 availability solutions, and OS/400 operating system options that add to the reliability of the system in an availability environment. Application software and how it affects an availability solution are also discussed. Significant players in the solution are the business partners who provide the high availability middleware. In addion to discussing their products, a checklist is provided to help to establish a planning foundation. Note: A service offering is available from IBM for examining and recommending availability improvements. Contact your IBM marketing representative for further information. The team that wrote this Redpaper This Redpaper was produced by contributions from a team of specialists from around the world working at the International Technical Support Organization, Rochester Center. Susan Powers is a Senior I/T Specialist at the International Technical Support Organization, Rochester Center. Prior to joining the ITSO in 1997, she was an AS/400 Technical Advocate in the IBM Support Center specializing in a variety of communications, performance, and work management assignments. Her IBM career began as a Program Support Representative and Systems Engineer in Des Moines, Iowa. She holds a degree in mathematics, with an emphasis in x High Availability on the AS/400 System: A System Manager’s Guide education, from St. Mary’s College of Notre Dame. She served as the project leader for this redbook. Nick Harris is a Senior Systems Specialist for the AS/400 system in the International Technical Support Organization, Rochester Center. He specializes in server consolidation and the Integrated Netfinity Server. He writes and teaches IBM classes worldwide in areas of AS/400 system design, business intelligence, and database technology. He spent 11 years as a System Specialist in the United Kingdom AS/400 Business and has experience in S/36, S/38, and the AS/400 system. Nick served to outline the requirements and set much of the direction of this Redpaper. Ellen Dreyer Andersen is an Certified IT Specialist in Denmark. She has 21 years of experience working with the AS/400 and System/3x platforms. Since 1994, Ellen has specialized in AS/400e Systems Management with a special emphasis on performance, ADSM/400, and high availabiity solutions. Sue Baker is a Certified Consulting I/T Specialist working on the Advanced Technical Support team with IBM in Rochester. She has worked over 15 years with IBM mid-range system customers, in the industries of manufacturing, transportation, distribution, education, and telecommunications. She currently focuses on developing and implementing performance, capacity planning, and operations management techniques needed in the more complex multiple system and high availabiliy customer environment. David Mee is a Strategic Accounts Project Manager in the Global Strategic Accounts Group of Vision Solutions. He specializes in application and database design, as well as integration and implementation of high availability solutions worldwide. He has over 15 years of experience in IBM Midrange systems, and holds a computer science degree with additional certificates from UCI and UCLA in RPG, Cobol, Pascal, C and Visual Basic programming languages. He writes and teaches classes on high availability, mirroring, and application resiliency for Vision Solutions. Thanks to the following people for their invaluable contributions to this project: Steve Finnes, Project Sponsor Segment Manager, AS/400 Brand Bob Gintowt, RAS Architecture Availability/Recovery and Limits to Growth, IBM Rochester laboratory Fred L. Grunewald Vision Solutions, Inc. Glenn Van Benschoten, Director Product Marketing Lakeview Technology Michael Warkentin, Senior Product Specialist DataMirror xi Comments welcome Your comments are important to us! We want our Redpapers to be as helpful as possible. Please send us your comments about this or other Redbooks in one of the following ways: • Use the online evaluation form found at http://www.redbooks.ibm.com/ • Send your comments in an Internet note to redbook@us.ibm.com xii High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 1 Part 1. What is high availability? Part 1 of this Redpaper discusses what high availability is. Levels of availability are discussed, as well as outage types, factors comprising an availability plan, and examples of high availability solutions. 2 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 3 Chapter 1. Background Early systems were considered available when they were up and running. As the demands of business, communications, and customer service grew, systems had to be up and running through the normal working day (usually 8 to 10 hours). Failures during this working period were not acceptable. In availability terms, this was a 5 x 8 service (five days at 8 hours each day). If a system was unavailable during this period, rapid recovery was necessary. Backups would be restored and the system and the database were inspected for integrity. This process could take days for larger databases. These ocurrences eventually led to the definition of availability. In general, availability means the amount of service disruption that is acceptable to the end user. This Redpaper provides insight into the challenges and choices a system manager may encounter when embarking on a project to make a business more highly available. This Redpaper does not provide a detailed technical setup of OS/400 or application products. This information is covered in other technical publications, for example the AS/400 Software Installation guide, SC41-5120. 1.1 When to consider a high availability solution When considering if a high availability solution is right for you, ask yourself these questions: • Will we benefit from using synchronized distributed databases? • Do our users need access to the AS/400 system 24 hours a day, 365 days a year? • Do our users operate in different time zones? • Is there enough time for nightly backups, scheduled maintenance, or installing new releases? • If our telephone sales application is not always up and running, will we lose our customers to the competition? • Is there a single point of failure for any data center? • Can we avoid the loss of data or access to the system in the event of a disaster or sabotage? • When the production machine is overloaded, can we move some users to a different machine for read only jobs? A high availability solution can benefit any of these situations. 1.1.1 What a high availability solution is High (or continuous) availability systems usually include an alternate system or CPU that mirrors some of the activity of the production system, and a fast communications link. These systems also include replication or mirroring software and enough DASD to handle the volume of data for a reasonable recovery time as shown in Figure 1 on page 4. 4 High Availability on the AS/400 System: A System Manager’s Guide Figure 1. The basics of a high availability solution Table 1 outlines some of the requirements of an HA solution. Tally these features to help simplify your investigation of high availability solutions. Table 1. Requirements of a high availability solution Features Solution A Solution B Solution C Data Propagator 24 x 7 availability No Eliminate downtime for backup and maintenance No Replication of database Yes Replication of other objects No Data replication to non-AS/400 systems Yes Handle unplanned outages No Automatically switch users to a target system No Workload distribution Yes Error recovery Yes Distribution to multiple AS/400 systems Yes Commitment control support Production Machine Mirroring to one more AS/400s Backup Machine 2 Backup Machine Background 5 Figure 2 illustrates a business operating with high availability. Figure 2. A highly available business Note the redundancy of communications links, servers, the distributed work environment, and the use of backup power. 1.2 What high availability is High availability is a much maligned phrase because it can have different meanings and is dependant on discussion variables. This Redpaper takes a holistic approach by discussing high availability as it applies to an organization, rather than an individual product or feature. This broadens the scope tremendously and prevents tunnel vision. Sync checks Yes Filtering of mirrored objects Yes (DB only) Execute remote commands No OptiConnect support Yes Utilize Remote Journals Yes Features Solution A Solution B Solution C Data Propagator Network Distribution site Manufacturing site Machine room B AC UPS AC UPS Application servers Machine room A Application servers Database servers Database servers Routers Routers Customers Suppliers Shop floor data collection WAN Network Network LAN Network Network 6 High Availability on the AS/400 System: A System Manager’s Guide High availability in this context states that an organization, or part of an organization, can respond to a third-party request using the organization’s normal business model. Normal is defined as including set levels of availability. Requests on the business can be anything, such as a sales call, an inventory inquiry, a credit check, or an invoice run. High availability is achieved by having an alternative system that replicates the availability of the production system. These systems are connected by high-speed communications. High availability software is used to achieve the replication. Chapter 10, “High availability business partner solutions” on page 111, discusses these solutions for the AS/400e system. 1.2.1 Levels of availability Information systems experience both planned and unplanned outages. Systems can be classified according to the degree to which they cope with different types of outages. Your system can be as available as it is planned or designed to be. Orient your implementation choices toward your desired level of availability. These levels include: • High availability: High availability relates to keeping an application available during planned business (service) hours. Systems provide high availability by delivering an acceptable or agreed upon level of service to the business during scheduled periods of operation. The system is protected in this high availability type of environment to recover from failures of major hardware components, such as a CPU, disks, and power supplies when an unplanned outage occurs. This involves redundancy of components to ensure that there is always an alternative available if something breaks. It also involves conducting thorough testing to ensure that any potential problems are detected before they can affect the production environment. • Continuous operations: Continuous operations means that a system can provide service to its users at all times without outages (planned or otherwise). A system that has implemented continuous operations is capable of operating 24 hours a day, 365 days a year with no scheduled outage. This does not imply that the system is highly available. An application can run 24 hours a day, 7 days a week and still be available only 95% of the time because of unscheduled outages. When unscheduled outages occur, they are typically short in duration and recovery actions are unnecessary or minimal. The prerequisite for continuous operations is that few or no changes can be made to the system. In a normal production environment, this is a very unlikely scenario. • Continuous availability: This type of availability is similar to continuous operations. Continuous availability is a combination of high availability and continuous operations. This means that the applications remain available across planned and unplanned system outages and must be implemented on system, application, and business levels. Continuous availability systems deliver an acceptable or agreed upon service 7 days a week, 24 hours a day. They add to availability provided by fault-tolerant systems by tolerating both planned and unplanned outages. With Background 7 continuous availability, you can avoid losing transactions. End users do not have to be aware that a failure or outage has occurred in the total environment. In reality, when people say they need continuous availability, they usually mean that they want the application to be available at all times during the agreed service hours, regardless of problems with, or changes to, the underlying hardware or software. What makes this more stringent than high availability is that the service hours get longer and longer, to the point that there is no time left for making changes to any of the system components. Note: The total environment consists of the computer, the network, workstations, applications, telephony, site facilities, and human resources. The levels of protection that a total environment offers depends on how many of these functions are wrapped into the integrated solution. 1.3 Determining your availability requirements When most people are first asked how much availability they require, they often reply that they want continuous availability. However, the high cost of continuous availability often makes such requirements unrealistic. The question usually comes down to how much availability someone can afford. There are not many applications that can justify the cost of 100% availability. The cost of availability increases dramatically as you get closer to 100%. Moving from 90% to 97%, for example, probably costs nothing more than better processes and practices, and very little in terms of additional hardware or software. Moving from 97% to 99.9% requires investing in the latest hardware technology, implementing very good processes and practices, and committing to staying current on software levels and maintenance. At the highest extreme, 99.9% availability equates to 8.9 hours downtime a year. 99.998% equates to just 10 minutes unplanned downtime a year. Removing that last ten minutes of downtime is likely to be more costly than moving from 99.9% to 99.998%. What may be more beneficial, and less expensive, is to address planned outages. In an IBM study, planned outages accounted for over 90% of all outages. Of the planned ones, about 40% are for hardware, software, network, or application changes. Appendix F, “Cost components of a business case” on page 169, helps you decide if the value of availability to the application justifies the expense. 1.4 Determining how high you need to go It was previously mentioned that the former version of availability translated into a customer’s access to the business. A 9 a.m. to 5 p.m. time frame is a valid form of availability requirement. However, today the terms “24 x 7” or “24 x 365” are more commonly used. Even high availability is often substituted by the term “continuous availability”. The term The 9s is also popular and means a 99% availability. On its face, this seems like a high requirement. However, if you analyze what this value means in business terms, it says that your process is available 361.35 days per year. In 8 High Availability on the AS/400 System: A System Manager’s Guide other words, for 3.65 days, the process is not available. This equates to 87.6 hours, which is a huge and unacceptable amount of time for some businesses. In recent press articles, up to $13,000 a minute has been quoted as a potential loss. This is a $68,328,000 a year lost potential. It is not difficult to justify a high availability solution when confronted by these sorts of numbers. However, it is one of the lessons to learn as well as an obstacle to block and tackle. The business can potentially lose this amount of money. When planning for a high availability solution, convincing the business to commit to even a fraction of this amount is difficult. However, use any recent unplanned or planned outages to estimate the cost. It is recommended that you start with a departmental analysis. For example, how much would it cost the business if the salespeople could not accept orders for two hours due to a system failure? To continue using the 9s, add .9% to your figure of 99%. This now equals 99.9% availability. Although this is obviously very close to 100%, it still equates to 8.76 hours. The AS/400e has been quoted as offering 99.94% availablility or 5.1 hours. This is according to a recent Gartner Group report, “AS/400e has the highest availability of any stand-alone, general business server: 99.94 percent” (“Platform Availability Data: Can You Spare a Minute?” Gartner Group, 10/98). This applies to the hardware and operating system and unplanned outages. It does not apply to application or scheduled outages. 1.5 Estimating the value of availability The higher the level of availability, the higher the investment required. It is important to have a good understanding of the dollar value that IT systems provide to the business, as well as the costs to the business if these systems are not available. This exercise can be time consuming and difficult when you consider the number of variables that exist within the company. Some companies delay the analysis. Once the value of the availability of your IT services is determined, you have an invaluable reference tool for establishing availability requirements, justifying appropriate investments in availability management solutions, and measuring returns on that investment. The estimation process should: • Analyze by major application or by services provided: The major cost of an outage is the cumulative total of not having the applications available to continue business. • Determine the value of system availability: It is not easy to determine the cost of outages. The inaccessibility of each application or program has varying effects on the productivity of its users. Start with a reasonable estimation of what each critical application is worth to the business. Some applications are critical throughout major portions of the day, while others can be run any time or on demand. • Look at direct versus indirect costs: Direct costs are the time and revenue lost directly because a system is down. Indirect costs are those incurred by another department or function as a result of an outage. For example, a Background 9 marketing department may absorb the cost of a manufacturing line being shut down because the system is unavailable. This is an indirect cost of the outage, but it is nonetheless a real cost to the company. • Consider tangible versus intangible costs: Tangible costs are direct and indirect costs that can be measured in dollars and cents. Intangible costs are those for which cash never changes hands, such as lost opportunity, good will, market share, and so on. • Analyze fixed versus variable costs: Fixed costs are direct, indirect, tangible, or intangible costs that result from a failure, regardless of the outage length. Variable costs are those that vary with the duration of the down time, but that are not necessarily directly proportional. For more detailed calculations and methodology, refer to So You Want to Estimate the Value of Availability, GG22-9318. 1.6 iSeries factors for maximum availability The IBM ~ iSeries and AS/400e systems are renown for their availability due to a number of factors: • Design for availability: A single iSeries server delivers an average of 99.9+% availability. According to data collected by IBM over the last two years, AS/400 and iSeries owners have experienced an average of less than nine hours of unplanned down time per year. Figure 3 indicates two out of three factors of unplanned outages that can be affected by proper design. IBM delivers a very reliable server because the IBM Development team designs, creates, builds, tests, and services the iSeries and AS/400 systems as a single entity. • Effective system management process: As noted in Figure 4, 90 percent available translates to 36 days. Lack of attention to system management disciplines and processes affect the availability achieved. Availability solutions, such as clusters, are undermined when system management processes are lacking or nonexistent. An effective system management strategy ties heavily into automation, such as an automatic archival of data, continuous system auditing, responding to security exposures, and monitoring error logs, backup and restores, and so on. A quote by Gartner Group puts this in perspective: By the year 2003, “100% availability will remain elusive as user-controlled disciplines have an 40.0% 40.0% 20.0% Hardware, disasters, power Operator Error Application Failure Figure 3. Unplanned outage factors Availability Percentage 99.9999 99.999 99.99 99.9 99 90 Total Outage Per Year 32 seconds 5 minutes 53 minutes 8.8 hours 87 hours (3.6 days) 876 hours (36 days) = = = = = = Figure 4. Unplanned outage factors 10 High Availability on the AS/400 System: A System Manager’s Guide ever-greater relative impact on achieving availability” (Gartner Group, June Conference, Dallas, Tx., 1997). An investment in system management disciplines, automation, and application recovery is necessary. Just a few additional hours of yearly downtime reduces availability from 99.99% availability to 99.9%. • Increase automation: Increased availability means a reduction in the possibility of errors and recovery delays, and an increase in consistency. Human errors can create more down time than hardware failures or power outages. More effective automation through the use of automation software and tools can help offset an overburdened staff and allow them to attend to more unique and critical decisions and tasks. As availability requirements increase, investments in automation must also increase. • Exploit availability techniques and applications designed for availability: Decrease unplanned outages and their effects by utilizing server availability options (for example, disk protection) and OS/400 functions, such as access path protection, file journaling, and user auxiliary storage pools (ASPs). Target a phased approach at increasing application resiliency (recoverability) and availability. As a general rule, an application on a non-clustered system is difficult to recover. A cluster solution cannot overcome a poor application design. Use applications that incorporate commitment control or other transaction methods, and use application recovery to decrease application recovery times and database integrity issues (incomplete transactions). You must use journaling and application recovery techniques to achieve high availability at a minimum. More sophisticated and highly available applications also use commitment control. Each technique is a building block foundation to all highly available environments. • Implement special solutions to meet your availability goals: To reach your availability goals, special solutions, such as iSeries or AS/400 clusters with monitoring, automatic switchover, and recovery automation, are implemented to control both planned and unplanned outages. If you sidestep the issues described above, even sophisticated options like clusters may not provide the highest possible availability levels. Small outages, such as recovering or reentering transactions, add up. 1.7 Scheduled versus unscheduled outage Many Information Technology departments have a good understanding of disaster recovery protection, which includes unscheduled outages. It can also involve anything from dual site installations to third-party vendors offering disaster recovery suites. Most of these installations provide protection from fires, floods, a tornado or an airplane crash on the site. The instance of the dangers affecting the site are very rare, but, if they do happen, they are catastrophic. Scheduled outages can impact a business’ financial well-being. Strong focus is warranted to plan for and minimize the time involved for scheduled outages. There has been little focus in this area, except for hardware and software vendors. Background 11 1.7.1 Scheduled outages A scheduled outage is a form of planned business unavailability. Scheduled outages include a production line shutdown for maintenance, the installation of a new PABX, resurfacing the car park, or the entire sales team leaving town for a convention. All of these can influence the smooth operation of the business and, therefore, the customers. In I/T terms, these outages may be due to a hardware upgrade, operating system maintenance or upgrade, an upgrade of an application system, network maintenance or improvements, a workstation maintenance or upgrade, or even a nightly backup. The focus of outage discussion is shifting from unplanned outages (disasters, breakdowns, floods, and the like), to planned outages (primarily nightly backups, but also upgrades, OS/400 updates, application updates, and so forth). A growing solution is to consolidate individual systems onto large systems, rather than install distributed systems in departments or subsidiaries. This suggests that you may have users in different time zones, which can further minimize or eliminate the time available for routine operations, such as a backup. With a high availability solution and mirrored systems, customers can perform their daily backup on the backup system and let the users and work be done on the production system at the same time. 1.7.2 Unscheduled outages Unscheduled outages include obscure occurrences. As mentioned earlier, these can include such things as fires, floods, storms, civil unrest, sabotage, and other assorted happenings. These outages are fairly well recognized and many organizations have disaster recovery plans in place to account for these types of occurrences. Testing these disaster plans should not be overlooked. We identify this in Appendix G, “End-to-end checklist” on page 175. 1.8 Comparison to planned preventive maintenance (PPM) When researching this Redpaper, the authors found that there are only a few publications that relate to high availability for Information Infrastructure within organizations. They then looked laterally and found some good sources regarding practices and processes in the engineering aspect. For many years, most manufacturing companies ran their businesses based on sound planned preventive maintenance programs. These programs cover the plant systems and services that support the manufactured product. Companies have long understood terms such as resiliency, availability, mean time to failure, and cost of failure. To define the planned preventive maintenance schedule for a production line is a very complex operation. Some simple examples of the high level tasks that need to be performed include: • Documenting a business plan: This includes business forecasts, current product forecasts, new product forecasts, and planned business outages (vacation, public holidays). 12 High Availability on the AS/400 System: A System Manager’s Guide • Documenting environmental issues: This includes the frequency of power failures, fires, storms, floods. This also includes civil issues, such as unrest, strikes, layoffs, and morale. • Documenting the processes used: This includes throughput, hours of operation, longevity of the product, and planned product changes. • Taking inventory of all parts involved in the process: This includes purchase costs, purchase dates, history, quality, meantime to failure, replacement cycles, and part availability. • Documenting the maintenance and replacement process • Estimating the cost for running the process and comparing it to the value of the product • Estimating the affordability of a planned preventive maintenance program • Documenting the resources required to manage the process: This includes job specifications, critical skills, and external skills. These tasks can be compared to the following tasks in the Information Management area. Imagine these refer to an application that supports one part of the business: • Documenting the business plan: This includes business forecasts, current business forecasts that the application supports, business growth in this application, planned business outages (vacation, public holidays). • Documenting environmental issues: This includes the frequency of power failures, fires, storms, floods. This also includes civil issues, such as unrest, strikes, layoffs, and morale. • Documenting the processes used: This includes throughput, hours of operation, longevity of the application, and planned application changes. • Taking inventory of all parts involved in the application: This includes purchase costs, purchase dates, history, quality, application failures, replacement cycles, hardware required to run the application, software maintenance schedules, software fix times, time required to implement ad hoc fixes, and developer availability. • Documenting the maintenance and replacement process • Estimating the cost for running the application and comparing it to the value of the business area • Estimating the afford ability of making the application highly available program • Documenting the resources needed to manage the application: This includes application specifications, job specifications, critical skills, and external skills. This information is parsed into smaller sizes that can be applied to any business. 1.9 Other availability definition considerations Designing a solution for high availability requires a practical knowledge of the business. The solution should fit the design of the business operations and structure. Does the organization have an accurate and approved business Background 13 process model? If it does, a major portion of the solution design is easy to plan. If there is no business process model, do not make assumptions at this stage. It is critical to get accurate information since this is the foundation of your plan. Identify the systems and end users of each part of the business process model. Once identified, you have the names of people to relate, in practical terms, just how high is high availability. 14 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 15 Chapter 2. Developing an availability plan For maximum availability, it is very important that planning is performed for the operating system, hardware, and database, as well as for applications and operational processes. As indicated in Figure 5, the level of availability is affected by products and use of technology. An unreliable system, poor systems management, security exposures, lack of automation, and applications that do not provide transaction recovery and restart capability weakens availability solutions. Achieving the highest possible level of availability is accomplished using clustering software, hardware solutions, and the planning and management of high availability products. An availability plan also needs to consider other factors that influence the end result, such as organizational and political issues. It is important to understand the challenges involved in each implementation phase to find the most appropriate tools and techniques for the customer environment. The need for higher availability can be relaxed by determining what a business actually needs and what is possible using the available technology. In reality, most applications can withstand some planned outage, either for batch work, backups, and reorganization of files, or to affect application changes. In most cases, the application is expected to be available at the host, or perhaps on the network that is owned and controlled by the organization. However, continuous availability in the environment outside the control of the organization (that is, on the Internet) is not normally included in the business case. This chapter explores the basic recommendations for planning for system availability. Start by looking at the needs of the business and the information necessary to support it. These actions are separated into the following areas: • Reviewing the business plan • Understanding the human resources issues • Dealing with third party contracts 2.1 The business plan This section gives the reader an insight into the information that is gathered from the business. Some of this is found in the business plan and some from investigating individual departments. Why do you need all this information? To make the business and application highly available, you must take a wider view than just maintaining access to the information (system). It is not enough to have the most highly available system if the telephone system fails and no one can contact your company. 2) Effective Systems Management 4) Application Design and Techniques 5) iSeries and AS/400 Clusters 1) Products, designed for reliability 3) AUTOMATION Level of Availability 100% C osts Figure 5. Essentials for maximum server availability 16 High Availability on the AS/400 System: A System Manager’s Guide When you gather this information or interview staff within the company, you may be questioned as to why you need this information. Some co-workers may deny access to the information itself while some co-workers may roadblock your access to the people with the information. In cases such as these, your executive sponsor plays a very important role. The roadblocks you find can be easily avoided with the sponsor’s help. 2.1.1 Project scope and goal definition The business case, the driving reason for the high availability project, can be used as the starting point for defining the goals and understanding the scope of the project. Requirements and goals should be gathered from any department or party affected by system management. The project team must prioritize the requirements according to their urgency, importance, and dependencies, and to be able to define the short and long term goals. Success criteria and completion criteria should be defined appropriately. Quantifiable results are essential. Even if it is an intangible benefit, such as customer satisfaction, you need to provide concrete methods to measure them. To verify the requirements and goals, several starting points can be used: • Look for established solutions for special requirements. • Find out whether it is possible to develop a solution in a reasonable amount of time and with a reasonable amount of resource. • Visit reference sites. • Discuss the requirements with experienced consulting teams. After this work is done, the first draft of the project definition is ready. Show it to the executive steering committee to get feedback as to whether it reflects their vision. The results from this meeting are the central theme for elaborating on the detailed project definition. Input for this step includes: • Requirements • All information • Type of politics • Vision The result is a clear and unambiguous definition of the scope and the goal definition. 2.2 Human resources Producing an availability plan is not a single-threaded process, and it can not be built on one person’s view. It is critical to build a team. The team should represent the needs of those they represent. Members must understand the department problems, be a voice for their challenges, be able to communicate their needs, and support an action plan. A good project group is critical to this type of planning project. The activities of the business are diverse, represented by members bringing a diverse set of needs and views regarding both the problems and solutions, Developing an availability plan 17 There will be some very tough times, long hours, and unease as the project progresses. Determination, empathy, and leadership are valued skills for the team makeup. Take a look at existing resources with an emphasis on your existing staff. In many cases, the skills required can be found within your staff. In other cases, it makes more sense to look outside the staff for the required talents. The final makeup of the team may represent both personnel inside and outside the company (or the department, if the study is of a more-focused nature). 2.2.1 Project organization The organization of the project team can be categorized in two parts: the members of the team, and the other organization parties involved. Many different skills are required on the team. Criteria for the suitability of each team member includes their skills as well as their availability and personal characteristics, such as how they work on a team, negotiation skills, and knowledge of the technical solution, as well as the workings of the company. A suitable project manager should be selected from within or outside the company. The project manager is the “face” of the project and is responsible for the total health of the project, ensuring that the objectives and goals are met in the highest quality fashion within the specified time schedule and budget. When the team is built, a lot of communication is necessary. Regular status meetings should be held to communicate and assess the actual status of the project, checkpoint the progress, make short-term decisions, and assess remaining activities. The sponsoring executives should be informed of the status in regular meetings. Consider providing an office workspace for the project team, and supply it with applicable reference material for the subject area and company. Sometimes management chooses a project manager from outside the company. Sometimes an organization expert joins the project team to provide the project manager with knowledge about the internal structures of the company. 2.3 Communication and sponsorship The support of the company’s management to the project is critical to its success. The ultimate responsibility for the project should lay with an executive steering committee, or an executive serving as a contact between the team and executives. The steering committee represents the management involvement and sponsorship. It can be used to solve complications with other parts of the organization, communicate expectations to affected company parties, and provide decision making power. The vision of the project scope and objectives should be made clear to everyone involved. This vision must be broadly communicated through the entire organization so that everyone can be aware of the consequences the project has for them. Reactions, especially negative, should be taken seriously and should be used to reach a clearer project vision. Pay special attention to communication because this can prompt people to bring their ideas into the project. 18 High Availability on the AS/400 System: A System Manager’s Guide Non-technical aspects should be emphasized in the project plan so that non-technical people involved can better adopt the project as their own. If it is appropriate, consider having a customer or supplier representative on the team. 2.4 Service level agreements Service level agreements are contracts with business departments within your company and, in many cases, businesses with whom you have an external contractual relationship. If your network resources are not up and running, you can not keep these commitments. Within your company, service requirements probably differ by department. For example, your accounting department may be able to tolerate down time of the accounting applications for up to one hour before it starts negatively impacting department operation. Likewise, marketing may agree to a calendar three-hour down time, but it can allow only an hour of down time on a customer-reference database. You make service level agreements to track your IT organization’s performance against business requirements. Based on these agreements, set priorities and allocate limited IT resources. By linking your IT department to your service desk, you can manage complex relationships among user problems, corporate assets, IT changes, and network events. System management can provide a centralized view of your current asset inventory. This enables your analysts to correctly analyze and resolve problems. Help desk analysts work with administrators to plan and manage the effects of such IT changes as deploying and upgrading applications. System management involves tracking, logging, and escalating user interactions and requests. 2.5 Third party contracts Service level agreements outside your company can mean a loss of business to a competitor if you can’t meet your commitments. This can have serious consequences for your company. Most organizations have contracts with external suppliers. However, the third parties are not all typically under the control of one department. As a Systems Manager implementing a high availability solution, you interface with many different aspects of your organization to gather the required information and possibly change the contracts to meet your new needs. Contractors or resources utilized from outside the company represents programmers, technicians, operators, or a variety of consultants. 2.5.1 Application providers The AS/400 system gets its name from Application System. The majority of AS/400 customers have one or many applications installed on their AS/400 system and the attached workstations. When reviewing the application, establish how the provider supports your business and determine the required level of support. This can range from a 9 a.m. to 5 p.m. telephone support provider contract, to employing developers Developing an availability plan 19 skilled with the particular application. The more critical the application, the higher level of skills that are required to perform problem determination that enables the fix to be resolved in the shortest possible time. The application provider should be able to define the rough release schedule for the application. This allows you to plan for application system updates. They should also be able to provide varying levels of fix support if the application fails in some way. Develop an escalation process for critical failures. Think through and document what steps to follow to recover from a critical failure. Note: This information applies to applications written outside the company. The same considerations apply for applications developed in-house. 2.5.2 Operating system provider Operating system suppliers are similar to application providers. However, the opportunity to enhance the operating system code typically occurs more frequently than application providers code. Enhancements to the OS/400 operating system typically occur with an annual frequency. Updates occur more frequently on application software. 2.5.3 Hardware providers Hardware provided by non-IBM distribution channels can include: • CPUs • Towers • Tapes • DASD • IOPs • Workstations • Printers Contracting with hardware providers is a relatively simple method to utilize resources outside the company. The contractor’s reliability is normally well known and high. Maintenance organizations are highly skilled and can provide high levels of service, depending on the cost. At minimum, the hours of coverage for support should match your planned availability requirements. Many large customers have arrangements for key supplies to be “warehoused” at the customer site. In other words, the supplies are owned by the maintainer but are stored at the customer site. Be careful when ordering new hardware. Order only with a goal that ensures availability of spare parts early in the product life. Demand for a product can be so great that there are no spare parts available. 2.5.4 Peripheral equipment The hardware components of a computing system go beyond the central processing unit and controllers. To reach end users, a computing system includes peripheral equipment, such as workstations, printers, routers, and modems. 20 High Availability on the AS/400 System: A System Manager’s Guide Maintaining peripheral equipment can be difficult. You must judge the benefits of maintaining parts and components on-site for fast replacement, compared to a repair contract for the main equipment or a combination of both. Consider the longevity of the peripherals. For example, when printers, displays, modems and such are replaced, it is not uncommon to update the technology. Sometimes, a replacement is made as an upgrade of functionality, either because the former model is withdrawn from sales and support, or the needs of the users require more function than the broken unit. It is not uncommon (perhaps even typical) for the replacement unit to provide equivalent or more function for less cost to the business. At first glance, replacing technology may not seem to be a big problem. For example, consider the case of a printer connected to a personal computer that is used by others within the network. Assume this departmental printer fails and is non-repairable. One solution is to simply replace the printer. However, after the new printer arrives and is connected, it needs a new driver loaded on the server operating system. The load of the new driver can raise a number of significant issues: • Is an IPL required to be recognized by the system? • Is a configuration required? • Do the applications work with the new driver? • What if the load causes the server to fail? This circumstance can result in driver loads across the whole network that reduce the hours of productive business. Therefore, even the smallest components of the total environment can have a major impact. It is advised that you plan for some redundancy, including when and how you carry out bulk replacements. 2.5.5 Facilities Facilities include machine rooms, power supplies, physical security, heating and air-conditioning, office space, ergonomics, and fire and smoke prevention systems to name but a few. The facilities that complete the total environment are as complex as the applications and computer hardware. 2.5.5.1 Site services It is important to work alongside the site facilities personnel and to understand contracts and service levels. For example, turning off the air conditioning for maintenance can potentially have a disastrous effect on the systems in the machine room. The contract with facilities should include spare air-conditioners or the placement of temporary mobile conditioners. 2.5.5.2 Machine rooms A good system manager has a well documented understanding of the machine rooms and the equipment within. The same redundancy requirements should be placed on critical services just as there are on computer systems. Developing an availability plan 21 2.5.5.3 UPS The use of Uninterruptible Power Supplies (UPS) has grown tremendously over the past ten years. The cost of a small UPS has reduced and they are now very affordable. When considering a UPS, keep in mind these three broad areas: • Machine Room: The machine room is a prime candidate for a UPS because it often requires continuous power. The ultimate solution is to generate your own power. Ideally, the national power supply serves only as a standby, with limited battery backup. Some customers do have this arrangement, but it is an expensive solution. There are simpler solutions that provide nearly the same level of service. Consider switching to a generated power environment. When the power fails, the standby generator starts. Unfortunately, there is a time lag before the system comes on line. Therefore, an interim battery backup is needed to support the systems while the generated power comes online. To register that the power has failed and then switch between battery, generator, and back to normal power requires a complicated and expensive switch. It may also require links to the systems to warn operators of the power failure. Another area that is often overlooked is the provision over power supplies for other equipment in the machine room, for example, consoles, printers, and air-conditioning. It is not enough to have a UPS for the system if there is no access to the console. In an extreme condition, you may be unable to shut the system down before the battery power fails if there is no UPS support for the console. • The site: When considering the site, you must look at all areas, for example: – If you have a disaster recovery service, is there space to park the recovery vehicle in the car park close to the building to attach network cabling? – In some buildings, access to the machine room is a problem and systems must be moved through windows via a crane because the lifts are too small or cannot take the weight. – Are there high availability facilities in the general office space, such as an emergency telephone with direct lines in case the PABX fails. This allows the business to continue even if the desk phones fail. When developing the availability plan, document these restrictions and plan for circumventions. • The workstation: Key users may need backup power to their workstation if there is no full standby generation. Looking at workstations from an ergonomic view, you may find potential issues that can result in long term unavailability of the human resources. An example is repetitive strain injury. This can severely impact your critical human resources in a company and cost a significant amount in litigation. It is worth investing time and money to solve these problems when planning for availability. 22 High Availability on the AS/400 System: A System Manager’s Guide 2.6 Verifying the implementation Verifying (by testing) the proposed high availability solution before putting it into production cannot be over-emphasized. Some of the activities involved in testing the implementation are explained in the following list (several areas of verifying, or testing, are involved): • Build a prototype: A prototype is a simulation of a live production. Develop a prototype to test the quality of the high availability solution. Make sure it can be easily reproduced. This lowers the risk of disturbing production during installation rollout. • Regression tests: Ensure that the replicated solution works in the same way as the prototype by performing a number of regression tests in the new environment. These same tests should be used in the production environment to ensure the quality of the final high availability implementation. • Disaster recovery test: Develop routines to ensure that a disaster recovery is smooth and as fast as possible. This requires a real crash test with detailed documentation of the steps required to get the system up and running, including how long it takes and where to find backup media and other required material. • Volume test: Few companies have the resources to build a test environment large enough to do realistic tests with production-like volumes. Some recovery centers and business partners offer the use of their environments for performing tests in larger volumes. It is an important step to help ensure that the system behaves as expected during the actual production. 2.6.1 Documenting the results To retest the systems after a problem has been fixed, it is important to have every test situation well documented. The documentation should include: • Hardware requirements • Software requirements • HAV requirements • A test case category • Tools required to perform the test • A list of steps to execute the test case • Results of the test (pass or fail) • The name of the individual executing the test case • The date on which the test case was executed • Notes taken while the case is executed • Anticipated results for each step of the test case • Actual results • Comments on general notes for each test case. • A list of any problem records opened in the event that the test was not successful 2.7 Rollout The testing sequence ends with the confirmed rollout of the solution. The rollout is another milestone in the overall project. For the first time, the production environment is actually affected. A well designed rollout strategy is crucial. Many things can impact the success of the rollout. Carefully check all prerequisites. Developing an availability plan 23 In a typical rollout, a great number of people are affected. Therefore, communication and training is important. For the rollout into production volumes, sufficient support is required. A minor incident can endanger the production rollout. The basic infrastructure of the company influences the rollout process. The situation differs depending on the industry area. The risk of problems increases with the number of external factors and the complexity of the system. The ideal case is a monogamous environment with all equipment owned by the company, including a high-performance network. All success factors can then be planned and controlled within individual departments. The main issue of the rollout is characterized by finding the appropriate rollout strategy. The project can begin in phases or all at once. A test scenario can validate the proper rollout. As the final proof of concept, a pilot rollout should be considered. Consider business hours and critical applications. Minimize the risk by taking controllable steps. The rollout can include down time for the production system. Therefore, timing must be negotiated with all concerned people. The rollout can take place inside or outside of business hours. Remember that some of the required prerequisites may not be available at these times. A project planning tool is helpful. Lists of resources, availability of resources, time restrictions, dependencies, and cost factors should be documented. Provide a calendar of activities, black-out dates, and milestones. 24 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 25 Chapter 3. High availability example solutions As companies increase their dependency on technology to deliver services and to process information, the risk of adverse consequences to earnings or capital from operational failures increases. If technology-related problems prevent a company from assessing its data, it may not be able to make payments on schedule, which would severely affect its business. Costly financial penalties may incur, the company’s reputation may suffer, and customers may not be able to deliver services or process information of their own. The company may even go out of business all together. Technology-related problems can increase: • Transaction risk: This arises from problems with service or product delivery • Strategic risk: This results from adverse business decisions or improper implementation of those decisions • Reputation risk: This has its source in negative public opinion • Compliance risk: This arises from violations of laws, rules, regulations, prescribed practices, or ethical standards Every customer has their own unique characteristics. The implementation of a high availability solution is customized to customer needs and budgets, while keeping in mind the risks of encountering and recovering from problems. This chapter describes examples of customer requirements and the high availability solutions they choose. 3.1 A high availability customer: Scenario 1 A Danish customer with 3,000 users on a large AS/400 system had difficulty finding time to complete tasks that required a dedicated system for maintenance. These tasks included such operations as performing nightly backups, installing new releases, and updating the hardware. One reason this challenge occurred is because the customer′s AS/400 system was serving all of their retail shops across Scandinavia. As these shops extended their hours, there was less and less time for planned system outages. They solved their problem by installing mirroring software on two AS/400 systems. To increase availability, the customer bought a second AS/400 system and connected the two machines with OptiConnect/400. Next, they installed mirroring software, and mirrored everything on the production machine to the backup machine. In the event of a planned or an unplanned system outage, the system users could switch to the backup machine in minutes. This solution made it easy for the shops to expand their hours and improve sales without losing system availability or sacrificing system maintenance. 26 High Availability on the AS/400 System: A System Manager’s Guide 3.2 A large financial institution: Scenario 2 Financial services typically require the most highly available solutions. Risks affect every aspect of banking, from interest rates the bank charges to the computers that process bank data. All banks want to avoid risk, but the risk of equipment failure and human error is possible in all systems. This risk may result from sources both within and beyond the bank’s control. Complete details of and a general outline of the SCAAA Bank’s new hot backup configuration are provided in the following list: • An original production AS/400e Model 740 in Madison • A Model 730 backup AS/400e at IBM’s Chicago Business Recovery Services Center • Both machines running the same level of OS/400 (V4R5) • High availability software applications transferring data and objects from the customer applications on the production machine to those on the backup machine • TCP/IP communications protocol used for communications with the BRS Center • A T1 line from the Madison branch to IBM’s POP server in Chicago, where IBM is responsible for communications from the POP server to the BRS center SAAA Bank of Madison, Wis., is no exception. SAAA specializes in commercial and institutional banking, and it provides comprehensive financial services to other banks, governments, corporations, and organizations. It has received AAA, Aaa and AAA ratings from Standard and Poor’s, Moody’s, and IBCA (Europe’s leading international credit rating agency), respectively. It has offices around the world, with over 5,200 employees woking in commercial centers worldwide. The Madison branch provides services to U.S. companies and subsidiaries of German corporations in North America. The Information Systems department of the Madison branch is intimately involved with the actual business of the bank, not just its computers. On a typical day, the workload consists of foreign exchange transactions, money market transactions, securities transactions, options transactions, and loans. The IS department sees every transaction that goes through and does the utmost to ensure that each one executes accurately and promptly. Given the profitability of the Madison branch, the IS department plays a major role in the success of the bank as a whole. The core data is largely stored on an IBM AS/400e system. Protecting the bank’s data means eliminating all single points of failure on that platform. Prior to 1997, the bank took the following steps to ensure business continuity: • Running regular backups, even making midday backups • Adding a redundant token-ring card to prevent system failure • Eliminating cabling problems by using an intelligent hub • Providing dual air conditioning systems to address environmental variables High availability example solutions 27 • Installing universal power supplies (UPSs) to permit system operations during electrical failure • Providing RAID5 to maintain parity information across multiple disks • Installing a second RAID controller to manage the array when the original controller fails • Leasing twenty seats at the IBM Business Recovery Services (BRS) Center in Sterling Forest, NY, so that the essential work of the bank can be carried out in the event of a disaster After recognizing the bank’s exposure to possible data loss, the bank chose MIMIX Availability Management Software. Running high availability software over a WAN to the BRS Center protects from all risks of data loss. The high availability software solution at SAAA Bank was modified to meet the demands of the bank’s daily schedule. This adjustment was necessary because the applications required journaling to be turned off at the end of the day to facilitate close-of-business processing. This is a characteristic that makes them somewhat difficult to replicate. The bank accomplished this through an elaborate process. During the day, the bank turned on AS/400e journaling capabilities, but at the end of the business day, the bank shut down MIMIX and the journal files for both systems. At about 6:30 p.m., the bank ran close-of-business processing on both machines simultaneously. This way their databases remained synchronized. About two hours later, when the processing was complete, the bank restarted journaling on both machines and MIMIX was brought back up. The next morning, high availability software started replicating the day’s transactions. The entire process was automated, but an operator was always available in case of an emergency. The flexibility of MIMIX enabled the bank to keep the databases in synch at all times. In 1998, a potentially serious incident occurred at the bank. An operator working on the test system meant to restore some data to the test libraries. Unfortunately, they were copied and loaded into the production system. This overwrote all payments that had to be sent out later that afternoon, leaving the bank in a critical position. After detecting the error, the operator immediately switched over to the mirrored system on the backup machine located at the BRS center. That system held a mirrored, real time copy of all the bank’s transactions for the day. By clicking a button, the operator initiated a full restore from the backup system to the production system, ensuring synchronization of the two databases. This process required only about 25 minutes. Without high availability application software (MIMIX) the bank would have had to restore from a mid-day backup (which requires about 10 hours worth of work). In addition, about 20 bank staff members would have had to return to the bank and re-enter all the lost transactions at overtime rates. The estimated costs of these activities was around $28K. Even greater costs would have resulted from the one day interest penalties for late or nonexistent outgoing payments. The fastest and most comprehensive way to ensure maximum uptime and high availability is through added redundancy and fault-tolerant models. With the 28 High Availability on the AS/400 System: A System Manager’s Guide implementation of high availability software (MIMIX) at the BRS Center, the bank achieved the highest level of disaster prevention and continuous operations in over 20 years. The strategy provided the availability management, flexibility, and reliability needed to maintain business continuity in the event of a disaster at the location. 3.2.1 Benefits Looking back, SAAA Bank found that using high availability management software at the BRS center provided these benefits: • Maintained uninterrupted business processing, despite operator errors and environmental disasters, such as floods, fires, powers outages, and terrorist acts • Dramatically reduced the cost of disasters, especially disasters that arise at the user level • Sustained the bank’s reputation for reliability: Because the bank acts as a clearing house for several other banks, SAAA Bank had to ensure that the system was operational and that payments were sent out in a timely manner • Simplified the recovery process: High availability software freed the IS department from lengthy restore projects and eliminated the need to spend time and money re-entering lost data • Enabled the IS department to switch to the backup database at the BRS center when required 3.3 A large retail company: Scenario 3 Retailing is a new area for very high availability. As retail companies enter the area of e-commerce, the business demands are considerable. Companies operating in a global marketplace must have their systems online 24 hours a day, every day, all year long. This section describes an example of a large retail company, referred to as EDA Retail for the purposes of the discussion. EDA Retail is a Danish lumber wholesale business. They also own and run a chain of hardware retail stores across Denmark, Norway, and Sweden, supplying everything for the do-it-yourself person. These hardware stores run a Point-of-Sale (POS) solution from IBM, which connects to a large AS/400 system. At the time this information was gathered, the system was a 12-way model 740 with 1 TB of DASD. All inventory and customer data is stored on this one AS/400 system. The connection to the AS/400 system is the lifeline to all of the stores. If the connection to the AS/400 system is down, or if the AS/400 system is not up and running at all times, EDA Retail employees cannot take returns, check inventory, check customer data, or perform any related functions while selling goods to their customers. The unavailability of information developed into a unacceptable problem for EDA Retail. They had grown from a big to a very large enterprise during the four years from when they installed the AS/400 system. Management determined that a breakdown of the AS/400 system would cost the enterprise about 40,000 U.S. High availability example solutions 29 dollars per hour. In addition, planned shutdowns, such as that necessary for the daily backup, system maintenance, upgrades, and application maintenance were becoming a problem. The problem multiplied when a chain of stores in Sweden was taken over and expected to run on the EDA Retail’s system. These acquired stores had longer opening hours than the corresponding stores in Denmark, which increased the necessity for long opening hours of the AS/400 system at EDA Retail. The IBM team did a thorough analysis of EDA Retail’s installation in order to develop an environment with no single point of failure. At this time, the IBM team responsible for the customer proposed an extra system for backup, using dedicated communication lines between the two systems. The proposed solution involved a backup AS/400 system in a new, separate machine room of its own, and an OptiConnect line between the two AS/400 systems. A key feature in this environment was a High Availability solution from Vision Solutions. This solution mirrors all data on the system to a target (backup) system on a real time basis. In addition to the necessary hardware, this allows EDA Retail to become immune to disasters, planned shutdowns (for example, system maintenance), and unplanned shutdowns (such as system failures). In the case of a shutdown, either planned or unplanned, EDA Retail can switch users to the backup system. Within thirty minutes of the breakdown, operations continue. The primary test of the project was a Role Swap, in which roles are switched between the two systems. The system normally serving as the source system becomes the target system, and vice versa. When the roles are switched and mirroring is started in reverse mode, the user’s connection to the new source system is tested to determine if they can run their applications as they normally would. A success is illustrated when the correct data, authorizations, and other objects or information are successfully mirrored, with the transactions also mirrored to the new target system. The successful completion of a second test of this nature marked the end of the implementation project and the beginning of normal maintenance activities. The users had access to the information they required even while the backup was taken. Their information remained available at all times except during the role swap. 3.4 A small manufacturing company: Scenario 4 This solution applies to manufacturing and to any small business with limited information technology resources. These firms do not typically operate seven days a week, but they may operate 24 hours per day, Monday through Friday. The business requires that the systems be available when they are needed with a minimum of intervention. If the system fails, they need a backup system on which to run. They can then contact their external I/T services supplier for support. This support can take a day or so to arrive. In the meantime, they run their business with unprotected information. 30 High Availability on the AS/400 System: A System Manager’s Guide It is important to note that they are still running in this situation. The third party services provider then arrives, fixes the problem, and re-synchronizes the systems. DMT Industries is a manufacturer of medical supplies which has become highly globalized over the last several years. Their globalization raises the demand on the IT division to provide a 24 x 7 operation. DMT Industries has various platforms installed. One of the strategic business applications, which must be available at all times, is running on the AS/400 platform. A total of 1,500 users from all over the world have access to the system, which is located in Denmark. Until early 1998, DMT Industries could offer only 20 hours of system availability per day. The remaining four hours were used for the nightly backup. As it became necessary to offer 24 hour availability, they decided to implement a high availability solution. The installation then consisted of two identical AS/400 systems, located in separate machine rooms, a dedicated 100 Mbit Ethernet connection between them, and software from Vision Solutions. This enabled DMT Medical to perform their nightly backup on the backup system to which all production data was mirrored. They simply stopped the mirrored data from the production machine from being applied on the backup system while the backup ran. The users could continue operations on the production system as required. 3.5 A distribution company: Scenario 5 This example represents a three-tiered SAP solution (reportedly the largest site worldwide). TVBCo is a distribution company utilizing J.D. Edwards (JDE) applications. The company focuses on expanding knowledge about human protein molecules and transforming them by means of biotechnological methods and clinical tests. TVBCo had developed into a completely integrated worldwide pharmaceutical business. Its European Distribution Center in Holland houses a central IBM AS/400e production system, accessed by a large number of TVBCos branches in Europe. A second AS/400e machine is used for testing, development, and backup. The production AS/400e system plays a critical role in TVBCos European operations, so the company wanted to reduce its downtime to a minimum. In cooperation with IBM, TVBCo started developing an availability management plan and purchased two AS/400e systems. MIMIX availability management software solution was installed. High availability software immediately replicated all production system transactions on a defined backup system and performed synchronization checking to verify data integrity. Critical data and other items were always present and available on the backup system so that users could switch to that system and continue working if an unplanned outage occured. Aside from the immediate safekeeping of all data, high availability software maintained an accurate, up-to-date copy of the system setup on the back-up machine. For that purpose, all user profiles, subsystem descriptions, and other High availability example solutions 31 objects were replicated. High availability software also controlled the production system and the actual switchover in case of a failure. The backup machine monitored whether the production machine was still on. The moment it detected that something was wrong, it warned the system operator, who could decide to switch over. When that decision was made, all necessary actions had already been set out in a script so that the operator only had to intervene in exceptional cases. The TVBCo customized procedure included sending warnings to all users, releasing the backup database, activating subsystems and backup users, and switching interactive users to the backup system. TVBCo soon decided to start using high availability software as its worldwide standard for processing and invoicing. The data changed continuously, and a traditional backup could not be produced because too few stable points existed in the day. With the chosen high availability solution, the backup was made while the file was active. Therefore, taking the application offline was not necessary. In addition, the high availability solution backup ran, on average, only a couple of seconds (sometimes even less) behind production. Contrast this if a tape were made every hour (the backup could have run up to one hour behind in real time). The second application, an electronic data interchange (EDI) package, automates the purchase of supplies. This package retrieves batch data from the computer networks managed by the company’s suppliers. This process is not continuous so a more traditional backup method may be suitable. However, after estimating the considerable effort required to adapt the batch protocols and DDM files, it was decided to use high availability software. Because their chosen high availability software solution kept the backup data current (almost to the second), the company did not need to request that the batches be sent again when a problem occured with the production machine. Also, high availability software allowed the backup to be made in batches (for example, to relieve the network during peak hours). The third critical application replicated by high availability software is a DSI logistics program. Since this application not only generates packing lists and is used for updating the contents of the warehouse in the J.D. Edwards database, it must have information about the most recent setup of the warehouse. This information must, of course, remain available when the production equipment breaks down. Otherwise, the orders may be processed but the forklift operators would not know where to unload the merchandise. Like the J.D. Edwards application, the input and output of this application (and, therefore, the replicating) is a real time process. The data from the J.D. Edwards application and the IBM application are copied to the backup while the file is active. The batch processes from a variety of platforms, are copied to the AS/400e production system at scheduled intervals. With high availability software, it is not necessary to take the application offline because the backup is made while the file is active. High availability software also controls the production system and actual switchover in case of a failure. TVBCos data center is supported by a staff of 70. The hardware consists of two AS/400e Model 500 systems connected with TCP on ethernet adapters. 32 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 33 Part 2. AS/400 high availability functions Many areas of a system implementation contribute to the availability rating. Each option provides differing degrees of availability. Part II discusses the hardware, storage options, operating system features, and network components that contribute to a high availability solution. Appendix F, “Cost components of a business case” on page 169, provides a reference to compare the availability options of journaling, mirrored protection, device parity protection, and others. 34 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 35 Chapter 4. Hardware support for single system high availability The selection of hardware components provides a basis for system availability options. This chapter discusses some of the considerations for protecting your data and available features and tools from a hardware perspective on a single system and options using multiple systems. Hardware selections greatly contribute to a system’s high availability characteristics. This chapter discusses: • Data protection options • Concurrent maintenance • Hot spares • OptiConnect • Clusters • LPAR • Power options • Tape devices 4.1 Protecting your data Your first defense against data loss is a good backup and recovery strategy. You need a plan for regularly saving the information on your system. In addition to having a working backup and recovery strategy, you should also employ some form of data protection on your system. When you think about protecting your system from data loss, make the following considerations: • Recovery: Can you retrieve the information that you lost, either by restoring it from backup media or by creating it again? • Availability: Can you reduce or eliminate the amount of time that your system is unavailable after a problem occurs? • Serviceability: Can you service it without affecting the data user? Disk protection can help prevent data loss and keep your system from stopping if you experience a disk failure. The topics that follow provide information on the different types of disk protection, as well as using the types with one another. 4.2 Disk protection tools Several disk availability tools are available for reducing or eliminating system downtime. They also help with data recovery after a disk failure. The tools include: Although disk protection can reduce downtime or make recovery faster, it is not a replacement for regular backups. Disk protection cannot help you recover from a complete system loss, a processor failure, or a program failure. Remember 36 High Availability on the AS/400 System: A System Manager’s Guide • Mirrored protection • Device parity protection • Auxiliary storage pools (ASPs) • Others These disk protection methods help protect your data. You can use these methods in different combinations with one another. Your choice of disk tools determines your level of disk protection and vice versa. Figure 6 shows the different levels of availability. Figure 6. Levels of availability High availability, as shown in Figure 6, includes mirroring, RAID-5, offline backup, UPS, CPM, clustering, journaling, auxiliary storage pools, SMAPP, commitment control, and save-while-active components. Topics not covered in this Redpaper are discussed in The System Administrator’s Companion to AS400 Availability and Recovery, SG24-2161, and the Backup and Recovery, SC41-5304. 4.3 Disk mirroring Mirrored protection is an OS/400 high availability function in the licensed internal code that protects data from loss due to failure, or due to damage to a disk-related component. Mirrored protection is a high availability software function that duplicates disk-related hardware components to keep the AS/400 system available if one of the disk components fails. It prevents a loss of data in case of a disk-related hardware failure. Mirroring is used on any model of the AS/400 system and is a part of the Licensed Internal Code (LIC). Bus 2 IOP Mirroring Bus 3 IOP Controller Controller * Journaling * Auxiliary storgae pools * SMAAP * Commitment control Clustered systems Continuous power Backup power supply UPS Tape Array 1 Array 2 Disk Bus controller RAID - 5 Hardware support for single system high availability 37 Different levels of mirrored protection are possible, depending on what hardware is duplicated. The hardware components that can be duplicated include: • Disk units (to provide the lowest (relative) level of availability) • Disk controllers • Disk I/O processors (IOP) • Buses (to provide the highest (relative) level of availability) Mirroring protection is configured by the ASP. For optimum protection, there must be an even number of components at each level of protection. The system remains available during a disk, controller, IOP, or bus failure if the failing component and hardware components that are attached to it are duplicated. When you start mirrored protection or add disk units to an ASP that has mirrored protection, the system creates mirrored pairs using disk units that have identical capacities. Data is protected because the system keeps two copies of data on two separate disk units. When a disk-related component fails, the system can continue to operate without interruption by using the mirrored copy of the data until the failed component is repaired. The overall goal is to protect as many disk-related components as possible. To provide maximum hardware redundancy and protection, the system attempts to pair disk units that are attached to different controllers, input/output processors, and buses. If you have a multi-bus system or a large single-bus system, consider using mirrored protection. The greater the number of disk units attached to a system, the more frequently disk-related hardware failures occur. This is because there are more individual pieces of hardware that can fail. Therefore, the possibility of data loss or loss of availability as a result of a disk or other hardware failure is more likely. Also, as the amount of disk storage on a system increases, the recovery time after a disk storage subsystem hardware failure increases significantly. Downtime becomes more frequent, more lengthy, and more costly. Mirroring can be used on any AS/400 system model. The system remains available during a failure if a failing component and the hardware components that are attached to it have been duplicated. See 4.8, “Concurrent maintenance” on page 54, to better understand the maintenance aspect. Remote mirroring support allows you to have one mirrored unit within a mirrored pair at the local site, and the second mirrored unit at a remote site. For some systems, standard DASD mirroring remains the best option. Evaluate the uses and needs of your system, consider the advantages and disadvantages of each type of mirroring support, and decide which is best for you. Refer to B.2.3, “Mirrored protection: How it works” on page 151, for a more detailed description. 4.3.1 Standard mirrored protection Standard DASD mirroring support requires that both disk units of the load source mirrored pair (unit 1) are attached to the multi-function I/O processor (MFIOP). This option allows the system to initial program load (IPL) from either load source 38 High Availability on the AS/400 System: A System Manager’s Guide in the mirrored pair. The system can dump main storage to either load source if the system terminates abnormally. However, since both load source units must be attached to the same I/O processor (IOP), controller level protection is the best mirroring protection possible for the load source mirrored pair. How the AS/400 system addresses storage Disk units are assigned to an auxiliary storage pool (ASP) on a unit basis. The system treats each storage unit within a disk unit as a separate unit of auxiliary storage. When a new disk unit is attached to the system, the system initially treats each storage unit within as non-configured. Through Dedicated Service Tools (DST) options, you can add these nonconfigured storage units to either the system ASP or a user ASP of your choice. When adding non-configured storage units, use the serial number information that is assigned by the manufacturer to ensure that you are selecting the correct physical storage unit. Additionally, the individual storage units within the disk unit can be identified through the address information that can be obtained from the DST Display Disk Configuration display. When you add a nonconfigured storage unit to an ASP, the system assigns a unit number to the storage unit. The unit number can be used instead of the serial number and address. The same unit number is used for a specific storage unit even if you connect the disk unit to the system in a different way. When a unit has mirrored protection, the two storage units of the mirrored pair are assigned the same unit number. The serial number and the address distinguish between the two storage units in a mirrored pair. To determine which physical disk unit is being identified with each unit number, note the unit number assignment to ensure correct identification. If a printer is available, print the DST or SST display of your disk configuration. If you need to verify the unit number assignment, use the DST or SST Display Configuration Status display to show the serial numbers and addresses of each unit. The storage unit that is addressed by the system as Unit 1 is always used by the system to store licensed internal code and data areas. The amount of storage that is used on Unit 1 is quite large and varies depending on your system configuration. Unit 1 contains a limited amount of user data. Because Unit 1 contains the initial programs and data that is used during an IPL of the system, it is also known as the load source unit. The system reserves a fixed amount of storage on units other than Unit 1. The size of this reserved area is 1.08 MB per unit. This reduces the space available on each unit by that amount. 4.3.2 Mirrored protection: Benefits With the best possible mirrored protection configuration, the system continues to run after a single disk-related hardware failure. On some system units, the failed hardware can sometimes be repaired or replaced without having to power down the system. If the failing component is one that cannot be repaired while the system is running, such as a bus or an I/O processor, the system usually continues to run after the failure. Maintenance can be deferred and the system can be shut down normally. This helps to avoid a longer recovery time. Even if your system is not a large one, mirrored protection can provide valuable protection. A disk or disk-related hardware failure on an unprotected system leaves your system unusable for several hours. The actual time depends on the Hardware support for single system high availability 39 kind of failure, the amount of disk storage, your backup strategy, the speed of your tape unit, and the type and amount of processing the system performs. 4.3.3 Mirrored protection: Costs and limitations The main cost of using mirrored protection lies in additional hardware. To achieve high availability, and prevent data loss when a disk unit fails, you need mirrored protection for all the auxiliary storage pools (ASPs). This usually requires twice as many disk units. If you want continuous operation and data loss prevention when a disk unit, controller, or I/O processor fails, you need duplicate disk controllers as well as I/O processors. A model upgrade can be done to achieve nearly continuous operation and to prevent data loss when any of these failures occur. This includes a bus failure. If Bus 1 fails, the system cannot continue operation. Because bus failures are rare, and bus-level protection is not significantly greater than I/O processor-level protection, you may not find a model upgrade to be cost-effective for your protection needs. Mirrored protection has a minimal reduction in system performance. If the buses, I/O processors, and controllers are no more heavily loaded on a system with mirrored protection than they would be on an equivalent system without mirrored protection, the performance of the two systems should be approximately the same. In deciding whether to use mirrored protection on your system, evaluate and compare the cost of potential downtime against the cost of additional hardware over the life of the system. The additional cost in performance or system complexity is usually negligible. Also consider other availability and recovery alternatives, such as device parity protection. Mirrored protection normally requires twice as many storage units. For concurrent maintenance and higher availability on systems with mirrored protection, other disk-related hardware may be required. Limitations Although mirrored protection can keep the system available after disk-related hardware failures occur, it is not a replacement for save procedures. There can be multiple types of disk-related hardware failures, or disasters (such as flood or sabotage) where recovery requires backup media. Mirrored protection cannot keep your system available if the remaining storage unit in the mirrored pair fails before the first failing storage unit is repaired and mirrored protection is resumed. If two failed storage units are in different mirrored pairs, the system is still available. Normal mirrored protection recovery is done because the mirrored pairs are not dependent on each other for recovery. If a second storage unit of the same mirrored pair fails, the failure may not result in a data loss. If the failure is limited to the disk electronics, or if the service representative can successfully use the Save Disk Unit Data function to recover all of the data (a function referred to as “pump”), no data is lost. 40 High Availability on the AS/400 System: A System Manager’s Guide If both storage units in a mirrored pair fail causing data loss, the entire ASP is lost and all units in the ASP are cleared. Be prepared to restore your ASP from the backup media and apply any journal changes to bring the data up to date. 4.3.4 Determining the level of mirrored protection The level of mirrored protection determines whether the system continues running when different levels of hardware fails. The level of protection is the amount of duplicate disk-related hardware that you have. The more mirrored pairs that have higher levels of protection, the more often your system is usable when disk-related hardware fails. You may decide that a lower level of protection is more cost-effective for your system than a higher level. The four levels of protection, from lowest to highest, are as follows: • Disk unit-level protection • Controller-level protection • Input/output processor-level protection • Bus-level protection When determining what level of protection is adequate, consider the relative advantages of each level of protection with respect to the following considerations: • The ability to keep the system operational during a disk-related hardware failure. • The ability to perform maintenance concurrently with system operations. To minimize the time that a mirrored pair is unprotected after a failure, you may want to repair failed hardware while the system is operating. During the start mirrored protection operation, the system pairs the disk units to provide the maximum level of protection for the system. When disk units are added to a mirrored ASP, the system pairs only those disk units that are added without rearranging the existing pairs. The hardware configuration includes the hardware and how the hardware is connected. The level of mirrored protection determines whether the system continues running when different levels of hardware fail. Mirrored protection always provides disk unit-level protection that keeps the system available for a single disk unit failure. To keep the system available for failures of other disk-related hardware requires higher levels of protection. For example, to keep the system available when an I/O processor (IOP) fails, all of the disk units attached to the failing IOP must have mirrored units attached to different IOPs. The level of mirrored protection also determines whether concurrent maintenance can be done for different types of failures. Certain types of failures require concurrent maintenance to diagnose hardware levels above the failing hardware component. An example would be diagnosing a power failure in a disk unit that requires resetting the I/O processor to which the failed disk unit is attached. In this case, IOP-level protection is required. The level of protection you get depends on the hardware you duplicate. If you duplicate disk units, you require disk unit-level protection. If you also duplicate disk unit controllers, you require controller-level protection. If you duplicate Hardware support for single system high availability 41 input/output processors, you require IOP-level protection. If you duplicate buses, you require bus-level protection. Mirrored units always have, at least, disk unit-level protection. Because most internal disk units have the controller packaged along with the disk unit, they have at least controller-level protection. 4.3.4.1 Disk unit-level protection All mirrored storage units have a minimum of disk unit-level protection if they meet the requirements for starting mirrored protection (storage units are duplicated). If your main concern is protecting data and not high availability, disk unit-level protection may be adequate. The disk unit is the most likely hardware component to fail, and disk unit-level protection keeps your system available after a disk unit failure. Concurrent maintenance is often possible for certain types of disk unit failures with disk unit-level protection. Figure 7 shows disk unit-level protection. The two storage units make a mirrored pair. With disk unit-level protection, the system continues to operate during a disk unit failure. If the controller or I/O processor fails, the system cannot access data on either of the storage units of the mirrored pair, and the system is unusable. Figure 7. Disk unit level protection 4.3.4.2 Controller-level protection If the planned disk units do not require a separate controller, you already have controller-level protection for as many units as possible and do not need to do anything else. If your planned disk units do require a separate controller, add as many controllers as possible while keeping within the defined system limits. Then balance the disk units among them according to the standard system configuration rules. Bus Input/Output Programmer Disk Unit Disk Unit Controller 42 High Availability on the AS/400 System: A System Manager’s Guide To keep your system available when a controller fails, consider using concurrent maintenance. The controller must be dedicated to the repair action in this process. If any disk units attached to the controller do not have controller-level protection, concurrent maintenance is not possible. To achieve controller-level protection, all disk units must have a mirrored unit attached to a different controller. Most internal disk units have their controller packaged as part of the disk unit, so internal disk units generally have at least controller-level protection. Use problem recovery procedures in preparation for isolating a failing item or to verify a repair action. Figure 8 illustrates controller-level protection. Figure 8. Input/output controller-level protection The two storage units make a mirrored pair. With controller-level protection, the system can continue to operate after a disk controller failure. If the I/O processor fails, the system cannot access data on either of the disk units, and the system is unusable. 4.3.4.3 IOP-level protection If you want IOP-level protection and you do not already have the maximum number of IOPs on your system, add as many IOPs as possible while keeping within the defined system limits. Then, balance the disk units among them according to the standard system configuration rules. You may need to add additional buses to attach more IOPs. To achieve I/O processor-level protection, all disk units that are attached to an I/O processor must have a mirrored unit attached to a different I/O processor. On many systems, I/O processor-level protection is not possible for the mirrored pair for Unit 1. Bus Input/Output Programmer Disk Unit Disk Unit Controller Controller Hardware support for single system high availability 43 IOP-level protection is useful to: • Keep your system available when an I/O processor fails • Keep your system available when the cable attached to the I/O processor fails • Concurrently repair certain types of disk unit failures or cable failures. For these failures, concurrent maintenance needs to reset the IOP. If any disk units that are attached to the IOP do not have IOP-level protection, concurrent maintenance is not possible. Figure 9 illustrates IOP-level protection. Figure 9. IOP-level protection The two storage units make a mirrored pair. With IOP-level protection, the system continues to operate after an I/O processor failure. The system becomes unusable only if the bus fails. 4.3.4.4 Bus-level protection If you want bus-level protection, and you already have a multiple-bus system, nothing must be done. If your system is configured according to standard configuration rules, the mirrored pairing function pairs up storage units to provide bus-level protection for as many mirrored pairs as possible. If you have a single-bus system, you can add additional buses as a feature option on systems supporting multiple buses. Bus-level protection can allow the system to run when a bus fails. However, bus-level protection is often not cost-effective because of the following problems: • If Bus 1 fails, the system is not usable. • If a bus fails, disk I/O operations may continue, but so much other hardware is lost (such as work stations, printers, and communication lines) that, from a practical standpoint, the system is not usable. Bus Input/Output Programmer Disk Unit Controller Disk Unit Controller Input/Output Programmer 44 High Availability on the AS/400 System: A System Manager’s Guide • Bus failures are rare compared with other disk-related hardware failures. • Concurrent maintenance is not possible for bus failures. To achieve bus-level protection, all disk units that are attached to a bus must have a mirrored unit attached to a different bus. Bus-level protection is not possible for Unit 1. Figure 10 illustrates bus-level protection. Figure 10. Bus-level protection The two storage units make a mirrored pair. With bus-level protection, the system continues to operate after a bus failure. However, the system cannot continue to operate if Bus 1 fails. 4.3.5 Determining the hardware required for mirroring To communicate with the rest of the system, disk units are attached to controllers, which are attached to I/O processors, which are attached to buses. The number of each of these types of disk-related hardware that are available on the system directly affects the level of possible protection. Bus 1 Input/Output Programmer Disk Unit Controller Bus 2 Input/Output Programmer Disk Unit Controller Hardware support for single system high availability 45 To provide the best protection and performance, each level of hardware should be balanced under the next level of hardware. That is, the disk units of each device type and model should be evenly distributed under the associated controllers. The same number of controllers should be under each I/O processor for that disk type. Balance the I/O processors among the available buses. To plan what disk-related hardware is needed for your mirrored system, plan the total number and type of disk units (old and new) that are needed on the system, as well as the level of protection for the system. It is not always possible to plan for and configure a system so that all mirrored pairs meet the planned level of protection. However, it is possible to plan a configuration in which a very large percentage of the disk units on the system achieve the desired level of protection. When planning for additional disk-related hardware, perform the following steps: 1. Determine the minimum hardware that is needed for the planned disk units to function. Plan for one disk unit size (capacity) at a time. 2. Plan the additional hardware needed to provide the desired level of protection for each disk unit type Planning the minimum hardware needed to function Various rules and limits exist on how storage hardware can be attached. The limits may be determined by hardware design, architecture restrictions, performance considerations, or support concerns. For each disk unit type, first plan for the controllers that are needed and then for the I/O processors that are needed. After planning the number of I/O processors needed for all disk unit types, use the total number of I/O processors to plan for the number of buses that are needed. 4.3.6 Mirroring and performance When mirrored protection is started, most systems show little difference in performance. In some cases, mirrored protection can improve performance. Generally, functions that mostly perform read operations experience equal or better performance with mirrored protection. This is because read operations have a choice of two storage units to read from. The unit with the faster expected response time is selected. Operations that mostly perform write operations (such as updating database records) may see slightly reduced performance on a system that has mirrored protection because all changes must be written to both storage units of the mirrored pair. Therefore, restore operations are slower. In some cases, if the system ends abnormally, the system cannot determine whether the last updates were written to both storage units of each mirrored pair. If the system is not sure that the last changes were written to both storage units of the mirrored pair, the system synchronizes the mirrored pair by copying the data in question from one storage unit of each mirrored pair to the other storage unit. The synchronization occurs during the IPL that follows the abnormal system end. If the system can save a copy of main storage before it ends, the synchronization process takes just a few minutes. If not, the synchronization process can take much longer. The extreme case could be close to a complete synchronization. 46 High Availability on the AS/400 System: A System Manager’s Guide If you have frequent power outages, consider adding an uninterruptible power supply (UPS) to your system. If main power is lost, the UPS allows the system to continue. A basic UPS supply provides the system with enough time to save a copy of main storage before ending. This prevents a long recovery. Both storage units of the load source mirrored pair must be powered by the basic UPS. 4.3.7 Determining the extra hardware required for performance Mirrored protection normally requires additional disk units and input/output processors. However, in some cases, you may need additional hardware to achieve the level of desired performance. Use these points to decide how much extra hardware you may need: • Mirrored protection causes a minor increase in central processing unit usage (approximately 1% to 2%). • Mirrored protection requires storage in the machine pool for general purposes and for each mirrored pair. If you have mirrored protection, increase the size of your machine pool by approximately 12 KB for each 1 GB of mirrored disk storage (12 KB for 1 GB DASD, 24 KB for 2 GB DASD, etc.). • During synchronization, mirrored protection uses an additional 512 KB of memory for each mirrored pair being synchronized. The system uses the pool with the most storage. • To maintain equivalent performance after starting mirrored protection, your system should have the same ratio of disk units to I/O processors as it did before. To add I/O processors, you may need to upgrade your system for additional buses. Note: Because of the limit on buses and I/O processors, you may not be able to maintain the same ratio of disk units to I/O processors. In this case, system performance may degrade. 4.4 Remote DASD mirroring support Standard DASD mirroring support requires that both disk units of the load source mirrored pair (Unit 1) are attached to the Multi-function I/O Processor (MFIOP). This allows the system to IPL from either load source in the mirrored pair and allows the system to dump main storage to either load source if the system ends abnormally. However, since both load sources must be attached to the same I/O Processor (IOP), the best mirroring protection possible for the load source mirrored pair is controller-level protection. To provide a higher level of protection for your system, use remote load source mirroring and remote DASD mirroring. Remote DASD mirroring support, when combined with remote load source mirroring, mirrors the DASD on local optical buses with the DASD on optical buses that terminate at a remote location. In this configuration, the entire system, including the load source, can be protected from a site disaster. If the remote site is lost, the system can continue to run on the DASD at the local site. If the local DASD and system unit are lost, a new system unit can be attached to the set of DASD at the remote site, and system processing can be resumed. Remote DASD mirroring, like standard DASD mirroring, supports mixing device-parity-protected disk units in the same ASP with mirrored disk units. The device parity DASD can be located at either the local or the remote site. However, Hardware support for single system high availability 47 if a site disaster occurs at the site containing the device parity DASD, all data in the ASPs containing the device parity DASD is lost. Remote mirroring support makes it possible to divide the disk units on your system into a group of local DASD and a group of remote DASD. The remote DASD is attached to one set of optical buses and the local DASD to another set of buses. The local and remote DASD can be physically separated from one another at different sites by extending the appropriate optical buses to the remote site. 4.4.1 Remote load source mirroring Remote load source mirroring support allows the two disk units of the load source to be on different IOPs or system buses. This provides IOP- or bus-level mirrored protection for the load source. However, in such a configuration, the system can only IPL from, or perform a main storage dump to, the load source attached to the MFIOP. If the load source on the MFIOP fails, the system can continue to run on the other disk unit of the load source mirrored pair, but the system is not able to IPL or perform a main storage dump until the load source attached to the MFIOP is repaired and usable. 4.4.2 Enabling remote load source mirroring To use remote load source mirroring support, remote load source mirroring must first be enabled. Mirrored protection must then be started for ASP 1. If remote load source mirroring support is enabled after mirrored protection has already been started for ASP 1, the existing mirrored protection and mirrored pairing of the load source must not change. Remote load source mirroring support can be enabled in either the DST or the SST environment. If you attempt to enable remote load source mirroring, and it is currently enabled, the system displays a message that remote load source mirroring is already enabled. There are no other errors or warnings for enabling remote load source mirroring support. If the remote load source is moved to the MFIOP, the IOP and system may not recognize it because of the different DASD format sizes used by different IOPs. If the remote load source is missing after it has been moved to the MFIOP, use the DST Replace disk unit function to replace the missing load source with itself. This causes the DASD to be reformatted so that the MFIOP can use it. The disk unit is then synchronized with the active load source. Remote load source mirroring may be disabled from either DST or SST. However, disabling remote load source mirroring is not allowed if there is a load source disk unit on the system that is not attached to the MFIOP. If you attempt to disable remote load source mirroring support and it is currently disabled, the system displays a message that remote load source mirroring is already disabled. 4.4.3 Using remote load source mirroring with local DASD Remote load source mirroring can be used to achieve IOP-level or bus-level protection of the load source mirrored pair, even without remote DASD or buses on the system. There is no special setup required, except for ensuring that a disk unit of the same capacity as the load source is attached to another IOP or bus on the system. To achieve bus-level protection of all mirrored pairs in an ASP, configure your system so that no more than one-half of the DASD of any given capacity in that ASP are attached to any single bus. To achieve IOP-level 48 High Availability on the AS/400 System: A System Manager’s Guide protection of all mirrored pairs in an ASP, have no more than one-half of the DASD of any given capacity in the ASP attached to any single IOP. There is no special start mirroring function for remote load source support. The system detects that remote load source mirroring is enabled and automatically pairs up disk units to provide the best level of possible protection. It is not possible to override or influence the pairing of the disk units other than by changing the way the hardware of the system is connected and configured. Normal mirroring restrictions that concern total ASP capacity, an even number of disk units of each capacity, and other such considerations, apply. 4.4.3.1 Remote DASD mirroring: Advantages Advantages of remote DASD mirroring include: • Providing IOP-level or bus-level mirrored protection for the load source • Allowing the DASD to be divided between two sites, mirroring one site to another, to protect against a site disaster 4.4.3.2 Remote DASD mirroring: Disadvantages Disadvantages of remote DASD mirroring include: • A system that uses Remote DASD Mirroring is only able to IPL from one DASD of the load source mirrored pair. If that DASD fails and cannot be repaired concurrently, the system cannot be IPLed until the failed load source is fixed and the remote load source recovery procedure is performed. • When Remote DASD Mirroring is active on a system, and the one load source the system can use to IPL fails, the system cannot perform a main storage dump if the system ends abnormally. This means that the system cannot use the main storage dump or continuously-powered main store (CPM) to reduce recovery time after a system crash. It also means that the main storage dump is not available to diagnose the problem that causes the system to end abnormally. 4.5 Planning your mirroring installation If you decide that remote DASD mirroring is right for your system, prepare your system and then start site-to-site mirroring. Determine whether your system is balanced and meets standard configuration rules. The system must be configured according to the standard rules for the mirrored pairing function to pair up storage units to provide the best possible protection from the hardware that is available. Plan for the new units to add for each ASP. If you plan to start mirrored protection on a new system, that system is already configured according to standard configuration rules. If you are using an older system, it may not follow the standard rules. However, wait until after attempting to start mirrored protection before reconfiguring any hardware. When considering mirrored protection, review these planning steps: 1. Decide which ASP or ASPs to protect. 2. Determine the disk storage capacity requirements. 3. Determine the level of protection that is needed for each mirrored ASP. 4. Determine what extra hardware is necessary for mirrored protection. 5. Determine what extra hardware is needed for performance. Hardware support for single system high availability 49 In general, the units in an ASP should be balanced across several I/O processors, rather than all being attached to the same I/O processor. This provides better protection and performance. Plan the user ASPs that have mirrored protection and determine what units to add to the ASPs. Refer to Chapter 5, “Auxiliary storage pools (ASPs)” on page 63, for more information about ASPs. 4.5.1 Comparing DASD management with standard and remote mirroring For the most part, managing DASD with remote mirroring is similar to managing DASD with standard mirroring. The differences are in how you add disk units and how you restore mirrored protection after a recovery. Adding disk units Unprotected disk units must be added in pairs just as with general mirroring. To achieve remote protection of all added units, one half of the new units of each capacity of DASD should be in the remote group and one half in the local group. Single device-parity protected units may be added to ASPs using remote mirroring. However, the ASP is not protected against a site disaster. 4.6 Device parity protection Device parity protection is a high availability hardware function (also known as RAID-5) that protects data from loss due to a disk unit failure or because of damage to a disk. It allows the system to continue to operate when a disk unit fails or disk damage occurs. The system continues to run in an exposed mode until the damaged unit is repaired and the data is synchronized to the replaced unit. To protect data, the disk controller or input/output processor (IOP) calculates and saves a parity value for each bit of data. Parity protection is built into many IOPs. It is activated for disk units that are attached to those IOPs. Device parity involves calculating and saving a parity value for each bit of data. Conceptually, the parity value is computed from the data at the same location on each of the other disk units in the device parity set. When a disk failure occurs, the data on the failing unit is reconstructed using the saved parity value and the values of bits in the same location on other disks. The system continues to run while the data is being reconstructed. Logically, the implementation of device parity protection is similar to the system checksum function. However, device parity is built into the hardware. Checksum, on the other hand, is started or stopped using configuration options on the AS/400 system menu. If a failure occurs, correct the problem quickly. In the unlikely event that another disk fails in the same parity set, you may lose data. Recommendation 50 High Availability on the AS/400 System: A System Manager’s Guide The overall goal of device parity protection is to provide high availability and to protect data as inexpensively as possible. If possible, protect all the disk units on your system with device parity protection or mirrored protection. This prevents the loss of information when a disk failure occurs. In many cases, you can also keep your system operational while a disk unit is being repaired or replaced. Before using device parity protection, note the benefits that are associated with it, as well as the costs and limitations. Some device parity protection advantages: • It can prevent your system from stopping when certain types of failures occur. • It can speed up your recovery process for certain types of failures, such as a site disaster or an operator or programmer error. • Lost data is automatically reconstructed by the disk controller after a disk failure. • The system continues to run after a single disk failure. • A failed disk unit can be replaced without stopping the system. • Device parity protection reduces the number of objects that are damaged when a disk fails. Some device parity protection disadvantages: • It is not a substitute for a backup and recovery strategy. • It does not provide protection from all types of failures, such as a site disaster or an operator or programmer error. • Device parity protection can require additional disk units to prevent slower performance. • Restore operations can take longer when you use device parity protection. System checksum is another disk protection method similar to device parity. Checksum is not supported on RISC systems, and it is not discussed in this Redpaper. You can find information on checksum in Backup and Recovery, SC41-5306. Note Device parity protection is not a substitute for a backup and recovery strategy. Device parity protection can prevent your system from stopping when certain types of failures occur. It can speed up your recovery process for certain types of failures. But device parity protection does not protect you from many types of failures, such as system outages that are caused by failures in other disk-related hardware (for example, disk controllers, disk I/O processors, or a system bus). Remember Hardware support for single system high availability 51 For information on planning for device parity protection, refer to Appendix B, “Planning for device parity protection” on page 147. 4.6.1 How device parity protection affects performance Device parity protection requires extra I/O operations to save the parity data. This may cause a performance problem. To avoid this problem, some IOPs contain a non-volatile write cache that ensures data integrity and provides faster write capabilities. The system is notified that a write operation is complete as soon as a copy of the data is stored in the write cache. Data is collected in the cache before it is written to a disk unit. This collection technique reduces the number of physical write operations to the disk unit. Because of the cache, performance is generally about the same on protected and unprotected disk units. Applications that have many write requests in a short period of time, such as batch programs, can adversely affect performance. A single disk unit failure can adversely affect the performance for both read and write operations. The additional processing that is associated with a disk unit failure in a device parity set can be significant. The decrease in performance is in effect until the failed unit is repaired (or replaced) and the rebuild process is complete. If device parity protection decreases performance too much, consider using mirrored protection. These topics provide additional details on how a disk unit failure affects performance: • Disk unit failure in a device parity protection configuration • Input/output operations during a rebuild process • Read operations on a failed disk unit • Write operations on a failed disk unit 4.6.1.1 Disk unit failure in a device parity protection configuration The write-assist device is suspended when a disk unit failure occurs in a subsystem with device parity protection. If the write-assist device fails, it is not used again until the repair operation is completed. The performance advantage of the write-assist device is lost until the disk unit is repaired. The subsystems with device parity protection are considered to be exposed until the synchronization process completes after replacing the failed disk unit. While the disk unit is considered exposed, additional I/O operations are required. 4.6.1.2 Input/output operations during a rebuild process I/O operations during the rebuild (synchronization) process of the failed disk unit may not require additional disk I/O requests. This depends on where the data is read from or written to on the disk unit that is in the synchronization process. For example: • A read operation from the disk area that already has been rebuilt requires one read operation. • A read operation from the disk area that has not been rebuilt is treated as a read operation on a failed disk unit. • A write operation to the disk that has already been rebuilt requires normal read and write operations (two read operations and two write operations). • A write operation to the disk area that has not been rebuilt is treated as a write operation to a failed disk unit. 52 High Availability on the AS/400 System: A System Manager’s Guide Note: The rebuild process takes longer when read and write operations to a replaced disk unit are also occurring. Every read request or every write request interrupts the rebuild process to perform the necessary I/O operations. 4.6.2 Using both device parity protection and mirrored protection Device parity protection is a hardware function. Auxiliary storage pools and mirrored protection are software functions. When you add disk units and start device parity protection, the disk subsystem or IOP is not aware of any software configuration for the disk units. The software that supports disk protection is aware of which units have device parity protection. These rules and considerations apply when mixing device parity protection with mirrored protection: • Device parity protection is not implemented on ASP boundaries. • Mirrored protection is implemented on ASP boundaries. • You can start mirrored protection for an ASP even if it currently has no units that are available for mirroring because they all have device parity protection. This ensures that the ASP is always fully protected, even if you add disks without device parity protection later. • When a disk unit is added to the system configuration, it may be device parity protected. • For a fully-protected system, you should entirely protect every ASP by device parity protection, by mirrored protection, or both. • Disk units that are protected by device parity protection can be added to an ASP that has mirrored protection. The disk units that are protected by device parity protection do not participate in mirrored protection (hardware protects them already). • When you add a disk unit that is not protected by device parity protection to an ASP that has mirrored protection, the new disk unit participates in mirrored protection. Disk units must be added to, and removed from, a mirrored ASP in pairs with equal capacities. • Before you start device parity protection for disk units that are configured (assigned to an ASP), you must stop mirrored protection for the ASP. • Before you stop device parity protection, you must stop mirrored protection for any ASPs that contain affected disk units. • When you stop mirrored protection, one disk unit from each mirrored pair becomes non-configured. You must add the non-configured units to the ASP again before starting mirrored protection. 4.7 Comparing the disk protection options There are several methods for configuring your system to take advantage of the disk protection features. Before selecting the disk protection options that you want to use, compare the extent of protection that each one provides. Hardware support for single system high availability 53 Table 2 provides an overview of the availability tools that can be used on the AS/400 system to protect against different types of failure. Table 2. Availability tools for the AS/400 system Be aware of the following considerations when selecting disk protection options: • With both device parity protection and mirrored protection, the system continues to run after a single disk failure. With mirrored protection, the system may continue to run after the failure of a disk-related component, such as a controller or an IOP. • When a second disk failure occurs, meaning that the system has two failed disks, the system is more likely to continue to run with mirrored protection than with device parity protection. With device parity protection, the probability of the system failing on the second disk failure can be expressed as P out of n. Here, P is the total number of disks on the system, and n is the number of disks in the device parity set that had the first disk failure. With mirrored protection, the probability of the system failing on the second disk failure is 1 out of n. What is needed Device parity protection Mirrored protection User ASPs Protection from data loss due to disk-related hardware failure Yes - See Note 1 Yes Yes - See Note 4 Maintain availability Yes Yes No Help with disk unit recovery Yes - See Note 1 Yes Yes - See Note 4 Maintain availability when disk controller fails See Note 3 Yes - See Note 2 No Maintain availability when disk I/O processor fails No Yes - See Note 2 No Maintain availability when disk I/O bus fails No Yes - See Note 2 No Site disaster protection No Yes - See Note 5 No Notes: 1. Load source unit and any disk units attached to the MFIOP are not protected. 2. Depends on hardware used, configuration, and level of mirrored protection. 3. With device parity protection using the 9337 Disk Array Subsystem, the system becomes unavailable if a controller is lost. 4. With device parity protection using the IOP feature, the system is available as long as the IOP is available. 5. Configuring ASPs can limit the loss of data and the recovery to a single ASP. 6. For site disaster protection, remote mirroring is required. 54 High Availability on the AS/400 System: A System Manager’s Guide • Device parity protection requires up to 25% additional disk capacity for storage of parity information. The actual increase depends on the number of disk units that are assigned to a device parity set. A system with mirrored protection requires twice as much disk capacity as the same system without mirrored protection. This is because all information is stored twice. Mirrored protection may also require more buses, IOPs, and disk controllers, depending on the level of protection that you want. Therefore, mirrored protection is usually a more expensive solution than device parity protection. • Usually, neither device parity protection or mirrored protection has a noticeable effect on system performance. In some cases, mirrored protection actually improves system performance. The restore time to disk units protected by device parity protection is slower than the restore time to the same disk devices without device parity protection activated. This is because the parity data must be calculated and written. 4.8 Concurrent maintenance Concurrent maintenance is the process of repairing or replacing a failed disk-related hardware component while using the system. Concurrent maintenance allows disks, I/O processors, adapters, power supplies, fans, CD-ROMs, and tapes to be replaced without powering down the server. On systems without mirrored protection, the system is not available when a disk-related hardware failure occurs. It remains unavailable until the failed hardware is repaired or replaced. However, with mirrored protection, the failing hardware can often be repaired or replaced while the system is being used. Concurrent maintenance support is a function of system unit hardware packaging. Not all systems support concurrent maintenance. Mirrored protection only provides concurrent maintenance when it is supported by the system hardware and packaging. The best hardware configuration for mirrored protection also provides for the maximum amount of concurrent maintenance. It is possible for the system to operate successfully through many failures and repair actions. For example, a failure of a disk head assembly does not prevent the system from operating. A replacement of the head assembly and synchronization of the mirrored unit can occur while the system continues to run. The greater your level of protection, the more often concurrent maintenance can be performed. On some models, the system restricts the level of protection for Unit 1 and its mirrored unit to controller-level protection only. Under some conditions, diagnosis and repair can require active mirrored units to be suspended. You may prefer to power down the system to minimize the exposure of operating with less mirrored protection. Some repair actions require that the system be powered down. Deferred maintenance is the process of waiting to repair or replace a failed disk-related hardware component until the system can be powered down. The system is available, although mirrored protection is reduced by whatever hardware components have failed. Deferred maintenance is only possible with mirrored protection or device parity protection. Hardware support for single system high availability 55 4.9 Redundancy and hot spare The basic rule for making a server system highly available is to use redundant parts where needed and affordable. Just like the basic idea behind Redundant Array of Independent Disks (RAID), all parts in a server are subject to be a single point of failure. These part can include: • The CPU • The power supply • The main logical board • The main memory • Adapter cards These are all parts that, if even one fails, the overall system is rendered unusable. To decrease the time involved in replacing a defective component, some customers consider implementing what is known as a hot spare. In effect, the customer keeps a local inventory of any component that either: • Has a higher failure rate than usual • Has a long lead-time when a replacement is required Note: The term hot spare typically refers to a disk unit. However, the same concept applies to a hot site or another system used for recovery. Planning for spare disk units Spare disk units can reduce the time the system runs without mirrored protection after a disk unit failure of a mirrored pair. If a disk unit fails, and a spare unit of the same capacity is available, that spare unit can be used to replace the failed unit. The system logically replaces the failed unit with the selected spare unit. It then synchronizes the new unit with the remaining good unit of the mirrored pair. Mirrored protection for that pair is again active when synchronization completes (usually less than an hour). However, it may take several hours (from the time a service representative is called until the failed unit is repaired and synchronized) before mirrored protection is again active for that pair. To make full use of spare units, you need at least one spare unit of each capacity that you have on your system. This provides a spare for any size of disk unit that may fail. 4.10 OptiConnect: Extending a single system An OptiConnect cluster is a collection of AS/400 systems connected by dedicated fiber optic system bus cables. The systems in an OptiConnect cluster share a common external optical system bus located in an expansion tower or frame. The system providing the shared system bus is called the hub system. Each system that plugs into this shared bus with an OptiConnect Bus Receiver card is called a satellite system. Each satellite system dedicates one of its external system buses that connects to the receiver card in the hub system’s expansion tower or rack. The term OptiConnect link refers to the fiber optic connection between systems in the OptiConnect cluster. The term path refers to the logical software established connection between two OptiConnect systems. An OptiConnect network always 56 High Availability on the AS/400 System: A System Manager’s Guide consists of at least two AS/400 systems. One of the AS/400 systems is designated as the hub and at least one other system is designated as a satellite. There are two levels of redundancy available in an OptiConnect cluster: • Link redundancy: Link redundancy is an optical bus hardware feature. Any two systems attached to the hub system shared bus can establish a path between them, including paths to the hub system itself. You can establish path redundancy by configuring two hub systems in the OptiConnect cluster. Each satellite uses two buses to connect with two hub systems. OptiConnect software detects the two logical paths between the two systems and uses both paths for data flow. If a path failure occurs, the remaining path picks up all of the communication traffic. • Path redundancy: The OS/400 infrastructure for any system determines the logical path to another system. It does this by designating which system bus each of the systems that form the path uses. The link between any two satellite systems does not depend on the hub system bus. The two systems use the bus, but the hub system is not involved. Link redundancy is determined by the system models. For OptiConnect clusters, link redundancy is always provided when the extra fiber optic cable is installed. For path redundancy, an extra set of OptiConnect receiver cards and an extra expansion tower or frame are required along with another set of cables. OptiConnect for OS/400 is an AS/400-to-AS/400 communication clustering solution. It combines unique OptiConnect fiber bus hardware and standard AS/400 bus hardware with unique software. It uses distributed data management (DDM) to allow applications on one AS/400 system to access databases located on other AS/400 systems. The AS/400 systems that contain the databases are the database servers. The remote systems are considered the application client or clients. In most cases, the hub also acts as the database server. Since all systems can communicate with each other (providing that the hub is active), any system can be the client. Some OptiConnect configurations have AS/400 systems that act simultaneously as a server and a client. However, any system can act as a database server. OptiConnect for OS/400 is a communications vehicle. OptiConnect for OS/400 products provide AS/400 systems with physical links for a high availability clustered solution. OptiConnect for OS/400 components support the infrastructure for applications to conduct data exchanges over high speed connections. OptiConnect for OS/400 does not offer high availability with applications that utilize the hardware links. OptiConnect for OS/400 can be the transport mechanism for in-house developed applications, business partner software, or remote journal support. Further information on OptiConnect is found in 6.8, “Bus level interconnection” on page 82, and 6.8.1, “Bus level interconnection and a high availability solution” on page 84. Hardware support for single system high availability 57 4.11 Cluster support For planned or unplanned outages, clustering and system mirroring offer the most effective solution. For customers requiring better than 99.9% system availability, AS/400 clusters are viable. Cluster solutions connect multiple AS/400 systems together with various interconnect fabrics, including high-speed optical fiber, to offer a solution that can deliver up to 99.99% system availability. High availability is achieved with an alternative system that replicates the availability of the production system. These systems are connected by high-speed communications and use replication software to achieve this. They also require enough DASD to replicate the whole or critical part of the production system. With the entire system replicated, the mirrored system can enable more than just a disaster recovery solution. Combining these clusters with software from AS/400 high-availability business partners (such as those described in Chapter 10, “High availability business partner solutions” on page 111) improves the availability of a single AS/400 system by replicating business data to one or more AS/400 systems. This combination can provide a disaster recovery solution. Clusters are a configuration or a group of independent servers that appear on a network as a single machine. As illustrated in Figure 11, a cluster is a collection of complete systems that work together to provide a single and unified computing resource Figure 11. Cluster definition This cluster group is managed as a single system or operating entity and is designed specifically to tolerate component failures and to support the addition or subtraction of components in a way that is transparent to users. Clusters allow you to efficiently group systems together to set up an environment that provides availability that approaches 100% for critical applications and critical data. Resources can be accessed without regard to location. A client interacts with a cluster as if it were a single system. 58 High Availability on the AS/400 System: A System Manager’s Guide With the introduction of clusters, the AS/400e system offers a continuous availability solution if your business demands operational systems 24 hours a day, 365 days a year (24 x 365). This solution, called OS/400 Cluster Resource Services, is part of the OS/400 operating system. It provides failover and switchover capabilities for your systems that are used as database servers or application servers. If a system outage or a site loss occurs, the functions that are provided on a clusters server system can be switched over to one or more designated backup systems that contain a current copy (replica) of your critical resource. The failover can be automatic if a system failure should happen, or if you can control how and when the transfer takes place by manually initiating a switchover. Cluster management tools control the cluster from anywhere in the network. End users work on servers in the cluster without knowing or caring where their applications are running. In the event of a failure, Cluster Resource Services (CRS), which is running on all systems, provides a switchover. This switch causes minimal impact to the end user or applications that are running on a server system. Data requesters are automatically rerouted to the new primary system. You can easily maintain multiple data replications of the same data. Any AS/400 model that can run OS/400 V4R4 or later is compatible for implementing clustering. You must configure Transmission Control Protocol/Internet Protocol (TCP/IP) on your AS/400e systems before you can implement clustering. In addition, you can purchase a cluster management package from a High Availability Business Partner (HAV BP) that provides the required replication functions and cluster management capabilities. Refer to AS/400 Clusters: A Guide to Achieving Higher Availability, SG24-5194, for further information. 4.12 LPAR hardware perspective Logical partitions allow you to run multiple independent OS/400 instances or partitions. Figure 12 shows a basic LPAR configuration. For V4R5, each partition has has its own processors, memory, and disks. For V5R1, resources can be share between partitions. With logical partitioning, you can address multiple system requirements in a single machine to achieve server consolidation, business unit consolidation, and mixed production and test environments. You can run a cluster environment on a single system image. LPAR support is available on n-way symmetric multiprocessing iSeries models 8xx and AS/400 models 6xx, Sxx, and 7xx. See 7.5, “Logical Partition (LPAR) support” on page 93. Primary partition Development environment #1 SYS1 OS/400 OS/400 Production environment SYS2 PPPPP PP SYS3 Development environment #2 OS/400 P New release testing 840 8-way AS/400 with 3 partitions V4R5 V4R4 V4R5 Figure 12. A basic LPAR configuration Hardware support for single system high availability 59 By itself, LPAR does not provide a significant availability increase. It can, however, be used to complement other availability strategies. See 7.5, “Logical Partition (LPAR) support” on page 93, for a discussion of LPAR from an OS/400 view. 4.12.1 Clustering with LPAR support Since each partition on an LPAR system is treated as a separate server, you can run a cluster environment on a single system image. One cluster node per CPU can exist within one LPAR system. Clustering partitions can provide for a more cost efficient clustering solution than multiple systems. However, an LPAR clustered environment increases single points of failure. For example, if the server’s primary partition becomes unavailable, all secondary partitions also become unavailable (the opposite is not true). In some environments, LPAR ideally lends itself to situations where both a local and remote backup server is desired. A good example is when a business works to provide its own disaster recovery capability. The highest level of availability is obtained with two separate servers. Figure 13 shows that, with clustering active, data is replicated to both the local backup server and the remote server. In the event of a disaster (or the need for the entire local hardware to be powered off), the remote backup server is available. In some cases, this is more cost-efficient (including floor space) than separate servers. Integrated availability options In most cases, it is recommended that integrated availability solutions be used with a cluster to further mask or reduce downtime and to increase a cluster's efficiency. Consider the following list: • Disk protection: Device Parity Protection (RAID-5) and OS/400 Disk Mirroring. • Auxiliary storage pools (ASPs) • Access path protection • Logical Partitions (LPAR) In all cases, it is highly recommended that these integrated availability options be used in a clustered environment, as well as on a standalone iSeries or AS/400 server. Primary partition Production environment SYS1 OS/400 OS/400 Backup environment 2 SYS2 PP PPPPPP Local 8-way iSeries (partitioned) SYS3 Backup environment 3 and remote hot backup for disaster recovery OS/400 PPPP Remote 4-way AS/400 HABP Replication HABP Replication HABP Replication Figure 13. LPAR, local and remote iSeries and AS/400 cluster 60 High Availability on the AS/400 System: A System Manager’s Guide 4.13 UPS An uninterruptible power supply (UPS) provides auxiliary power to the processing unit, disk units, the system console, and other devices that you choose to protect from power loss. When you use a UPS with the AS/400 system, you can: • Continue operations during brief power interruptions (brown outs). • Protect the system from voltage peaks (white outs). • Provide a normal end of operations that reduces recovery time when the system is restarted. If the system abnormally ends before completing a normal end of operations, recovery time is significant. Normally, a UPS does not provide power to all local workstations. The UPS also usually does not provide power to modems, bridges, or routers that support remote workstations. Consider supplying alternate power to both workgroups since the inability of worker access to information disrupts productivity. You can avoid such disruption with proper availability and recovery implementation. Also, design your interactive applications to handle the loss of communication with a workstation. Otherwise, system resources are used in an attempt to recover devices that have lost power. Refer to Chapter 12, “Communications Error Recovery and Availability” in The System Administrator’s Companion to AS/400 Availability and Recovery, SG24-2161, for more information on resources used during device recovery. The programming language reference manuals provide examples of how to use the error feedback areas to handle workstations that are no longer communicating with the application. Backup and Recovery, SC41-5304, describes how to develop programs to handle an orderly shutdown of the system when the UPS takes over. 4.14 Battery backup Most (but not all) AS/400 models are equipped with a battery backup. Based on the system storage size, relying on a battery backup for enough time for an orderly shutdown is not sufficient. The battery capacity typically varies between 10 and 60 minutes. The useful capacity depends on the application requirements, main storage size, and system configuration. Consider the reduction of capacity caused by the natural aging of the battery and environmental extremes of the site when selecting the battery. The battery must have the capacity to maintain the system load requirements at the end of its useful life. Refer to Backup and Recovery, SC41-5304, for power down times for the advanced series systems. Refer to the AS/400 Physical Planning Reference, SA41-5109, for power down times for the AS/400 Bnn-Fnn models. Also, refer to the Physical Planning Reference for later AS/400 models at the Web site: http://www.as400.ibm.com/tstudio/planning/index.rf.htm Hardware support for single system high availability 61 4.15 Continuously powered main storage On V3R6 systems and later, AS/400 systems are equipped with a System Power Control Network (SPCN) feature. This provides the Continuously Powered Main Storage (CPM) function. During a power fluctuation, the transition to CPM mode is 90 seconds after an initial 30 second waiting period. The internal battery backup provides sufficient power to keep the AS/400 system up for the 120 seconds until the transition to the CPM is complete. With CPM enabled, the battery provides sufficient power to shut down the system and maintain the contents of memory for up to 48 hours after a power loss without user interface or control. The transition to CPM is irreversible. CPM interrupts the processes at the next microcode end statement and forces as many updates to the disk as it can. During the next IPL, it restores main storage and attempts to complete outstanding updates. Preserving main storage contents significantly reduces the amount of time the system requires to perform an IPL after a power loss. CPM operates outside of transaction boundaries. You can use the CPM feature along with a UPS (or the battery backup). If the system detects that the UPS can no longer provide sufficient power to the system, the data currently in memory is put into “sleep” mode. The CPM storage feature takes control and maintains data in memory for up to 48 hours. With the CPM feature, the system automatically initiates an IPL after power is restored. CPM is a viable feature. Choosing to use CPM depends on your expectations of your local power and battery backup or generator to maintain power at all times. Refer to Backup and Recovery, SC41-5304, for more information on CPM requirements. 4.16 Tape devices For information on what tape devices are available for each AS/400 model, and the hardware and software requirements to support each model, refer to the iSeries Handbook, GA19-5486, and iSeries and AS/400e System Builder, SG24-2155. For save and restore performance rates, see Appendix C, “Save and Restore Rates of IBM Tape Drives for Sample Workloads”, and Section 8.1, “Save and Restore Performance” in the AS/400 Performance Capabilities Manual at: http://publib.boulder.ibm.com/pubs/pdfs/as400/V4R5PDF/AS4PPCP3.PDF 4.16.1 Alternate installation device On V4R1 (and later) systems, you can use a combination of devices that are attached on the first system bus, as well as additional buses. The alternate installation device does not need to be attached to the first system bus. For example, the 3590 tape drive can be positioned up to 500 meters or two kilometers away. This enables a physical security improvement since users who are allowed access to the machine room may be different than those operating the tape drives. 62 High Availability on the AS/400 System: A System Manager’s Guide You can select an alternate installation device connected through any I/O bus attached to the system. When you perform a D-mode IPL (D-IPL), you can use the tape device from another bus using the Install Licensed Internal Code display. For example, if you have a 3590 attached to another bus (other than Bus 1), you can choose to install from the alternate installation device using the Install Licensed Internal Code display and then continue to load the LIC, OS/400, and user data using the alternate installation device. Note: Set up alternate installation device support prior to performing a D-IPL. System Licensed Internal Code (SLIC) media is necessary to perform the D-IPL that restores and installs from the tape device. Some models (typically with 3590 tape devices attached) experience a performance improvement when using an alternate installation device for other save and restore or installation operations. This is caused by having the tape drive on a different IOP than the one to which the load source unit is attached. On systems prior to V4R1, the alternate installation device is only supported using devices attached to the first system bus. The first system bus connects to the service processor IOP. Typically, this is where the optical or tape devices used for installations are attached. Before using the alternate installation device, ensure that it is defined on a bus other than system Bus 1. You must enable the device. When installing from the alternate installation device, you need both your tape media and the CD-ROM media containing the Licensed Internal Code. Recommendation © Copyright IBM Corp. 2001 63 Chapter 5. Auxiliary storage pools (ASPs) An auxiliary storage pool (ASP) is a software definition of a group of disk units on your system. This means that an ASP does not necessarily correspond to the physical arrangement of disks. Conceptually, each ASP on your system is a separate pool of disk units for single-level storage. The system spreads data across the disk units within an ASP. If a disk failure occurs, you need to recover only the data in the ASP that contained the failed unit. Prior to V5R1, here are two types of ASPs: • System auxiliary storage pool • User auxiliary storage pools Your system may have many disk units attached to it that are optionally assigned to an auxiliary storage pool. To your system, the pool looks like a single unit of storage. The system spreads objects across all disk units. You can use auxiliary storage pools to separate your disk units into logical subsets. When you assign the disk units on your system to more than one ASP, each ASP can have different strategies for availability, backup and recovery, and performance. ASPs provide a recovery advantage if the system experiences a disk unit failure resulting in data loss. If this occurs, recovery is only required for the objects in the ASP that contained the failed disk unit. System objects and user objects in other ASPs are protected from the disk failure. There are also additional benefits and certain costs and limitations that are inherent in using ASPs. 5.1 Deciding which ASPs to protect Because mirrored protection is configured by auxiliary storage pool, the ASP is the user’s level of control over single-level storage. Mirrored protection can be used to protect one, some, or all ASPs on a system. However, multiple ASPs are not required in order to use mirrored protection. Mirrored protection works well when all disk units on a system are configured into a single ASP (the default on the AS/400 system). In fact, mirroring reduces the need to partition auxiliary storage into ASPs for data protection and recovery. However, ASPs may still be recommended for performance and other reasons. To provide the best protection and availability for the entire system, mirror all ASPs in the system. Consider the following situations: • If the system has a mixture of some ASPs with and without mirrored protection, a disk unit failure in an ASP without mirrored protection severely limits the operation of the entire system. Data can be lost in the ASP in which the failure occurred. A long recovery may be required. Independent ASPs are introduced at V5R1. At the time this Redpaper was written, the appropriate information was not available. Note 64 High Availability on the AS/400 System: A System Manager’s Guide • If a disk fails in a mirrored ASP, and the system also contains ASPs that are not mirrored, data is not lost. However, in some cases, concurrent maintenance may not be possible. The disk units that are used in user ASPs should be selected carefully. For best protection and performance, an ASP should contain disk units that are attached to several different I/O processors. The number of disk units in the ASP that are attached to each I/O processor should be the same (that is, balanced). ASPs are further discussed in Chapter 5, “Auxiliary storage pools (ASPs)” on page 63. 5.1.1 Determining the disk units needed A mirrored ASP requires twice as much auxiliary storage as an ASP that is not mirrored. This is because the system keeps two copies of all the data in the ASP. Also, mirrored protection requires an even number of disk units of the same capacity so that disk units can be made into mirrored pairs. On an existing system, note that it is not necessary to add the same types of disk units already attached to provide the required additional storage capacity. Any new disk units may be added as long as sufficient total storage capacity and an even number of storage units of each size are present. The system assigns mirrored pairs and automatically moves the data as necessary. If an ASP does not contain sufficient storage capacity, or if storage units cannot be paired, mirrored protection cannot be started for that ASP. The process of determining the disk units needed for mirrored protection is similar for existing or new systems. Review the following points to plan disk requirements: 1. Determine how much data each ASP contains. 2. Determine a target percent of storage used for the ASP (how full the ASP will be). 3. Plan the number and type of disk units needed to provide the required storage. For an existing ASP, you can plan a different type and model of disk unit to provide the required storage. After planning for all ASPs is complete, plan for spare units, if desired. Once you know all of this information, you can calculate your total storage needs. The planned amount of data and the planned percent of storage used work together to determine the amount of actual auxiliary storage needed for a mirrored ASP. For example, if an ASP contains 1 GB (GB equals 1,073,741,824 bytes) of actual data, it requires 2 GB of storage for the mirrored copies of the data. If 50% capacity is planned for that ASP, the ASP needs 4 GB of actual storage. If the planned percent of storage used is 66%, 3 GB of actual storage is required. One gigabyte of real data (2 GB of mirrored data) in a 5 GB ASP results in a 40% auxiliary storage utilization. Total planned storage capacity needs After planning for the number and type of storage units needed for each ASP on the system, and for any spare storage units, add the total number of storage units of each disk unit type and model. Auxiliary storage pools (ASPs) 65 The number planned is the number of storage units of each disk unit type, not the number of disk units. The following section provides a more detailed description. 5.2 Assigning disk units to ASPs If you decide that you want more than one auxiliary storage pool (ASP), make the following determinations for each ASP: • How much storage do you need? • What disk protection (if any) should you use? • Which disk units should be assigned? • Which objects should be placed in the ASP? The Workstation Customization Programming book, SC41-5605, provides information to help you with these considerations. This book is only available online at the AS/400 Library at: http://as400bks.rochester.ibm.com At the site, click AS/400 Information Center. Select your language and click GO! Click V4R4 and then click Search or view all V4R4 books. Enter the book number in the seach field and click Find. Finally, click the appropriate publication that appears. When you work with disk configuration, you may find it helpful to begin by making a list of all the disks and disk-related components on your system. You can put this information in a chart like Table 3, or you may want to draw a diagram. Table 3. Disk configuration example chart 5.3 Using ASPs User ASPs are used to manage the following system performance and availability requirements: • Provide dedicated resources for frequently used availability objects, such as journal receivers • Allow online and unattended saves. • Place infrequently used objects, such as large history files, on disk units with slower performance IOP Controller Unit Type and model Type and model Capacity Resource name Name of mirrored pair 1 00 01 1 6602-030 1031 1 DD001 1 10 01 2 6602-074 733 1 DD019 1 10 02 3 6602-070 1031 1 DD036 1 00 02 6 6602-030 1031 1 DD002 1 10 03 4 6602-074 773 3 DD005 1 10 04 5 6602-074 773 3 DD033 66 High Availability on the AS/400 System: A System Manager’s Guide 5.3.1 Using ASPs for availability Different parts of your system may have different requirements for availability and recovery. For example, you may have a large history file that is changed only at the end of the month. The information in the file is useful but not critical. You may put this file in a separate library in a user ASP that does not have any disk protection (mirrored protection or device parity protection). You could omit this library from your daily save operations and choose to save it only at the end of the month when it is updated. Another example would be documents and folders. Some documents and folders are critical to the organization and should be protected with device parity protection or mirrored protection. They can be put in a protected user ASP. Others are kept on the system to provide information but do not change very often. They can be in a different user ASP with a different strategy for saving and for protection. 5.3.2 Using ASPs to dedicate resources or improve performance If you are using user ASPs for better system performance, consider dedicating the ASP to one object that is very active. In this case, you can configure the ASP with only one disk unit. However, it usually does not improve performance to place a single device-parity protected unit in a user ASP because the performance of that unit is affected by other disk units in the device parity set. Refer to Figure 14 for a visual example of multiple ASPs. Figure 14. Auxiliary storage pools Allocating one user ASP exclusively for journal receivers that are attached to the same journal can improve journaling performance. By having the journal and database files in a separate ASP from the attached receivers, there is no contention for journal receiver write operations. The units that are associated with the ASP do not have to be repositioned before each read or write operation. Journaling uses as many as 10 disk arms when writing to a journal receiver. Configuring an ASP with more than 10 arms does not provide any additional performance advantage for journaling. However, if you do have an ASP with more than 10 arms, journaling uses the 10 fastest arms. If you add more disk units to Load Source System ASP User ASP used for save/archive on compressed DASD User ASP used for journal receivers ASP 1 ASP 2 ASP 3 Auxiliary storage pools (ASPs) 67 the ASP while the system is active, the system determines whether to use the new disk units for journal receivers the next time the change journal function is performed. Another method for improving performance is to make sure that there are enough storage units in the user ASP to support the number of physical input and output operations that are done against the objects in the user ASP. You may have to experiment by moving objects to a different user ASP and then monitor performance in the user ASP to see if the storage units are used excessively. If the units show excessive use, you should consider adding more disk units to the user ASP. 5.3.3 Using ASPs with document library objects You can place document library objects (DLOs) in user ASPs. The possible advantages of placing DLOs in user ASPs are: • The ability to reduce save times for DLOs and to separate them by their save requirements. • The ability to separate DLOs by availability requirements. Critical DLOs can be placed in user ASPs that are protected by mirrored protection or device parity protection. DLOs that change infrequently can be placed in unprotected ASPs with slower drives. • The ability to grow to a larger number of documents. If you have V3R7 or a later release of the OS/400 licensed program, you can run multiple SAVDLO or RSTDLO procedures against different ASPs. If you have V4R1 or a later release of the OS/400 licensed program, you can run multiple SAVDLO operations on the same ASP. One approach for placing DLOs in user ASPs is to leave only system DLOs (IBM-supplied folders) in the system ASP. Move other folders to user ASPs. The system folders do not change frequently, so they can be saved infrequently. You can specify an ASP on the SAVDLO command. This allows you to save all the DLOs from a particular ASP on a given day of the week. For example, you could save DLOs from ASP 2 on Monday, DLOs from ASP 3 on Tuesday, and so on. You could save all changed DLOs daily. The recovery steps if you use this type of save technique would depend on what information was lost. If you lost an entire ASP, you would restore the last complete saved copy of DLOs from that ASP. You would then restore the changed DLOs from the daily saves. When you save DLOs from more than one ASP in the same operation, a different file and a sequence number are created on the tape for each ASP. When you restore, you must specify the correct sequence number. This makes it simple to restore the changed DLOs only to the ASP that was lost without needing to know all the folder names. These restrictions and limitations apply when placing DLOs in user ASPs: • When using a save file for a save operation, you can save DLOs from only one ASP. • When using an optical file for a save operation, you can save DLOs from only one ASP. 68 High Availability on the AS/400 System: A System Manager’s Guide • If you are saving to a save file and you specify SAVDLO DLO(*SEARCH) or SAVDLO DLO(*CHG), you must also specify an ASP, even if you know the results of you search are found in a single ASP. • Documents that are not in folders must be in the system ASP. • Mail can be filed into a folder on a user ASP. Unfiled mail is in the system ASP. Note: When you specify DLO(*SEARCH) or DLO(*CHG) for the SAVDLO command, specify an ASP, if possible. Specifying an ASP saves system resources. 5.3.4 Using ASPs with extensive journaling If journals and files being journaled are in the same ASP as the receivers and the ASP overflows, you must end journaling of all files and recover from the overflowed condition for the ASP. Backup and Recovery describes how to recover an overflowed ASP. If the journal receiver is in a different ASP than the journal, and the user ASP that the receiver is in overflows, perform the following steps: 1. Create a new receiver in a different user ASP. 2. Change the journal (CHGJRN command). 3. Save the detached receiver. 4. Delete the detached receiver. 5. Clear the overflowed ASP without ending journaling. 6. Create a new receiver in the cleared ASP. 7. Attach the new receiver with the CHGJRN command. 5.3.5 Using ASPs with access path journaling If you plan to use explicit access path journaling, IBM recommends that you first change the journal to a journal receiver in the system ASP (ASP 1) for a few days. Start access path journaling to see storage requirements for the receiver before you allocate the specific size for a user ASP. 5.3.6 Creating a new ASP on an active system Beginning with V3R6 of the OS/400 licensed program, you can add disk units while your system is active. When you add disk units to an ASP that does not currently exist, the system creates a new ASP. If you choose to create a new user ASP while your system is active, be sure you understand the following considerations: • You cannot start mirrored protection while the system is active. The new ASP is not fully protected unless all of the disk units have device parity protection. • You cannot move existing disk units to the new ASP while your system is active.The system must move data when it moves disk units. This can be done only through Dedicated Service Tools (DST). • The system uses the size of an ASP to determine the storage threshold for the journal receivers that are used by system-managed access-path protection (SMAPP). When you create an ASP while your system is active, the size of the disk units that you specify on the operation that creates the ASP is considered the size of the ASP for SMAPP. For example, assume that you add two disk units to a Auxiliary storage pools (ASPs) 69 new ASP (ASP 2). The total capacity of the two disk units is 2,062 MB. Later, you add two more disk units to increase the capacity to 4,124 MB. For the purposes of SMAPP, the size of the ASP remains 2,062 MB until the next time you perform an IPL. This means that the storage threshold of your SMAPP receivers is lower and the system must change receivers more often. Usually, this does not have a significant impact on system performance. The system determines the capacity of every ASP when you perform an IPL. At that time, the system makes adjustments to its calculations for SMAPP size requirements. 5.3.7 Making sure that your system has enough working space When you make changes to your disk configuration, the system may need working space. This is particularly true if you plan to move disk units from one ASP to another. The system needs to move all the data from the disk unit to other disk units before you move it. There are system limits for the amount of auxiliary storage. If your system does not have sufficient interim storage, begin by cleaning up your disk storage. Many times, users keep objects on the system, such as old spooled files or documents, when these objects are no longer needed. Consider using the automatic cleanup function of Operational Assistant to free some disk space on your system. If cleaning up unnecessary objects in auxiliary storage still does not provide sufficient interim disk space, another alternative is to remove objects from your system temporarily. For example, if you plan to move a large library to a new user ASP, you can save the library and remove it from the system. You can then restore the library after you have moved disk units. Here is an example of how to accomplish this: 1. Save private authorities for the objects on your system by typing SAVSECDTA DEV (tape-device). 2. Save the object by using the appropriate SAVxxx command. For example, to save a library, use the SAVLIB command. Consider saving the object twice to two different tapes. 3. Delete the object from the system by using the appropriate DLTxxx command. For example, to delete a library, use the DLTLIB command. 4. Recalculate your disk capacity to determine whether you have made sufficient interim space available. 5. If you have enough space, perform the disk configuration operations. 6. Restore the objects that you deleted. 5.3.8 Auxiliary storage pools: Example uses The following list explains how ASPs are used to manage system performance and backup requirements: • You can create an ASP to provide dedicated resources for frequently used objects, such as journal receivers. 70 High Availability on the AS/400 System: A System Manager’s Guide • You can create an ASP to hold save files. Objects can be backed up to save files in a different ASP. It is unlikely that both the ASP that contains the object and the ASP that contains the save file will be lost. • You can create different ASPs for objects with different recovery and availability requirements. For example, you can put critical database files or documents in an ASP that has mirrored protection or device parity protection. • You can create an ASP to place infrequently used objects, such as large history files, on disk units with slower performance. • You can use ASPs to manage recovery times for access paths for critical and noncritical database files using system-managed access-path protection. 5.3.9 Auxiliary storage pools: Benefits Placing objects in user ASPs can provide several advantages, including: • Additional data protection: By separating libraries, documents, or other objects in a user ASP, you protect them from data loss when a disk unit in the system ASP or other user ASPs fails. For example, if you have a disk unit failure, and data contained on the system ASP is lost, objects contained in user ASPs are not affected and can be used to recover objects in the system ASP. Conversely, if a failure causes data that is contained in a user ASP to be lost, data in the system ASP is not affected. • Improved system performance: Using ASPs can also improve system performance. This is because the system dedicates the disk units that are associated with an ASP to the objects in that ASP. For example, suppose you are working in an extensive journaling environment. Placing libraries and objects in a user ASP can reduce contention between the journal receivers and files if they are in different ASPs. This improves journaling performance. However, placing many active journal receivers in the same user ASP is not productive. The resulting contention between writing to more than one receiver in the ASP can slow system performance. For maximum performance, place each active journal receiver in a separate user ASP. • Separation of objects with different availability and recovery requirements: You can use different disk protection techniques for different ASPs. You can also specify different target times for recovering access paths. You can assign critical or highly used objects to protected high-performance disk units. You may assign large low-usage files, like history files, to unprotected low-performance disk units. 5.3.10 Auxiliary storage pools: Costs and limitations There are some specific limitations that you may encounter when using auxiliary storage pools (ASPs): • The system cannot directly recover lost data from a disk unit media failure. This situation requires you to perform recovery operations. • Using ASPs can require additional disk devices. • Using ASPs requires you to manage the amount of data in an ASP and avoid an overflowed ASP. • You need to perform special recovery steps if an ASP overflows. • Using ASPs requires you to manage related objects. Some related objects, such as journals and journaled files, must be in the same ASP. Auxiliary storage pools (ASPs) 71 5.4 System ASP The system automatically creates the system ASP (ASP 1). This contains disk Unit 1 and all other configured disks that are not assigned to a user ASP. The system ASP contains all system objects for the OS/400 licensed program and all user objects that are not assigned to a user ASP. Note: You can have disk units that are attached to your system but are not configured and are not being used. These are called non-configured disk units. There are additional considerations that you should be aware of regarding the capacity of the system ASP and protecting your system ASP. These are explained in the following sections. 5.4.1 Capacity of the system ASP If the system ASP fills to capacity, the system ends abnormally. If this occurs, you must perform an IPL of the system and take corrective action (such as deleting objects) to prevent this from re-occurring. You can also specify a threshold that, when reached, warns the system operator of a potential shortage of space. For example, if you set the threshold value at 80 for the system ASP, the system operator message queue (QSYSOPR) and the system message queue (QSYSMSG) are notified when the system ASP is 80% full. A message is sent every hour until the threshold value is changed, or until objects are deleted or transferred out of the system ASP. If you ignore this message, the system ASP fills to capacity and the system ends abnormally. A third method for preventing the system ASP from filling to capacity is to use the QSTGLOWLMT and QSTGLOWACN system values. 5.4.2 Protecting your system ASP IBM recommends that you use device parity protection or mirrored protection on the system ASP. Using disk protection tools reduces the chance that the system ASP will lose all data. If the system ASP is lost, addressability to objects in every user ASP is also lost. You can restore the addressability by restoring the entire system or by running the Reclaim Storage (RCLSTG) command. However, the RCLSTG command cannot recover object ownership. After you run the command, the QDFTOWN user profile owns all objects found without ownership intact. You can use the Reclaim Document Library Object (RCLDLO) command procedure to recover ownership of document library objects. 5.5 User ASPs Grouping a set of disk units together and assigning that group to an auxiliary storage pool (ASP) creates a user ASP. You can configure user ASPs 2 through 16. They can contain libraries, documents, and certain types of objects. There are two types of user ASPs: • History file • Non-library user ASPs Once you have ASPs configured, you should protect them by using mirroring or device parity protection. 72 High Availability on the AS/400 System: A System Manager’s Guide 5.5.1 Library user ASPs Library user ASPs contain libraries and document library objects (DLOs). It is recommended that you use library user ASPs because the recovery steps are easier than with non-library user ASPs. You should be familiar with the following regulations regarding library user ASPs: • Do not create system or product libraries (libraries that begin with a Q or #)or folders (folders that begin with a Q) in a user ASP. Do not restore any of these libraries or folders to a user ASP. Doing so can cause unpredictable results. • Library user ASPs may contain both libraries and document library objects. The document library for a user ASP is called QDOCnnnn (here, nnnn is the number of the ASP). • Journals and files that are being journaled must be in the same ASP. Place the journal receivers in a different ASP. This protects against the loss of the files and the receivers if a disk media failure occurs. • Journaling cannot be started on an object (STRJRNPF or STRJRNAP command) if the journal (object type *JRN) and the object to be journaled are in different ASPs. • Journaling cannot be started again for a file that is saved and then restored to a different ASP that does not contain the journal. The journal and the file must be in the same ASP for journaling to be automatically started again for the file. • No database network can cross ASP boundaries. • You cannot create a file in one ASP that depends on a file in a different ASP. All based-on physical files for a logical file must be in the same ASP as the logical file. The system builds access paths only for database files in the same ASP as the based-on physical file (temporary queries are not limited). Access paths are never shared by files in different ASPs. Record formats are not shared between different ASPs. Instead, a format request is ignored and a new record format is created. • You can place an SQL collection in a user ASP. You specify the target ASP when you create the collection. • If the library user ASP does not contain any database files, set the target recovery time for the ASP to *NONE. This would be true, for example, if the library user ASP contains only libraries for journal receivers. If you set the access path recovery time to *NONE, this prevents the system from doing unnecessary work for that ASP. The Backup and Recovery Guide, SC41-5304, describes how to set access path recovery times. Non-library user ASPs Non-library user ASPs contain journals, journal receivers, and save files with libraries that are in the system ASP. If you are assigning access path recovery times for individual ASPs, you should set the target recovery time for a non-library user ASP to *NONE. A non-library user ASP cannot contain any database files and cannot, therefore, benefit from SMAPP. If you set an access path recovery time for a non-library user ASP to a value other than *NONE, the system performs extra work with no possible benefit. Backup and Recovery Guide, SC41-5304, describes how to set access path recovery times. Auxiliary storage pools (ASPs) 73 Using ASPs can require protecting user ASPs. Keep the following points in mind regarding user ASP protection: • All ASPs, including the system ASP, should have mirrored protection or consist entirely of disk units with device parity protection to ensure that the system continues to run after a disk failure in an ASP. • If a disk failure occurs in an ASP that does not have mirrored protection, the system may not continue to run. This depends on the type of disk unit and the error. • If a disk failure occurs in an ASP that has mirrored protection, the system continues to run (unless the both storage units of a mirrored have failed). • If a disk unit fails in an ASP that has device parity protection, the system continues running as long as no other disk unit in the same device parity set fails. • System limits are set for auxiliary storage. During an IPL, the system determines how much auxiliary storage is configured on the system. The total amount is the sum of the capacity of the configured units and their mirrored pairs (if any). Disk units that are not configured are not included. The amount of disk storage is compared to the maximum that is supported for a particular model. • If more than the recommended amount of auxiliary storage is configured, a message (CPI1158) is sent to the system operator’s message queue (QSYSOPR) and the QSYSMSG message queue (if it exists on the system). This message indicates that too much auxiliary storage exists on the system. This message is sent once during each IPL as long as the amount of auxiliary storage on the system is more than the maximum amount supported. 74 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 75 Chapter 6. Networking and high availability One of the major items to consider for availability is the network. When planning for a network, capacity and accessibility are addressed, just as capacity and accessibility are planned for the system itself. Your company needs a stable computing environment to fuel its business growth and to sharpen its advantage in a competitive marketplace. When major hubs or routers are down, users have difficulty accessing key business applications. The ability to carry on business, such as enrolling new members to a bank, or to respond to member inquiries, is impacted. To minimize this effect, aim for stringent metrics (for example, 99.5 percent availability of the operational systems). Ultimately, the network must be sound and recoverable to support core business applications. Employ a carefully planned comprehensive network management solution that is: • Scalable and flexible, no matter how complicated the task • Provides around-the-clock availability and reliability of applications to users, no matter where they’re located • Capable of building a solid network foundation, no matter how complex the system This chapter comments on the various network components of an HA solution and how they can affect the overall availability. 6.1 Network management Networks typically comprise a wide variety of devices, such as hubs, routers, bridges, switches, workstations, PCs, laptops, and printers. The more different components and protocols there are in a network, the more difficult it is to manage them. Problem detection and response can be at a local or remote level. Tools can help you correct problems at the source before users are affected and minimize downtime and performance problems. Management tools provide a detailed view of the overall health and behavior of your network's individual elements. These elements, such as routers, servers, end systems, data, event flow, and event traffic are useful to record. When network problems occur, use management tools to quickly identify and focus on the root cause of the error. The focal point of control is where all network information is viewed to provide a complete picture of what is happening. This console or server provides monitoring and alert actions, as well as maintenance capabilities. With proper network management, the time to respond to and resolve a network error is reduced. 76 High Availability on the AS/400 System: A System Manager’s Guide 6.2 Redundancy Availability is measured as the percentage of time that online services for a critical mass of end users can function at the end-user level during a customer's specified online window. When systems are combined into a cluster, the supporting network must be able to maintain the same level of availability as the cluster nodes. Communication providers must deliver guarantees that they will be available for, and have sufficient capacity for, all possible switch scenarios. There must be alternate network paths to enable the cluster services to manage the cluster resources. Redundant paths can prevent a cluster partition outage from occurring. Redundant network connections are illustrated in Figure 15. Figure 15. Redundant network connections As shown in Figure 15, the adapter card can be a LAN, WAN, or ATM adapter. Duplicates are installed, as well as duplicate memory and disk units. Indeed, System A serves as a duplicate system to System B, and vice versa. When a network connection between two machines that are part of the business process does not function, it does not necessarily mean the business process is in danger. If there is an alternative path and, in case of malfunctioning, the alternative path is taken automatically, monitoring and handling these early events assures a minimum breakage. 6.3 Network components Protocol, physical connection, the hardware and software needed, the backup options available, and network design are all factors to consider for a highly available network. Memory card Memory card Processor Card Memory card Memory card Adapter Card (redundant) Adapter Card (active) Processor Card System B Processor LAN, WAN, or ATM cards Mirrored disks Mirrored disks System A Processor Network Adapter Card (redundant) Adapter Card (active) Networking and high availability 77 Components involved in a network include: • Protocols: – Bisynchronous – Asynchronous – Systems Network Architecture (SNA): The predominate protocol for connecting terminals to host systems. SNA has strong support for congestion control, flow control, and traffic prioritization. It is known to provide stable support for large and complex networks. However, SNA is not capable of being routed natively across a routed network. Therefore, it must be encapsulated to flow across such a network. – Transmission Control Protocol/Internet Protocol (TCP/IP): TCP/IP is the protocol used to build the world’s largest network, the Internet. Unlike SNA, all hosts are equal in TCP/IP. The protocol can be carried natively through a routing network. TCP/IP is the de facto published standard. This allows many vendors to interoperate between different machines. Poor congestion control, flow control, traffic prioritization, and a lack of controls makes it difficult to guarantee a response time over a wide area network. – Internetwork Packet Exchange (IPX) was developed for NetWare: NetWare is a network operating system and related support services environment introduced by Novell, Inc. It is composed of several different communication protocols. Services are provided in a client/server environment in which a workstation (client) requests and receives the services that are provided by various servers on the network. Since this is usually a secondary protocol, the requirements need to be analyzed. However, they may need to be considered for transport only. – Communication lines: Communication lines support the previously described protocols. Special interfaces may be required to connect to the AS/400e system, remote controllers, personal computers, or routers. The type and speed of the line depends on the requirement of performance and response times. Considerations for such facilities include: • Leased: Private set of wires from the telephone company. These wires can be: – Point-to-point: Analog or digital – Multi-point: Phone company provides the connection of multiple remote sites through the central telephone offices • Integrated Services Digital Network (ISDN): Basic or primary access • Frame relay: An internal standard. The re-routing of traffic is the responsibility of the frame relay provider. For a fail-safe network, all endpoints should have a dial backup or ISDN connection. • Virtual Private Networks (VPN): Utilize a service provider that provides encryption and uses the Internet network for the transmission of data to the other site. VPN allows secure transfers over TCP/IP (IPSec) and multiprotocol (L2TP) networks. • Multiprotocol Switched Services: ATM or LAN • Hardware: A factor that may limit the speed in which communication lines are connected. These factors include: 78 High Availability on the AS/400 System: A System Manager’s Guide – Controllers: Manage connections for legacy terminals and personal computers with twinax support cards – Bridge – Routers: Allows encapsulation of SNA data into TCP/IP and routes it over a communication network – Frame Relay Access Devices (FRADs): Used to buffer SNA devices from a frame relay network. It also channels SNA, BSC, Asynch, and multiprotocol LAN traffic onto a single frame relay permanent virtual circuit and eliminates the need to have separate WAN links for traditional and LAN traffic. – Terminal Adapter: Used in an ISDN network to buffer the device from the physical connection – Modems: Used for analog lines – DSU and CSU: Used for digital lines – Switches: WAN switches for networks carrying high volumes of traffic at a high speed. These are networks that cope with both legacy and new emerging applications in a single network structure. The network mixes data, voice, image, and video information, and transports different traffic types over a single bandwidth channel to each point. • Software: When the network consists of LANS, routers, and communication lines, a management tool such as Operation Control/400 with Managed System Services/400, Nways Workgroup Manager, or IBM Netfinity is advised to help manage the network. From routers to remote controllers, review all network components. If there are critical requirements for remote access, you must provide alternative network paths. This can be as simple as a dialup link or as complex as a multi-path private network. 6.4 Testing and single point of failure The problem with managing a complex computing system is that anything can go wrong with any components at any time. In any development shop, applications are tested. Network, hardware, and external link recovery should be tested with the same emphasis. Just as planning for redundancy, recovery, are a must for high availability, so is testing possible scenarios. In a highly available environment, testing is particularly more critical. By employing a highly available solution, a business makes a serious commitment to exceptional levels of reliability. These levels not only rely on hardware and software, but they can only be achieved with stringent problem and change management. Identified problems must be replicated and a change must be put in place and tested before further presenting a further production risk. Testing should involve obvious things (such as hardware, network, and applications), and less obvious components (such as the business processes associated with the computer systems, facilities, data, and applications from external providers, as well as accurately documented job responsibilities). Consider your implementation test environment and your ongoing problem or change test environment when implementing a highly available solution. A Networking and high availability 79 second system or separate LPAR partition can be used for operating system and application test environments. Figure 16 illustrates a very simple HA environment. There is a two-node cluster or replication environment. A separate system is available for development that has no links to the cluster. Development cannot test changes in a clustered environment. To enable tested changes to be made to the cluster, a fourth system is added. Changes can then be tested on the development systems before moving these changes into production. Figure 16. Cluster test scenario Note: Figure 16 illustrates a basic customer setup. The routers, LANs, WANs, etc., required to simulate the real environment are not shown. Creating a separate cluster with two smaller systems meets most testing needs. However, some hardware or peripherals may not work with smaller systems. The production environment can only truly be tested with actual and careful function duplication. Testing needs to involve a planned switchover, a failover (unplanned switchover), the rejoin of the systems, and adding a system to the cluster. Be sure to re-test these scenarios after each system upgrade and any major change to any of the cluster components. A single network adapter card in a server system that works in a client/server environment is a single-point-of-failure (SPOF) for this server. Likewise, a single SCSI adapter connecting to an external storage system is a SPOF. If a complete server fails within a group of several servers, and the failed server cannot be easily and quickly replaced by another server, this server is an SPOF for the server group or cluster. The straightforward solution is that adapter cards can be made redundant by simply doubling them within a server and making sure a backup adapter becomes active if the primary one fails. System B Production System C Backup System D Development 2 System A Development 1 80 High Availability on the AS/400 System: A System Manager’s Guide CPU, power supplies, and other parts can be made redundant within a server too. This requires that the backup adapter becomes active if the primary one fails. These components can also be made redundant by requiring special parts that are not very common in the PC environment and thus quite expensive. However, in cooperation with a software agent (the HA software), two or more servers in an HA cluster can be set up to replace each other in case of a node outage. When cluster nodes are placed side-by-side, a power outage or a fire can affect both. Consequently, an entire building, a site, or even a town (earthquakes or other natural disasters) can be an SPOF. Such major disasters can be handled by simply locating the backup nodes at a certain distance from the main site. This can be very costly and IT users must be careful to evaluate their situation and decide which SPOFs to cover. General considerations for testing a high availability environment are described in Appendix F, “Cost components of a business case” on page 169. 6.5 Hardware switchover A hardware switchover may occur in case of a planned role swap or an unplanned role swap (for example, a system outage). The role swap typically involves 5250 device switching from the failed system to the backup system. Note: A hardware switchover can also be called a role swap. Figure 17 shows an example IP address assignment for two systems in a cluster preparing for a hardware switchover. Figure 17. IP address takeover between two systems in a cluster In a clustered environment, the typical tasks for enabling a hardware switchover are: 1. Quiesce the system: a. Remove users from the production system. b. Ensure all jobs in the production job queues have completed. c. Hold those job queues. Production system   1.3.22.426 1.3.22.427 1.3.22.322 1.3.22.327 Server 1 Server 2 1.3.22.120 (Inactive) IP Address takeover 1.3.22.120 (Active) Networking and high availability 81 d. Check synchronization of database transactions. e. End subsystems. 2. End high availability applications on the source machine. Make sure all journal receiver entries are complete. 3. End high availability applications on the target machine. 4. Switch between the source and target systems. 5. Switch the network to the target system. 6. Start application and transaction mirroring in reverse mode on the target system. 7. Connect users to the target system. 8. Make necessary changes and updates to the source system and start that system. 9. Switch the roles of the source and target systems again. 10.Switch the network. 11.Start mirroring in normal model. Note: Switchover capabilities are enhanced at V5R1. However, they are not covered in this Redpaper. 6.6 Network capacity and performance As business requirements have evolved, a greater dependency has been put on information technology (IT) strategies to remain competitive in the marketplace. More and more, the marketplace means an e-business environment. Network performance directly corresponds to business performance. This increased reliance on the network creates the need for high availability within the infrastructure. New and evolving applications, such as interactive white boarding, video conferences, and collaborative engineering, have significantly increased the need for bandwidth capacity. In addition to bandwidth requirements, these applications create very different traffic patterns from traditional client/server applications. Many of these new applications combine voice, video, and data traffic and have driven the convergence of these infrastructures. As this convergence occurs, the volume of time-sensitive traffic increases. Instantaneous rerouting of traffic must occur to maintain the integrity of the application. Reliability and fault tolerance are critical to maintain continuous network operations. 6.7 HSA management considerations with networking Networking management is associated with network performance, particularly in regards to HSA and Continuous Operations support and in a mission-critical application environment. After the network is set up, monitor it to be sure it runs at maximum efficiency. 82 High Availability on the AS/400 System: A System Manager’s Guide The primary concerns of continuity are associated with a server’s communications pointers to other servers and their respective clients. When networks are consolidated, a common complication is the duplication of network addresses for existing devices. Each of these potential problem areas are readily identified with proper network management tools. 6.7.1 Network support and considerations with a HAV application Many AS/400 application environments take advantage of several OS/400 communication facilities. The primary concern with these facilities is not so much involved with day-to-day operation activities but is more likely to be involved with the role swap (redirection) of the database function from the production system to a backup system. Moving executing jobs from one physical AS/400 system unit to another requires that pointers managed for these systems are adjusted in a quick and efficient manner. Several communications facilities are involved: • OptiConnect • TCP/IP • SNA Each facility utilizes certain name attributes or address pointer conventions to indicate another AS/400 system’s presence in a network. Implementing continuous operations support requires that the name attributes or pointer conventions be changed to reflect that the current state of business is operating on another AS/400 system. Scenario with Tivoli network management in action This section illustrates a fictitious example of what a network management tool can do. It is noon, the peak traffic time for an international travel agency Web site. Thousands of customers worldwide access the site to book airline, hotel, and car reservations. Without warning, the Web site crashes, literally shutting down the agency’s lucrative e-business trade and closing the doors to customers. With each passing minute, tension mounts because the company stands to lose millions of dollars in revenue. Tivoli instantly sifts through enormous amounts of data for the root cause of the problem. Moments later, it pinpoints the source of the problem: a failing router. Immediately, the company’s e-business traffic is automatically rerouted through an auxiliary router. Meanwhile, system administrators recycle the router and resolve the problem. Downtime is minimized, customers remain satisfied, and a near-disaster is averted. The agency is back online and back in business. This example illustrates how a network management tool like Tivoli improves system availability. 6.8 Bus level interconnection Bus level interconnetion consists of hardware and software support between two AS/400 systems at the application level. This path provides a high performance bandwidth for client systems requiring local database access time. By using this feature of the AS/400 system, many customers have been able to use multiple systems to work as a single application image to clients. The SAP R/3 Networking and high availability 83 application, with its three-tier configuration and OptiConnect, is an example of how bus level interconnection is used for this purpose. Figure 18 depicts a three-tiered configuration showing the bus level interconnection arrangement with two bus owners (referred to as a hub). This arrangement provides redundancy for the shared bus path. Figure 18. Three-tiered configuration: OptiConnect assignments In a shared bus redundancy arrangement, it is not necessary for the application servers to be the bus owners because there is no technical reason why the database servers can’t assume that role. The only rule you must adhere to with this arrangement is that either application or database servers must perform this role as a pair. You can’t have one application server and one database server assume the bus owner role in the redundancy arrangement. If you choose a primary application and a database server for this assignment, there may come a time when both those systems must go through some sort of dedicated process or a power sequence. Application Server SAP Client Host: IP@2 SAP Client Host: IP@2 Backup Application Server Database Server Backup Database Server Client LAN/WAN IP@2 IP@3 IP@2 (inactive) Bus Owner 1 Sysnam=APPL1 Bus Owner 2 Sysnam=APPL2 84 High Availability on the AS/400 System: A System Manager’s Guide This operation can not be allowed to occur because any DDM and SQL procedures through OptiConnect (for example) cease and would, therefore, affect availability of the backup systems when continuous operations is required for the application user. When the bus owner goes to a restricted state or power sequence, all OptiConnect traffic between remote systems stops. Note: HSL OptiConnect is introduced at V5R1. However, it is not covered in this Redpaper. 6.8.1 Bus level interconnection and a high availability solution Besides providing high-speed data throughput, bus level interconnection allows OMS/400 to transport data to the backup system efficiently and quickly so that exposure to data loss is very minimal. One shared bus between an application server and two database server systems can accommodate the data requests from the application server. This is done on behalf of the clients to the primary database server while concurrently supporting the data mirroring path between the primary database server and backup database server. However, better resiliency in this example would be provided if both database servers each supported a shared bus to the application server providing a dual bus path (also referred to as shared bus redundancy). OMS/400 supports the use of OptiConnect for data replication to the backup system. When configuring your application database library in OMS/400, the specification for Opticonnect requires only the System Name from the OS/400 Network Attribute System Name. All pointers to the remote system and the apply processes are built by the configuration process. 6.8.2 TCP/IP This protocol is also supported by OMS/400. When OptiConnect cannot be used, or is not in operation, this path can also serve as a route for data replication. Primarily, this path is used by SAM/400 for verification of the primary database server’s operation and signals the redirection of the primary database system’s IP address to the backup database server system in the event of an unplanned outage. In the application environment, it is the path that the client uses for application requests from the application server and the path used between the application server and database server to access stream files prevalent in this environment (also called the access network). Networking and high availability 85 Figure 19. Three-tiered configuration: IP assignments Figure 19 shows the suggested ethernet LANs for supporting client traffic through the access network (Client LAN/WAN) path. This occurs while operations, development, and SAM/400 use the AS/400 LAN for additional traffic for testing the availability of the production system. Note: In relationship to Figure 19, the OptiConnect arrangement has been removed from the diagram to keep it uncluttered. However, you can still assume that all four AS/400 systems are connected in that manner. The LANs can be bridged for network redundancy. However, security would have to be implemented to keep client traffic out of the AS/400 LAN because of direct accessibility to the database servers. It should be noted that SAM/400 can use all of these protocols in combination to test the validity of an unplanned outage. Figure 19 shows several IP address assignments. This is designed to keep the client interface consistent with one address/host name. In the event of a role swap with an application server, the client operator does not need to be concerned with configuration tasks and does not need to use a second icon on their desktop for access to a backup system. Application Server SAP Client Host: IP@2 SAP Client Host: IP@2 Backup Application Server Database Server Backup Database Server Client LAN/WAN AS/400 LAN IP@2 IP@3 IP@2 (inactive) IP@4 IP@6 (no name) IP@5 IP@4 (inactive) IP@7 IP@8 86 High Availability on the AS/400 System: A System Manager’s Guide For the application server that must use a backup database server, it also uses the same IP address as that of the primary database server. The role swap procedures outlined in SAM/400 manages the ending and starting of different interfaces. This option facilitates the identification of each system in its new role throughout the LAN. Note that the respective backup systems have inactive IP addresses (as reflected in the interface panel of the TCP configuration menu). These addresses are started during the role swap procedure when the backup system becomes the production environment. The additional IP address for the primary database server is not generally assigned a host name which is the case with other addresses. This ensures that the address is not being used by the clients through the access network. The main purpose is for the reversed role the primary database server plays when it has stopped operating (for planned or unplanned outages) and it must now be synchronized with the backup database server. Naturally, for continuous operations, the business moves to the backup database server and the database located there begins processing all transactions while the primary database server system is being attended to. After awhile, the primary database server files become aged and must be refreshed with the current state of business before returning to operation. One method is to save the files from the backup system to the production system. However, this method would impact the user’s availability while they are still using the backup database server. OMS/400 is designed to reverse the roles of the primary and backup system to reversed target and reversed source respectively. That means the backup system can capture and send the database changes back to the production system so that it can catch up and be current with the business operation. The function that the additional IP address plays here is that its allows the system to connect to the AS/400 LAN (as shown in Figure 19) without having to use its normal interface (at this time, it is inactive) when TCP services are required. All this occurs while the business is still using the backup system to run its applications. Once the systems are equal, the user can then return them to their original roles by scheduling a planned role swap. Usually, you would select a time of day when activity is low or when work shifts are changing. © Copyright IBM Corp. 2001 87 Chapter 7. OS/400: Built-in availability functions This chapter reviews existing OS/400 functions, specifically those that enhance system availability, databases, and applications. These functions form the foundation of the AS/400 system. Note: This chapter only summarizes these functions. The System Administrator’s Companion to AS/400 Availability and Recovery, SG24-2161, provides more details on OS/400 functions designed for availability. This chapter also outlines clustering and LPAR. Introduced in OS/400 V4R4, these features provide system redundancy and expand availability and replication options. 7.1 Basic OS/400 functions Some availability functions for the AS/400 system were architected into the initial design (some were carried over from the preceding System/38). Some of these functions include: • Auxiliary storage pools (ASPs) • Journaling • Commitment control These functions were delivered when the system main storage and processing power were small in comparison to the resource needs of the applications that enabled them. Therefore, many applications were developed without these functions. As these applications grew in functionality and user base, the task of enabling the functions grew quickly. Now, newer functions are available to provide nearly 100% availability. These new functions are founded on the historic functions and are not enabled by many applications. Therefore, application developers must revisit their applications and enable these historic functions to enable the new functions. ASPs are discussed in Chapter 5, “Auxiliary storage pools (ASPs)” on page 63. Journaling and commitment control are discussed in this section. 7.1.1 Journaling Journals define which files and access paths to protect with journal management. This is referred to as journaling a file or an access path. A journal receiver contains journal entries that the system adds when events occur that are journaled, such as changes to database files. This process is shown in Figure 20 on page 88. 88 High Availability on the AS/400 System: A System Manager’s Guide Figure 20. The journaling process Journaling is designed to prevent transactions from being lost if your system ends abnormally or has to be recovered. Journal management can also assist in recovery after user errors. Journaling provides the ability to roll back from an error to a stage prior to when the error occurred if both the before and after images are journaled. To recover, the restore must be done in the proper order: 1. Restore your backup from tape. 2. Apply journaled changes. We recommend that you keep a record of which files are journaled. Use journal management to recover changes to database files that have occurred since your last complete save. A thorough discussion of journaling is beyond the scope of this chapter. For more information about journaling, refer to OS/400 Backup and Recovery, SC41-5304. 7.1.2 Journal receivers with a high availability business partner solution Business partner high availability solutions incorporate the use of journaling. A description of journal receivers with OMS/400 and ODS/400 is explained in this section. Journal receivers are used by OMS/400 and ODS/400. OMS/400 transmits the image of database records recorded there for use by the apply jobs on the target database server. ODS/400, on the other hand, uses the entries recorded in the OS/400 audit journal (QAUDJRN) to tell it what object operations must be performed on the target AS/400 system. The issue with journal receivers is that, FILEA FILEB FILEC Receiver #1 Journal Receiver #3 Receiver #2 Journal CHANGES DELETES A D D S OS/400: Built-in availability functions 89 when transactions occur in the applications, and objects are manipulated, the resulting entries placed in the receivers make them grow in size until they reach the maximum threshold of 1.9 GB. Reaching the maximum threshold must be avoided because the business applications and system functions that use the journaled objects cease operation. They will not resume operation until the filled receiver has been detached from the journal and a new receiver is created and attached. Typically, the AS/400 user can take advantage of the system’s ability to manage the growth and change of receivers while the applications are running. Or, they may elect to write their own management software. However, either solution overlooks one important aspect: synchronization with the reader function that transmits the journal entry to a backup system (OMS/400) or performs an object function (ODS/400) on behalf of the backup system. If, for any reason, the production system cannot communicate the data and object changes to the backup system, no receiver can be deleted until replication resumes and all journal entries have been processed by the high availability application. Therefore, if the AS/400 user elects one of the first two options to manage receivers, they could inadvertently delete those receivers before their contents were interrogated by high availability software for replication and synchronization purposes. Placing journal receivers in user ASPs minimizes the impact journaling may place on some application environments, especially if it has never been used for a given application environment. Since the write I/O changes from asynchronous (with regards to the program execution) to synchronous (where the program producing the write activity must actually wait for the record to be written to the journal receiver) latency is introduced and program execution may increase elapsed time. This result can be seen in batch applications that produce many significant write operations to a database being journaled. Using user ASPs only for the journal receivers allows for quick responses to the program from the DASD subsystems. The only objects being used in that ASP are the journal receivers and the most typical operation is write I/O. Therefore, the arms usually are positioned directly over the cylinders for the next contiguous space allocation when a journal receiver extent is written to the disk. Obviously, following this recommendation places more responsibility on the user for managing receivers and the DASD space they utilize. The associated journal must remain in the same ASP as the database it is recording. To implement the user ASP solution, create a new library in the user ASP and, during the next In such a case, Vision Suite recommends the use of its Journal Manager. This Vision Suite feature changes journal receivers based on a policy established by the user. It also coordinates between the reader jobs for replication and the user’s requirements to free auxiliary storage space and save receivers offline. Once it has been implemented, the user’s interface for this requirement simply observes the Work with Disk Status (WRKDSKSTS) and monitors the user auxiliary storage pool (ASP) utilization threshold. Note 90 High Availability on the AS/400 System: A System Manager’s Guide Change Journal (CHGJRN) operation, specify the new receiver to be qualified to that library. 7.2 Commitment control Commitment control is an extension of journal management. It allows you to define and process a group of changes to resources, such as database files or tables, as a logical unit of work (LUW). Logically, to the user, the commitment control group appears as a single change. To the programmer, the group appears as a single transaction. Since a single transaction may update more than one database file, when the system is in a network, a single transaction may update files on more than one system. Commitment control helps ensure that all changes within the transaction are completed for all affected files. If processing is interrupted before the transaction is completed, all changes within the transaction are removed. Without commitment control, recovering data for a complex program requires detailed application and program knowledge. Interrupted programs cannot easily be re-started. To restore the data up to the last completed transaction, typically a user program or utility, such as a Data File Utility (DFU), is required to reverse incomplete database updates. This is a manual effort, it can be tedious, and it is prone to user error. Commitment control ensures that either the entire group of individual changes occur on all participating systems, or that none of the changes occur. It can assist you with keeping your database files synchronized. 7.2.1 Save-while-active with commitment control Using the save-while-active function while commitment control processing is active requires additional consideration. When an object is updated under commitment control during the checkpoint processing phase of a save-while-active request, the system ensures that the object is saved to the media at a commitment boundary. All objects that have reached a checkpoint together are saved to the media at the same common commitment boundary. It is important to make sure that all performance considerations have been correctly implemented in this situation. Otherwise, the system may never be able to reach a commitment boundary. It may not be able to obtain a checkpoint image of the objects to be saved. Procedures need to be specified to ensure that all of the objects reach a checkpoint together and all of the objects are saved in a consistent state in relationship to each other. If the checkpoint versions of the objects are not at an application boundary, user-written recovery procedures may still be necessary to bring the objects to an application boundary. Refer to OS/400 Backup and Recovery, SC41-5304 for details on coding for commitment control and save-while-active. 7.3 System Managed Access Path Protection (SMAPP) An access path describes the order in which records in a database files are processed. A file can have multiple access paths if different programs need to see the records in different sequences. If your system ends abnormally when access OS/400: Built-in availability functions 91 paths are in use, the system may have to rebuild the access paths before you can use the files again. This is a time-consuming process. To perform an IPL on a large and busy AS/400 system that has ended can take many hours. You can use journal management to record changes to access paths. This greatly reduces the amount of time it takes the system to perform an IPL after it ends abnormally. Access path protection provides the following benefits: • Avoids rebuilding access paths after most abnormal system ends • Manages the required environment and makes adjustments as the system changes if SMAPP is active • Successful even if main storage cannot be copied to storage Unit 1 of the system ASP during an abnormal system end • Generally faster and more dependable than forcing access paths to auxiliary storage for the files (with the FRCACCPTH parameter) The disadvantages of access path protection include: • Increases auxiliary storage requirements • May have an impact on performance because of an increase in the activity of the disks and processing unit • Requires file and application knowledge for recovery. There is a small additional processor overhead if *RMVINTENT is specified for the RCVSIZOPT parameter for user-created journals. However, the increase in storage requirements for access path journaling is reduced by using *RMVINTENT. • Normally requires a significant increase in the storage requirements for journaling files. The increase with SMAPP is less than when access paths are explicitly journaled. Two methods of access-path protection are available: • System management access-path protection (SMAPP) • Explicit journaling of access paths An access path (view) describes the order in which records in a database file are processed. A file can have multiple access paths if different programs need to see the records in different sequences. If your system ends abnormally when access paths are in use, the system may have to rebuild the access paths before you can use the files again. This can be a time-consuming process, since an IPL on a large busy server that had ended abnormally may take many hours. Two methods of access-path protection are available: • OS/400 System Managed Access Path Protection (SMAPP) • Explicit journaling of access paths 7.4 Journal management You can use journal management to recover the changes to database files or other objects that have occurred since your last complete save. Use a journal to define what files and access paths you want to protect with journal management. This is often referred to as journaling a file or an access path. A journal receiver 92 High Availability on the AS/400 System: A System Manager’s Guide contains the entries (called journal entries) that the system adds when events occur that are journaled, such as changes to database files, changes to other journaled objects, or security-relevant events. Use the remote journal function to set up journals and journal receivers on a remote AS/400 system. These journals and journal receivers are associated with journals and journal receivers on the source system. The remote journal function allows you to replicate journal entries from the source system to the remote system. The main purpose of journal management is to assist in recovery. You can also use the information that is stored in journal receivers for other purposes, such as: • An audit trail of activity that occurs for database files or other objects on the system • Assistance in testing application programs. You can use journal entries to see the changes that were made by a particular program. Figure 21 shows the steps involved for journaling. Figure 21. The Journaling Process 7.4.1 Journal management: Benefits Benefits of journal management can include: • A reduction in the frequency and amount of data saved • Improved ability and speed of recovery from a known point to the failure point • Provides file synchronization if the system ends abnormally 7.4.2 Journal management: Costs and limitations Disadvantages of journal management include: • An increase in auxiliary storage requirements • Can have an impact on performance because of an increase in the activity of disks and the processing unit MEMORY 1 6 2 4 Journal Receiver Synch Point 6 3 5 ASP1 ASP2 Before image After image OS/400: Built-in availability functions 93 • Requires file and application knowledge for recovery Refer to the OS/400 Backup and Recovery, SC41-5304 for further information. 7.5 Logical Partition (LPAR) support In an n-way symmetric multi-processing iSeries or AS/400e server, logical partitions allow you to run multiple independent OS/400 instances or partitions, each with their own processors, memory, and disks. Note: With OS/400 V5R1, a single processor can be sliced for sharing across partitions. You can run a cluster environment on a single system image. With logical partitioning, you can address multiple system requirements in a single machine to achieve server consolidation, business unit consolidation, and mixed production and test environments. Figure 22 shows an LPAR configuration with resources shared across partitions. Figure 22. Example LPAR configuration Each logical partition represents a division of resources in your AS/400e system. Each partition is logical because the division of resources is virtual rather than physical. The primary resources in your system are its processors, memory (main storage), I/O buses, and IOPs. An LPAR solution does not offer a true failover capability for all partitions. If the primary partition fails, all other partitions also fail. If there are multiple secondary partitions backing each other up, they have the capability to failover between partitions. These secondary partitions are nodes and are a cluster solution. However, they are not a separate server implementation. LPAR cannot provide the same level of availability as two or more node cluster solutions. 830 4-way 2 GB memory A 2-way 1 GB B 2-way 1 GB 94 High Availability on the AS/400 System: A System Manager’s Guide See 4.12, “LPAR hardware perspective” on page 58, for discussion of LPAR from a hardware perspective. 7.6 Cluster support and OS/400 The ultimate availability solution consists of clustered systems. OS/400 V4R4 introduced clustering support. This support provides a common architected interface for application developers, iSeries software providers, and high availability business partners to use in-building high availability solutions for the iSeries and AS/400 server. The architecture is built around a framework and is the foundation for building continuous availability solutions for both the iSeries and AS/400 servers. Clustering provides: • Tools to create and manage clusters, the ability to detect a failure within a cluster, and switchover and failover mechanisms to move work between cluster nodes for planned or unplanned outages • A common method for setting up object replication for nodes within a cluster (this includes the data and program objects necessary to run applications that are cluster-enabled) • Mechanisms to automatically switch applications and users from a primary to a backup node within a cluster for planned or unplanned outages Clustering involves a set of system APIs, system services, and exit programs as shown in Figure 23. Data replication services and the cluster management interface are provided by IBM HABPs. How clustering works, and planning for clusters, is described in AS/400 Clusters, A Guide to Achieving Higher Availability, SG24-5194. Cluster Management Provided by HABPs Data Resiliency Replication technology provided by HABPs Cluster Resource Services Base OS/400 cluster functions from IBM APIs Application Resiliency High availability cluster enabled applications Figure 23. Cluster overview © Copyright IBM Corp. 2001 95 Chapter 8. Performance Performance is an availability issue from several view points. First and foremost, to the end user, poor performance can be significantly more frustrating than a system outage. Poor performance can even result in lost sales. An example would be a customer calling to check the availability or price of an item. If the response of the service representative (or a Web-enabled application) is too long, the customer looks elsewhere. Poor performance can be caused by any number of factors, which can be any computing component between the end-user incoming request and the delivery of the response. The timing is affected by the performance of the communication links, routers, hubs, disk arms (service time), memory, and the CPU. Service agreements can hold both parties accountable, with the parties being the service provider (you the business), and the receiver or requestor of the service (the customer). Do not delay the planning for performance for your high availability environment until after implementing the system and applications. Plan for it prior to installation. Set your expectations and guidelines accordingly. It is easy to suggest that, by implementing a backup machine, the spare capacity on the backup can give extra cycles to a sluggish application. This is very rarely the case. Investigating the proposed performance of the new environment should reap dividends during the implementation phase. This section discusses the implication of performance on the various levels of availability. Note: Performance ratings of save and restore hardware and software options can be found in the AS/400 Performance Capabilities Reference. This can be accessed online at: http://www.ibm.com/eserver/iseries/library 8.1 Foundations for good performance This section briefly describes the fundamental elements of good performance on the AS/400e servers. 8.1.1 Symmetric multiprocessing (SMP) In the AS/400e world, SMP has multiple meanings. First and foremost, it is a hardware deployment capability. iSeries processors and some AS/400e processors can be purchased in 2-way, 4-way, 8-way, 12-way, 18-way, and 24-way configurations. In the industry, all the n-way processor configurations are referred to as SMP processor systems. Due to the architecture of the AS/400 server and OS/400, applications and utilities are able to take advantage of the SMP models without overt programming efforts. However, it is possible to obtain even higher levels of throughput by redesigning batch processes to take advantage of multiple processors. Another use of the term SMP in the AS/400e world refers to a feature of the operating system called DB2 Symmetric Multiprocessing for AS/400. This feature enables a dynamic build of access paths or views for queries (including 96 High Availability on the AS/400 System: A System Manager’s Guide OPNQRYF and the query manager) utilizing parallel I/O and parallel processing across all available processors. Plan carefully for the use of this feature because it can significantly increase overall CPU utilization in addition to increased physical I/O operations. To learn more about the DB2 Symmetric Multiprocessing for AS/400 feature, refer to the iSeries Handbook, GA19-5486 and the Performance Capabilities Reference, which can be accessed online at: http://publib.boulder.ibm.com/pubs/pdfs/as400/V4R5PDF/AS4PPCP3.PDF 8.1.2 Interactive jobs Just as you would not implement a major application change without analysis of the potential performance impact, you need to determine the potential impact of implementing high availability on your interactive workload. Ask the following questions: • What effect will the proposed implementation have on interactive performance? • If journaling is to be activated, what will its impact be? • Have you decided to rewrite your applications to ensure data integrity by utilizing commitment control? What will this performance impact be? The answers to these questions must be analyzed and fed back to the business plan and incorporated into your service level agreements. 8.1.3 Batch jobs Batch jobs are another key area for high availability. This is one place that backup machines may have a positive effect on the performance of the primary system. You may be able to redirect work from your primary system to the backup system. This is most feasible for read only work. Other types of batch jobs could be very difficult to alter to take advantage of a second system and a major re-write may be necessary. If you choose to utilize your backup system for read only batch work, make sure that you understand the impact of these jobs on the high availability business partner apply processes. If the work you run on the backup system interferes in any way with these apply processes, you may reduce your ability to switchover or failover in a timely manner. You need to consider the impact of activating journaling on your batch run times and explore the possibility of incorporating commitment control to improve run time in a journaled environment. This is discussed in more detail in 8.2.3, “Application considerations and techniques of journaling” on page 99. 8.1.4 Database Consider the types of database networks utilized by your application when understanding performance. Do you have multiple database networks, a single database network, multiple database networks across multiple servers, or a single database network across multiple servers? Performance 97 As stated, the major performance impact for a database is the start of journaling. Each database operation (except read operations) involves journal management. This adds a physical I/O and code path to each operation. 8.2 Journaling: Adaptive bundling Journaling’s guarantee of recover ability is implemented with extra I/O and CPU cycles. Since V4R2 of OS/400, the technique of adaptive bundling has been used to reduce the impact of these extra I/O operations. This means that journal writes are often grouped together for multiple jobs, in addition to commit cycles. Refer to Figure 24 for a simple illustration. Figure 24. Adaptive bundling Unless you have taken specific actions, a single batch job can minimally take advantage of adaptive bundling. A high penalty is paid in extra I/O with every record update performed. It is normal to see an increase in both I/O and CPU utilization after turning on journaling. Even on a well-tuned system, the CPU utilization increase can be as high as 30%. This increase in utilization increases response time. An even higher degradation is common for single threaded batch jobs that do not run commitment control. Journaling increases the number of asynchronous writes. The effect of these asynchronous writes is shown on the Transition Report of Performance Tools. Modules QDBPUT, QDBGETKY, and QDBGETSQ show evidence of this asynchronous I/O request. Joe1 Job 1 Job 2 Average Bundle Size Comm Line Memory 98 High Availability on the AS/400 System: A System Manager’s Guide To reduce the impact of journaling: • Group inserts under a commit cycle • Group inserts • Split a batch job into several jobs and run them in parallel 8.2.1 Setting up the optimal hardware environment for journaling Building a sound hardware environment for your journaled environment can minimize the impact of journaling. Start by creating a user auxiliary storage pool (ASP) that utilizes mirrored protection. Depending on the amount of storage required for your journal receivers, and the release of OS/400 you are running, you can allocate between 6 and 200 total arms to this user ASP. These disk arms should have dedicated IOPs with at least .5 MB of write cache per arm. Regardless, these arms should be the fastest available on your system. Note: The total stated number of arms are not “operational” numbers after turning on mirroring. If you are running a release of OS/400 prior to V4R5, the maximum number of disk arms the system can efficiently use for parallel I/O operations for journaling is 30. In V4R5, if you utilize the *MAXOPT parameter on the receiver size (RCVSIZOPT) keyword, the number of used disk arms increases to 200. 8.2.2 Setting up your journals and journal receivers Refer to Section 6.2 “Planning and Setting Up Journaling” in the Backup and Recovery Guide, SC41-5304, for detailed information. Also keep the following points in mind: • The journal and journal receiver objects should not be in the same library as the files they are to journal. • The journal object (*JRN) should not reside in the user ASP of the journal receiver (*JRNRCV) objects. • Isolate journal receiver writes from system managed access path protection (SMAPP) writes by specifying RCVSIZOPT(*RMVINTENT) on the CRTJRN and CHGJRN commands. In addition to isolating the SMAPP I/O operations to arms dedicated for that activity, your journal receivers will not fill as quickly. The system uses two-thirds of allocated ASP arms for JRNRCV objects and the remaining one-third for SMAPP entries. • Suppress open and close journal entries by utilizing OMTJRNE(*OPNCLO) on the STRJRNPF command. • Use system managed receivers by specifying MNGRCV(*SYSTEM) on the CRTJRN and CHGJRN commands to enable better system performance during the change of journal receivers. You can ensure that your business partner package maintains control over the actual changing of journal receivers by specifying a threshold on your journal receivers that is larger than the size specified in the partner package. The MNGRCV(*SYSTEM) requires the parameter THRESHOLD be specified on the CRTJRNRCV command. Performance 99 8.2.2.1 Determining the number of journals and receivers Generally speaking, you always have multiple journal and journal receivers. Some strategies for determining the number of journals and journal receivers you have include: • By application: To simplify recovery, files that are used together in the same application should be assigned to the same journal. In particular, all the physical files underlying a logical file should be assigned to the same journal. Starting in V3R1, all files opened under the same commitment definition within a job do not need to be journaled to the same journal. If your major applications have completely separate files and backup schedules, separate journals for the applications may simplify operating procedures and recovery. • By security: If the security of certain files requires that you exclude their backup and recovery procedures from the procedures for other files, assign them to a separate journal, if possible. • By function: If you journal different files for different reasons, such as recovery, auditing, or transferring transactions to another system, you may want to separate these functions into separate journals. Remember, a physical file can be assigned to only one journal. If you have user ASPs with libraries (known as a library user ASP), all files assigned to a journal must be in the same user ASP as the journal. The journal receiver may be in a different ASP. If you place a journal in a user ASP without libraries (non-library user ASP), files being journaled must be in the system ASP. The journal receiver may be in either the system ASP or a non-library user ASP with the journal. See the section titled “Should Your Journal Receivers Be in a User ASP?” in the Backup and Recovery Guide, SC41-5304, for more information about the types of ASPs and restrictions. Remember to consult the Backup and Recovery Guide, SC41-5304 for restore or recovery considerations when setting up your environment. Even though you set up this environment to minimize the need to ever fully restore your system, you may have to partially restore within your own environment or fully restore if you take advantage of the Rochester Customer Benchmark Center or a disaster/recovery center. 8.2.3 Application considerations and techniques of journaling Database options that have an impact on journaling and system performance are: • The force-write ratio (FRCRATIO) parameter for physical files that are journaled. This allows the system to manage when to write records for the physical file to disk because, in effect, the journal receiver has a force-write ratio of one. • Record blocking when a program processes a journaled file sequentially (SEQONLY(*YES)). When you add or insert records to the file, the records are not written to the journal receiver until the block is filled. You can specify record blocking with the Override with Database File (OVRDBF) command or in some high-level language programs. This is a standard and good performance practice that significantly helps the performance of journaling too. 100 High Availability on the AS/400 System: A System Manager’s Guide • Use OMTJRNE(*OPNCLO)) to reduce the number of journal entries. If you choose to omit open journal entries and close journal entries, note the following considerations: – You cannot use the journal to audit who has accessed the file for read only purposes. – You cannot apply or remove journal changes to open boundaries and close boundaries using the TOJOBO and TOJOBC parameters. – Another way to reduce the number of journal entries for the file is to use shared open data paths. This is generally a good performance recommendation regardless of journaling activity. • Utilize the Batch Journal Caching PRPQ. This offering: – Forces journal entries to be cached in memory for most efficient disk writes – Is designed to reduce journaling's impact on batch jobs – Is selectively enabled Additional information about the Batch Journal Caching PRPQ can be found in Appendix C, “Batch Journal Caching for AS/400 boosts performance” on page 153. 8.3 Estimating the impact of journaling To understand the impact of journaling on the capacity of a system, consider the processes involved. Additional overhead is involved for disk and CPU activity and additional storage is required in preparation for potential recovery. 8.3.1 Additional disk activity Consider that each row updated, added, or deleted has either one or two journal entries. While this is an asynchronous I/O operation, your disk arm response time can increase. This causes degradation to your production workload. Under certain circumstances, these asynchronous I/O operations become synchronous I/O operations and cause your application to wait for them to complete before they can continue. 8.3.2 Additional CPU Each update, add, or delete operation utilizes additional CPU seconds to complete. The ratio of CPU per logical I/O is a key factor in determining the additional CPU required for journaling. 8.3.3 Size of your journal auxiliary storage pool (ASP) Depending on how accurate you want your estimate of space requirements to be, follow one of the two methodologies explained in this section to estimate your space requirements. 8.3.3.1 Using weighted average record length Perform the following steps to use a weighted average record length: 1. Determine the average number of entries per time period (number of days or hours) worth of receiver entries you want or need available. Performance 101 To take advantage of the journal’s ability to protect your data and to provide an audit trail, you may want to keep more than a few hours worth of receiver entries for transmission to a secondary system. Once the data is written to the disk, the only expense involved beyond the disk space consumed is the cycles required to retrieve the entries for further analysis. 2. Determine the weighted average record length. Add 155 to this weighted average record length. 3. Multiply the results from step 2 by the results from step 1 and divide by 1,024 to determine the KB. Or, divide by 1,048,576 to determine the MB required. 8.3.3.2 Using actual changes logged by file management The file description contains information useful for calculating storage usage. To utilize this information: 1. Execute a CL program to capture FD information. 2. Execute an RPG program to translate the date to a table for further calculations. 3. Rerun CL in as you did in step 1. 4. Execute an RPG program to add a second set of FD information to a table. 5. Calculate requirements based on information in the file. 8.4 Switchover and failover Highly available systems involve clustering. When a production system is switched over to the backup system, either for a planned or an unplanned outage, the time required to make this switchover is critical. To reduce the time involved in this switchover process, consider the networks and performance as explained in the following section. Networks and performance The performance of a communications network provides acceptable (or non-acceptable) response time for the end users. Response time provides the perception for the end user of the reliability and availability of the system. In general, to improve performance: • Avoid multiple layers of communications. • Avoid communication servers (such as Microsoft SNA server, IBM Communication Server, or NetWare SNA server). • Use Client Access/400 or IBM eNetwork Personal Communications for AS/400 where possible. • Use a native protocol instead of ANYNET. The Best/1 licensed program, a component of 5769-PT1, can capture information on a communication line and predict utilization and response times. 102 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 103 Part 3. AS/400 high availability solutions Combining the features of OS/400 system and network hardware with AS/400e high availability software produced by IBM business partners is an important method for improving a single systems availability. Part III discusses these components and it also explores the considerations for writing applications to support a highly available environment. 104 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 105 Chapter 9. High availability solutions from IBM The foundation of all high availability functions is OS/400. The AS/400 development and manufacturing teams continue to improve the AS/400 system for feature, function, and reliability options with each release of OS/400. In particular, the OS/400 remote journal feature enhances high availability solutions by enabling functions below the machine interface (MI) level. Note: Prior to OS/400 V4R2, remote journal functions were coded into application programs. APIs and commands are available in V4R2 and V4R3 respectively. Cluster solutions connect multiple AS/400 systems together using various interconnect fabrics, including high-speed optical fiber. Clustered AS/400 systems offer a solution that can deliver up to 99.99% system availability. For planned or unplanned outages, clustering and system mirroring offer the most effective solution. IBM business partners that provide high systems availability tools complement IBM availability tools with replication, clustering, and system mirroring solutions. The IBM (application) contribution to AS/400 High Availability Solutions includes the IBM DataPropagator Relational Capture and Apply for AS/400 product. From this point on, this product is referred to as DataPropagator/400. This chapter describes the IBM package and its benefits as a minimal High Availability Solution. 9.1 IBM DataPropogator/400 The IBM solution that fulfills some of the requirements of an HSA solution is DataPropagator/400. Note: The DataPropagator/400 product was not designed as a high availability solution. In some cases, it can cover the needs for data availability, as discussed in this chapter. DataPropagator/400 is a state-of-the-art data replication tool. Data replication is necessary when: • Supplying consistent real time reference information across an enterprise • Bringing real time information closer to the business units that require access to insulate users from failures elsewhere on the network • Reducing network traffic or the reliance on a central system • On-demand access disrupts production or response • Migrating systems and designing a transition plan to move the data while keeping the systems in sync • Deploying a data warehouse with an automated movement of data • Current disaster plan strategies do not adequately account for site-failure recovery DataPropagator/400 is not a total High Availability Solution because it only replicates databases. It does not replicate all of the objects that must be mirrored for a true High Availability Solution in a dynamic environment. 106 High Availability on the AS/400 System: A System Manager’s Guide However, consider DataPropagator/400 for availability functions in a stable environment where the following criteria can be met: • Only the database changes during normal production on the AS/400 system. • Such objects as user profiles, authorities, and other non-database objects are saved regularly on the source system and restored on the target system when changed. In other words, in a stable environment, where only the database changes, replicating the database to a backup system and transferring users manually to this system may be a sufficient availability and recovery plan (Figure 25). Figure 25. Usage of DataPropagator/400 9.1.1 DataPropagator/400 description IBM DataPropagator/400 automatically copies data within and between IBM DB2 platforms to make data available on the system when it is needed. The IBM DataJoiner product can be used in addition to the DataPropagator/400 product to provide replication to several non-IBM databases. Immediate access to current and consistent data reduces the time required for analysis and decision making. DataPropagator/400 allows the user to update copied data, maintain historical change information, and control copy impact on system resources. Copying may involve transferring the entire contents of a user table (a full refresh) or only the changes made since the last copy (an update). The user can also copy a subset of a table by selecting the columns they want to copy. Making copies of database data (snapshots) solves the problem of remote data access and availability. Copied data requires varying levels of synchronization with production data, depending on how the data is used. Copying data may even be desired within the same database. If excessive contention occurs for data access in the master database, copying the data offloads some of the burden from the master database. Why replication? Reengineer business processes Improve decision making Speed application deployment Increase online throughput Improve system availability Support audit requirements Support data warehousing Support mobility AS/400 OS/2 Windows NT AIX HP-UX Solaris System DB2 Oracle Sybase Informix Microsoft SQL server Ordering/billing IMS DB2 DB2 Manufacturing Mobile MVS VM/VSE High availability solutions from IBM 107 By copying data, users can get information without impacting their production applications. It also removes any dependency on the performance of remote data access and the availability of communication links. DataPropagator/400 highlights include: • An automatic copy of databases • Full support for SQL (enabling summaries, derived data, and subsetted copies) • During a system or network outage, the product restarts automatically from the point where it stopped. If this is not possible, a complete refresh of the copies can be performed if allowed by the administrators. Also, for example, if one of the components fails, the product can determine that there is a break in sequence of the data being copied. In this case, DataPropagator/400 restarts the copy from scratch. • Open architecture to enable new applications • DataPropagator/400 commands that support AS/400 system definitions • Full use of remote journal support in V4R2 9.1.2 DataPropagator/400 configuration In the database network, the user needs to assign one or more roles to their systems when configuring the DataPropagator/400 environment. These roles include: • Control server: This system contains all of the information on the registered tables, the snapshot definitions (the kind of data you want to copy and how to copy it), the ownership of the copies, and the captures in reference to registrars and subscribers. • Data server: This contains the source data tables. • Copy server: This is the target system. Depending on the structure of the company, the platforms involved, and customer preferences, a system in the network can play one or more of these three roles. DataPropagator/400, for example, works powerfully on a single AS/400 system, which, at the same time, serves as the Control, Copy, and Data Server. 9.1.3 Data replication process With DataPropagator/400, there are two steps to the data replication process: • The Capture process: This is for reading the data. • The Apply process: This is for applying updated data Figure 26 and Figure 27 on page 108 illustrate these processes. 108 High Availability on the AS/400 System: A System Manager’s Guide Figure 26. The DataPropagator/400 Capture process Figure 27. The DataPropagator/400 Apply process Features included in the Capture and Apply processes include: • Support for the remote journal function to offload the source CPU • Automated deletion of journal receivers • Replication over a native TCP/IP-based network • Multi-vendor replication with DataJoiner (replication to and from Oracle, Sybase, Informix, and Microsoft SQL Server databases) • Integration with the Lotus Notes databases 9.1.4 OptiConnect and DataPropagator/400 DataPropagator/400 is based on a distributed relational database architecture (DRDA) and is independent of any communications protocol. Therefore, it uses OptiConnect and any other media without additional configuration. Administration Target BASE Control COPY COPY COPY APPLY Unit of Work Change Data Control CAPTURE Journal Base tables Column selection After image or before and after image Operational System Administration BASE Unit of Work Change Data Control CAPTURE Journal Operational System Control Target PL/User Copy HISTORY STAGING APPLY AGGREGATE Base and Copy tables Internal and Repetition Column and Row selection Computed Columns Aggregations Append or Replace High availability solutions from IBM 109 9.1.5 Remote journals and DataPropagator/400 DataPropagator/400 takes advantage of an operating system’s remote journal function. With remote journals, the capture process is run at the remote journal location to offload the capture process overhead from the production system. The Apply process does not need to connect to the production system for differential refresh because the DataPropagator/400 staging tables reside locally rather than on the production system. In addition, because the DataPropagator/400 product is installed only on the system that is journaled remotely, the production system no longer requires a copy of DataPropagator/400. 9.1.6 DataPropagator/400 implementation DataPropagator/400 is most beneficial for replicating data to update remote databases. One real-life example of this is a customer in Denmark who had a central AS/400 system and stored all production data, pricing information, and a customer database on it. From this central machine, data was distributed to sales offices in Austria, Germany, Norway, and Holland (each of which operated either small AS/400 systems or OS/2 PCs). Each sales office received a subset of the data that was relevant to their particular office. See 3.1, “A high availability customer: Scenario 1” on page 25, for a description of this customer scenario. 9.1.7 More information about DataPropagator/400 For more information about IBM DataPropagator/400 solutions, refer to DataPropagator Relational Guide, SC26-3399, and DataPropagator Relational Capture and Apply/400, SC41-5346. Also, visit the IBM internet Web site at: http://www.software.ibm.com/data/dbtools/datarepl.html 110 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 111 Chapter 10. High availability business partner solutions High availability middleware is the name given to the group of applications that provide replication and management between AS/400e systems and cluster management middleware. IBM business partners that provide high system availability tools continue to complement IBM availability offerings of clustering and system mirroring solutions. Combining clusters of AS/400 systems with software from AS/400 high-availability business partners improves the availability of a single AS/400 system by replicating business data to one or more AS/400 systems. This combination can provide a disaster recovery solution. This chapter explores the applications provided by IBM High Availability Business Partners (HABPs), DataMirror, Lakeview Technology, and Vision Solutions. This chapter also discusses the options these companies provide to support a highly or continuously available solution. Note: For customers requiring better than 99.9% system availability, AS/400 clusters with high-availability solutions from an IBM Business Partner are recommended. 10.1 DataMirror Corporation DataMirror Corporation is an IBM business partner with products that address a number of issues, such as data warehousing, data and workload distribution, and high availability. DataMirror products run on IBM and non-IBM platforms. The DataMirror High Availability Suite uses high performance replication to ensure reliable and secure delivery of data to backup sites. In the event of planned or unplanned outages, the suite ensures data integrity and continuous business operations. To avoid transmission of redundant data, only changes to the data are sent to the backup system. This allows resources to be more available for production work. After an outage is resolved, systems can be resynchronized while they are active. The DataMirror High Availability Suite contains three components: • DataMirror High Availability (HA) Data • ObjectMirror • SwitchOver System Figure 28 on page 112 illustrates the components of the DataMirror High Availability Suite. The corresponding DataMirror HA Suite Source Menu is shown in Figure 29 on page 112. Some of the software vendors mentioned in this chapter may have products with functions that are not directly related to the high availability issues on the AS/400 system. To learn more about these products, visit these vendors on the World Wide Web. You can locate their URL address at the end of the section that describes their solution. Note 112 High Availability on the AS/400 System: A System Manager’s Guide Figure 28. DataMirror High Availability Suite Figure 29. DataMirror HA Suite Source Menu 10.1.1 DataMirror HA Data DataMirror HA Data mirrors data between AS/400 production systems and failover machines for backup, recovery, high systems availability, and clustering. Object Changes Database files and or AS/400 objects User Application Role Swap Database files and or AS/400 objects Role Swap Meta Data Object Capture User Application Filtering Communications Manager Object Capture Data Transformation Communications Manager Transformation Server Object Mirror SwitchOver System Source Target High availability business partner solutions 113 A user can replicate entire databases or individual files on a predetermined schedule in real time or on a net change basis. They can refresh the backup machine nightly or weekly as required. They can also use DataMirror HA Data to replicate changes to databases in real time so that up-to-the-minute data is available during a scheduled downtime or disaster. DataMirror HA Data software is a no-programming-required solution. Users simply install the software, select which data to replicate to the backup system, determine a data replication method (scheduled refresh or real time), and begin replication. At the end of a system failure, fault tolerant resynchronization can occur without taking systems offline. DataMirror HA Data supports various high availability options, including workload balancing, 7 x 24 hour operations availability, and critical data backup. Combined with Data Mirror ObjectMirror software and SwitchOver System, a full spectrum of high availability options is possible. 10.1.2 ObjectMirror ObjectMirror enables critical application and full system redundancy to ensure access to both critical data and the applications that generate and provide data use. ObjectMirror supports real time object mirroring from a source AS/400 system to one or more target systems. It provides continuous mirroring as well as an on-demand full refresh of AS/400 objects that are grouped by choice of replication frequency and priority. ObjectMirror features include: • Grouping by choice to mirror like-type objects based on frequency or priority • Continuous real time mirroring of AS/400 objects • Intelligent replication for guaranteed delivery to backup systems even during a system or communication failure • Object refresh on a full-refresh basis as needed • Fast and easy setup, including an automatic registration of objects • Ability to send an object or group of objects immediately without going through product setup routines 10.1.3 SwitchOver System The SwitchOver System operates on both the primary and backup AS/400 systems to monitor communications or system failures. During a failure, the SwitchOver System initiates a logical role switch of the primary and backup AS/400 systems either immediately or on a delayed basis. A Decision Control Matrix in the SwitchOver System allows multiple line monitoring, detailed message logging, automated notification, and user-exit processing at various points during the switching process. An A/B switch (shown in Figure 30 on page 114) allows the user to automatically switch users and hardware peripherals, such as twinax terminals, printers, and remote controllers. 114 High Availability on the AS/400 System: A System Manager’s Guide Figure 30. DataMirror SwitchOver system 10.1.4 OptiConnect and DataMirror The DataMirror High Availability Suite supports SNA running over OptiConnect between multiple AS/400 systems. After OptiConnect is installed on both source and target AS/400 systems, the user needs to create controllers and device descriptions. After controllers and devices are varied on, the user simply specifies the device name and remote location used in the DataMirror HA Data or Object Mirror target definition. Files or objects that are specified can then be defined, assigned to the target system, and replicated. 10.1.5 Remote journals and DataMirror The DataMirror High Availability (HA) Suite is capable of using the IBM remote journal function in OS/400. The architecture of the DataMirror HA Suite allows the location of the journal receivers to be independent from where the production (source) or failover (target) databases reside. Therefore, journal receivers can be located on the same AS/400 system as the failover database. This allows the use of DataMirror intra-system replication to support remote journals. Customers can invoke remote journal support in new implementations. Additionally, the existing setup can be modified if remote journal support was not originally planned. 10.1.6 More information about DataMirror To learn more about DataMirror products, visit DataMirror on the Internet at: http://www.datamirror.com Remote Systems Mainframe ??? ??????????? Async, BSC, SDLC, X.25, ISDN, Frame Relay using V24, V35, X21 X25, T1/T2 5394 5494 Twinax attached terminals, printers 5250 PC cards LCW .... Twinax attached terminals, printers 5250 PC cards SWITCH A B SwitchOver System 1234 1234 DATA DATA OS/400 OS/400 Token Ring Ethernet FDDI 5394 PC Routers or Bridges Remote LAN PC High availability business partner solutions 115 10.2 Lakeview Technology solutions Lakeview Technology, an IBM business partner, offers a number of products to use in an AS/400 high availability environment. Their high availability suite contains five components: • MIMIX/400 • MIMIX/Object • MIMIX/Switch • MIMIX/Monitor • MIMIX/Promoter The following sections highlight each of these components. 10.2.1 MIMIX/400 MIMIX/400 is the lead module in the Lakeview Technology MIMIX high availability management software suite for the IBM AS/400 system. It creates and maintains one or more exact copies of a DB2/400 database by replicating application transactions as they occur. The AS/400 system pushes the transaction data to one or more companion AS/400 systems. By doing this, a viable system with up-to-date information is always available when planned maintenance or unplanned disasters bring down the primary system. MIMIX/400 also supports intra-system database replication. Figure 31 shows the basic principles of the MIMIX/400. Figure 31. The basics principles of Mimix/400 The key functions of MIMIX/400 are: • Send: This function scrapes the source system journal and sends the data to one or more target systems. This function offers the following characteristics: – Is written in ILE/C for high performance – Uses CPI-C to provide a low-level generic interface that keeps CPU overhead to a minimum Source AS/400 system Target AS/400 system CPI-C Com munications CPI-C Com munications Applications Journaling is the key A Data B Data Journal receiver 116 High Availability on the AS/400 System: A System Manager’s Guide – Supports filtering to eliminate files from MIMIX copies and to optimize communication throughput, auxiliary storage usage, and performance on the target system – Generates performance stamps, which continue throughout the replication cycle, for a historic view of performance bottlenecks • Receive: This function collects transactions from the Send function. The Receive function stores and manages the transactions on the backup system for processing using the Apply function. The Receive function offers these features: – A temporary staging step where transactions are pushed off the sending system as soon as possible to eliminate the load from the source system CPU – Fast performance because it is written in ILE/C – Variable length log space entries to make the most of available CPU and DASD resources – Filtering capabilities for greater capacity exists on the target system to boost performance on the Send side • Apply: This function reads all transactions and updates the duplicate databases on the target systems accordingly. The Apply function supports the following features: – Offers a file or member control feature to manage file name aliases and define files to the member level (files are locked during the Apply process for maximum configuration flexibility and to prevent files from being unsynchronized) – Opens up to 9,999 files simultaneously within a MIMIX Apply session – Supports record lengths up to 32 K in size – Manages DB2/400 commitment control boundaries during the Apply and Switch processes – Uses a log process to protect against data loss during a source system outage – Includes a graphical status report of source and target system activity. It displays the report in an easy-to-read format for operators to quickly identify MIMIX operating environment issues. • Synchronize: This function verifies that the target system has recorded exact copies of the source system data. The Synchronize function supports the following features: – Offers keyed synchronization to keep target and source databases in synchronization with the unique “key” field in each record – Provides support tools to analyze and correct file synchronization errors by record • Switch: This function prepares target systems for access by users during a source system outage. The Switch function performs the following tasks: – Defines systems, journals, fields, and data areas. The Send, Receive, and Apply sessions are linked into a logical unit called a data group. High availability business partner solutions 117 – Uses a data group manager to reverse the direction of all MIMIX/400 transmissions during an outage – Offers a journal analysis tool to identify transactions that may be incomplete after an outage 10.2.2 MIMIX/Object The MIMIX/Object component creates and maintains duplicate images of critical AS/400 objects. Each time a user profile, device description, application program, data area, data queue, spooled file, PC file, image file, or other critical object is added, changed, moved, renamed, or deleted on an AS/400 production system, MIMIX/Object duplicates the operation on one or more backup systems. The key elements of MIMIX/Object include: • Audit Journal Reader: This element scrapes the source system security audit journal for object operations and passes them to the distribution reader. The features of the Audit Journal Reader include: – Management of objects within a library by object and type; document library objects (DLOs) by folder path, document name, and owner; and integrated file system objects by directory path and object name – Management of spooled-file queues based on their delivery destination – Explicit, generic (by name), and comprehensive (all) identification of library, object, DLO, and integrated file system names – An “include” and “exclude” flag for added naming precision – Integrated file system control to accommodate hierarchical directories, support long names, and provide additional support for byte stream files • Distribution Reader: This element sends, receives, confirms, retries, and logs objects to history and message queues. The features of the Distribution Reader include: – Multi-thread asynchronous job support to efficiently handle high volumes of object operations – A load-leveling journal monitor to automatically detect a large back log for greater parallelism in handling requests – A history log to monitor successful distribution requests; offering reports by user, job, and date; and provides effective use of time for improving security control and management analysis – A failed request queue to provide error information, and for deleting and retrying options for ongoing object integrity and easy object resolution – An automatic retry feature to resubmit requests when objects are in use by another application until the object becomes available – Automatic management of journal receivers, history logs, and transaction logs to minimize the use of auxiliary storage • Send Network Object: This element relies on the Audit Journal Reader, which interactively saves and restores any object from one system to another. It offers simplified generic distribution of objects manually or automatically through batch processing. 118 High Availability on the AS/400 System: A System Manager’s Guide 10.2.3 MIMIX/Switch The MIMIX/Switch component detects system outages and initiates the MIMIX recovery process. It automatically switches users to an available system where they can continue working without losing information or productivity. The key elements of MIMIX/Switch include: • Logical Switch: This element controls the physical switch, communication and device descriptions, network attributes, APPC/APPN configurations, TCP/IP attributes, and timing of the communication switchover. The features of the Logical Switch include: – User exits to insert user-specified routines almost anywhere in the command stream to customize the switching process – A message logging feature to send status messages to multiple queues and logs for ensuring the visibility of critical information • Physical Switch: This element automatically and directly communicates with the gang switch controller to create a switchover. The features of the Physical Switch include: – A custom interface to the gang switch controller to switch communication lines directly – An operator interface to facilitate manual control over the gang switch controller – Remote support to initiate a switch through the gang switch controller from a distance – Interface support of twinax, coax, RJ11, RS232, V.24, V.35., X.21, DB9, or other devices that the user can plug into a gang switch • Communications Monitor: This element tracks the configuration object status to aid in automating retry and recovery. An automatic verification loop ensures that MIMIX/Switch only moves users to a backup system when a genuine source system outage occurs. 10.2.4 MIMIX/Monitor The MIMIX/Monitor component combines a command center for the administration of monitor programs and a library of plug-in monitors so the user can track, manage, and report on AS/400 processes. MIMIX/Monitor regulates the system 24 hours a day. It presents all monitor programs on a single screen with a uniform set of commands. This minimizes the time and effort required to insert or remove monitors or change their parameters. The MIMIX/Monitor also accepts other data monitoring tools created by customers and third-party companies into its interface. The user can set the programs included with MIMIX/Monitor to run immediately, continually at scheduled intervals, or after a particular event (for example, a communications restart). MIMIX/Monitor includes prepackaged monitor programs that the user can install to check the levels in an uninterrupted power supply (UPS) backup system, or to evaluate the relationship of MIMIX to the application environment. High availability business partner solutions 119 10.2.5 MIMIX/Promoter The MIMIX/Promoter component helps organizations maintain continuous operations while carrying out database reorganizations and application upgrades, including year 2000 date format changes. It uses data transfer technology to revise and move files to production without seriously affecting business operations. MIMIX/Promoter builds copies of database files record-by-record, working behind the scenes while users maintain read-and-write access to their applications and data. It allows the user to fill the new file with data, change field and record lengths, and, at the same time, keep the original file online. After copying is complete, MIMIX/Promoter moves the new files into production in a matter of moments. This is the only time when the application must be taken offline. Implementing an upgrade also requires promoting such non-database objects as programs and display files. To handle these changes, many organizations use change management tools, some of which can be integrated with MIMIX/Promoter′ s data transfer techniques. 10.2.6 OptiConnect and MIMIX MIMIX/400 integrates OptiConnect for OS/400 support for the IBM high-speed communications link, without requiring separate modules. The combination of MIMIX and OptiConnect provides a horizontal growth solution for interactive applications that are no longer contained on a single machine. OptiConnect delivers sufficient throughput for client/server-style database sharing among AS/400 systems within a data center for corporate use. MIMIX/400 complements the strategy by making AS/400 server data continuously available to all clients. 10.2.7 More Information About Lakeview Technology For more information about the complete Lakeview Technology product line, visit Lakeview Technology on the Internet at: http://www.lakeviewtech.com 10.3 Vision Solutions: About the company Vision Solutions was founded in 1990 by two systems programmers working at a hospital IT staff in California. They recognized the need for a dual systems solution that would exploit the rich OS/400 architecture and provide an application for managing business integrity using dual AS/400 systems. Originally known as Midrange Information Systems, Inc., the name was changed to Vision Solutions, Inc. in July 1996. Today it has grown into an international company with development staff and facilities in the Netherlands, South Africa, and the United States. It employs over 150 people worldwide. 10.3.1 Vision Solutions HAV solutions When you consider the costs of purchasing additional assets in the form of hardware, software, and consulting services to expand the hours of operations, to increase the scope of a business’ growth capability, and to allow greater utilization of a business solution on the AS/400 platform, using dual systems for mirroring the business application system is a prudent business decision. If the 120 High Availability on the AS/400 System: A System Manager’s Guide focus is strictly on the disaster aspects of dual systems, the decision to go with this solution is never quick or easy to make. By expanding the view of why a business must use dual systems to the other advantages of continuous operations support, improved availability from dedicated backup processes and workload balancing, the decision process leads to a wise business project. Vision Solutions supports this effort with its management and integrity facilities that are built directly into Vision Suite. One main advantage of using Vision Suite for your HSA and Continuous Operations requirements is that many of its application integrity features exceed the requirements of most AS/400 mission critical applications today. When considering dual systems, pursue your evaluations with due diligence and use the following criteria: • Integrity: How do you know the backup system is equal to your production? Do you employ more analysts to write additional utilities for monitoring or support that use your existing staff? Or, will this decision software reside in the HSA solution? • Performance: How much data can you push to the other system to minimize in-flight transactions being lost during an unplanned outage using the minimum possible CPU? If your application employs OS/400 Remote Journaling and Clustering, does the HSA vendor demonstrate live support of this capability? • Performance: What happens when your network stops or your backup system fails for an indefinite period of time? Can you catch up quickly to protect your business? • Ease of Use: Can my existing operations staff use this application? • Application Support: As an application evolves and possibly extends its use of the rich OS/400 architecture, will your HSA application be able to support those extensions without customized software? Pursue this criteria for your business needs and commitments with regards to continuous operations and business integrity. 10.3.2 Vision Suite There are three components to Vision Suite: • Object Mirroring System/400 (OMS/400) • Object Distribution System/400 (ODS/400) • System Availability Monitor/400 (SAM/400) The requirements for Vision Suite necessitate the use of the OS/400 Journal function for both OMS/400 and ODS/400. Journaling gives Vision Suite the ability to deliver real time database transactions and event driven object manipulations to a backup AS/400 system. While some AS/400 application environments have journaling already active, the integrated Vision Suite Journal Manager can relieve the user of the journal receiver management function. Figure 32 on page 121 illustrates a typical configuration between a production system and a backup system. The journal receivers provide the input to Vision Suite for replication to the backup Database Server system. 10.3.2.1 OMS/400 This component replicates and preserves the Application Integrity established in your software design. It immediately transfers the transitional changes that occur High availability business partner solutions 121 in your data areas, data queues, and physical files to a backup AS/400 system. As the application database is manipulated by the programs I/O requests, and these operations are recorded in the journal by OS/400, OMS/400 transports the resulting journal entries of those requests to a backup database server in real time. This minimizes any data loss due to an unplanned outage. As shown in Figure 32, the Reader/Sender function takes the journal entry over to the backup system through various communications media supported in OS/400. Any communications media utilizing the SNA or TCP/IP protocols and OptiMover are supported by Vision Suite. Figure 32. Typical Vision Suite configuration The Receiver function places the journal entry into the Router function so that it may be assigned to an Apply Queue. The Apply Queue writes the record image captured in the journal entry into the appropriate data object located on the backup system. All objects in a given Database Server are distributed evenly across multiple Apply Queues provided for that single database. The equal separation of files (according to logical relationships in DB2/400 or applications) ensures an even use of CPU and memory resources among each Apply Queue. 10.3.2.2 Application integrity In addition to the requirement of delivering data efficiently and quickly, OMS/400 manages the synchronization and integrity of the database and data objects by using several background processes that do not require operator control and management. Integrity of your database between each physical file, data area, LAN Production System Backup System System ASP User ASP Journal Receivers SQL Tables, Stream files, objects Reader/Sender Communications Media Receiver System ASP Router Apply Apply SQL Tables, Stream files, objects 122 High Availability on the AS/400 System: A System Manager’s Guide and data queue, along with the applications, is a cornerstone for successful role swaps. Role swap is Vision terminology for moving the business processes, which can be end users, batch jobs, or a combination of both application environments from one AS/400 server to another. Assurances for complete Application integrity on your backup systems allow you to quickly declare disasters instead of hoping for something else to happen. No integrity issue of any type ensures that planned or unplanned outages occur with a quick role swap supporting continuous operations at minimum downtime. Furthermore, the journal receivers generated from all of the database activity impacts the auxiliary storage space in a short time if these objects are not managed. Vision Suite features its Journal Manager function that creates journal environments for your mission critical databases. It can completely control the entire management role of journal receivers on both the production and backup AS/400 server. This ensures that the client is free for other AS/400 maintenance functions. 10.3.2.3 ODS/400 The second component of Vision Suite preserves the Environment Integrity developed for your application by replicating all supported object types of that application. This is an important consideration for any AS/400 system requiring continuous operations. While database transactions are complex and numerous compared to object manipulations, change management of your application environment must be duplicated on the backup system to ensure a timely and smooth role swap (move the business from the production system to the backup system). To ensure Environment Integrity, a user should choose to perform change management on each individual system or use ODS/400 to replicate the various object changes to those systems. Increasingly, object security has heightened the need for ODS/400. In pre-client (or PC Support) days, typical security control was managed through 5250 session menus. However, today’s WAN and Internet/intranet network environments utilize many application tools that are built on ODBC and OLE database interfaces. AS/400 IT staffs must meet the challenge of taking advantage of OS/400 built-in object security. This involves removing public authority from mission critical objects and interjecting the use of authorization lists and group profiles. While the AS/400 system maintains a high-level interface for this work, the interlocking relationships of objects, databases, and users (both local and remote) become complex. ODS/400 maintains this complex environment so that its integrity is preserved on your backup system. In mission critical AS/400 applications, the main focus for continuous operations and high availability is the integration of the database with its associated software and related security object authorizations and accesses. 10.3.2.4 SAM/400 The final component of Vision Suite is SAM/400. Its main purpose is to monitor the production (or source application) system heartbeat and condition the role swap when all contact is lost. It has ancillary functions for keeping unwarranted users from accessing the backup system when it is not performing the production function. It also provides user exits for recovery programs designed by the Professional Services staff for specific recovery and environment requirements. High availability business partner solutions 123 Vision Solutions, Inc. products operate on two or more AS/400 systems in a network and use mirroring techniques. This ensures that databases, applications, user profiles, and other objects are automatically updated on the backup machines. In the event of a system failure, end users and network connections are automatically transferred to a predefined backup system. The Visions products automatically activate the backup system (perform a role swap) without any operator intervention. With this solution, two or more AS/400 systems can share the workload. For example, it can direct end-user queries that do not update databases to the backup system. Dedicated system maintenance projects are another solution benefit. The user can temporarily move their operations to the backup machine and upgrade or change the primary machine. This High Availability Solution offers an easy and structured way to keep AS/400 business applications and data available 24 hours a day, 7 days a week. The Vision Solutions, Inc. High Availability Solution (called Vision Suite) includes three components: • Object Mirroring System (OMS/400) • Object Distribution System (ODS/400) • System Availability Monitor (SAM/400) The following sections highlight each component. 10.3.3 OMS/400: Object Mirroring System The Object Mirroring System (OMS/400) automatically maintains duplicate databases across two or more AS/400 systems. Figure 33 illustrates the OMS/400 system. This system uses journals and a communication link between the source and target systems. Figure 33. Object Mirroring System/400 User application programs Journal receivers User databases OMS/400 sender User query programs User Space User databases OMS/400 receiver Router Source system Target AS/400 124 High Availability on the AS/400 System: A System Manager’s Guide The features of the OMS/400 component include: • Automatic repair of such abnormal conditions as communication, synchronization, or system failure recoveries • Synchronization of enterprise-wide data by simulcasting data from a source system to more than 9,000 target destinations • User space technology that streamlines the replication process • An optional ongoing validity check to ensure data integrity • Automatic restart after any system termination • Automatic filtration of unwanted entries, such as opens and closes • The power to operate programs or commands from a remote system • The ability to dynamically capture data and object changes on the source system and copy them to the target system without custom commands or recompiles • The option to create an unlimited number of prioritized AS/400 links between systems • Total data protection by writing download transactions to tape • Support of RPG/400 for user presentation, and ILE/C for system access, data transmission, and process application • Full support of the IBM OptiConnect/400 system • Global journal management when a fiber optic bus-to-bus connection is available • The use of CPI-C to increase speed of data distribution using minimal CPU resources 10.3.4 ODS/400: Object Distribution System The Object Distribution System (ODS/400) provides automatic distribution of application software, authority changes, folders and documents, user-profile changes, and system values. It also distributes subsystem descriptions, job descriptions, logical files, and output queue and job queue descriptions. ODS/400 is a partner to the OMS/400 system, and it provides companies with full system redundancy. It automatically distributes application software changes, system configurations, folders and documents, and user profiles throughout a network of AS/400 computers. ODS/400 supports multi-directional and network environments in centralized or remote locations. For maximum throughput, ODS/400 takes advantage of bi-directional communications protocol and uses extensive filters. 10.3.5 SAM/400: System Availability Monitor The System Availability Monitor (SAM/400) can switch users from a failed primary system to their designated secondary system without operator intervention. SAM/400 works in conjunction with OMS/400 and ODS/400, continuously monitoring the source system. In the event of a failure, SAM/400 automatically redirects users to the target system. This virtually eliminates downtime. High-speed communications links, optional electronic switching hardware, and High availability business partner solutions 125 SAM/400 work together to switch users to a recovery system in only a few minutes. SAM/400 offers: • Continuous monitoring of all mirrored systems for operational status and ongoing availability • A fully programmable response to react automatically during a system failure, which reduces the dependence on uninformed or untrained staff • The ability to immediately and safely switch to the target system, which contains an exact duplicate of the source objects and data during a source system failure (unattended systems are automatically protected 24 hours a day, 7 days a week) • User-defined access to the target system based on a specific user class or customized access levels The SAM/400 component offers: • Up to ten alternate communication links for monitoring from the target system to the source system • Automatic initiation of user-defined actions when a primary system failure occurs • Exit programs to allow the operator to customize recovery and operations for all network protocols and implementations Figure 34 illustrates the SAM/400 monitoring process. Figure 34. SAM/400 structure Users are allowed to access applications at the “End” point. 10.3.6 High Availability Services/400 High Availability Services/400 (HAS/400) consists of software and services. The HAS/400 solution is comprised of: SAM/400 Traffic Role Swap Start conversation Yes Source system Target AS/400 SAM/400 Remote switch controller A/B switch panel Is the other system active?? No Do we have a hardware switch ? Call user exit pgm Yes 126 High Availability on the AS/400 System: A System Manager’s Guide • Analysis of the customer environment in terms of system availability needs and expectations, critical business applications, databases, and workload distribution capabilities • An implementation plan written in terms of solution design and the required resources for its deployment • Installation and configuration of the software products and the required hardware • Education for the customer staff on operational procedures • Solution implementation test and validation • Software from Vision Solutions, Inc., as previously described 10.3.7 More information about Vision Solutions, Inc. For more information about Vision Solutions, Inc. products, visit Vision Solutions on the Internet at: http://www.visionsolutions.com © Copyright IBM Corp. 2001 127 Chapter 11. Application design and considerations Applications are regarded as business-critical elements. The viewpoint of systems management is changing from a component view to an application view. Here are a few considerations that are now made at the application level: • The entire application, or parts of the application, must be distributed. • The application has to be monitored to guarantee availability. • Operations, such as scheduling jobs and doing backups, are recommended. • User profiles must be created, given access to applications, changed, and deleted. If a system is unavailable, and rapid recovery is necessary, backups are restored and the system and the database are inspected for integrity. The recovery process could take days for larger databases. This scenario requires improvement in the areas of restore speed and application recovery. Both of these areas can be costly to implement. High speed tape drives are very expensive items and, for very large databases, they may not show enough restore time improvement to meet user demands. Application recovery requires a lot of development effort and is, therefore, very costly. This, in itself, may also degrade availability. To make the application more available, considerable additional processing and I/O should be available. This means the response time degrades unless there is an abundance of computing resources. In the past, application recovery was at the bottom of the list of availability tasks. The size of required systems would be too expensive for the business to justify. These days, with the vast improvements on price and performance power, solutions exist that provide a high level of availability. In addition, businesses are now able to define the cost of system outage more accurately. From a user point-of-view (both a system end user, and a receiver of services), the availability of a system relates to the information available at a given time. A customer holding for a price lookup considers the efficiency of the answer as an indication of the quality of the business. Designing applications for high availability is a comprehensive topic and textbooks have been dedicated to this topic alone. From a high level, some of the considerations are discussed in this chapter. Areas that are covered include application checkpointing design, considerations, and techniques (including CL programs), for the interactive environment. 11.1 Application coding for commitment control You can use commitment control to design an application so that it can be started again if a job, an activation group within a job, or the system ends abnormally. The application can be started again with the assurance that no partial updates are in the database due to incomplete logical units of work from a prior failure. There are numerous documents that describe the use of commitment control and journaling. The OS/400 Backup and Recovery, SC41-5304, contains journaling and commitment control requirements. IBM language-specific manuals include: 128 High Availability on the AS/400 System: A System Manager’s Guide • DB2 for AS/400 SQL Programming, Version 4, SC41-5611 • ILE C for AS/400 Programmer’s Guide, Version 4, SC09-2712 • ILE COBOL for AS/400 Programmer’s Guide, Version 4, SC09-2540 • ILE RPG for AS/400 Programmer’s Guide, Version 4 SC09-2507 These manuals contain information about using commitment control for a particular language. Various Redbooks and “how to” articles are found throughout IBM-related web sites. These sites include: • http://www.news400.com “Safeguard Your Data with RI and Triggers”, Teresa Kan December, 1994 (page 55) • http://www.news400.com “AS/400 Data Protection Methods”, Robert Kleckner, December 1993 (page 101) 11.2 Application checkpointing In general, application checkpointing is a method used to track completed job steps and pick up where the job last left off before a system or application failure. Using application checkpointing logic, along with commitment control, you can provide a higher lever of resiliency in both applications and data, regardless of whether they are mission critical in nature. Throughout the existence of IBM midrange systems, application checkpointing has been used to help recover from system or application failures. It is not a new subject when it comes to IBM midrange computing, specifically on the AS/400 system. However, there are some new features. Unlike commitment control, application checkpointing has no system level functions that can be used to automate recovery of an application. If commitment control is used, and the job stream has multiple job steps, the application needs to know which job has already run to completion. Application checkpoints help the programmers to design recovery methods that can prevent the restarted jobs from damaging the database by writing duplicate records. Remember that there is additional information written on concepts and methods for application checkpoints, as well as recovery with or without journaling and commitment control. The following sections define recovery methods for applications that work in any high availability environment for the AS/400 system (including clustering). Most of the concepts also work for other (non-AS/400) platforms. 11.3 Application checkpoint techniques Techniques for application checkpointing and recovery vary for every program. Whether you use Cobol, RPG, SQL or C as the application language, the methods employed for application checkpointing remain constant. Without journaling and commitment control capabilities, programmers devise their own tracking and recovery programs. This section describes an example of how this is done. Application design and considerations 129 11.3.1 Historical example The following scenario describes a customer environment that runs the sales force from a System/36 in the early 1980s. The customer’s remote sales representatives dial into a BBS bulletin board system from their home computer, upload the day’s orders, and request the sales history for the clients they are to visit the next day. The next morning, the same remote sales representative dials into the BBS to retrieve the requested information. BBS systems and modems were not reliable in the early 1980s. Many of the transfers ended abnormally. Application programmers devised a method to update data areas with specific job step information after the job steps completed. If the Operator Control Language (OCL) job starts and finds an error code in the data area, the program logic jumps to the last completed step (as indicated by the data area) and starts from there. This is a primitive form of application checkpointing, but it works. Later applications utilized log files. Programs were designed to retrieve information from the log file if the last job step was not successful. Using the program name, last completed job step, current and next job step, as well as the total job steps, the programmer determined where to start the program. The program itself contained recovery subroutines to process if the recovery data area contained information that a job failure occurred. As for the data, temporary files were created containing the before image of a file. Reading the required record, then writing the temporary file prior to any updates or deletes, the information was written back to its original image if the job failed. At the completion of the job, all temporary files were removed. With the high availability products on the market today, a more efficient design is possible. A permanent data file includes the data area logic. High availability products mirror data areas and data queues. However, the HA applications work off of journal information. Note: Data areas and data queues can not be journaled in OS/400 V4R4. Moving any checkpoint logic from a data area to a data file operates with more efficiency and provides a higher restart capability than data areas. 11.4 Application scenarios The following paragraphs explain application checkpointing methods in various scenarios. The methods described are not the only possible options for application checkpoints. However, they provide a good starting point for managing your high availability environment with application checkpoints that work in any HA environment. 11.4.1 Single application For a single application, checkpoints are established by adding recovery logic to the program to handle the commit and roll back functions. The job’s Control Language (CL) program needs to include checks for messages that indicate an incomplete or open Logical Unit of Work (LUW). 130 High Availability on the AS/400 System: A System Manager’s Guide Testing for incomplete job runs is the primary requirement of application checkpoints. Some simple testing of control information for an error code prior to running the start or end commit command prevents users from getting erroneous messages. If the control information is clean, run the Start Commitment Control (STRCMTCTL) function and change the control information to uncompleted. If the control information has an error code, act on it by performing either a commit or rollback. Most often, the action is a rollback. At the completion of the program, execute a commit and then change the control information to indicate a successful update. 11.4.2 CL program example This example uses CL programs. It is assumed that the Logical Unit of Work (LUW) includes all I/O operations that this program performs. If a rollback takes place, all changes are removed from the system. If there is a higher complexity to the application, such as multiple levels of application calls, or many updates, inserts, and deletes, you should consider this a multiple application program. Also, in this example, a data area is used for the control information. For High Availability, it is recommended that you have the control information in a data file. Mirroring record information is more efficient than data areas or data queues because the current release of OS/400 (V4R4) does not support journaling data areas or data queues. Successful and complete recovery from a system failure is more likely if the recovery information is contained in a mirrored file. If this job is used in a multi-step job stream, place true application checkpoint functions into it. To do this, create a checkpoint-tracking file. The checkpoint-tracking file used to track the job steps must include information about the job and where to start. PGM DCL &OK *CHAR 1 RTVDTAARA CONTROL &OK 1 1 IF COND(&OK *NE ‘ ‘) THEN(ROLLBACK) CHGDTAARA &OK ‘E’ STRCMTCTL CALL UPDPGM RTVDTAARA CONTROL &OK 1 1 IF COND(&OK *NE ‘ ‘) THEN(ROLLBACK) IF COND(&OK *EQ ‘ ‘) THEN(COMMIT) ENDCMTCTL CHGDTAARA &OK ‘ ’ ENDPGM Basic CL program model 131 Chapter 12. Basic CL program model The following model contains the basic information for most jobs. Additional information can and should be tracked for better recovery: * Program Name * Current Job Step * Previous Job Step * Next Job Step * Total Job Steps * Job Start time * Job Name * User * Last processed record key information The information listed here should help you determine where you are, where you were, and where to go next. You can also determine how far into the job you are and who should be notified that the job was interrupted. 12.1 Determining a job step The diagrams shown in Figure 35 and Figure 36 illustrate how to determine a job step. Figure 35. Determining a job step (Part 1 of 3)) Figure 36. Determining a job step (Part 2 of 3) All programs have some basic flow. Using Commitment Control, the data is protected with the LUW, Commit, and Rollback. The diagram shown in Figure 37 on page 132 shows these components. Open Read Calculate Write EOJ Return Open file Read file Modify data Write file EOJ Return CC/LUW No 132 High Availability on the AS/400 System: A System Manager’s Guide Figure 37. Determining a job step (Part 3 of 3) To determine the key points for the job step markers, notice how the data is read, manipulated, and written. Also look at any end-of-job processing. If the job fails while processing the data, start at the last completed processed record. This means that the section where the data is read into the program is a “key point” for your job step. Most applications read the data first, so this is the first job step. If there is a section of the program prior to the read that must be recalculated, it should be the first point of the job step. After determining where the data is read, look at where the data is written. In this example, commitment control provides the second key point. When the data is committed to a disk, it can’t be changed. By placing the next job step at this point, a restart bypasses any completed database changes and moves to the next portion of the program. If the program writes thousands of records, but performs a commit at every 100, the checkpoint tracking information should include some key elements for the last committed record. This information should be collected for step 1 after every commit. With the information collected this way, a restart can go to step 1 and set the initial key value to start reading at the last committed point. End of job processing can cause many application restart attempts to fail. The reasons for this should appear obvious. If the job does not have an end of job (EOJ) summary or total calculations to perform, then step 2 is the last job step. However, if summary reports and total calculations must be performed (for example, for an invoice application), some added logic is needed. Most summary or total calculations are performed on data that is collected and calculated from a previously performed file I/O. They are usually stored in a table or in an array in memory. If the job fails before EOJ calculations are made, memory allocated for the job is released. Take this into consideration when working with commitment control’s logical unit of work. The logic for determining the recovery job steps may need to be “Start everything over”. With improper commitment control logic, data loss is possible. If, on the other hand, the job is extremely long running or is a critical job, write the summary information into a work file. The process of completing a job step and writing out the work file occurs before EOJ processing and can be considered its own job step. If you use a work file for array or table information, place the name of the work file in the tracking file as well. Include restart logic in the program to reload any tables or arrays for EOJ processing. Open file Read file Modify data Write file EOJ Return CC/LUW No Step 1 Step 2 Step 3 Basic CL program model 133 Keep in mind that high availability solution providers can track and mirror files that are created and deleted on the fly. However, processing overhead is much higher and the chance of “in-flight transactions” can occur at the time of failure. This results in lost data. Therefore, it is recommended that work files be created and maintained instead of created and deleted. Using the Clear Physical File Member (CLRPFM) function on the AS/400 system maintains the journal link with the file and reduces the overhead of saving and restoring file information caused by a create and delete. Create temporary files in the QTEMP library on the AS/400 system. QTEMP can not be mirrored by high availability solution providers. This library can contain true temporary files in which the contents can be easily recreated. If a job fails, temporary files do not affect the program outcome. The last checkpoint in the job should take place immediately before the end of, or return to, the previous job step. This checkpoint can clear any error flags, reset the program name, or remove the tracking record from the checkpoint tracking file. Creating logging functions for history reporting can also be performed from this checkpoint. 12.1.1 Summary of the basic program architecture The examples and logic defined in 11.3, “Application checkpoint techniques” on page 128, work within the AS/400 system on all release levels of OS/400 from Version 1 onward. If you plan on moving to a clustered environment available from IBM (from OS/400 V4R4 onward), the model described here provides a good start for cluster ready. Add additional tracking information to start the users at a specific point in the application, such as tracking screens and cursor positions. When writing out the checkpoint-tracking file, reserve space for this information so the recovery process can place the user as close to where they left off as possible. Information for screens and cursor positions, as well as job information, user information, and file information can be retrieved from the Information Data Structures (INFDS) provided by OS/400. A wealth of information can be retrieved from the INFDS, including screen names, cursor positions, indicator maps, job information, and more. Part of the cluster ready requirements is to restart the users where they last left off. Using application checkpoints accomplishes this. 12.2 Database At the core of the AS/400e system is the relational database. When multiple systems are involved, a portion or a copy of the data can reside on the remote system. This is called a distributed database. 12.2.1 Distributed relational database Application coding for distributed databases require more involved and complicated logic than programs designed for accessing data on a single system. Application checkpoint logic adds further complexity to the program logic. Both of these situations require a seasoned programmer analyst. This should be someone with patience, persistence, and experience. This section addresses 134 High Availability on the AS/400 System: A System Manager’s Guide some of the concerns of programming in a distributed relational database environment. Note: Using the ideas and concepts discussed in this chapter, and deploying them into any distributed application model, does not change the amount of work for the programmer. Application checkpointing is not concerned with where the data is. Rather, it is concerned with where the application runs. If you use DRDA as the basis of the distributed database, the applications reside with the data on all systems. DRDA submits a Unit of Work (UoW) that executes a sequence of database requests on the remote system. In other words, it calls a program and returns the information. DRDA uses a two-phase commit for the database. This means that the data is protected from failure over the communications line, as well as system or job failures. If the program on the remote system performs proper application checkpointing when the remote system, remote application, or communications, fails during the processing of UoW, the restart picks up from where it left off. Adding a “from location” field and “to location” field in the checkpoint-tracking file allows reports information that better defines the locations of the jobs running. It also helps isolate communication fault issues with the application modules that start and stop communications. It is recommended that application checkpointing in a DRDA environment be setup in a modular fashion. A recovery module that controls the reads and writes to the checkpoint-tracking file make analysis and recovery easier. Take special care to ensure that the database designers include the checkpoint-tracking file in the original database design. This enables the recovery module to treat each existence of the tracking file as an independent log for that local system. A future release of DB2/400 UDB Extended Enterprise Edition (EEE) will include scalability for Very Large Database (VLDB) support. Like DRDA, the VLDB database is distributed over different systems. Unlike DRDA, the access to the data is transparent to the program. No special communication modules or requester jobs need to be created and maintained. Since the environment looks and feels like a local database, the application checkpointing logic must treat it like a local database running in the multiple application mode. For more details on the workings of DB2/400 and DRDA, refer to DB2/400 Advanced Database Functions, SG24-4249. 12.2.2 Distributed database and DDM You can use DDM files to access data files on different systems as a method of having a distributed database. When the DDM file is created, a “shell” of the physical file resides on the local system. The shell is a pointer to the data file on the remote system. The program that reads from, and writes to, this file does not know the data is located on a different system. Since DDM files are transparent to the application, approach application checkpointing logic as discussed in 11.4.1, “Single application” on page 129. Basic CL program model 135 12.3 Interactive jobs and user recovery The information and logic necessary to recover from a system or application failure is no different between interactive jobs and batch jobs. There is a difference in how much user recovery is needed. There are three basic parts to every job: • Data • Programs • Users Use commitment control and journaling to address data recovery. The file layout for the checkpoint-tracking file described previously has a space reserved for the user name. This user name field can be used to inform the user that the job was abnormally ended and that a restart or recovery process will run. If this were an interactive job with screen information, the chance of the user getting back to where they left off is not high. To correct this deficit, the recovery process tracks more information to place the user as close to the point they left off as possible. The addition of current screen information, cursor positions, array information, table pointers, and variables can be stored in the tracking file along with all the other information. The AS/400 system stores all of the information it needs to run the application accessible by the programmer. The programmer simply needs to know where to find it. The Information Data Structure (INFDS) in RPG contains most, if not all, of the information required to get the user back to the screen from where they left off. Use the Retrieve Job Attribute (RTVJOBA) command to retrieve job attributes within the CL of the program. Retrieve system values pertinent to the job with the Retrieve System Value (RTVSYSVAL) command. Internal program variables are in control of the application programmer. Recovery logic within the application can retrieve the screen and cursor positions, the run attributes, and system values and write them to the tracking file along with key array, table, and variable information. In the event of a failure, the recovery logic in the program determines the screen the user was on, what files were open, what the variable, array, and key values were, and even place the cursor back to the last used position. 12.4 Batch jobs and user recovery and special considerations Unlike interactive jobs, batch jobs have no user interface that needs to be tracked and recovered. Since the user interface is not a concern, the checkpoint tracking process defined in 11.4.1, “Single application” on page 129, and 12.2.1, “Distributed relational database” on page 133, should suffice. In general, this is true. This section describes additional vital information about when the recovery environment includes high availability providers software. These considerations include: 136 High Availability on the AS/400 System: A System Manager’s Guide • Job queue information for the batch jobs cannot be mirrored: This means that if you submit multiple related jobs to a single threaded batch queue, and the system fails before all those jobs are completed, restarting may not help. • Determining what jobs have been completed and what jobs still need to run: If you have created a checkpointing methodology with the points described previously, you have a tracking record of the job that was running at the time of the failure. Using this information, restart that job and then manually submit the remaining jobs to batch. If this was a day-end process, determining the jobs that still need to be run should not be complicated. If it was a month-end process, the work to restart all the jobs consumes more time but it can be achieved with few or no errors. If this was a year-end process, the work to restart all the necessary jobs in the correct order without missing vital information can be very time consuming. A simple solution is to store tracking information for batch jobs in the application checkpoint-tracking file. The added work in the recovery file is minimal, yet the benefits are exponential. Within the job that submits the batch process, a call to the application checkpointing job name, submitted time, submitted queue, and submitted status is ideal. The application checkpoint module writes this information to the checkpoint-tracking file. When the job runs in batch mode, it modifies the status to “Active”. Upon completion, remove the record or mark as was done in the tracking file to complete a log of submitted jobs. Within any high availability environment, the tracking information is processed almost immediately. In the event of a system failure, the recovery module interrogates the tracking file for submitted jobs that are still in a JOBQ status and automatically resubmit them to the proper job queue in the proper order (starting with the job with an “Active” status). This possibility prevents any user intervention, therefore eliminating “user error”. 12.5 Server jobs The nature of a server job is very robust. To maintain reliability, server jobs should be able to recover from most, if not all, error conditions that can cause normal jobs to fail. Since server jobs run in a batch environment, the recovery process itself is identical to the batch process described in 12.4, “Batch jobs and user recovery and special considerations” on page 135. However, additional considerations for error recovery are necessary for the recovery file. With application checkpointing built into the server job, error conditions can be logged in the tracking file and corrections to either the server job or the client jobs can be made based on what is logged. Using application checkpoints to isolate faults and troubleshoot error conditions is an added advantage to a well-designed recovery process. If the server job fails, connection to the client can still exist. If this open connection is not possible, there may be a way to notify the client to re-send the requested information or Unit-of-Work (UoW). Either way, the server job must track the information in the checkpoint-tracking file. Basic CL program model 137 12.6 Client Server jobs and user recovery Client Server jobs come in many different models, including thin clients, fat clients, and other clients that include attributes of both fat and thin clients. Even though they have a different label, these client server jobs are either a batch job or an interactive job. The environment that the job runs in dictates the type of recovery to perform. Most client server applications rely on the client to contact the server to request information from the server. Thin clients contact the server and pass units of work for the server to perform and report back. Fat clients request data and process the information themselves. “Medium” clients perform various aspects of each method. Recovery for the client server jobs should be mutually exclusive. If the client job fails, the connection to the server can still exist. The client may be able to pick up where it left off. If this open connection is not possible, there may be a way to notify the server to re-send the request for information. Either way, the client job must track the information in the checkpoint-tracking file. If the processing of the client request pertains to a long running process, it may be best to design that particular job as a thin client. With a thin client design, the processing is performed on the server side where application checkpointing tracks and reports the job steps. In this case, recovery on the client includes checking whether the communications is still available. If not, then submit the request again from the beginning. If the processing of the client pertains to critical information, the design should lean towards the fat client model. If the client is a fat client model, the application checkpointing logic described in this book should suffice. Note: The nature of a client server relationship varies greatly. It is worth the time to determine whether the recovery process in a client server environment is necessary prior to writing the recovery steps. Thin clients perform much faster in a restart mode if they are simply started again with absolutely no recovery logic. For example, if a client process makes one request to the server for information, adding recovery logic can double or even triple the amount of time required to make that request. 12.7 Print job recovery In the standard program model, information is collected, processed, and written. The process of writing the information typically occurs during the end of job (EOJ), after all the data is collected. In this case, adding a checkpoint at the beginning of the EOJ processing restarts the printing functions in the event of a restart. The scenario is described in “Distributed relational database” on page 133. Exceptions to this rule include programs that collect and write “detailed” information as the job runs. Again, 12.2.1, “Distributed relational database” on 138 High Availability on the AS/400 System: A System Manager’s Guide page 133, describes how a job step defined at the proper locations recreates every function within the steps. Even with a detailed and proper running recovery function in place, there is no way to “pick up where you left off” in a print job. The print file itself is closed when the job runs. Rewriting to it is not possible. With proper application checkpoints in place, the print information should not be lost. It is, however, duplicated to some extent. If a print job within that application requires a specific name, that name should be tracked in the checkpoint-tracking file and proper cleanup should be performed prior to the job running again. © Copyright IBM Corp. 2001 139 Part 4. High availability checkpoints Part IV discusses miscellaneous items that are helpful when implementing a high availability solution. Included in this part is information on a Batch Caching solution, a discussion of the management of disk storage, device parity features, and a checklist of items to consider when implementing your high availability solution. 140 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 141 Appendix A. How your system manages auxiliary storage For many businesses, computers have replaced file cabinets. Information critical to a business is stored on disks in one or more computer systems. To protect information assets on your AS/400 system, you need a basic understanding of how it manages disk storage. On the AS/400 system, main memory is referred to as main storage. Disk storage is called auxiliary storage. Disk storage may also be referred to as DASD (direct access storage device). Many other computer systems require you to take responsibility for how information is stored on disks. When you create a new file, you must tell the system where to put the file and how big to make it. You must balance files across different disk units to provide good system performance. If you discover later that a file needs to be larger, you need to copy it to a location on the disk that has enough space for the new larger file. You may need to move other files to maintain system performance. The AS/400 system is responsible for managing the information in auxiliary storage. When you create a file, you estimate how many records it should have. The system places the file in the location most conducive for good performance. In fact, it may spread the data in the file across multiple disk units. When you add more records to the file, the system assigns additional space on one or more disk units. The system uses a function that is called virtual storage to create a logical picture of how the data looks. This logical picture is similar to how data is perceived. In virtual storage, all of the records that are in a file are together (contiguous), even though they may be physically spread across multiple disk units in auxiliary storage. The virtual storage function also keeps track of where the most current copy of any piece of information is (whether it is in main storage or in auxiliary storage). Single-level storage is a unique architecture of the AS/400 system that allows main storage, auxiliary storage, and virtual storage to work together accurately and efficiently. With single-level storage, programs and system users request data by name rather than by where the data is located. Disk storage architecture and management tools are further described in AS/400 Disk Storage Topics and Tools, SG24-5693. A.1 How disks are configured The AS/400 system uses several electronic components to manage the transfer of data from a disk to main storage. Data must be in main storage before it can be used by a program. 142 High Availability on the AS/400 System: A System Manager’s Guide Figure 38. Components used for data transfer Figure 38 shows the components that are used for data transfer. The components include: • Bus: The bus is the main communications channel for input and output data transfer. A system may have one or more buses. • I/O processor: The input/output processor (IOP) is attached to the bus. The IOP is used to transfer information between main storage and specific groups of controllers. Some IOPs are dedicated to specific types of controllers, such as disk controllers. Other IOPs can attach more than one type of controller (for example, tape controllers, and disk controllers). • Disk controller: The disk controller attaches to the IOP and handles the information transfer between the IOP and the disk units. Some disk units have built-in controllers. Others have separate controllers. • Disk unit: Disk units are the actual devices that contain the storage units. Hardware is ordered at the disk-unit level and each disk unit has a unique serial number. A.2 Full protection: Single ASP A simple and safe way to manage and protect your auxiliary storage is to perform the following tasks: • Assign all disk units to a single auxiliary storage pool (the system ASP). • Use device parity protection for all disk units that have the hardware capability. • Use mirrored protection for the remaining disk units on the system. With this method, your system continues to run even if a single disk unit fails. When the disk is replaced, the system can reconstruct the information so that no data is lost. The system may also continue to run when a disk-related hardware component fails. Whether your system continues to run depends on your B U S Processor Main Storage Input/Output Processor (IOP) Disk Controller Disk Unit Storage Unit Input/Output Processor (IOP) Disk Controller Disk Unit Storage Unit How your system manages auxiliary storage 143 configuration. For example, the system continues to run if an IOP fails and all of the attached disk units have mirrored pairs that are attached to a different IOP. When you use a combination of mirrored protection and device parity protection to fully protect your system, you increase your disk capacity requirements. Device parity protection requires up to 25% of the space on your disk units to store parity information. Mirrored protection doubles the disk requirement for all disks that do not have the device parity protection capability. Figure 39 shows an example of a system with full protection. The system has 21 disk units. All of the disk units are assigned to the system ASP. The system assigns unit numbers to each configured disk on the system. Notice that the mirrored pairs share a common unit number. Figure 39. Full protection: Single ASP A.3 Full protection: Multiple ASPs You may want to divide your disk units into several auxiliary storage pools. Sometimes, your overall system performance may improve by having user ASPs. For example, you can isolate journal receivers in a user ASP. Or, you can place history files or documents that seldom change in a user ASP that has lower performance disk units. You can fully protect a system with multiple ASPs by performing the following tasks: • Use device parity protection for all disk units that have the hardware capability. • Set up mirrored protection for every ASP on the system. You can set up mirrored protection even for an ASP that has only disk units with device parity System ASP Load Source 6602 Unit 1 Unit 1 Unit 3 Unit 3 Unit 2 Unit 2 Unit 4 Unit 4 Unit 1 6602 9336 9336 6602 6602 9336 9336 Unit 5 Unit 6 Unit 7 Unit 8 9337 Unit 13 Unit 14 Unit 16 9337 6603 Unit 11 Unit 10 Unit 15 Unit 17 Unit 12 Unit 9 Legend Unit n - Unit n = Mirrored pair Unit n Unit protected by device parity protection 144 High Availability on the AS/400 System: A System Manager’s Guide protection. That way, if you add units that do not have device parity protection in the future, those units are automatically mirrored. Note: You must add new units in pairs of units with equal capacity. Before configuring this level of protection, be sure that you know how to assign disk units to ASPs. Figure 40 shows an example of two ASPs. Both ASPs have device parity protection and mirrored protection defined. Currently, ASP 2 has no mirrored units. Figure 40. Full protection: Multiple ASPs A.4 Partial protection: Multiple ASPs Sometimes, full protection (using a combination of device parity protection and mirrored protection) may be too costly. If this happens, you need to develop a strategy to protect the critical information on your system. Your objectives should be to minimize the loss of data and to reduce the amount of time in which critical applications are not available. Your strategy should involve dividing your system into user ASPs and protecting only certain ASPs. Note, however, that if the system is not fully protected and an unprotected disk unit fails, serious problems can occur. The entire system can become unusable, end abnormally, require a long recovery, and data in the ASP that contains the failed unit must be restored. Before configuring this level of protection, be sure that you know how to assign disk units to ASPs. System ASP Load Source 6602 Unit 1 Unit 1 Unit 3 Unit 3 Unit 2 Unit 2 Unit 4 Unit 4 Unit 1 6602 9336 9336 6602 6602 9336 9336 Unit 5 Unit 6 Unit 7 Unit 8 9337 Unit 13 Unit 14 Unit 16 6603 Unit 10 Unit 11 9337 Unit 15 Unit 17 Unit 12 Unit 9 ASP 2 Legend Unit n - Unit n = Mirrored pair Unit n Unit protected by device parity protection How your system manages auxiliary storage 145 The following list provides suggestions for developing your strategy: • If you protect the system ASP with a combination of mirrored protection and device parity protection, you can reduce or eliminate recovery time. The system ASP, and particularly the load source unit, contain information that is critical to keeping your system operational. For example, the system ASP has security information, configuration information, and addresses for all the libraries on the system. • Think about how you can recover file information. If you have online applications, and your files change constantly, consider using journaling and placing journal receivers in a protected user ASP. • Think about what information does not need protection. This is usually information that changes infrequently. For example, history files may need to be online for reference, but the data in the history files may not change except at the end of the month. You could place those files in a separate user ASP that does not have any disk protection. If a failure occurs, the system becomes unusable, but the files can be restored without any loss of data. The same may be true for documents. • Think about other information that may not need disk protection. For example, your application programs may be in a separate library from the application data. It is likely the case that the programs change infrequently. The program libraries could be placed in a user ASP that is not protected. If a failure occurs, the system becomes unusable, but the programs can be restored. Two simple guidelines can summarize the previous list: 1. To reduce recovery time, protect the system ASP. 2. To reduce loss of data, make conscious decisions about which libraries must be protected. Figure 41 on page 146 shows an example of three ASPs. ASP 1 (system ASP) and ASP 3 have device parity protection and mirrored protection defined. Currently, ASP 3 has no mirrored units and ASP 2 has no disk protection. In this example, ASP 2 could be used for history files, reference documents, or program libraries. ASP 3 could be used for journal receivers and save files. 146 High Availability on the AS/400 System: A System Manager’s Guide Figure 41. Different protection for multiple ASPs System ASP Load Source 6602 Unit 1 Unit 1 Unit 3 Unit 4 Unit 2 Unit 2 Unit 12 Unit 12 Unit 1 6602 9336 9336 6602 6602 9336 9336 Unit 5 Unit 6 Unit 7 Unit 8 9337 Unit 13 Unit 14 Unit 16 6603 Unit 10 Unit 11 9337 Unit 15 Unit 17 Unit 12 Unit 9 ASP 2 ASP 3 Legend Unit n - Unit n = Mirrored pair Unit n Unit protected by device parity protection © Copyright IBM Corp. 2001 147 Appendix B. Planning for device parity protection If you intend to have a system with data loss protection and concurrent maintenance repair, plan to use one of the following configurations: • Mirrored protection and device parity protection to protect the system ASP. • Mirrored protection for the system ASP and device parity protection for user ASPs. • Mirrored protection and device parity protection to protect the system ASP and user ASPs. Note: You can use device parity protection with disk array subsystems as well as with input-output processors (IOP). For each device parity protection set, the space that is used for parity information is equivalent to one disk unit. The minimum number of disk units in a subsystem with device parity protection is four. The maximum number of disk units in a subsystem with device parity protection is seven, eight, or 16, depending on the type. A subsystem with 16 disk units attached has two device parity protection sets and the equivalent of two disk units dedicated to parity information. For more information about device parity protection, see Backup and Recovery, SC41-5304. B.1 Mirrored protection and device parity protection to protect the system ASP This section illustrates an example of a system with a single auxiliary storage pool (ASP). The ASP has both mirrored protection and device parity protection. When one of the disk units with device parity protection fails, the system continues to run. The failed unit can be repaired concurrently. If one of the mirrored disk units fails, the system continues to run using the operational unit of the mirrored pair. Figure 42 on page 148 shows an example of mirrored protection and device parity protection used in the system ASP. 148 High Availability on the AS/400 System: A System Manager’s Guide Figure 42. Mirrored protection and device parity protection to protect the system ASP B.2 Mirrored protection in the system ASP and device parity protection in the user ASPs You should consider device parity protection if you have mirrored protection in the system ASP and are going to create user ASPs. The system can tolerate a failure in one of the disk units in a user ASP. The failure can be repaired while the system continues to run. Figure 43 shows an example of a system ASP with device parity. Legend Unit n - Unit n = Mirrored pair Unit n Unit protected by device parity protection System ASP Load Source 6602 Unit 1 Unit 1 Unit 3 Unit 3 Unit 2 Unit 2 Unit 4 Unit 4 Unit 5 Unit 1 Unit 6 Unit 7 Unit 8 6602 9336 9336 6602 6602 9336 9336 9337 Unit 13 Unit 14 Unit 15 Unit 16 Unit 17 6603 Unit 9 Unit 10 Unit 11 Unit 12 9337 Planning for device parity protection 149 Figure 43. Mirrored protection in the system ASP and device parity protection in the user ASPs B.2.1 Mirrored protection and device parity protection in all ASPs If you have all ASPs protected with mirrored protection, to add units to the existing ASPs, also consider using device parity protection. The system can tolerate a failure in one of the disk units with device parity protection. The failed unit can be repaired while the system continues to run. If a failure occurs on a disk unit that has mirrored protection, the system continues to run using the operational unit of the mirrored pair. Figure 44 on page 150 shows an example of mirrored protection and device parity protection in all ASPs. System ASP Load Source 6602 Unit 1 Unit 1 Unit 3 Unit 3 Unit 2 Unit 2 Unit 4 Unit 4 Unit 5 Unit 1 Unit 6 Unit 7 Unit 8 6602 9336 9336 6602 6602 9336 9336 9337 Unit 9 Unit 10 Unit 11 Unit 12 9337 User ASP 2 Unit 13 Unit 14 Unit 15 Unit 16 Unit 17 6603 User ASP 3 Legend Unit n - Unit n = Mirrored pair Unit n Unit protected by device parity protection 150 High Availability on the AS/400 System: A System Manager’s Guide Figure 44. Mirrored protection and device parity protection in all ASPs B.2.2 Disk controller and the write-assist device The disk controller for the subsystems with device parity protection performs an important function for write operations. The controller keeps a list of all uncommitted data written to the write-assist device that has not been written to the data disk or the parity disk. Use this list during a power failure on the AS/400 system. Write requests and the write-assist device A write request to the subsystems with device parity protection starts three write operations. Data to be written to the disk units is first stored in the buffer in the disk controller. From this buffer, the data is sent to the write-assist device, the data disk, and the parity disk. The following actions occur during a write request: 1. A write operation to the write-assist device: Data is written to the write-assist device sequentially. A write operation to the write-assist device does not require parity calculation. The disk controller (identifier and disk address) adds the header information. Trailing information is added for the data before writing to the write-assist device. The header information can be used during a power failure. Normally, the write operations to the write-assist device are completed before the write operations to the disk units. The disk controller sends a completion message to storage management that allows the application to continue. The System ASP Load Source 6602 Unit 1 Unit 1 Unit 3 Unit 3 Unit 2 Unit 2 Unit 4 Unit 4 Unit 5 Unit 1 Unit 6 Unit 7 Unit 8 6602 9336 9336 6602 6602 9336 9336 9337 Unit 9 Unit 9 Unit 10 Unit 10 Unit 11 Unit 12 Unit 13 Unit 14 Unit 15 9337 6603 User ASP 2 Legend Unit n - Unit n = Mirrored pair Unit n Unit protected by device parity protection Planning for device parity protection 151 data that is written on the write-assist device is marked as uncommitted on the disk controller. Note: The write operation to the data disk and the parity disk continues in the background until the data is successfully written and is marked as committed in the disk controller. 2. A write operation to the disk unit. • For data, the operation: – Reads the original data – Writes the new data • For parity data, the operation: – Reads the original parity information – Compares the new data with the original data and the original parity to calculate the new parity – Writes the new parity information The write operation to the data disk usually completes before the write operation to the parity disk. The write operation to the data disk does not have to wait for the parity calculation. The delay between the writing of new data and the writing of the new parity information is known as delayed parity. 3. Data is marked as committed data when it is successfully written to both the data disk unit and the parity disk unit. 4. A completion message is sent to storage management only if the write operation on the write-assist device or the data disk unit has not already sent a message. The performance for this type of write operation depends on disk contention and the time that is needed to calculate the parity information. B.2.3 Mirrored protection: How it works Since mirrored protection is configured by ASP, all ASPs must be mirrored to provide for maximum system availability. If a disk unit fails in an ASP that is not mirrored, the system can’t be used until the disk unit is repaired or replaced. The start mirrored pairing algorithm automatically selects a mirrored configuration to provide the maximum protection at the bus, I/O processor, or controller level for the hardware configuration of the system. When storage units of a mirrored pair are on separate buses, they have maximum independence or protection. Because they do not share any resource at the bus, I/O processor, or controller levels, a failure in one of these hardware components allows the other mirrored unit to continue operating. Any data written to a unit that is mirrored is written to both storage units of the mirrored pair. When data is read from a unit that is mirrored, the read operation can be from either storage unit of the mirrored pair. Because it is transparent to the user, they don’t know from which mirrored unit the data is being read. The user is also not aware of the existence of two physical copies of the data. If one storage unit of a mirrored pair fails, the system suspends mirrored protection to the failed mirrored unit. The system continues to operate using the 152 High Availability on the AS/400 System: A System Manager’s Guide remaining mirrored unit. The failing mirrored unit can be physically repaired or replaced. After the failed mirrored unit is repaired or replaced, the system synchronizes the mirrored pair by copying current data from the storage unit that has remained operational to the other storage unit. During synchronization, the mirrored unit to which the information is being copied is in the resuming state. Synchronization does not require a dedicated system and runs concurrently with other jobs on the system. System performance is affected during synchronization. When synchronization is complete, the mirrored unit becomes active. © Copyright IBM Corp. 2001 153 Appendix C. Batch Journal Caching for AS/400 boosts performance In the year 2000, a Programming Request for Price Quotation (PRPQ) offering became available to improve the performance of an AS/400e system when journals are involved. It is known as Batch Journal Caching for AS/400 PRPQ, and the order number is 5799-BJC. It installs and runs correctly on any national language version. C.1 Overview The Batch Journal Caching for AS/400 PRPQ can provide a significant performance improvement for batch environments that use journaling. Benefits include: • It changes the handling of disk writes to achieve the maximum performance for journaled database operations. • By caching journal writes in main memory, it can greatly reduce the impact of journaling on batch run time by eliminating the delay in waiting for each journal entry to be written to disk. This PRPQ is an ideal solution for customers with batch workloads who use journaling as part of a high availability solution to replicate database changes to a backup system. C.2 Benefits of the Batch Journal Caching PRPQ Applications that perform large numbers of database add, update, or delete operations should experience the greatest improvement when this PRPQ is active. Although it is directed primarily toward batch jobs, some interactive applications may also benefit. Applications using commitment control should see less improvement because commitment control already performs some journal caching. With traditional non-cached journaling in a batch environment, each database record added, updated, or deleted by the batch job causes a new journal entry to be constructed in main memory. The batch job then waits for each new journal entry to be written to disk to assure recovery. This results in a large number of disk writes. The Batch Journal Caching PRPQ provides the ability to selectively enable a new variation of journal caching. It changes the handling of disk writes to achieve the maximum performance for journaled database operations. Both the journal entries and the corresponding database records are cached in main memory, thereby delaying writing journal entries to disk until an efficient disk write can be scheduled. This prevents most database operations from being held up while waiting for the synchronous write of journal entries to disk. By more aggressively caching journal writes in main memory, it can: • Greatly reduce the impact of journaling on batch run time by reducing the delay in waiting for each journal entry to be written to disk. 154 High Availability on the AS/400 System: A System Manager’s Guide • Avoid the problems and costs associated with making application changes (such as adding commitment control) to improve batch performance in these environments. C.2.1 Optimal journal performance For optimal journal performance, many factors beyond using this PRPQ should be considered, including: • The number and type of disk units and disk controllers • Amount of write cache • Placement of journal receivers in user auxiliary storage pools (ASPs) • Application changes C.3 Installation considerations The prerequisites and limitations of the Batch Journal Cache PRPQ are identified here. C.3.1 Prerequisites This PRPQ runs under the latest levels of operating system. Install either: • OS/400 V4R5 with PTFs MF24863, MF24866, MF24870, and SF63192 • OS/400 V4R4 with PTFs MF24293, MF24626, and SF63186 C.3.2 Limitations This batch cache type of journaling differs from traditional journaling and can affect the recover ability of the associated database files. Because journal entries are temporarily cached in main memory, a few recent database changes that are still cached and not yet written to disk can be lost in the rare event of a severe system failure where the contents of main memory are not preserved. This type of journaling may not be suitable for interactive applications where single system recovery is the primary reason for using journaling. Also, it may not be suitable where it is unacceptable to lose even one recent database change in the rare event of a system failure in which the contents of main memory are not preserved. Batch journal caching is primarily intended for situations where journaling is used to enable database replication to a second system (for example, for high availability or Business Intelligence applications) under a heavy workload like batch processing. This also applies to heavy interactive work with applications that do not employ commitment control. This function can also be selectively enabled to optimize performance when running nightly batch workloads. It can then disabled each morning to optimize recoverability when running interactive applications. This can speed up some nightly batch jobs without sacrificing robust recovery for daytime interactive work. C.3.3 For more information After installing the PRPQ software product, read the README member of the README AS/400 file in the QBJC library for instructions on this product. Use DSPPFM FILE(QBJC/README) MBR(README) to display the file. For further information, contact your IBM marketing representative. © Copyright IBM Corp. 2001 155 Appendix D. Sample program to calculate journal size requirement D.1 ESTJRNSIZ CL program esj1: pgm dclf estjrnsiz/lastipl call qwccrtec /* retrieve last IPL information */ CPYSPLF FILE(QPSRVDMP) TOFILE(ESTJRNSIZ/LASTIPL) DLTSPLF FILE(QPSRVDMP) loop: rcvf monmsg msgid(CPF0864) exec(goto cmdlbl(endit)) if (%sst(&lastipl 103 17) *ne ’ ’) + then(chgdtaara lastipl %sst(&lastipl 103 17)) goto loop endit: call pfildtl endpgm D.2 NJPFILS RPGLE program FQPRINT O F 132 PRINTER OFLIND(*INOF) USROPN FPFILRPT O E Printer OFLIND(*IN90) D* Include error code parameter D/COPY QSYSINC/QRPGLESRC,QUSEC Dlstlib s 10A Dlstfil s 10A Dipltim s z Dtimipl ds D ccipl 2A D yyipl 2A D sep1 1A INZ(’-’) D mmipl 2A D sep2 1A INZ(’-’) D ddipl 2A D sep3 1A INZ(’-’) D hhipl 2A D sep4 1A INZ(’.’) D nnipl 2A D sep5 1A INZ(’.’) D ssipl 2A D sep6 1A INZ(’.’) D msipl 6A INZ(’000000’) Dlastipl ds D iplmo 1 2A D iplda 4 5A D iplyr 7 8A D iplhr 10 11A D iplmi 13 14A D iplse 16 17A Dtimestamp s z INZ(*SYS) Dqmbrovr s 1A Dqmbrfmt s 8A Dqmbrdovr s 9B 0 INZ(4096) Dnummbrs s 4B 0 Dnumobjs s 4B 0 Dobjtolist s 20 INZ(’*ALL *ALLUSR ’) DFIRST_ERR S 1 INZ(’0’) Dobj_count s 9 0 INZ(1) Dmbr_count s 9 0 INZ(1) Dobjspcnam s 20A INZ(’OBJECTS ESTJRNSIZ ’) Dmbrspcnam s 20A INZ(’MEMBERS ESTJRNSIZ ’) Dext_attr s 10A Dspc_name s 20A Dspc_size s 9B 0 INZ(1) Dspc_init s 1 INZ(x’00’) Dobjlstptr s * Dmbrlstptr s * Dobjspcptr s * Dmbrspcptr s * DARR s 1 BASED(objlstptr) DIM(32767) DRCVVAR s 8 DRCVVARSIZ s 9B 0 INZ(8) DARRm s 1 BASED(mbrlstptr) DIM(32767) DRCVVARm s 8 156 High Availability on the AS/400 System: A System Manager’s Guide DRCVVARSIZm s 9B 0 INZ(8) D* Common list header DOUSH0100 DS BASED(OBJSPCPTR) D OUSUA 1 64 User area D OUSSGH 65 68B 0 Size generic header D OUSSRL 69 72 Structure rel level D OUSFN 73 80 Format name D OUSAU 81 90 API used D OUSDTC 91 103 Date time created D OUSIS 104 104 Information status D OUSSUS 105 108B 0 Size user space D OUSOIP 109 112B 0 Offset input parm D OUSSIP 113 116B 0 Size input parm D OUSOHS 117 120B 0 Offset header secti D OUSSHS 121 124B 0 Size header section D OUSOLD 125 128B 0 Offset list data D OUSSLD 129 132B 0 Size list data D OUSNBRLE 133 136B 0 Number list entries D OUSSEE 137 140B 0 Size each entry D OUSSIDLE 141 144B 0 CCSID list ent D QUSCID 145 146 Country ID D OUSLID 147 149 Language ID D OUSSLI 150 150 Subset list indicat D OUSERVED00 151 192 Reserved D* Common list header DMUSH0100 DS BASED(MBRSPCPTR) D MUSUA 1 64 User area D MUSSGH 65 68B 0 Size generic header D MUSSRL 69 72 Structure rel level D MUSFN 73 80 Format name D MUSAU 81 90 API used D MUSDTC 91 103 Date time created D MUSIS 104 104 Information status D MUSSUS 105 108B 0 Size user space D MUSOIP 109 112B 0 Offset input parm D MUSSIP 113 116B 0 Size input parm D MUSOHS 117 120B 0 Offset header secti D MUSSHS 121 124B 0 Size header section D MUSOLD 125 128B 0 Offset list data D MUSSLD 129 132B 0 Size list data D MUSNBRLE 133 136B 0 Number list entries D MUSSEE 137 140B 0 Size each entry D MUSSIDLE 141 144B 0 CCSID list ent D MUSCID 145 146 Country ID D MUSLID 147 149 Language ID D MUSSLI 150 150 Subset list indicat D MUSERVED00 151 192 Reserved D* Structure for OBJL0200 DQUSL020002 DS BASED(objlstptr) D QUSOBJNU00 1 10 Object name used D QUSOLNU00 11 20 Object lib name use D QUSOBJTU00 21 30 Object type used D QUSIS01 31 31 Information status D QUSEOA 32 41 Extended object attr D QUSTD06 42 91 Text description D QUSUDA 92 101 User defined attr D QUSERVED22 102 108 Reserved D* File Definition Template (FDT) Header D* This section is always located at the beginning of the returned data. DQDBQ25 DS 4096 Header info D QDBFYRET 1 4B 0 Bytes returned D QDBFYAVL 5 8B 0 Bytes available D*QDBFHFLG 2 D QDBBITS27 9 10 Attribute bytes D QDBBITS1 9 9 D* QDBRSV100 2 BITS D* QDBFHFPL00 1 BIT D* QDBRSV200 1 BIT D* QDBFHFSU00 1 BIT D* QDBRSV300 1 BIT D* QDBFHFKY00 1 BIT D* QDBRSV400 1 BIT D* QDBFHFLC00 1 BIT D* QDBFKFSO00 1 BIT D* QDBRSV500 1 BIT D* QDBFHSHR00 1 BIT D* QDBRSV600 2 BITS D* QDBFIGCD00 1 BIT Sample program to calculate journal size requirement 157 D* QDBFIGCL00 1 BIT D QDBRSV7 11 14 reserved D QDBLBNUM 15 16B 0 # data members D*QDBFKDAT 14 D QDBFKNUM00 17 18B 0 D QDBFKMXL00 19 20B 0 D* QDBFKFLG00 1 D QDBBITS28 21 21 D* QDBRSV802 1 BIT D* QDBFKFCS02 1 BIT D* QDBRSV902 4 BITS D* QDBFKFRC02 1 BIT D* QDBFKFLT02 1 BIT D QDBFKFDM00 22 22 D QDBRSV1000 23 30 keyed seq ap D QDBFHAUT 31 40 public aut D QDBFHUPL 41 41 pref storage unit D QDBFHMXM 42 43B 0 max members D* Maximum Members (MAXMBRS) D QDBFWTFI 44 45B 0 max file wait time D QDBFHFRT 46 47B 0 FRCRATION D QDBHMNUM 48 49B 0 # members D QDBRSV11 50 58 reserved D* Reserved. D QDBFBRWT 59 60B 0 max recd wait time D*QDBQAAF00 1 D QDBBITS29 61 61 add’l attrib flags D* QDBRSV1200 7 BITS D* QDBFPGMD00 1 BIT D QDBMTNUM 62 63B 0 tot # recd fmts D*QDBFHFL2 2 D QDBBITS30 64 65 add’l attrib flags D* QDBFJNAP00 1 BIT D* QDBRSV1300 1 BIT D* QDBFRDCP00 1 BIT D* QDBFWTCP00 1 BIT D* QDBFUPCP00 1 BIT D* QDBFDLCP00 1 BIT D* QDBRSV1400 9 BITS D* QDBFKFND00 1 BIT D QDBFVRM 66 67B 0 1st supported VRM D QDBBITS31 68 69 add’l attrib flags D* QDBFHMCS00 1 BIT D* QDBRSV1500 1 BIT D* QDBFKNLL00 1 BIT D* QDBFNFLD00 1 BIT D* QDBFVFLD00 1 BIT D* QDBFTFLD00 1 BIT D* QDBFGRPH00 1 BIT D* QDBFPKEY00 1 BIT D* QDBFUNQC00 1 BIT D* QDBR11800 2 BITS D* QDBFAPSZ00 1 BIT D* QDBFDISF00 1 BIT D* QDBR11900 3 BITS D QDBFHCRT 70 82 file level indicato D QDBRSV1800 83 84 D QDBFHTXT00 85 134 file text descript D QDBRSV19 135 147 reserved D*QDBFSRC 30 D QDBFSRCF00 148 157 D QDBFSRCM00 158 167 D QDBFSRCL00 168 177 source file fields D* Source File Fields D QDBFKRCV 178 178 access path recover D QDBRSV20 179 201 reserved D QDBFTCID 202 203B 0 CCSID D QDBFASP 204 205 ASP D* X’0000’ = The file is located in the system ASP D* X’0002’-X’0010’ = The user ASP the file is located in. D QDBBITS71 206 206 complex obj flags D* QDBFHUDT00 1 BIT D* QDBFHLOB00 1 BIT D* QDBFHDTL00 1 BIT D* QDBFHUDF00 1 BIT D* QDBFHLON00 1 BIT D* QDBFHLOP00 1 BIT D* QDBFHDLL00 1 BIT 158 High Availability on the AS/400 System: A System Manager’s Guide D* QDBRSV2101 1 BIT D QDBXFNUM 207 208B 0 max # fields D QDBRSV22 209 284 reserved D QDBFODIC 285 288B 0 offset to IDDU/SQL D QDBRSV23 289 302 reserved D QDBFFIGL 303 304B 0 file generic key D QDBFMXRL 305 306I 0 max record len D FMXRL1 305 305A D QDBRSV24 307 314 reserved D QDBFGKCT 315 316B 0 file generic key D field count D QDBFOS 317 320B 0 offset to file scop D array D QDBRSV25 321 328 reserved D QDBFOCS 329 332B 0 offset to alternate D collating sequence D table D QDBRSV26 333 336 reserved D QDBFPACT 337 338 access path type D QDBFHRLS 339 344 file version/releas D QDBRSV27 345 364 reserved D QDBPFOF 365 368B 0 offset to pf speciD fic attrib section D QDBLFOF 369 372B 0 offset to LF speciD fic attrib section D QDBBITS58 373 373 D* QDBFSSCS02 3 BITS D* QDBR10302 5 BITS D QDBFLANG01 374 376 D QDBFCNTY01 377 378 sort sequence table D QDBFJORN 379 382B 0 offset to jrn D section D* Journal Section, Qdbfjoal. D QDBFEVID 383 386B 0 initial # distinct D values an encoded D vector AP allowed D QDBRSV28 387 400 reserved D*The FDT header ends here. D*Journal Section D*This section can be located with the offset Qdbfjorn, which is located in the FDT heade DQDBQ40 DS jrn section D QDBFOJRN 1 10 jrn nam D QDBFOLIB 11 20 jrn lib nam D*QDBFOJPT 1 D QDBBITS41 21 21 jrn options flags D* QDBR10600 1 BIT D* QDBFJBIM00 1 BIT D* QDBFJAIM00 1 BIT D* QDBR10700 1 BIT D* QDBFJOMT00 1 BIT D* QDBR10800 3 BITS D QDBFJACT 22 22 jrn options D* ’0’ = The file is not being journaled D* ’1’ = The file is being journaled D QDBFLJRN 23 35 last jrn-ing date D QDBR105 36 64 reserved D* Structures for QDBRTVFD D* Input structure for QDBRTVFD API header section DQDBRIP DS Qdb Rfd Input Parms D*QDBRV 1 1 varying length D QDBLORV 2 5B 0 Len. o rcvr var D QDBRFAL 6 25 Ret’d file & lib D QDBFN00 26 33 Format name D QDBFALN 34 53 File & lib name D QDBRFN00 54 63 Recd fmt name D QDBFILOF 64 64 File override flag D QDBYSTEM 65 74 System D QDBFT 75 84 Format type D*QDBEC 85 85 varying length D* Retrieve member information structure D*Type Definition for the MBRL0100 format of the userspace in the QUSLMBR API DQUSL010000 DS BASED(mbrlstptr) D QUSMN00 1 10 Member name D*Record structure for QUSRMBRD MBRD0200 format DQUSM0200 DS 4096 D QUSBRTN03 1 4B 0 Bytes Returned D QUSBAVL04 5 8B 0 Bytes Available D QUSDFILN00 9 18 Db File Name Sample program to calculate journal size requirement 159 D QUSDFILL00 19 28 Db File Lib D QUSMN03 29 38 Member Name D QUSFILA01 39 48 File Attr D QUSST01 49 58 Src Type D QUSCD03 59 71 Crt Date D QUSSCD 72 84 Src Change Date D QUSTD04 85 134 Text Desc D QUSSFIL01 135 135 Src File D QUSEFIL 136 136 Ext File D QUSLFIL 137 137 Log File D QUSOS 138 138 Odp Share D QUSERVED12 139 140 Reserved D QUSNBRCR 141 144B 0 Num Cur Rec D QUSNBRDR 145 148B 0 Num Dlt Rec D QUSDSS 149 152B 0 Dat Spc Size D QUSAPS 153 156B 0 Acc Pth Size D QUSNBRDM 157 160B 0 Num Dat Mbr D QUSCD04 161 173 Change Date D QUSSD 174 186 Save Date D QUSRD 187 199 Rest Date D QUSED 200 212 Exp Date D QUSNDU 213 216B 0 Nbr Days Used D QUSDLU 217 223 Date Lst Used D QUSURD 224 230 Use Reset Date D QUSRSV101 231 232 Reserved1 D QUSDSSM 233 236B 0 Data Spc Sz Mlt D QUSAPSM 237 240B 0 Acc Pth Sz Mlt D QUSMTC 241 244B 0 Member Text Ccsid D QUSOAI 245 248B 0 Offset Add Info D QUSLAI 249 252B 0 Length Add Info D QUSNCRU 253 256U 0 Num Cur Rec U D QUSNDRU 257 260U 0 Num Dlt Rec U D QUSRSV203 261 266 Reserved2 D* Record structure for data space activity statistics DQUSQD DS D QUSNBRAO 1 8I 0 Num Act Ops D QUSNBRDO 9 16I 0 Num Deact Ops D QUSNBRIO 17 24I 0 Num Ins Ops D QUSNBRUO 25 32I 0 Num Upd Ops D QUSNBRDO00 33 40I 0 Num Del Ops D QUSNBRRO00 41 48I 0 Num Reset Ops D QUSNBRCO 49 56I 0 Num Cpy Ops D QUSNBRRO01 57 64I 0 Num Reorg Ops D QUSNAPBO 65 72I 0 Num APBld Ops D QUSNBRLO 73 80I 0 Num Lrd Ops D QUSNBRPO 81 88I 0 Num Prd Ops D QUSNBRRK 89 96I 0 Num Rej Ksel D QUSNRNK 97 104I 0 Num Rej NKsel D QUSNRGB 105 112I 0 Num Rej Grp By D QUSNBRIV 113 116U 0 Num Index Val D QUSNBRII 117 120U 0 Num Index Ival D QUSVDS 121 124U 0 Var Data Size D QUSRSV107 125 192 Reserved 1 C* Set things up C EXSR INIT C* Start mainline process C* set pointer to first object C EVAL objlstptr = objspcptr pt to b1 of usrsp C EVAL objlstptr = %addr(arr(OUSOLD + 1)) pt to entry 1 C EVAL numobjs = OUSNBRLE C* process all entries C DO numobjs C EVAL libn = qusolnu00 C EVAL filn = QUSOBJNU00 C IF QUSEOA = ’PF ’ only PF types C EVAL QDBFALN = filn + libn C CALL ’QDBRTVFD’ get full details C parm QDBQ25 C parm 4096 QDBLORV C PARM QDBRFAL C parm ’FILD0100’ QDBFN00 C parm QDBFALN C parm ’*FIRST ’ QDBRFN00 C parm ’0’ QDBFILOF C parm ’*LCL ’ QDBYSTEM C parm ’*EXT ’ QDBFT C parm QUSEC C IF QUSBAVL > 0 160 High Availability on the AS/400 System: A System Manager’s Guide C MOVEL ’QDBRTVFD’ APINAM 10 C EXSR APIERR C END C* Have FD info, test for SRC versus DTA C testb ’4’ QDBBITS1 10 11 10 on = DTA C* 11 on = SRC C *in10 ifeq *on is a data file C setoff 10 C* is the file already journaled? C QDBFJORN ifgt 0 C* yes, get jrn info C eval QDBQ40 = %SUBST ( QDBQ25 C : QDBFJORN + 1 C : %SIZE( QDBQ40 )) C eval jrnnam = QDBFOJRN C eval jrnlib = QDBFOLIB C if QDBFJACT = ’0’ C eval jrnact = ’N’ C end C if QDBFJACT = ’1’ C eval jrnact = ’Y’ C end C else C eval jrnnam = *blanks C eval jrnlib = *blanks C eval jrnact = ’ ’ C end C* now get member list C EVAL spc_name = mbrspcnam C CALL ’QUSLMBR’ C parm spc_name C parm ’MBRL0100’ mbr_fmt 8 C parm QDBFALN C parm ’*ALL ’ mbr_nam 10 C parm ’0’ mbr_ovr 1 C parm QUSEC C IF QUSBAVL > 0 Any errors? C MOVEL ’QUSLMBR’ APINAM C EXSR APIERR C END C* resolve pointer C CALL ’QUSPTRUS’ C PARM SPC_NAME C PARM MBRSPCPTR C PARM QUSEC C* Check for errors on QUSPTRUS C QUSBAVL IFGT 0 C MOVEL ’QUSPTRUS’ APINAM 10 C EXSR APIERR C END C EVAL mbrlstptr = mbrspcptr C EVAL mbrlstptr = %addr(arrm(MUSOLD + 1)) C EVAL nummbrs = MUSNBRLE C DO nummbrs C EVAL mbrn = QUSMN00 C EVAL QDBFALN = FILN + LIBN C CALL ’QUSRMBRD’ C PARM QUSM0200 C PARM QMBRDOVR C parm ’MBRD0200’ QMBRFMT C parm QDBFALN C parm QUSL010000 C parm ’0’ QMBROVR C parm QUSEC C IF QUSBAVL > 0 C MOVEL ’QUSRMBRD’ APINAM 10 C EXSR APIERR C END C eval QUSQD = %SUBST ( QUSM0200 C : QUSOAI + 1 C : QUSLAI ) C* have detail info, now create data C eval rcdlen = qdbfmxrl C* calc seconds since IPL C timestamp subdur ipltim runsec:*S 10 0 C* calc ave ops per sec C QUSNBRIO add QUSNBRUO rcdops C eval rcdops = rcdops + QUSNBRDO Sample program to calculate journal size requirement 161 C QUSNBRRO00 add QUSNBRRO01 mbrops C rcdops IFGT 0 C mbrops ORGT 0 C rcdops div runsec avrrcdops C mbrops div runsec avrmbrops C avrrcdops add avrmbrops jrnsec C rcdlen add 155 jrnsiz C eval jrnsiz = (jrnsiz * jrnsec * 86400) / 1048576 C END C EXSR dodetail C eval avrrcdops = 0 C eval avrmbrops = 0 C eval jrnsec = 0 C eval jrnsiz = 0 C EVAL mbrlstptr = %addr(arrm(MUSSEE + 1)) incr to next ent C END C END C END C EVAL objlstptr = %addr(arr(OUSSEE + 1)) C END C* End mainline process C EXSR DONE C* * * Subroutines follow * * * C* INIT subroutine C INIT BEGSR C OPEN QPRINT C exsr wrthead C z-add 16 qusbprv set err code struct C to omit exceptions C* Does user space exist for OBJECT list? C eval spc_name = objspcnam C eval ext_attr = ’QUSLOBJ ’ C EXSR USRSPC C* Does user space exist for MEMBER list? C eval spc_name = mbrspcnam C eval ext_attr = ’QUSLMBR ’ C EXSR USRSPC C* Retrieve last IPL time derived from previous step C *DTAARA DEFINE LASTIPL LASTIPL 17 C IN LASTIPL C move iplyr yripl 2 0 C move iplyr yyipl C move iplmo mmipl C move iplda ddipl C move iplhr hhipl C move iplmi nnipl C move iplse ssipl C IF yripl > 88 C move ’19’ ccipl C ELSE C move ’20’ ccipl C END C MOVEL TIMIPL IPLTIM C* Fill the user space with object list C eval spc_name = objspcnam C call ’QUSLOBJ’ C parm spc_name C parm ’OBJL0200’ fmtnam 8 C parm objtolist C parm ’*FILE ’ objtype 10 C parm QUSEC C* Any errors? C IF QUSBAVL > 0 C MOVEL ’QUSLOBJ’ APINAM C EXSR APIERR C END C* Get a resolved pointer to the user space C CALL ’QUSPTRUS’ C PARM SPC_NAME C PARM OBJSPCPTR C PARM QUSEC C* Check for errors on QUSPTRUS C QUSBAVL IFGT 0 C MOVEL ’QUSPTRUS’ APINAM 10 C EXSR APIERR C END C ENDSR C* 162 High Availability on the AS/400 System: A System Manager’s Guide C* USRSPC subroutine C USRSPC BEGSR C* Verify user space exists C CALL ’QUSROBJD’ C PARM RCVVAR C PARM RCVVARSIZ C PARM ’OBJD0100’ ROBJD_FMT 8 C PARM SPC_NAME C PARM ’*USRSPC’ SPC_TYPE 10 C PARM QUSEC C* Errors on QUSROBJD? C IF QUSBAVL > 0 C IF QUSEI = ’CPF9801’ user space not foun C CALL ’QUSCRTUS’ create the space C PARM SPC_NAME C PARM EXT_ATTR 10 C PARM SPC_SIZE C PARM SPC_INIT C PARM ’*ALL’ SPC_AUT 10 C PARM *BLANKS SPC_TEXT 50 C PARM ’*YES’ SPC_REPLAC 10 C PARM QUSEC C PARM ’*USER’ SPC_DOMAIN 10 C* Errors on QUSCRTUS? C IF QUSBAVL > 0 C MOVEL ’QUSCRTUS’ APINAM 10 C EXSR APIERR C END C* else error occurred accessing the user space C ELSE C MOVEL ’QUSROBJD’ APINAM 10 C EXSR APIERR C END C END C ENDSR C* APIERR subroutine C APIERR BEGSR C* If first error found, then open QPRINT *PRTF C IF NOT %OPEN(QPRINT) C OPEN QPRINT C ENDIF C* Print the error and the API that received the error C EXCEPT BAD_NEWS C EXSR DONE C ENDSR C* DONE subroutine C DONE BEGSR C WRITE TTLSEP C WRITE TTLS C EVAL *INLR = ’1’ C RETURN C ENDSR C* WRTHEAD subroutine C wrthead begsr C WRITE AFT1 C WRITE AFT2 C WRITE AFT3 C WRITE AFT4 C ENDSR C* DODETAIL subroutine C DODETAIL BEGSR C IF *IN90 = *ON C EXSR WRTHEAD C EVAL *IN90 = *OFF C EVAL lstlib = *blanks C END C* C IF LSTLIB <> LIBN C WRITE AFMBR C EVAL LSTLIB = LIBN C GOTO GETOUT C END C IF LSTFIL <> FILN C WRITE AFNOLIB C EVAL LSTFIL = FILN C goto GETOUT C END C WRITE AFNOFIL Sample program to calculate journal size requirement 163 C GETOUT TAG C add jrnsec tjrnsec C add jrnsiz tjrnsiz C ENDSR OQPRINT E BAD_NEWS 1 O ’Failed in API ’ O APINAM O ’with error ’ O QUSEI D.3 Externally described printer file: PFILRPT A*%%*********************************************************************** A*%%TS RD 20010115 123614 SMBAKER REL-V4R4M0 5769-PW1 A*%%FI+10660100000000000000000000000000000000000000000000000000 A*%%FI 0000000000000000000000000000000000000000000000000 A*%%*********************************************************************** A R AFT1 A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SKIPB(001) A SPACEA(001) A 3 A DATE(*YY) A EDTWRD(’0/ / ’) A 86 A ’Journal size estimate’ A 170 A ’Page: ’ A +0 A PAGNBR A*%%*********************************************************************** A*%%SS A*%%CL 001 A*%%*********************************************************************** A R AFT2 A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A 80 A ’---------Record---------’ A 107 A ’--------Member--------’ A 131 A ’-----Journal Estimate-----’ A*%%*********************************************************************** A*%%SS A*%%*********************************************************************** A R AFT3 A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A 42 A ’Record’ A +3 A ’----------Journal----------’ A 82 A ’Number of’ A 97 A ’Average’ A 107 A ’Number of’ A 122 A ’Average’ A 131 A ’Average ops’ A 151 A ’MB per’ A*%%*********************************************************************** A*%%SS A*%%*********************************************************************** A R AFT4 A*%%*********************************************************************** 164 High Availability on the AS/400 System: A System Manager’s Guide A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A SPACEA(001) A 3 A ’Library’ A 16 A ’File’ A 29 A ’Member’ A 42 A ’Length’ A +3 A ’Name’ A +8 A ’Library’ A +5 A ’Act’ A 81 A ’Operations’ A 94 A ’per second’ A 107 A ’Operations’ A 119 A ’per second’ A 132 A ’per second’ A 154 A ’day’ A*%%*********************************************************************** A*%%SS A*%%CL 001 A*%%*********************************************************************** A R AFMBR A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A LIBN 10A O 3 A FILN 10A O 16 A MBRN 10A O 29 A RCDLEN 5S 0O 43 A EDTCDE(3) A JRNNAM 10A O +3 A JRNLIB 10A O +2 A JRNACT 1A O +3 A RCDOPS 11S 0O 80 A EDTCDE(3) A AVRRCDOPS 7S 1O 96 A EDTCDE(3) A MBROPS 10S 0O 107 A EDTCDE(3) A AVRMBROPS 7S 1O 121 A EDTCDE(3) A JRNSEC 11S 1O 131 A EDTCDE(3) A JRNSIZ 10S 3O 146 A EDTCDE(3) A*%%*********************************************************************** A*%%SS A*%%SN JRNACT x A*%%*********************************************************************** A R AFNOLIB A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A FILN 10A O 16 A MBRN 10A O 29 A RCDLEN 5S 0O 43 A EDTCDE(Z) A JRNNAM 10A O +3 A JRNLIB 10A O +2 A JRNACT 1A O +3 A RCDOPS 11S 0O 80 A EDTCDE(3) A AVRRCDOPS 7S 1O 96 Sample program to calculate journal size requirement 165 A EDTCDE(3) A MBROPS 10S 0O 107 A EDTCDE(3) A AVRMBROPS 7S 1O 121 A EDTCDE(3) A JRNSEC 11S 1O 131 A EDTCDE(3) A JRNSIZ 10S 3O 146 A EDTCDE(3) A*%%*********************************************************************** A*%%SS A*%%SN JRNACT x A*%%*********************************************************************** A R AFNOFIL A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A MBRN 10A O 29 A RCDLEN 5S 0O 43 A EDTCDE(Z) A JRNNAM 10A O +3 A JRNLIB 10A O +2 A JRNACT 1A O +3 A RCDOPS 11S 0O 80 A EDTCDE(3) A AVRRCDOPS 7S 1O 96 A EDTCDE(3) A MBROPS 10S 0O 107 A EDTCDE(3) A AVRMBROPS 7S 1O 121 A EDTCDE(3) A JRNSEC 11S 1O 131 A EDTCDE(3) A JRNSIZ 10S 3O 146 A EDTCDE(3) A*%%*********************************************************************** A*%%SS A*%%SN JRNACT x A*%%*********************************************************************** A R TTLSEP A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A 129 A ’--------------’ A 146 A ’-----------’ A*%%*********************************************************************** A*%%SS A*%%*********************************************************************** A R TTLS A*%%*********************************************************************** A*%%RI 00000 A*%%*********************************************************************** A SPACEB(001) A TJRNSEC 11S 1O 131 A EDTCDE(3) A TJRNSIZ 10S 3O 146 A EDTCDE(3) A*%%*********************************************************************** A*%%SS A*%%CP+999CRTPRTF A*%%CP+ FILE(ESTJRNSIZ/PFILRPT) A*%%CP+ DEVTYPE(*SCS) A*%%CP PAGESIZE(*N 192 *N ) A*%%CS+999CRTPRTF A*%%CS+ FILE(QTEMP/QPRDRPT ) A*%%CS+ DEVTYPE(*SCS) A*%%CS PAGESIZE(*N 132 *N ) A*%%*********************************************************************** 166 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 167 Appendix E. Comparing availability options This appendix can help you compare availability options so that you can make decisions about what to protect and how. Journaling, mirroring, and device parity protection are compared by the extent of data loss, recovery time, and performance impact. Recovery time by failure type, and availability options by failure type are identified. E.1 Journaling, mirroring, and device parity protection Table 4 compares several important attributes of journaling physical files, mirrored protection, and device parity protection, including how they affect performance, the extent of loss, and recovery time. Table 4. Journaling physical files, mirrored protection, and device parity attribute comparison E.2 Availability options by time to recover Table 5 shows which availability options can reduce the time needed to recover from a failure. The number of plus (+) signs in a column indicates an option’s impact on the time to recover compared to the other options. An option with more pluses has greater relative impact. Table 5. Availability options by time to recover Attribute Physical file journaling Mirrored protection Device parity protection Data loss after a single disk failure Loss of file data is determined by currency of backup None None Recovery time after a single disk unit failure Potentially many hours None to a few hours None to a few hours Performance impact Minimal to significant Minimal, except some read operations improve Minimal, except restore operations degrade Option DASD System Power loss Program failure Site loss Save operation + + + + + Journal files ++ ++ ++ + Access path protection ++ ++ ++ UPS +++ User ASPs ++ Device parity protection +++ Mirrored protection +++ Dual systems +++ + ++ 168 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 169 Appendix F. Cost components of a business case Creating an accurate business case for some IT applications is not trivial. This is certainly the case when justifying a high availability solution. Many of the benefits provided by high availability are intangible. To help you create a business case for improving the availability of an application, this appendix provides a list of costs (both for providing availability and those associated with outages) that can be used as part of that business case. Without a detailed study of your business, it is difficult to know whether an outage in your company will have a similar impact. Use this appendix as a guideline to justify the need for further study. F.1 Costs of availability The costs for providing an improvement in high availability are very intangible. The value of availability is much harder to ascertain. One of the first steps is to study the current availability statistics and understand which objective to improve. It is conceivable that a single faulty component, such as a local display, creates an invalid perception that the availability problem is within an application. Review sources of information, such as problem management reports, system logs, operator logs, and so on, to identify the outages over the past year. Verify this list with the application owner to ensure that you both have the same perception of the current availability. Next, identify and categorize the root cause for every outage, both planned and unplanned. From this list, identify the items with the highest impact on availability. It is only when you understand what is causing the application to be unavailable at a given moment that you can effectively create a plan to improve that area. Use this information to identify which causes of outages must be addressed to gain the availability improvement you desire. The plan is likely to include some change in processes, and it may also involve hardware and software changes. The following sections provide information on some of the contributing factors for costs. F.1.1 Hardware costs A component failure impact analysis can be done to identify the single point of failure and the components that, if lost, would have a serious impact on the application availability. The only way to provide continuous operations is to have redundancy for all critical components. Take the result of that study and discuss the results with the sponsors. There may be some identified components that are addressed as an expected upgrade process. Size and price the remaining components. At a minimum, consider the console hardware component. 170 High Availability on the AS/400 System: A System Manager’s Guide F.1.2 Software Just as redundant hardware is required to provide for continuous operations, redundant software is also required. Additional licenses of some programs may be required. Does the current application need to be updated to support the high availability solution? Is this the time to add this application to help manage operations and availability? Are additional licenses required? When evaluating costs, consider: • Application change control • Change and problem management • Utilities F.2 Value of availability The value of availability (or the cost of unavailability) is more difficult than arriving at the cost or providing that availability. For example, if you lose a network controller, what is the cost impact of the loss? This depends on many things: • Which applications do the users in the affected area use? If the application developers access a test system, the cost will be lower. • Which shift did the outage occur? What time? • How long did it take to recover? • How do you report availability? F.2.1 Lost business The amount of business lost because of an outage can vary from individual transactions, to the actual loss of a customer. If the amount of business transacted by your application is consistent, compare the average value of business with the amount of business transacted on the day an application outage occurred. It is difficult to tie the loss of a customer to an application outage. If you have a relatively small number of high-value customers, you probably have a close relationship with them and they may make you aware of why they moved their business elsewhere. If you are in the retail industry, it is unlikely that you can produce a definite figure for the number of customers lost due to application unavailability. One possible method, however, is to follow a series of application outages and determine if it was followed by a trend in the amount of repeat business. Either of these analysis methods require working with the application owner to obtain and record the required information. It is likely that a single outage may result in lost transactions, where a series of outages may result in lost customers. Therefore, the cost of each outage is also impacted by the frequency of outages. Cost components of a business case 171 F.3 Image and publicity As businesses become more computerized and visible on the Internet, electronic links between supplies and customers are becoming a standard. The availability of applications becomes more visible to those outside the company. With a click of the mouse, customers can go on to another source for their goods or services. Recurring application outages are known quickly by customers (existing and potential). This impacts your validity for winning new contracts or renewing existing ones. Poor availability leads to bad publicity, which is very difficult to rectify. To make matters worse, potential customers can be anywhere in the world. You should modify your view of the outage impact to reflect this. It is nearly impossible to assess the cost of poor publicity caused by poor availability. If you have a public relations department, they may be aware of existing negative publicity and should have some idea of the cost to improve the public perception of the company. F.4 Fines and penalties Fines and penalties imposed as a result of application unavailability is an objective number to obtain. In some industries, application availability is monitored by controlling bodies. Companies are expected to maintain a certain level of availability, for example, the airline industry. There are also moves within the financial world to encourage companies to maintain high levels of availability. Extended outages can lead to fines from the governing bodies. F.5 Staff costs There may be significant staff costs both during and after an outage. Depending on the application affected, prolonged outages can have a significant financial cost in lost productivity. For example, for an application controlling a factory production line, there could be many people sitting around not able to do their job. Overtime may need to be paid to catch up on target productivity. Identify the users of a given application, estimate the impact of an application outage on those people, and then multiply by their average salary per hour to provide a rough idea of the cost in terms of lost productivity. Factor in the overtime rate if lost productivity is made up by overtime to give a rough recovery cost. Add the cost of the IT staff involved in recovering from the outage, and factor in any additional availability hours that may be required by the end users to catch up on their work. F.6 Impact on business decisions Depending on the type of application, the cost of an application outage varies considerably. This depends on how timely the information must be and the type of data involved. The loss of a business intelligence application varies from very 172 High Availability on the AS/400 System: A System Manager’s Guide small to very large. For order processing in a factory working on just-in-time, the impact can be significant. If items are not ordered in time, the whole factory could halt production due to the lack of one key component. Work with the application owners to identify the impact of the application unavailability for a given amount of time. Identify both a worst and best case scenario, and cost of reach. Then, agree on realistic average cost. F.7 Source of information To obtain some data to help in these calculations, check these sources: • Application: If a business case was created for the application when it was first developed, there may be some cost benefit estimates in that document that are useful. Factor in the age of the document, however, to determine the applicability of the figures. • Disaster recovery: If your company has a disaster recovery agreement, it is likely that a business case had to be presented in relation to that expenditure. It should contain estimates of the impact of a system outage. Additionally, when the business case includes the application or system level, use those figures or extrapolate an idea of the financial cost of the loss of a given application. Speak to the developers of the disaster recovery business case to see where they obtained the financial impact information used to build the case. • Transaction values: Transactions for an average day can be recorded and used to assess the impact of an outage for an application. Record the number of transactions per day for each application. You can at least identify the change in the number of transactions if an outage occurs. If you can agree on an average value per transaction, this allows you to more readily estimate the financial impact of lost transactions. • Industry surveys: Data processing firms and consultants have produced studies over the years, with example costs for outages. Their reports can include a cost range per hour and an average cost per hour. F.8 Summary Every industry and every company has unique costs and requirements. Tangible outage costs are only one part of the equation. You may be fortunate enough to suffer no unplanned outages, and yet still require better application availability. To maintain competitiveness, if your competitors are offering 24 x 7 online service, you may have no choice but to move in that direction. This may be the case even if there is currently not enough business outside the normal business hours to justify the change by itself. Additional availability can give you access to a set of customers you currently do not address (for example, people on shift work who do their business in the middle of the night). Internet shopping introduces a completely new pattern of consumer behavior. Previously, once a customer was in your shop, they were more inclined to wait ten minutes for the system to come back than to get back in the car and drive even minutes to your competitor. Now shopping is done by clicking a computer mouse. If you have better availability than your competitors, you have the opportunity to pick up customers from competing sites while their system is down. Cost components of a business case 173 Some retail outlets switch to a credit authorization business when the first provider experiences any interruption. They switch to another provider once the next interruption happens. If your systems have better availability, you have the opportunity to pick up a competitor’s business when they experience an outage. If your customer support systems are available 24 x 7, you have flexibility to have fewer call center staff. Once your customers realize they can get service any time, the trend tend to favor fewer calls during the day with more in the off-peak hours. This allows you to reduce the number of operators required to answer the volume of calls at a given time. If you can spread the workload associated with serving your customers over a longer period of time, the peak processing power required to service that workload decreases. This can defer an additional expense to upgrade your system. Due to mergers or business growth, you may be required to support multiple time zones. As the number of zones you support increases, the number and duration of acceptable outage times rapidly decreases. A successful full business case includes these considerations, and others that are more specific to your company and circumstances. Most importantly, a successful availability project requires total commitment on the part of management and staff to work together towards a common goal. 174 High Availability on the AS/400 System: A System Manager’s Guide © Copyright IBM Corp. 2001 175 Appendix G. End-to-end checklist This chapter provides a guide to the tasks and considerations needed when planning a new high availability solution. It is not a definitive list and looks different for each customer, depending on the particular customer situation and their business requirements. Use it as a guide to help you consider factors influencing the success of a high availability solution. Note: A service offering is available from IBM to examine and recommend improvements for availability. Contact your IBM marketing representative for further information. G.1 Business plan The investment in a high availability solution is considerable. It is critical that this investment is reflected back to the Business Plan. This will ease the justification of the solution, and in the process will display the value of the information technology solution to the business. Does a valid business plan exist? • Tactical Plan Do you have a tactical plan? • Strategic Plan Do you have a strategic plan? G.1.1 Business operating hours Define the current operating hours of all parts of the business, no matter how insignificant. Suppliers and customers should also be included in this information. How long can the customer business survive in the extent of a systems failure? Describe the survivability of the various different parts of the business, and rank their criticality. • Business operating hours: – Current normal operating hours – Operating hours by application/geography – Planned extensions to operating hours – Extensions by application/geography • Business processes: Do business processes exist for the following areas: – Information Systems Standards – Geographic standards – Centralized or local support – Language – Help desk – Operating systems – Applications 176 High Availability on the AS/400 System: A System Manager’s Guide G.2 High availability project planning Major points for a successful project plan are: • Objective of the project: Prepare an accurate and simple definition of the project and its goals. • Scope of the project: Define the scope of the project. This definition will more than likely be broken down to several major sub-projects. • Resources: Are there sufficient resources to manage and facilitate the project? • Sponsorship: Is there an Executive Management sponsor for the project? • Communication: – Does the project have an effective communications media for the project? – Communications to the business, sponsors, and customers – Communications and collaboration of parties working on the project? • Cost management: – Is there a budget for the project? – Has the budget been established based on cost of outage? G.3 Resources Soft resources are a critical success factor. • Current skills: Do you have an accurate skills list? Do you have job specifications for the current skills? • Required skills: What skills are required for the new operation? • Critical skills: Do all these skills exist? Do you have a job specification for all critical skills? • Skills retention: – Vitality Are existing resources encouraged to maintain their skills currency? Is there a skills development plan for existing resources? – Loss from the department • Are you likely to lose resources as result of the project? • Are these critical skills? • Through lack of interest in new mode of operation? • Through learning skills valuable outside the business? G.4 Facilities Are the facilities listed in this sections available? • Testing the recovery plan – Do you have tested recovery plan? – Are there any activities planned during the project that could impact it • Types of planned and unplanned activities? End-to-end checklist 177 G.4.1 Power supply Ask the following questions about your power supply: • Reliability: – Is the local power supply reliable? – How many outages in the past year? – How many weather related? • Quality: Is the local power supply company committed to maintaining a quality service? • Power switchover: Do you require power switchover? • UPS: – Are there any UPSs in the installation? – Do they qualify with the particular power supply options? – Do they have the capacity to meet the new demands? • Servers: – Is a UPS required for the Servers? – How many? – What type (battery/generator)/size/duration of backup supply/location? – Are the systems aware of power failure? – Do programs need to be developed to allow the systems to reaction to power failures? • Clients: – Do clients need to have UPS? – What type (battery/generator)/size/duration of backup supply/location? • Other equipment: – Is there other equipment that needs UPS? – Consoles/Printers/PABX/Network Devices/Machine facilities? • Generated supply: – Will you be running a generated backup? – What configuration will be running? – Local power - generated backup – Generated power - local backup G.4.2 Machine rooms Ask the following questions regarding machine rooms. • Flooring: – What is the current standard for your machine room? – Fully accessible or semi-accessible • Fire Protection: – What is the current fire protection method? – Halon/Water/CO2 • Air-conditioning: – Is the machine room air-conditioned? – At what point will the equipment fail without air-conditioning? – Will the machine room ever achieve this condition with-out air-conditioning? 178 High Availability on the AS/400 System: A System Manager’s Guide – Is there spare capacity in the air-conditioning system? – Is there redundancy in the current air-conditioning? • Contracts: – Do contracts and services levels exist for machine rooms? – Do these need amending for the new system? G.4.3 Office building Ask the following questions regarding your office building: • Workstations: – Have the workstations been assessed for their ergonomic function? – Will the changes to the current system effect your liability for your employees ergonomics? • Cabling systems: – Does the facility have a structured cabling system? – Will the existing cabling system accommodate the new system? • PABX: – Is the current PABX capable of operating in the new environment? – Does the existing PABX have redundancy? – Is redundancy in the PABX a requirement? G.4.4 Multiple sites Answer the following questions regarding multiple sites. • Are there multiple local sites (within the same site)? • Are any of the local sites located across a public space? (highway, area) • Are any remote sites less than 2km away? • Are any remote greater than 2km away? • Are any sites across an international border? • Do the remote sites have a different PTT? G.5 Security In this section, answer the following questions regarding security: • Policy: – Does a detailed security policy exist? – Is there a site security function? – Does this function set policy for I/T security? • Physical security: – Is access protection provided? – Is there a documented process for dismissal employees? – Does this process also limit risk to systems prior to dismissal? • Office space: Is the office space protected? End-to-end checklist 179 • Machine room: – Is the machine room protected? – Does the protection system provide an audit trail? • Site: – Does the site have physical security? – Will the new system require this to be updated? • Operating system security: – Do the systems implement security? – What level is implemented? – What level is required? – What applications at what level? • AS/400 security levels: – What level of AS/400 security is implemented? – What level is required? • Personal computer security: – What forms of PC security have been implemented? – Do these levels need changing? • Remote users: – What type of remote security has been implemented? – How do remote users access the systems? – Intranet? – Internet? • Network security: What network security is in place? • Printing: – Are there secure printing requirements? – Are the printers producing secure output located in a controlled environment? G.6 Systems in current use Document the model, processor feature, interactive feature, main storage, DASD capacity, DASD arms, protection, and ASP. G.6.1 Hardware inventory Prepare the inventory list of the all the hardware components in your current system, including: • Processors • DASD subsystems: DASD Space available • Mirroring • RAID-5 • Third Party solutions 180 High Availability on the AS/400 System: A System Manager’s Guide • Tape subsystems – Multiple tapes subsystems versus tape library – Data transfer rate • IOPs and Towers G.6.2 Redundancy What level on redundancy are you planning? • System (cluster) • Tower • Bus • I/O processor • Controller • Adapter G.6.3 LPAR • Are you planning to use LPAR support for the project? • Transition support with LPAR? • Server consolidation support with LPAR? G.6.4 Backup strategy Even with a continuously available solution backups are necessary. Is there a new backup strategy in plan? Does this plan include the following components? • Media Management • Media storage • Automated backup • Performance • Save-while-active • Incremental backups • Disaster recovery backup G.6.5 Operating systems version by system in use • Interactive CPU, batch CPU, DASD utilization, network utilization LAN/WAN. • Are there multiple operating systems versions in this plan? • Do you plan to rationalize these into the same version? • Is there a plan for bringing systems to the same version G.6.6 Operating system maintenance • Servers: Do you have maintenance contracts for your server operating systems? • Clients: Do you have maintenance agreements in place for your client software? G.6.7 Printers • Do you have a list of your printer inventory? • Are the printers supported with a maintenance agreement? • Will the printers support the new environment? End-to-end checklist 181 G.7 Applications in current use Inventory of server based applications • Application provider • Version • Number of users • Database size in MB • Data transfer requirements Inventory of client based applications • Application provider • Version • Number of users • Database size in MB • Data transfer requirements Application maintenance • Support contracts – Do support contract exist for all applications? – What is the support level? – Are there react time guarantees? – Does this meet the new availability requirements? • Frequency of update – Does the application have updates? – What is the frequency of update? G.7.1 Application operational hours current Ask youself the following questions: • By application what are the operational hours? • Are there any processing peak periods that extend these operating hours? Application processing peaks. Application processing map An application processing map shows the time plan of when various applications run on both server and client machines. It shows the interleaving of applications and should identify any periods that have an opportunity for very high or very low requirement. • Interactive • Batch • During the day • Day end • Month end • quarter end • year end • fiscal year end Application operational hours requirement: 5x8, 7x24, 24x365 What are the assessed operational requirements of each application? 182 High Availability on the AS/400 System: A System Manager’s Guide Growth information; applications, database, users What is the growth information for each application and its associated database and users? Splitting an application Is the splitting of the application across multiple machines a consideration? © Copyright IBM Corp. 2001 183 Appendix H. Special notices This publication is intended to help customers, business partners, and IBMers who are looking to implement a high availability solution for their AS/400 system. It provides planning checklists and background information to understand the facets of planning for a high availability solution, implementing a high availability solution, and managing the project installation itself. The information in this publication is not intended as the specification of any programming interfaces that are provided by OS/400. See the PUBLICATIONS section of the IBM Programming Announcement for OS/400 for more information about what publications are considered to be product documentation. References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product, program or service. Information in this book was developed in conjunction with use of the equipment specified, and is limited in application to those specific hardware and software products and levels. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact IBM Corporation, Dept. 600A, Mail Drop 1329, Somers, NY 10589 USA. Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. The information about non-IBM ("vendor") products in this manual has been supplied by the vendor and IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating 184 High Availability on the AS/400 System: A System Manager’s Guide environments may vary significantly. Users of this document should verify the applicable data for their specific environment. This document contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples contain the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. Reference to PTF numbers that have not been released through the normal distribution process does not imply general availability. The purpose of including these reference numbers is to alert IBM customers to specific information relative to the implementation of the PTF when it becomes available to each customer according to the normal IBM PTF distribution process. The following terms are trademarks of the International Business Machines Corporation in the United States and/or other countries: The following terms are trademarks of other companies: C-bus is a trademark of Corollary, Inc. in the United States and/or other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and/or other countries. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States and/or other countries. PC Direct is a trademark of Ziff Communications Company in the United States and/or other countries and is used by IBM Corporation under license. ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States and/or other countries. (For a complete list of Intel trademarks, see http://www.intel.com/tradmarx.htm) UNIX is a registered trademark in the United States and/or other countries licensed exclusively through X/Open Company Limited. SET and the SET logo are trademarks owned by SET Secure Electronic Transaction LLC. Other company, product, and service names may be trademarks or service marks of others. e (logo)® IBM  Redbooks Redbooks Logo APPN AS/400 AS/400e AT CT Current DataJoiner DataPropagator DB2 DRDA Netfinity Nways OS/2 OS/400 RPG/400 SAA SP System/36 System/38 XT 400 Lotus Lotus Notes Notes Tivoli