High Availability and Production Redundancy

What It Means to Your Credit Union

Since 2002, CU*Answers has been investing significantly in hardware and other resources in order to create an environment where redundancy is part of production-a constant state of readiness to replace our primary operating resources with stand-in and backup resources.

This is more than just crisis management. The world has become a 24x7 up-time environment. We encourage consumers and CU members to count on services that are ready when they need them, and to rely on this instant access. Many organizations are starting to base their reputations on being part of the customer's lifestyle. This trend will not reverse itself and therefore is key to long-term success for our credit unions and for CU*Answers.

Therefore, starting with its 2003 Business Plan, CU*Answers has consistently built redundancy into its normal production cycles. The first step was to build High Availability capability for the iSeries to allow us to retain Production iSeries support even while undergoing routine equipment management such as hardware maintenance, operating system changes, facility changes, and the like.

What is a Production Redundancy Rollover?

In a nutshell, a second iSeries system is kept in a constant state of readiness so that the entire CU*Answers network and all of its TCP/IP components can be switched to run on the backup system. Because the backup system constantly mirrors all activity on the main system, this cut-over means no loading of backup tapes to recover operation—there is a smooth, seamless switch from one physical iSeries system to the other using the freshest data possible.

This is not a stand-in routine. This is the full production environment where all TCP/IP network components are redirected to a matching iSeries computer. Tellers use live files in CU*BASE; members see true, real-time balances in It’s Me 247 and CU*TALK, and all other TCP/IP-ready interfaces work as usual.

A key component of effective High Availability is that it must be considered a normal operations mode, with regularly scheduled rollovers not only for special situations, but also as proof of our redundant readiness.

Another benefit to the High Availability system is that during rollovers we can perform operating system upgrades and other maintenance on our main Production iSeries. This previously would have required that the entire system be taken off line, without even stand-in services. With High Availability, we can now continue providing normal Production iSeries support even during many different types of maintenance activities.

Are All Network Services Rolled?

No. Only third-party vendors that have worked with us to allow TCP/IP interfaces can be rolled to the backup system. CU*Answers services such as CU*BASE, It’s Me 247, CU*TALK, CU*EasyPay!, and TCP/IP ATM switches such as C0-Op and NYCE already allow the switch.

At present, there are still a few vendors, including some ATM/Debit card switches and on-line check ordering systems that have not yet made the switch to TCP/IP. Therefore, these systems will operate in an off-line, stand-in mode during any rollover to our backup system.

Two New iSeries Systems Installed

In January, 2007, CU*Answers installed two brand new IBM i5 Model 550 computers, replacing both our existing Production and High Availability units. The i5 Model 550 represents the latest IBM iSeries technology available in the marketplace, and positions us very well for the next several years of client and application growth. Over the past decade, we have come to expect the highest level of performance and reliability from our IBM servers, and we are confident the i5 Model 550’s will be rock-solid like all of their predecessors.

Our Progress So Far

Initial implementation was completed during December, 2002, with our first true rollover on Sunday, December 15, 2002. Our staff was excited to watch teller transactions post and to view ongoing home banking and audio response activity. Moving the entire network across required an interruption of 30 minutes.

We subsequently have performed numerous High-Availability (HA) role-swaps on weekends as well as highly successful mid-week rollovers, the first of which took place on May 26, 2004.

The next milestone was to move the two iSeries systems in 2005 to separate physical data centers, thereby providing a new level of business interruption protection for all CU*Answers credit union clients. This was a major investment for our CUSO, and one that signaled a real step up for the types of organizations we will be running as the 24x7 world becomes more a reality for all of us.

In 2006, we began the making further improvements by streamlining the rollover process. The first step was to automate the exchange of IP addresses necessary to complete a High Availability rollover. This smoothed the process and significantly reduced the time required to perform exchanging activity from the production system to the High Availability system and back again.

In 2007, we implemented a virtual role-swap practice program and performed four HA rollovers as planned; then in 2008 we demonstrated both the readiness of the HA team and effectiveness of the project.

On 2/21/08, we discovered that an important component of our production system was failing and required replacement. In the past, this would have resulted in system down-time and possible delays in processing. Thanks to the commitment of our clients in the HA program, we rolled to the HA system (in fact, remained swapped for an additional period when IBM encountered repair issues) and carried on with imperceptible impact to users.

Along with additional planned role-swaps, we also performed an extended rollover at the end of September, 2008 spanning an entire week of processing. It also spanned processing for the end of the quarter during which all month-end processing and member statement production activities were originated from the HA rollover center.

What's Next?

For details on our most recent HA program activities, check out the results of our last rollover test. Remember to include the results of our rollovers in your Board Meeting minutes and as a part of your due diligence and contingency planning documentation!

We will continue to improve and expand the high availability program. We are steadily working on new tools to improve the rollover process and to increase the quality of the HA application.

See the schedule of upcoming High Availability rollover tests for information on the dates planned for our role-swap activities. We will continue to provide updates on our ongoing efforts to provide the best system redundancy available for our clients.

What can my credit union do?

This is not just a CU*Answers initiative. Every credit union should be planning for a new level of redundant services in today's financial service world. How can you ensure that everything that is changing at your credit union on a daily basis is backed up right to the minute so that electronic services offered after hours can be recreated quickly and without manual re-posting? Imagine losing an entire day's work in today's world and having to recreate it after the fact.

First and foremost, begin reviewing all of your processes and how they fit in with constantly being ready to respond to member needs based on redundant and alternative systems. Are you ready to lose a branch? What will your vendors do if they have to route information to a secondary location? Where would your staff work? Do your members know how you would respond on their behalf?

In other words, join CU*Answers in preparing for a fluid environment that is constantly ready to serve your members. Talk with your staff and your Board about this effort and have a little patience as we go through the steps and the learning curve in creating a network that goes far beyond what your financial service competition is offering their customers or members.

When the CU*Answers production redundancy system began actively supporting constant rollovers and real production processing time in 2004, all credit unions took a step forward in serving their members both today and in the future. Every credit union's business plan should include this initiative.

If you would like to learn more about what production redundancy means to your credit union, your Board, and your members, please feel free to contact Randy Karnes at ceo@cuanswers.com or at 616-285-5711 ext. 101.