The purpose of this blog is to highlight how excessive frugality and lack of IBM i (iSeries/AS400) expertise can have grave consequences for users. Over the decades I have too frequently encountered unfortunate circumstances of costly disruption and lost critical data due to inadequate support and safeguards. This is just one recent example.
Disclaimer: iSeries and AS400 are servers. IBM i is an operating system. I use these terms interchangeably to make it easy for folks to find this information when searching on the web.
I work with hundreds of SMB (small to medium) and family businesses that run their business with IBM i POWER servers (AS400/iSeries). They are commonly frugal. Frugality is part of their strategy to beat competitors as well as remain profitable.
Even so, frugal choices can sometimes yield bad outcomes.
I recently spoke a New York-based IBM i administrator whose company transitioned from their IBM POWER5 server to a new ERP system.
He explained that to save money they bought a used 9406-520 with IBM i V6.1 in 2010.
Even though the new ERP system is functional, they still need access to historical data from that 9406-520.
Fast forward to January 2020, the 9406-520 had 2 hard disks fail that took down the server for about 2 weeks. While a non-IBM maintenance company was able to replace the failed disk units, when it came time to restore the system the most current Save21 backup on hand was 10 years old. Yikes!
(Like many others with the title, this administrator in charge of the care and feeding of the IBM i, had very little actual IBM i training. My sense is that he had already had a disk failure in a RAID environment and did not understand the console error message. So when the second disk drive failed, the system was “toast.” Had he responded to the first disk error message, this lost system would most likely not have happened.)
After restoring the Save21 and their most current daily saves, he discovered his company had lost the entire Accounts Payable file for the last 10 years. YIKES!!
He confessed that he wants to move his IBM i system to the cloud because neither he nor anyone else in the company have IBM i expertise to performance regular and complete backups.
He is faced with a very expensive forensic data accounting challenge to rebuild 10 years of lost accounts payable information to satisfy auditors and respond to compliance issues. YIKES!!!
As dramatically scary as this recent scenario is, I hear painful outcomes like this about 2-3 times per year.
What is missing in most all cases is proper IBM i administrative documentation that includes:
1) More frequent full system saves (Save21),
2) Daily backups that are properly scheduled, logged and saved,
3) Testing backups to make sure the media is reliable and the backups can be restored.
Ideally, I would like to see these users on IBM hardware and software support with the most current supported version of the IBM i OS.
In addition, in cases like this I would like the IBM i administrator to understand how to read console error messages. My sense is that the console had been sending error messages of predictive disk failure notices that went unheeded.
How does stuff like this happen?
What I generally see is that the former smart IBM i (AS400/iSeries) expert (IT Manager or Administrator) set the system up 10-20 years ago and ultimately moved on. The next 1-2 generations of IBM i administrators learned the basic daily routines to keep the server running without an understanding of the IBM i OS, console error messages, and how to prevent unexpected downtime.
Further, one of the strengths of IBM POWER servers – exceptional reliability – lulls users to believe that this server will run trouble free forever.
I wish these frugal, naïve users would have different practices to avoid these terrible outcomes. Despite my efforts to share how to better support their system, my advice falls on deaf ears – or the owners are unwilling to listen to the concerns of their operators
Either way, I wish it were different.
Need help to avoid situations like this. Please email me at blosey@source-data.com or call me at 714-593-0387. There are simple, basic steps you can take to avoid these unfortunate outcomes.
Leave a Reply