Client/Server Under Glass by Diane Fromme

One of the most successful competitors in "Steeltown" is Dofasco, Inc. From modest beginnings a little over 80 years ago, the $2.6 billion company has grown to be one of the major integrated steel producers in North America, with a record of innovation in both process and product technology.

But toughing it out in the industry that has been the heart and soul of the city has been anything but easy, particularly as new technology and global competition have created a sea change in steel-making in recent years.

Never one to lag, Dofasco has responded to these changes by turning to its information systems operation for greater efficiencies. The key to its success has been a very aggressive switch to a distributed client/server environment, which took place from 1993 to early 1996. "We have moved 70% of our mainframe cycles to client/server, with uptime as good and better than that on the mainframe," says Eric Wilson, General Manager of IT Services.

Leading analysts predict a maximum uptime of 95% for client/server systems, yet Dofasco has achieved what Wilson believes to be over 99% availability. How did Dofasco meet the monumental challenge of re-engineering from mainframe to client/server with such excellent results? With 13 manufacturing business units spread out over an 800-acre campus and running round the clock, a carefully planned systems management infrastructure and tight IT team collaboration were essential to their success. "We didn't have the luxury of 'going down' to make this transition," says Wilson.

REFINING DOFASCO'S BUSINESS

Several industry trends fueled the FIP. "We were looking at an oversupply of steel to a market mature enough that steel consumption had leveled off," says Grigsby. "In addition, Dofasco was facing competition from a new breed of low-cost steel mill. The mini mill, which produces steel from steel scrap instead of raw ore, penetrated some of our lower-grade markets."

Dofasco closed down some production facilities that weren't going to be competitive over the long term. By 1993, the company had gotten significantly smaller, reducing its staff from 12,500 to 7,000.

While the restructuring efforts were taking shape, it became obvious that a new technology infrastructure was necessary. "Company surveys indicated that to do more with less, end users wanted to derive more value from information systems - they wanted IS to be more efficient," says Wilson.

Dofasco's FIP and end-user demands put the company in a ready position for IS change. "This presented a chance to redo our entire application system, which in turn forced us to look at our computing infrastructure," says Wilson. "Our VMS and MVS mainframes had served their purpose in the past, but they could not be changed easily enough or become more flexible to meet current IS demands."

The company was experiencing several classic mainframe issues. "We had a lot of data integrity problems, with databases all over the place," says Wilson. "In addition, our businesses were dealing with different flavours of systems infrastructure. The bottom line was we had a very unsatisfied user community."

This re-engineering opportunity allowed Dofasco's IT team to ask which platforms would allow the company to progress successfully into the future. 'The basic question became, should we stick with the mainframe or move to something else?" says Grigsby. "We wanted to do this once and only once, because the business and financial implications of rebuilding this infrastructure again were too great."

During the last quarter of 1992, Grigsby finally put a stake in the ground. Dofasco would build its new system in an open, distributed client/server environment. "I fundamentally could not see rebuilding Dofasco's information systems in the mainframe world," says Grigsby. "We needed an infrastructure with which we could migrate into the future."

In addition, the demands of a changing business required upgrading of all Dofasco's key applications, including logistics, human resources (HR), maintenance management, payroll and finance. "Each of the business units needed changes and improvements quickly," says Grigsby. "Due to these demands, we could not deal with the applications in a sequential manner, but rather had to take on several projects in parallel. We had to bite the bullet and do this all at once," he says.

INFORMATION TECHNOLOGY GOALS

"From a business perspective, we had a primary goal of getting our data house in order," says Wilson. "We had to establish a standard, corporate data architecture as opposed to a myriad of individual data models. Without corporate computing standards and change management, application and infrastructure changes were thrown over the fence for someone to put in place," says Wilson. "In the past, this procedure mainly resulted in inefficiencies. In the future it would become a serious obstruction to the future rollout of modern networks and client/server-based applications."

Dofasco's secondary IS goal involved increasing the cycle time for development changes to its key business area applications. "Our business applications were homegrown. Now we had the chance to buy current versions of off-the-shelf applications, and update them only when new versions are delivered. In fact, we made a 'buy, not build' decision with a goal of buying 80% of all applications."

From a management perspective, Wilson describes Dofasco's IS vision as a distributed, client/server environment managed with a glass-house mentality. "Everyone thought the glass house was so good in the 80s. These practices weren't all applicable to the client/server model, so we wanted to create the 90s version of the glass house," says Wilson.

RESIDENT EXPERTISE

Grigsby describes Dofasco's IT area as a three-legged stool. "Each leg of the stool is important in order to keep it level," he says. "Our application development group focuses on solutions. Our infrastructure services group keeps everything running. And our business support group works with our clients to ensure we're defining the right products for them, delivering those products and maximizing the value of those products."

Dofasco tried to capitalize on the strengths of its own people. "I'm not an advocate of pushing mainframe people out the door while you're bringing client/server people in the other door," says Grigsby. "At the end of the day, all you really have is your people. If you haven't developed the right working relationships with them, technology is not going to do it for you."

Tom Hilbig, Dofasco's Systems Management Project Manager, agrees. "These are the people that are going to be left with the process and the technologies once the project is done," he says. "It's important to involve them early in the decisions, so that they take ownership of the new system and its results." This team-oriented approach, which Dofasco later augmented with the help of a systems management consultant, encouraged IT employees to be proponents of the new system instead of skeptics. "We married the skills and experience of our distributed process control experts with the discipline of our mainframe team. After all, if you can figure out MVS, you can figure out anything," says Wilson.

BUILDING THE GLASS HOUSE

Dofasco's CIO set high standards for the IT turnover. "We set a goal of establishing a network-based, client/server environment that would run more effectively than a mainframe environment," says Grigsby. "We had to be better than the old world was." That challenge made Dofasco's IT team look long and hard at how to achieve such results.

"It boiled down to two clear strategies," says Wilson. "My team had to partner with competent platform suppliers, and we had to implement a distributed systems-management project to control the environment and maximize uptime."

Dofasco had started picking its new infrastructure and database platforms by early 1993, following the "buy, not build" rule. On the operating system side, Dofasco wanted the platform that offered the most choices or solutions. "Following this criterion, the choice became either a proprietary operating system such as VMS or MVS, or one of the many UNIX systems," says Wilson. "Since we wanted to get away from proprietary environments and move to open systems, we decided to go with a UNIX system. Since a mainstream vendor was important to us, we selected HP's flavor of UNIX.

On the hardware side, Dofasco implemented 15 HP 9000 business servers supporting about 3,500 PC systems. On the database side, Dofasco chose Oracle. "Oracle is a large company that would still be there when we were done with our mainframe to client/server transition," explains Wilson.

"These were clearly business-driven decisions," says Wilson. "We were looking for companies who could be our commercial partners for mission-critical business applications."

After choosing its vendors, Dofasco documented the piece of its IT plan that described how the team would move and reengineer 70% of its mainframe applications into the new environment. As part of this plan, the company separated its vertical applications from its horizontal applications, user interfaces, and the "glue" between systems (i.e., the infrastructure to manage the project). "We had two activities going on in parallel, creating a time-clock by which the systems management infrastructure had to be in place before our vertical applications came live," says Wilson. "We knew that without distributed systems management we might have serious application availability issues, which would not be acceptable to our customers."

SYSTEMS MANAGEMENT PROJECT

With a continued predilection to buy instead of build, Dofasco looked for an off-the-shelf systems management solution. "We could not find any well-documented case studies on how to implement a large client/server environment," says Hilbig. "At this point we became disillusioned regarding existing solutions- a silver bullet here was a pipe dream."

"Many vendors were eager to work with us, but no one had a solution when we needed it. We couldn't wait around on vendor promises," says Wilson. "We had two re-engineering efforts running in parallel, and our application development team wouldn't slow down for us. That's when we decided to build our own systems management infrastructure."

In the summer of 1993, Dofasco asked for help. The firm sent out RFPs to three different companies, each of which had very different approaches to the problem. Dofasco chose to work with HP's Professional Services Organization (PSO). "We picked the vendor that seemed to have the best methodology or process to go about putting in systems like this," says Wilson. "HP is a very process-driven company. Between their expertise and our in-house expertise we jointly developed an efficient systems management process."

Using a waterfall-style methodology, Dofasco's systems management team specified a stepwise process for the systems management project. The project spanned from March 1994 through the beginning of 1996, with some slack time between phases. It involved several groups within IS, spanning Computer Support, Business Support and Application Development. "We had to fight the temptation to go off and just buy tools so we could fix things and get on with it. In our phased implementation plan, the tools were the last thing we tackled," says Hilbig. "Design had to come first."

Phase I of the systems management project involved architectural implementation and planning. "We defined the problem and came up with a conceptual solution. This design was a framework model that accounted for the integration between various functions and departments as well as the data flow between various tasks," says Hilbig. "We then used the framework model to clearly identify where automated tools were necessary to provide the data to enable particular functions." Dofasco's systems management framework included functions that would be consistent across all platforms: change and configuration management, problem and fault management, performance management, security management, and operations management.

Phase II focused on tool selection. Here, several teams defined low-level tool requirements, chose tools to meet those specifications, and built a tool implementation plan. "The Kepner-Tregoe (KT) decision analysis methodology was very valuable during our tool selection process," says Hilbig. "KT is a matrix-driven way to clear biases from the decision analysis process. We picked all our tools in six weeks."

Phase III kicked off a series of implementation projects. "We broke our processes into manageable projects. For example, we implemented our problem management tools, detailed business processes relevant to that function, and trained the users," says Hilbig.

"This framework shows why we were able to implement systems management despite our downsized numbers," says Wilson. "Now we can have one skill set manage security across all platforms, the next manage changes across all platforms, and so on. We achieve our goal of uniformly managing applications across business units."

FEDERAL-PROVINCIAL-TOWNSHIP MODEL

Wilson describes one use of the F-P-T model in the application development area. "A Township issue would be what tool you're going to pick to test software or manage your code. But when you deliver that code to the infrastructure, it must meet a Provincial or Federal standard," he says.

On the operations side, Dofasco has several platform-specific groups. "We let these groups separately define their tool requirements. That is a Township level of autonomy," says Hilbig. "But all tools had to be capable of sending alerts to our Central Event Manager (CEM), an enterprise console that was a Federal requirement. From these alerts, the CEM automatically creates and routes trouble tickets in our problem management system."

The beauty of the F-P-T system is that the IS infrastructure still gets high-level requirements met, while the individual IS teams have the autonomy and control over their domains.

MELDING TOOLS AND SYSTEMS

Dofasco's CEM tells operations staff members the state of the environment. They get a warning if a fault is occurring. If there is an error signaling a problem, it gets passed to the problem management system through trouble tickets, which identify if the issue is one for computer support (e.g. the data base has died and must be restarted) or for application development (e.g. an application fault).

Another useful operations tool is Unison's Maestro, which acts as Dofasco's central job scheduler across all of its UNIX platforms. "People getting into client/server often think there's no longer a need for a batch window," comments Hilbig. "We found out the hard way that this was not true."

"Our new tools allow users to organize jobs more efficiently, and get to the source of problems more quickly," says Wilson. "In the past, it came down to us to identify where problems were. The automated alert identification helps us be more proactive - in fact, we feel we're actually preventing a lot of problems by monitoring events."

BEATING THE CLOCK

"Our Computerized Maintenance Management System (CMMS) automates Dofasco's workforce of 3,000 maintenance people," he says. "The application touches about 4,000 total users, with 1,000 concurrent users accessing the Oracle database at any given time. At the time it was ready to be deployed in January 1996, we believe this was the largest client/server application in Canada. There was a lot of skepticism about this turn-on."

"We came live about two weeks after Christmas. The application was virtually bumpless - it came off without a hitch, and it's still running today," says Wilson. "The application itself was well designed, but without our system management infrastructure, we couldn't have experienced that type of reliability."

Grigsby stresses persistence. "Once you start this trip," he says, "you'd better make sure you have the intention and fortitude to take it the whole way. This is a big challenge that will stress your team's time, energy and patience."

Looking back from the other side, Dofasco's team feels the results have been worth the journey.

Diane Fromme is an experienced reporter specializing in high-technology and communication issues. She lives in Ft. Collins, Colorado.