Exceptional Planning & Situation Management Guide Complete Disaster Recovery
Hurricane destroyed company's data center facility and all its contents. Lack of resources, facility, hardware, and systems further complicated an already serious situation.
Calculated impact and performed situation assessment. Declared a disaster situation and engaged disaster recovery plan. Led all restoration and recovery efforts as well as directing communications for senior management. Engaged all necessary personnel including outside vendors.
Accomplished complete system recovery within 48 hours of declaring disaster situation. All systems and network access were available within 48 hours, and no data was lost.
Five-Criterion Tool Creates Fair & Clear Employee Evaluation
As data center manager, needed to create method to evaluate all data center employees for yearly performance review. However, because department was new and its function so different, there was no history or company precedent to use as guide.
Developed five-criterion evaluation process that included communication skills, technical knowledge, initiative, working with others, and follow-up skills. Worked with corporate Human Resources team to determine key employee behavior traits. Criteria eventually became part of standard culture, and employees were then evaluated based on criteria for yearly performance reviews.
With five-criterion process, implemented fair and clear method of employee evaluation. It also created metric to evaluate improvement and provide annual performance increases. Under evaluation process, only 3 of 15 original hires left company in seven years.
Involving Staff in Technical Decisions Secures Commitment to SAN Solution
Company needed new storage area network (SAN) because flood had destroyed data. To meet need, had to overcome high cost of technology, large disk space requirements, and limited support capabilities of internal staff.
Assembled technical team of vendor storage experts and staff members to determine necessary disk and switch requirements. Also, arranged training for vendor equipment support and internal staff.
New solution provided more than three times the number of terra bytes of storage than destroyed SAN environment and cost $200,000 less. In addition to vendor training, involving staff in technical decisions secured commitment to solution.
Data Center Goes Live on Time & on Budget with Careful Planning & Management
Company retained 85-90 data sites worldwide, and after implementing JD Edwards enterprise resource planning (ERP) system, company required central point for system access and consistency. Difficulty arose because plan, budget, facility, staff, and hardware to create access point had not been developed. Furthermore, time was a constraint because ERP site implementations were expensive and required a data center.
Presented senior management with build-out plan, budget, and facility scope. Then, worked with Human Resources team to define employee roles and responsibilities and began interviewing and hiring processes. In addition, hired architecture firm to assist with design details of facility, and hired and managed build-out vendors as well. Finally, defined computer hardware requirements, and purchased and planned hardware installation.
With such effective planning and vendor management, data center went live on budget and within 18-month timeline with all necessary employees on board.
Adept Leadership Sends New Data Center Live Four Months after Disaster
Company's original data center was lost in a natural disaster, and entire facility and contents had to be replaced. Building new data center environment and facility quickly was critical because each day company spent in recovery hot site cost $9,000.
Coordinated and collected all necessary technical resources and vendors to finalize new hardware requirements. Developed new system requirements and specified equipment. Worked with hardware vendors and established best prices and delivery options.
As a result of these leadership efforts, new data center system architecture was implemented and went live four months after disaster. Daily $9,000 hot site expenses were minimized. Secured new equipment with expanded capabilities for $1.2 million when original data center equipment cost $4 million.
Open and Honest Communication Guides Successful Data Center Outsourcing Effort
Company direction changed and executive management team required decentralized approach to data center operations. Fulfilling this need was difficult because staff's focus, commitment, and morale were at an all-time low.
Developed plan that coordinated both internal resources and vendors necessary to outsource data center. Communicated openly and honestly with staff throughout four-month project. As a result, entire data center was successfully moved on budget and on schedule.
Specifically, successfully moved all hardware and operation processes to outsourcing vendor with less than 24 hours of system down time. No data loss occurred, and production schedules were successfully implemented.
Training Technicians & Using Existing Equipment Resolves Trouble Ticket Assignments
Challenged with providing means to collect, document, and assign issue trouble tickets 24 hours a day, seven days a week worldwide. Specifically, company's worldwide help desk user base was active 24 hours a day because of different time zones. Furthermore, level-two technicians, who are more experienced and able to resolve more complex issues, were not centralized or always available when issues occurred.
Used existing computer operations resources that were available three shifts each day as user call point. Then, developed support assignment criteria matrix that appropriately assigned tickets to level-two support. Finally, created solution knowledge base within help desk software.
Ultimately, implementing help desk software provided ticket documentation and solution knowledge base. Moreover, decision to train technicians and use existing computer operations resources eliminated need to hire experienced senior-level support and better equipped level-one technicians to handle more complex issues.
Standardizing Hardware & Software Drastically Reduces Cost of PC Ownership
Company's desktop support was not effective or timely for 250 desktop users at corporate headquarters. However, each department had its own desktop hardware and software preference, so introducing change was difficult.
Collaborated with IT Director to create company policy for hardware and software standardization. Arranged cross-functional team of employees and IT support members to evaluate and select desktop hardware and software.
Standardizing hardware and software reduced company's total cost of ownership for desktop PC hardware, software, and support to one-third of industry standard.
New Scheduling & Payroll System Improves Accuracy & Speeds Payment
Company needed more efficient means for daily employees to clock in for shifts. Clock-ins required effective supervisor validation and time approval process. In addition, solution needed to provide ability to schedule employees on future shifts and create automatic time entry for payroll process.
Change was difficult for several reasons, including fact that employee group was resistant to change, technical infrastructure was old or non-existent, and coordinating training time was next to impossible because of vast number of employees with different schedules.
Began by defining scope for automated employee scheduling. Installed new swipe card clock in process using network-connected time clocks, and system automatically entered supervisor time approval of employee shift work into plant payroll.
Changes improved employee time clocking accuracy 20%, reduced time approval turnaround by 24 hours per supervisor, and provided valid work data for payment 24 hours sooner, improving payroll department productivity.
Plant First to Provide Extensive IT Services Companywide
Charged with providing all plant systems with access to electronic messaging and data sharing capabilities. However, no hardware or network software was available to connect computer services. Furthermore, internal users were resistant to change.
Gathered system requirements and created project plan to implement Microsoft-based local area network (LAN). Defined project scope, selected equipment, and purchased hardware. Successfully configured network server services and installed Microsoft Exchange e-mail.
Ultimately, implemented local area network, connecting all 50 plant PCs, which improved communication and productivity and made data sharing for statistical analysis much easier. This was first plant to provide these IT services companywide.
Tool Effectively Manages Large Data Center Budget & Prevents Over-Runs
Newly created data center was significant new department with large budget within company structure. However, managing this budget effectively was challenging because there was no past history on which to build, and expenses for IT services were spread across many site budgets and were difficult to isolate.
Worked closely with corporate accounting department to understand reporting needs. Then, collected all IT budgets and constructed master spreadsheet, effectively creating template for all old-cost budget detail. Finally, added in all new budget requirements, resulting in $10 million expense budget for data center.
Ultimately, developed and used template as budget standard, and never exceeded expense budget while using tool.
Successful Negotiations Save $30,000 on Life of New Disaster Recovery Contract
Company required new disaster recovery contract after losing data center to natural disaster. Specifically, disaster destroyed all existing equipment, rendering original contract null and void. However, vendor answered request for new contract with increased pricing model.
Worked with company's Chief Information Officer in developing contract strategy. Participated in extensive vendor meetings and negotiated five-year contract for $600 less per month than vendor's original request, saving $30,000 during life of contract. Furthermore, negotiated contract included new equipment, additional disk space, and network connections required by company.
|