MDS Sciex ROI Report
FOR EVERY DOLLAR INVESTED IN 6 THINKING HATS TRAINING,
MDS SCIEX REPORTS ROI $26.48 OR 2648%.
A Report from MDS SCIEX
CAN INNOVATION TOOLS INFLUENCE THE
NEW PRODUCT DEVELOPMENT PROCESS?
By:
- Ken Delcol, PMP, PEng – MDS SCIEX
- Suzanne Wolfe – MDS SCIEX
- Karima West – Nought Ltd.
BACKGROUND
MDS SCIEX is a division of MDS Inc., Canada’s largest and most diversified health and life sciences company. MDS SCIEX exports 90% of its products to more than 50 countries through a marketing alliance with PE Corp. MDS SCIEX was born out of the need to analyze material on the initial NASA Martian expedition project and has grown from a research organization to a world leader in the research, design and manufacture of mass spectrometers. The organization has prided itself on its research talent and on its earlier ability to deliver innovative solutions to the market place. In the early 1990’s, MDS SCIEX lost its ability to deliver new ideas to the market in a timely fashion. However, it never lost its ability to generate new ideas. By the mid 1990’s significant effort was directed at improving the organization’s ability to deliver new ideas to the marketplace through the use of recognized time-to-market techniques.
MDS SCIEX employs more than 350 highly skilled professionals, with nearly half of our workforce in scientific research, product engineering and software development. The early culture of MDS SCIEX was focused on the development of fundamental science and is changing to one that must focus on the building of commercial products. The current design culture is stimulated by difficulty and complexity, dislikes routine activities and tends to trivialize objections to design directions. Project teams at MDS SCIEX are persistent, seeking new challenges once a problem is solved. They are pragmatic, constantly demanding proof and debating every angle. They prefer to generate their own systems and process changes rather than adapt to changes imposed externally.
THE CHALLENGE
The new millennium will find most organizations facing tremendous competitive pressures. Their struggle to maintain their market positions is affecting their ability to grow as more resources are focused on short rather than long-term issues. Gone are the days when restructuring, re-engineering, right sizing, mergers and acquisitions alone were enough to improve would add to the bottom line. Leading business gurus are predicting that the single most critical factor for business success in the next century will be innovation.
The definition of innovation used in this paper has a business orientation:
Innovation is the translation of a new idea into a product and/or service which is delivered to the marketplace within a window of opportunity and which meets the market’s quality, performance and cost expectations, and the organization’s profit requirements.
By late 1997 MDS SCIEX was looking for ways to build a competitive advantage through improved innovation strategies. Like most companies, MDS SCIEX believed that employee development programs aimed at innovation and creativity skills would have a positive impact on the organization’s bottom line. An innovation skills program, accompanied by a rigorous impact assessment would provide the opportunity to test this belief.
SETTING THE STAGE
In early 1998, the MDS-SCIEX Product Development group had an objective to develop a product to capture a market segment from a major competitor within a narrow window of opportunity. Additional challenges included the lack of familiarity with the technology by the Product Development group, the existence of a competitor’s product in the market, the need to compress the development schedule, the need to expand the vendor group, and the expectations of MDS SCIEX’s executives relying on the project to create a new third product line for the organization. The project to deliver this product was code named Athena and was to become the largest single project in MDS SCIEX’s history.
The most important aspect of the project was the need for the Product Development group to work very closely with the Research group. This would ensure that the critical technical characteristics of the product were retained in the transfer from the research breadboard to a fully commercialized product. This would require both engineers and scientists to collaborate outside their natural domains of expertise.
As is traditional in most organizations, design conflicts between engineers, scientists, manufacturing, and vendors existed in abundance. These conflicts surfaced regularly at the project review meetings held during the development of the product, resulting in delays in design and manufacturing, cost overruns, both organizational and interpersonal conflicts, and lack of consensus in the final design. An analysis of previous projects revealed that most design review conflicts centered on the following:
- Presentation of incomplete or erroneous information.
- Poor handling of questions (adversarial tone, poor topic focus, interruptions and tangents).
- Generalizing and/or exaggerating issues.
- Inability of the review process to generate consensus, identify new ideas or find acceptable solutions.
The Athena group needed to find a way to promote the timely generation and constructive review of ideas, proposals, and design solutions while avoiding the conflicts of past design review meetings. With the Athena project constraints and objectives clear, the stage was now set to allow the opportunity to attempt to improve the product development process with an innovation skills program.
CREATING THE ENVIRONMENT
The innovation program was aimed at a variety of design issues i.e. creating new options, working collaboratively to solve problems, and building consensus. A core group of senior managers selected and promoted an innovation skills program based on Edward de Bono’s Six Thinking Hats.
*****************************************************
The main reasons for selecting the Six Thinking Hats included the following properties of the technique:
- harnesses focused thinking within a flexible process (appealing to most engineers)
- discounts the belief that creativity is the domain of only selected people
- generates consensus
- depersonalizes criticism
All project staff including external design consultants assigned to the Athena project were trained in the use of the tool. The training was provided by the senior managers who selected the tool and became certified to deliver the employee training. In addition, the training was also rolled out to other areas of the organization.
The Athena project manager encouraged his team to use their new skills for appropriate project-related work, and planned the agendas of all Concept, Preliminary, Code and Critical Design Review meetings to incorporate the new technique. Team members were also encouraged to use the technique on an ad hoc basis in their daily work.
EVALUATION DESIGN
The innovation skills impact study was designed to gather as much useful information about the learning and application of the skills within the context of the Athena project (see West, 1999 for complete details). Assessment tools and processes were developed to be as non-intrusive as possible, to avoid interfering with product development schedules and to permit naturalistic observation of the flow of work. All team members were informed of the purpose of the impact study and were asked permission to be surveyed and interviewed. All responses were compiled in strict confidentiality. Twenty-eight individuals, consisting of the initial core Athena group, were followed for this case study.
FRAMEWORK FOR EVALUATING TRAINING
The Kirkpatrick (1998) 4-level program evaluation model was selected for this study based on its ease of use and acceptance in the assessment field. This model was supplemented with Phillips (1997) `return-on-investment’ model to add a financial dimension to the evaluation. The five levels of evaluation used in this study are defined as:
Level 1 – Reaction A measure of participant satisfaction and intention to apply the skills.
Level 2 – Learning A measure of transfer of knowledge or development of skill or change in attitude.
Level 3 – Behavior A measure of change in on-the-job activity or application of the learning in real situations.
Level 4 – Results The business impact achieved by program participants as they successfully apply the skills (changes in output, quality, costs, time, customer satisfaction etc.).
Level 5 – Return-on-Investment Compares the monetary benefits of the program with the program costs.
TRAINING OBJECTIVES
Based on interviews with senior Athena project management staff, behavioral objectives were set for the impact study. These were based on observed shortcomings of previous design projects and indicated in what areas the team was expected to improve:
Design Review Meetings
- Less adversarial atmosphere.
- Increase in potential solutions generated for all design concerns raised.
- Greater group satisfaction with process.
- Design Documentation
- Clearer documentation of group work and thinking.
- Clearer description of thinking process for pre-meeting distribution.
ASSESSMENT TOOLS AND PROCESS
The assessment tools for this study were custom-designed to generate data to determine if the training objectives were met and to fit with each of Kirkpatrick’s and Phillips’ five levels of evaluation. Assessment was done prior to training, immediately after training, throughout the development of the product, and finally as product development wound down.
- PRE-TRAINING ASSESSMENT [PTA] FORM
In order to capture the group’s impressions of the design review process as it stood prior to the innovation skills program, team members were asked to rate design review meetings they had attended in terms of 20 meeting elements including meeting management issues, interpersonal communication and climate issues, generation and analysis of ideas, and quality of outcome. - SIX THINKING HATS TRAINING ASSESSMENT [6HTA] FORM
Immediately following their training, team members were asked to assess the training and trainers, indicate whether they had retained the content of the training, and give their impression of the value of these skills to their work at MDS SCIEX, as well as a prediction of where and when they might use them. - SIX THINKING HATS MEETING OBSERVATION [6HMOB] FORM
During the five months following their training, the Athena team was required to use the new approach in their design review meetings. Approximately 50% of design review meetings were observed in terms of structure and adherence to the agenda, use of new terminology, tone and content of comments. - FIVE MONTH FOLLOW-UP INTERVIEW PROTOCOL
As the design review phase of the study wound down, each team member participated in a 20-minute interview about their attitudes toward and use of the innovation skills since their training. They were asked to rate the Athena design review meetings in the same way they had rated previous meetings prior to their training.
RESULTS AND INTERPRETATION:
Organizational learning experts suggest that in order to create lasting change within an organization, the change should not be implemented from the top, but rather introduced from the middle or lower levels, in projects where the new approaches are most valuable and relevant to staff (Senge, 1999). The changes should be cultivated carefully, allowing self-reinforcing processes free reign and clearing away potential change barriers. The innovation skills program successes may be due in large part to an attempt to take such an approach to introducing change: supporting one cross-functional group’s efforts to improve its own effectiveness and efficiency.
Originally, senior Product Development staff introduced the Six Thinking Hats program to improve perceived shortcomings in the design review process. Of note was that this desire to improve was echoed by the remaining Product Development staff. Measured before the skills training, their average rating of 20 design review meeting elements ranged from “could use improvement” to “adequate”. Both management and staff tended to agree on which were the problem elements (e.g., tolerance for all viewpoints, exploration of ideas, building constructively on each others’ ideas, quality of decisions).
A shared commitment to the Six Thinking Hats program as a potential remedy was also clear by the end of the training days: 76 % of the group felt the techniques would improve their work and 86 % agreed with senior staff that the tools would improve the design review process. Management and staff consensus of both the need to improve and the relevance of the improvement strategy no doubt fueled the success of the innovation skills program at MDS SCIEX.
Organizational learning experts have found that deep and lasting change occurs only with personal growth, and the commitment of all group members to self-generated change as opposed to compliance to top-down demanded change. The Athena project development team began to manifest this by actively using the Six Thinking Hats skills over the five (5) months of intensive design review meetings observed for the impact study. During the first design review meetings, adherence to the Six Thinking Hats agendas was strictly enforced by the project manager, but soon most participants were also using the terminology to preface their contributions and to regulate each other’s comments, often with good-natured humour. They would draw attention to violations of the agenda colour in use (“Hey, you’re red hatting my green hat!”), ask to revisit an agenda colour already completed (“I just thought of another white hat piece of information we forgot to consider”), and not hesitate to move the pre-set agenda along (“I think we’ve exhausted the black hats for this one”).
After five months of use on the Athena project, average ratings of the 20 design review meeting elements measured before the skills training increased substantially, often an entire satisfaction level. Selected pre-and post-program ratings are shown below in (Exhibit 2 (Meeting Efficiency) and Exhibit 3 (Atmosphere for Innovation)X). A full 90% agreed that their work had improved because of the new approach, 90 % thought their co-workers’ work had improved, and 95% predicted they would use the skills in the next four months.
RETURN ON TRAINING INVESTMENT
When asked to rate the contribution of the new innovation program to the improved functioning of the Product Development team, participants had no difficulty assigning a percentage to their perceptions (only five declined to estimate). Responses ranged from 10% to 80% of effectiveness attributed to the new program, with an average of 40%. Participants indicated that their estimates were conservative, considering that other factors contributed to the group’s improved efficiency (such as experience gained on previous projects, more familiarity with team members etc.) making it difficult to compare across projects. Even as a conservative estimate, a 40% perceived improvement in group functioning as a result of one initiative is substantial.
Current methods of calculating the return on Human Resources Development (HRD) initiatives, particularly for “soft skills” training, suggest converting qualified participant estimates of improved efficiency into cost savings based on staff salaries, benefits and overhead (SBO) savings (Phillips, 1997). The perceived 40% efficiency improvement is calculated on the projected SBO that the project would have incurred had the team performed at the historical efficiency rate. Based on the team’s efficiency improvement estimate, MDS SCIEX interpreted the actual team SBO incurred as 60% of the projected SBO, as follows:
$519 / day – Average team member salary, benefits and overhead (SBO)
$519 x 28 staff = $14,532 / day – Total team SBO
$14,532 x 100 days = $1,453, 200 – Actual team SBO for 5 months of product development
$1,453,200 / .60 = $2,422,000 – Projected team SBO, assuming 40% efficiency improvement
PROGRAM BENEFITS
Projected SBO – Actual SBO = $2,422,000 – $1,453, 200 = $ 968,800
The investment portion of the return-on-investment calculation includes all costs associated with the innovation skills program, as follows:
PROGRAM COSTS
- “trainer training” pro-rated for two Athena sessions: $ 600
- trainer’s daily SBO x 2 training days: $ 1,540
- training day cost
- total team SBO for one day $14,532
- cost of training materials ($125 x28) $ 3,500
- participant and instructor meals ($5 x 30) $ 150
- impact study design, data collection and interpretation $18,089 – $35,261
Considering only the conservative estimate of the impact of the program provided by the participants, the return-on-investment within the 5 months of use of the skills on one project is as follows:
ROI (%) = Net Program Benefits x 100 = (Benefits – Costs) x100Program Costs
Costs= $968,800 – $35,261 x 100 = $933,539 x 100 = 2648% $35,261 $35,261
This calculation indicates that for every dollar spent on the innovation skills program for the Athena group, MDS SCIEX reaped an additional $26.48, after the cost of the program had been recovered, based on team members estimates of improved efficiency. This is a substantial return. Furthermore, this simplified calculation in no way encompasses the full bottom-line impact of the program.
Consistent improvements in design review meeting efficiency and atmosphere, see Exhibits 2 and 3, are difficult to quantify in dollars, but no doubt had significant effects on the project goals. There is also no direct or easy way to assign a value to any given idea or solution that emerged from the design review meetings as a result of the new innovation tools.
The impact study reveals that meeting agendas were better managed during the Athena project than in previous projects, new ideas were encouraged and explored to a greater extent, team members participated more and behaved in a less adversarial manner, the quality of decisions improved, and ideas were built upon more constructively, among many other improvements. These results indicate that the potential for innovation was enhanced and barriers removed as a result of the application of the tools, setting the stage for improved group effectiveness in innumerable tangible and intangible ways.
PROJECT OUTCOMES
The design phase took one and one-half months longer than planned while accommodating 15 change requests and being initially under staffed. The first Athena units were delivered to Manufacturing less than one year from the start of design. Other similarly staffed MDS SCIEX projects would have taken well over a year to accomplish the Athena design effort. The Athena units have a radically different architecture, which addresses all of the shortcomings of the Research breadboard units including transportation, safety, and manufacturability. The performance of the commercialized units have exceed their design specifications and that of the top breadboard units. All agency tests were readily met.
The Six Thinking Hats technique was instrumental in the development of a radically different mechanical architecture for the Athena units. This involved the development of over 36 different high-level system architectures and up to eight different concepts for each subsystem. The bulk of the concept work was completed in five weeks when compared to eight weeks on less complicated subsystems on other projects. These project outcomes corroborate the team’s estimation that the Athena product development process was 40% more efficient as a result of the new innovation tools.
Many of the problems discussed earlier with design reviews were avoided with the use of the Six Thinking Hats technique. The project team was not overwhelmed when a given design was found to have problems since the review meeting also provided possible solutions. The general feeling at the end of most design reviews was optimistic even when problems were raised. The biggest endorsement of the tool is the use of the process by other project teams in their design review meetings and in the development of new design concepts.
CONCLUSIONS
This paper has demonstrated the following three critical ideas:
- Innovation tools can positively impact the product development process. Without the Six Thinking Hats technique, the Athena project would not have succeeded in meeting its aggressive delivery schedule.
- The use of innovation tools results in new behaviors, which can be measured; hence their effectiveness can be assessed. The Six Thinking Hats technique has succeeded in creating new behaviors related to the design review process and in helping to create a more supportive environment for innovation.
- Program evaluation models provide an effective framework for assessing the impact of innovation skills training. The assessment models used to monitor the Athena project have provided a quantitative approach to analyzing the qualitative and quantitative impact of an innovation technique.
The investment in innovation tools to help improve the product development process was a sound investment and the impacts of these tools can be easily assessed and quantified by the users of the new process themselves.
REFERENCES
Six Thinking Hats is a registered trade mark with Advanced Practical Thinking Training Inc.
Kirkpatrick, D.L. (1998). Evaluation Training Programs: The Four Levels. 2nd ed. Berret-Koehler: San Francisco.
Phillips, J.J. (1997). Return On Investment In Training And Performance Improvement Programs. Gulf: Houston.
Senge, P. (1999). The Dance Of Change: The Challenges To Sustaining Momentum In Learning Organizations. Doubleday/Currency: New York.
West, K. (1999). Group Knowledge Creation In A High Technology Setting. Doctoral dissertation in preparation, Ontario Institute for Studies in Education of the University of Toronto.