The process of evaluating and choosing the most qualified control system integrator (CSI), at the best value, may be the most critical decision that affects overall project success.
Obviously most end-users are not in the business of being a control system integrator, and therefore many do not have the expertise to properly compare and contrast CSI's in a competitive, evaluated bid process.
Good Process, Good Choice
With a well-defined selection process in place, the most
qualified CSI rises to the top. A good choice here ensures
healthy collaboration and project success down the road.
Today's control systems often include intrusion monitoring , alarming, and the automation of processes in the event of these alarms; activities that may directly impact the safety and security of the public at large. Choosing the most qualified CSI is serious business.
These categories allow you to develop a decision matrix (Figure 1):
Â· Eligibility & Unique Features
Â· Project Understanding
Â· Hardware and Software Recommendations
Â· Project Approach
Figure 1: Decision Matrix
Choosing a qualified control system integrator is serious business
--a good decision matrix can help.
Eligibility and Unique Features
Eligibility: This category may be useful for scoring a company's stability beyond that established as minimum requirements. In other words, if you require a CSI to be in business for more than five years and the CSI you're evaluating has been in business for 30 years, your grade here could reflect this advantage. Areas to consider include:
Â· Firm's annual integration services gross revenue
Â· Firm's Size (# of employees)
Â· Number of Engineers on staff
Â· Turnover ratio (for the past 5 years)
Â· Years of service & experience per employee
Â· Financial Rating (Dunn & Bradstreet, etc.)
Â· Adequate Errors and Omissions insurance, as well as minimum bonding capacity
Â· Licensed Engineering Firm (length of time)
Â· Listed UL-508 panel shop, integral to the CSI
Â· Evidence internal company practices, policies and standards (may include Mission & Vision statement)
Â· CSIA Registered Member status
Unique features: This category gives you an area to score a company's unique characteristic that can be of value to your facility. Consider features that may not necessarily fit well into any other category. Overall company reputation or industry involvement in support organizations, (such as the AWWA, WEF, PNCWA, ISA, CSIA, OAWU, ERWA, APWA and MCAA) could be graded in this section.
Does the company seem to have a good understanding of your organization's goals, both short term and long term? Is the CSI providing in their scope of supply what was specifically specified?
Hardware & Software Recommendations: In order to grade this category, you must first know something about the different hardware and software manufacturers that are on the market and available. You may decide to make your selections first, thereby eliminating this category. All CSI would then be required to provide the same equipment. Common hardware manufacturers include Rockwell Allen-Bradley, Siemens, Modicon and others. Software manufacturers include Wonderware, RSView, Intellution and many more. Several items to consider when selecting these components include:
- Manufacturer size/longevity
- Commitment to R&D
- New products/year
- % Market share
- Municipal references
- Availability of support personnel (inside & outside the company)
- Ease of use
- Cost (initial capital & cost of ownership over its lifetime),Communication medium, such as leased line, radio and spread spectrum telemetry may also be pre-selected, or graded in this section, if the project involves SCADA.
How does the company plan to approach the project? Will each of the project elements be met with the professionalism and the kind of expertise you're looking for? Are the tasks and the project schedule realistic and well laid out? Have project milestones been identified and a plan presented that will accomplish them?
Proposal Format: Is the proposal formatted and outlined well? Professional? Readable? Easily understood? This document represents the company's first impression, putting their "best foot forward." It could give you clues to the company's approach to the O&M manuals and other communication documents vital to your projects success.
Support Services: Does the CSI have good post project support? Does the company have a dedicated full-time field service manager? Support 24/7? Does it have the resources necessary to provide the level of service you require? Do they offer an annual maintenance program? How did the firm's references rate its post project support? Is it clear in the proposal that post project customer service is recognized by the CSI as a significant advantage to their organization?
Experience: Does the company and its employees have experience with projects of similar size and complexity? Be sure to consider each element (or task) of your project, especially any design requirements. It's one thing to have completed a project under a consultant's direction, its quite another for a CSI to perform all design-required elements in-house. There also exists quite a gap in experience with respect to SCADA vs. SCADA incorporating full Telemetry
References: Perhaps one of the most important "validators" of a CSI's performance. Is there good feedback on the company's past performance? Assure yourself that the references are representative of the type of project you are considering. Does the CSI develop good relationships with its contractors, consultants, customers and venders? Does the CSI come highly recommended by its references and others you have contacted through your own relationships?
A Reasonable Price
Does the price seem reasonable? Keep in mind that the price listed at this point in the process may be near impossible to determine, especially if the project has yet to be fully designed.
An artificially low price may be a sign that either the CSI has not included (or the CSI does not understand) the full project scope, or perhaps an initial low price is used as a ploy to lure you in; anticipating they will "change-order" the owner and end users throughout project implementation in order to make up the difference.
On the other hand, the price may be on the high side, conservatively taking into account areas yet to be well defined, such as the number of screens, reports, installation requirements and degree of involvement of the municipality.
It is highly recommended that price weigh little, to not at all, on the decision during this stage of the evaluation. We believe the best approach is to focus on selecting the most qualified CSI, such as the case when requesting an Statement of Qualification (SOQ) process. A highly respected and highly recommended CSI could not maintain its good reputation if they were to over inflate their pricing. A CSI unwilling to provide the best value would quickly fall out of favor, scoring very poorly on several of the categories previously covered.
Improving the List and Weighting the Categories
You may decide to add, combine or delete from this list. Vital to the criteria is the "weight" given to each category. Determining the importance, or weight, of each category is the initial step in completing this process. Our experience led us to the following values: Eligibility, 15%; Project Specifics (understanding, recommendations, approach and presentation), 33%; Support Service, 20%; Experience & References, 28%, followed by Price at 5%.
Price weighted at only 5%? Bear in mind that if you're evaluating a CSI's for a performance based, design-build project, your project is not yet designed. A solid price at this point is nearly impossible to ascertain. Your goal at this stage should be to select the most qualified CSI, and leave the determining and negotiating of the price to be taken care of at a later date. It is recommended that an owner seek a SOQ that does not include price, rather than a request for proposal (RFP) that may have a request for pricing.
Once you have assigned the appropriate weight per category, photocopy your matrix sheet, one per CSI, or use an Excel workbook, one sheet per integrator. Then rate each category based on a numerical grade, 0--100. Zero meaning they missed the mark completely, 100 being the highest possible score for that category.
First, score each CSI independently, methodically moving through their proposal, grading each category; one after the other. The second step fine-tunes the grades across the board.
In step two, select one particular category and compare grades among all the CSI's in that particular area. This will allow you to fine-tune the grades, as you may discover that you gave the same numerical value to two different CSI's. Comparing them now, side-by-side, may reveal that one CSI is more complete, say in the instance of "Project Approach." Fine-tune their grades by moving either one of them up or down, or whatever you find to be the best solution.
Final Score is next calculated by simply multiplying each category's grade by its associated, relative weight. Once you know the final score for each category, add them up for that CSI's individual Final Total.
The most qualified integrator rises to the top, having obtained the highest Final Total. Moving forward and awarding this most qualified CSI may be plainly obvious. If not, a second phase in your selection process is necessary.
Face to Face
Phase two involves face-to-face interviews. You may choose this step based on the total number of proposals, very close Final Totals, or other factors that lead you down this path. You'll want to short list the top scoring two or three CSIs and arrange to interview them at your facility, or perhaps better yet, their facility.
Several benefits accrue from this more involved stage, such as the opportunity for your organization to get a feel for the "personality" of the firm and how communication and interaction throughout the project may be conducted. Certainly items such as Project Understanding, Project Approach, and Price can and should be discussed in further detail at this stage.
A similar matrix should be developed for each of the interviews. Regardless of how you score the interviews, by investing the time and effort to go through the process as we have outlined, there is little doubt that one particular firm will stand out amongst the others, a firm you are eager to work with and begin a long term relationship with for the delivery and support of your new automation and control project.
Lastly, you may want to develop a section, or determine through the Phase Two, (interview) process, certain intangibles. These intangibles could include how your organization "feels" about the CSI. In other words, are they trustworthy? Do you have the confidence that they can perform all aspects of your project within budget and on-time? Was the CSI professional throughout the process? Did their personalities mesh well with those in your company?
As discussed previously, we believe that addressing these critical areas, asking the right questions and simply taking the time to move through this process will reveal one or two companies that shine above the others. This will make your decision-making process much clearer and far less risky then the alternative "dart-board" or low bid process.