In my previous post, SharePoint versus Documentum as the Dominant Pharma Platform, I described what may be one of the most significant changes in pharma ECM: the emergence of Microsoft SharePoint as a platform for managing controlled documents in the life sciences industry. The next 2 years are an inflection point, and will probably mark the beginning of a major shift in the document management (DM) platform profile for the pharma industry.
In this post, I will try to help you decide which platform to choose. I strongly advise you to do a formal evaluation of the options – a bakeoff between FirstDoc/Documentum, FirstPoint/SharePoint, and NextDocs/SharePoint. And below I explain exactly how to conduct such an evaluation.
If you’re a pharma company, DM platform choice is a very important issue because:
- Which platform you choose in the next few years as you upgrade or replace your major applications will probably be the platform you will live with for the next 7 to 10 years.
- You must therefore manage long-term risk and cost, particularly the risk that Documentum or the application suite built on it (CSC FirstDoc) will not be an adequate platform in 3, 5, or 7 years – and you’ll have to replace it in a risky, costly manner after investing heavily in it and in CSC FirstDoc. Documentum and FirstDoc are the proven market leaders, and while they may be perfectly adequate now, it’s unclear whether that will continue to be the case.
- You must also manage short-term risk and cost – particularly the risk that either SharePoint or the application suites built on it (CSC FirstPoint or NextDocs) will not be adequate in the short term. The short term is right now – or whenever you start replacing or upgrading your pharma applications. SharePoint as the platform for FirstPoint or NextDocs (or another entrant) may be a good long-term bet, but it’s a nonstarter if SharePoint and those two application suites can’t cobble together a great solution right now when you need it.
If you ask us what you should do, we’d be happy to provide you with recommendations. But one of those recommendations will be to have the relevant vendors demonstrate their products against your requirements, with your usage scenarios and documents. It’s Doculabs’ standard methodology for evaluating vendor solutions, but made more stringent through the inclusion of additional steps that focus on the exact questions that are most relevant to an evaluation of FirstDoc/Documentum versus FirstPoint/SharePoint versus NextDocs/SharePoint.
Here’s how you should do it.
1. Clarify and prioritize your company’s business requirements and standards of acceptability. Unless two of the candidates are obvious rejects, leaving a winner by default, you will need a principled way to clarify and resolve tradeoffs between your requirements. You will also need to decide between short-term versus long-term benefits, costs, and risk. And you must decide how much risk and cost you are willing to accept in order to attain a projected level of benefit. Your company may choose a different solution if it is risk-minimizing versus benefit-maximizing, or if it focuses on the near term versus the long term. Your evaluation team will not have productive conversations unless you’re all using the same measuring stick.
2. Define your general evaluation categories, which are the general types of requirements. The evaluation should be comprehensive and include the following categories:
- Functional Requirements are the capabilities required to fulfill your defined business requirements.
- Technical Requirements are the specific IT specifications (e.g. operating environments, protocols, databases) and capabilities (e.g. scalability, performance, reliability, security, and integration) required to fulfill the functional requirements.
- Vendor Requirements are the ability of the supplier to provide and support a successful solution through the initiative’s lifecycle. These include the vendor’s stability, its strategic direction relevant to your firm and pharma, and its ability and willingness to provide support and services.
- Resource Requirements include the overall cost of the solution, but also other indicators of overall resources, including effort, time, expertise, and people.
3. Define the specific requirements within each requirement category. You should have many requirements for each of the four requirements categories described in #2 above. Your Functional Requirements evaluation, for example, should include requirements for user capabilities, administrative capabilities, e-signature capabilities, reporting capabilities, audit and compliance requirements (e.g. Part 11 compliance), etc. These will become line items in the RFP and will also become input to the instructions you provide to finalist vendors as they prepare their solution demonstrations in the subsequent phase of your evaluation process.
4. Identify vendor-specific issues relevant to the specific requirements. It will facilitate the evaluation process if strengths and weaknesses of each candidate vendor can be identified with respect to the requirement. For example, with respect to the Technology Requirement “scalability,” some products may be questionable because of the architecture of the product, or because of the platform they use, or both. This is a commonly voiced concern about SharePoint as a pharma platform. So address it directly.
5. Determine how each requirement will be tested. The requirements are such that they are amenable to some evaluation methods, but not to others. For example, some requirements can be assessed only by working with the product, while others are best assessed by looking at the state of the market. We recommend combining four methods to evaluate the requirements:
- Market research and opinion, such as from industry analysts and consultants. This is most useful for evaluating past historical performance of the vendors, market trends, etc. (We’d be happy to answer your questions, by the way.)
- Vendor attestation, typically gathered in an RFP-type process. This is most useful for acquiring very specific information about the proposed solutions, in writing, that the vendors will then clarify and demonstrate during testing.
- Reference sites, which you should at least contact, if not visit, to discuss the experiences other organizations have had with the vendor’s solution. Note that it’s more useful to conduct such interviews without vendor “handlers” present.
- Testing, which should be done in your environment using your specific DM usage scenarios.
For each requirement, determine which method or combination of testing methods should be used. (I plan to provide some of the more important requirements, vendor-specific issues, and best evaluation methods in a future blog post.) Be sure you have a fairly clear idea regarding what constitutes “success” or “failure” for each requirement.
6. Define the scaling and metrics for the evaluation. The two key metrics are the grading scale for the vendors against the requirements, and the weighting or prioritization for each of the requirements:
- With respect to the grading scale, we have found a 5-point scale to be most useful. Many organizations make the mistake of making the grading criteria complex. But a simple scale from poor to excellent works best for all requirements:
1 = Poor
2 = Fair
3 = Good
4 = Very Good
5 = Excellent
- With respect to weighting, there is no single best method. What constitutes a best method for weighting depends on a few things, particularly whether there is a significant difference in the importance of the various requirements. We recommend a 5-point scale for weighting:
1 = Relevant but low importance
2 = Moderate importance
3 = Significant importance
4 = High importance
X = Extremely high importance (i.e. a deal-breaker)
- Some requirements (such as vendor stability or platform scalability) may be deemed of extremely high importance, meaning a low score on these requirements will eliminate the vendor from further consideration, regardless of their score on any other item. Note that for these criteria, a very high score may not be necessary – only a “non-low” score. After identifying any such requirements, set the threshold score. Typically, “Poor” (and sometimes “Fair”) on these items are considered to be exclusionary scores.
- To score the remaining vendors, simply multiple the grades (1-5) by the relevant numerical weights (1-4). This basic approach typically achieves the desired differentiation between candidates, and appropriately rewards those vendors who consistently score higher across requirements.
- The “winning” alternative should both be the highest-scoring candidate, and also be above an agreed-upon threshold of adequacy for a “go” decision. It is often useful for the evaluation group to agree upon clear cases of where this threshold might be set, but it’s often not useful to set a hard line (“Total scores below X fail, while scores at or above X are fine”) before seeing the scores themselves.
7. Perform the evaluation using the four methods described in #5 above. Doculabs can provide detailed recommendations on how to most effectively conduct these evaluations.
8. Score the evaluations and recommend a course of action.
As I mentioned, the methodology outlined above is based on Doculabs’ standard bake-off process, but with the exact questions you’ll need to ask around your requirements and how to test the vendor and their solutions with respect to each requirement. When a technology decision requires this level of due diligence, it’s the process we recommend because it ensures a thorough-going approach to the evaluation of the options. No question: Your FirstDoc/Documentum versus FirstPoint/SharePoint versus NextDocs/SharePoint decision deserves this degree of due diligence.