Stupid Vendor Tricks, Part 1: Doing a POC in a “Lab”

After 32 years in the IT industry I’ve seen just about every trick a vendor can play on a prospective client. This is the first in a series of the most egregiously bad, deceptive or just plain wrong Stupid Vendor Tricks I’ve seen over the years.

Stupid Vendor Tricks, Part 1: Doing a POC in a “Lab”

The best way to evaluate a potential new product is to install it in your production environment and put it through its paces, using at least a subset of your actual data. This is variously called an onsite demo, a proof-of-concept (or POC) or an evaluation. I am a big advocate of doing this type of onsite test. It can tell you a great deal more about a solution than any PowerPoint deck or Webex demo could ever do.

A variation on this is the so-called “lab” POC, where a vendor sets up a lab environment supposedly similar to the prospective client’s. There are numerous justifications for doing this, and some are legitimate, as long as the client understands that this is nothing like a real-world test. Sometimes a vendor will propose a lab POC as a way to shortcut the evaluation process. Beware of this proposal.

A Test-Drive Without Leaving The Showroom

Wikipedia defines a laboratory as “a facility that provides controlled conditions in which scientific research, experiments, and measurement may be performed.” The crucial words are “controlled conditions“. Let me pose a question: You, dear reader, are most likely a professional who manages an organization’s information technology environment. Would you consider your production shop to be a “controlled” environment? By contrast, a lab POC is like test driving a new car without ever leaving the showroom.

Sparing You The Sausage-Making

Have you ever heard the old adage that if you ever see sausage being made, you’ll never eat sausage again? Almost without fail, lab POC’s go perfectly, as they should, considering that vendors can tilt the table as far in their favor as necessary in order to make the proposed solution appear to work. (If the POC goes sideways for any reason it demonstrates a serious lack of competency on the part of the vendor, IMHO.) What the vendor is doing is sparing you the sausage-making; you generally don’t have to see how difficult the solution is to install, configure and/or successfully integrate with other products.

A Sure-Fire Path to Buyer’s Remorse

The trouble usually starts when a client makes an actual purchase decision based on a lab POC. To extend the new car analogy, once you get it out on the highway you may find that the ride is lousy, handling is poor and the engine is underpowered, all things you can’t learn by sitting behind the wheel in a showroom. I have seen lab POC’s go swimmingly, only to result in production installations that take months of work to get running properly.  Some never do.

Just Say No…

When a vendor suggests a lab POC as a way to shortcut your decision process on a particular solution, it may sound attractive, but resist the urge – make them install it at your shop on your network and test it with your data. Then you will find out how well the solution really works.