Linux Today: Linux News On Internet Time.

Benchmarketing 101 -- Or FUD with Numbers

Jul 23, 1999, 06:13 (17 Talkback[s])
(Other stories by Eric Lee Green)

[ The opinions expressed by authors on Linux Today are their own. They speak only for themselves and not for Linux Today. ]

By Eric Lee Green

Welcome to Benchmarketing 101, boys and girls! Today we learn a new word: Benchmarketing. Can you say benchmarketing? Good! Now we learn how to do it!

What is benchmarketing? Benchmarketing is a simple application of the principles of FUD, specifically, FUD principle #1, "exaggeration" (see the FUD 101 HOWTO for a detailed explanation). What you have to do is find some area, no matter how trivial, where your product is better than the opponent's. It doesn't matter that other products are better than yours in that area. It doesn't even matter whether the benchmark is related to how the product is used. All that matters is bashing the opponent.

Here's an example of benchmarketing:

Herring vs. Windows NT

- Herring Windows NT
Swimming Speed: 5 mph 0.1 mph
Cost: $1.35 $500
TCO (year): $492.75 * $6,180 **
* based on one tin of herring per day, 365 days per year
** based on 1998 IDC report on TCO for Windows NT
Windows NT represents poor value proposition
What Consumers Want Herring Windows NT
Nutrition YES NO
Ease of Preparation YES NO
Pleasing Texture YES NO
Taste YES NO
Goes with crackers YES NO
Preferred by penguins YES NO
Easy to use YES ???

Herring represents a better value proposition than Windows NT. It has a lower initial cost, a lower total cost of operation, and has a 500% speed advantage over Windows NT. In addition, it has all those nifty features that consumers want, like taste, nutrition, and a pleasing texture, while Windows NT has none of those features. In conclusion, please buy herring for all of your computing needs.

Part Deaux: HUH?!

By now, boys and girls, you have probably figured out that you were duped. Let's detail exactly how you were duped, and how you can tell you're being benchmarketed tomorrow.
  1. Statistics unrelated to use: The statistics gathered above are totally unrelated to the use of either Windows NT or herring as a computing platform.
  2. Ignoring superior competitors: A shark is a faster swimmer than a herring, but we conveniently left it out of our chart.
  3. Irrelevant features: The features gathered are totally irrelevant to the use of herring or Windows NT as a computing platform.
  4. Leaving out statistics: We deliberately left out any statistics that might make herring look bad as a computing platform. For example, the herring's brain is less powerful at number-crunching than any known microprocessor, thus making herring a poor choice for mathematical programming. We don't mention that, just as Microsoft never mentions that NT 4.0 is ill-suited for use as a corporate firewall when compared to FreeBSD or Linux.
  5. Conclusions not following from the facts: The conclusion reached (that herring was preferable to Windows NT as a computing platform) was not supported by any of the facts mentioned. What we are trying to do here is snow you with facts, and then slip in a conclusion and hope (FUD distraction method #1) that the fact that we've sandwiched the false statement between true facts will distract you enough so that you don't spot that it is false.

Part Tres: Conclusions

Whenever you see benchmarks, you should be careful to examine whether the benchmarks involved are applicable to the problem set that you're trying to solve. If they are not, then the benchmark is irrelevant.

Secondly, you should examine how meaningful the benchmark is. If the benchmark says that solution (a) does 'x' operations per second, solution (b) does 'x+1000' operations per second, but only need to do 'x/10' operations per second, then the fact that one solution can do more operations per second than the other is meaningless. Both will fill your needs, and you should choose which one is most cost-effective for your situation. Why buy a Mercedes when all you need is a Honda Civic?

For example, if you're trying to set up a low-cost web server for your small business to feed a T1 line, a $499 eMachine 333 running Linux will saturate that T1 just as assuredly as a $35,000 quad-processor Dell with four network cards and Windows NT. This is a case where a benchmark (the Dell is MUCH faster than the eMachine) is irrelevant.

Finally: Be aware. Benchmarketing is nothing new. IBM was a past master of benchmarketing back in the 70's, for example (they invented 'TPS', Transactions Per Second, to quantify how their mainframes were superior to the competition). Whenever you see benchmarks in the press, there is a 90% certainty that you are seeing benchmarketing in action. Be careful out there.

Eric Lee Green is the networking and systems guru for Enhanced Software Technologies Inc., "The Bru Guys". He deeply regrets the use of a hotmail.com EMAIL address but hastily explains that he's had it since before Hotmail was sold to Microsoft, and further excuses it by saying that he'd prefer SPAM to burden Microsoft rather than his employer.