Are software companies shielding themselves from academic scrutiny?

Last month I was fortunate enough to attend the Professional Association of SQL Server users (PASS) summit in Seattle Washington. The conference boast of being the largest gathering data pro in the word. Day two of the conference started with a keynote address from Adjunct MIT Prof David Dewitt. Dr. Dewitt has made significant contributions towards advancing data profession. Dewitt did not disappoint as he delivered a masterful presentation on comparing different vendor offerings of cloud data warehouses. After the presentation I stumbled across some interesting information surrounding a technical paper published by Dr. Dewitt in 1982. The story goes he developed a new benchmark test for measuring database performance. The results of test showed Oracle performed poorly. The CEO of Oracle, Larry Ellison, apparently was not thrilled with the publication. He took his frustrations to Dewitt's boss, the University department chair, and demanded Dewitt be fired. Unfortunately for Larry no one explained to him how tenure works. Shortly after the ordeal Oracle instituted the "DeWitt Clause" into their database system's end user license agreement (ELUA). The clause specifically prohibits naming the software in any form of benchmark study without permission, effectively giving themselves the ability to prevent the publication of poor test results. Both Microsoft and Sysbase quickly implemented their own versions of the Dewitt Clause shortly afterward too.

Current State of Benchmarking

The DeWitt clause undoubtedly provides a hurdle for comparing systems for academics and data pros but it's not the only problem. The process of bench-marking is just too slow in today's fast development cycles. I can't see results remaining relevant for over two years. All the important pieces of a database system are constantly evolving e.g. internals code base, hardware, operating system.

As long as software giants have control over what gets published I fear consumers will pay the price. I would like to see the DeWitt clause removed from future ELUAs. Scrutiny from people like Dr. DeWitt should be embraced and used to build a better system.