In today’s oversaturated market of anti-spyware/malware/adware applications, it is becoming increasingly difficult for users to determine which applications will perform best for their specific needs. Thus, they look for standardized and legitimate “comparative tests” of these applications.
Testing anti-spyware applications is not an easy task. It is imperative that those who are going to undertake the task of testing need to have the skills to perform the tests competently and to test the products in real-world situations. Otherwise they are not performing a service to users. Users also need to examine the credibility of the party testing the applications and not simply “look at the numbers.” Currently, most tests are not comparing “apples to apples” because every anti-spyware application uses different methods of reporting the “numbers” of infections detected and removed.
There are standardized and widely accepted elements of any investigative report. These include an Introduction, Materials and Methods or Procedures, Results, Discussion and usually, but not always, Conclusions.
The most critical elements of an adequate report or investigation are to provide the reader with the Materials and Methods used which would allow others to duplicate the experiment or investigation to determine the validity of the results; that is, are the results reproducible in the hands of others using the same procedures (Materials and Methods). Thus, the methodology used in any investigation must be of sufficient detail to allow any interested parties the opportunity to independently validate the results.
In using non-standardized methods, it is critical to provide detailed procedure in order to ensure validation by allowing reproduction of the results by others.
When or if it is determined that the methodology is itself flawed or contains documented errors, this invalidates the results and casts serious concerns on other components or elements of the entire methodology and on the results.
The level of detail cannot be assumed or taken on faith. Therefore, it is of the utmost importance to provide a level of detail which removes any ambiguity as to how something was done and to provide and detail the safeguards used to ensure that the procedures were indeed followed.
In examining the testing methodology used by recent tests by Malware-Test.com, it is unclear whether their own procedures were followed when flaws or errors are discovered as detailed elsewhere. This casts doubt on how other elements were carried out. Furthermore, it is one thing to say how you are carrying out the testing and another to actually follow the protocol. Thus, alleged transparency by providing the purported methodology cannot in and of itself be accepted on faith and can be extremely misleading particularly in view of any demonstrated inadequacies.
Malware testing is certainly a daunting task and adequate documentation of methodology is the single most important element in validation of the results. When testing is performed by individuals one can accept and or excuse minor inadequacies. However, when the results are performed by alleged experts, in testing facilities which exist for testing purposes, they must be held to the highest standards.