Are Suites really “Sweet”?

The proliferation of “Software Suites” to deal with the ever increasing “malware” problems raises serious issues for users of these products.

As the market for anti-virus software has stabilized, the major anti-virus software companies are entering other sections of the malware market in order to expand their user base and also to secure their current customers.

Depending upon the vendor, these suites now encompass not only the traditional anti-virus area, but have also expanded into the Firewalls, Anti-Spyware, Anti-Trojan, Anti-Keylogger, Anti-Phishing And Anti-Spam areas. This shift in emphasis and expansion is one result of the stabilization if not the decline of the anti-virus market and the movement of those writing malicious software into the lucrative adware/spyware area. In addition, as the distinction between anti-virus and anti-spyware products has blurred, considerable overlap of software capabilities has developed.

At first appraisal by computer users, the evolution of suites appears as a desirable solution to a continually evolving problem. The question arises whether these suites are really serving the end user’s needs?

In order to ensure adequate protection and to protect their customer base, the software suites appear to have adopted a philosophy that there is no need to co-exist with other competing programs. Thus users attempting to maintain dual means of protection are thwarted both by slowdowns of their computer system as well as by direct conflicts with users’ competing products. The use of resources by the suites is heavy even without competing components. The adage “jack of all trades, but master of none” seems appropriate here because the results of numerous tests clearly demonstrate that no single product can be relied upon to provide the required protection because rapid changes characteristic of the spyware area are in sharp contrast to the previous viral area. Thus users must seek and use multiple products to provide them the required margin of safety.

Co-existing with suites is a challenge because of the aforementioned problems, not the least of which is the heavy use of system resources by the “suite” products. The result of this dilemma is that users must rely upon products which are “light” on system resources. We have realized this requirement, as no single product can catch every infection on a given day, and have specifically designed our product, SUPERAntiSpyware, to be light on system resources and co-exist with existing security solutions. I am certain other companies will follow suit. Users will ultimately dictate the verdict on whether “suites are sweet.”.

The Importance of Testing Methodology

Testing Methodology

In today’s oversaturated market of anti-spyware/malware/adware applications, it is becoming increasingly difficult for users to determine which applications will perform best for their specific needs. Thus, they look for standardized and legitimate “comparative tests” of these applications.

Testing anti-spyware applications is not an easy task. It is imperative that those who are going to undertake the task of testing need to have the skills to perform the tests competently and to test the products in real-world situations. Otherwise they are not performing a service to users. Users also need to examine the credibility of the party testing the applications and not simply “look at the numbers.” Currently, most tests are not comparing “apples to apples” because every anti-spyware application uses different methods of reporting the “numbers” of infections detected and removed.

There are standardized and widely accepted elements of any investigative report. These include an Introduction, Materials and Methods or Procedures, Results, Discussion and usually, but not always, Conclusions.

The most critical elements of an adequate report or investigation are to provide the reader with the Materials and Methods used which would allow others to duplicate the experiment or investigation to determine the validity of the results; that is, are the results reproducible in the hands of others using the same procedures (Materials and Methods). Thus, the methodology used in any investigation must be of sufficient detail to allow any interested parties the opportunity to independently validate the results.

In using non-standardized methods, it is critical to provide detailed procedure in order to ensure validation by allowing reproduction of the results by others.

When or if it is determined that the methodology is itself flawed or contains documented errors, this invalidates the results and casts serious concerns on other components or elements of the entire methodology and on the results.

The level of detail cannot be assumed or taken on faith. Therefore, it is of the utmost importance to provide a level of detail which removes any ambiguity as to how something was done and to provide and detail the safeguards used to ensure that the procedures were indeed followed.

In examining the testing methodology used by recent tests by Malware-Test.com, it is unclear whether their own procedures were followed when flaws or errors are discovered as detailed elsewhere. This casts doubt on how other elements were carried out. Furthermore, it is one thing to say how you are carrying out the testing and another to actually follow the protocol. Thus, alleged transparency by providing the purported methodology cannot in and of itself be accepted on faith and can be extremely misleading particularly in view of any demonstrated inadequacies.

Malware testing is certainly a daunting task and adequate documentation of methodology is the single most important element in validation of the results. When testing is performed by individuals one can accept and or excuse minor inadequacies. However, when the results are performed by alleged experts, in testing facilities which exist for testing purposes, they must be held to the highest standards.