74 replies to this topic

[Trilobite]

UPDATE August 1, 2004: I have updated the number of infected test files to 758. I have added an attachment containing a chart of which test files were identified or missed as containing malicious code. (Replaced on 8/02 with an easy to read text file instead of a html in a zip file ) My testing of several thousand infected test files has been postponed due to a hard disk crash.

I recently acquired a few more possible viruses (I am now up to 758 possible Trojan, backdoor, and virus infected files). All of these test files were acquired from in the wild, mostly through cleaning of infected computers and through catching them in the act. There are no known zoo viruses in these tests. Each and every test file was identified as containing malicious code by at least one of these antivirus programs.

Please note that some of these files represent several variations of the same Trojan/virus/malware. It is also possible for one file to contain the code of more than one Trojan, virus or malware. There might also be a few duplicates.

All of the scans were preformed with the following options (if available): scan all files, scan compressed executables, scan inside archive files, and high heuristics.

(The software name and virus definition date precede each test result)

AntiVir Personal Edition (AVPE)
Program: v6.26.00.00 VDF-File v6.26.0.53 from 07.30.20043
647 Possible Viruses/malware/Trojans Found in 647 files out of a total of 758 files!
Approximately 85.36% detection. (Based on number of infected files, not number of infections)

Avast! 4, VPS file version: July 29, 2004 - [0431-2]
620 Possible Viruses/malware/Trojans Found in 607 files out of a total of 758 files!
Approximately 80.08% detection. (Based on number of infected files, not number of infections)

AVG 7.0.253 Professional, Virus Base 264.1.0 7-29-2004:
532 Possible Viruses/malware/Trojans Found in 523 files out of a total of 758 files!
Approximately 69.00% detection. (Based on number of infected files, not number of infections)

eTrust Anti-Virus, 30-07-2004:
575 Possible Viruses/malware/Trojans Found in 576 files out of a total of 758 files!
Approximately 75.99% detection. (Based on number of infected files, not number of infections)
585 Possible Viruses/malware/Trojans Found using non-standard scan!

F-PROT ANTIVIRUS, 30 July 2004
673 Possible Viruses/malware/Trojans Found in 667 files out of a total of 758 files! (Infected: 496, Suspicious: 117)
Approximately 88.00% detection. (Based on number of infected files, not number of infections)

Kaspersky Anti-Virus, Updated: 30-07-2004:
756 Possible Viruses/malware/Trojans Found in 750 files out of a total of 758 files!
Approximately 98.94% detection. (Based on number of infected files, not number of infections)

McAfee, Virus data file v4382 created Jul 28 2004:
756 Possible Viruses/malware/Trojans Found in 745 files out of a total of 758 files!
Approximately 98.28% detection. (Based on number of infected files, not number of infections)

nod32, (20040730) NT
630 Possible Viruses/malware/Trojans Found in 623 files out of a total of 758 files!
Approximately 82.19% detection. (Based on number of infected files, not number of infections)
633 Possible Viruses/malware/Trojans Found using ‘Advanced heuristics’!

Panda Titanium 2004 Anti-Virus, Updated: 07-30-2004:
699 Possible Viruses/malware/Trojans Found in 688 files out of a total of 758 files!
Approximately 90.77% detection. (Based on number of infected files, not number of infections)

Symantec's Norton Antivirus, 7/30/2004:
740 Possible Viruses/malware/Trojans Found in 701 files out of a total of 758 files!
Approximately 92.48% detection. (Based on number of infected files, not number of infections)

Attached is a chart of which test files were identified or missed as containing malicious code by each AV program.

Did your favorite antivirus perform poorly in this test? There are a lot of factors that could have caused this:
Each AV program uses a different virus database. Some containing more malicious signatures than others, meaning some AV programs will have higher detection rates than others.
Each AV company has their own interpretation of what constitutes malware. Some AV companies only want their product to target primarily viruses and worms, and to a lesser degree Trojans and exploits, and to an even lesser degree (or not at all), spyware, hijackers and adware. For example, if you look at the attached chart, you may notice that several of the tested AV programs miss a significant number of the Trojans.
Considering there are roughly 100,000 (or more) unique infections in the wild, a population sample of 758 infected files may not accurately represent true detection rates of AV programs.
Could poor detection of certain AV programs be due to ‘zoo’ viruses in this test sample?
Not likely. First of all, many AV programs will detect zoo viruses. Second, all of these test files were obtained from within the ‘wild’, meaning that all of these files exist outside of laboratories and they have been (unfortunately) released out into the real world. ‘Zoo’ viruses are proof of concept viruses or otherwise unreleased viruses and generally do not exist outside of controlled laboratories. There are no known zoo viruses in these tests.
These test are NOT to determine which AV software is superior, this is just a test on 758 POSSIBLE Trojan, backdoor, and virus infected files.

Please, do not PM, e-mail or otherwise ask for any of these files. These are live viruses, they can do serious damage to your system and others. They are NOT available for sale or trade and will NOT be distributed to anyone. ALL requests for these files WILL BE IGNORED!

Attached Files

•  AV_detection_tests.txt   85.41KB   5560 downloads

Edited by Trilobite, 02 August 2004 - 02:11 PM.

[lonewolf]

I think a more accurate test, to determine how good the different scanners are, would be to use 90 different viruses. 9 is just not enough to get an accurate picture on their true performance.

[Trilobite]

I agree, the more the better, however so far all I have to test on is nine. But even with only 9 viruses, several of the antivirus programs that I tested, failed to detect all or most of them.

[jerry4dos]

Ahh -- comparison testing! Just my cup of tea!

I conducted some similar tests on my small (10) virus collection, with nearly the same results the first time through.

Trilobite, when you conducted your tests, did you use the "On-Demand Scan" feature, or the "On-Access Scan"? (These are McAfee's terms.) I found the "On-Access" feature of ALL the different versions less reliable than the "On-Demand" scan, with McAfee being the best of the lot at intercepting stuff "on the fly."

Also, did you set your scan options to include things like "scan inside archives," "scan compressed executables," and "dumb scan of all files"? When I did this with F-PROT, it spotted all-but-one file that McAfee flagged as a TROJAN ("JS/Keylog-Briss.ldr").

I really like the F-PROT package that includes a DOS-based scanner. It lets me scan and trap things without loading any of my Win95 GUI files.

Jerry

[Trilobite]

Please post your results. The more tests that are conducted with different viruses, the better the comparisons will be.

I initiated an active "On-Demand Scan" of the test files. All of my scans included the highest detection level like scan inside archived ‘ziped’ files and compressed exe files. My F-Prot test did include the “dumb file scan”.

Presently, I prefer to use McAfee’s superdat commandline virus scanner as my dos scanner.

[mrrockford]

Howdy,

Please run your test with AntiVir's AVPE, available for download at www.free-av.com . I would like to see how it matches up to the others.

[Sanzio]

I was running Norton but now have Sophos and am much happier with results. I would interested to see what you think in your next test.

[Trilobite]

I have updated the results at the top of this thread to include AntiVir's AVPE.
As the computer that I use to test this software on has no Internet connection, I was unable to update to the most recent virus definitions. However since all of the possible viruses that I tested this software on are older than the virus definitions used (06-18-2004) this should not be a problem.

I have not tested Sophos yet, I might give it a try after the holiday weekend.

Edited by Trilobite, 02 July 2004 - 03:40 PM.

[Trilobite]

I have acquired quite a few new test files and have completely redone the test.
The updated results can be found in the first post of this topic.

[Paranoid]

Results are kind of expected so far.

[Trilobite]

UPDATE July 15, 2004: Added 7 new test files (Netsky, Zafi, Bagle…), updated definitions and rescanned. Added results for Panda Titanium 2004 and Nod32. Removed eTrust (I have been unable to obtain the latest trial version or updates). The next update should be a more accurate as it will use ~3,000 infected test files instead of 43.

Edited by Trilobite, 15 July 2004 - 05:48 PM.

[NAMOR]

Hey Trilobite,

You think you can run the NOD32 test again , but with the /AH (Advanced Heuristic) option on? I just want to see if there is a difference deep heuristics and advanced.

NOD32 AH special shell extension here:
http://www.wildersse...read.php?t=9776

[Trilobite]

I’ll give it a go, if it works with the trial version that is. It may be a few days before I can run the test, I am presently setting up to run tests on a rather large virus collection on DVD (somewhere between 3,000 and 30,000 infected files).

Please note that the virus definitions for Nod32 are a month out of date. It is the most recent update that I have been able to obtain with the trial version.

I should have mentioned this in the results but all of the scans were preformed with the following options (if available): scan all files, scan compressed executables, scan inside archive files, and high heuristics.

[NAMOR]

no hurry, thanks Trilobite...

[Trilobite]

@ NAMOR,

It took some finagling, but I finally was able to update the NOD32 definitions. I was previously unable to obtain the updates because the computer I am testing on is isolated from the network for obvious security reasons. I was also able to finally download and update the trial version of eTrust. The updated results have been edited into my above post.

Both the deep and advanced Heuristics for NOD32 found 26 infected files. I will rerun the scans when I have everything setup for the large virus collection that is currently on DVD. This may take a few days/weeks as my test computer is currently lacking in a DVD drive.

[NAMOR]

Thanks Trilobite, I look forward to your next test.. Are you getting you samples for the VX site?

[Trilobite]

Are you getting you samples for the VX site?

No. Because of the damaging nature of some of the viruses and worms, these files will not be for sale or trade and will not be given to anyone.

I hope I will still be able to run the tests on the large collection. Our plans were to burn them to DVD in order to transfer them to the test computer, however the hard disk that the viruses are initially stored on reported an i/o error yesterday. I do not know yet how many of the files, if any, were salvageable.

[Trilobite]

I have updated and changed the format of the results. I have added a number of infected test files. It is now 374. This should yield more accurate results. Because of this, it is no longer practical to list the individual results for each file.
I have not yet sorted the test files and as such, there may be a few duplicates.
Please note, I have not yet had the time to update all of the definitions, however all definitions used are within a week and a half of each other. I will update all definitions and rescan when I have transferred the entire collection to my test PC.

[Paranoid]

So what are you going to call your tests?

Something catchy?

[Trilobite]

So what are you going to call your tests?

Something catchy?

Bad pun!

But on the other hand, that is kind of infectious.

[Alya]

Trilobite
Did you use regular or extended anti-viral bases for Kaspersky when you did the test?

[Trilobite]

Did you use regular or extended anti-viral bases for Kaspersky when you did the test?

I am unfamiliar with the two different versions that you mention.
Could you post more information?

For my tests, I used the cumulative, weekly and daily updates to the virus definitions.

[Paranoid]

Trilobite
Did you use regular or extended anti-viral bases for Kaspersky when you did the test?

Shouldn't make a difference. Unless the test sample has non-virus/worm/trojans.

[KinG]

I am unfamiliar with the two different versions that you mention.
Could you post more information?

from Internet, standard databases - download standard anti-virus database from the Kaspersky Lab update server. Standard anti-virus database allows you to detect all currently existing malware and disinfect objects and data infected with such malware.

from Internet, extended databases - download extended anti-virus database from the Kaspersky Lab update server. Apart from detecting all currently existing malware, extended database allows you to detect programs providing to intruders remote access to your data.

For my tests, I used the cumulative, weekly and daily updates to the virus definitions.

That is just when the update has been given out. http://www.kaspersky...apter=146271794

Edited by KinG, 25 July 2004 - 01:46 PM.

[Trilobite]

Currently, the test files consist of worms, viruses, macro viruses, Trojans, downloader Trojans, vbs, 2 boot viruses, at least 1 BIOS virus (Chernobyl), 2 or 3 java worms and there might be 2 or 3 Linux viruses. The vast majority are Win32 worms.

Ideally, I would like to list or chart which infections were found in each file for each AV program. The problem arises when I use a large collection, not all of the AV programs create easily accessible log files which makes copying the data by hand very tedious. Plus I am not sure exactly how large one post can be on this forum, the logs may not all fit in one nice, neat post. I think what I may do is rename the files with name of the virus they contain (i.e. W32_NetSky.p@mm.vir) and then attach a text file containing a listing of the test files. if I have time (and there is room) I may include some of the logs in the attached text file.

Thanks KinG.
I might attempt to get the extended databases and test to see if there is any difference. If it is only available via Internet update, then it might not be possible to test as my test computer is completely isolated from any networks for obvious security reasons.

[KinG]

Here are some more information:
About how to perform antivirus database
Kinds of updates
Additional anti-virus databases
I do not think there is a SuperSecure database option anymore or maybe it is already included in the extended. Also, I believe the updates they offer on their website are standard.

If it is only available via Internet update, then it might not be possible to test as my test computer is completely isolated from any networks for obvious security reasons.

How are you getting updates then?

[Trilobite]

How are you getting updates then?

I download the updates to disk and update the scanners manually.

[Alya]

Trilobite
You can also download extended updates to disk just the same way as regular ones and update the scanner manually. Just change "updates" to "updates_ext" in your KAV Updater URL's (see http://www.kaspersky...pter=146235718)

Paranoid
Yes, you are right. But just in case any of his pests classified as spyware/malware by KAV (although from his description doesn't sound like it).

I'm just somewhat disappointed that McAfee outperformed KAV
And looking for a "reason" why this could be happeneing.

[Trilobite]

@Alya,
Thanks.
I’m in the process of updating my tests. It may be a few day/weeks before I post the next update.

If anyone is interested, attached is a listing of the test files that I am currently using. The list is still not completely sorted and there are a number of variations of single viruses and the possible duplicate. Where possible, I have renamed the infected file to the name of the virus/worm/Trojan and have added an extension of .txt to avoid accidental execution of the file.

[Paranoid]

I'm just somewhat disappointed that McAfee outperformed KAV  
And looking for a "reason" why this could be happeneing.

McAfee has being in this business as long as anyone so they have a broad database, most tests I'm familar with give them good results in terms of coverage. Usually 1st or 2nd with KAV. So I'm not surprised they do well.

[Trilobite]

McAfee has a very large virus database. I’ll have to check it against Kaspersky. I think they are fairly close. I do know that McAfee’s, and possibly Kaspersky’s, databases are larger than Symantec’s database.

I did a quick scan with McAfee fully updated and out of the 759 test files listed in my previous post, McAfee missed only 3 or 4 infections (out of 759*). Please note that this does not mean that McAfee missed 3 or 4 files. McAfee reports a number of the test files as having more than one different infection. At my last count, McAfee missed somewhere between 14-23 files.

Edited by Trilobite, 27 July 2004 - 10:00 AM.

[wreck]

I would be curious to see what each individual A-V had to say in defense of their application. I suspect that each would claim that some nasties are listed along with another listing or the virus is actually a trojan and not looked for. Are the infections you tested all viruses or are some trojans??

[Trilobite]

I would be curious to see what each individual A-V had to say in defense of their application. I suspect that each would claim that some nasties are listed along with another listing or the virus is actually a trojan and not looked for. Are the infections you tested all viruses or are some trojans??

The classification does vary from AV to AV. As for the type of test files, see this list. It is attached to a previous post in this thread

I have attempted to rename the files to the name of the virus/Trojan that McAfee detects in them. Some of the others have been renamed using Kaspersky. I have not renamed most of the ‘trojan’ files yet.

[Paranoid]

A very common answer from companies would be

"We are aware of the viruses in your sample, but most of them are zoo viruses, and are not found outside the labs (in wild), as such it would serve no practical purpose in including them."

Lots of people have done antivirus tests with lots and lots of downloaded malware, and many of their results are attacked, because of poor methodology.

[wreck]

Thanks, Paranoid. That was pretty much what I was thinking. In the "real world", I would bet that the major names in A-V that update daily are pretty much the same in effectiveness.

[Trilobite]

A very common answer from companies would be

"We are aware of the viruses in your sample, but most of them are zoo viruses, and are not found outside the labs (in wild), as such it would serve no practical purpose in including them."

Common, perhaps. In this test however, incorrect.
To my knowledge there are no zoo viruses in my test, let alone most of them.

[Paranoid]

I don't mean to be aggressive, trilobyte but what specific procedures do you do to ensure that each and every one of your virus samples are really in the wild?

We are talking about some 700+ of them after all.

[Trilobite]

This collection is a combination of infected files that I have collected from in the wild and infected files that another person has collected from in the wild. None of these samples were obtained from antivirus companies or antivirus testing organizations, which are pretty much the only places you will even find a zoo virus.

Now, I don't mean to be aggressive, paranoyd but what specific procedures do you do to come to the conclusion that the vast majority of my virus samples are zoo viruses? After all, you do not even have access to this collection in order to properly make such a determination.

[Paranoid]

This collection is a combination of infected files that I have collected from in the wild and infected files that another person has collected from in the wild.  None of these samples were obtained from antivirus companies or antivirus testing organizations, which are pretty much the only places you will even find a zoo virus.

Places where you get zoo viruses are hacker sites.

Now, I don't mean to be aggressive,

Really? Here you do sound aggressive.

paranoyd but what specific procedures do you do to come to the conclusion that the vast majority of my virus samples are zoo viruses?  After all, you do not even have access to this collection in order to properly make such a determination.

Because I have seen a lot of such tests, where people go to various shall we say "dark" sites, download all the stuff there, than make no effort to verify if the samples are what they say they are, unloose their scanners on them and then announce their results.

[Trilobite]

@paranoid,
I am not going to feed the trolls.
I have made myself clear on the content of these test files and how they were obtained.

@All
I apologize for the petty bickering of the last several posts.

[Paranoid]

Trilobyte

I think that asking for clarification about the nature and origin of your test files is far from trolling. If you wish your test to be treated seriously, questions of such nature will be asked. (And no, you didn't really answer)

I did not perceive any "bickering", until in your last post.

[NAMOR]

Nice test again..

I see something strange about NOD32 with all the test I have viewed. This AV always seems to either be at the top of the list or at the very bottom. Anywho... Here is another AV for you to try if you like Trilobite called MKS_Vir 2004. It seem to be an AV from a polish company which now has a distrubutor for the US and Canada... It's been a hot topic at other forums... Haven't tried it myself though...

http://www.stormbyte.com/

Edited by NAMOR, 31 July 2004 - 09:15 AM.

[Paranoid]

Nature of the test set, NOD is weak on trojans.

[NAMOR]

thank you Paranoid... Guess it's good that I also have KAV 4.5 (on demand only) and boclean then.

[Alya]

Trilobite
I was very intrigued by your tests, so I got McAfee VirusScan Enterprise 8 and tested it against KAV Personal Pro 4.5.0.48

These are my results:

Wild Viruses
(I disinfected several computers using KAV and accumulated 391 viruses in my KAV "Infected" folder from these disinfections. Computers belong to people with limited understanding of computer security and pretty average computer usage patterns - fair E-mail usage, browsing using Internet Explorer, word processors, some downloading esp games, one person had Kazaa. This KAV "Infected" folder was scanned with McAfee VirusScan).

KAV: 391 infected files
McAfee: 102 infected files

Then I wanted to see if McAfee would pick up anything after KAV. I just finished disinfecting one computer with KAV (which picked up 33 viruses there in few hundred infected files). So I scanned that computer with McAfee. McAfee picked up one virus only (W32/Sdbot.worm.gen.i) in one infected file.

Zoo Viruses
I downloaded and scanned three hacker sites (total number of viruses on each site is, of course, unknown):

Site A (viruses/trojans)
KAV: 25 viruses, 40 infected files
McAfee: 33 infected files

Site B (viruses/trojans)
KAV: 251 viruses, 354 infected files
McAfee: 38 infected files

Site C (exploits)
KAV: 1 virus, 1 infected file (exploit)
McAfee: 3 infected files (exploits)

Edited by Alya, 31 July 2004 - 06:03 PM.

[Trilobite]

I have updated the results of my tests. The first post of this tread has been edited accordingly.

I have updated the number of infected test files to 758. I have added an attachment containing a chart of which test files were identified or missed as containing malicious code. You will need an archival utility such as winzip or winrar to uncompress the attachment. My testing of several thousand infected test files has been postponed due to a hard disk crash.

@ NAMOR,
Thanks. I’ll look into testing MKS_Vir 2004.

@ Alya,
I didn’t see any difference between the regular and extended Kaspersky databases. The detection is still very good with the regular database.

I disinfected several computers using KAV and accumulated 391 viruses in my KAV "Infected" folder from these disinfections.

Viruses, Trojans and malware in general are very easy to pick up this way. This is how I obtained a lot of mine. But then again, these types of files don’t exactly hide. By nature, most of them self replicate and attempt to infect other systems.

[NAMOR]

TO Trilobite or anyone else,

Have you seen test results for either of these Open Source AV programs? Might be interesting to see how well they compare to the "name-brand" programs.

http://www.clamwin.com/

http://www.openantivirus.org/

Edited by NAMOR, 01 August 2004 - 06:12 PM.

[billaku]

Please consider adding:


F-Secure Anti-Virus 2004.
Virus Bulletin states that F-Secure Anti-Virus has two main scan engines: F-Prot and Kaspersky.

Would be interesting seeing how the 3 compare: Kapspersky, F-Prot, F-Secure.


Softwin BitDefender


Also interested in ClamAV being added.


Much thanks for any consideration.

[Trilobite]

I have been busy recently and haven’t had the time to run any new tests with other AV programs and probably won’t have time to update the scans for a while. Not to mention that several of the AV trial versions have expired. I will make note of all of the requested AV programs and if time permits, I will test them.

I still do not have any word on the recovery of the crashed hard disk…the bulk of the several thousand infected files were stored on it and we have been working on finding the backups of that drive and on the data recovery of that drive.