• Announcements

    • Budfred

      IE 11 copy/paste problem

      It has come to our attention that people using Internet Explorer 11 (IE 11) are having trouble with copy/paste to the forum. If you encounter this problem, using a different browser like Firefox or Chrome seems to get around the problem. We do not know what the problem is, but it seems to be specific to IE 11 and we are hopeful that Microsoft will eventually fix it.
    • Budfred

      PLEASE READ - Reversing upgrade   02/23/2017

      We have found that this new upgrade is somewhat of a disaster.  We are finding lots of glitches in being able to post and administer the forum.  Additionally, there are new costs associated with the upgrade that we simply cannot afford.  As a result, we have decided to reverse course and go back to the previous version of our software.  Since this will involve restoring it from a backup, we will lose posts that have been added since January 30 or possibly even some before that.    If you started a topic during that time, we urge you to make backups of your posts and you will need to start the topics over again after the change.  You can simply paste the copies of your posts that you created at that point.    If you joined the forum this month, you will need to re-register since your membership will be lost along with the posts.  Since you have a concealed password, we cannot simply restore your membership for you.   We are going to backup as much as we can so that it will reduce inconvenience for our members.  Unfortunately we cannot back everything up since much will be incompatible with the old version of our software.  We apologize for the confusion and regret the need to do this even though it is not viable to continue with this version of the software.   We plan to begin the process tomorrow evening and, if it goes smoothly, we shouldn't be offline for very long.  However, since we have not done this before, we are not sure how smoothly it will go.  We ask your patience as we proceed.   EDIT: I have asked our hosting service to do the restore at 9 PM Central time and it looks like it will go forward at that time.  Please prepare whatever you need to prepare so that we can restore your topics when the forum is stable again.
Sign in to follow this  
Followers 0
Trilobite

Antivirus tests

75 posts in this topic

UPDATE August 1, 2004: I have updated the number of infected test files to 758. I have added an attachment containing a chart of which test files were identified or missed as containing malicious code. (Replaced on 8/02 with an easy to read text file instead of a html in a zip file ;) ) My testing of several thousand infected test files has been postponed due to a hard disk crash.

 

I recently acquired a few more possible viruses (I am now up to 758 possible Trojan, backdoor, and virus infected files). All of these test files were acquired from in the wild, mostly through cleaning of infected computers and through catching them in the act. There are no known zoo viruses in these tests. Each and every test file was identified as containing malicious code by at least one of these antivirus programs.

 

Please note that some of these files represent several variations of the same Trojan/virus/malware. It is also possible for one file to contain the code of more than one Trojan, virus or malware. There might also be a few duplicates.

 

All of the scans were preformed with the following options (if available): scan all files, scan compressed executables, scan inside archive files, and high heuristics.

 

(The software name and virus definition date precede each test result)

 

AntiVir Personal Edition (AVPE)

Program: v6.26.00.00 VDF-File v6.26.0.53 from 07.30.20043

647 Possible Viruses/malware/Trojans Found in 647 files out of a total of 758 files!

Approximately 85.36% detection. (Based on number of infected files, not number of infections)

 

Avast! 4, VPS file version: July 29, 2004 - [0431-2]

620 Possible Viruses/malware/Trojans Found in 607 files out of a total of 758 files!

Approximately 80.08% detection. (Based on number of infected files, not number of infections)

 

AVG 7.0.253 Professional, Virus Base 264.1.0 7-29-2004:

532 Possible Viruses/malware/Trojans Found in 523 files out of a total of 758 files!

Approximately 69.00% detection. (Based on number of infected files, not number of infections)

 

eTrust Anti-Virus, 30-07-2004:

575 Possible Viruses/malware/Trojans Found in 576 files out of a total of 758 files!

Approximately 75.99% detection. (Based on number of infected files, not number of infections)

585 Possible Viruses/malware/Trojans Found using non-standard scan!

 

F-PROT ANTIVIRUS, 30 July 2004

673 Possible Viruses/malware/Trojans Found in 667 files out of a total of 758 files! (Infected: 496, Suspicious: 117)

Approximately 88.00% detection. (Based on number of infected files, not number of infections)

 

Kaspersky Anti-Virus, Updated: 30-07-2004:

756 Possible Viruses/malware/Trojans Found in 750 files out of a total of 758 files!

Approximately 98.94% detection. (Based on number of infected files, not number of infections)

 

McAfee, Virus data file v4382 created Jul 28 2004:

756 Possible Viruses/malware/Trojans Found in 745 files out of a total of 758 files!

Approximately 98.28% detection. (Based on number of infected files, not number of infections)

 

nod32, (20040730) NT

630 Possible Viruses/malware/Trojans Found in 623 files out of a total of 758 files!

Approximately 82.19% detection. (Based on number of infected files, not number of infections)

633 Possible Viruses/malware/Trojans Found using ‘Advanced heuristics’!

 

Panda Titanium 2004 Anti-Virus, Updated: 07-30-2004:

699 Possible Viruses/malware/Trojans Found in 688 files out of a total of 758 files!

Approximately 90.77% detection. (Based on number of infected files, not number of infections)

 

Symantec's Norton Antivirus, 7/30/2004:

740 Possible Viruses/malware/Trojans Found in 701 files out of a total of 758 files!

Approximately 92.48% detection. (Based on number of infected files, not number of infections)

 

Attached is a chart of which test files were identified or missed as containing malicious code by each AV program.

 

Did your favorite antivirus perform poorly in this test? There are a lot of factors that could have caused this:

Each AV program uses a different virus database. Some containing more malicious signatures than others, meaning some AV programs will have higher detection rates than others.

Each AV company has their own interpretation of what constitutes malware. Some AV companies only want their product to target primarily viruses and worms, and to a lesser degree Trojans and exploits, and to an even lesser degree (or not at all), spyware, hijackers and adware. For example, if you look at the attached chart, you may notice that several of the tested AV programs miss a significant number of the Trojans.

Considering there are roughly 100,000 (or more) unique infections in the wild, a population sample of 758 infected files may not accurately represent true detection rates of AV programs.

Could poor detection of certain AV programs be due to ‘zoo’ viruses in this test sample?

Not likely. First of all, many AV programs will detect zoo viruses. Second, all of these test files were obtained from within the ‘wild’, meaning that all of these files exist outside of laboratories and they have been (unfortunately) released out into the real world. ‘Zoo’ viruses are proof of concept viruses or otherwise unreleased viruses and generally do not exist outside of controlled laboratories. There are no known zoo viruses in these tests.

These test are NOT to determine which AV software is superior, this is just a test on 758 POSSIBLE Trojan, backdoor, and virus infected files.

 

Please, do not PM, e-mail or otherwise ask for any of these files. These are live viruses, they can do serious damage to your system and others. They are NOT available for sale or trade and will NOT be distributed to anyone. ALL requests for these files WILL BE IGNORED!

AV_detection_tests.txt

Edited by Trilobite

Share this post


Link to post
Share on other sites

I think a more accurate test, to determine how good the different scanners are, would be to use 90 different viruses. 9 is just not enough to get an accurate picture on their true performance.

Share this post


Link to post
Share on other sites

I agree, the more the better, however so far all I have to test on is nine. But even with only 9 viruses, several of the antivirus programs that I tested, failed to detect all or most of them.

Share this post


Link to post
Share on other sites

Ahh -- comparison testing! Just my cup of tea!

 

I conducted some similar tests on my small (10) virus collection, with nearly the same results the first time through.

 

Trilobite, when you conducted your tests, did you use the "On-Demand Scan" feature, or the "On-Access Scan"? (These are McAfee's terms.) I found the "On-Access" feature of ALL the different versions less reliable than the "On-Demand" scan, with McAfee being the best of the lot at intercepting stuff "on the fly."

 

Also, did you set your scan options to include things like "scan inside archives," "scan compressed executables," and "dumb scan of all files"? When I did this with F-PROT, it spotted all-but-one file that McAfee flagged as a TROJAN ("JS/Keylog-Briss.ldr").

 

I really like the F-PROT package that includes a DOS-based scanner. It lets me scan and trap things without loading any of my Win95 GUI files.

 

Jerry

Share this post


Link to post
Share on other sites

Please post your results. The more tests that are conducted with different viruses, the better the comparisons will be.

 

I initiated an active "On-Demand Scan" of the test files. All of my scans included the highest detection level like scan inside archived ‘ziped’ files and compressed exe files. My F-Prot test did include the “dumb file scan”.

 

Presently, I prefer to use McAfee’s superdat commandline virus scanner as my dos scanner.

Share this post


Link to post
Share on other sites

Howdy,

 

Please run your test with AntiVir's AVPE, available for download at www.free-av.com . I would like to see how it matches up to the others.

Share this post


Link to post
Share on other sites

I was running Norton but now have Sophos and am much happier with results. I would interested to see what you think in your next test.

Share this post


Link to post
Share on other sites

I have updated the results at the top of this thread to include AntiVir's AVPE.

As the computer that I use to test this software on has no Internet connection, I was unable to update to the most recent virus definitions. However since all of the possible viruses that I tested this software on are older than the virus definitions used (06-18-2004) this should not be a problem.

 

I have not tested Sophos yet, I might give it a try after the holiday weekend.

Edited by Trilobite

Share this post


Link to post
Share on other sites

I have acquired quite a few new test files and have completely redone the test.

The updated results can be found in the first post of this topic.

Share this post


Link to post
Share on other sites

UPDATE July 15, 2004: Added 7 new test files (Netsky, Zafi, Bagle…), updated definitions and rescanned. Added results for Panda Titanium 2004 and Nod32. Removed eTrust (I have been unable to obtain the latest trial version or updates). The next update should be a more accurate as it will use ~3,000 infected test files instead of 43.

Edited by Trilobite

Share this post


Link to post
Share on other sites

I’ll give it a go, if it works with the trial version that is. It may be a few days before I can run the test, I am presently setting up to run tests on a rather large virus collection on DVD (somewhere between 3,000 and 30,000 infected files).

 

Please note that the virus definitions for Nod32 are a month out of date. It is the most recent update that I have been able to obtain with the trial version.

 

I should have mentioned this in the results but all of the scans were preformed with the following options (if available): scan all files, scan compressed executables, scan inside archive files, and high heuristics.

Share this post


Link to post
Share on other sites

@ NAMOR,

 

It took some finagling, but I finally was able to update the NOD32 definitions. I was previously unable to obtain the updates because the computer I am testing on is isolated from the network for obvious security reasons. I was also able to finally download and update the trial version of eTrust. The updated results have been edited into my above post.

 

Both the deep and advanced Heuristics for NOD32 found 26 infected files. I will rerun the scans when I have everything setup for the large virus collection that is currently on DVD. This may take a few days/weeks as my test computer is currently lacking in a DVD drive.

Share this post


Link to post
Share on other sites
Are you getting you samples for the VX site?

No. Because of the damaging nature of some of the viruses and worms, these files will not be for sale or trade and will not be given to anyone.

 

I hope I will still be able to run the tests on the large collection. Our plans were to burn them to DVD in order to transfer them to the test computer, however the hard disk that the viruses are initially stored on reported an i/o error yesterday. :grrr: I do not know yet how many of the files, if any, were salvageable.

Share this post


Link to post
Share on other sites

I have updated and changed the format of the results. I have added a number of infected test files. It is now 374. :bounce: This should yield more accurate results. Because of this, it is no longer practical to list the individual results for each file.

I have not yet sorted the test files and as such, there may be a few duplicates.

Please note, I have not yet had the time to update all of the definitions, however all definitions used are within a week and a half of each other. I will update all definitions and rescan when I have transferred the entire collection to my test PC.

Share this post


Link to post
Share on other sites
So what are you going to call your tests?

 

Something catchy?

Bad pun!

 

But on the other hand, that is kind of infectious. :worm:

 

:2tu::rolleyes:

Share this post


Link to post
Share on other sites
Did you use regular or extended anti-viral bases for Kaspersky when you did the test?

I am unfamiliar with the two different versions that you mention.

Could you post more information?

 

For my tests, I used the cumulative, weekly and daily updates to the virus definitions.

Share this post


Link to post
Share on other sites
Trilobite

Did you use regular or extended anti-viral bases for Kaspersky when you did the test?

Shouldn't make a difference. Unless the test sample has non-virus/worm/trojans.

Share this post


Link to post
Share on other sites
I am unfamiliar with the two different versions that you mention.

Could you post more information?

 

from Internet, standard databases - download standard anti-virus database from the Kaspersky Lab update server. Standard anti-virus database allows you to detect all currently existing malware and disinfect objects and data infected with such malware.

 

from Internet, extended databases - download extended anti-virus database from the Kaspersky Lab update server. Apart from detecting all currently existing malware, extended database allows you to detect programs providing to intruders remote access to your data.

 

For my tests, I used the cumulative, weekly and daily updates to the virus definitions.

 

That is just when the update has been given out. http://www.kaspersky.com/avupdates?chapter=146271794

Edited by KinG

Share this post


Link to post
Share on other sites

Currently, the test files consist of worms, viruses, macro viruses, Trojans, downloader Trojans, vbs, 2 boot viruses, at least 1 BIOS virus (Chernobyl), 2 or 3 java worms and there might be 2 or 3 Linux viruses. The vast majority are Win32 worms.

 

Ideally, I would like to list or chart which infections were found in each file for each AV program. The problem arises when I use a large collection, not all of the AV programs create easily accessible log files which makes copying the data by hand very tedious. Plus I am not sure exactly how large one post can be on this forum, the logs may not all fit in one nice, neat post. I think what I may do is rename the files with name of the virus they contain (i.e. W32_NetSky.p@mm.vir) and then attach a text file containing a listing of the test files. if I have time (and there is room) I may include some of the logs in the attached text file.

 

Thanks KinG.

I might attempt to get the extended databases and test to see if there is any difference. If it is only available via Internet update, then it might not be possible to test as my test computer is completely isolated from any networks for obvious security reasons.

Share this post


Link to post
Share on other sites

Here are some more information:

About how to perform antivirus database

Kinds of updates

Additional anti-virus databases

I do not think there is a SuperSecure database option anymore or maybe it is already included in the extended. Also, I believe the updates they offer on their website are standard.

If it is only available via Internet update, then it might not be possible to test as my test computer is completely isolated from any networks for obvious security reasons.

How are you getting updates then?

Share this post


Link to post
Share on other sites
How are you getting updates then?

I download the updates to disk and update the scanners manually.

Share this post


Link to post
Share on other sites

Trilobite

You can also download extended updates to disk just the same way as regular ones and update the scanner manually. Just change "updates" to "updates_ext" in your KAV Updater URL's (see http://www.kaspersky.com/extraavupdates?chapter=146235718)

 

Paranoid

Yes, you are right. But just in case any of his pests classified as spyware/malware by KAV (although from his description doesn't sound like it).

 

I'm just somewhat disappointed that McAfee outperformed KAV :(

And looking for a "reason" why this could be happeneing.

Share this post


Link to post
Share on other sites

@Alya,

Thanks.

I’m in the process of updating my tests. It may be a few day/weeks before I post the next update.

 

If anyone is interested, attached is a listing of the test files that I am currently using. The list is still not completely sorted and there are a number of variations of single viruses and the possible duplicate. Where possible, I have renamed the infected file to the name of the virus/worm/Trojan and have added an extension of .txt to avoid accidental execution of the file.

Share this post


Link to post
Share on other sites

I'm just somewhat disappointed that McAfee outperformed KAV :( 

And looking for a "reason" why this could be happeneing.

 

 

McAfee has being in this business as long as anyone so they have a broad database, most tests I'm familar with give them good results in terms of coverage. Usually 1st or 2nd with KAV. So I'm not surprised they do well.

Share this post


Link to post
Share on other sites

McAfee has a very large virus database. I’ll have to check it against Kaspersky. I think they are fairly close. I do know that McAfee’s, and possibly Kaspersky’s, databases are larger than Symantec’s database.

 

I did a quick scan with McAfee fully updated and out of the 759 test files listed in my previous post, McAfee missed only 3 or 4 infections (out of 759*). Please note that this does not mean that McAfee missed 3 or 4 files. McAfee reports a number of the test files as having more than one different infection. At my last count, McAfee missed somewhere between 14-23 files.

Edited by Trilobite

Share this post


Link to post
Share on other sites

I would be curious to see what each individual A-V had to say in defense of their application. I suspect that each would claim that some nasties are listed along with another listing or the virus is actually a trojan and not looked for. Are the infections you tested all viruses or are some trojans??

Share this post


Link to post
Share on other sites
I would be curious to see what each individual A-V had to say in defense of their application. I suspect that each would claim that some nasties are listed along with another listing or the virus is actually a trojan and not looked for. Are the infections you tested all viruses or are some trojans??

The classification does vary from AV to AV. As for the type of test files, see this list. It is attached to a previous post in this thread

 

I have attempted to rename the files to the name of the virus/Trojan that McAfee detects in them. Some of the others have been renamed using Kaspersky. I have not renamed most of the ‘trojan’ files yet.

Share this post


Link to post
Share on other sites

A very common answer from companies would be

 

"We are aware of the viruses in your sample, but most of them are zoo viruses, and are not found outside the labs (in wild), as such it would serve no practical purpose in including them."

 

Lots of people have done antivirus tests with lots and lots of downloaded malware, and many of their results are attacked, because of poor methodology.

Share this post


Link to post
Share on other sites

Thanks, Paranoid. That was pretty much what I was thinking. In the "real world", I would bet that the major names in A-V that update daily are pretty much the same in effectiveness.

soapbox.gif

Share this post


Link to post
Share on other sites
A very common answer from companies would be

 

"We are aware of the viruses in your sample, but most of them are zoo viruses, and are not found outside the labs (in wild), as such it would serve no practical purpose in including them."

Common, perhaps. In this test however, incorrect.

To my knowledge there are no zoo viruses in my test, let alone most of them.

Share this post


Link to post
Share on other sites

I don't mean to be aggressive, trilobyte but what specific procedures do you do to ensure that each and every one of your virus samples are really in the wild?

 

We are talking about some 700+ of them after all.

Share this post


Link to post
Share on other sites

This collection is a combination of infected files that I have collected from in the wild and infected files that another person has collected from in the wild. None of these samples were obtained from antivirus companies or antivirus testing organizations, which are pretty much the only places you will even find a zoo virus.

 

Now, I don't mean to be aggressive, paranoyd but what specific procedures do you do to come to the conclusion that the vast majority of my virus samples are zoo viruses? After all, you do not even have access to this collection in order to properly make such a determination.

Share this post


Link to post
Share on other sites
This collection is a combination of infected files that I have collected from in the wild and infected files that another person has collected from in the wild.  None of these samples were obtained from antivirus companies or antivirus testing organizations, which are pretty much the only places you will even find a zoo virus.

 

Places where you get zoo viruses are hacker sites.

 

Now, I don't mean to be aggressive,

 

Really? Here you do sound aggressive.

 

paranoyd but what specific procedures do you do to come to the conclusion that the vast majority of my virus samples are zoo viruses?  After all, you do not even have access to this collection in order to properly make such a determination.

 

Because I have seen a lot of such tests, where people go to various shall we say "dark" sites, download all the stuff there, than make no effort to verify if the samples are what they say they are, unloose their scanners on them and then announce their results.

Share this post


Link to post
Share on other sites

@paranoid,

I am not going to feed the trolls.

I have made myself clear on the content of these test files and how they were obtained.

 

@All

I apologize for the petty bickering of the last several posts.

Share this post


Link to post
Share on other sites

Trilobyte

 

I think that asking for clarification about the nature and origin of your test files is far from trolling. If you wish your test to be treated seriously, questions of such nature will be asked. (And no, you didn't really answer)

 

I did not perceive any "bickering", until in your last post.

Share this post


Link to post
Share on other sites

Nice test again..

 

I see something strange about NOD32 with all the test I have viewed. This AV always seems to either be at the top of the list or at the very bottom. Anywho... Here is another AV for you to try if you like Trilobite called MKS_Vir 2004. It seem to be an AV from a polish company which now has a distrubutor for the US and Canada... It's been a hot topic at other forums... Haven't tried it myself though...

 

http://www.stormbyte.com/

Edited by NAMOR

Share this post


Link to post
Share on other sites

Trilobite

I was very intrigued by your tests, so I got McAfee VirusScan Enterprise 8 and tested it against KAV Personal Pro 4.5.0.48

 

These are my results:

 

Wild Viruses

(I disinfected several computers using KAV and accumulated 391 viruses in my KAV "Infected" folder from these disinfections. Computers belong to people with limited understanding of computer security and pretty average computer usage patterns - fair E-mail usage, browsing using Internet Explorer, word processors, some downloading esp games, one person had Kazaa. This KAV "Infected" folder was scanned with McAfee VirusScan).

 

KAV: 391 infected files

McAfee: 102 infected files

 

Then I wanted to see if McAfee would pick up anything after KAV. I just finished disinfecting one computer with KAV (which picked up 33 viruses there in few hundred infected files). So I scanned that computer with McAfee. McAfee picked up one virus only (W32/Sdbot.worm.gen.i) in one infected file.

 

Zoo Viruses

I downloaded and scanned three hacker sites (total number of viruses on each site is, of course, unknown):

 

Site A (viruses/trojans)

KAV: 25 viruses, 40 infected files

McAfee: 33 infected files

 

Site B (viruses/trojans)

KAV: 251 viruses, 354 infected files

McAfee: 38 infected files

 

Site C (exploits)

KAV: 1 virus, 1 infected file (exploit)

McAfee: 3 infected files (exploits)

Edited by Alya

Share this post


Link to post
Share on other sites

I have updated the results of my tests. The first post of this tread has been edited accordingly.

 

I have updated the number of infected test files to 758. I have added an attachment containing a chart of which test files were identified or missed as containing malicious code. You will need an archival utility such as winzip or winrar to uncompress the attachment. My testing of several thousand infected test files has been postponed due to a hard disk crash.

 

@ NAMOR,

Thanks. I’ll look into testing MKS_Vir 2004.

 

@ Alya,

I didn’t see any difference between the regular and extended Kaspersky databases. The detection is still very good with the regular database.

I disinfected several computers using KAV and accumulated 391 viruses in my KAV "Infected" folder from these disinfections.

Viruses, Trojans and malware in general are very easy to pick up this way. This is how I obtained a lot of mine. But then again, these types of files don’t exactly hide. By nature, most of them self replicate and attempt to infect other systems.

Share this post


Link to post
Share on other sites

I have been busy recently and haven’t had the time to run any new tests with other AV programs and probably won’t have time to update the scans for a while. Not to mention that several of the AV trial versions have expired. I will make note of all of the requested AV programs and if time permits, I will test them.

 

I still do not have any word on the recovery of the crashed hard disk…the bulk of the several thousand infected files were stored on it and we have been working on finding the backups of that drive and on the data recovery of that drive.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0