Most users got conscious and disabled the use of macros, so the virus couldn’t get executed and many mail providers blocked e-mails with attached Office documents.

This is not the case anymore, as macro viruses are very rare now, but a recent Word vulnerability has made DOC files dangerous again. This time the problem is not with macros inside the document, but a vulnerability that allows to execute malicious code when the document is open.

There is no patch yet for this vulnerability, as Microsoft won’t release it until June, so you should be extremely careful with documents you receive, specially if they are unexpected.

For now, this doesn’t seem too widespread, as only one attack has been detected against a company, and it was a very targeted one, directed specially to them, but it wouldn’t be strange to find it in the wild in some days.

It would be better if the server was able to avoid these messages being sent. Although some mail servers analyze the content of the message before delivering there are some other techniques which have been proposed to work against spam. Some of them are even standards, but haven’t usually been widely deployed. Let’s have a look at some advantages and disadvantages of them.

DNS blacklist. This is one of the most old and known techniques which tries to avoid the spam being delivered by checking if the computer sending it is a “probable” spammer. It looks up its IP address in an online service (for example, MAPS or ORBS) and if the address is listed the mail will be rejected (or at least, flagged as suspicious).

There are various online services, each one listing IP addresses depending on different factors. For example, some of them list dynamic and dial-up IP addresses, which usually should send the mails through their server. Other only list IP addresses which have sent spam in the past.

Some controversy has built around these services because sometimes IP addresses have been added in error and this makes a “legal” server unable to send mail to whoever uses this filtering system. Furthermore, spammers are trying to take down some of this sites, usually by DDoS them, so users can’t check them.

You can find more technical information about DNS Blacklists at the Wikipedia.

Greylisting. If a server uses greylisting when a new mail is received it will check the IP address of the sender, his mail address and the recipient mail address against a local database. If this combination has already been seen before the mail is delivered. If it hasn’t been seen before the message is rejected with a “Try later” message.

This works because most spammers will never retry to send the message but legitimate mail server will try again in a short time, so when they retry the mail will be accepted. This can be a really powerful technique while spammers don’t adapt to it (if many servers use this they will finally retry to send the mails).

The disadvantage of this technique is that it delays all messages coming from unknown sources, be it spam or not, which might not be suitable for everyone, especially online business. Even more, if the sending server is not configured correctly it might not retry to send the mail.

This can be combined with whitelisting (addresses which are always accepted) and blacklisting (addresses which are always rejected).

More information about greylisting at the Wikipedia and links to implementations for different servers.

SPF and DomainKeys. Many spammers use fake mail addresses as the remitent of the message and send the mail from hacked machines or open relays. To solve this one can build a list of IP addresses allowed to send mail from one domain. This is the way SPF works, adding a DNS registry which tells the authorized IP addresses to send mail for that domain. This solution is really simple but it requires the “big players” to use it in their mail servers, which hasn’t happened yet.

DomainKeys is another similar technique proposed by Yahoo which uses cryptography to authenticate the message and check it comes from the mail server indicated. One of the possible disadvantages of this method is that it requires more resources to check the cryptographic signatures, although this shouldn’t be a problem in servers with a low number of users. Also, as SPF it has to be implemented by most mail servers to be really useful.

More information about SPF in wikipedia or at their homepage. Also about DomainKeys in Wikipedia or at Yahoo.

But spammers know this and they wont’ allow their business to go down so easily. So what is the future of filter evasion? I have been thinking about some techniques which would probably evade most of current filters and perhaps it’s time to prepare against them before it’s too late.

The idea for this list came from a post by Vivek Jishtu where he explains how a spammer is using the Yahoo greeting cards to send his messages without being detected by filters. This service allows anyone to send a card to someone, who will be notified by e-mail and will receive a link to go to a site to view the card. In this card, the spammer can include arbitrary content so he can put his spam message there and as this will not pass through any filter it won’t be detected. So, if the user receiving the card visits the link he will see this (translated from Chinese by Google Translator):

But there are also other methods that spammers might use now or in the future (I’m not aware any of this is currently in use, but you never know).

The first technique is copied from viruses or worms which have used this for a long time. Instead of sending the content of the spam in the main body of the message, a ZIP file can be attached containing a text file with the advertisement from the spammer. If this becomes popular, Bayesian spam filters might be unable to detect it as the analyzed content can have no malicious word and can look innocuous. To be able to analyze the spam, the filter should decompress the ZIP file and search for text files inside it. This also can be avoided with another technique coming from the virus world, the use of ZIP files protected with a password, like the Bagle-J virus did. The user is told to open the ZIP file using a password contained in the main body, so the filter won’t be able to decompress the file but the user will.

Another technique, similar to the use of images instead of text, is sending their advertisements in attached files in some popular file format, like PDF or Microsoft Word files. Again, the content of the main body might be totally innocuous, asking the user to open the attached file. The filter will need to understand the file format to be able to extract the text and analyze it, which will consume resources from the computer, something sometimes not feasible in servers with lots of users.

These two techniques can be stopped by disallowing the use of attached files or, at least, restricting the formats accepted, as some servers already do to prevent the reception of viruses. We also can educate the users not to open attached files coming from unkown sources, although I doubt this will work as we can see with the expansion of some viruses which work this way.

Spammers could even do another loop and send their spam inside a PDF file compressed in a ZIP file protected by a password… OK, enough, enough,…

I don’t know if any of these or similar techniques will be used by spammers in a near future. If they do use them it will be harder to filter the spam but, at the same time, will mean we are winning a battle in this war against spam. We should better be prepared before it’s too late.

Of course, Bayesian filtering is not the only way to detect spam, although we have been concentrating on it. There are other techniques currently in use, which probably might be more effective against these new attacks and we’ll see them in another post.

We can split the filtering programs depending on where they work: on the server or on the client. The programs working on the server have some advantages, as they look at more mail messages (they see mail from all users in a system) it is easier and faster to train them. Furthermore, there is only one place to administer it, making the administrator task easier. At the same time, the users don’t need to receive the spam so they don’t spend additional bandwith and time. On the other hand, they are not so customizable by the user, which might prefer his own techniques to detect spam and false postivesand, if the user doesn’t have access to the server he will not be able to install it.

One of the most known server-side filtering software is SpamAssassin, which uses different checks to test for spam, one of them being Bayesian filtering. Each one of this tests adds or substracts a score from the mail and at the end of the runs this score will determine if the mail is spam or not. Amongst other these test include mail-header tests, text-content rules, white-lists and black-lists and collaborative databases, making this program one of the most accurate. This can also be used as client-side filtering, although the installation will not be as easy as others.

Another aproach to server-side filtering is the one used by ASSP which works with any king of mail server, as it stands as a proxy (getting the data and transmiting) in front of the real mail server and filters the data before it is delivered. It also uses Bayesian filtering and allows the settings of white-lists so you can define addresses which will be always accepted. It can also scan messages against viruses, which will drop even more malicious mail.

The last server-side software we are going to see is DSpam. This has some characteristics differentiating it from other Bayesian filters. In this case, the tokens are not only analyzed one by one, but also in pairs, which gives a better view to know if a mail is spam or not. It laso uses Bayesian Noise Reduction and other new approaches to filtering, which promise to give a hight detection rate. It includes a web-based interface to administer it, where each user can train it depending on the personal tastes.

If we don’t have access to the server or we don’t want to play with it, we can use a client-based filter, installed in our computer which will analyze the mail once it has been downloaded (or while downloading) and will flag it as spam or legitimate mail. The advantage of this kind of approach is that it can be highly integrated in our mail reader, so might be easier to use by the user.

If we use Thunderbird, it already includes a filter, as we saw in the last post about spam. This is really easy to use, as we only have to click a button to tell it if we think a message is spam and once it is trained it will move automatically all spam to a predefined folder, or can even delete it automatically (I don’t recommend it in case of false positives).

If we use Outlook instead of Thunderbird, one good option is SpamBayes. This software also uses some new approaches which are explained in the background page. One interesting characteristic of SpamBayes is that it doesn’t have only two states, spam and non-spam, but also a third one, unsure, when it doesn’t know how to classify a message. This way, we can choose what we want to do with it: keep it, delete it or use it to train the program. Although it includes a plugin for using it with Ooutlook, it can also be used with other programs as a proxy, and even in other operating systems like Linux or Mac OS X.

To finish this list, we are going to have a look at one of the first mail filters I used. It’s called POPFile and works as a proxy in front of the mail server. Our mail client will connect to POPFile and POPFile will connect to the mail server, analyzing the mail as it downloads. One of the things I like most about it is the ability to classify any kind of e-mail, not only spam. So, POPFile can distinguish between work-related mail, mail from our children,… or any other different classification we want to do. We only have to create the categories and assign some messages to each one to train it and it will classify the received e-mails. It also has a web-based interface to manage all of this.

The list of spam filters is quite long and this is only a selection of some of them. You will have to see which one fits you better and use it. Remember to always train it correctly before you do automatic actions on the mail received as you could lose some mails if you don’t do it correctly.

I’m not a mathematician, so I might make a few errors when trying to explain the theory behind the filters. Please forgive me. The article in Wikipedia explains it better than I can do it.

The main formula where Bayesian filtering stands is:

which says that the probability of an e-mail being spam given the words contained in it is equal to the probability of these words appearing in a spam message, multiplied by the probability of a message being spam divided by the probability of the words appearing in any message.

Wow, it looks quite complicated. One of the most known papers about this kind of filtering is A plan for spam from Paul Graham. We’ll see some code from it.

Well, to be able to calculate this result we need, in first place, to break the message in words, which are called tokens, from where the probabilities are taken. This partitions are really important, as they will affect the final result depending on how they are done. If we have the sentence It’s a shame we could break the words in It-s-a-shame or maybe in Its-a-shame or even It’s-a-shame and each of them might give different results when used.

Once the message is broken in tokens, we can calculate the Pr(word|spam) with the next code (this was code in Lisp originally):

(let ((g (* 2 (or (gethash word good) 0)))
      (b (or (gethash word bad) 0)))
   (unless (< (+ g b) 5)
     (max .01
          (min .99 (float (/ (min 1 (/ b nbad))
                             (+ (min 1 (/ g ngood))
                                (min 1 (/ b nbad)))))))))

When we have calculated the probabilities for all the tokens in the message, we get the most relevant ones (the ones which probability is farther from 0.5, so the nearest to 0 or 1). Paul decided to use the 15 most relevant and stores them in a list called probs, applying the next formula to it:

(let ((prod (apply #'* probs)))
  (/ prod (+ prod (apply #'* (mapcar #'(lambda (x)
                                         (- 1 x))
                                     probs)))))

If the result is bigger than 0.9 we consider that the e-mail is spam and classify it as such. So, although the theory may look hard once implemented it is far easier. Maybe the only problem with this code is it’s Lisp, which not so many people know about.

Let’s make this even easier by looking at the source code of Mozilla Thunderbird, the famous opensource mail reader, which includes a Bayesian module to classify mail. The implementation in Thunderbird is slightly different from the original, but the concept remains the same.

The algorithm is implemented in the file mozilla\mailnews\extensions\bayesian-spam-filter\src\nsBayesianFilter.cpp in the function classifyMessage. It’s implemented in C++, but we are seeing it in “pseudo-code”. It uses some different variables:

• mGoodCount: number of non-spam messages classified
• mBadCount: number of spam messages classified
• mGoodTokens: hash table with good tokens and number of times they have appeared
• mBadTokens: hash table with spam tokens and number of times they have appeared

Take care, as the same token might appear in both hash tables with different number of apparitions. For example, the word hello is equally probable in spam and non-spam messages. When the algorithm is not yet trained default values are assigned:

if (mGoodCount == 0 || mGoodTokens.count() == 0)
    message is spam
si (mBadCount == 0 || mBadTokens.count() == 0)
    message is not spam

If the algorithm has been trained then it’s applied with the next formula (adapted from Bugzilla):

for each token {
 hamcount = number of token appearances in non-spam
 spamcount = number of token appearances in spam 
 hamratio = hamcount / nGoodCount
 spamratio = spamcount / nBadCount
 
 prob = spamratio / (hamratio + spamratio)
 
 n = hamcount +  spamcount
 prob = (0.225 + n * prob) / (.45 + n)
 
 distance = abs(prob - 0.5)
 if (distance > = .1) {
  token.distance = distance
  token.prob = prob
 }
}

With this code, we have the probability for each token. This is saved in a list sorted by distance (distance is taken as the difference between probabilities) and the first 150 elements are taken. A probability distribution chi² is calculated and if the result is bigger or equal to 0.9 the message will be classified as spam.

But, we don’t need to know all of this unless we want to write one filter ourselves. There are lots of already available filters which work quite good and get rates of detection around 99%, sometimes even better than a human.

As Bayesian filtering is the most common used technique, this is what spammers try to escape more frequently. We told that Bayesian works by calculating the probability that a word is from spam or from legitimate mail, so what spammers do is modify the messages so they get more probability of being legitimate mail.

One of the ways to do this is insert random but common words in spam, so the spam words contribute less to the score and the message goes under the filter. We can see an example of a real spam:

The real content of the spam is contained at the bottom but at the beginning of the e-mail there are some lines with text which come from the novel The Master and Margarita and try to hide the fact that this is an spam.

Another way to try to evade the filters is by sending the content as an image. This technique is also used in the last example we have seen, but it’s a really common one, as we can see in this other e-mail:

Although this may look like an HTML email, in fact all the content is inside an image, with no text to be analyzed by the filters, so it gets more difficult to identify the message as spam because we have no words to compute the probability. Sometimes this technique works against the spammer, as it’s quite strange for a legitimate mail to contain only an image with a link, so some more advanced filters might detect this message as spam correctly.

One last technique is the use of unknown or made-up words to confuse the filter. As Bayesian works by looking the probability of already seen words and knowing if they are more likely to occur in legitimate mail or in spam, when an unknown word is found the filter can’t really know if it belongs to spam or not, so it can’t classify it correctly and the spam might just evade the filter. Let’s see an example:

We can see that instead of ordering the message says orderinq with a Q as the last letter, which looks quite similar to the G. Also, the word Viagra is not written with a V letter at the beginning, but with the slash and forward-slash symbols like this \ /. There are more example in these two sentences, as almost every word is modified to evade the filters.

Sometimes, it gets so difficult for spammers to be sure their junk will reach the recipient that the messages they sent have almost no sense and it is quite hard to know what they are really trying to advertise.

Can you guess what they are trying to say?

If we have a Bayesian filter which checks our e-mail it is a good idea to keep it updated and trained. It’s quite easy and shouldn’t consume a lot of our time, unless we receive incredible amounts of e-mail. To do this we should check from time to time the folder where spam is moved to check if there has been any false positive (that is, a legitimate mail message which has been classified as spam). If there is any, we must tell the filter that message is not spam so it changes the probabilities of the words included in it. Checking this folder from time to time is a good idea anyways, so we don’t lose any important e-mail which might have been miscategorised. It’s also important when we receive spam which is not filtered as such, not only delete it, but tell the filter that message is spam so we can keep it trained.

There are other methods to classify spam which are not based in Bayesian filters and we will see them in next posts.

The basis for this method is a previous training of the algorithm, where we must feed it with spam messages and legitimate mail telling which is which. With this data, the algorithm breaks the messages in words and assign a probability to each word for being in a spam message and another for being in a legitimate mail.

When a new message is received, it’s broken in words like the training messages and the saved probabilities of each word are analyzed with a formula called Naive Bayes, which returns a final probability for the mail being spam or not.

Most of the known mail classifier use, at least, this method, usually combined with others, but we can see this is a really powerful way of classifying.

Another approach to classification is the one used by Spamassassin which has a series of rules that assign some points when it applies to the mail. As more points are assigned the mail has more probability of being spam, and it is classified as such when it surpasses a threshold.

Spamassassin also uses the Bayesian filter but it’s not the only way to check for spam, as it usually has distinguishable characteristics which may make it different enough from legitimate mail to be easily classifiable.

But spammers are adapting to the measures, modifying the mails they send so they are not detected as spam by the filters and it’s necessary to tweak these filters and find new ways to throw spam to trash.

For an e-mail to be spam it must be sent without the consent of the recipient, that is, an e-mail with a commercial advertisement is not spam if you have asked for it. The legislation of each country is more specific as to what is spam and what is not.

The products which get more advertising in spam vary with time, but it is quite usual to receive spam about drugs like viagra or valium, about how to get fake college diplomas, how to get a mortgage or illegal software.

The problem of spam is economic. Sending spam is really cheap, so even if only a really small percentage of the receivers buy the product it’s still profitable. So, you must never buy products advertised this way, so spammers get the message that people don’t like to receive these kind of messages and won’t buy their products.

In the same way, the most expensive part of the spam is not payed by the spammer. He only has to find somewhere from where to send the spam and, once it has been send, he doesn’t have to pay anything more for it. But the message has to travel through other networks, has to be stored somewhere and has to be, finally, read or deleted. This has a cost in network bandwidth, in disk space occupied in, more importantly, in time spent by the final recipient having to classify and delete the e-mail.

For many people, the quantity of spam received is bigger than the quantity of legitimate mail, so they need some way to classify it automatically, as it almost gets impossible to do it by hand in a short time.

• Why people send this kind of mail
• Differents kinds of undesirable mail
• Methods to recollect e-mail dirs
• How they send this mail
• How to avoid receiving this kind of mail

If you know spanish I can only recommend going straight to read it. If you don’t, I’m going to write a series of posts about e-mail with some of the information in the post.

More info | Correo no deseado. Causas, tipologías y medidas de prevención.