These e-mails are known as hoaxes and, although they are send with a good intention, they are almost always false, a kind of urban legend spread through Internet.

You can spot this kind of e-mails because they say you will have a big loss if you don’t forward them, they are not signed, they promise some presents from a company or offer some difficult to believe information.

Some examples of these kind of messages:

• The Make A Wish Foundation, has agreed to donate 7 cents evertime this message is sent on.
• If you forward it to 20 friends, you will receive the brand new Ericsson R320 WAP-phone.
• DO NOT RELY ON YOUR ANTI-VIRUS SOFTWARE. McAFEE NOR NORTON CAN DETECT IT BECAUSE IT DOES NOT BECOME A VIRUS UNTIL JUNE 1ST. IT WILL BE TO LATE THEN. WHATEVER YOU DO, DO NOT OPEN THE FILE!!!

These e-mails have all been extracted from Break the chain, a site dedicated to recopilating them, so you can know if an e-mail you receive is a hoax or not.

You should never forward this letters to your friends, because they are very annoying, clutter up your inbox and many times, they can be used to get e-mail addresses to spam them. If your friends send them to you, you should tell them not to do it and why is it bad, redirecting them to Break the chain if necessary.

But spammers know this and they wont’ allow their business to go down so easily. So what is the future of filter evasion? I have been thinking about some techniques which would probably evade most of current filters and perhaps it’s time to prepare against them before it’s too late.

The idea for this list came from a post by Vivek Jishtu where he explains how a spammer is using the Yahoo greeting cards to send his messages without being detected by filters. This service allows anyone to send a card to someone, who will be notified by e-mail and will receive a link to go to a site to view the card. In this card, the spammer can include arbitrary content so he can put his spam message there and as this will not pass through any filter it won’t be detected. So, if the user receiving the card visits the link he will see this (translated from Chinese by Google Translator):

But there are also other methods that spammers might use now or in the future (I’m not aware any of this is currently in use, but you never know).

The first technique is copied from viruses or worms which have used this for a long time. Instead of sending the content of the spam in the main body of the message, a ZIP file can be attached containing a text file with the advertisement from the spammer. If this becomes popular, Bayesian spam filters might be unable to detect it as the analyzed content can have no malicious word and can look innocuous. To be able to analyze the spam, the filter should decompress the ZIP file and search for text files inside it. This also can be avoided with another technique coming from the virus world, the use of ZIP files protected with a password, like the Bagle-J virus did. The user is told to open the ZIP file using a password contained in the main body, so the filter won’t be able to decompress the file but the user will.

Another technique, similar to the use of images instead of text, is sending their advertisements in attached files in some popular file format, like PDF or Microsoft Word files. Again, the content of the main body might be totally innocuous, asking the user to open the attached file. The filter will need to understand the file format to be able to extract the text and analyze it, which will consume resources from the computer, something sometimes not feasible in servers with lots of users.

These two techniques can be stopped by disallowing the use of attached files or, at least, restricting the formats accepted, as some servers already do to prevent the reception of viruses. We also can educate the users not to open attached files coming from unkown sources, although I doubt this will work as we can see with the expansion of some viruses which work this way.

Spammers could even do another loop and send their spam inside a PDF file compressed in a ZIP file protected by a password… OK, enough, enough,…

I don’t know if any of these or similar techniques will be used by spammers in a near future. If they do use them it will be harder to filter the spam but, at the same time, will mean we are winning a battle in this war against spam. We should better be prepared before it’s too late.

Of course, Bayesian filtering is not the only way to detect spam, although we have been concentrating on it. There are other techniques currently in use, which probably might be more effective against these new attacks and we’ll see them in another post.

We can split the filtering programs depending on where they work: on the server or on the client. The programs working on the server have some advantages, as they look at more mail messages (they see mail from all users in a system) it is easier and faster to train them. Furthermore, there is only one place to administer it, making the administrator task easier. At the same time, the users don’t need to receive the spam so they don’t spend additional bandwith and time. On the other hand, they are not so customizable by the user, which might prefer his own techniques to detect spam and false postivesand, if the user doesn’t have access to the server he will not be able to install it.

One of the most known server-side filtering software is SpamAssassin, which uses different checks to test for spam, one of them being Bayesian filtering. Each one of this tests adds or substracts a score from the mail and at the end of the runs this score will determine if the mail is spam or not. Amongst other these test include mail-header tests, text-content rules, white-lists and black-lists and collaborative databases, making this program one of the most accurate. This can also be used as client-side filtering, although the installation will not be as easy as others.

Another aproach to server-side filtering is the one used by ASSP which works with any king of mail server, as it stands as a proxy (getting the data and transmiting) in front of the real mail server and filters the data before it is delivered. It also uses Bayesian filtering and allows the settings of white-lists so you can define addresses which will be always accepted. It can also scan messages against viruses, which will drop even more malicious mail.

The last server-side software we are going to see is DSpam. This has some characteristics differentiating it from other Bayesian filters. In this case, the tokens are not only analyzed one by one, but also in pairs, which gives a better view to know if a mail is spam or not. It laso uses Bayesian Noise Reduction and other new approaches to filtering, which promise to give a hight detection rate. It includes a web-based interface to administer it, where each user can train it depending on the personal tastes.

If we don’t have access to the server or we don’t want to play with it, we can use a client-based filter, installed in our computer which will analyze the mail once it has been downloaded (or while downloading) and will flag it as spam or legitimate mail. The advantage of this kind of approach is that it can be highly integrated in our mail reader, so might be easier to use by the user.

If we use Thunderbird, it already includes a filter, as we saw in the last post about spam. This is really easy to use, as we only have to click a button to tell it if we think a message is spam and once it is trained it will move automatically all spam to a predefined folder, or can even delete it automatically (I don’t recommend it in case of false positives).

If we use Outlook instead of Thunderbird, one good option is SpamBayes. This software also uses some new approaches which are explained in the background page. One interesting characteristic of SpamBayes is that it doesn’t have only two states, spam and non-spam, but also a third one, unsure, when it doesn’t know how to classify a message. This way, we can choose what we want to do with it: keep it, delete it or use it to train the program. Although it includes a plugin for using it with Ooutlook, it can also be used with other programs as a proxy, and even in other operating systems like Linux or Mac OS X.

To finish this list, we are going to have a look at one of the first mail filters I used. It’s called POPFile and works as a proxy in front of the mail server. Our mail client will connect to POPFile and POPFile will connect to the mail server, analyzing the mail as it downloads. One of the things I like most about it is the ability to classify any kind of e-mail, not only spam. So, POPFile can distinguish between work-related mail, mail from our children,… or any other different classification we want to do. We only have to create the categories and assign some messages to each one to train it and it will classify the received e-mails. It also has a web-based interface to manage all of this.

The list of spam filters is quite long and this is only a selection of some of them. You will have to see which one fits you better and use it. Remember to always train it correctly before you do automatic actions on the mail received as you could lose some mails if you don’t do it correctly.

I’m not a mathematician, so I might make a few errors when trying to explain the theory behind the filters. Please forgive me. The article in Wikipedia explains it better than I can do it.

The main formula where Bayesian filtering stands is:

which says that the probability of an e-mail being spam given the words contained in it is equal to the probability of these words appearing in a spam message, multiplied by the probability of a message being spam divided by the probability of the words appearing in any message.

Wow, it looks quite complicated. One of the most known papers about this kind of filtering is A plan for spam from Paul Graham. We’ll see some code from it.

Well, to be able to calculate this result we need, in first place, to break the message in words, which are called tokens, from where the probabilities are taken. This partitions are really important, as they will affect the final result depending on how they are done. If we have the sentence It’s a shame we could break the words in It-s-a-shame or maybe in Its-a-shame or even It’s-a-shame and each of them might give different results when used.

Once the message is broken in tokens, we can calculate the Pr(word|spam) with the next code (this was code in Lisp originally):

(let ((g (* 2 (or (gethash word good) 0)))
      (b (or (gethash word bad) 0)))
   (unless (< (+ g b) 5)
     (max .01
          (min .99 (float (/ (min 1 (/ b nbad))
                             (+ (min 1 (/ g ngood))
                                (min 1 (/ b nbad)))))))))

When we have calculated the probabilities for all the tokens in the message, we get the most relevant ones (the ones which probability is farther from 0.5, so the nearest to 0 or 1). Paul decided to use the 15 most relevant and stores them in a list called probs, applying the next formula to it:

(let ((prod (apply #'* probs)))
  (/ prod (+ prod (apply #'* (mapcar #'(lambda (x)
                                         (- 1 x))
                                     probs)))))

If the result is bigger than 0.9 we consider that the e-mail is spam and classify it as such. So, although the theory may look hard once implemented it is far easier. Maybe the only problem with this code is it’s Lisp, which not so many people know about.

Let’s make this even easier by looking at the source code of Mozilla Thunderbird, the famous opensource mail reader, which includes a Bayesian module to classify mail. The implementation in Thunderbird is slightly different from the original, but the concept remains the same.

The algorithm is implemented in the file mozilla\mailnews\extensions\bayesian-spam-filter\src\nsBayesianFilter.cpp in the function classifyMessage. It’s implemented in C++, but we are seeing it in “pseudo-code”. It uses some different variables:

• mGoodCount: number of non-spam messages classified
• mBadCount: number of spam messages classified
• mGoodTokens: hash table with good tokens and number of times they have appeared
• mBadTokens: hash table with spam tokens and number of times they have appeared

Take care, as the same token might appear in both hash tables with different number of apparitions. For example, the word hello is equally probable in spam and non-spam messages. When the algorithm is not yet trained default values are assigned:

if (mGoodCount == 0 || mGoodTokens.count() == 0)
    message is spam
si (mBadCount == 0 || mBadTokens.count() == 0)
    message is not spam

If the algorithm has been trained then it’s applied with the next formula (adapted from Bugzilla):

for each token {
 hamcount = number of token appearances in non-spam
 spamcount = number of token appearances in spam 
 hamratio = hamcount / nGoodCount
 spamratio = spamcount / nBadCount
 
 prob = spamratio / (hamratio + spamratio)
 
 n = hamcount +  spamcount
 prob = (0.225 + n * prob) / (.45 + n)
 
 distance = abs(prob - 0.5)
 if (distance > = .1) {
  token.distance = distance
  token.prob = prob
 }
}

With this code, we have the probability for each token. This is saved in a list sorted by distance (distance is taken as the difference between probabilities) and the first 150 elements are taken. A probability distribution chi² is calculated and if the result is bigger or equal to 0.9 the message will be classified as spam.

But, we don’t need to know all of this unless we want to write one filter ourselves. There are lots of already available filters which work quite good and get rates of detection around 99%, sometimes even better than a human.

As Bayesian filtering is the most common used technique, this is what spammers try to escape more frequently. We told that Bayesian works by calculating the probability that a word is from spam or from legitimate mail, so what spammers do is modify the messages so they get more probability of being legitimate mail.

One of the ways to do this is insert random but common words in spam, so the spam words contribute less to the score and the message goes under the filter. We can see an example of a real spam:

The real content of the spam is contained at the bottom but at the beginning of the e-mail there are some lines with text which come from the novel The Master and Margarita and try to hide the fact that this is an spam.

Another way to try to evade the filters is by sending the content as an image. This technique is also used in the last example we have seen, but it’s a really common one, as we can see in this other e-mail:

Although this may look like an HTML email, in fact all the content is inside an image, with no text to be analyzed by the filters, so it gets more difficult to identify the message as spam because we have no words to compute the probability. Sometimes this technique works against the spammer, as it’s quite strange for a legitimate mail to contain only an image with a link, so some more advanced filters might detect this message as spam correctly.

One last technique is the use of unknown or made-up words to confuse the filter. As Bayesian works by looking the probability of already seen words and knowing if they are more likely to occur in legitimate mail or in spam, when an unknown word is found the filter can’t really know if it belongs to spam or not, so it can’t classify it correctly and the spam might just evade the filter. Let’s see an example:

We can see that instead of ordering the message says orderinq with a Q as the last letter, which looks quite similar to the G. Also, the word Viagra is not written with a V letter at the beginning, but with the slash and forward-slash symbols like this \ /. There are more example in these two sentences, as almost every word is modified to evade the filters.

Sometimes, it gets so difficult for spammers to be sure their junk will reach the recipient that the messages they sent have almost no sense and it is quite hard to know what they are really trying to advertise.

Can you guess what they are trying to say?

If we have a Bayesian filter which checks our e-mail it is a good idea to keep it updated and trained. It’s quite easy and shouldn’t consume a lot of our time, unless we receive incredible amounts of e-mail. To do this we should check from time to time the folder where spam is moved to check if there has been any false positive (that is, a legitimate mail message which has been classified as spam). If there is any, we must tell the filter that message is not spam so it changes the probabilities of the words included in it. Checking this folder from time to time is a good idea anyways, so we don’t lose any important e-mail which might have been miscategorised. It’s also important when we receive spam which is not filtered as such, not only delete it, but tell the filter that message is spam so we can keep it trained.

There are other methods to classify spam which are not based in Bayesian filters and we will see them in next posts.

The basis for this method is a previous training of the algorithm, where we must feed it with spam messages and legitimate mail telling which is which. With this data, the algorithm breaks the messages in words and assign a probability to each word for being in a spam message and another for being in a legitimate mail.

When a new message is received, it’s broken in words like the training messages and the saved probabilities of each word are analyzed with a formula called Naive Bayes, which returns a final probability for the mail being spam or not.

Most of the known mail classifier use, at least, this method, usually combined with others, but we can see this is a really powerful way of classifying.

Another approach to classification is the one used by Spamassassin which has a series of rules that assign some points when it applies to the mail. As more points are assigned the mail has more probability of being spam, and it is classified as such when it surpasses a threshold.

Spamassassin also uses the Bayesian filter but it’s not the only way to check for spam, as it usually has distinguishable characteristics which may make it different enough from legitimate mail to be easily classifiable.

But spammers are adapting to the measures, modifying the mails they send so they are not detected as spam by the filters and it’s necessary to tweak these filters and find new ways to throw spam to trash.

For an e-mail to be spam it must be sent without the consent of the recipient, that is, an e-mail with a commercial advertisement is not spam if you have asked for it. The legislation of each country is more specific as to what is spam and what is not.

The products which get more advertising in spam vary with time, but it is quite usual to receive spam about drugs like viagra or valium, about how to get fake college diplomas, how to get a mortgage or illegal software.

The problem of spam is economic. Sending spam is really cheap, so even if only a really small percentage of the receivers buy the product it’s still profitable. So, you must never buy products advertised this way, so spammers get the message that people don’t like to receive these kind of messages and won’t buy their products.

In the same way, the most expensive part of the spam is not payed by the spammer. He only has to find somewhere from where to send the spam and, once it has been send, he doesn’t have to pay anything more for it. But the message has to travel through other networks, has to be stored somewhere and has to be, finally, read or deleted. This has a cost in network bandwidth, in disk space occupied in, more importantly, in time spent by the final recipient having to classify and delete the e-mail.

For many people, the quantity of spam received is bigger than the quantity of legitimate mail, so they need some way to classify it automatically, as it almost gets impossible to do it by hand in a short time.

This is a big annoyance for the owners of the servers, as they will be probably black-listed and will not be able to send legitimate mail, causing a disruption of the service for legitimate users.

In a first instance, spammers used mail servers which were incorrectly configured and allowed anyone to send e-mail through it (technically, it is known as relaying mail). It’s very cheap to use this technique, as to send massive amounts from the server the spammer only needed to send it once. Fortunately, nowadays most administrators configure their servers correctly and only allow authorized users to send e-mail, so spammers needed to find another way to send their junk. If you administer an e-mail server and you don’t have secured it against relaying you should check how to disable it.

The most common used technique nowadays is relay mail through botnets. Botnets are groups of compromised computers controlled remotely by the attacker and spammers use them to send the e-mails to the world. Unfortunately, there are a lot of botnets in Internet and it’s quite cheap to find someone who controls one and sends the e-mails for us.

For this reason, it’s important to protect our computer so it doesn’t get used to spam all the world. Also, some ISPs implement filters so e-mail from their users can only be send through their server (technically, they close the outbound TCP port 25). This way, they can’t send spam from that account but this is also an annoyance for more advanced users, which sometimes need to use other e-mail servers as they might have different accounts in other places.

One of the most common ones is by browsing the web. Spammers send their computers to spider the web, that is navigate and follow links, retrieving the text in the pages and analyzing it looking for e-mail addresses. They usually only look for addresses which match the pattern user@server.tld, so if we write our address in some webpage, be it our personal website, in the comments section of another site or anywhere else, it’s easy some of this robots find it and we begin receiving undesired mail.

Another method is analyzing chain letters. These are usually full of working e-mail addresses, as they are send to all the addressbook and when forwarded these list is not deleted, filling it with more and more addresses as it is being forwarded.

Some time ago, Usenet News were a really popular service where people could read and send messages. These messages contain a header with the e-mail address of the sender, so spammers collected messages and analyzed them to get addresses. Nowadays, Usenet is not so used as before and the ones who use it are more knowledgeable, so I suspect these method is falling into oblivion, although it might be used by some spammers.

There have always been dishonest companies and some of them sell their databases to spammers, so depending on where we get registered we might be giving away our e-mail address to someone unknown. Depending on the country, there might be severe laws to prevent this, but it’s not always the case.

Another method is getting the address used when registering a domain. When you register a domain (like www.example.com) you have to provide three addresses (might be the same) which are lately made public so people can contact you about the domain. As it’s really easy to get them, spammers only have to get a list of domains and scan them for addresses.

Finally, one of the most used ones is just guessing or, we might say, bruteforcing. That is, try different addresses hoping they work. As it’s really cheap to send and e-mail they don’t lose almost anything for trying a really big number of addresses, even if most of them don’t work. You can find some examples of this in some of the spam received, when looking at the destination you find a lot of e-mail addresses very similar to yours.

There are other techniques not so widely in use, so these are the most important ones. From some of them we can protect ourselves, but there’s nothing we can do to protect from the other, so we have to simply trust other people to do it for us.

Spam: the classic and oldest type of undesired mail. In fact, any kind of e-mail sent massively and without the consent of the receiver is considered to be spam, but to distinguish them we usually call it spam when it’s some kind of advertising trying to sell legal or illegal products.

Phishing: this is a technique used to collect sensitive information from users, such as passwords or bank account details. E-mail of this type tries to disguise as legitimate mail but points to fake webservers where you are asked to enter the information.

Viruses: in old times viruses spread through floppy disks but with the rising of the use of e-mail creators have changed the distribution method and worms (as these kinds of viruses are known) are nowadays one of the most common type of virus.

Chain letters: these usually come from people we know, so it’s easier to trust them, but almost always contain false information. We can distinguish them because they try to expand some kind of rumour, such as non-deletable viruses, methods of obtaining free presents from some companies or threats of something bad happening to us if they are not forwarded to a specified quantity of people.

Trojans: similar to viruses but usually not send massively, only to an intended recipient as a method of gaining control of his computer or information stored in it.

There are some more kinds of undesired mail but these are the most important ones. We are going to have a look at each of them and discover how they work and how to avoid them.