spread the knowledge
MIMEDefang
“Übersicht“
Vorbereitungen
Zuerst die Installation Vorbereiten, damit MimeDefang für ClamAV die korrekten Berechtigungen haben wird:
vi /etc/make.conf
Folgendes hinzufügen
#-----------------------------------------------# # ClamAV # #-----------------------------------------------# CLAMAVUSER=mailnull CLAMAV_CLAMD_SOCKET=/var/spool/MIMEDefang/clamd.sock
Installation
cd /usr/ports/mail/mimedefang make install clean
Installation mit Clamav Enabled:
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Options for mimedefang 2.72_2 x x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x x x [*] CLAMAV Enable Clamav x x x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu x < OK > <Cancel> x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
SpamAssasin Config:
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Options for p5-Mail-SpamAssassin 3.3.2_6 x x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x x x [*] AS_ROOT Run spamd as root (recommended) x x x x [*] SPAMC Build spamd/spamc (not for amavisd) x x x x [ ] SACOMPILE sa-compile x x x x [*] DKIM DKIM/DomainKeys Identified Mail x x x x [*] SSL Build with SSL support for spamd/spamc x x x x [*] GNUPG Install GnuPG (for sa-update) x x x x [ ] MYSQL Add MySQL support x x x x [ ] PGSQL Add PostreSQL support x x x x [*] RAZOR Add Vipul's Razor support x x x x [ ] SPF_QUERY Add SPF query support x x x x [ ] RELAY_COUNTRY Relay country support x x x x [ ] DCC Add DCC support (see LICENSE) x x x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu x < OK > <Cancel> x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
sa-update dann auch gleich laufen lassen…
===> Installing rc.d startup script(s) config: no rules were found! Do you need to run 'sa-update'? ******************************************************* * _ _ _ _______ ______ __ _ _____ __ _ ______ * * | | | |_____| |_____/ | | | | | | ____ * * |__|__| | | | _ | _| __|__ | _| |_____| * * * ******************************************************* * You must install rules before starting spamd! * ******************************************************* Do you wish to run sa-update to fetch new rules [N]? Y
ClamAV nehm ich auch die default Settings:
/usr/ports/security/clamav
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Options for clamav 0.97.4 x x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x x x [*] ARC Enable arch archives support x x x x [*] ARJ Enable arj archives support x x x x [*] LHA Enable lha archives support x x x x [*] UNZOO Enable zoo archives support x x x x [*] UNRAR Enable rar archives support x x x x [*] LLVM Enable JIT Bytecode compiler (bundled LLVM) x x x x [ ] LLVM_PORTS Enable JIT Bytecode compiler (from Ports) x x x x [*] TESTS Run compile-time tests (req. python) x x x x [ ] MILTER Compile the milter interface x x x x [ ] LDAP libmilter was built with LDAP x x x x [ ] ICONV Enable ICONV support x x x x [ ] STDERR Print logs to stderr instead of stdout x x x x [ ] EXPERIMENTAL Build experimental code x x x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj x tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqu x < OK > <Cancel> x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
Die Installation schliesst nun mit folgendem Output ab:
To finish installation please follow these steps: 1. Edit /usr/local/etc/mimedefang/mimedefang-filter 2. Reconfigure sendmail to use MIMEDefang, for example add this lines to your sendmail.mc: MAIL_FILTER(`mimedefang', `S=local:/var/spool/MIMEDefang/mimedefang.sock, F=T, T=C:15m;S:4m;R:4m;E:10m')dnl define(`confINPUT_MAIL_FILTERS', `mimedefang')dnl rebuild sendmail.cf and install it. Also, you can install some anti-virus software to check your mail, for example: security/clamav Open source AV package with automatic virus definition updates security/f-prot F-Prot Antivirus for BSD Workstations security/vscan Evaluation version of a DOS/Windows/Linux file virus scanner ===> Compressing manual pages for mimedefang-2.72_2 ===> Registering installation for mimedefang-2.72_2 ===> SECURITY REPORT: This port has installed the following files which may act as network servers and may therefore pose a remote security risk to the system. /usr/local/bin/mimedefang-multiplexor This port has installed the following startup scripts which may cause these network services to be started at boot time. /usr/local/etc/rc.d/mimedefang If there are vulnerabilities in these programs there may be a security risk to the system. FreeBSD makes no guarantee about the security of ports included in the Ports Collection. Please type 'make deinstall' to deinstall the port if this is a concern. For more information, and contact details about the security status of this software, see the following webpage: http://www.mimedefang.org/
Startup Script
vi /usr/local/etc/rc.d/mimedefang
folgendes anpassen:
# "yes" turns on the multiplexor sender checking function
MX_SENDER_CHECK=yes
# "yes" turns on the multiplexor recipient checking function
MX_RECIPIENT_CHECK=yes
# Set to yes if you want to use an embedded Perl interpreter MX_EMBED_PERL=yes # Maximum # of requests a process handles # MX_REQUESTS=200 # machen wir mal von 100 auf 50 runter... vielleicht doch nicht # das bedeuted glaub ich einfach, dass nach dieser anzahl requests # der prozess / slave gekillt wird. MX_REQUESTS=100 # Maximum number of processes to run (mail received while this many # processes are running is rejected with a temporary failure, so be # wary of how many emails you receive at a time). This applies only # if you DO use the multiplexor. The default value of 2 is probably # too low; we suggest 10 instead #MX_MAXIMUM=6 MX_MAXIMUM=8 # Limit slave processes' resident-set size to this many kilobytes. Default # is unlimited. #MX_MAX_RSS=10000 ## default ist unlimitiert, also lassen wir es jetzt unlimited - wir haben ja einen neuen server! ## MX_MAX_RSS=20000 # Limit total size of slave processes' memory space to this many kilobytes. # Default is unlimited. #MX_MAX_AS=30000 ## default ist unlimitiert, also lassen wir es jetzt unlimited - wir haben ja einen neuen server! ## MX_MAX_AS=50000
# Set to yes if you don't want MIMEDefang to see invalid recipients.
# Only works with Sendmail 8.14.0 and later.
MD_SKIP_BAD_RCPTS=yes
Nun Startup Script aktivieren:
vi /etc/rc.conf
hinzufügen:
#-----------------------------------------------# # MimeDefang # #-----------------------------------------------# mimedefang_enable=YES
Fehler in Version 2.63
In der neueren Version (2.63) hat es irgend einen Fehler drin. und wenn man den Script starten will, macht er nichts. d.h. Mimedefang wird nicht gestartet und es gibt auch keinen Output beim ausführen. Wenn man aber folgende Zeilen (am Ende des Scripts) auskommentiert, dann gehts:
das Auskommentieren ist WICHTIG! sonst könnte es sein, dass es Startup Fehler gibt. Vielleicht zuerst ohne auskommentieren versuchen, gibts Fehler, auskommentieren und wieder testen.
#if type run_rc_command > /dev/null 2>&1
#then
# # NetBSD/FreeBSD compatible startup script
# run_rc_command "$1"
# exit $RETVAL
#fi
# See how we were called.
case "$1" in
start)
start_it
;;
Konfiguration
Mimedefang
vi /usr/local/etc/mimedefang/mimedefang-filter
$AdminAddress = 'postmaster@myhost.com'; $AdminName = "MyHost Spam Fighters"; $ClamdSock = "/var/spool/MIMEDefang/clamd.sock"; $DaemonAddress = 'mimedefang@myhost.com';
Ansonsten alles so lassen wie es ist. Nun folgende Routine im Config File hinzufügen. Hier ist wichtig, dass der Filter überhalb allen anderen steht. Das heisst, gleich überhalb filter_begin(). So werden alle anderen Checks übersprungen. Will man nur den SpamAssassin Check überspringen, reicht ein Eintrag gleich überhalb filter_end
"if ($Features{"SpamAssassin"}) {"
return if($RelayAddr eq "127.0.0.1" || $RelayAddr =~ /^130.225/);
Das überspringt SpamAssassin checks, sowie alle anderen Tests in filter_end() unterhalb dieser Line.
sub filter_relay {
my($hostip, $hostname, $helo) = @_;
# Do not scan locally-originating mail
if ($hostip eq "127.0.0.1" or
$hostip =~ /^81.94.97./) {
return ('ACCEPT_AND_NO_MORE_FILTERING', 'ok');
}
return('CONTINUE', 'ok');
}
This causes mail originating from 127.0.0.1 or the subnet 81.94.97/24 to be accepted without any further filtering. Adjust as appropriate for your network.
und noch dass die returns an automailer nicht gefiltert werden:
sub filter_recipient {
my ($recipient, $sender, $ip, $hostname, $first, $helo,
$rcpt_mailer, $rcpt_host, $rcpt_addr) = @_;
if ($recipient =~ /^<?automailer@myhost.com>?$/i) {
return ('ACCEPT_AND_NO_MORE_FILTERING', 'ok');
}
return ('CONTINUE', "ok");
}
Fertig sieht das also so aus:
# added by teslina sub filter_relay { my($hostip, $hostname, $helo) = @_; # Do not scan locally-originating mail if ($hostip eq "127.0.0.1" or $hostip eq "217.150.250.113" or $hostip =~ /^80.74.159./) { return ('ACCEPT_AND_NO_MORE_FILTERING', 'ok'); } return('CONTINUE', 'ok'); } sub filter_recipient { my ($recipient, $sender, $ip, $hostname, $first, $helo, $rcpt_mailer, $rcpt_host, $rcpt_addr) = @_; if ($recipient =~ /^<?automailer@myhost.com>?$/i) { return ('ACCEPT_AND_NO_MORE_FILTERING', 'ok'); } return ('CONTINUE', "ok"); } #EOF teslina #*********************************************************************** # %PROCEDURE: filter_begin # %ARGUMENTS: # $entity -- the parsed MIME::Entity # %RETURNS: # Nothing # %DESCRIPTION: # Called just before e-mail parts are processed #*********************************************************************** sub filter_begin { my($entity) = @_;
nun noch unter:
if ($hits >= $req) {
action_change_header("X-Spam-Score", "$hits ($score) $names");
folgende Zeile hinzufügen, damit das Subject geändert wird, sofern spam:
action_change_header('Subject', "[ *****SPAM***** ] $Subject");
das sieht dann fertig so aus:
if ($hits >= $req) {
action_change_header("X-Spam-Score", "$hits ($score) $names");
# added by teslina
action_change_header('Subject', "[ *****SPAM***** ] $Subject");
# EOF teslina
md_graphdefang_log('spam', $hits, $RelayAddr);
Optimierungen
Spam Mails sind in der Regel nicht > also 50kb. Aus performance gruenden, sollte der gescannte text daher nicht unbedingt grösser sein (default in mimedefang ist 100kb)
# Spam checks if SpamAssassin is installed
if ($Features{"SpamAssassin"}) {
if (-s "./INPUTMSG" < 100*1024) {
# Only scan messages smaller than 100kB. Larger messages
# are extremely unlikely to be spam, and SpamAssassin is
ändern in
# Spam checks if SpamAssassin is installed
if ($Features{"SpamAssassin"}) {
if (-s "./INPUTMSG" < 50*1024) {
# Only scan messages smaller than 50kB. Larger messages
# are extremely unlikely to be spam, and SpamAssassin is
Sendmail
Nun Sendmail noch einmal konfigurieren
cd /etc/mail/my_prefs vi corky.mc
Folgendes rauslöschen (falls vorhanden)
INPUT_MAIL_FILTER(`spamassassin',`S=local:/var/run/spamass-milter.sock, F=, T=C:15m;S:4m;R:4m;E:10m')dnl define(`confINPUT_MAIL_FILTERS', `spamassassin')dnl
Folgendes hinzufügen
INPUT_MAIL_FILTER(`mimedefang', `S=local:/var/spool/MIMEDefang/mimedefang.sock, T=C:15m;S:4m;R:4m;E:10m')dnl
unbedingt darauf achten, dass MAIL_FILTER **VOR** MAILER(`cyrusv2′) etc. steht!
Weitere Infos zu den Mail Filters hier:
- http://tutorials.mediaslut.ch/index.php?title=Installation_GreyListing_(RelayDelay)#Sendmail
- http://www.sendmail.org/m4/tweaking_config.html
Damit sub filter_relay auch funktioniert, muss man hier MX_SENDER_CHECK auf YES stellen
vi /usr/local/etc/rc.d/mimedefang.sh-dist
# "yes" turns on the multiplexor sender checking function MX_SENDER_CHECK=yes
MimeDefang Tuning
Verwendet man einen embedded Perl interpreter, geht alles schneller. Einfach im Config so einstellen. mehr muss man nicht machen:
# Set to yes if you want to use an embedded Perl interpreter MX_EMBED_PERL=yes
# Maximum # of requests a process handles # alle 500 requests wird ein neuer slave prozess gestartet # die zahl sollte recht hoch sein. denn wenn ein perl prozess gestartet werden # muss braucht das mehr ressourcen als wenn 1 prozess einfach mehr handelt. MX_REQUESTS=500
Tuning Tipps #1
So, summarizing some of the previous messages... If you looking for all reasonable ways to maximize system performance: Mount / as noatime Mount /etc/mail as ramdisk, noatime [restore backup at boot time] Mount /tmp as ramdisk, noatime Mount /var as noatime Mount /var/spool/MIMEDefang as ramdisk, noatime. Mount /var/spool/mqueue/qf as noatime [on disk 1] Mount /var/spool/mqueue/df as noatime [on disk 2] Mount /var/spool/mqueue/xf as ramdisk, noatime [only if you split the qdir's] Mount /var/mail normally Remember to make periodic backups of /etc/mail so you have something to restore after a crash or reboot. SCSI disks are notably faster than IDE disks. Striping is notably faster than not striping. Higher RPM disks are faster than lower RPM disks (but 2x the RPM does not mean 2x the actual performance). Hardware RAID is faster than software RAID. IDE RAID is a low-cost option, and there are 10K RPM IDE disks now. IDE is fine until you're trying to push >>100K messages/day or really big surges. Between MIMEDefang and all of these ramdisks, you'll need a lot of memory. At least 2GB of RAM. If you start paging and swapping, performance will drop considerably (keep in mind that disk is something like 1000x slower than RAM!). Set MX_REQUESTS to something like 500 and use embedded Perl if it works under your O/S version. As seen in the recommended filter, don't run SpamAssassin on messages larger than 100K. Use greylisting. If your e-mail correspondents don't use Novell Groupwise, Communigate Pro or Symantec Raptor firewalls you can use the MUCH more efficient "tempfail after RCPT TO" form of greylisting. Otherwise take note that the message is greylisted and then tempfail the message at the top of filter_begin. IMPORTANT NOTE: action_tempfail just makes a note to tempfail the message, it doesn't happen right then so you _should_ do "return if message_rejected();" at the top of filter, filter_multipart and filter_end. Drop executable attachments, don't bother to virus-scan them ... just drop them. Use daemonized virus scanners. I've seen clamd scan a message in about 10% of the CPU time it took clamscan to scan a message. Don't bother to scan a message you know will be rejected. Validate all input (HELO, MAIL FROM, RCPT TO) as much as possible. Read all the various threads over the last couple months that cover what constitutes "too much validation". These tests may let you reject 10-15% of incoming mail as spam without ever receiving the message body (a _big_ win). Run a local caching nameserver on your filter. That will get rid of a lot of network traffic to the DNSBL's and generally improve performance a bit. Use dccifd instead of dccproc, you'll save on an exec() for every scanned message. Consider running a DCC server locally, the public servers are always overloaded. Consider dropping messages that exceed a predetermined SpamAssassin score (10-20 is a good range, it depends on your environment). If you don't deliver the message, that's just one less CPU/memory/disk hit. CPU is important, but not as important as you might think. A single P4 2.0GHz can handle more than 100K messages/day if the rest of the system is balanced. Don't go for that quad-processor 3.2GHz Xeon system with 4MB on-die cache with 8 SCSI disks in a RAID 0+1 array, 32GB RAM and dual gig-Ethernet NICs unless you're trying to figure out how to push a million messages/day through one box. Frankly it'll be cheaper to have a dozen 1u P4 2.8GHz 40GB IDE systems than it would be to buy that one monster box ... AND you'll have better reliability with a dozen expendable boxes. Greylisting becomes slightly more interesting over a dozen boxes, you better know (or learn) to use a real database system at that point. Should we make "how to make a system running MIMEDefang go faster?" a FAQ entry? Chris Myers Networks By Design
Tuning Tipps #2
1) Run a caching DNS server on the local box, and load your RBL's into it whenever possible. Every time you don't have to hit the network, you're saving time. 2) Run a DCC server locally (can't do this with Pyzor or Razor). The public DCC servers are way overloaded, which means timeouts and retries. 2a) Use dccifd instead of dccproc, you'll save on at least an exec(). 3) If this is a dedicated server, make /tmp a shmfs ... virtually nothing besides SA will be using it anyway. Make sure any Cron scripts that do reporting don't eat up /tmp when they run. 4) Make sure you've got enough RAM on the box for the number of processes you are running. 5) Use greylisting ( http://projects.puremagic.com/greylisting/), you don't have to implement the full spec (especially the minimum time before accepting a new sender/recipient pair) to get most of the benefit. If you get a lot of spam, this may reduce your load by 50% (or more!). 6) Use HELO validation (some spammers pretend to be "you" in the HELO), you can kick them early in the SMTP conversation. 7) Drop messages with a high SA score (>5, >7, >10, >20 ... depends on what you call "high" in your environment). If you don't deliver the message, you're saving load. 8) Drop executable attachments without virus-scanning them. Virus scanning is killing you, do everything possible to avoid it. 9) Make sure your SpamAssassin database isn't getting HUGE, the keyword database has a nasty tendency to learn bits of undecoded base64. I've seen my database get up to 200MB with older versions of SpamAssassin. The Mail::SpamAssassin::Conf manpage describes the configuration keywords you need to control the database size. 10) Put your SpamAssassin database in shmfs. Reading/writing a 10MB, or 100MB, file for every message is a big hit to performance. Obviously you want to back it up to disk occasionally and restore the backup whenever the system boots. Remember that disk is 1000x slower than RAM. Let me repeat myself on #8: Virus scanning is killing you, do everything possible to avoid it. Your box, if you use the suggestions above (esp. #5), should be able to handle a minimum of 250,000 messages/day and probably 2x-3x that ... without virus scanning. If you're dropping executable attachments, you barely need virus scanning anyway (.zip and other archive files come to mind). Perlcache and speedycgi-type technologies aren't going to help you. They obtain speed by making the perl process persistent (no fork()/exec()/Perl startup every time you run the perl script), which MIMEDefang already does. See MX_REQUESTS in the MIMEDefang configuration file, which can safely be in the hundreds. You're still looking for solutions to speed up per-message processing by SECONDS, not milliseconds, so don't spend a lot of effort on Net::DNS yet. Keep an eye on RAM, as you implement these speedups you'll find that your system is handling more and more sendmail/MIMEDefang processes simultaneously during peak times. If you run out of RAM and start paging/swapping you're going to be hating life. Chris Myers Networks By Design
ClamAV
ClamAV wurde bereits mit MimeDefang installiert. Nun kommt die Konfiguration. Das File befindet sich hier:
vi /usr/local/etc/clamd.conf #LocalSocket /var/run/clamav/clamd.sock LocalSocket /var/spool/MIMEDefang/clamd.sock #User clamav User mailnull
Das Config File für die Datenbank Updates (freshclam) ist hier zu finden:
/usr/local/etc/freshclam.conf
Das kann man nun bei Bedarf editieren. Ich lasse es für’s erste bei den Default Einstellungen.
Nun ClamAV im Startup Script aktivieren:
vi /etc/rc.conf
#-----------------------------------------------# # ClamAV # #-----------------------------------------------# clamav_freshclam_enable="YES" clamav_clamd_enable="YES" clamav_clamd_socket="/var/spool/MIMEDefang/clamd.sock"
ClamAV Daemon und auto update starten:
/usr/local/etc/rc.d/clamav-clamd start /usr/local/etc/rc.d/clamav-freshclam start
Nun die Virus DB aktualiseren:
# freshclam
Der Output sieht dann in etwa so aus:
ClamAV update process started at Wed Apr 11 14:57:02 2012
main.cvd is up to date (version: 54, sigs: 1044387, f-level: 60, builder: sven)
WARNING: getfile: daily-14651.cdiff not found on remote server (IP: 65.19.179.67)
WARNING: getpatch: Can’t download daily-14651.cdiff from database.clamav.net
WARNING: getfile: daily-14651.cdiff not found on remote server (IP: 194.186.47.19)
WARNING: getpatch: Can’t download daily-14651.cdiff from database.clamav.net
WARNING: getfile: daily-14651.cdiff not found on remote server (IP: 200.236.31.1)
WARNING: getpatch: Can’t download daily-14651.cdiff from database.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Downloading daily.cvd [100%]
daily.cvd updated (version: 14770, sigs: 146256, f-level: 63, builder: guitar)
Downloading bytecode.cvd [100%]
bytecode.cvd updated (version: 168, sigs: 38, f-level: 63, builder: edwin)
Database updated (1190681 signatures) from database.clamav.net (IP: 208.72.56.53)
Clamd successfully notified about the update.
So, ClamAV ist nun ready.
Grundsätzlich ist es jetzt so, dass MimeDefang die ClamAV Checks durchführt. Man braucht in der Sendmail Config also nur den MimeDefang Milter rein zu tun. Allerdings kann es passieren, dass durch die Mail Modifikation von MimeDefang dazu kommt, dass gewisse Viren nicht erkannt werden. Wenn man also super paranoid ist, sollte man im Sendmail mehrere Milter in dieser Reihenfolge verwenden:
- ClamAV-Milter
- MimeDefang Milter (mit ClamAV Virus Check aktiviert)
- ClamAV Milter nochmals laufen lassen
Da ich nicht ganz so krass paranoid bin, reicht mir der MimeDefang Milter – für den Moment wenigstens – völlig aus 😉
Weitere Infos zu MimeDefang und ClamAV gibt es hier.
SpamAssassin
Installation
SpamAssassin wurde bereits mit MimeDefang installiert. Falls nicht, kann man das so noch nachholen (Config Optionen siehe oben)
cd /usr/ports/mail/p5-Mail-SpamAssassin make install clean
Konfiguration
Da wir SpamAssassin über MimeDefang aufrufen, muss SpamAssassin über das MimeDefang Config File konfiguriert werden. Das Default SpamAssassin Config File wird ignoriert.
vi /usr/local/etc/mimedefang/sa-mimedefang.cf
Hier nun die gewünschten Anpassungen machen. Zum Beispiel:
###########################################################################
# Add your own customised scores for some tests below. The default scores are
# read from the installed "spamassassin.cf" file, but you can override them
# here. To see the list of tests and their default scores, go to
# http://spamassassin.taint.org/tests.html .
#Enable Bayes
use_bayes 1
use_bayes_rules 1
bayes_path /etc/mail/spamassassin/bayes/bayes
bayes_file_mode 0777
bayes_auto_learn 1
bayes_auto_learn_threshold_nonspam 0.0
bayes_auto_learn_threshold_spam 15.0
use_razor2 1
use_dcc 1
use_pyzor 1
# SWINOG Blacklist
# http://antispam.imp.ch
#
# Swinog URIRBL - http://antispam.imp.ch/05-uribl.php
urirhsbl URIBL_SC_SWINOG uribl.swinog.ch. A
body URIBL_SC_SWINOG eval:check_uridnsbl('URIBL_SC_SWINOG')
describe URIBL_SC_SWINOG URI's listed in uribl.swinog.ch.
tflags URIBL_SC_SWINOG net
score URIBL_SC_SWINOG 0 0.900 0 1.500
# Swinog DNSRBL - http://antispam.imp.ch/06-dnsbl.php
header RCVD_IN_SWINOG_SPAM eval:check_rbl('swinog', 'dnsrbl.swinog.ch.', '127.0.0.3')
describe RCVD_IN_SWINOG_SPAM Listed in dnsrbl.swinog.ch.
tflags RCVD_IN_SWINOG_SPAM net
score RCVD_IN_SWINOG_SPAM 2.500
# ImproWare IP-Spamliste - http://antispam.imp.ch/04-spamlist.html
# Sendmail Config noch anpassen, damit diese Spams automatisch blockiert werden
header RCVD_IN_IMP_SPAMLIST eval:check_rbl('spamrbl', 'spamrbl.imp.ch.', '127.0.0.5')
describe RCVD_IN_IMP_SPAMLIST Listed in spamrbl.imp.ch
tflags RCVD_IN_IMP_SPAMLIST net
score RCVD_IN_IMP_SPAMLIST 0.100
# ImproWare IP-Wurmliste - http://antispam.imp.ch/03-wormlist.html?lng=0
# Sendmail Config noch anpassen, damit diese Spams automatisch blockiert werden
header RCVD_IN_IMP_WORMLIST eval:check_rbl('wormrbl', 'wormrbl.imp.ch.', '127.0.0.5')
describe RCVD_IN_IMP_WORMLIST Listed in wormrbl.imp.ch
tflags RCVD_IN_IMP_WORMLIST net
score RCVD_IN_IMP_WORMLIST 0.100Das sind noch zusätzliche Config möglichkeiten. In der Regel sollte man rund 1000 mails haben, damit bayes richtig läuft. zum test kann man das aber sicher auch mal runterschrauben.
required_hits 10.0 ok_locales en rewrite_subject 0 skip_rbl_checks 1 score HABEAS_SWE 2.0 use_razor2 1 use_dcc 0 use_pyzor 1 use_bayes 1 auto_learn 1 bayes_path /etc/mail/spamassassin/bayes/bayes bayes_auto_expire 1 bayes_auto_learn_threshold_nonspam 0.5 bayes_auto_learn_threshold_spam 5.5 bayes_expiry_max_db_size 100000 bayes_file_mode 0644 bayes_ignore_header X-Spam-Status: bayes_ignore_header X-Spam-Score: bayes_journal_max_size 10240 bayes_journal_max_size 5120000 bayes_learn_to_journal 1 bayes_min_ham_num 100 bayes_min_spam_num 100 pyzor_timeout 4 razor_timeout 4
Optimierung
Diese Flags sollten auf 0 gesetzt werden. Siehe Link: [1]
use_razor2 0 use_dcc 0 use_pyzor 0
Sendmail Config
Damit die SWINOG/IMP Blacklist Daten korrekt abgewiesen werden, noch folgende Einträge machen:
FEATURE(`dnsbl',`bl.spamcop.net', `"Spam blocked see: http://spamcop.net/bl.shtml?"$&{client_addr}', `t')dnl
FEATURE(`dnsbl',`dsn.rfc-ignorant.org',`550 You do not accept bounces - http://www.rfc-ignorant.org/')dnl
FEATURE(`enhdnsbl', `wormrbl.imp.ch', `451 tempfail - see http://antispam.imp.ch/03-wormlist.html
$&{client_addr} is infected with a virus or worm !', `t', `127.0.0.5.')
FEATURE(`enhdnsbl', `spamrbl.imp.ch', `451 tempfail - see http://antispam.imp.ch/04-spamlist.html
$&{client_addr} has sent spam and may be a possible spamsource!', `t', `127.0.0.5.')
AutoLearn
Erstmal machen wir im IMAP spezielle Verzeichnisse, wo alle Mails reinkopiert werden, die Spam sind, jedoch nicht als solches erkannt wurden. Ich arbeite in diesem Beispiel mit den Verzeichnissen xJunk/Spam und xJunk/NoSpam.
Danach die spezielle cyrus learn (sa-learn-cyrus) software downloaden [2]:
cd /usr/local/etc/mail/spamassassin/ fetch http://www.pollux.franken.de/uploads/media/sa-learn-cyrus-0.3.5.tar.gz tar xzfv sa-learn-cyrus-0.3.5.tar.gz rm sa-learn-cyrus-0.3.5.tar.gz chown -R root:wheel sa-learn-cyrus-0.3.5/ cd sa-learn-cyrus-0.3.5/
Nun Config File anpassen:
vi sa-learn-cyrus.conf
lock_file = /tmp/sa-learn-cyrus.lock simulate = no spam_folder = 'xJunk.Spam' ham_folder = 'xJunk.NoSpam' # path with system-wide SA preferences site_config_path= /usr/local/etc/mail/spamassassin # SA configuration file prefs_file = /usr/local/etc/mimedefang/sa-mimedefang.cf # path to sa-learn learn_cmd = /usr/local/bin/sa-learn initial_letter = no purge_cmd = /usr/local/cyrus/bin/ipurge
# SA user and group user = root group = wheel
Danach noch kurz das Script anpassen:
vi sa-learn-cyrus
alle config file paths
/etc/spamassassin
ändern in
/usr/local/etc/mail/spamassassin/sa-learn-cyrus-0.3.5
:%s//etc/spamassassin//usr/local/etc/mail/spamassassin/sa-learn-cyrus-0.3.5
Dann noch
'imap:base_dir' => '/var/spool/imap',
ändern in
'sa:learn_cmd' => '/usr/local/bin/sa-learn',
Und
# Base directory of IMAP spool (below that mailboxes are located) base_dir = /var/spool/cyrus/mail
ändern in
# Base directory of IMAP spool (below that mailboxes are located) base_dir = /var/spool/imap
mkdir /etc/mail/spamassassin mkdir /etc/mail/spamassassin/bayes chmod 777 /etc/mail/spamassassin/bayes cp /root/.spamassassin/* /etc/mail/spamassassin/bayes/ rm -rf /root/.spamassassin/ ln -s /etc/mail/spamassassin/bayes /root/.spamassassin
Falls wir auf einem anderen Server bereits eine grössere Bayes DB haben, einfach die Files auf den neuen Server kopieren:
cd /etc/mail/spamassassin/bayes scp user@oldserver.com:/etc/mail/spamassassin/bayes/* . chown mailnull:wheel *
Dann im /usr/local/etc/mimedefang/sa-mimedefang.cf:
bayes_path /etc/mail/spamassassin/bayes/bayes bayes_file_mode 0777
Jetzt noch einen Cronjob machen, der die mails auto-learnt
crontab -e
# spam auto-learning aktivieren 59 03 * * * /usr/local/etc/mail/spamassassin/sa-learn-cyrus-0.2.3sa-learn-cyrus
Spamfolder leeren
Die alten Spams können automatisch aus dem SPAM Folder gelöscht werden.
Perl Script
Einfach folgendes Perl Script als Cron aufrufen. Mit -d 7 werden alle Mails, die älter als 7 Tage sind, vom Folder gelöscht.
vi /global/cronjobs/global/cyrus_spam.sh
#!/usr/bin/perl
my @SPAM_D=`cd /var/spool/imap/user/; find . -name xJunk`;
foreach my $DIR (@SPAM_D){
$DIR =~ s///./g;
$DIR =~ s/../user./g;
$DIR =~ s/n//g;
$cmd[0] = "su cyrus -c '/usr/local/cyrus/bin/ipurge -f -d 7 $DIR'";
print "Executing @cmd ...n";
$result = system(@cmd);
print "nPurging Spam mails using command '@cmd' failed: $?" unless $result == 0 ;
}
crontab -e
# spam auto-learning aktivieren immer morgens um 4 - da er recht viel ressourcesn braucht (high load)
0 04 * * * /usr/local/etc/mail/spamassassin/sa-learn-cyrus-0.3.5/sa-learn-cyrus > /dev/null 2>&1
0 05 * * * /global/cronjobs/global/cyrus_spam.sh > /dev/null 2>&1
Cyrus interne Lösung
Dazu sollte dieser Eintrag in den Events von cyrus.conf reichen – dies funktioniert jedoch NICHT!. Folgende Anleitung ist nur zur Info. Als Wildcart hab ich * sowie % gesehen – auch verschiedene Arten des Aufrufes der Mailbox (also user/*/xJunk oder user.*.xJunk oder user.%.xJunk) hab alles durchgespielt, hat nichts funktioniert. Der Event wurde einfach nicht aufgerufen. Vielleicht liegts ja daran, dass es nicht Punkt 01h geschieht, sondern vielleicht 1h30 oder so… keine ahnung. Hab Cyrus auch neu gestartet nach der Änderung.
vi /usr/local/etc/cyrus.conf
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
delprune cmd="cyr_expire -E 3" at=0400
# this is only necessary if caching TLS sessions
tlsprune cmd="tls_prune" at=0400
# purge spam older than 14 days at 1AM
purgespam cmd="ipurge -f -d 7 user/*/xJunk" at=0100
}
Starten / Testen
Alles starten
/usr/local/etc/rc.d/mimedefang start
Config updaten
cd /etc/mail make rcsendmail restart
Testen
- sende mail per mailprogramm raus an gmail oder so.
- sende mail von gmail zurück
kommt das alles an, läuft sendmail schon mal 😉
Jetzt noch testen, ob SPAM erkannt wird. Sende ein mail mit dem Content (Subject &/oder Body)
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
Das ist der [GTUBE] Test-String. Die Mail sollte nun als Spam markiert sein und ggf. direkt in den xJunk Ordner verschoben worden sein.
Clamav kann man mit folgendem EICAR Teststring testen (einfach in den BODY einer Mail kopieren)
X5O!P%@AP[4PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
Per Default Einstellung werden Virus Mails ohne weitere Infos gelöscht:
Apr 11 16:53:07 corky mimedefang.pl[13705]: q3BEr6Wd013920: MDLOG,q3BEr6Wd013920,virus,Eicar-Test-Signature,209.85.214.174,<xxx@gmail.com>,<xxx@server.com>,testvirus
Apr 11 16:53:07 corky mimedefang.pl[13705]: q3BEr6Wd013920: Discarding because of virus Eicar-Test-Signature
Apr 11 16:53:07 corky sm-mta[13920]: q3BEr6Wd013920: Milter: data, discard
Apr 11 16:53:07 corky sm-mta[13920]: q3BEr6Wd013920: discardedFehlerbehebungen
Out of Memory
Problem:
Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: Out of memory during request for Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: 16392 Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: bytes, total sbrk() is Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: 12288000 Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: bytes! Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: Callback called exit. Nov 11 10:22:01 corky sendmail[70019]: kAB9M15p070019: from=root, size=254,, nrcpts=1, msgid=<200611110922.kAB9M15p070019@corky.shoe.org>, relay=root@localhost Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: END failed--call queue aborted at /usr/local/bin/mimedefang.pl line 27. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: Callback called exit at /usr/local/bin/mimedefang.pl line 27. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: END failed--call queue aborted at /usr/local/bin/mimedefang.pl line 43. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: Callback called exit at /usr/local/bin/mimedefang.pl line 43. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: END failed--call queue aborted at /usr/local/bin/mimedefang.pl line 50. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: Callback called exit at /usr/local/bin/mimedefang.pl line 50. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: END failed--call queue aborted at /usr/local/bin/mimedefang.pl line 73. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: Callback called exit at /usr/local/bin/mimedefang.pl line 73. Nov 11 10:22:01 corky mimedefang-multiplexor[69898]: Slave 0 stderr: END failed--call queue aborted at /usr/local/bin/mimedefang.pl line 1.
MIMEDefang hat zuwenig Speicher zugewiesen bekommen. Lösung: Im Startupscript die Limits erhöhen -> entweder auskommentieren (UNLIMITED) oder einfach grösser machen 10000kb sind 10 MB Speicher etc.
vi /usr/local/etc/rc.d/mimedefang.sh-dist
# Limit slave processes' resident-set size to this many kilobytes. Default # is unlimited. #MX_MAX_RSS=10000 MX_MAX_RSS=20000
# Limit total size of slave processes' memory space to this many kilobytes. # Default is unlimited. #MX_MAX_AS=30000 MX_MAX_AS=50000
Auskommentieren / Unlimited machen ist nicht sehr gescheit – da sonst der Mailserver alles für sich beansprucht:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 50715 mysql 20 0 334M 128M kserel 0 247.1H 35.50% 35.50% mysqld 70385 mailnull -8 0 33716K 32844K piperd 1 1:19 17.14% 17.14% perl 70370 mailnull 113 0 31108K 30244K RUN 0 0:48 16.60% 16.60% perl 70418 mailnull 114 0 30692K 29772K RUN 1 0:42 13.82% 13.82% perl 70839 mailnull -8 0 29040K 28172K piperd 0 0:21 11.82% 11.82% perl 71124 mailnull -8 0 26508K 25664K piperd 0 0:07 10.52% 9.52% perl 70353 mailnull -8 0 34828K 33972K piperd 0 1:25 7.86% 7.86% perl 71257 www 100 0 18452K 12732K select 0 0:01 4.44% 2.64% httpd 71318 root 124 0 13672K 9488K RUN 1 0:01 11.84% 2.15% php
ran out of memory — possible DoS attack due to complex MIME?
Das ist eigentlich das Selbe wie oben beschrieben. Er bekommt zu wenig Memory zugewiesen.
May 12 20:10:19 corky mimedefang-multiplexor[63195]: Slave 0 stderr: Out of memory! May 12 20:10:19 corky mimedefang-multiplexor[63195]: Slave 0 ran out of memory -- possible DoS attack due to complex MIME? May 12 20:10:19 corky mimedefang[63211]: Error from multiplexor: ERR No response from slave May 12 20:10:19 corky mimedefang-multiplexor[63195]: Reap: slave 0 (pid 63212) exited normally with status 1 (SLAVE DIED UNEXPECTEDLY)
Da wir jetzt einen neuen Server haben, der das alles handeln können sollte, kommentieren wir MX_MAX_AS und MX_MAX_RSS aus, damit er selber entscheiden kann. Folgendes hab ich noch auf dem Netz gefunden. Eventuell sollte man MX_IDLE und MX_QUEUE_TIMOUT / MX_QUEUE_SIZE auch noch machen. Bis heute (14.05.09) läuft es aber stabil ohne diese Zusätze
## commented out: #MX_MAX_RSS=10000 #MX_MAX_AS=30000 # Maximum number of processes to run (default installed was 10) MX_MAXIMUM=7 # Number of seconds a process should be idle before checking for # minimum number and killed # MX_IDLE=300 MX_IDLE=60 ## un-commented the following 2 directives: # Multiplexor queue size -- default is 0 (no queueing) MX_QUEUE_SIZE=10 # Multiplexor queue timeout -- default is 30 seconds MX_QUEUE_TIMEOUT=30
local socket name /var/spool/MIMEDefang/mimedefang.sock unsafe
in
tail -f /var/log/maillog
gibts folgende Fehler:
Jun 13 14:08:00 corky mimedefang-multiplexor[57046]: Starting slave 1 (pid 57066) (2 running): Bringing slaves up to minSlaves (2) Jun 13 14:08:19 corky sm-mta[57085]: n5DC8Jmm057085: Milter (mimedefang): local socket name /var/spool/MIMEDefang/mimedefang.sock unsafe Jun 13 14:08:19 corky sm-mta[57085]: n5DC8Jmm057085: Milter (mimedefang): to error state
Hier ist das Problem, dass Sendmail nicht neu gestartet wurde, nachdem der Milter gerestartet wurde. Daher einfach mehrfach Sendmail stoppen, den milter stoppen – kurz warten (einfach ein paar Sekunden) und schauen, dass sendmail und mimedefang wirklich aus der Prozessliste verschwunden sind.
/usr/local/etc/rc.d/mimedefang.sh stop rcsendmail stop
-> Prozessliste checken, danach neu starten
/usr/local/etc/rc.d/mimedefang.sh start rcsendmail start
Danach sollte der Fehler verschwunden sein.
Connection refused by /var/spool/MIMEDefang/mimedefang.sock
Jun 16 12:28:33 corky sm-mta[24836]: n5GASXpO024836: Milter (mimedefang): error connecting to filter: Connection refused by /var/spool/MIMEDefang/mimedefang.sock Jun 16 12:28:33 corky sm-mta[24836]: n5GASXpO024836: Milter (mimedefang): to error state
Das hatte ich mal nachdem ich eine neue Version von MimeDefang installiert hab. Lösen konnte ich es, indem ich im startup Script den Part „NetBSD/FreeBSD compatible startup script“ auskommentiert hab.
#if type run_rc_command > /dev/null 2>&1 #then # # NetBSD/FreeBSD compatible startup script # run_rc_command "$1" # exit $RETVAL #fi
Falls es danach immernoch nicht geht, auch mal checken, wie es aussieht, wenn ich im startup script eine andere location fuer das mimedefang.sock eingeben würde, obs danng ehen würde? oder mimedefang.sock löschen? darf man das überhaupt?
Bayes DB wird ignoriert
Per default müssen mind. 200 Spam und 200 Ham Messages in der Bayes db gespeichert sein, damit Bayes angewendet wird. Um herauszufinden, wie viele spams mit sa-learn bereits gespeichert wurden, kann man diesen Befehl benutzen:
sa-learn --dump magic
0.000 0 3 0 non-token data: bayes db version 0.000 0 2554 0 non-token data: nspam 0.000 0 1 0 non-token data: nham 0.000 0 122400 0 non-token data: ntokens 0.000 0 1170407332 0 non-token data: oldest atime 0.000 0 1244902676 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 0 0 non-token data: last expiry atime 0.000 0 0 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count
In diesem Beispiel sind 2554 Spams und 1 Ham in der DB. In diesem Zustand wird BAYES also NICHT angewendet (da zu wenig HAMs). Dies kann man umgehen, indem man in der Config den Default wert überschreibt.
vi /usr/local/etc/mimedefang/sa-mimedefang.cf
bayes_min_ham_num 1 bayes_min_spam_num 300
(nach dem change, mimedefang UND sendmail neu starten – und sa-update nochmals ausführen)
sa-update -v
wenn Bayes noch nicht angewendet wird, gibt es auch noch kein bayes_journal:
ll /etc/mail/spamassassin/bayes/ -rw-rw-rw- 1 mailnull wheel 327680 Jun 13 16:54 bayes_seen -rw-rw-rw- 1 root wheel 2641920 Jun 13 16:54 bayes_toks
Sobald Bayes aktiv ist, gibt es noch eine zusätzliche datai: journal:
-rw-rw-rw- 1 mailnull wheel 3048 Jun 13 17:07 bayes_journal -rw-rw-rw- 1 mailnull wheel 327680 Jun 13 16:54 bayes_seen -rw-rw-rw- 1 root wheel 2641920 Jun 13 16:54 bayes_toks
bayes_journal kann nicht geschrieben werden
Das Journal File wird gelöscht, sobald die Daten mit der DB gesynct sind. Das File sollte jedoch nach einem Weilchen wieder auftauchen. Tut es das nicht, könnte dies ein Permission problem sein. MimeDefang muss die Permission haben, das bayes_journal zu schreiben. Daher sollte der Ordner mailnull gehören:
drwxrwxrwx 2 mailnull wheel 512 Feb 18 17:11 bayes/ root@corky(/etc/mail/spamassassin)>
tut es das nicht, mach
chown mailnull:wheel /etc/mail/spamassassin/bayes
mimedefang-multiplexor[1511]: s6R2b1pY040689: Slave 3 stderr: main::rebuild_entity() called too early to check prototype at /usr/local/bin/mimedefang.pl line 805.
Nach einem MimeDefang Upgrade auf Version 2.74 kam dieser Fehler. Lösen kann man das, indem man das Mimedefang Startupscript wie folgt ändert:
# Start mimedefang printf "%-60s" "Starting $prog: " rm -f $SOCKET > /dev/null 2>&1 $PROGDIR/$prog -P $PID -R $LOOPBACK_RESERVED_CONNECTIONS -m $MX_SOCKET `[ -n "$SPOOLDIR" ] && echo "-z $SPOOLDIR"`
bei MX_SOCKET -y hinzufügen:
# Start mimedefang
printf "%-60s" "Starting $prog: "
rm -f $SOCKET > /dev/null 2>&1
$PROGDIR/$prog -P $PID -R $LOOPBACK_RESERVED_CONNECTIONS
-m $MX_SOCKET -y
`[ -n "$SPOOLDIR" ] && echo "-z $SPOOLDIR"`
Quelle: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=971523
