Postfix/amavisd perfomance – server crash
- February 4th, 2007
- Posted in Computer Related Notes
- Write comment
Yesterday, I had interesting mail problem that I believe is still occurring today only on a much smaller scale. One on my servers crashed a couple times. I believe this is because the server was overloaded trying to process too many mail messages at once.
I ended up having to add the following line to main.cf on both of my mail servers:
smtp-amavis_destination_concurrency_limit = 2
I found this in the amavisd documentation for a postfix installation. I never went into the performance tweaks documented. I had about 200 messages queued up on my secondary server. Whenever, I tried to process them, my primary server got pounded. I decided to process the couple messages that looked legitimate, and delete all the others. So far, with the new settings mentioned above and deleting the pending unimportant messages in the secondary queue, things have been working fine. I guess I should RTFM a bit more on postfix and amavisd. Especially, since I am running these domains on older hardware.
The mail server ran out of memory again. I am not exactly sure what is going with this thing. I know one domain is getting a lot of mail. It looks like somebody is spoofing the domain name. Well, all the mail for domain is forwarded to another address at a differnet ISP. I am theorizing that they are stopping me from forwarding them due to volume. As a result, I get a bounce that I then try to forward, creating a loop situation. In addition to the changes I mentioned before, last night I made the following changes to prevent accepting and sending mail to unknown domains.
master.cf:
127.0.0.1:10025 inet n – n – – smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=reject_unknown_sender_domain
-o smtpd_recipient_restrictions=permit_mynetworks,reject,reject_unknown_sender_domain
Also, I decided not to forward the mail for the domain, and just keep it locally stored on the server until the bounces subside. Several are still coming in an hour. I have had it like this since last night. Hopefully, they will slow down in the next couple days. The server has been fine so far. We will see how it is through the day.
The mail is still coming in pretty quickly. I believe I have figured out what the problem was with the number of messages. I think the domain was spoofed and used to send out spam. Those that bounced (many of them) reporting that the spam was stopped, were sent back to my mail server which forwarded them to the mail server at the remote location. Well, this mail server eventutally blocked mine out due to too mail messages being sent in a certain period of time. This happens with this hosting site once in a while, but not in these numbers. Therefore, when the message was rejected I got another bounce that I tried to forward too. This caused a kind of loop that I have been able to prevent by accepting the mail locally at my mail server. I just have to let this ride out in the few days. It should calm down by the end of the week. My mail logs did have the “Connection refused” messages in the log, so I am fairly certain this was the problem.
It has been 6 days since last post anything on this. I am still getting quite a few bounced messages sent to this address. Not sure what to do about it. I haven’t had much time to look in to much further, since I figured out a way to keep the mail server running.
Had the same issue with a different domain the last couple days. The problem is the size of the body_checks file. Postfix needs to check for too much in the body of each message and it just killing the machine when we are hit with a lot of mail. Needless to say, I have removed all the entries from the file. I am not sure how to proceed. Hardware is not an option.
I think I finally looked at more closely at how to limit the number of processes postfix will run at one time. The default is 100. Using atop, I was to find that the smaller machine was running out of memory at 27 processes. I won’t know if this worked until I notice when we get out next blast. One happened on 8/25/2007 again. I changed it 10 on the smaller machine and 20 on the one with more memory using the following method:
Modify /etc/postfix/master.cf to limit the number of postfix processes. This
should prevent the server from running out of memory when we are hit with a
blast of spam.
vi /etc/postfix/master.cf
…
# JGZ 8/30/2007 – to limit the number of postfix processes
# To prevent running out of memory when we get hit with a blast of spam.
#smtp inet n – n – – smtpd
smtp inet n – n – 10 smtpd
…