Problem with SPAM mail that I can't block

Question asked by Gabriele Maoret - SERSIS - 7/30/2019 at 4:42 AM

Answered

Since a few days I continue to find emails similar to this one in the queue of our SMARTERMAIL server and I can't understand where they come from.
Is someone else experiencing the same pain and how can they be solved?

From: "System Administrator"
To: mathiasbaskdws@banksyariahku.com
Subject: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:
Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Delivered:
=?UTF-8?Q?Applicalo_e_i_funghi_spariscono:_spray_per_le_unghie_efficace_con_ingredienti_naturali?=
Date: Tue, 30 Jul 2019 13:39:30 +0200
Message-ID: <1d391e1152b24c6a9fbb684c108e58e2@banksyariahku.com>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary=b3d68e6529fb413cbb8b97d1d99df022
X-SmarterMail-MessageType: Bounce
X-Exim-Id: 6ba1f596360f40caaf07704d007fbc6f

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

25 Replies

Reply to Thread

Sébastien Riccio Replied

7/30/2019 at 9:49 AM

This looks like an infinite loop of internal system messages. Is the sender or recipient mailbox full ?

Sébastien Riccio System & Network Admin https://swisscenter.com

Kyle Kerst Replied

7/30/2019 at 2:45 PM

Employee Post Marked As Answer

Sebastien, you are correct this appears to be a looping issue we're aware of and are troubleshooting in another environment currently. I will reach out directly to the OP with a custom build that should correct these issues.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Gabriele Maoret - SERSIS Replied

8/5/2019 at 11:29 PM

Updated to latest release 7153, this issue is still here...

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

Kyle Kerst Replied

8/6/2019 at 7:43 AM

Employee Post

Correct gmaoret, we are still investigating these issues. If you have experienced this in your environment I recommend submitting a ticket so we can collect further diagnostic information. Thanks in advance!

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Tony Scholz Replied

8/7/2019 at 7:06 AM

Employee Post

Hello,

The developers are still working on resolving this issue.

Thank you

Tony Scholz System/Network Administrator SmarterTools Inc. www.smartertools.com

FrankyBoy Replied

8/7/2019 at 11:38 AM

Same problem for Us

Gabriele Maoret - SERSIS Replied

8/8/2019 at 8:35 AM

OK, glad to see that this issue is not only mine...

I hope that a solution is near because there are tons of mail in my spool...

FrankyBoy Replied

8/8/2019 at 12:30 PM

Any updates about this bug? Causing a lot of problems...

Tim DeMeza Replied

8/8/2019 at 4:04 PM

We are also experiencing this issue.

Sébastien Riccio Replied

8/12/2019 at 7:56 AM

We have an open ticket for this issue since 10 July 2019. Unfortunately no fix so far.

Sébastien Riccio System & Network Admin https://swisscenter.com

Kyle Kerst Replied

8/12/2019 at 2:29 PM

Employee Post

Hello everyone, sorry for the delay in response. We've been waiting for feedback from a couple of environments to see if these fixes have worked, and we are seeing positive results in a few of them. As such, I've included this custom build below along with minor upgrade instructions. Please give this a shot, and let us know if you continue to have problems. Be sure to double-check that you have a valid from address configured for all messages under Settings>System Messages as well.

Download: http://www.smartertools.com/downloads/SmarterMail/CustomBuilds/100.0.7158.27437/SmarterMail_7158.exe

Instructions: https://portal.smartertools.com/kb/a2855/minor-upgrade-of-smartermail.aspx

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Sébastien Riccio Replied

8/17/2019 at 1:09 AM

Hello. We're running custom build 7164. Woken up this night by spool monitoring alert this night at 2am. 16k+ mails in the queue, 98% were mails looping with Failed: Failed: Failed: Failed.

The issue is still present and it's really becoming a big problem.

Up to (and including) build 7125 this issue didn't exists. It's really getting a huge problem. Is it not possible to reverse changes made to the spool code or whatever that broke this. It's really basic functionalities of a mail server that should be rock solid!

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

8/17/2019 at 2:39 AM

I guess this is another weekend that we'll be stuck with the issue on production servers having to actively monitor the queue to avoid spool filling up because of this... :(

I would revert back to 7125, before problem appeared. But we can't as it has bugs for mailing list that our customers complained about...

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

8/17/2019 at 2:56 AM

As a side effect, we have now to clear mailboxes of users impacted by the problem. They are filled with ten of thousands of this:

Bonus: You can't search for "System Administrator" keyword and remove the mails as it returns no result. So it's a pain to clean their mailboxes without deleting legit mails.

Also why the mails are dated 08/18/19 when we're 08/17/19 ??

This issue drives us crazy and we have to spend an unacceptable amount of time fixing the aftermath.

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

8/17/2019 at 3:08 AM

It's a few hours we cleaned the spool and here it starts again!

Where is SmarterTools team? They know the issue exists and they really leave us in this situation ??

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

8/17/2019 at 4:00 AM

We're hitting the worst scenario ever.

contact@domain has forward to: (domain is local to smartermail)

user1@domain

user2@domain

user3@domain

user4@domain

user3@domain mailbox is full. So a notification is sent to contact@domain.

contact@domain forward the user3 mailbox full notification to:

user1@domain

user2@domain

user3@domain

user4@domain

user3@domain mailbox is still full. So a notification is sent (again) to contact@domain (endless loop)

user1, user3, user4 mailboxes are too filled with the forwarded mailbox full notifications for user3 and become full too (seems notifications are delivered even if mailbox is full) which generates more notifications that are looping forever filling each mailbox exponentially

The delivery log grows exponentially too, the spool raises to more than 100k mails (before we decide to stop/kill the server).

Logs are full of "mailbox is locked" as they are constantly being accessed.

What the hell we never had such problem up to and including 7125... We've opened a ticket explaining this on 18 july...

This is a major fail

Sébastien Riccio System & Network Admin https://swisscenter.com

Gregory Liénard Replied

8/18/2019 at 2:17 AM

Same problem over here.

Problem is that authentication is bypassed, but nowhere, not in the header, raw, html or HDR of the message in the spool is the IP address of the sender mentionned. I would say that this is critical info a System Administrator needs to detect which internal server is being abused to send spam.

This is an example of the HDR:

Written email@academyeven.icu email@academyeven.icu retry: 0;08/18/2019 12:09:24 from: email@academyeven.icu auth: bypass creationdate: 08/18/2019 12:09:24 containsLocalDeliveries: False

SMTP In relay settings is set to nobody and all of our users must authenticate. There is no whitelist of IPs in Security at all. How does it come that auth: bypass? What does this mean? We do not have a user email@academyeven.icu or that domain in our smartermail server. How is it possible that this mail is being sent???

Sébastien Riccio Replied

8/19/2019 at 1:14 AM

Gregory, I think it's because these hdr and eml files are the files of non delivery reports and they are sent by the system administrator, that's why auth is bypass.

It's not really a spam problem. It happens when for exemple a recipient mailbox is full and a notification is sent about it.

Sébastien Riccio System & Network Admin https://swisscenter.com

David Finley Replied

8/20/2019 at 6:59 PM

Same thing happening here and on Build 7165. Any other suggestions?

http://www.interactivewebs.com

Tim DeMeza Replied

8/21/2019 at 8:03 AM

I opened a support ticket a while ago and I am on custom build .7164. For me, the issues seem to be resolved.

However, I have a possible related issue.. I am trying to do an archive search, and it is taking a really, really long time. Is it possible all the "FAILED" spam iterations that kept showing up in the spool are archived? If so, any way to clean these. I have isolated the slow searches to the past 2 months which seems to coincide with this issue.

Any assistance or ideas are appreciated.

Kyle Kerst Replied

8/21/2019 at 10:53 AM

Employee Post

Hello everyone, sorry for the delay. We've received a new custom build and have tested to confirm this resolves the looping issues in all of the scenarios we've had reported so far. Can you please test with this build and let us know if you see any further issues or scenarios that cause the behavior?

Download: http://www.smartertools.com/downloads/SmarterMail/CustomBuilds/100.0.7172.16219/SmarterMail_7172.exe

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

David Finley Replied

8/21/2019 at 11:47 PM

I installed the 7172 version and it appears to have fixed this problem.

http://www.interactivewebs.com

Sébastien Riccio Replied

8/23/2019 at 1:50 AM

Yes, it seems build 7172 is way better at handling these infite bounce loops.
We're still trying different scenarios that we had before, to be sure they are all handled now.

Sébastien Riccio System & Network Admin https://swisscenter.com