mail delivery issues since build 7153

8/14/2019 at 5:51 AM

Might be related to this?

Your message has been delivered to the following recipients

8/14/2019 at 8:23 AM

Looks like are we have similar issues with our SmarterMail system:
We are receiving the System-Messages with Delivery-Notification for all outgoing emails, but only for the automatically by our SmarterMail/Events automatically added "journal@"-recipient/account used for our message-archiving software.

Our Events "Journal Incoming" and "Journal Outgoing" - both just do "Add recipient" journal@ourdomain.de
and under "Spool" and "Waiting for Delivery" are many, many hundreds/thousounds of emails.... most of them dealing with delivering to our local journal-useraccount (looks like only this "last/added recipient" ist failing, originally recipients are already ok/success!
Could even be that some kind of messages having "looping issues" because often many, many, many textlines with "Failed" :-(

I can even see "mailbox is locked" messages in the delivery log for the journal-account, re-indexing was no solution, still seeing this issues/SystemMessages with delivery failure
[2019.08.14] 17:20:46.631 [40279] Add message to mailbox call failed. Reason: Access to the grp file is locked, monitor enter failed.
[2019.08.14] 17:20:46.631 [40279] Delivery attempt to journal@xxxxxxxxxx.de to folder 'Inbox' failed. Mailbox is locked.

I tried with SystemMessages and "empty from address", and even with entering "postmaster as the from-address", this did not make any difference concerning the messages/failure-emails.

And our users do think, that there original/destination was not working, they do not notice that in our case it is just our "journal"-recipient failing!

tested with v17 / build 7153 and even with the MAPI-beta 7159
looks like the bug/issue is not solved/fixed yet.

Kyle Kerst Replied

8/14/2019 at 9:34 AM

Employee Post

Thomas, if you're still having looping issues please deploy the custom build below which includes fixes for this as well as some automated forwarding delay problems:

Download: http://www.smartertools.com/downloads/SmarterMail/CustomBuilds/100.0.7164.17721/SmarterMail_7164.exe

Instructions: https://portal.smartertools.com/kb/a2855/minor-upgrade-of-smartermail.aspx

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

B

Bruce Replied

8/14/2019 at 12:51 PM

I am getting 1,000 of emails looping on multiple SmarterMail servers with the subject line;

Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:Failed: Failed: Failed: Failed: Failed: Failed:Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed:

For each loop this gets longer.

Will this custom build fix this?

Employee Replied

8/14/2019 at 12:53 PM

Employee Post

Hi Bruce. Yes, build 7164 will fix this looping.

B

Bruce Replied

8/14/2019 at 1:01 PM

Thanks, Rod, for the quick reply.

Thomas Lange Replied

8/14/2019 at 1:43 PM

Thanks Kyle for the link to this custom build.
(btw: I suppose the MAPI-beta needs an updated/fixed build, too)

Bruce: I just installed custom build 7164 - looks good now!
But keep in mind in case your "Spool"-queue is already filled with thousounds of messages (we had 3.000 upto 8.000 Waiting for delivery!) you need to stop the SmarterMail service, then RENAME the Spool-Directory, and restart the SmarterMail-service. SmarterMail will then re-create a new spool-directory, including it´s subdirectories.

Keep in mind you could loose emails: what was in the old-spool will not being send anymore! , thats why I suggest a rename of this folder and not a delete, to have the chance to manually check content/files.

B

Bruce Replied

8/14/2019 at 6:17 PM

I was able to clean the spool queues, but I have just been woken at 2am by a throttle notification and checked the spool on one of the servers and it has occurred again, 1,000's of emails with 'Failed: Failed:' subjects.

It looks like 7164 has not fixed the problem and the loop is still occurring.

B

Bruce Replied

8/14/2019 at 6:57 PM

It looks to be when you get a Failed Delivery notification from 'System Administrator' to a mailbox that is full that because the Failed Notification can not be delivered it then gets stuck in a loop.

S

Stefan Mössner Replied

8/15/2019 at 1:54 AM

Hello all,

I had the same issue with SmarterMail, build 7153. The new custom build 7164 and deleting the spool files helped me to get back a performant running SmarterMail.

Thank You.

B

Bruce Replied

8/15/2019 at 3:15 AM

I have not seen any more since 2am this morning, so I am hoping it was just a hangover from the previous version.

S

Stefan Mössner Replied

8/16/2019 at 1:57 AM

Hello all,

unfortunatelly it only worked for nearly one day. Again there were some mails with "Failed:". I'm using some addresses of my internal mail domain which doesn't exist as mailboxes for notifications from firewalls, IP cameras etc. I had to delete these mails for stopping SmarterMail to try to send them again and again. In the past releases of SmarterMail there wasn't such an issue. Why isn't SmarterMail stopping to send these mails after a few retries?

Kind Regards.

B

Bruce Replied

8/16/2019 at 2:14 AM

I am also still getting this but only now on one server, which has 12,000 mailboxs, but this issue only seems to affect one mailbox that sends email notifications but the mailbox is full so failed emails cannot be returned, so you end up with an email loop with the subject line;

Failed: Failed: Failed: Failed: Failed: Failed: Failed:

B

Bruce Replied

8/16/2019 at 2:26 AM

OK, just found another mailbox that this is also happening too which is also rejecting email as the mailbox is full.

As a temporary fix I am increasing the size of the affected mailboxes so that they can receive the 'Failed' email and stop the loop.

B

Bruce Replied

8/19/2019 at 6:47 AM

Is there a fix for this yet.

The issue was fixed for when a mailbox does not exist, but not for when a mailbox is full.

S

Stefan Mössner Replied

8/19/2019 at 3:22 PM

It's even not fixed when the sending mailbox doesn't exist.

Kyle Kerst Replied

8/19/2019 at 3:50 PM

Employee Post

Hello everyone, thanks for keeping us posted on your results here. We're not seeing the same looping behavior during testing, so I suggest submitting a support ticket so we can take a closer look at this. One thing that works after you've upgraded to 7164 is to add an SMTP Block under Settings>Security>SMTP Blocks for the offending from addresses, then remove the already existing messages in the spool. Once cleared out you should be able to remove the SMTP Block and resume normal operation. Please give that a shot and reach out to support if these issues persist. Thanks!

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

S

Stefan Mössner Replied

8/19/2019 at 10:37 PM

Hello Kyle,

I don't think that this will help because this behavior happens again and again. I get monitoring mails every day and so there are mails in the spool folders with "Failed" in the subject even if they are delivered correctly.

This is a new behavior since build 7153. I can't open a support ticket because I use the free version of SmrterMail.

Kind Regards.

B

Bruce Replied

8/20/2019 at 12:21 AM

The SMTP Blocking is not an option as these are local mailboxes that send the emails.

I will explain the behaviour I am seeing in the hope that you can replicate it or know how to fix it.

When the email fails to be received =SmarterMail creates a 'Failed' email from the System Administrator. I think this is the 'Delivery Status Failure' email in the 'System Messages'.

If the 'Failed' email from the System Administrator can not be delivered because the sender's mailbox is full it ends up in a loop with SmarterMail sending 'Failed' emails for the 'Failed' emails.

I have fixed the issue on my system by when I spot the 'Failed' emails in the spool I increase the amount of diskspace on the full mailbox so they can receive the 'Failed' email from the System Administrator thus stopping the loop.

Sébastien Riccio Replied

8/20/2019 at 3:30 AM

SM build 7125 was not affected.

The problem appeared in custom build 7139 that we had to install to fix newsletter issues after customer complaints.

I opened a ticket explaining precisely the issue (same as bruce is describing here), that was 20 july (1 month ago).
I also specified in follow-ups that it appeared after build 7139 installation, so they are aware of it, to better locate the code changes that could be the source of the problem.

After a few support ping-pong resulting in new custom build to install the problem is still here.

In the meantime 7153 was publicly released containing the problem... ( I really don't get it why they released a public version with a _known_ major issue )

Since a month we're spending an unbelievable amount of time each day to monitor/clean spool of these NDR before queue starts to be filled with 100K and more of these FAIL mails and creating locking problems on the mailboxes, and even server lockups...

Everybody is angry and tired here, our tech staff, our customers (threating to use another service...)

This really need to be top 1 issue to be resolved.

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

8/20/2019 at 7:30 AM

Cool. Crossing fingers

Sébastien Riccio System & Network Admin https://swisscenter.com

C

Carlos Ramon Garcia Enriquez Replied

8/20/2019 at 8:46 AM

Web Team what new version is?

Sébastien Riccio Replied

8/20/2019 at 12:18 PM

Really?

Sébastien Riccio System & Network Admin https://swisscenter.com

David Finley Replied

8/20/2019 at 6:31 PM

Why not release these builds as you fix issues?

http://www.interactivewebs.com

B

Bruce Replied

8/21/2019 at 12:16 AM

Ron are you saying not to install this latest fix?

Is anyone else seeing that webmail auto refesh is broken?

David Finley Replied

8/21/2019 at 12:39 AM

I just tested on Safari on Build 7165 (Aug 14, 2019)
- No problem deleting. Found it worked as one would expect.

http://www.interactivewebs.com

B

Bruce Replied

8/21/2019 at 2:51 AM

Thanks David.

I have updated 3 servers with Build 7165 and not seeing an issue with deleting emails.

After installing an upgrade of SmarterMail you do need to refresh your browser in order for auto-refresh to start working again in the webmail, it might have been this that caused Ron's delete problem.

B

Bruce Replied

8/21/2019 at 5:38 AM

Build 7165 is worse than it was previously for us.

Get a huge number of these Failed emails now.

The issue has shifted back from a problem with full mailboxes to non-existing mailboxes again.

B

Bruce Replied

8/21/2019 at 5:44 AM

Previously every time SmarterTools came out with a new major release it would be buggy and unusable for the first 6 months which was bad enough.

Now it is every month you release a buggy and unstable version and you then release custom builds all month to try and fix the issues before it all starts over the next month!

Come on SmarterTools get some software testers as if this pattern continues will certainly be moving to a competitor and stop paying you $5,000 per year to be your beta testers.

Kyle Kerst Replied

8/21/2019 at 4:26 PM

Employee Post

Hello everyone, sorry for the delay here. We've identified a fix for the few remaining scenarios causing these issues and I've included the custom build below. Please give this a shot and let us know if you see any further problems.

Download: http://www.smartertools.com/downloads/SmarterMail/CustomBuilds/100.0.7172.16219/SmarterMail_7172.exe

Instructions: https://portal.smartertools.com/kb/a2855/minor-upgrade-of-smartermail.aspx

As to the refresh issues, we're not seeing these problems here. Can you submit a support ticket on that one?

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

S

Stefan Mössner Replied

8/22/2019 at 10:12 PM

Hi Kyle,

it looks like this build (7172) solved the issue. Tonight there were no more mails with "Failed" in the spool folders. Thank you for solving the issue.

Kind Regards.

B

Bruce Replied

8/23/2019 at 12:57 AM

Looks like the issue is resolved for us also with build 7172.

Thank you.

C

Christian Schmit Replied

8/23/2019 at 1:34 AM

We are also on build 7172 since last night and everything seems to be working as it should.

Sébastien Riccio Replied

8/23/2019 at 1:49 AM

Yes, it seems build 7172 is way better at handling these infite bounce loops.

We're still trying different scenarios that we had before, to be sure they are all handled now.

Sébastien Riccio System & Network Admin https://swisscenter.com

d

domains Replied

8/26/2019 at 8:10 AM

We have applied Build 7172 but are still having mail rejections with a "Mailbox is locked" error

Kyle Kerst Replied

8/26/2019 at 9:15 AM

Employee Post

Hello everyone! Thanks for the updates on your results with this, I'm glad to hear we were able to get to the bottom of this and resolve it. Please let us know if you run into any additional scenarios that result in this behavior.

Hello Domains, The mailbox locked behavior is an expected result if an existing delivery session is taking place, or POP access is active on the account. Does the account experiencing those errors typically receive a large amount of email? You may want to adjust your retry intervals to me something like this for best results as well (Settings>General>Spool)

1, 1, 5, 5, 15, 30, 30, 30, 30, 60, 90, 120, 240, 480, 960, 1440, 2880

These retry intervals account for local delivery/forwarding delays as well as legitimate (longer) failures to deliver. Please give that a shot and if you continue to have issues please submit a ticket so we can get to the bottom of this.

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

d

domains Replied

8/26/2019 at 9:43 AM

Kyle,

Yes the accounts get a very large volume of email but no POP account access. I have tweaked the spool settings and will monitor. A trouble ticket was submitted last week, but it hasn't been resolved

Sébastien Riccio Replied

8/26/2019 at 3:15 PM

Still not convinced why retry intervals are involved in this ....

Sébastien Riccio System & Network Admin https://swisscenter.com

Kyle Kerst Replied

8/26/2019 at 3:22 PM

Employee Post

Domains, happy to hear you were able to get the retry intervals corrected. I'll take a look at your ticket and see if I can offer further guidance there as well.

Sebastien, the retry intervals are a necessary change due to some modifications to how we handle delivery sessions re: forwarding and system message notification delivery. These retry intervals should be present by default on new installations, but will require the update on existing for optimal performance.

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

Sébastien Riccio Replied

8/26/2019 at 3:44 PM

Hello Kyle, we would be interrested to understand why these changes were necessary? Thank you

Edit: Retries are in case of failure ? It's kinda misleading

Sébastien Riccio System & Network Admin https://swisscenter.com

S

Stefan Mössner Replied

8/26/2019 at 9:54 PM

Hi Kyle,

thank you for giving us the information about the need of changing the spool settings on existing installations. Do we have to restart SmarterMail after changing this?

Kind Regards.

Kyle Kerst Replied

8/27/2019 at 7:40 AM

Employee Post

Sebastien, from what I understand these changes were to essentially make the spooling/delivery process more efficient, eliminating or mitigating as much as possible cases where we would have to wait to deliver in the past. I can gather more specific details for you on this though if you can ping me on your existing ticket. Retry intervals are designed to handle delivery failures first and foremost, but apply to delivery in general, and will be used for all delivery session. This is why we have a couple of short retries in there at the start, followed by more typical durations to account for legitimate failures or locked mailboxes.

Stefan, happy to help! You should not need to restart the SM service to allow these changes to take effect.

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

Alex Clarke Replied

8/27/2019 at 12:58 PM

So... will a future build of SM automatically adjust the retry intervals, or is this a change that we will manually need to make?

Neal Culiner Replied

8/27/2019 at 1:24 PM

If SmarterTools ran the builds 1 week prior to public release would this situation occur?

Alex Clarke Replied

8/27/2019 at 1:35 PM

If SmarterTools ran the builds 1 week prior to public release would this situation occur?

Probably. 1 week isn't long enough to test a release and some bugs only expose themselves under certain conditions.

Kyle Kerst Replied

8/27/2019 at 1:57 PM

Employee Post

Alex, the retry intervals should be updated in new installs going forward.

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

Alex Clarke Replied

8/27/2019 at 2:08 PM

Thanks for the confirmation Kyle!

Any idea when the next release will be please?

Kyle Kerst Replied

8/27/2019 at 3:51 PM

Employee Post

You're very welcome Alex, always happy to help and will provide information as we get it :-) The next release date isn't something I'm privy to at the moment but I would expect it in the very near future as we've been focused on bug fixes pretty heavily over the last week or so and theres a lot of good stuff incoming. Have a great day!

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

P

Patrick Mattson Replied

8/28/2019 at 10:05 AM

OK so i understand I got two emails stuck doing the failed: failed: failed. When will the fix be released officially? I have run into issues since upgrading from 14 and have been afraid to upgrade since.

Kyle Kerst Replied

8/28/2019 at 10:20 AM

Employee Post

Patrick, this is the loop behavior being discussed above, and I recommend performing a minor upgrade to the custom build I linked previously:

Download: http://www.smartertools.com/downloads/SmarterMail/CustomBuilds/100.0.7172.16219/SmarterMail_7172.exe

Instructions: https://portal.smartertools.com/kb/a2855/minor-upgrade-of-smartermail.aspx

This build has now been running in several environments for a period of time with no further looping. The minor upgrade itself should be entirely painless, but be sure to take a full backup of your environment before getting started:

https://portal.smartertools.com/kb/a2643/backup-and-restore-smartermail.aspx

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

Dillon Yonash Replied

10/17/2019 at 1:35 PM

Can you confirm that these fixes have been rolled into the standard (non-custom) release of build 7188? I think this describes what I was seeing in 7153, but I didn't find this thread until today. I installed 7188 earlier this week.

Kyle Kerst Replied

10/17/2019 at 4:56 PM

Employee Post

These fixes were rolled into 7188, and you should no longer see long delivery delays. What do these delays look like? Are you seeing them stall in the spool or are you receiving user complaints of delayed delivery?

Kyle Kerst IT Manager SmarterTools Inc. www.smartertools.com

B

Bruce Replied

10/18/2019 at 12:18 AM

Looks like the issue still exists in Build 7188 under some circumstances.

I had a loop of 22,000 emails with the subject line;

Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: Failed: ..........................

This occurred after disabling a mail domain. The emails that failed were sending an email to the same address as sent the email.

The delivery log for one of these emails is below, which I have redacted the real email address and replaced with example@example.com;

[2019.10.15] 17:14:11.800 [65092] Delivery started for example@example.com (via bypass) at 17:14:11 [2019.10.15] 17:14:17.826 [65092] Added to SpamCheckQueue (0 queued; 4/150 processing) [2019.10.15] 17:14:17.826 [65092] [SpamCheckQueue] Begin Processing. [2019.10.15] 17:14:19.717 [65092] Starting Spam Checks. [2019.10.15] 17:14:19.717 [65092] Skipping spam checks: User authenticated [2019.10.15] 17:14:19.717 [65092] Spam Checks completed. [2019.10.15] 17:14:19.717 [65092] Removed from SpamCheckQueue (0 queued or processing) [2019.10.15] 17:14:20.857 [65092] Added to LocalDeliveryQueue (0 queued; 3/50 processing) [2019.10.15] 17:14:20.857 [65092] [LocalDeliveryQueue] Begin Processing. [2019.10.15] 17:14:20.857 [65092] Starting local delivery to example@example.com [2019.10.15] 17:14:20.874 [65092] Delivery for example@example.com to example@example.com has bounced. Reason: The domain is disabled [2019.10.15] 17:14:20.998 [65092] DSN email written to 56365102 with status failed to example@example.com [2019.10.15] 17:14:21.013 [65092] Delivery for example@example.com to example@example.com has completed (Bounced) [2019.10.15] 17:14:21.013 [65092] Removed from LocalDeliveryQueue (0 queued or processing) [2019.10.15] 17:14:23.873 [65092] Removing Spool message: Killed: False, Failed: False, Finished: True [2019.10.15] 17:14:23.873 [65092] Delivery finished for example@example.com at 17:14:23 [id:56365092]

L

Long Chung Replied

4/15/2020 at 3:36 AM

The version I use is: internal version 7219. Delays often occur, and a return message appears: 601 attempts to send the message to the following IP: xxx.xx.xx.xx

I have issued a support vote, but it has not been resolved after a long time. I think this is due to the bad design of the mail software itself, because it is normal to modify the SMTP gateway to send it out.

This SMTP gateway is another host in the same environment, so it is not a problem of the environment but the software itself.

54 Replies

Reply to Thread