6
SM 8580 upgrade?? Any issues so far?
Question asked by Brian Bjerring-Jensen - 6/2/2023 at 5:28 AM
Unanswered
Just saw the release notes.

Anyone tried it yet?

61 Replies

Reply to Thread
5
Updated my biggest server a few hour ago from 8451.
All seem ok... finger crossed!!!
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
1
Mike Mulhern Replied
Anyone know if the MAPI issues are resolved?
1
MAPI has worked for us flawlessly since 8545 upgrade.
2
MGWallace Replied
In a couple weeks I may update of our servers with many clients on it if all continues to go well with those who've updated their servers. Thank you for updating and reporting what happens!

Cheers.
3
Updated to 8552 on June 2nd directly from "old SM"  build 8451.

I have no new problems, for now everything is pretty good, even if I have some observations to make... I gave more details here: 
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
1
Tan Replied
Just upgraded but there is tons of cpu load. Support taking a look at it but the slowness and unresponsive is driving me nuts. Hope support can find the root cause.

Previously was OOM issue and was told to upgrade now instead of OOM become CPU issue.

Update:
CPU is stable now, here is what SM responded. Strange, how come you guys not having the same issue

====
The issue was that the URIBLs were activating Spam assassin and spam assassin was doing lots of regex lookups while checking emails for spam. The developers think that this can be optimized and will be working on it. In the meantime, I have disabled your URIBLS in Settings->Antispam->URIBLS
====

0
Manuel Martins Replied
HI,
I get this error on SMTP log, does anybody know why ?

rsp: 501 Syntax error, syntax: "MAIL FROM:" "<" address ">" / "<>" [SP Mail-parameters] CRLF

1
Jay Dubb Replied
Aside from the initial RAM and CPU spikes on "first login" how is resource consumption moving forward?  

We brought on a large client (currently in the 1,700 mailbox range) in late 2021 and We/They suffered through some pretty agonizing MAPI issues during the first year of service-- enough that they notified us they were going to break contract and were already getting quotes from 2 other large ESPs.  Fortunately the biggest pain points were fixed and we were able to retain the account.  There are still MAPI problems that forced them to convert dozens of users to IMAP + CalDAV/CardDAV.  That has been enough to give some relief, and things have been fairly quiet the past couple months, thankfully.

Quiet enough that we are VERY afraid to move off 8451 (Feb 20, 2023) and jump back into a potential fire.

We've read about the MAPI issues on the new version that persisted coming out of Beta.  Some say "fixed with version __" while others say there are still some lingering problems.  We have to know MAPI is "right" before risking the upgrade.

Another huge concern is CPU, and especially RAM consumption, going way up. Our Smartermail server hosting around 1,800 mailboxes with 7.5 TB of mailbox data, is a resource HOG already.  It runs on a Dell R440 bought new in 2022.  Dual Xeon, 8 x 2 TB Enterprise SAS (not SATA) SSDs.  During the average weekday, resource consumption is:

  • 20 Xeon cores sustained at 50-60% with spikes to 95%.
  • 68-70 GB of sustained RAM consumption (80 GB total installed) with daily spikes to 76-78 GB.
If the new version requires more CPU and RAM than 8451, we'll have to lay out another 5-figures for a fatter server... something we are NOT particularly willing to do given our investment in the current server last year.

Thoughts, insights, opinions?

0
This is what we see. Very easy on ressources. Look into the spamlists and they are the worst culprits regarding ressource use.

1
Lasse B. Replied
There is apparently a problem with the Spam Checker module, where it checks intermediate mail server IP addresses if the sender's mail server IP address is whitelisted. Intermediate mail server IP addresses are rarely in SPF records (e.g. an IPv6 address) and therefore SPF validation fails. I have a case open about it, as the users in Webmail see a warning on all mails from e.g. Microsoft Outlook 365.

I would just let you know.
2
Jay Dubb Replied
@Brian Bjerring-Jensen, your graph is similar to the resource consumption on our serve too, until we added a MAPI customer with around 1,700 mailboxes.  Now, THIS is what our consumption looks like on a typical weekday afternoon.



So you can see why we're more than a little worried about what things will look like on the new version.
 
 

0
J Lee Replied
I'm seeing server crashes every few days, and one else seeing this?
J. Sebastian Lee Service2Client LLC 6333 E Mockingbird Ste 147 Dallas, TX 75214 - 877.251.3273
0
Manuel Martins Replied
Hi,
Our Server is working 25-40% CPU and RAM. No crashes fortunatly!! 

My major concern for now is that we see a very high increse on Disk Bandwidth Usage since upgraded to this new release, 2 weeks ago, the old version did not used to use so many Disk Bandwidth and Webmail is much slower than it was.

1
We are on 8566 now. Apart URIBLs bug (se the other thread here...), everything is working good.

Yes, we also are seeing a (fairly) big increment in CPU, RAM and disk I/O usage, but it's not a problem by now.

Sure I hope that CPU and RAM usage will be more optimized in the future...
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
0
Jay Dubb Replied
@Gabriele , when did you first upgrade to the new version track?  Was that recently and maybe the system is still grinding through its "first use" conversions, or was that a while back and the increased RAM/CPU/Disk is a permanent thing now?
0
I upgraded over a month ago... The increased resource usage is really permanent, but I've seen small improvements with almost every build, so I'm hopeful they can make more improvements in the future
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
0
Seeing very little use on 8566

After the inital reboot everything settles down and is running rock solid.No issues with MAPI or sync at all.

Now I am fighting SPAM since a lot of shit is coming through...
0
Jay Dubb Replied
I'm seeing a number of reports of much higher disk I/O.  Does anybody know the root cause of that, beyond the one-time first-login update that happens?  Our mail server is using an array of 8 x Enterprise SAS SSDs on a fat caching controller.  Even with that very FAT (and expensive, these SAS SSDs are $2,000 each) disk subsystem, there are times where disk IO is so heavy (build 8451) that it noticeably affects performance, particular for webmail users.

We really WANT to upgrade to the new version, but so many reports of increased CPU, RAM, and disk I/O have us extremely concerned about taking that irreversible step forward.  We were expecting improved efficiencies over the old version, but it's not looking to be the case.
 
1
Hi Jay! We are on 8566 now and efficiency is really improved over the first buils of the "new" SmarterMail.

CPU, RAM and disk I/O are still higher than the "old", but not so high, we see about 1,4-1,5x on our server

The first builds where sometimes over 3-4x... 
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
0
Chris Danks Replied
We just upgraded to 15 June build and our server keeps crashing. I’ll try what @tan said above. 
3
ActorMike Replied
Trusted Senders go to Junkmail and those browser notice nag messages that cannot be permanently dismissed are a nuisance. 

Image Caption
1
J Lee Replied
2nd, the Trusted Sender does not work all the time.
J. Sebastian Lee Service2Client LLC 6333 E Mockingbird Ste 147 Dallas, TX 75214 - 877.251.3273
0
Chris Danks Replied
have you opened a ticket with smartertools so they can see the issue themselves and then fix it for everyone else in the next update?
3
ActorMike Replied
They already know about the trusted senders issue. Obviously this is a serious problem since valid email can get dumped into junkmail. Hopefully a quick fix. Regretting updating now.
0
Chris Danks Replied
Yeah I too really regret updating!
I wish smartertools would have a beta/rc/stable build process as it feels like i have upgraded to a beta build. 
0
ActorMike Replied
What happened to the view raw header function? We cannot report spam messages anymore.
0
Andrew Barker Replied
Employee Post
ActorMike,

The View Header option is available in the ... menu in the message view. If you want View Raw Content, that is accessible by right clicking on the message in the message list.
Andrew Barker Software Developer SmarterTools Inc. www.smartertools.com
2
ActorMike Replied
Andrew Barker  True but it only shows the message header now and not the raw content in the message body, so we cannot report the message to spamcop! We've been able to view the raw message forever!

Add this to the list of downgrades.

0
Kyle Kerst Replied
Employee Post
For all admins seeing Trusted Sender issues - please open a ticket with us so we can help dig into it further. Trusted Senders are working in the vast majority of environments we've looked at, and the ones where it isn't usually ends up being a configuration change being required to account for improvements in these areas. 

@ActorMike - you can use the ...>Download EML to get the full raw content (including headers as well) for spam submissions if needed. These can be opened in a text editor if need be, but usually you can just zip up the EML and submit that to your antispam vendor.
Kyle Kerst IT Coordinator SmarterTools Inc. www.smartertools.com
0
ActorMike Replied
I submitted a ticket for the Trusted senders issue and I guess I misunderstood and thought it was a bug. How can we get this addressed? 
0
Mark K. Replied
On Server 2019 I removed the Windows Defender Antivirus Role and it helped
0
Webio Replied
@ActorMike just like someone else suggested view raw source is available still but not under three dots button above message but above message list.

When it comes to trusted senders IMHO it might be related to return-path.

SMTP log on incoming gateway:

...
2023.06.28 11:21:24.202 [69.169.224.126][27770915] cmd: MAIL FROM:<01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com>
2023.06.28 11:21:24.202 [69.169.224.126][27770915] senderEmail(1): 01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com
2023.06.28 11:21:24.202 [69.169.224.126][27770915] rsp: 250 OK <01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com> Sender ok
2023.06.28 11:21:24.202 [69.169.224.126][27770915] Sender accepted. Weight: 0. Block threshold: 90.
2023.06.28 11:21:24.233 [69.169.224.126][27770915] cmd: RCPT TO:<RECIPIENTEMAILWASHERE>
2023.06.28 11:21:24.342 [69.169.224.126][27770915] rsp: 250 OK <RECIPIENTEMAILWASHERE> Recipient ok
2023.06.28 11:21:24.373 [69.169.224.126][27770915] cmd: DATA
2023.06.28 11:21:24.373 [69.169.224.126][27770915] Performing PTR host name lookup for 69.169.224.126
2023.06.28 11:21:24.436 [69.169.224.126][27770915] PTR host name for 69.169.224.126 resolved as b224-126.smtp-out.eu-central-1.amazonses.com
2023.06.28 11:21:24.436 [69.169.224.126][27770915] rsp: 354 Start mail input; end with <CRLF>.<CRLF>
2023.06.28 11:21:24.482 [69.169.224.126][27770915] senderEmail(2): message@moje.zalando.pl parsed using: Zalando <message@moje.zalando.pl>
Delivery log on incoming gateway:
2023.06.28 11:21:52.453 [89968425] CMD: MAIL FROM:<01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com> RET=HDRS ENVID=febcfc6e-a2af-4563-bbc3-657ac7576008 SIZE=102494
2023.06.28 11:21:52.484 [89968425] RSP: 250 OK <01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com> Sender ok
Delivery log on primary server:

2023.06.28 11:21:52.362 [40553307] Delivery started for 01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com at 11:21:52
2023.06.28 11:21:55.830 [40553307] Added to SpamCheckQueue (0 queued; 3/150 processing)
2023.06.28 11:21:55.830 [40553307] [SpamCheckQueue] Begin Processing.
2023.06.28 11:21:55.830 [40553307] Blocked Sender Checks started.
2023.06.28 11:21:55.830 [40553307] Blocked Sender Checks completed.
2023.06.28 11:21:55.830 [40553307] Spam Checks started.
2023.06.28 11:21:55.830 [40553307] Spam Checks skipped: Spam checks done by inbound gateway
2023.06.28 11:21:55.830 [40553307] Spam Checks completed.
2023.06.28 11:21:56.705 [40553307] Removed from SpamCheckQueue (1 queued or processing)
2023.06.28 11:21:59.674 [40553307] Added to LocalDeliveryQueue (0 queued; 2/150 processing)
2023.06.28 11:21:59.674 [40553307] [LocalDeliveryQueue] Begin Processing.
2023.06.28 11:21:59.690 [40553307] Starting local delivery to RECIPIENTEMAILWASHERE
2023.06.28 11:21:59.721 [40553307] Process delivery status notification step from local recipient success. Recipient: [RECIPIENTEMAILWASHERE], Notify: [failure], Delivered: [True], Forwarded: [False], Deleted: False
2023.06.28 11:21:59.721 [40553307] Delivery for 01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com to RECIPIENTEMAILWASHERE has completed (Delivered to Junk Email) Filter: Spam (Weight: 33), Action (Global Level): MoveToFolder
2023.06.28 11:21:59.721 [40553307] End delivery to RECIPIENTEMAILWASHERE (MessageID: <01070189014eba02-e17d6307-9131-4e22-9386-02235475965d-000000@eu-central-1.amazonses.com>)
So as you can see email from return path is being used for verification (it looks that way) during checking process BUT when I go to junk mail in webmail and I'm opening this message it shows trusted sender (message@moje.zalando.pl) info.
0
Tan Replied
@Actormike since you are on the latest build, can you help me to check on this?

0
MGWallace Replied
Has anyone updated to the newest build (8587) as of July 6, 2023?
If so, how has this improved the CPU and RAM issues?
Also, how about the Trusted Senders going to junk mail?
Again, thank you to everyone who are assisting with these newest issues builds.
Cheers.
0
J Lee Replied
J. Sebastian Lee Service2Client LLC 6333 E Mockingbird Ste 147 Dallas, TX 75214 - 877.251.3273
0
ActorMike Replied
M.G., there is no note of the trusted senders going to junkmail being fixed, unfortunately.

Build 8587 (Jul 6, 2023)

  • Fixed: An issue were a single appointment in a user's account caused sync issues and exceptions in SmarterMail.
  • Fixed: An occasional blank compose/reply window when SmarterMail is running in a virtual directory in IIS.
  • Fixed: Better handling of string comparisons in MAPI related to Turkish Windows servers.
  • Fixed: Category sync issues in IMAP.
  • Fixed: Email migrated from Exchange 2010 has the wrong time in the sent folder.
  • Fixed: Folders created through any EWS client create those folders in the wrong location on the server.
  • Fixed: For some emails, the attachment icon doesn't appear in webmail, and the attachment doesn't appear in MAPI, EAS, and EWS clients.
  • Fixed: In News Feeds, the RSS title, "Last Updated" timestamp, and RSS summary are missing on all feeds.
  • Fixed: Meeting Invitations from MAPI to MAPI that include attachments lack Accept/Decline buttons and duplicate the attachments.
  • Fixed: Messages that are forwarded by automated forwarding or content filters have a mismatch between Return Path and From address.
  • Fixed: Setting a report start date to 01/01/1800 leads to MailService.exe consuming ~5GB of memory until restart.
  • Fixed: URIBL's sometimes use excess CPU and take too long to do their checks.


1
Ben Rowland Replied
I installed 8580 yesterday. I had been having multiple outages per day and extremely slow performance on the previous build. I have found performance greatly improved with 8580 so far and no outages. Webmail had been unusably slow and is now working again.

I see 8587 now released but somehow the version downloaded and installed yesterday was 8580 instead of 8587. So I must have installed it after they posted the release notes but before posting the new installer.

Regardless, the performance is better, at least for me.
1
MGWallace Replied
@ActorMike: That is why I asked about the junk mail issue as well, as they didn't include it... thinking the URIBL's fix of taking to long and using up CPU would somehow be a fix to this issue as well.

Hopefully this new build will be doing what it is intended to do soon!

@Ben Rowland: Thank you for your update too! Very encouraging!
4
Martin Schaible Replied
Updated to 8587 today directly from old build 8451.

Everything went good, no issues so far. The logs are looking good.

I had to update the IDS rules to my needs only.
3
Brian Bjerring-Jensen Replied
On 8587 and no issues reported so far.
1
Alessandro Pereira Replied
We have a serious problem with excessive CPU usage in Build 8587 (July 6), we didn't have this problem in previous versions.
We disabled the URIBLS and also several RBL to be able to work again.
1
Thats because they are stuck in the spooler and the blacklist is very slow to load.
0
Tan Replied
@Alessandro Pereira, do you want to log ticket with SM because this issue technically should be fixed.
1
ActorMike Replied
Does the new build address the problem with Trusted senders going to junkmail that fail DMARC? A trusted sender should NEVER go to junkmail for any reason.
1
Jay Dubb Replied
Reviving an old threat here, now that several months have passed since the Beta went RTM.  I know there were problems with RAM and CPU consumption, and various bugs/breakage.  

Have these all been resolved now, to where it's a safe bet to upgrade?  Our SM server carries some very high-value/high-dollar clients that we absolutely cannot afford to upset, and we've held off upgrading (sticking with the devil we know) because of the troubles being talked about here in the SM community on the new version.  The release notes have been much less dramatic lately, so we're thinking it might be OK to jump in now.

Would love to hear opinions on the new version, good or bad.  For those who upgraded, would you do it again or do you wish you'd stayed on the "old" version (8451)?  Any words of warning or advice?
 
0
Mike Mulhern Replied
I'm currently on 8629 and it has been stable for me.
0
MGWallace Replied
RAM and CPU consumption are fixed, it has been stable for us... BUT when we upgraded to the newest build we ran into issues... those are be addressed now and will hopefully be addressed on the next update coming out.

We upgraded from an older build (Build 8451) to the newest current build (Build 8664), hence the issues.
1
Jay Dubb Replied
@M.G. Wallace - that would be our starting point also (8451, the last of the "old" builds).  What issues did you run into going from 8451 straight to 8664?

0
ActorMike Replied
I can't imagine having CPU and RAM issues.. We have a pretty old processor with 40 logical processors, runs IIS, MySql, Smartermail and sits on zero percent CPU.

0
Reto Replied
@ActorMike is this a HP server? Verify with Perfomance Monitor Processor\_Total if you have really a flat 0% usage. There is a bug where Task Manager shows no cpu usage. Had it to fix on some of our servers too :-)
0
MGWallace Replied
@Jay Dubb The issues we experienced will be addressed hopefully in the next build as the amazing team were able to duplicate those issues on their end. So, I would hold off till everything is fixed. 

One issue was: if an account never had a folder or file created for emails, an example being, the account never captured incoming emails, this included some admin accounts and accounts used for sending emails from websites) stopped working. those accounts needed to be removed and set back up again. 
(SmarterTools Dev team were able to fix this, so this should be in the next build)

This issue also lead to the entire domain not showing up in SmarterMail, if the admin account never had a folder created for emails. All other accounts within the domain sent and received emails, but from the SmarterMail admin site, you were not able to see the accounts. There is a fix to this, but it takes time, as in our case!

Another issue was: IDS Rules were duplicating upon every restart, meaning, when if you changed or updated any of the original Security IDS Rules upon restarting SmarterMail they added Original IDS Rules back in again, keeping the ones you already changed. So in our case, with all the domains which needed to be adjusted, to get them to show again, we ended up seeing 30 extra IDS Rules, especially for 'Password Brute Force by Email' and 'Password Retrieval Brute Force' in our case.
(SmarterTools Dev team still looking into this)
1
ActorMike Replied
@Reto there is no bug. Obviously, on occasion, there is a CPU spike, especially when loading a program on the server. Pretty much all servers are overkill nowadays for typical use.



0
Jay Dubb Replied
@M.G. - Thank you for that insight.  We're excited to move on to the new(er) version (the "er" part meaning there's yet another beta underway now) from this 10-month old version, although it has been fairly stable... save for the occasional CPU/RAM spike that kills the server.

@ActorMile - our server used to look like that, when we hosted POP3 and some IMAP.  But once we added a client who is currently consuming around 1,800 mailboxes using MAPI, EAS and Webmail, this is what the server looks like Mon-Fri.  And believe it or not, for a Monday, this is actually lighter than usual.

0
Jack. Replied
@Jay Dubb, wow, 55.8 Mbps. Is that network usage just for SmarterMail?
0
Jay Dubb Replied
@Jack ... Yep, just Smartermail.  It's a busy server, although 55 Mbps is a tad high.  It was still Monday morning in the time zone of our busiest customer when I took that screenshot, so that's daily volume plus retrieving what came in over the weekend.  Normal weekday traffic is in the 30-45 Mbps range, sustained, during business hours.
 
2
Tim Uzzanti Replied
Employee Post
Always nice to see customers servers doing a ton of work!  
Tim Uzzanti CEO SmarterTools Inc. www.smartertools.com
0
Jay Dubb Replied
@Tim , something we've noticed is a ton of disk activity related to indexing.  Sitting with the ResMon console open on the Disk tab, there is almost constant indexing activity on large accounts (especially the ones above 50 GB in size).  ResMon shows which files are being accessed, and it's not just recent days, it's old stuff that is months and years old where almost certainly there has been no activity.  Several mailboxes are so large (60-85 GB) that when the indexer finally reaches the end, it almost immediately starts over again.  There is so much index activity that it generates about 1 GB per second in read activity by itself.  Our owner/senior admin has had some tickets open on that before, without much resolution.  

Are there any optimizations being planned to cut down on the amount of indexing done?  Maybe some logic that doesn't re-index grp files that haven't been touched since the last run?  As busy as the server looks from a CPU and network traffic standpoint, the disk activity is even higher.  We had to buy a new server ($30k) with 8 x SAS Enterprise SSDs (which were $2k each!!) on a controller with huge cache, because our 8 x SAS 10k spinners on a striped volume could not keep up with the I/O.  

We discussed this internally, and concluded that we'd like an option to reduce indexing activity on an admin-selectable sliding scale.  Since old emails are almost never touched, we'd be willing to forego constant re-indexing on older mail, and allow the system to index on-the-fly if a user access old mail (at a slight performance penalty at the moment of access while indexing is done), in exchange for a reduced amount of regular indexing.

I don't want to hijack this thread into a discussion of indexing, but felt it was worth mentioning.
 
3
ActorMike Replied
Really don't understand why we still have trusted senders going to junkmail! This is a HUGE problem caused by Dmarc or something. We need our trusted senders in our inbox regardless of the reason/excuse.
2
Tim Uzzanti Replied
Employee Post
Jay,

I have no idea what your situation is based on seeing Task Manager.  If you want to open a support incident, we would be more than happy to look at your machine and review for good or bad.

In almost all cases, SmarterMail handles on similar hardware more than 2x the number of accounts and mail traffic than competing mail servers and in comparison to Exchange, its about 4x.  tWith the BETA, it will be even more!

Again, we would be happy to look at your machine and see if you have things setup well and/or there are some bottlenecks etc.
Tim Uzzanti CEO SmarterTools Inc. www.smartertools.com
4
Tim Uzzanti Replied
Employee Post
Mike,

If you would like to discuss Trusted Senders and / or our new Trust Shield that is in the BETA, please open a new thread.

There are a lot of reasons you need to be VERY worried about letting your users see emails from spoofed addresses pretending to be your Trusted Senders.  
Tim Uzzanti CEO SmarterTools Inc. www.smartertools.com
0
ActorMike Replied
Tim- Thanks for the explanation; I see your point.

Reply to Thread