SM 8495: alarm "Your system clock is more than 15 seconds off" even if my server is in perfect sync

Problem reported by Gabriele Maoret - SERSIS - 4/5/2023 at 1:49 PM

Resolved

First issue with the new version: we continuosly get alarm "Your system clock is more than 15 seconds off":

BUT the server is in perfect sync with POOL.NTP.ORG, as you can see here:

It seems that SmarterMail consider UTC time, but we are in Italy... so we are in CET (Cnetral EEuropean Time - UTC+1) in autumn/winter, and NOW we are in CEST (Central European Summer Time - UTC+2), the light saving time that we use in spring/summer.

Is there perhaps a setting somewhere to tell SmarterMail what timezone to consider?

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

44 Replies

Reply to Thread

Kyle Kerst Replied

4/5/2023 at 1:52 PM

Employee Post

Hi Gabriele! The server should use the OS time and timezone settings, so you should not need to set anything manually. I'm going to do some testing on this scenario and I'll let you know what I find out.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Kyle Kerst Replied

4/5/2023 at 2:10 PM

Employee Post

I ran through this under our last dev build and the current release build (to match your testing) but was not able to replicate in the what I believe is the same configuration. Below you'll see the results of the powershell command Get-TimeZone, and the NTP query you ran as well:

The offset on the NTP results looks to be about the same as well, but I don't see this chronic warning you are getting. On that note, I believe Windows uses just the one timezone for CET now and just offsets for daylight savings time. Can you check the Get-TimeZone command results and post here too? Per this list below you are in the W. Europe Standard Time default timezone in Windows:

https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/default-time-zones?view=windows-11

I had no issues testing under that timezone either. Perhaps we have a locale issue in Windows as well - I can test that next.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Gabriele Maoret - SERSIS Replied

4/5/2023 at 2:14 PM

This is the result in my server:

Per Microsoft indication, we are in W. Europe Standard Time , maybe not exactly the strictly same of CET:

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

Gabriele Maoret - SERSIS Replied

4/5/2023 at 2:27 PM

Strange: the alarm has now stopped appearing...

I don't know...

I'll let you know if it happens again...

Kyle Kerst Replied

4/5/2023 at 2:59 PM

Employee Post

Interesting, I wonder if the NTP results were temporarily askew and resolved themselves at some point. Do you see any NTP related errors or warnings in the Event Viewer in Windows?

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Brian Bjerring-Jensen Replied

4/19/2023 at 3:24 PM

Currently seeing the same thing right after we log in.... Build 8504.

It started appearing today.

Kyle Kerst Replied

4/19/2023 at 4:43 PM

Employee Post

I recommend checking your system time against a valid NTP server from Microsoft, as well as your timezone and other settings. We were not able to reproduce this issue here, and have had no tickets submitted on this.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Vahn Babigian Replied

4/19/2023 at 4:45 PM

Kyle, remember when you where working on another ticket for me you brought the 15 second notification to my attention. After re-syncing with the timeserver I still got the message. I had to reboot the server to make it go away. The question to ask is why we never got this message prior to the upgrade to build 8495.

Brian Bjerring-Jensen Replied

4/20/2023 at 12:09 AM

I am getting this constantly now. Its syncing towards 0.dk.pool.ntp.org and no issues there.

This is frustrating... to say the least.

Brian Bjerring-Jensen Replied

4/20/2023 at 1:18 AM

this could be related to the MAPI issues since when I sync the time and restart outlook, then messages are not stuck in outbox.

Brian Bjerring-Jensen Replied

4/20/2023 at 2:12 AM

I created a task in task scheduler to run every minute syncing the clock on the server.

cmd.exe w32tm /resync /rediscover

And it hasnt given an error message since.

echoDreamz Replied

4/20/2023 at 1:02 PM

We stopped using the Windows time service a while back, always had issues with it and servers being out 20-30 seconds. Switched to using the Meinberg NTP client with one of their LANTIME rackmount servers and all servers have been extremely accurate for ~10 years.

Brian Bjerring-Jensen Replied

4/20/2023 at 11:35 PM

8504 is throwing the error despite clocks in perfect sync.(at least here).

Gabriele Maoret - SERSIS Replied

4/21/2023 at 12:18 AM

It hasn't happened to me since the first initial alarms...

It stopped after I synced the server to POOL.NTP.ORG and then restarted the server

Brian Bjerring-Jensen Replied

4/22/2023 at 9:50 AM

Havent had any alarms yet. But running background sync via task scheduler every minute....to avoid it.

Brian Bjerring-Jensen Replied

4/22/2023 at 1:56 PM

Yup. I'll report back

Brian Bjerring-Jensen Replied

4/23/2023 at 8:24 AM

HAvent seen it since upgrade to 8510. I will keep an eye out for it though since I believe some of the MAPI functionality is relying on a stable NTP function.

Brian Bjerring-Jensen Replied

5/12/2023 at 9:55 AM

8531 just updated getting the error again.

echoDreamz Replied

6/4/2023 at 10:27 PM

UPDATE.. restarted the SM service and the issue went away.

Just updated to 8552, logged in and instantly received the 15 seconds off notice. Our mail server is extremely accurate (as are all servers on our network, as we have a local GPS-sync'd time server).

Your time is exact!

The difference from Time.is was +0.000 seconds (±0.029 seconds)

My desktop is also...

Your time is exact!

The difference from Time.is was +0.000 seconds (±0.021 seconds).

Brian Bjerring-Jensen Replied

6/5/2023 at 1:59 AM

We havent seen it since 8531. Currently on 8552.

Manuel Martins Replied

7/6/2023 at 4:59 AM

Hi,

We are now using Build 8580 and getting this Error also!!

We use NetTime aplication Service and our Server's Time is Perfectly Sync. We are on UTC + 0:00 timezone.

echoDreamz Replied

7/13/2023 at 9:48 PM

This has came back after we did the 8594 upgrade... And again, our server time is highly accurate. We restarted the server, restarted the service 3 times, finally on the 3rd service restart the notice went away and everything started syncing properly.

Something is 100% wrong with SM and this notice.

Your time is exact!

The difference from Time.is was -0.001 seconds (±0.030 seconds).

Gabriele Maoret - SERSIS Replied

7/14/2023 at 12:32 AM

This is a strange error and there is probably something to check at the server (or, maybe, client?) level...

It never happened to me after the first alarms, and I did all the updates version by version...

Now I'm on the latest 8594 and still everything works fine...

Brian Bjerring-Jensen Replied

7/14/2023 at 2:25 AM

We havent gotten them either...

Have you checked if windows time is running?

echoDreamz Replied

7/14/2023 at 8:59 AM

We do not use the Windows Time service, never been a fan of it and had issues with it in the past, We use https://www.meinbergglobal.com/english/sw/ntp.htm which is from the same people who make our local time server. Regardless, the time on the server is exact, always is, always has been. https://www.time.gov/ shows our server time is off by +0.001s - Which is not +/- 15 seconds, not even close.

J. LaDow Replied

7/14/2023 at 11:14 AM

We were on the latest version for a week before the notifications started appearing.

We're also in perfect sync.

They've been becoming more frequent.

As our server load increased with adding a couple domains, that's when they started. The notice appears randomly. Like something in SM's check is affected by delay in CPU load in a return function or something along those lines.

We're not a large installation -- so far, 180 users (of which 130 are actual users, the rest are domain admin accounts that will never be used) across 50 domains - and the notifications didn't start appearing until about 2 days ago after the server was rebooted from the latest MS updates. We don't think it's the updates. The reboot flushed something out in SM and when it re-loaded that's when they started.

We will be making some changes and rebooting tonight - will report back.

It does appear that ANY resolution you attempt to implement requires a full server reboot in order to see if the changes/fix is picked up by SmarterMail -- a service restart and IIS restart does not cover it.

MailEnable survivor / convert --

J. LaDow Replied

7/15/2023 at 12:09 PM

FOLLOW-UP:

After switching to NIST time servers in Windows settings, and a full reboot of the server, this issue has not reappeared yet. Will update if this changes. Currently on v100.0.8580.22538 (Jun 29, 2023)

MailEnable survivor / convert --

echoDreamz Replied

7/15/2023 at 5:10 PM

Glad a reboot worked for you J. We did a reboot and SM started and still complained, took a reboot and 3 restarts of the service to finally get SM started without the warning. Something is definitely wrong somewhere with how SM is tracking time.

If SM is querying a time source and using it to compare against the local server time, could be there an issue with the source SM is getting time from?

J. LaDow Replied

7/15/2023 at 5:32 PM

Our reboot kept the notification quiet for about 12 hours.

Now it's back.

MailEnable survivor / convert --

echoDreamz Replied

7/15/2023 at 5:47 PM

Taking a little peek using DotPeek, I see SM has a function called GetAveragedUtcTime and is using NTP to pull a remote time using pool.ntp.org, time.nist.gov and time.google.com.

The NTP pool is terrible, heard way too many horror stories about it and systems getting poor time from it, also, Google smears time, so probably not great to mix them with non-smearing sources?

I am assuming the issue is coming from a poor response from pool.ntp.org? Would it be possible for ST to allow us to set our own NTP servers to pull time from?

echoDreamz Replied

7/15/2023 at 8:49 PM

Wrote up a quick C# app that queries pool.ntp.org every 1 minute, it's been running now for about 2 hours, had it write out a message containing the returned UTC time and the current UTC time whenever the time received was +- 5 seconds from the current system time (which is currently +0.021s out).

After about 2 hours, I have had 18 outputs each ranging from 5 seconds to 33 seconds difference, pool.ntp.org just seems to not be reliable.

I get why they are doing it, they should alert you if your system time is wrong, but it's warning when it's not and I believe it is due to pool.ntp.org, I think they should make the NTP servers configurable (if not via the interface, at least a setting in the JSON).

J. LaDow Replied

7/16/2023 at 2:37 AM

Great analysis!

MailEnable survivor / convert --

Brian Bjerring-Jensen Replied

7/16/2023 at 5:06 PM

I run a .bat file every 5 mins

@echo off
w32tm /query /peers
sc config w32time start= auto
w32tm /config /syncfromflags:manual /manualpeerlist:"time.nist.gov,0.dk.pool.ntp.org,1.dk.pool.ntp.org,2.dk.pool.ntp.org,3.dk.pool.ntp.org"
w32tm /config /reliable:yes
net stop w32time
net start w32time
w32tm /resync /nowait

For that very reason.

echoDreamz Replied

7/16/2023 at 7:29 PM

We just use the client from our NTP server manufacturer - https://www.meinbergglobal.com/english/sw/ntp.htm been using it for ~15 years, never had a single issue. No batch files etc. need, it just does it's job and it does it very well. The issue here is not with our server's time, it's with SM thinking the time is off, when it's not. We do not use the ntp pool, but SM does and I am wondering if that is where the false idea that the system's time is off.

Matt Petty Replied

7/17/2023 at 7:24 AM

Employee Post

Hmm maybe we can swap ntp pool out for Meta/Facebook. They've got a pretty robust NTP network from what I've read. Ultimately this is just a warning. We do prevent the server from running if we detect a difference of more than 12 hours and this was to prevent mixing items into the wrong days (grp files for example), ultimately the warning about 15 seconds is just a warning and besides being annoying in the corner has no effect on the system.

Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com

kevind Replied

7/17/2023 at 8:11 AM

Why not change it to 90 seconds or something to get rid of the annoying warning? That should be enough to work with most network time systems and if your message's timestamp is off by 1 minute, no big deal.

Employee Replied

8/1/2023 at 12:40 PM

Employee Post

Hi everyone,

I wanted to let you know that our public release on Thursday will have some changes in this area. We've removed pool.ntp.org from the list, so now the hard-coded options are:

time.cloudflare.com
time.google.com
time.apple.com
time.nist.gov

I hope this helps! We'll look forward to hearing your feedback when the next release is published.

Employee Replied

8/1/2023 at 12:47 PM

Employee Post

kevind,

In response to your question about changing the warning to 90 seconds...

It's my understanding that the reason we have such a small window is because some features within SmarterMail are very time sensitive, including the two-step authentication codes and auth tokens used with impersonation and logins.

I can certainly bring this up as a discussion item, though.

Kind regards,

echoDreamz Replied

8/1/2023 at 1:18 PM

I still believe the NTP servers should not be hard-coded, but at least the pool is gone, this should hopefully fix the issue.

Tim Uzzanti Replied

8/1/2023 at 1:44 PM

Employee Post

There is a way to configure it but you could screw things up really bad. Sometimes we need to keep people from harming themselves, it's not going to be documented. There are a variety of gotcha's with some of these services, some of which a handful of people are seeing.

Tim Uzzanti CEO SmarterTools Inc. www.smartertools.com

echoDreamz Replied

8/1/2023 at 5:33 PM

If it was just an entry in the main JSON file for SM, then sure, it's on us, but we run our own local NTP server, so no need to call out to others to verify time. Sorta like the Windows Registry, you can mess stuff up, but if you know what you are doing, hey great, have fun.

John C. Reid Replied

10/24/2023 at 8:16 AM

Today is October 24th 2023, and I am running SmarterMail Enterprise 100.0.8664.22089 (Sep 21, 2023.)

My server is perfectly synced with time.nist.gov, and I am still seeing this issue. In fact, I just saw it for the first time after rebooting. The points I have pulled from the thread above are:

- perhaps removing pool.ntp.org didn't fix the issue.
- time.google.com is still included even though that is mixing smeared time with non-smeared time servers.
- SmarterTools treats us like children and won't allow server administrators to control and manage their own servers.
- Often things that SmarterTools claims to have fixed will creep back in to later releases.

echoDreamz Replied

10/24/2023 at 9:52 AM

John,

Weird, i havent seen the issue since removing ntp.org... sucks it still happens for some.

- Agreed with your Google point, mixing of time sources that smear and dont smear isnt the best

- 100% agree that this should be a setting, even if it was manually in some JSON file, not the UI

- Or allow it to be disabled completely for those admins that know their time is perfect

John C. Reid Replied

10/24/2023 at 10:08 AM

@Ron Raley So are you saying that the release that "did not happen Thursday" fixed it for you when it did release?

I'm thinking maybe this is a retro bug as I have seen at least half a dozen supposably fixed issues return in later releases in the last 8 or so years we have been using SmarterMail. In fact one of those issues had to do with webmail customers not being able to save to drafts or send email, and SmarterTools kept pointing to that being a time issue also.

Back to Community Threads

Please leave this box unchecked

Reply to Thread

Enter the verification text