[8125] Exception Getting DKIM signature

Problem reported by Martin Schaible - 4/8/2022 at 10:32 AM

Known

Hello
I see a lot of these errors in the devliery log:

04:27:28.592 [77881088] Exception getting DKIM signature System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei SmarterTools.MailSigning.DKIM.DKIM.<GetWorkingPemStringFromCurrentPemString>g__ExportPrivateKeyPemStr|16_0(RSACryptoServiceProvider csp)
   bei SmarterTools.MailSigning.DKIM.DKIM.GetWorkingPemStringFromCurrentPemString(String pemk)
   bei MailService.RelayServer.RemoteDeliverySession.GetDkimSignature()

Then i checked about 10 domains for the DKIM settings -> Email Signing -> View Record.
Any previous data is gone. I get the default Text Record value only:

v=DKIM1; k=rsa; h=sha256; p=

Then i opened the settings, only the cancel button is enabled

What happend here?
Cheers

Martin

26 Replies

Reply to Thread

Sébastien Riccio Replied

4/9/2022 at 4:52 AM

Hello Martin,

Your post got me curious and I've checked our delivery log aswell:

It's flooded with the same error :(

[2022.04.09] 00:00:17.353 [59454870] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.
[2022.04.09] 00:00:17.353 [59454860] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.
[2022.04.09] 00:00:17.353 [59454850] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.
[2022.04.09] 00:00:17.353 [59454840] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.
[2022.04.09] 00:00:17.353 [59454872] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.
[2022.04.09] 00:00:17.353 [59454862] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.
[2022.04.09] 00:00:17.353 [59454852] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.

Looking deeper at one of the occurence, I can see it happen when our customers are sending mails.

[2022.04.09] 00:00:17.353 [59454852] Sending remote mail from somelocal@mailbox.ch
[2022.04.09] 00:00:17.353 [59454852] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.

Sending a mail to check-auth@verifier.port25.com from one of the affected domain shows the messages aren't signed with DKIM anymore and that would explain the different support case from our customers about delivery issues.

==========================================================
Summary of Results
==========================================================
SPF check: pass
"iprev" check: pass
DKIM check: none

I've checked the DKIM configuration of one of the customer domain triggering the error and it's same as you:

Many domains are affected but not all domains, for example this one seems ok:

Now... Checking settings.json for a bogus domain, shows that the keys are gone from the files

as opposed to a DKIM working domain

What the ... How are we recovering from this situation ?

- Generating new keys ?

We don't have the rights on every domain name to apply changes at DNS level so they match

- Recover the keys from archived settings.json ?

We would need to check all domains (5000+) to evaluate which are affected then restore only the dkim keys part and re-inject them into the live settings.json. This is gonna be a huge work.

Great, my weekend is now trashed...

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

4/9/2022 at 5:11 AM

So a bit of follow up:

For tessting recovery, I handled manually a bogus domain dkim keys restoration :

1) Grab keys from the domain archived settings.json files (keys were present there)

2) Injected the keys in the domain current settings.json

3) Reloaded domain

At this point the key is displayed again when you go to " DKIM settings -> Email Signing -> View Record ".

BUT

As soon as you send a mail with the domain the keys are gone again from settings.json !

Like something trigger a rewrite of the keys in settings.json and fails to do so (not for every domains though...)

So it looks at this point it's useless to recover every affected domain as it gets trashed again right away.

Houston, we are in deep s*1t.

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

4/9/2022 at 5:19 AM

Additional infos:

This doesn't seem to be related to latest build but should originate from a change in an older build.

Checking the logs back until the oldest log we have on the system, the issue was already present 10th march 2022...

[2022.03.10] 00:00:16.885 [55571132] Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.

I unfortunately don't have older logs to be sure when it started. Maybe do you have older logs available, Martin ?

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

4/9/2022 at 5:39 AM

Okay okay...

I found some interresting log entries in "error log" for the domains I've manually fixed and right after sending a mail from these domains:

[2022.04.09] 14:00:54.547 Performed DKIM private/public key fix for somedomain.ch
[2022.04.09] 14:02:42.834 Performed DKIM private/public key fix for someotherdomain.ch

Looks like SmarterMail introduced some "key fix" attempting to do I-dunno-what to DKIM keys it thinks a fix is required. But obviously the fix result in the keys being completly removed.

This explains why after restoring the original keys, it scratches them right away again ...

So we're stuck (again) not able to do anything to recover the situation without SmarterMail fixing the ... fix.

Sébastien Riccio System & Network Admin https://swisscenter.com

Martin Schaible Replied

4/9/2022 at 5:41 AM

Bonjour Sébastien
So i am not alone with this. I pretty sure, that others are suffering from the same and then SmarterTools will have a fire in the house.

I have access to older logs and i also run a change log, which version of SmarterMail was installed on a certain day. So i can probably identify the version which caused this. I will do this later on.

Fortunately i run only a few hundred domains, but i don't want to add the DKIM data again. That will cost a lot of time.

I think, this is now an emergency.

Sébastien Riccio Replied

4/9/2022 at 7:10 AM

Hello Martin,

That would be cool if you are able to find approxmatively when it started.

It could be from Build 8097 (Mar 3, 2022) -

Fixed: Emails sent via SMTP are not signed with DKIM using the "All Fields" setting.

Or maybe even Build 8025 (Dec 21, 2021)

Fixed: DKIM signing is now working as expected.

But I hope its not as old as this... :)

Yes this is an emergency. On my side I've opened a ticket...

Kind regards

Sébastien Riccio System & Network Admin https://swisscenter.com

Martin Schaible Replied

4/9/2022 at 9:19 AM

I fixed some important domains, which solved the problems. But that isn't a solution, if you have thousands of domains.
The exception isn't a regular "error message". It is a .NET exception which points into the direction, that no error handler is there to deal with the situation. The result ist, that the mail will not be sended and the sender does not get any message, what happened. This is bad.

I don't know, how many mails where not sended, but it explains some issues.

I think it's alright to say, that i'm really pissed off.

Sébastien: I will check the logs later.

Sébastien Riccio Replied

4/9/2022 at 9:33 AM

Martin,

When you say you fixed some domains, how did you proceed ? Generating a new key or recovering previous keys ?

Were you able to successfully verify that DKIM was working again for these fixed domains ?

Yes, I think it's alright to say. I am also very pissed atm.

Thanks a lot.

Kind regards,

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

4/9/2022 at 9:52 AM

I updated my python SmarterMail integrity checker with checks for this issue...

26 domains on 5006 seems to be affected ... It seems not that much catastrophic, but still....

Sébastien Riccio System & Network Admin https://swisscenter.com

Martin Schaible Replied

4/9/2022 at 12:22 PM

I generated the keys new and updated the DNS. It was the fastest way to get some important domans back to life.

I checked around 50 Domains and 45 where afected. Not nice.

Checking now the logs. Needs some time to get RegEx'ed :-)

Martin Schaible Replied

4/9/2022 at 12:44 PM

The problem started with Build 8097. That's a month ago in my case. WTF.

What i have learned:
Do more checks after an Update.
Create a checklist for parsing logs for errors.
Create a script which feeds the monitoring system if .NET exceptions are occuring.

Martin Schaible Replied

4/13/2022 at 1:03 PM

Strange that SmarterTools does not react on this major problem.
Kinda disappointing.

Kyle Kerst Replied

4/13/2022 at 2:49 PM

Employee Post

Good afternoon, and thanks for your patience on this while we did some further analysis on the problem. We are aware that this issue is affecting a couple of environments in our current public release and apologize for the inconvenience this is causing. In these cases please remove and regenerate those affected DKIM keys and these domains should return to mail signing as expected. In the meantime, we are working on a fix that should prevent these issues arising again in the future. However, we don't believe you should run into this again once regenerated. Please don't hesitate to reach out to us in support if you run into further issues with this.

Kyle Kerst IT Coordinator SmarterTools Inc. www.smartertools.com

Douglas Foster Replied

4/17/2022 at 7:44 PM

Kyle's assertion that this is a one-time problem (for each installation) appears to be correct. My DKIM configuration had been stable for years, until it vanished during a previous upgrade. I was upset, but I proceeded with regenerating the signature to resolve the problem. Subsequently, I upgraded to build 8125 and the problem did not reappear.

Sébastien Riccio Replied

4/17/2022 at 9:26 PM

Hello Douglas,

Yes, regenerating the keys hopefully fixes the issue permanently (well at least until the next new issue with dkim...), but it's kinda annoying to have no other way for fixing it, when you don't have control over the DNS entries of the affected domains.

We have to contact each affected customer to so they publish the new public key in their domain.

Sébastien Riccio System & Network Admin https://swisscenter.com

Kyle Kerst Replied

4/18/2022 at 7:29 AM

Employee Post

Thanks for your follow-up on this guys, glad to hear that does the trick. We do understand and apologize for the inconvenience, we can see this as a pain point, specially in hybrid environments where DNS modifications might take time to implement. I can say we have some check/verification code in place at this point that should keep things in good shape going forward, so you shouldn't have to deal with it again!

Kyle Kerst IT Coordinator SmarterTools Inc. www.smartertools.com

Douglas Foster Replied

4/18/2022 at 7:42 AM

Some time back, I put in a wishlist for DKIM rollover to be less traumatic. At present, I have to break the DKIM configuration to obtain new DNS settings, but I cannot activate the new settings until SM sees that the public key has propagated to DNS. It should be possible to rollover without disruption. Two options: (a) generate new settings without breaking the old ones, then switchover as a separate step, or (b) generate settings externally with OpenSSL or equivalent, deploy them to DNS in advance, then import the settings into SM.

Not that this request would have simplified this problem, since SM is discarding the old settings. But fixing this problem reminded me of how much I dislike the current design.

Kyle Kerst Replied

4/18/2022 at 9:13 AM

Employee Post

I do believe we have a regenerate option incoming in the near future but I'll confirm that for you!

Kyle Kerst IT Coordinator SmarterTools Inc. www.smartertools.com

Manuel Martins Replied

4/27/2022 at 7:50 AM

Hi,

MXToolbox says that can't validate my DKIM Hash Signature.

What is Wrong ?

Thanks

Tony Scholz Replied

4/27/2022 at 10:12 AM

Employee Post

Hello Manuel.

According to the screen shot the body was adjusted somehow from the original signing to when/where it arrived.

This can be do to the message being adjusted by a spam filter, av scanning adding a message, etc.. Do you have any filters or AV that could be adjusting the message?

Thank you

Tony

Tony Scholz System/Network Administrator SmarterTools Inc. www.smartertools.com

Manuel Martins Replied

4/27/2022 at 10:32 AM

Hello Tony,

No, I don't think i have any Filter or AV that can change the message.

I found some post on the internet stating that MXToolBox DKIM checker as a BUG.

email - DKIM validating, but mxtoolbox reports as "DKIM-Signature Not Verified"? - Server Fault

Don't use MxToolbox to verify DKIM : msp (reddit.com)

I Tested with others Checkers and it seems to be OK!

https://dkimvalidator.com/

https://www.mail-tester.com/

Thanks Tony.

Patrick Mattson Replied

6/3/2022 at 10:02 AM

I started getting complaints about emails not coming in, they gave me a couple emails to look at.

After searching the logs I too am seeing a lot of errors related to these emails.

Exception getting DKIM signature System.NullReferenceException: Object reference not set to an instance of an object.

What happened to my DKIM record for my customers?

I was fine until I upgraded to the latest version over the weekend.

Douglas Foster Replied

6/3/2022 at 10:58 AM

You need to recreate your DKIM settings. This started quite a few versions back. Support said somewhere in this forum that after they are recreated the first time, the problem will not happen again. My experience seems to confirm that assertion, although it is frustrating that the problem happens at all. In an early version, it allowed me to pick me own scope ID. In the latest versions, SmarterMail assigns the scope IDs. I am guessing (without proof) that the new version expects a certain scope ID format, so it discards the ones that I don't recognize.

Patrick Mattson Replied

6/3/2022 at 11:53 AM

Does SmarterTools ever let me have a free weekend?

Sébastien Riccio Replied

6/4/2022 at 2:49 AM

No, not really

Sébastien Riccio System & Network Admin https://swisscenter.com

JerseyConnect Team Replied

6/7/2022 at 7:17 AM

On one hand I'm fortunate we found this thread before upgrading, so we could regenerate the DKIM sigs ahead of time. On the other hand I just sigh that DKIM was broken for months without any logging to let us know it was a problem.

Back to Community Threads

Please leave this box unchecked

26 Replies

Reply to Thread

Tags

Related Knowledge Base Articles