Let me start this off by stating that in 2020 I'm using an old version of SmarterMail (V 15.7), on Windows 2016 Server Standard. I won't go into the reasons why I'm still on an older version of SmarterMail, but I mention it because it's my hope that this issue has long since been fixed in newer, more current versions. However if it is still an issue, this may prove valuable to you.
Below is a PowerShell script I wrote by cobbling together a bunch of code snippets as well as figuring out some of these things on my own. In the comments are attributions to the URLs that I've taken code snippets from, but I may have missed some. This script runs on PowerShell version 5.1, but it may run on others.
Let me also say that this isn't a fix, but more like a really complicated workaround. However, it has worked every time I've had to use it, which is about a dozen times in the last few months. That being the case, this script is NOT perfect, I offer no guarantees, warranties or pinky swears that it will work in your environment, and I can offer no support on this. So if it breaks your server, it's your responsibility.
Enough disclaimers. :)
Here's the problem I've been experiencing. Intermittently, Cyren will start to stack up messages and eventually the server will completely stop processing incoming email. In the past the only thing I could do is restart the server completely, however, this is never desirable.
During my investigation as to why this was happening, I got to thinking about how might be able to detect this issue, and recover the mail server without rebooting.
Whenever this would happen, I would start to see error messages in the delivery log like the following:
"09:58:15 Error from Cyren: Error: Starting Scan Message thread. Exception: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start()
at CyrenClient.Program.ReceivedDataFromSmarterMail(String Data)"
First there would be a few dozen errors, but the server would still operate normally. No user complaints. Then over the course of a couple of days, the number of errors would skyrocket to hundreds or thousands, and that's when the spool would fill up with messages stuck in the "Spam Check" state (if memory serves).
Now that I recognized a pattern, I could start to address the problem.
This script is meant to be run on repetitive basis, every few minutes. In my environment I would want to run it every 1 minute, but I'm still not totally convinced it won't cause a catastrophe.
Here's an overview of what the script does:
- Writes to a log file specifically for this script
- Sends emails at specific times to alert admins to the state of the script and reporting the state of the service
- Checks to make sure the delivery log exists
- Reads through the delivery log looking for the specific Cyren error text
- If no Cyren errors are found, the script exits
- If the script finds Cyren errors, it attempts to kill the Cyren process (MailService_SubProcess)
- The script loops three times waiting for 10 seconds each time for Cyren to restart
- If after the third time Cyren doesn't restart, a critical alert email, SMS and PushOver alert is sent.
- If the process starts as expected, an email is sent stating that the service has been restarted.
- The delivery log is altered to prevent the script from picking up the prior errors and causing multiple restarts
A note on the email/messaging in this script. I use email for the non-critical alerts. For critical alerts, I use email, email to SMS and a service called PushOver, which I find is faster than the other methods, and harder to ignore. PushOver hasn't paid me anything to say this, but I wouldn't say 'no' if they did. :)
One last thing, this script won't run as-is. You will need to change a few things that will be unique to your environment. Some examples are, but not limited to:
- File Paths
- Email Addresses
- SMTP Username
- SMTP Password
- SMTP Port Numbers
The password in this script is in plain text.... I know. Ok. I know, I know, I know. No, I didn't include a real password, no ???@domain.com isn't my email address and I have a whole truckload of 'I know' over here for you, so I don't want to hear it. :)
Seriously though, I have not found a good way to store a password in PowerShell that doesn't require a DAILY manual input. So I'll make you a deal, you're allowed to scold me in the comments only if you can offer a secure solution that doesn't involve daily manual re-inputting, can exist inside this script, doesn't use third-party tools and you let me share it with anyone and everyone I choose.
Yeah. That's what I thought. :P
Here's the script:
CyrenStayAlive.ps1
# This script is designed to run twice a day off production hours.
# As of the time of this writing the times preferred are 6:00 AM and 8:15 PM
$Y = Get-Date -Format "yyyy"
$M = Get-Date -Format "MM"
$D = Get-Date -Format "dd"
$H = Get-Date -Format "HH"
$Mn = Get-Date -Format "mm"
$S = Get-Date -Format "ss"
# Test Log File
#$SMLogFile = "X:\Users\username\Desktop\2020.04.14-delivery.log"
$SMLogFile = "X:\SmarterMail\Logs\"+$Y+"."+$M+"."+$D+"-delivery.log"
$LogFile = "X:\Support\CyrenStayAlive_"+$Y+$M+$D+".log"
$global:emailbody="*** CyrenStayAlive Started @ ${H}:${Mn}:${S} on ${Y}${M}${D} ***"
# Separate email addresses with a comma
$AlertAddresses = "email@domain.com","networkadmin@domain.com"
$CriticalAlertAddresses = "?????@pomail.net","###@mms.att.net"
# Source: https://stackoverflow.com/questions/7834656/create-log-file-in-powershell
Function LogWrite {
Param ([string]$logstring)
$Stamp = (Get-Date).toString("yyyy-MM-dd @ HH:mm:ss")
$Line = "$Stamp $logstring"
Add-content $LogFile -value $Line
}
LogWrite "Start *********"
# Check to see if the log file exists. If so continue, if not, send an error.
If ([System.IO.File]::Exists($SMLogFile)) {
LogWrite "SmarterMail Log File Exists! File:" $SMLogFile
Write-Host "SmarterMail Log File Exists! File:" $SMLogFile
$global:emailbody+="SmarterMail Log File Exists! File:$SMLogFile`n"
}
ELSE {
LogWrite "SmarterMail Log File Does Not Exist! File:" $SMLogFile
Write-Host "SmarterMail Log File Does Not Exist! File:" $SMLogFile
$global:emailbody+="SmarterMail Log File Does Not Exist! File: $SMLogFile`n"
# For Production, this should be EXIT
# Exit
Break
}
# Source: https://stackoverflow.com/questions/6623433/detect-number-of-processes-running-with-the-same-name
Function numInstances([string]$process)
{
@(get-process -ea silentlycontinue $process).count
}
Function SendEmail ($strRecipient, [string]$strSubject, [string]$strBody) {
# Separate recipients with a comma
$fromaddress = $env:COMPUTERNAME+"_noreply@domain.com"
$emailserver = "smtp.domain.com"
$emailusername = "email@domain.com"
$emailpassword = ConvertTo-SecureString -Force -AsPlainText "MakeAReallyLongSecurePassword89273"
$emailcredential = New-Object System.Management.Automation.PSCredential($emailusername,$emailpassword)
# Yes. This is super insecure, however, the above method was a massive pain in the a$$. The secure string was expiring every day, which prevents the script from running on a schedule.
# If you have a better method to secure the password that doesn't expire every 24 hours, please implement it and let me know what you did.
foreach ($EmailRecipient in $strRecipient) {
try {
Write-Host "Attempting to send email...."
Write-Host "To: " $EmailRecipient
Write-Host "Subject: " $strSubject
Write-Host "Body: " $strBody
Send-MailMessage -To $EmailRecipient -From $fromaddress -Subject $strSubject -Body $strBody -Credential $emailcredential -SmtpServer $emailserver -Port 587
Write-Host "Email Succeeded!!!!"
LogWrite "Email Succeeded!!!!"
}
catch {
Write-Host "Email Attempt Failed!!!!"
LogWrite "Email Attempt Failed!!!!"
LogWrite "To: " $EmailRecipient
LogWrite "Subject: " $strSubject
LogWrite "Body: " $strBody
}
}
}
Function KillProcessAndCheck ([string]$process) {
Write-Host "Beginning KillProcessAndCheck...."
LogWrite "Beginning KillProcessAndCheck...."
$global:emailbody+="Beginning KillProcessAndCheck....`n"
Try {
Write-Host "Attempting to kill $process...."
LogWrite "Attempting to kill $process...."
$global:emailbody+="Attempting to kill $process.... ${Y}${M}${D} @ ${H}:${Mn}:${S}`n"
Stop-Process -name $process -Force
# Wait for process to stop gracefully
Write-Host "Waiting for $process to stop gracefully"
$global:emailbody+="Waiting for $process to stop gracefully`n"
Wait-Process -Name $process
}
Catch {
Write-Host "$process Kill Attempt Failed!!!!"
LogWrite "$process Kill Attempt Failed!!!!"
$global:emailbody+="$process Kill Attempt Failed!! ${Y}${M}${D} @ {H}:${Mn}:${S}`n"
SendEmail $CriticalAlertAddresses "Critical Mail Alert Cyren" $global:emailbody
SendEmail $AlertAddresses "Critical Mail Alert Cyren" $global:emailbody
# For Production, this should be EXIT
Exit
#Break
}
Write-Host "$process Killed Successfully!!!!"
LogWrite "$process Killed Successfully!!!!"
$global:emailbody+="$process Killed Successfully!!!!`n"
for ($x='' ;$x.length -le 3;$x=$x+'x'){
Write-Host $x
#Start-Sleep -Milliseconds 20
Write-Host "Waiting for process to restart Automatically"
$global:emailbody+="$x`nWaiting for process to restart Automatically`n"
Start-Sleep -s 10
$CyrenProcs = numInstances $process
If ($CyrenProcs -eq 1) {
Write-Host "$process Auto-Restart Successful!!!!"
LogWrite "$process Auto-Restart Successful!!!!"
$global:emailbody+="$process Auto-Restart Successful!!!!`n"
(get-content X:\SmarterMail\Logs\"$Y"."$M"."$D"-delivery.log) | foreach-object {$_ -replace "Error from Cyren:", "Error from Cyren+:"} | set-content X:\SmarterMail\Logs\"$Y"."$M"."$D"-delivery.log
#SendEmail $CriticalAlertAddresses "Mail Alert Cyren" $global:emailbody
SendEmail $AlertAddresses "Mail Alert Cyren" $global:emailbody
# For Production, this should be EXIT
Exit
#Break
}
}
Write-Host "$process did NOT Auto-Restart!!!!"
LogWrite "$process did NOT Auto-Restart!!!!"
$global:emailbody+="$process did NOT Auto-Restart!!!!`n"
SendEmail $CriticalAlertAddresses "Critical Mail Alert Cyren" $global:emailbody
SendEmail $AlertAddresses "Critial Mail Alert Cyren" $global:emailbody
# For Production, this should be EXIT
# Exit
Break
}
$CyrenProcess = "MailService_SubProcess"
#$CyrenProcess = "Notepad"
$CyrenProcs = numInstances $CyrenProcess
If ($CyrenProcs -eq 0) {
Write-Host "Critical Error: $process not running. Service or Server Restart may be required...."
LogWrite "Critical Error: $process not running. Service or Server Restart may be required...."
$global:emailbody+="Critical Error: $process not running. Service or Server Restart may be required....`n"
SendEmail $CriticalAlertAddresses "Critical Mail Alert Cyren" $global:emailbody
SendEmail $AlertAddresses "Critical Mail Alert Cyren" $global:emailbody
# For Production, this should be EXIT
# Exit
Break
}
$SearchString = "Error from Cyren: Error: Starting Scan Message thread. Exception: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown."
$CyrenErrors = Select-String -Path $SMLogFile -Pattern $SearchString | Measure-Object -Line | Select-Object -expand Lines
If ($CyrenErrors -eq 0) {
LogWrite "No Cyren Errors found."
Write-Host "No Cyren Errors found."
# For Production, this should be EXIT
Exit
#Break
}
ElseIf ($CyrenErrors -gt 0) {
LogWrite "Cyren Errors found:" + $CyrenErrors
Write-Host "Cyren Errors found:" $CyrenErrors
$global:emailbody+="Cyren Errors found: $CyrenErrors `n"
#SendEmail $AlertAddresses "Mail Alert Cyren" $global:emailbody
KillProcessAndCheck $CyrenProcess
# For Production, this should be EXIT
Exit
#Break
}
Else {
LogWrite "Some other thing happened."
Write-Host "Some other thing happened."
$global:emailbody+="Some other thing happened.`n"
SendEmail $CriticalAlertAddresses "Critical Mail Alert Cyren" $global:emailbody
SendEmail $AlertAddresses "Critical Mail Alert Cyren" $global:emailbody
# For Production, this should be EXIT
Exit
#Break
}