I am sharing some tested API code, because the API documentation is a little cryptic and this code represents a lot of trial and error.
This script walks through some or all domains to extract basic information about each user, and uses it to generates two CSV files, one for users and aliases, another for mailing lists. Coded fields are not currently converted to text, but the code has dictionaries that define the code-to-description translation for those who do not want coded output. All necessary customizations have been isolated to one highlighted location in the code.
Important details that I learned along the way, which apply to any API effort:
- You can do everything with one system admin login as long as you add these tokens to the headers:
For domain functions: "X-SmarterMailDomain" : <mydoman>
For user functions: "X-SmarterMailDomain" : <mydomain>, "X-SmarterMailUser" : <myuser>
This gimmick avoids the failure that will occur if you try to perform an API login using an account that has 2-factor authentication enabled. (System admin accounts can be locked down by IP address, so lockdown by 2FA is presumably not needed.)
- When an API call requires a dictionary object with selection criteria, all keywords must be supplied, as there are no default values. If you leave something out, the API call will blow up with an unhelpful error. Here is an example of valid selection criteria for the /domain-list-search function when all usable results are desired:
json={"skip": 0,"take": 9999,"search": "","sortField": "domainName","sortDescending": False,"includeErrored": False}
- The impersonate user function is very slow, requiring several seconds. When building this script, I wanted to extract the creation date for each account, and that data element is only available with a user-context query. If the script is run with AdditionalUserData=False, it will race through several thousand addresses in a few seconds, but it will not extract the date-created, because it only uses domain impersonation. If the script is run with AdditionalUserData=True, the same set of several thousand accounts will take hours to process, because of the delay caused by thousands of per-user impersonation calls.
If you just want to use the account list function, you can now scroll past the rest of my prose to the Python code.
However, there has been a lot of interest in creating a tool to purge bad email. That concept was on my planning horizon, but I have given up the quest. For anyone who wants to resume that quest, I provide these notes about using this script as a jumping off point:
- You will need prior knowledge of which accounts received the bad email. This probably means that you are out of luck unless you have an incoming gateway that captures message metadata into a database.
- You will also need to make assumptions about where the message is stored in the user's folder structure. Suggested folders to search are Inbox and Deleted Items. Searching every folder is not likely to be an acceptable process if the user has many gigabytes of data.
- You will need a user interface to define the selection criteria for messages that you want to purge.
- You will need to convert your message selection criteria into a SQL query against your message metadata. This will return the user accounts to be inspected.
- You will also need to convert your message selection criteria into the dictionary object used in the API for selecting messages from the user's mailbox.
- Next, you will rewrite my per-user information query to become a per-user mailbox search query.
- For each message selected by the API, you will probably want a user interface to verify that you are about to delete a problem message and not deleting an acceptable message by accident.
- Finally, delete the message. When manually impersonating a user, I have to delete a message once to move it from Inbox to Deleted Items, then again to delete it permanently. I have not checked the API to determine whether a one-step purge is possible or not.
Given all of this complexity, I decided that manual impersonation was a sufficient tool for me.
Here is the code for extracting account information:
*********************************************************************
import re
import sys
from functools import reduce
import requests
import pyodbc
import os
from datetime import datetime
def datetextfix(datetext):
datetext = datetext.strip()
if datetext[0:10] == '0001-01-01':
return ''
elif len(datetext) >= 19:
return datetext[0:10] + ' ' + datetext[12:7]
elif len(datetext) >= 10:
return datetext[0:10]
else:
return datetext
def checkexpired(timetext):
rslt = False
if timetext[0] > '0':
try:
x = re.search('\\d{4}\\-\\d{1,2}\\-\\d{1,2}',timetext, re.IGNORECASE)
if x: # strip off fractions of a second
nowtime = datetime.now(timezone.utc)
x1a = x.span()[0]
x2a = x.span()[1]
datestring = timetext[x1a:x2a].strip()
timestring = timetext[x2a+1:]
y = re.search('\\d{1,2}\\:\\d{1,2}\\:\\d{1,2}',timestring, re.IGNORECASE)
if y:
y1a = y.span()[0]
y2a = y.span()[1]
timestring = timestring[y1a:y2a].strip()
datevalue = date.fromisoformat(datestring)
timevalue = time.fromisoformat(timestring)
newvalue = datetime.combine(datevalue,timevalue,timezone.utc)
if newvalue < nowtime:
rslt = True
else:
rslt = False
except:
rslt = False
return rslt
def authenticate_smartermail(loginaccount: str,password: str):
# If the token is already cached, return it.
admintoken = []
res = requests.post(
url=serveruri + "auth/authenticate-user",
json={
"username": loginaccount,
"password": password,
},
)
res_json = res.json()
if res.status_code < 200 or 299 < res.status_code:
raise Exception(res_json)
else:
admintoken["access_token"] = res_json["accessToken"]
admintoken["refresh_token"] = res_json["refreshToken"]
admintoken["access_token_expires"] = res_json["accessTokenExpiration"]
admintoken["refresh_token_expires"] = res_json["refreshTokenExpiration"]
return admintoken
# Bottom of authenticate_smartermail() function
class AuthToken:
def __init__(self):
self.uri = serveruri
self.AccessToken = None
self.TokenRefresh = None
self.TokenExpiration = None
self.RefreshExpiration = None
self.Headers = ''
self.username = ''
self.password = ''
self.ImpersonateUsername = None
self.ImpersonateAccessToken = None
self.ImpersonateTokenRefresh = None
self.ImpersonateTokenExpiration = None
self.ImpersonateRefreshExpiration = None
self.ImpersonateHeaders = ''
self.ShowAuthData = False
self.impersonateObject = None
def login_user(self,username,password):
self.admintoken = []
self.username = username
self.password = password
res = requests.post(
url=f"{self.uri}/auth/authenticate-user",
json={
"username": username,
"password": password
},
)
res_json = res.json()
if res.status_code < 200 or 299 < res.status_code:
raise Exception(res_json)
else:
self.AccessToken = res_json["accessToken"]
self.TokenRefresh = res_json["refreshToken"]
self.TokenExpiration = res_json["accessTokenExpiration"]
self.RefreshExpiration = res_json["refreshTokenExpiration"]
self.Headers = {'Authorization': f"Bearer {self.AccessToken}"}
self.Headers = {}
return res.status_code
# End of login_user method
def impersonate_user(self, user_email):
atchar = user_email.find("@")
thisdomain = user_email[atchar+1:]
uri = f"{self.uri}/settings/domain/impersonate-user/"
payload = {"email": user_email}
self.Headers = {'Authorization': f"Bearer {self.AccessToken}", "X-SmarterMailDomain" : thisdomain}
# print("impersonating",user_email)
response = requests.post(uri, headers=self.Headers, json=payload)
if response.status_code < 200 or 299 < response.status_code:
raise Exception(response.json())
else:
impersonate_data = response.json()
# Update the API class instance with impersonation tokens and expiration
self.ImpersonateUsername = user_email
self.ImpersonateAccessToken = impersonate_data['impersonateAccessToken']
self.ImpersonateTokenRefresh = impersonate_data['impersonateRefreshToken']
self.ImpersonateTokenExpiration = impersonate_data.get('impersonateAccessTokenExpiration')
self.ImpersonateRefreshExpiration = impersonate_data.get('impersonateRefreshTokenExpiration')
self.ImpersonateHeaders = {'Authorization': f"Bearer {self.ImpersonateAccessToken}"}
self.impersonateObject = impersonate_data # Store the full response if needed
# if self.ShowAuthData:
# print("Impersonate Data:", impersonate_data) # Show the auth data if required
return response.status_code
# End of impersonate_user method
# End of AuthToken class definition
AccountStatus = {
0:'Undefined',
1:'Enabled',
2:'DisabledAllowMail',
3:'DisabledDisallowMail',
4:'CriticallyErrored'
}
AccountTypes = {
0:"Undefined",
1:"User",
2:"DomainAdmin",
3:"PrimaryDomainAdmin",
4:"Alias",
5:"MailingList"
}
ListStatuses = {
1:'Enabled',
2:'Disabled',
3:'CriticallyErrored'
}
ListPosting = {
0:"Anyone",
1:"Subscribers",
2:"Moderator",
}
DisabledAccounts = 0
OverQuotaAccounts = 0
InvalidAccounts = 0
LoginFailures = 0
BadAddresses = 0
ExternalDomains = 0
AliasCount = 0
ExtraAddresses = 0
rsltcount = 0
# ******* Edit these parameters *******************************
servername = 'smartermail.example.com' # Server to query
adminusername = 'admin' # System admin account
adminpassword = 'password' # System admin password
userlistfile = "userlist.csv" # User and domain details filename
mailinglistfile = "mailinglist.csv" # Mailing list details
FetchAllDomains = True # If True, process all domains. if false, use mydomainlist instead
AdditionalUserData=False
mydomainlist = [ # Domains to process if FetchAllDomains is false
'mydommain1.com', # Also used if domain list fetch fails for any reason
'mydomain2.com',
'mydomain3.com'
]
# *************************************************************
serveruri = "https://" + servername + "/api/v1"
admintoken = AuthToken()
loginresult = admintoken.login_user(adminusername,adminpassword)
if FetchAllDomains == True:
print("retrieving domain list")
domainlist = []
thisurl = admintoken.uri + "/settings/sysadmin/domain-list-search"
domainresponse = requests.post(
url=thisurl
, headers={"Authorization": "Bearer " + admintoken.AccessToken}
, json={"skip": 0,"take": 9999,"search": "","sortField": "domainName","sortDescending": False,"includeErrored": False}
)
if domainresponse.status_code >= 200 and domainresponse.status_code <=299 :
domain_data = domainresponse.json()
domainresults = domain_data['results']
for itm in domainresults:
print(itm["name"])
domainlist.append(itm["name"])
else:
print(domainresponse)
print("domain list not retrieved")
# end: domain list fetch block
if FetchAllDomains == False:
domainlist = mydomainlist
#
outfile = open(userlistfile,'w')
outfile.write("domain,userName,displayName,status,accountType,authType,isWebmailEnabled,lastLoginProtocol,createDate,lastLoginTime,isEasEnabled,easLastLogin,isMapiEwsEnabled,mapiLastLogin,ewsLastLogin,isPopEnabled,popLastLogin,isImapEnabled,imapLastLogin,smtpLastLogin,aliasTargetCount,aliasIsCatchAll,aliasIncludeAllDomainUsers,showInGAL,bytesUsed,bytesAllowed,bytesUsedPercent,acceptedNewestPolicy,description,aliasInternalOnly,aliasShowAsRoom,aliasSendFrom\r")
mlfile = open(mailinglistfile,'w')
mlfile.write("ListAddress,ListID,Status,Disabled,Subscribers\r")
# blanktime = "0001-01-01T00:00:00"
blanktime = ""
# zerotime = datetime.fromisoformat('1970-01-01T00:00:00Z')
zerotime = ""
for thisdomain in domainlist:
# Begin User and Alias list extract
listresponse = requests.post(
url=admintoken.uri + "/settings/domain/account-list-search",
headers={
"Authorization": "Bearer " + admintoken.AccessToken, "X-SmarterMailDomain" : thisdomain
},
json = {"skip":0,"take":3000,"search":None,"sortfield" : "userName", "sortDescending": False, "searchFlags": ["users",'aliases']}
)
if listresponse.status_code >= 200 and listresponse.status_code <=299 :
response_data = listresponse.json()
results = response_data['results']
UserTotal = response_data['totalCount']
UserCount = 0
for rsltdict in results:
UserCount = UserCount + 1
if 'userName' in rsltdict: acct_userName = rsltdict['userName']
else: acct_userName = '<none>'
if 'displayName' in rsltdict: acct_displayName = rsltdict['displayName']
else: acct_displayName = None
if 'status' in rsltdict: acct_status = rsltdict['status']
else: acct_status = -1
if 'accountType' in rsltdict: acct_accountType = rsltdict['accountType']
else: acct_accountType = -1
if 'authType' in rsltdict: acct_authType = rsltdict['authType']
else: acct_authType = -1
if 'isEasEnabled' in rsltdict: acct_isEasEnabled = rsltdict['isEasEnabled']
else: acct_isEasEnabled = False
if 'isMapiEwsEnabled' in rsltdict: acct_isMapiEwsEnabled = rsltdict['isMapiEwsEnabled']
else: acct_isMapiEwsEnabled = False
if 'lastLoginTime' in rsltdict: acct_lastLoginTime = rsltdict['lastLoginTime']
else: acct_lastLoginTime = blanktime
if 'lastLoginProtocol' in rsltdict: acct_lastLoginProtocol = rsltdict['lastLoginProtocol']
else: acct_lastLoginProtocol = ''
if 'isWebmailEnabled' in rsltdict: acct_isWebmailEnabled = rsltdict['isWebmailEnabled']
else: acct_isWebmailEnabled = False
if 'easLastLogin' in rsltdict: acct_easLastLogin = rsltdict['easLastLogin']
else: acct_easLastLogin = blanktime
if 'mapiLastLogin' in rsltdict: acct_mapiLastLogin = rsltdict['mapiLastLogin']
else: acct_mapiLastLogin = blanktime
if 'ewsLastLogin' in rsltdict: acct_ewsLastLogin = rsltdict['ewsLastLogin']
else: acct_ewsLastLogin = blanktime
if 'isPopEnabled' in rsltdict: acct_isPopEnabled = rsltdict['isPopEnabled']
else: acct_isPopEnabled = False
if 'popLastLogin' in rsltdict: acct_popLastLogin = rsltdict['popLastLogin']
else: acct_popLastLogin = blanktime
if 'isImapEnabled' in rsltdict: acct_isImapEnabled = rsltdict['isImapEnabled']
else: acct_isImapEnabled = False
if 'imapLastLogin' in rsltdict: acct_imapLastLogin = rsltdict['imapLastLogin']
else: acct_imapLastLogin = blanktime
if 'smtpLastLogin' in rsltdict: acct_smtpLastLogin = rsltdict['smtpLastLogin']
else: acct_smtpLastLogin = blanktime
if 'aliasTargetCount' in rsltdict: acct_aliasTargetCount = rsltdict['aliasTargetCount']
else: acct_aliasTargetCount = 0
if 'aliasIsCatchAll' in rsltdict: acct_aliasIsCatchAll = rsltdict['aliasIsCatchAll']
else: acct_aliasIsCatchAll = False
if 'aliasIncludeAllDomainUsers' in rsltdict: acct_aliasIncludeAllDomainUsers = rsltdict['aliasIncludeAllDomainUsers']
else: acct_aliasIncludeAllDomainUsers = False
if 'showInGAL' in rsltdict: acct_showInGAL = rsltdict['showInGAL']
else: acct_showInGAL = False
if 'bytesUsed' in rsltdict: acct_bytesUsed = rsltdict['bytesUsed']
else: acct_bytesUsed = -1
if 'bytesAllowed' in rsltdict: acct_bytesAllowed = rsltdict['bytesAllowed']
else: acct_bytesAllowed = -1
if 'bytesUsedPercent' in rsltdict: acct_bytesUsedPercent = rsltdict['bytesUsedPercent']
else: acct_bytesUsedPercent = -1
if 'acceptedNewestPolicy' in rsltdict: acct_acceptedNewestPolicy = rsltdict['acceptedNewestPolicy']
else: acct_acceptedNewestPolicy = False
if 'description' in rsltdict: acct_description = rsltdict['description']
else: acct_description = ''
if 'aliasInternalOnly' in rsltdict: acct_aliasInternalOnly = rsltdict['aliasInternalOnly']
else: acct_aliasInternalOnly = False
if 'aliasShowAsRoom' in rsltdict: acct_aliasShowAsRoom = rsltdict['aliasShowAsRoom']
else: acct_aliasShowAsRoom = False
if 'aliasSendFrom' in rsltdict: acct_aliasSendFrom = rsltdict['aliasSendFrom']
else: acct_aliasSendFrom = ''
if acct_isEasEnabled == None: acct_isEasEnabled = False
if acct_isMapiEwsEnabled == None: acct_isMapiEwsEnabled = False
if acct_isWebmailEnabled == None: acct_isWebmailEnabled = False
if acct_isPopEnabled == None: acct_isPopEnabled = False
if acct_isImapEnabled == None: acct_isImapEnabled = False
if acct_easLastLogin == None: acct_easLastLogin = zerotime
if acct_ewsLastLogin == None: acct_ewsLastLogin = zerotime
if acct_popLastLogin == None: acct_popLastLogin = zerotime
if acct_mapiLastLogin == None: acct_mapiLastLogin = zerotime
if acct_imapLastLogin == None: acct_imapLastLogin = zerotime
if acct_smtpLastLogin == None: acct_smtpLastLogin = zerotime
if acct_lastLoginTime == None: acct_lastLoginTime = zerotime
# if acct_easLastLogin == blanktime: acct_easLastLogin = zerotime
# if acct_ewsLastLogin == blanktime: acct_ewsLastLogin = zerotime
# if acct_popLastLogin == blanktime: acct_popLastLogin = zerotime
# if acct_mapiLastLogin == blanktime: acct_mapiLastLogin = zerotime
# if acct_imapLastLogin == blanktime: acct_imapLastLogin = zerotime
# if acct_lastLoginTime == blanktime: acct_lastLoginTime = zerotime
# acct_easLastLogin_text = f"{acct_easLastLogin : %Y-%m-%d %H:%M:%S}"
# acct_ewsLastLogin_text = f"{acct_ewsLastLogin : %Y-%m-%d %H:%M:%S}"
# acct_popLastLogin_text = f"{acct_popLastLogin : %Y-%m-%d %H:%M:%S}"
# acct_mapiLastLogin_text = f"{acct_mapiLastLogin : %Y-%m-%d %H:%M:%S}"
# acct_imapLastLogin_text = f"{acct_imapLastLogin : %Y-%m-%d %H:%M:%S}"
# acct_smtpLastLogin_text = f"{acct_smtpLastLogin : %Y-%m-%d %H:%M:%S}"
# acct_lastLoginTime_text = f"{acct_lastLoginTime : %Y-%m-%d %H:%M:%S}"
if acct_userName == None: acct_userName =''
if acct_displayName == None: acct_displayName = ''
if acct_lastLoginProtocol == None: acct_lastLoginProtocol =''
if acct_description == None: acct_description =''
if acct_status == None: acct_status =-1
if acct_accountType == None: acct_accountType =-1
if acct_authType == None: acct_authType =-1
if acct_aliasTargetCount == None: acct_aliasTargetCount =-1
if acct_bytesUsed == None: acct_bytesUsed =-1
if acct_bytesAllowed == None: acct_bytesAllowed =-1
if acct_bytesUsedPercent == None: acct_bytesUsedPercent =0
if acct_aliasIncludeAllDomainUsers == None: acct_aliasIncludeAllDomainUsers = False
if acct_showInGAL == None: acct_showInGAL = False
if acct_acceptedNewestPolicy == None: acct_acceptedNewestPolicy = False
if acct_aliasInternalOnly == None: acct_aliasInternalOnly = False
if acct_aliasShowAsRoom == None: acct_aliasShowAsRoom = False
if acct_aliasSendFrom == None: acct_aliasSendFrom = False
if acct_aliasIsCatchAll == None: acct_aliasIsCatchAll = False
addresscount = 1
if acct_accountType == 4:
addresscount = rsltdict['aliasTargetCount']
acctstat = 1
if acct_status == 1:
userenabled = True
else:
userenabled = False
rsltdict['createDate'] = zerotime
if acct_accountType==4:
if rsltdict['aliasInternalOnly']==1:
userenabled = False
elif AdditionalUserData == True :
if admintoken.accessTokenExpiration <= datetime.now():
admintoken = AuthToken()
loginresult = admintoken.login_user(adminusername,adminpassword)
getheaders = {"Authorization": "Bearer " + admintoken.AccessToken, "X-SmarterMailDomain" : thisdomain, "X-SmarterMailUser" : rsltdict['userName'] + '@' + thisdomain}
userresponse = requests.get(
url=admintoken.uri + "/settings/user?" + rsltdict['userName'] + '@' + thisdomain,
headers=getheaders
)
if userresponse.status_code >= 200 and userresponse.status_code <=299 :
userresults = userresponse.json()
userresdata = userresults["userData"]
# print(userresdata)
if 'createDate' in userresdata: rsltdict['createDate' ] = userresdata['createDate']
else:
print("could not obtain create date",acct_userName)
if rsltdict['createDate'] == None: rsltdict['createDate'] = zerotime
if rsltdict['createDate'] == blanktime: rsltdict['createDate'] = zerotime
acct_createTime = rsltdict['createDate']
print('address:',rsltdict['userName'] + '@' + thisdomain,"enabled",userenabled,"type",acct_accountType,"addresscount",addresscount)
outfile.write(thisdomain + ',' + acct_userName +
',"' + acct_displayName + '"' +
',' + str( acct_status )+
',' + str( acct_accountType ) +
',' + str( acct_authType ) +
',' + str( acct_isWebmailEnabled ) +
',' + str(acct_lastLoginProtocol) +
',' + datetextfix( acct_createTime )+
',' + datetextfix( acct_lastLoginTime ) +
',' + str( acct_isEasEnabled )+
',' + datetextfix( acct_easLastLogin ) +
',' + str( acct_isMapiEwsEnabled ) +
',' + datetextfix( acct_mapiLastLogin ) +
',' + datetextfix( acct_ewsLastLogin ) +
',' + str( acct_isPopEnabled ) +
',' + datetextfix( acct_popLastLogin ) +
',' + str( acct_isImapEnabled ) +
',' + datetextfix( acct_imapLastLogin ) +
',' + datetextfix( acct_smtpLastLogin )+
',' + str( acct_aliasTargetCount ) +
',' + str( acct_aliasIsCatchAll ) +
',' + str( acct_aliasIncludeAllDomainUsers ) +
',' + str( acct_showInGAL ) +
',' + str( acct_bytesUsed )+
',' + str( acct_bytesAllowed ) +
',' + str( acct_bytesUsedPercent ) +
',' + str( acct_acceptedNewestPolicy )+
',"' + acct_description + '"' +
',' + str( acct_aliasInternalOnly ) +
',' + str( acct_aliasShowAsRoom ) +
',' + str( acct_aliasSendFrom )+ '\r'
)
if UserTotal == UserCount:
pass
# print(thisdomain,"--user total",UserTotal,"matches user count",UserCount)
else:
# print(thisdomain,"--error user total",UserTotal,"does not match user count",UserCount)
LoginFailures = LoginFailures + 1
# End: Perfrom user lookup
else:
# print("get user return error:",response.status_code)
LoginFailures = LoginFailures + 1
# End User and Alias list extract
# Begin Mailing List extract
print("retrieving mailing lists for",thisdomain)
mlresponse = requests.get(
url=admintoken.uri + "/settings/domain/mailing-lists/list",
headers={
"Authorization": "Bearer " + admintoken.AccessToken
, "X-SmarterMailDomain" : thisdomain
},
)
if mlresponse.status_code >= 200 and mlresponse.status_code < 299:
response_data = mlresponse.json()
# print(mlresponse.status_code,response_data)
results = response_data['items']
ListCount = 0
for rsltdict in results:
ListCount = ListCount + 1
listid = rsltdict['id']
liststatus = rsltdict['status']
listsubs = rsltdict['listSubscriberCount']
lostposters = rsltdict['digestSubscriberCount']
listpostercount = rsltdict['posterCount']
listbannedcount = rsltdict['bannedUserCount']
listaddress = rsltdict['listAddress']
listmoderator = rsltdict['moderatorAddress']
listperms = rsltdict['postingPermissions']
listdisabled = rsltdict['disabled']
if liststatus == 1 or listdisabled == False:
listenabled = 1
else:
listenabled = 0
print('list',listaddress+'@'+thisdomain,listid,liststatus,listdisabled,listsubs)
mlfile.write(listaddress+'@'+thisdomain+","+str(listid)+","+str(liststatus)+","+str(listdisabled)+","+str(listsubs)+"\r")
print(thisdomain,"mailing list count",ListCount)
else:
print("mailing list return error:",mlresponse.status_code)
LoginFailures = LoginFailures + 1
# End Mailing List extract
# Bottom of Domain loop
if LoginFailures == 0:
print("No errors.")
else:
print("Failure count:", LoginFailures)
outfile.close()
mlfile.close()
exit(LoginFailures)