32 Replies

12/23/2024 at 12:05 PM

We update the GEO IP DB every 3 hours for that exact reason.

So should SM. Otherwise we have a lot of false positives over time in GEO IP blocking and IDS/IPS.

Sébastien Riccio Replied

12/23/2024 at 1:05 PM

GeoIP database is not updating itself regulary/automatically in SM ??

Sébastien Riccio

System & Network Admin

12/23/2024 at 1:34 PM

Only when installed.

Zach Sylvester Replied

12/23/2024 at 1:52 PM

Employee Post

Hello Everyone,

I discussed with my team the possibility of enabling SmarterMail to automatically update the GEO database. However, due to a licensing restriction with MaxMind, this is not possible.

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

https://github.com/maxmind/geoipupdate/blob/main/doc/GeoIP.conf.md

Sébastien Riccio Replied

12/23/2024 at 2:11 PM

Zach,

As far as I know, it would only require the server admin to provide a MaxMind key for the updates, which you can obtain for free by registering to their service.

At least it's how we're doing it with geoipudate on Linux.

For the one not willing to register -> use bundled databases

For the one with an account -> use the autoupdated db's

I understand people frustration being stuck with an old db because they can't upgrade smartermail (due to the new issues it will bring) only for the purpose of having fresh geo db's.

Kind regards.

Sébastien Riccio

System & Network Admin

Sébastien Riccio Replied

12/23/2024 at 2:19 PM

By the way, is that the file used for GeoIP in SM ?

If yes:

Can't we setup on our own an automatic updater for it ? Would it need to restart smartermail at every update though ?

If no:

Where are the db files located, to consider updating them by other means ?

Thanks

Sébastien Riccio

System & Network Admin

Zach Sylvester Replied

12/23/2024 at 2:37 PM

Employee Post

Hi Sebastien,

Thank you for reaching out!

You're correct—the main challenge is that users would need to register and provide their own API keys, among other requirements. Since you're open to doing that, I’ll revisit this topic with my team and gather their thoughts on it.

The other key obstacle is that we’re not using the default database; instead, we’ve implemented custom techniques to optimize lookups for better speed and efficiency. That’s why a direct replacement isn’t feasible. However, if needed, you could technically work around this by installing the latest version of SmarterMail on a new server, stopping the service on both servers and copying the GeoIP.dat file from the new version to the old one. This would allow the older version to utilize the updated database.

That said, I’ll discuss with my team the possibility of introducing a method to dynamically upgrade the GeoIP.dat file in future updates.

Let me know if you have any additional thoughts or questions!

Best regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

Gabriele Maoret - SERSIS Replied

12/23/2024 at 2:41 PM

So, if I understand correctly, as a workaround, I can use a "dummy" installation of a new release and copy this "GeoIP.dat" file from there to my working server, correct?

Can you confirm this for me, so I can try this as a workaround?

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

Zach Sylvester Replied

12/23/2024 at 2:44 PM

Employee Post

Hi Gabriele,

That’s correct—the database updates only include new information without altering the underlying format.

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

Gabriele Maoret - SERSIS Replied

12/23/2024 at 2:47 PM

Very good, I'll give it a try

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

D

David Fisher Replied

12/23/2024 at 2:57 PM

Hi,

Why is the GeoIP.dat file the same size of 1,036,433 bytes and dated Jan 31, 2019 on every build?

Even my SmarterMail v15.7 is this size and date along with build 9007 & 9119

Thanks!

-dave

Zach Sylvester Replied

12/23/2024 at 3:01 PM

Employee Post

Hi David,

I’m not entirely sure about that. Based on our release notes, the last time we upgraded the GeoIP database was with Build 9035 on September 26, 2024. The updated database is committed to our codebase, so it should definitely be different.

If you suspect it might be an older version, you can try uninstalling SmarterMail, deleting the GeoIP.dat file, and then reinstalling SmarterMail. Alternatively, you can follow the steps I mentioned earlier to replace the file manually.

Let me know if you have any questions or run into issues!

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

12/23/2024 at 3:06 PM

We have a maxmind licensekey... why not implement the option to put a key there and let us decide how often we would like a refresh.

Otherwise it wont make sense to have it there in the first place if its old data...

Zach Sylvester Replied

12/23/2024 at 3:12 PM

Employee Post

Hi Brian,

As I mentioned earlier, I’ll bring this up with the team for further discussion. The main challenge lies in managing the complexity of building out the integration while ensuring a seamless experience.

One of SmarterMail’s key advantages is minimizing the need for manual configurations, as simplicity and ease of use are central to our offering. Competing mail servers often require more manual effort, and we aim to stand out by making things as effortless as possible for our users.

I’ll share your suggestion with the team and see if we can explore a solution that aligns with our philosophy.

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

https://support.maxmind.com/hc/en-us/articles/4408216157723-Database-Formats

12/23/2024 at 3:14 PM

Thanks Zack as always :)

Merry christmas to you and the team.

Sébastien Riccio Replied

12/23/2024 at 3:23 PM

Hmmm,

Another question/suggesion. If the DB is that .dat file, it means you're using the old legacy db format.

It's being quite being deprecated in favor of MMDB files.

Also:

The MaxMind MMDB (MaxMind DB) database format offers several advantages over the legacy DAT files:

Performance

MMDB files provide significantly faster IP address lookups compared to DAT files. The MMDB format utilizes an efficient binary search tree algorithm specifically optimized for IP address queries

1

4

. This results in blazingly fast lookups, outperforming even database management systems for IP address searches.

Versatility

MMDB files support both IPv4 and IPv6 addresses, mapping them to data records within a single file

1

. This eliminates the need for separate files for different IP versions.

Efficiency

The binary format of MMDB files is more space-efficient compared to plaintext formats like CSV or JSON

4

. This allows for more compact storage of large IP datasets.

Standardization

The MMDB format has become an industry standard for IP geolocation and related datasets

4

. Many tools and libraries across different programming languages support reading MMDB files directly.

Data Integrity

As a binary format, MMDB files are less prone to accidental modifications or corruption compared to plaintext formats. This helps maintain data integrity, especially for mission-critical applications.While MMDB files excel at IP lookups, it's worth noting that they are specialized for this purpose and lack the flexibility of plaintext formats for general data manipulation or exploration

4

. However, for applications primarily focused on efficient IP address queries, the MMDB format offers clear advantages over the older DAT files.

Maybe you would not need to slightly modify the legacy format to have better performances, but switch to use the MMDB format.

Sébastien Riccio

System & Network Admin

Zach Sylvester Replied

12/23/2024 at 3:28 PM

Employee Post

Hi Sebastien,

That’s a great observation. You’re correct that the legacy .dat format is becoming less common, and we’ve been taking steps to make our .dat file more efficient. However, the advantages of switching to the MMDB format, as you pointed out, are worth considering. I’ll definitely bring this up with my team for further discussion.

Thank you for sharing the additional context and resources—it’s helpful for evaluating potential improvements.

Wishing you and your team and everyone else a Merry Christmas from all of us at SmarterTools!

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

Gabriele Maoret - SERSIS Replied

12/24/2024 at 12:50 AM

@Zach Sylvester :

I installed a fresh copy of SmarterMail 9119, but I can't find the "GeoIP.dat" file anywhere (I searched the ENTIRE C: drive)...

Are you sure this is the file to use for this workaround?

If so, where can I find it?

NOTE: this file isn't even there when searching my production SmarterMail 8930 server...

I think that this "GeoIP.dat" file is NOT used anymore (maybe it was in older versions of SmarterMail...) and with the new versions it does NOT even exist...

Maybe only those who upgraded from very old versions can find it, and that's why those who have it see it as dated 2019 (in fact we were NOT using SmarterMail prior 2020 on this server, and maybe that's why I can't find the file...)...

So I think that SmarterMail currently uses a totally different method to locate the nationality of the IPs, and that this "GeoIP.dat" file is an UNUSED remnant of old installations...

Please Zach, check well and try to give us correct and useful information...

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

Zach Sylvester Replied

12/24/2024 at 7:32 AM

Employee Post

Hi Gabriele,

Thank you for following up, and I apologize for the outdated information earlier. After taking a closer look, it seems that the GeoIP data is now embedded within our assembly files, meaning it’s included in our DLLs and cannot be accessed directly.

Currently, the only way to update the GeoIP data is by upgrading to a newer version of SmarterMail. However, I’ll bring this up with the team to explore the possibility of integrating MaxMind’s API for dynamic updates in the future.

Thank you for pointing this out, and I appreciate your patience!

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

Gabriele Maoret - SERSIS Replied

12/24/2024 at 7:54 AM

This is a shit...

I cannot update to latest version because of blocking MAPI bugs, so I must stay on 8930 until you release a build that works with my customers (8930 works almost perfectly)...

But so I cannot use Geo IP blocking because I cannot update the geo IP database...

Very bad

I suggest you to really have a GeoIP database that can be updated (manually or automated) NOT embedded/hardcoded...

Embed data hardcoded into DLL (or EXE) is, and always be, a bad practice in programming...

PLEASE change the badge "NOT A PROBLEM" into something will else, because this is definitely a (not solved) problem!!!

Gabriele Maoret - Head of SysAdmins and CISO at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

Zach Sylvester Replied

12/24/2024 at 8:21 AM

Employee Post

Hi Gabriele,

I understand your frustration, and I appreciate your feedback. The decision to embed the GeoIP data in the DLL was intentional, as it simplifies installation and reduces the overhead of managing external files. While this is not a bug but a deliberate design choice, I understand how it impacts users who need to stay on older versions.

That said, SmarterMail is designed to work as a complete package, and dynamic updates to components like GeoIP data are not part of the current implementation. I will share your concerns with the team, but I cannot make any guarantees about changes to this functionality.

Thank you for your understanding.

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

12/24/2024 at 8:23 AM

Then disable it and dont use it if it isnt updated regularly.

False positives and wrong blocking info in IDS is shit and a BIG issue....

J

J. LaDow Replied

12/24/2024 at 8:46 AM

This exact issue is yet another in the pile that warrants an LTS version where BUGFIXES and "version-locked" components (especially ones relating to security) can get updated. It is absolute sh*t that many of us cannot upgrade due to issues with newer versions going unresolved or being fixed only for new bugs to appear in the same version as a bug-fix. We didn't pay for enterprise grade software so that we could be the beta testers for million-dollar companies to enjoy the benefits of their "stable versions".

You tell people to upgrade, but there is nothing they can upgrade to - and you don't have a mechanism in place to update internal "security" components such as ClamAV or MaxMind for those who are version-locked due to introduced bugs. If MaxMind offers users a way to sign up a free license to update their lists I find it hard to believe that licensing is the only issue regarding updates. That sounds like some WordPress theme with bundled plugins garbage right there and not something I would expect here.

Vendor-lock-in is a bad policy.

MailEnable survivor / convert --

Zach Sylvester Replied

12/24/2024 at 9:21 AM

Employee Post

Hi Brian and J.,

I completely understand your frustration regarding the lack of dynamic updates. I’ll definitely bring this up with my team to evaluate the new database format and see if this is something we can implement moving forward.

As for the suggestion of an LTS version, I see where you’re coming from, and it’s a good idea in certain scenarios. That said, even an LTS version doesn’t always guarantee a completely bug-free experience. For example, Apple does extensive beta testing for iOS and macOS, yet bugs still slip into public releases. That’s why we’re focused on striking the right balance between stability and continuous improvement for all users.

Additionally, I’m updating the status of this post to Known because this is indeed a known limitation with the current implementation. I appreciate you both bringing attention to this, as it helps us better prioritize improvements.

Thank you again for sharing your thoughts—they provide valuable insights as we refine SmarterMail to better meet the needs of our users.

Kind regards,

Zach Sylvester

Software Developer
SmarterTools Inc.

J

J. LaDow Replied

12/24/2024 at 9:42 AM

Hey Zach - you brought up a VERY good and VALID point using Apple as an example.

I will counter it with this: When Apple does have a bug in one of their OS releases, they FIX THAT BUG AND RELEASE - they don't continue new features and enhancements in point releases - they use them to fix bugs (and they don't charge users for the right to upgrade to those bug fixes). THAT is a HUGE difference than what we are experiencing here.

Otherwise, you are completely right - but the point of LTS is that if a bug DOES appear, it is resolved without introducing new features or updates. The point of contention here is that with LTS, the risk of a bug is exponentially LESS because by definition, LTS does not get "new features" unless a major release - and in that time-period ONLY are "more bugs" accepted due to being a MAJOR release change. Take Node.JS as an example - or most any major Linux distribution such as Ubuntu. ALL have an LTS channel, and do so with a product that is freely released to end users.

MailEnable survivor / convert --

echoDreamz Replied

12/26/2024 at 10:53 PM

Agreed. We want an LTS channel too that focuses exclusively on bugs and security fixes. Almost every product we use has some form of different branches depending on your needs and we use LTS everywhere. SM is really the only one we have to worry about what is going to happen if we install this build or we really need fix X, but there is tons of changes and new things in the release, is it worth it?

Not being able to update the DB with our own paid databases has also been a problem with this new build/branch of SM. The MMDB format is also not new and has been around for a decade, so if lookup performance was an issue, this has been fixed for years. SM being able to automatically download DBs and reload them with our license key or even looking for changes to the geoIP database file and reloading it automatically would be fantastic.

12/27/2024 at 6:06 AM

@echodreamz

And expected for en Enterprise product.

Jeffrey Werner Replied

1/6/2025 at 3:27 PM

Employee Post

Hello everyone,

We’ve discussed integrating the ability for users who have MaxMind subscriptions to update it within SmarterMail itself. This has been added to our backlog. This will require us to move our conversion process for the MaxMind database into our database format into SmarterMail instead of the build process which produces the installers and overall releases. The MaxMind database isn’t designed for our high query volumes such as a mail server, and that is why this conversion is necessary.

Regarding the MAPI issue affecting a few users, progress has been limited because no customer has provided us with an account or a reproducible scenario to investigate. We currently have one potential report in support with data and steps, and we hope it will help us replicate the issue.

A majority of customers are running our latest versions, and this MAPI issue is continually referenced by just a couple of people, yet we haven’t been provided anything to help them out. It’s difficult to see the posts continue to reference an issue when you’re an organization that literally prioritizes bugs and wants to solve problems.

Jeffrey Werner

Software Developer

SmarterTools Inc.

echoDreamz Replied

1/6/2025 at 10:53 PM

I dont see why the maxmind MMDB database would not be fast enough for your lookups compared to the old .dat files.

I tossed together a quick test (using the MaxMind DatabaseReader) with a million randomly generated IPv4 addresses and another million IPv6 addresses, I was able to do a lookup against all 2 million records on my local machine in 6800ms. Which is going to be around ~300/ms, and that is single threaded. If I do them with multiple threads, I can get it down to 4100ms or about 500/ms.

Surely SM does not need that level of lookup performance?

Matt Petty Replied

1/7/2025 at 7:36 AM

Employee Post

We can do about 10000 lookups in 3ms, single thread, I just ran a test, 2million was 205ms. We've been using this format before their newer format and it's better so we've kinda just stuck with it.

Matt Petty

Senior Software Developer

SmarterTools Inc.