Max file size of grp files

Problem reported by WebControl GmbH - 10/21/2015 at 5:45 AM

Resolved

Hello SmarterMail community and developers,

currently I'm doing some benchmark tests with SmarterMail Version 14.3.5752 Free Edition. During that I've noticed something weird.

With a simple mail sending script I've sent mass mails with 5MB attachment to one mailbox. After some time I noticed that the mails stuck in the spool. I took a look in the delivery log and saw this error message:

Add message to mailbox call failed. Reason: Access to the grp file is locked

I searched for this error in the knowledge base and found the explanation, that this error appears, when a mailbox is currently being accessed via POP. But in my case there is no access via POP.

So I stopped my script, left the remaining mails in the spool and called it a day.

On the next day I saw that a bunch of mails were delivered, but again not all of them. The error in the delivery log was the same:

Add message to mailbox call failed. Reason: Access to the grp file is locked

I took a look in the filesystem and I was surprised, because the grp file from yesterday and today had almost the same size:

2015_10_13 - 2.738.144 KB
2015_10_14 - 2.738.141 KB

On the next day the same error and almost the same size.

My first suggestion was the filesystem but it cannot be the reason in my opinion. As storage server I use a Windows Server 2012 with a NTFS partition. SmarterMail access this via samba share. Although there is no problem creating files much bigger than 2,7 GB.

There are no space limits in SmarterMail for the domain and the user.

Is it possible that this filesize is a restriction in the code of SmarterMail or the .NET-Framework?

9 Replies

Reply to Thread

Bruce Barnes Replied

10/21/2015 at 6:26 AM

SmarterMail doesn't like anything else accessing its files, and shares cause additional issues. That's why external antivirus programs should be restricted from all of the SmarterMail directories.

Bruce Barnes ChicagoNetTech Inc brucecnt@comcast.net Phonr: (773) 491-9019 Phone: (224) 444-0169 E-Mail and DNS Security Specialist Network Security Specialist Customer Service Portal: https://portal.chicagonettech.com Website: https://www.ChicagoNetTech.com Security Blog: http://networkbastion.blogspot.com/ Web and E-Mail Hosting, E-Mail Security and Consulting

WebControl GmbH Replied

10/21/2015 at 9:30 AM

Hi Bruce,

according to your reply I installed ProcessMonitor https://technet.microsoft.com/de-de/sysinternals/bb896645.aspx to check if other applications maybe accessing the SmarterMail files. But it's only the SmarterMail Service involved. Also I stopped the replication on the storage server. After that I did another test with my mass mailing script:

Now the delivery stopped when the grp file was 2.500.077 KB. I restarted the SmarterMail service and now I see several errors of this type in the delivery log:

[2015.10.21] 18:17:38 [82305] Starting local delivery to user@domain.tld
[2015.10.21] 18:17:38 [82305] Exception: An attempt was made to move the file pointer before the beginning of the file.
[2015.10.21]
[2015.10.21]    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
[2015.10.21]    at System.IO.FileStream.SeekCore(Int64 offset, SeekOrigin origin)
[2015.10.21]    at System.IO.FileStream.Seek(Int64 offset, SeekOrigin origin)
[2015.10.21]    at MailStore.GroupFileAccess.GroupFile.AddHeaderItemAlreadyLocked(UInt32 uid, BinaryWriter bw)
[2015.10.21]    at MailStore.GroupFileAccess.GroupFile.AddMessage(String #Lrb, UInt32 #q9, Boolean #Psb, String& #Qsb, Int32& #9r)
[2015.10.21]    at MailStore.GroupFileAccess.GroupFile.Add(String emailFilePath, UInt32 uid, Boolean wait, String& headerText, Int32& size)
[2015.10.21]    at MailStore.Mailboxes.Mailbox.AddMessage(String messagePath, MessageFlags flags, Int32 size, String& headerText, String& failureReason)
[2015.10.21]    at MailStore.Mailboxes.MailboxManager.#eEb(String #c1i, String #90i, MessageFlags #Yu, Int32 #9r, Boolean #T9, String& #Qsb, String& #Jwi)
[2015.10.21]    at MailStore.Mailboxes.MailboxManager.AddMessage(String messagePath, String subFolderName, MessageFlags flags, Int32 size, Boolean createMailbox, String& headerText, String& failureReason)
[2015.10.21]    at RelayServer.DeliveryManager.#DEb(LocalRecipient #w5i, MimeReader #ABb, String #Qsb, String #4Db, String #3n, Boolean #x5i, Boolean #y5i)
[2015.10.21]    at RelayServer.DeliveryManager.#xEb(SpoolMessage #Nrb, LocalRecipient #w5i, String #4Db, Boolean #x5i, Boolean #y5i)
[2015.10.21]    at RelayServer.DeliveryManager.#SBb(SpoolMessage #Nrb)

Matt Petty Replied

10/21/2015 at 9:44 AM

Employee Post

Yes this is a limitation of .grp files. We have a task to change how we store the file pointer to allow us to write larger files.

Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com

WebControl GmbH Replied

10/21/2015 at 10:11 AM

Ok, thanks for checking this!

Richard Frank Replied

11/18/2015 at 8:09 AM

I noticed this too, but that was because the same mail was repeatedly posted to an inbox, causing the grp file to grow.
Actually I was glad the file couldn't grow beyond 2.x Gb else issues like I had can consume the space on the hdd / partition

markus Replied

11/7/2016 at 2:05 AM

Please allow me to jump in on this point, and add some additional questions:

1.) as I understand a grp file is created for each day.

So the size of this file is related to the users activity (new messages - deleted)

2.) So the mentioned file size limitation should fit close to all daily operations.

3.) But what happens when I try to migrate from a PST or another’s ISP Mailserver?

4.) Also - an that's why I'm currently searching and studying a lot of articles - this larger grp files quite often becomes notably defragmented but also locked by smartermail due to the mailbox users activity. Now defragmenting the entire disc becomes a real tricky task. The timeout on all that locked files stretches the entire process way to long when office times and backup processes should be avoided.
Stopping the entire service or at least blocking that access protocols that will lock the files and the defragment. In a ISP environment with some thousand users not an option. not even on Sunday early morning.

I understand that data must be stored somewhere and that multiple access attempts (new msgs from spool, pop3, imap, EWS, ...) must be managed to ensure data integrity. The question is if there should be files larger than - let's say - 500MB and the can be delocked without stopping the entire server.

Alex Clarke Replied

6/13/2019 at 3:22 PM

Can someone from ST confirm the max GRP filesize on version 17 (build 7090) please?

Tony Scholz Replied

6/13/2019 at 4:04 PM

Employee Post

Hello Alex,

Yes, the max size for a GRP file is 2 GB.

Thank you

Tony Scholz System/Network Administrator SmarterTools Inc. www.smartertools.com

Alex Clarke Replied

6/14/2019 at 9:24 AM

Thanks Tony.

This explains why uploading lots of emails (over IMAP) starts failing when the GRP files get to around 2GB in size.

It looks like there is a bug when uploading a selection of emails. For example, I've just uploaded 62 emails (around 10MB of data) and they've been added to a GRP file named 2019_6_14.grp.

This is wrong as they're sent items are all date from around 2012!

Don't suppose you've experienced this behaviour before have you?

Back to Community Threads

Please leave this box unchecked

9 Replies

Reply to Thread

Tags

Related Knowledge Base Articles