2
Compression
Idea shared by Charalampos Michael - 6/19/2024 at 5:16 PM
Under Consideration
Hello,
  Wouldn't be better to apply a fast compression to e-mails ?

Thank you

8 Replies

Reply to Thread
0
Tony Scholz Replied
Employee Post
Hello Charalampos Michael, 

Are you referring to the GRP files on Disk or all of the files on Disk? 
Tony Scholz System/Network Administrator SmarterTools Inc. (877) 357-6278 www.smartertools.com
0
The GRP which contains the e-mail data
0
Tony Scholz Replied
Employee Post
Thank you, I have escalated a request. 
Tony Scholz System/Network Administrator SmarterTools Inc. (877) 357-6278 www.smartertools.com
1
If this is something you are thinking of adding may I suggest:

1) Make it something that can be enabled / disabled - sometimes it is preferred to use storage level dedupe / compression
2) Have an option so if enabled, enable compression of GRP files over X days old - that way recent content that is more likely to be used will not have to be decompressed
3) Ensure user usage is tracked on the 'native' rather than compressed size
0
Windows Server can do this for entire disks.

Would I be correct, however, that this would deteriorate IOPS for SmarterMail?
0
Compression = smaller file (faster read) but more cpu usage (depends on the algorithm).

@Ron NTFS compression is too old i'm already compressing the smartmail folder with NTFS compression

@Tony i dunno if it's possible with your GRP form to exclude attachments from the compression.

Thank you

4
Matt Petty Replied
Employee Post
You are already compressing the files then if we compressed the files then NTFS wouldn't be able to compress as much. You can't double compress files, so atleast with your use case we'd be trading NTFS compression with our compression and losing performance because now both us and NTFS are gonna be doing compression work for no gain.

Our GRP's are built for random access, which means we don't actually have to read the entire file when we're getting data from it. You lose this capability when your compression is done at the file-level. This would require a restructure of how our email is stored in order to not decimate the performance of doing this random access method. 

I've got a super experimental side project that can store emails in a new way that can actually reduce the size of a typical SM size by HALF, WITHOUT compression. The more users and domains you have the better it scales and it even works across multiple servers. However, so far this has just been an experimental research project of mine.
Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
0
@Matt I don't intent to double compress if you add compression to GRP files.

Anyhow NTFS LZ77 algorithm is old and slow and i don't think it's the best for texts. I can image how much better job a PPMd or density or even ZStandard will do for e-mails.

btw, your experimental project sounds awesome.

Reply to Thread