11
Keep IMAP custom tags (like color tags)
Idea shared by Sébastien Riccio - 8/26/2020 at 3:53 AM
Completed
Hello,

We recently migrated some customers from their very old mail server to SmarterMail.
Some of them reported to us that on the old server they were able to set colors tags on their mails (from outlook mac).

This is an issue for them as they use a mailbox on multiple devices and on the old system the tags were reported on each device (they are saved through imap tags I guess).

Since they are on SmarterMail when they set a color flag on one device, the others devices are not aware of this. It seems SM is not accepting IMAP custom tags so it's only saved locally. And it's a problem for our customers that are already asking us to migrate them back on their old server.

To reproduce it, I've setup a smartermail test account on my emClient and added color tags to some mails. If I setup the same account on anther emClient, the tags aren't present.

I did the same test with another basic mail server (based on dovecot), and this is working perfectly. The tags are visible on both clients and are resilient to account removal/readd (which is not the case with SM).



Sébastien Riccio
System & Network Admin

23 Replies

Reply to Thread
1
Employee Replied
Employee Post
Thank you, @Sébastien, we already have a feature request for TAGS with IMAP. I will link this post to that task and update when we plan on implementing it.
1
Ah okay cool. Thanks a lot Robert.

Sébastien Riccio
System & Network Admin

1
Oh we need this badly too. We almost lost a fire department and had a pissed off fire chief on our hands when they moved from Windows 7 to 10 and found out the server didnt sync their custom tags. We were able to move their database over though to preserve them.
1
echoDreamz,

Please continue making them angy and let them know that we will take care of their e-mail needs at firehousesolutions.com :)

Ron
1
lol - Oh? Your SmarterMail installation magically has support for tags :).

Had the city IT guys contacted us before they decided to upgrade their systems to Windows 10, we could have assisted them in the migration of eM Client. Fire Chief wasn't mad at us, he was just irritated at the entire process of the upgrade to their new machines / Windows 10 from that weekend.

If SM supported tags, their new eM Client installs would have been perfectly fine.
1
Hello Robert,

Any ETA for this feature. Some customers would like to have an estimation when this will be available.

Thanks,
Sébastien
Sébastien Riccio
System & Network Admin

2
I was re-thinking about this feature request, because some customers asked us again when it will be available and was asking myself, with the little knowledge I have of how SmarterMail stores the data, how it could be done.

As far as I know flags for mails are stored in mailbox.cfg so that would maybe the place to add support for custom tags.

The problem with custom tags/keywords is that they are in fact custom so it would not be suitable to add a column in the cfg for each of them. Well There are some consensus about the name of certain tags but still.
Could a solution be to add a single column in mailbox.cfg, like for example imaptags, and store them as a json object in that column (as long as mailbox.cfg supports dynamically sized text fields?).

Then any imap client could store whatever tags it wants for a mail and that would be stored like this.
Could this makes sense ?

That would be the first step and then add then support for it on the imap service.

Next step which could be a plus would be to add support for some common custom tags in the webmail for example to colorize/categorize some mails, but the most important is to have at least the tags saved so they are reported between imap clients.

Kind regards
Sébastien Riccio
System & Network Admin

1
This function will be good add for us too!
Gabriele Maoret - SERSIS - Head of SysAdmins
Currently manages 3 SmarterMail installations (1 in cloud for SERSIS which provides service to a few hundreds 3rd party Mail Domains + 2 on premise to customers)
0
Tim Uzzanti Replied
Employee Post
I wanted to update the community on this request after we have reviewed the RFC's and tested a variety of clients and their capabilities. Thunderbird and older Outlooks somewhat support this behavior but with Outlook 2016 and later it has been deprecated and is somewhat broken.  We then evaluated whether there would be any value implementing this for newer protocols (having this data on the backend) and there is no value.  Based on this result, we will not be implementing moving forward.

Keep in mind, we do support categories in webmail and those categories will be synced with Outlook using a variety of protocols.  This is one of the reasons why Microsoft broke this older IMAP functionality to fully integrate what is much more thorough functionality with EWS and MAPI.

Hope this helps,
Tim Uzzanti
CEO
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
5
Hello Tim,

There is not only Outlook in life. Many eMail clients use IMAP keywords to store flags to keep them sync between different computers instead of storing them only locally.

Any current opensource IMAP daemon (Dovecot, courrier-imap, Cyrus IMAP, etc) supports it and it is also webmails like Roundcube uses this feature on the backend IMAP server to store categories flags.
(yes we had recently to install roundcube in front of smartermail as a workaround to the many issues our customers is currently having with webmail encoding issues)

We moved customers from our previous mail system to SmarterMail and had quite a bunch of complaint that this flag keeping ability has been lost since they are on SmarterMail.

Due to past and current issues we are now moving them again to a new opensource components based mail system we had to setup to replace SM.

We first decided to use SmarterMail for our mail service because we did not want to spend time on developing our own mail system based on existing componants and use our time on other projects.

In the end we unfortunately spend more time troubleshooting issues, doing detailed bug reports, losing existing customers (and even new sometimes when we mention SmarterMail) and decided recently to go back to a service we can fix ourselves and don't depend on another company goodwill.

This answer about the IMAP keywords implementations seems to confirm we're right.

Regards.

Sébastien Riccio
System & Network Admin

4
I would like also to add that for our web hosting provider business, and on 25k mail accounts, most of them are using only the good old historical protocols (mostly IMAP) and the webmail.

Despite we are offering freely EWS, and EAS without any additional profit  only a few are using it. Around 100 for EAS and fewer for EWS and none for MAPI.

So for our userbase we need these to be ironed as a priority and it seems it's not where the product seems to head. This feature request denial tends to confirm it. We also submitted some other tickets about improvements for these protocols that seems they will be never taken into account.

When we ask our leaving customers if they want to share a feedback about what made them decide to leave us, it often come back to happy with the web hosting but "too many recurring issues with mail and it is very important for us".

All this makes us think that it's time to face the reality. SM doesn't match our needs anymore and seems it never will, so better move on...

Sébastien Riccio
System & Network Admin

1
Thunderbird uses IMAP Keywords to store tags. IMAP Keywords are a standard and required feature of IMAP under the RFC. Therefore the lack of support for keywords makes the IMAP offering of SmarterMail not fit for purpose. This is not a feature request it is a bug report.
EAS, EWS and MAPI are minority protocols that are rarely used except for businesses with a lack of IT knowledge so the attitude that proper IMAP functionality is of no value is extremely irrational.
This product does have a place, but if IMAP is not supported properly it will not survive as IMAP is the email protocol of choice.
3
IMAP Keywords are part of the RFC (RFC3501), yes, but they are optional and not a requirement. It would be amazing if SM had this... we have quite a lot of people wanting this, clients who have moved from other platforms (specifically Linux) where tags/keywords are supported.
3
Yes they are optional as per RFC but in the reality many IMAP implementations provides them and many clients use them. IMAP is still by far the most used protocol for mailbox access and a mail server providing IMAP services should provide these optional but widely used features or will feel incomplete compared to other implementations.

Sébastien Riccio
System & Network Admin

2
Given that this thread is marked as a declined feature request when it should be a bug I have submitted a new bug report.
3
Tim, I believe SmarterTools missed an opportunity to shine on this particular request.

Here is a list of e-mail clients:

4
+1 in implementing IMAP Keywords.

Even if it's optional, it will be an amazing addition to SmarterMail
Gabriele Maoret - SERSIS - Head of SysAdmins
Currently manages 3 SmarterMail installations (1 in cloud for SERSIS which provides service to a few hundreds 3rd party Mail Domains + 2 on premise to customers)
3
I also attach myself with my small installation

2000 users all in imap who mainly use thunderbird and some emclients

I'm also having a hard time migrating users who use tags in thunderbird

I add that I have been administering other mail servers and am migrating to smartertools many clients from icewarp

Discovering that smartermail lacks the tags imap functionality was a really nasty surprise

in icewarp there is
in Dovecot, courrier-imap, Cyrus IMAP there is


I then verified that those who use thunderbird to access a gsuite account and use the tags, these are correctly displayed among the various thunderbirds that connect to the same account


Using emclient on gsuite with the same client the tags are correctly managed

Therefore, refusing this functionality means preventing the migration of gsuite accounts used by customers with imap clients and with tags
Sabatino Traini
      Chief Information Officer
Genial s.r.l.
Martinsicuro - Italy

0
+1 That was a bit what I tried to explain two years ago in the original request :)
Sébastien Riccio
System & Network Admin

6
Matt Petty Replied
Employee Post
Hello,

    Just poking this thread and setting it to "in-progress". The release we are working on has support for both colored flags (in some exchange clients) and colored categories already. We should hopefully be able to tie the IMAP stuff into this system. No promises yet on if these IMAP tags will sync directly to our colored email functionality but I'll try. At the very least we can persist what the client desires now.

Based on your feedback @Sabatino I will look into our IMAP retrieval/migration code, we should be able to pull these custom properties and retain them.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
4
Matt Petty Replied
Employee Post
Another update,

I finished the implementation for this. I tested using thunderbird+macmail. I set some flags, deleted the account, IMAP migrated the data to a new account, connected to thunderbird+macmail to the new account, and... the flags stayed! So this is good!

However, in my testing, even though eM Client has custom tags/flags and we advertised to the client that we'd accept custom tags they were not syncing any flags except the normal system flags. We have a very good and open channel of communication with eM Client and we'll let them know about this and maybe they can touch it up to work as expected.
More of my testing and implementation revealed that we will NOT be able to tie in Thunderbird and MacMail's custom tags/flags to our categories. So the flags set in one type of client may not sync to another type of client. No flag data should be lost.
For example, 
-mac mail's flags came over as $MailFlagBit0, $MailFlagBit1, $MailFlagBit2, etc
-thunderbird's flags came over as $label1, $label2, $label3, etc.
-mac mail wont see thunderbird flags, thunderbird wont see macmail flags
-webmail may not see any of these flags by clients
-no flag data should be lost if you use two different clients and modify the flags of an email, unless that client chooses to replace ALL the flags on the message, which I didn't see happen at least with the 2 clients I tried.

TL;DR:
Clients will retain their custom flags but the custom flags won't automatically map to other client's flags, unless their names match. We will now also pull all custom flags during a mailbox migration/retrieval and send all custom flags to the clients that request them via IMAP. Yay!

EDIT:
Only the first 5 main categories for thunderbird are $label1, $label2, etc.
If you delete all the default thunderbird tags and create new ones or rename the pre-existing ones to match categories (tags) in webmail they will sync. This will further sync with the exchange clients too. Theoretically, you can have Thunderbird, Outlook MAPI, Webmail, and Outlook for Mac all syncing the same flags/categories for mail messages, super cool!
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
2
I am happy to have helped to clarify why this was necessary

I wanted to report my experience with em client and thunderbird with another mailserver

If the labels have the same name they are correctly exchanged and displayed by both clients
Sabatino Traini
      Chief Information Officer
Genial s.r.l.
Martinsicuro - Italy

0
Matt Petty Replied
Employee Post
Hmm I may have misconfigured something with eM Client then. I noticed renaming the default categories or creating new ones caused Thunderbird to work more as expected. I didn't try some of these things like changing the defaults in eM Client. If I have time today I can try it again and I'll atleast put down a note to bring it up or QA it later. 
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com

Reply to Thread