12
Keep IMAP custom tags (like color tags)
Idea shared by Sébastien Riccio - 8/26/2020 at 3:53 AM
Completed
Hello,

We recently migrated some customers from their very old mail server to SmarterMail.
Some of them reported to us that on the old server they were able to set colors tags on their mails (from outlook mac).

This is an issue for them as they use a mailbox on multiple devices and on the old system the tags were reported on each device (they are saved through imap tags I guess).

Since they are on SmarterMail when they set a color flag on one device, the others devices are not aware of this. It seems SM is not accepting IMAP custom tags so it's only saved locally. And it's a problem for our customers that are already asking us to migrate them back on their old server.

To reproduce it, I've setup a smartermail test account on my emClient and added color tags to some mails. If I setup the same account on anther emClient, the tags aren't present.

I did the same test with another basic mail server (based on dovecot), and this is working perfectly. The tags are visible on both clients and are resilient to account removal/readd (which is not the case with SM).



Sébastien Riccio
System & Network Admin

36 Replies

Reply to Thread
1
Employee Replied
Employee Post
Thank you, @Sébastien, we already have a feature request for TAGS with IMAP. I will link this post to that task and update when we plan on implementing it.
1
Ah okay cool. Thanks a lot Robert.

Sébastien Riccio System & Network Admin https://swisscenter.com
1
Oh we need this badly too. We almost lost a fire department and had a pissed off fire chief on our hands when they moved from Windows 7 to 10 and found out the server didnt sync their custom tags. We were able to move their database over though to preserve them.
1
echoDreamz,

Please continue making them angy and let them know that we will take care of their e-mail needs at firehousesolutions.com :)

Ron
1
lol - Oh? Your SmarterMail installation magically has support for tags :).

Had the city IT guys contacted us before they decided to upgrade their systems to Windows 10, we could have assisted them in the migration of eM Client. Fire Chief wasn't mad at us, he was just irritated at the entire process of the upgrade to their new machines / Windows 10 from that weekend.

If SM supported tags, their new eM Client installs would have been perfectly fine.
1
Hello Robert,

Any ETA for this feature. Some customers would like to have an estimation when this will be available.

Thanks,
Sébastien
Sébastien Riccio System & Network Admin https://swisscenter.com
2
I was re-thinking about this feature request, because some customers asked us again when it will be available and was asking myself, with the little knowledge I have of how SmarterMail stores the data, how it could be done.

As far as I know flags for mails are stored in mailbox.cfg so that would maybe the place to add support for custom tags.

The problem with custom tags/keywords is that they are in fact custom so it would not be suitable to add a column in the cfg for each of them. Well There are some consensus about the name of certain tags but still.
Could a solution be to add a single column in mailbox.cfg, like for example imaptags, and store them as a json object in that column (as long as mailbox.cfg supports dynamically sized text fields?).

Then any imap client could store whatever tags it wants for a mail and that would be stored like this.
Could this makes sense ?

That would be the first step and then add then support for it on the imap service.

Next step which could be a plus would be to add support for some common custom tags in the webmail for example to colorize/categorize some mails, but the most important is to have at least the tags saved so they are reported between imap clients.

Kind regards
Sébastien Riccio System & Network Admin https://swisscenter.com
1
This function will be good add for us too!
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
0
Tim Uzzanti Replied
Employee Post
I wanted to update the community on this request after we have reviewed the RFC's and tested a variety of clients and their capabilities. Thunderbird and older Outlooks somewhat support this behavior but with Outlook 2016 and later it has been deprecated and is somewhat broken.  We then evaluated whether there would be any value implementing this for newer protocols (having this data on the backend) and there is no value.  Based on this result, we will not be implementing moving forward.

Keep in mind, we do support categories in webmail and those categories will be synced with Outlook using a variety of protocols.  This is one of the reasons why Microsoft broke this older IMAP functionality to fully integrate what is much more thorough functionality with EWS and MAPI.

Hope this helps,
Tim Uzzanti CEO SmarterTools Inc. (877) 357-6278 www.smartertools.com
5
Hello Tim,

There is not only Outlook in life. Many eMail clients use IMAP keywords to store flags to keep them sync between different computers instead of storing them only locally.

Any current opensource IMAP daemon (Dovecot, courrier-imap, Cyrus IMAP, etc) supports it and it is also webmails like Roundcube uses this feature on the backend IMAP server to store categories flags.
(yes we had recently to install roundcube in front of smartermail as a workaround to the many issues our customers is currently having with webmail encoding issues)

We moved customers from our previous mail system to SmarterMail and had quite a bunch of complaint that this flag keeping ability has been lost since they are on SmarterMail.

Due to past and current issues we are now moving them again to a new opensource components based mail system we had to setup to replace SM.

We first decided to use SmarterMail for our mail service because we did not want to spend time on developing our own mail system based on existing componants and use our time on other projects.

In the end we unfortunately spend more time troubleshooting issues, doing detailed bug reports, losing existing customers (and even new sometimes when we mention SmarterMail) and decided recently to go back to a service we can fix ourselves and don't depend on another company goodwill.

This answer about the IMAP keywords implementations seems to confirm we're right.

Regards.

Sébastien Riccio System & Network Admin https://swisscenter.com
4
I would like also to add that for our web hosting provider business, and on 25k mail accounts, most of them are using only the good old historical protocols (mostly IMAP) and the webmail.

Despite we are offering freely EWS, and EAS without any additional profit  only a few are using it. Around 100 for EAS and fewer for EWS and none for MAPI.

So for our userbase we need these to be ironed as a priority and it seems it's not where the product seems to head. This feature request denial tends to confirm it. We also submitted some other tickets about improvements for these protocols that seems they will be never taken into account.

When we ask our leaving customers if they want to share a feedback about what made them decide to leave us, it often come back to happy with the web hosting but "too many recurring issues with mail and it is very important for us".

All this makes us think that it's time to face the reality. SM doesn't match our needs anymore and seems it never will, so better move on...

Sébastien Riccio System & Network Admin https://swisscenter.com
1
Thunderbird uses IMAP Keywords to store tags. IMAP Keywords are a standard and required feature of IMAP under the RFC. Therefore the lack of support for keywords makes the IMAP offering of SmarterMail not fit for purpose. This is not a feature request it is a bug report.
EAS, EWS and MAPI are minority protocols that are rarely used except for businesses with a lack of IT knowledge so the attitude that proper IMAP functionality is of no value is extremely irrational.
This product does have a place, but if IMAP is not supported properly it will not survive as IMAP is the email protocol of choice.
3
IMAP Keywords are part of the RFC (RFC3501), yes, but they are optional and not a requirement. It would be amazing if SM had this... we have quite a lot of people wanting this, clients who have moved from other platforms (specifically Linux) where tags/keywords are supported.
3
Yes they are optional as per RFC but in the reality many IMAP implementations provides them and many clients use them. IMAP is still by far the most used protocol for mailbox access and a mail server providing IMAP services should provide these optional but widely used features or will feel incomplete compared to other implementations.

Sébastien Riccio System & Network Admin https://swisscenter.com
2
Given that this thread is marked as a declined feature request when it should be a bug I have submitted a new bug report.
3
Tim, I believe SmarterTools missed an opportunity to shine on this particular request.

Here is a list of e-mail clients:

4
+1 in implementing IMAP Keywords.

Even if it's optional, it will be an amazing addition to SmarterMail
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
3
I also attach myself with my small installation

2000 users all in imap who mainly use thunderbird and some emclients

I'm also having a hard time migrating users who use tags in thunderbird

I add that I have been administering other mail servers and am migrating to smartertools many clients from icewarp

Discovering that smartermail lacks the tags imap functionality was a really nasty surprise

in icewarp there is
in Dovecot, courrier-imap, Cyrus IMAP there is


I then verified that those who use thunderbird to access a gsuite account and use the tags, these are correctly displayed among the various thunderbirds that connect to the same account


Using emclient on gsuite with the same client the tags are correctly managed

Therefore, refusing this functionality means preventing the migration of gsuite accounts used by customers with imap clients and with tags
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy
0
+1 That was a bit what I tried to explain two years ago in the original request :)
Sébastien Riccio System & Network Admin https://swisscenter.com
6
Matt Petty Replied
Employee Post
Hello,

    Just poking this thread and setting it to "in-progress". The release we are working on has support for both colored flags (in some exchange clients) and colored categories already. We should hopefully be able to tie the IMAP stuff into this system. No promises yet on if these IMAP tags will sync directly to our colored email functionality but I'll try. At the very least we can persist what the client desires now.

Based on your feedback @Sabatino I will look into our IMAP retrieval/migration code, we should be able to pull these custom properties and retain them.
Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
4
Matt Petty Replied
Employee Post
Another update,

I finished the implementation for this. I tested using thunderbird+macmail. I set some flags, deleted the account, IMAP migrated the data to a new account, connected to thunderbird+macmail to the new account, and... the flags stayed! So this is good!

However, in my testing, even though eM Client has custom tags/flags and we advertised to the client that we'd accept custom tags they were not syncing any flags except the normal system flags. We have a very good and open channel of communication with eM Client and we'll let them know about this and maybe they can touch it up to work as expected.
More of my testing and implementation revealed that we will NOT be able to tie in Thunderbird and MacMail's custom tags/flags to our categories. So the flags set in one type of client may not sync to another type of client. No flag data should be lost.
For example, 
-mac mail's flags came over as $MailFlagBit0, $MailFlagBit1, $MailFlagBit2, etc
-thunderbird's flags came over as $label1, $label2, $label3, etc.
-mac mail wont see thunderbird flags, thunderbird wont see macmail flags
-webmail may not see any of these flags by clients
-no flag data should be lost if you use two different clients and modify the flags of an email, unless that client chooses to replace ALL the flags on the message, which I didn't see happen at least with the 2 clients I tried.

TL;DR:
Clients will retain their custom flags but the custom flags won't automatically map to other client's flags, unless their names match. We will now also pull all custom flags during a mailbox migration/retrieval and send all custom flags to the clients that request them via IMAP. Yay!

EDIT:
Only the first 5 main categories for thunderbird are $label1, $label2, etc.
If you delete all the default thunderbird tags and create new ones or rename the pre-existing ones to match categories (tags) in webmail they will sync. This will further sync with the exchange clients too. Theoretically, you can have Thunderbird, Outlook MAPI, Webmail, and Outlook for Mac all syncing the same flags/categories for mail messages, super cool!
Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
2
I am happy to have helped to clarify why this was necessary

I wanted to report my experience with em client and thunderbird with another mailserver

If the labels have the same name they are correctly exchanged and displayed by both clients
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy
0
Matt Petty Replied
Employee Post
Hmm I may have misconfigured something with eM Client then. I noticed renaming the default categories or creating new ones caused Thunderbird to work more as expected. I didn't try some of these things like changing the defaults in eM Client. If I have time today I can try it again and I'll atleast put down a note to bring it up or QA it later. 
Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
0
The problem with custom tags/keywords is that they are in fact custom so it would not be suitable to add a column in the cfg for each of them.

0
That's a bit why I had proposed years ago that an unique column containing a json with the custom flags could be added to the .cfg. 
Sébastien Riccio System & Network Admin https://swisscenter.com
3
Matt Petty Replied
Employee Post
We actually use properties.pids for this now, previously this was used only for MAPI but this update we've added some more general data to the file, EWS is now more utilizing the data within that file. This file is how we store a lot of the Outlook specific information, outlook tells us to store all sorts of stuff, some of that stuff we know and some of it is just binary garbage that we need to store but we can't really do anything with. For example on these things "Flagged" means a whole lot more than it used to, our web interface is are simple flags but in the clients for EWS and MAPI the flags can be colored, they can have due dates, can be assigned to other people, flags have to store a lot of dynamic data now and thus we've switched to PIDS to store this data. This is also one of the reasons why you can't downgrade from BETA to LIVE because we've made some changes to CFG to support this.
(This bit above is only for email)
The collab items Contacts, Notes, Contacts, and Calendars use a field in the JSON (folder-*.json) file, this isn't necessarily new but were doing a lot more with these now.
Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
1
SmarterMail Free 100.0.8482.26162 (mar 23, 2023) 

I installed the RC to explore the new features. Unfortunately I was unable to participate in the BETA.
One of the things I was looking forward to in this new version was support for imap tags

This is because I have several customers to migrate that use tags and the lack of support was blocking me.

I tried to migrate a box (using imapsync which is more comfortable for me - icewarp source sever)
and I found that:

1) Tags are migrated correctly. They are correctly displayed in the new webmail. By connecting in imap with tb the tags are correctly managed (even the personalized ones as long as they are recreated with the same name in tb).

2) I was however surprised with not working emclient (9.2.1628) not. Emclient connected in imap to the old icewarp tags are managed and displayed. Connected to the new sm, they are not handled at all. It shows me the set ones, If I set a new tag in emclient, the tag is not transferred to the server and in fact it is not displayed in webmail or in tb
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy
3
Tim Uzzanti Replied
Employee Post
We are reviewing prior to release.
Tim Uzzanti CEO SmarterTools Inc. (877) 357-6278 www.smartertools.com
1
Hi Tim.
Is there any news regarding imap tag support for emclient?
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy
2
Tim Uzzanti Replied
Employee Post
All done on our side and working!  emClient is going to release an update that enables the functionality fairly soon.  There will also be some other goodies for system and domain admins in relation to emClient as well.  More information shortly!
Tim Uzzanti CEO SmarterTools Inc. (877) 357-6278 www.smartertools.com
0
Tim Uzzanti Replied
Employee Post
We have been told, next minor release of emClient.  Could be a few weeks.  
Tim Uzzanti CEO SmarterTools Inc. (877) 357-6278 www.smartertools.com
0
I have installed the latest version of sm

SmarterMail Enterprise 100.0.8657.28945 (Sep 14, 2023)

I did various tests.
Imap tags work between em client and webmail

between two thunderbirds

The anomaly lies between thunderbird and webmail and emclient

In this case tags work if and only if you do not use the default tb tags but create your own in tb
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy
1
Andrew Barker Replied
Employee Post
Sabatino,

That is correct. When you apply a default tag to a message in Thunderbird, the tag is sent to SmarterMail as something like $Label1. Under the IMAP4 specification, the leading dollar sign indicates that this is a keyword. Since it's a keyword that only has meaning to Thunderbird, it doesn't get displayed by other clients.
Andrew Barker Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
0
Mmmm
An endless story. When I think I have the imap tags available I discover that there is another problem.

If you move a message or a set of messages from one folder to another from webmail, all tags are lost.

If you do it from emclient, it seems to work, but it's an illusion. In fact, when checking via webmail the messages no longer have the tag and in fact trying from another emclient or simply repairing the folder the tags are not there

:-(
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy
0
I would guess they use the folder's index (mailbox.cfg) to store the tags information and they probabgly forgot to include the freshly added custom tags in the move operation from one folder to another. I'd assume it can be easily fixed by adding it.

As for the repair I think you lose all other mail status flags anyway (read, replied, etc) because, if i'm not wrong, a repair basically rebuild mailbox.cfg based on the GRP files and the GRP files do not contain these informations. They are stored in mailbox.cfg which gets completly rebuilt when repairing.

Sébastien Riccio System & Network Admin https://swisscenter.com
0
Hi Sebastian

I looked into it further and it is a very particular case. I had also opened a ticket which I will now correct by indicating what I discovered.

I need to migrate a client from a local mail server with a slow internet connection.
The domain is already on my server but the messages are downloaded in pop to the client's local server.
Some mailboxes are large. So I decided to take steps and did some tests.

I basically created a tmp@dominio.ext box on my server

I did a sync (with migrate mailbox) of the client's mailbox on this test mailbox.

The idea was to have the customer start using the mailbox directly on my server and move the existing archive to him calmly.

Once the migration was finished I checked that the tags on tmp@domain.ext were present.
on this box any movement of messages from one folder to another works without losing the tags

Now having to move the archive to the customer's real mailbox, I thought about creating a sub folder on the customer's mailbox. copy the tmp contents into it and then make an attach folder.
And here comes the problem.
The tags are imported, but if I move a message from one folder to another they are lost. Even if I do a reindex first. If I rebuild this folder the tags disappear.

Ok, I understood the problem, another try.
Instead of doing with attach folder I did a retrival imap with subfolder destination.
Works. imported tags, lossless movement. Reindex and rebuild without loss of tags

Very particular case. I don't know what it depends on and if it could be useful to anyone
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy

Reply to Thread