4
/!\ Half-broken CalDAV probably since Build 7751
Problem reported by Sébastien Riccio - 4/8/2021 at 2:38 AM
Being Fixed
Hello SmarterTools,

Since middle March 2021 we had a bunch of support tickets about calendar synchronisation affecting different mail clients.

After playing find-the-common-thing between all these tickets, it looks to us that it is affecting customers that are using DAV to sync their Calendar and Contacts.

For example:
- an Outlook user using CalDavSynchronizer on Outlook
- a Mac user using Outlook configured with CalDav/CardDAV
- a Thunderbird user using some CalDAV/CardDDAV extension

All these started at the same time and looking at SM changelog I find this:

Build 7751 (Mar 22, 2021)
Fixed: Harmonizely connected via CalDAV does not sync fully due to non-standard GUIDs being used.
Fixed: Some appointments do not sync via CalDAV if the requested resource path is URL encoded.

It would match the date these customers started having problems.

I was personnally able to reproduce the issues With outlook+CalDavSynchronizer but had not yet confirmed for Mac mail and Thunderbird.

What I also can tell is that it seems it doesn't affect emClient configured in DAV mode, but it loooks like the only client unaffected so far.

Another observation is that when using Outlook and CalDavSynchroniser, when you create/edit/modify an appointment from Outlook (that get then synced to the server) it seems to loose the "owner" of the calendar event.
When it is able to sync, and when you then check in the webmail the created event, it is like it wasn't owned by the user it says is was created by "" (empty)
Also CalDavSynchroniser then throws exceptions due to 404 on objects:

It's a bit worrying because if our users starting to create or change objects from a caldav synchronized device and it lose the object "owner" on the way, it is going to go in a situation we will need to reassign somehow the objects to their owner....

Pleaes can you take a look at this ASAP ?

Thanks!


Sébastien Riccio
System & Network Admin

26 Replies

Reply to Thread
0
Stefano Replied
I've seen that problem some days ago with a Microsoft Teams invitation.
Deleted all the invites from the webmail except one, everything is working fine.
But this morning has happened to another user with a simple calendar appointment.
0
Matt Petty Replied
Employee Post
Are all the issues different? The only issue I could gleam from your report was the owner reseting on meetings? Is there more? I can make a task to test out these scenarios and attempt reproduction.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Stefano Replied
Hi Matt, that's what a client has given me.
The error is in Italian, but it seems to be the same error of Sébastien

0
Sébastien Riccio Replied
Hello Matt,

I condensed in one post, all issues from multiple customers that all started around from the same date and all involve using caldav for synchronisation, because I guess that's something around caldav implementation that changed around mid-march (the march 22nd update?).

Different customers using caldav on mac Mail, in thunderbird (windows) and outlook (windows) have all strange issues since mid march, when it was perfectly working for years before this date.

As multiple client software/synchronizer are affected, I would think that it's more a change on server side that is causing this.

Personnally I was able to replicate the Outlook+CalDavSynchronizer issue with a fresh account/fresh calendar just by adding item from outlook. It then throws 404 errors that you can see in the screenshots and also is doing weird things like events created and viewable in the webmail but they are not editable or deletable because "the owner should do it". It doesn't recognise the objects as owned by the user like if it lost ownership.

I was going to personally test macmail and thunderbird with the caldav extension but was short on time today.

Will try to do in a an hour or so.
Sébastien Riccio
System & Network Admin

0
Sébastien Riccio Replied
Matt,

Here is a HOWTO to reproduce some of the issues:

1) Use Outlook + CalDavSynchronizer on a SmarterMail account
2) Configure a Synchronization profile with immediate object synchronisation and two ways sync every 2 minutes, for example.
2) Create a event in Outlook (it will attempt to sync it after a few secs and popup an exception)


3) Close the window and try to manually sync the event again


3) Go to the webmail of the account and check the calendar.

The event will be there but looking like a tentative and owned by nobody (uneditable/deletable from webmail, etc)



That's the issue about Outlook/CalDavSynchronizer. 

But we have people with other client software with kinda the same issue when using CalDav.

Kind regards

Sébastien Riccio
System & Network Admin

0
Sébastien Riccio Replied
Here is how to reproduce the CalDav issue with MacMail:
- Configure the account as CalDav in Calendar App
- Create an event from Calendar app. Event will briefly appear Calendar and then suddently disappear
- Go to the user webmail, the created event appears in the user Calendar
- If you click on it on the web calendar, modify something like the location or description, then click save, nothing happens. The save buton has no effect.

Tested on BigSur with latest Calendar app. The same tests in Exchange mode (EWS) from the same Calendar app seems to have no issue.

This issue and the caldavsyncrhonizer+outlook tends to show something was broken with latest change s on the CalDav code (march 22 build?). Approx date when multiple CalDAV user started to open tickets about issues, coincidence ?

Kind regards.
Sébastien Riccio
System & Network Admin

0
Sébastien Riccio Replied
Hello,

TLDR;
- 7751 introduced an issue that triggers an exception when using Outlook+CalDavSynchronizer
- 7751 introduced an issue that makes freshly created events from Mac Calendar app with CalDAV disappear from Calendar, a few seconds after they are created (but still shows in webmail)
- With 7719 both of these isssues are not present
- The losing of event owner/author issue is already present in 7719 but is triggered by editing an existing event from Outlook+CalDav

So 7751 broke usability of CalDAV with multiple mail clients (looks like emClient is not affected).

Long story:

So we took the time to install 7719 on our test environment and go again for a test run to confirm an issue was introduced in 7751.

With 7719 and Outlook+CalDavSynchronizer the exception 404 popup that our customers are reporting when creating new events is gone and the event is created successfully. So this is 100% confirmed to be due to 7751 changes and confirm what our customer reported since mid march.

However the missing "owner" information is already present in 7719. Not when creating the event from Outlook+CalDav but it is lost when editing the recently created event from Outlook+CalDav.
It is then losing the creator information on the webmail as you can see here:


About the MacOS + Calendar (CalDAV) with 7719, it works correctly. As previously stated, with 7751 the created event is then visible on the webmail but disappear from Calendar.
This confirms what our customers reported (it started acting weirdly mid march 2021) 


Can we please be provided asap with an hotfix for these as we are again in trouble with our customers after installing a new production build...

Thanks a lot.

ps: sorry for the bold emphasis, but I wanted to highlight the most important informations.

Sébastien Riccio
System & Network Admin

0
Stefano Replied
Any update about it?
0
Sébastien Riccio Replied
Nothing yet... to workaround a bit the issue we had to have the customers using caldav switch, at least temporarily, to EWS or EAS depending the software they use.

But they keep asking why they have to do this as it was previously working for years with CalDAV... 
Sébastien Riccio
System & Network Admin

0
Matt Petty Replied
Employee Post
I've found the fix/issue introduced by 7751. I've got caldav sync working again, I am now attempting to repro and fix the issue with caldav sync causing issues with owner of meetings being lost, which was reported as always being an issue. I should have some more information later.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Sébastien Riccio Replied
Hello Matt, 

Thank you for the update and work on this. This is good news. 
Don't hesitate if you have a custom build available, we can then test it on our testing server and if everything seems good push it on prod.

Kind regards,

Sébastien Riccio
System & Network Admin

0
Matt Petty Replied
Employee Post
"However the missing "owner" information is already present in 7719. Not when creating the event from Outlook+CalDav but it is lost when editing the recently created event from Outlook+CalDav."

I sent an invite from matt@smartermail.io (WebMail) to test@smartermail.io (Outlook+Sync)
1. Accepted appointment from test@smartermail.io with Outlook+Sync
2. Checked webmail on test, owner is still retained.
3. Edit the "Location" field on test@smartermail.io with Outlook+Sync
4. Checked webmail on test, owned is still retained, location field is populated.

I was not able to reproduce the owner being lost, did I miss any steps? Is there a specific way they are modifying the appointment?
I'm performing this test with the 7751 fix in place, maybe me fixing the first issue somehow fixed this? But you mentioned reproducing the issue on 7719.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Sébastien Riccio Replied
Hello Matt,

If I remember correctly I did this:

- Created an appointment from Outlook+Sync
- Waited the event appear in webmail's calendar
- Edited the created appointment from Outlook and changed some informations
- Waiting the event was updated in webmail's calendar
- When updated it appeared as tentative and lost tracking of appointment author

If I understand correctly your test, you created the event from an invite, maybe the issue only occurs when you directly create a new appointment (not form an invite) from outlook+sync and then edit it.

Also be sure that the sync occured before every step, or triggers it manually with Synchronize now in CalDavSync.

Kind regards
Sébastien Riccio
System & Network Admin

0
Matt Petty Replied
Employee Post
Do you edit this appointment before the other attendee accepts it? Also when your viewing it in webmail are you viewing the owners or the invitees account? It might be easier in your steps if you state also what account each step uses.
1. Created invite in Outlook+Sync test@smartermail.io sent to matt@smartermail.io
2. Waiting for it to appear in webmail (on matt@smartermail.io). Checked the owner is correct on the invitee's webmail calendar, did not accept.
3. Edited the appointment on the owner, test@smartermail.io from Outlook+Sync, added a Location
4. Waited for the update (on matt@smartermail.io)  checked the owned is correct on the invitee's webmail calendar, still not accepted, the "Location" field was properly modified.

I tested the scenario from Outlook+Sync but still cannot get it to reproduce? Could you put a ton of detail into your steps, including the accounts used, atleast the usernames or some sort of place holder. Since this revolves around a meeting its important to make sure I'm logging in and doing the right steps on the correct users.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Sébastien Riccio Replied
Hello Matt,

In my tests all is done with only one user.

- user@domain.tld creates event in outlook
- wait for the event appearing in user@domain.tld webmail
- user@domain.tld edit the event in outlook and change for example the description
- wait for the event to update in user@domain.tld webmail

at this point the event should be owned by nobody and is seen like a tentative and can't be edited/deleted from webmail.

Can you just try these exact steps, all with a single account user doing actions on his own calendar from outlook+sync, not involving any other accounts.

If you are not able to reproduce it I can downgrade again our test server to 7519 (I have updated it back to latest after my tests) and attempt to do a gif/video catpure of the thing.

EDIT: I've rolled back our test server, if you want we can hook up for a teamviewer or anything else session on my computer so I can reproduce it while you're watching.
I should be available for this until ~23:00 CET, that would be 2pm Arizona time I guess.

Kind regards
Sébastien Riccio
System & Network Admin

0
Matt Petty Replied
Employee Post
How are you setting up a meeting with no invitee's? Isn't that just a regular appointment, regular appointments don't have a concept of an owner or tentative/accepted/etc. The fact it's creating it with an owner and tentative with no invitee's might be the real issue.

Appointments = no invitees, no owner, no status (tentative/accepted/declined)
Meetings = invitees, owner, has status.

EDIT:
I performed your steps, under no circumstance did I ever see anything about an owner when viewed in webmail either before or after the edit, I was also given the ability to edit the appointment before+after the change (because there is no owner). This is expected, as there should be no owner, since this isn't a meeting.
Also, you don't see the owner when your viewing your own appointment in webmail, even if it is a meeting. 

We only show the owner to INVITEE's of an appointment (meeting).

EDIT2:
It's 2:25pm now, but we can do a teamviewer tomorrow if you'd like. Should be available tomorrow. You can DM me the credentials, could hop into live chat and ask for me to get my attention.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Sébastien Riccio Replied
Hello Matt,

Yes it is an appointment, I don't remember talking about meetings ? Maybe I used the term event that is ambiguous, english is not my mothertongue.

I'm still up and sent you the credentials by DM.

Kind regards.
Sébastien Riccio
System & Network Admin

1
Sébastien Riccio Replied
Matt, sent you a DM on here with a link to the video capture.
Kind regards.
Sébastien Riccio
System & Network Admin

1
Matt Petty Replied
Employee Post
For anyone else following along, I also sent my capture following steps which didn't seem to replicate in the same way. I was mis-understanding the issue a little bit, it is an appointment that for some reason was turning into a meeting with no owner and a tentative status, which definitely shouldn't happen. There is a chance I have fixes that others don't have, might wanna try that custom build I can make for you.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
2
Sébastien Riccio Replied
Hello Matt, yes I'm staying up late these night to wait for a build to test asap on our test environnement (not production) and determine if these fixes you have (and that we dont) affect the reproductibility of this issue.

We have a lot of angry customers, the ones using Outlook+CalDav sync and Calendar on OSX with CalDAV telling that since mid march our service is unusable, when it was almost perfectly working before we updated.

Our CEO is getting very nervous about these issues with SmarterMail, I can't even propose to rollback to 7719 to avoid the new issues introduced in 7751 as it is strongly advised NOT to downgrade due to changes in grp files and so on... 

I'm sure you're doing your best to adress thiese issues. Still we're again in deep trouble due to new introduced issues and really need a fix for these before the weekend to keep control on the damage before it turns into a carnage.

Thank you for your help.
Sébastien Riccio
System & Network Admin

0
Paul Blank Replied
Working happily within its limitations, a few of my clients are still humming along on SM v15.7.6970. We are well aware of what doesn't work and what does, but what works just plain works. Split-domain with Office 365. No Outlook in sight. Third-party cloud filter. 

Besides backups, time spent on other than routine user maintenance is close to zero. The plan for this summer (which has been delayed by COVID considerations) is now to migrate users, a few at a time, to O365 while leaving SM intact for departed users, utility accounts, auto-mailing devices and the like. I kinda think we'll stay with SM v15 for awhile.
1
echoDreamz Replied
Yes, Paul, we are all very aware of your true love for v15.
0
Paul Blank Replied
It is as simple as this, for me: I sincerely wish I didn't have the reservations that I have, and I keep my SM licenses paid-for and up-to-date. But I don't have time to be fussing with mission-critical software that gives me and my clients conniptions. In the case of email, it includes the server and client components. I am fortunate that I often get to specify what my clients run. I understand that this is not always the case. Your mileage WILL vary.
4
echoDreamz Replied
You certainly have time to post on the forums about versions of software you are not running which does nothing for those that are reporting issues. We know, you run v15 and are happy, congrats, we will send you some cookies or something. Adding to a thread that you are running a major version of a product released 5 years ago doesnt do anything for anyone.
3
Sébastien Riccio Replied
Hey! May I have some cookies too ? :)
Sébastien Riccio
System & Network Admin

2
Sébastien Riccio Replied
Hello,

Here is an update about the status of this issue.

The main issue (exception with 404) on Outlook+CalDavSync and items created in OSX Calendar disappearing were due to the same issue, introduced in 7751.
This has been fixed in the custom build we received and I was able to confirm this is now back to normal.

--

For the second issue, where our appointments created from Outlook+CalDAV appears as tentative with unknown organizer, instead of appearing as simple appointments that Matt and Kyle wasn't able to reproduce are due to Outlook (or at least Outlook + CalDAV).

For my tests I was using my main Outlook profile which has already a main calendar, and adding the account I was testing in this profile. It looks like Outlook doesn't handle this correctly.

Creating a fresh Outlook profile with only the test account configured shows no issues.
Our affected customers for this particular issue probably already have another main calendar in their outlook profile and are using their SM account as additional account.

I tried different advanced options in CalDavSynchronizer to try to avoid the issue, but without success so far.

Kind regards.
Sébastien Riccio
System & Network Admin

Reply to Thread