Extreme slow search in webmail
Problem reported by Giorgio Borra - 9/29/2019 at 9:37 AM
Known
Hi,

after upgrade from v.16 to v.17 (now installed build is 7188) my search in webmail has become very slow.
My inbox folder, where I do the research, is 1,3 GB not so big...
The time needed to search for a single and simple word is 43 seconds.
I try to reindex the mailbox but the search times are the same.
Any suggestions ?
SmarterMail Enterprise on Windows Server 2012R2 standard, 1 domain , 20 webmail, installed on a VM very powerful with 8 CPU, 16 GB Ram, 6 dedicated disk in raid 5, SAS 10k, non stressed server and all patch and firmware updated (HP DL380 Gen9).
Thanks
Giorgio

30 Replies

Reply to Thread
1
JerseyConnect Team Replied
We've also experienced the same slower response times with the current search. I even opened a support ticket, but the only suggestion was to reindex the affected accounts. What I've found through my testing is that search results will initially take up to 50 seconds or more to return. If you then run additional searches for other words response times will eventually improve to 15 seconds or less by the 3rd or 4th subsequent search.

I've also found the search response times are better (within a few seconds) immediately after an account has finished reindexing.

We also have a similar setup, Windows 2012R2 on a VM with 6 CPU, 12 GB RAM, and 10k SAS storage.
3
Jade D Replied
Also experienced the same issue and I addressed this in a post about general performance issues with the latest version and it was suggested that I was wrong, and that smartermail 17 is the fastest best version available.....
0
Giorgio Borra Replied
The faster version of SmarterMail were 15.x
Version 17.x is the slower I ever had (for search element)
SM team, is there a way to manually delete index ? 
I notice that reidex process is very fast...just 4-5 seconds for reindex a 15,3 GB mailbox.
Thanks
Giorgio
2
Arthur G. Replied
We are currently on version 15 and are trying to hold off the upgrade for as long as is possible.

Surprised to see there is no response from ST team regarding this issue.
1
Derek Curtis Replied
Employee Post
We will have a minor release this week (non-MAPI related) that has a number of indexing and GRP fixes in it. This should resolve some oddities with indexing which, in turn, should speed up searching and advanced search. 
Derek Curtis
COO
SmarterTools Inc.
(877) 357-6278
0
Giorgio Borra Replied
Hi Derek,

I have installed new version 7236. 
The same search now take 30 seconds.
Good job, we have gain time. But is not enough. 
It would be very appreciate if the search were almost instantaneous as with Outlook.
Version 15.x was very very fast, a few seconds for a simple search.
Can you insert a time counter or a progress bar during the search ?

Giorgio


0
echoDreamz Replied
For me, SM search on a 5GB mailbox with many many many thousands of emails takes ~7 seconds. Pretty quick.
0
Cary Gouge Replied
Any update on improving the search and advanced search speeds via webmail?
2
Jade D Replied
Not too sure what you've changed on your server Christopher, but ours is dog slow.

Latest version of SM Build 7242 - 38GB Mailbox with mails dating back a decade.
Simple search takes around a minute, may be more. The search indicator status stops and then a little while later the results appear.
0
Webio Replied
Some suggestions I've presented here:


IMHO 99% of the time users are looking for certain sender emails or names so allowing mail admins to search form to look only in email addresses FROM fields would speed things up very much
0
Michael Muller Replied
You should all also check that your searches, if successful, show all results that should show. I'm finding that many searches won't find emails after a certain date. For me, it's September 19th, or thereabouts.

We, too, upgraded from 15 to 17, and are also finding the simple search has suffered.
---
Montague WebWorks
Powered by RocketFusion
0
Alex Clarke Replied
+1 for experiencing issues with search. 🙁
0
Michael Muller Replied
That said, the recent update did fix the webmail simple searches.
---
Montague WebWorks
Powered by RocketFusion
2
Giorgio Borra Replied
Installed version 7242, now search is fast. 3 seconds for a simple word.
Good Job ST !

Giorgio
0
echoDreamz Replied
Our search has always been quick, though we do use a quite large SSD-based CacheCade for our SM RAID.
0
Sérgio Rocha Replied
Hi,

We upgrade to v17 this weekend, and we have a client with 80GB mailbox and the index its externally slow and dont find most of the emails.

Regards,
 
1
Ionel Aurelian Rau Replied
We`ve found that at least for our users with very large mailboxes (30-50+ GB), simply issuing an re-index for that mailbox was not sufficient - users would still not be able to find some emails. What we did in order force a real, full re-index, was to delete all of the existing indexes manually and then force the re-index. After this, all emails would come up in a search. We did this a while back and do not remember the exact steps we did (if we stopped the SM Service, etc), so please do not do this unless you know what you are doing and unless you can find what are the exact correct steps.

However, even with the mailbox completely indexed, there is still an issue that I wrote again in the past: normal, in-folder search is much, much slower than Advanced Search. 

We love the Advanced Search and do not want it modified in any way in terms of performance (except if this can be further improved :) ). However, it would be good if the Normal Searched was at least as fast as the Advanced one.
0
Giorgio Borra Replied
Hi, we have create 20 folders on the inbox and for each folders about 10 subfolders.
We have move many mail into these (folders and subfolders) and reindex.
The mailbox have 16 GB data.
The result is a very slow search also in subfolders with 30 mail. If I search a simple word it takes about 20-30 seconds.
It seems that when I click on a single folder SM search in all the folders and subfolders present in inbox.
ST can you investigate ?
In another mailbox 15 GB, only inbox folder, the search is almost instantaneous.
Thanks
Giorgio Borra

0
Hootan Roosta Replied
Yes we experience the sluggishness as well. It is extremely slow.

Also in the advanced search we can't do anything with the result except open one by one we need more functionality once you get the results. Only delete is available. We need the "more action"  button (the one with the 3 dots) in the main result window that would apply when multiple items are selected. Like forwarding, marking as read. Flagging, etc...

0
Sérgio Rocha Replied
Lets keep positive that ST look to this issue in the future, We have a few client very unhappy with this, asking why an upgrade make the webmail worst.

0
Alain Néris Replied
Most of our clients and the people from my team are "happy" with SM15 (after a short but worrying trial of SM16, 2 years ago). From time to time, I think about V17, but, for now, I don't see the point of it.

According to my opinion, SM15 webmail is still a very effective professional "desktop" webmail. A popular webmail like yahoo webmail has nothing to envy, etc... And we do not need a fully responsive design interface for a "desktop" webmail.

I have a question for ST. Would it be possible to develop SM15 software and what would be the limits ?
3
echoDreamz Replied
ST is not going to keep developing on a now almost 4 year old product. Makes no sense. Focus development time and man hours on the current product and the future. You dont see the point in upgrading to a newer version, but want them to continue developing and improving v15, which they have... It's called SM17.
1
Alain Néris Replied
echoDreamz. 

Of course, I understand development time and man hours. It's the same for me in my own company and for everybody. But 4 years later and 2 releases ahead, there are still issues with slowness... Is it because of the technology ? About .NET, from 1.1 to 4.X... I know it quite well for my developments. And I know it is not so easy. Performance traps may occur. Is it because of file formats ? 

I just would like to understand. I often read the posts here waiting for The moment to upgrade.
1
Matt Petty Replied
Employee Post
Alain Néris, Maybe you could iterate more on specifically what kind of slowness are you seeing? We have made some changes that relate to speed on a system. Here's some.

-Internally we've reduced the number of locking between threads which allows many aspects of a user's account to be concurrently accessed, reducing the amount of spinners a user may see.
-We have a new internal system for handling all of our data, this allows us to more easily make file changes and control behavior consistently. (grp's-cfg's, stats, indexing are an exception to this)
-With this system we can now in many cases completely cache files internally in memory so that we don't need to read them from the file system.
-We implemented changes that allow many changes at the same time (or in a short period of time) to be rolled into ONE write to the disk, delayed cache writers.
-With the beta we are introduced a message caching mechanism allowing us to intake, move, and read thousands of messages in a matter of seconds. While messages are in this state they can be modified, indexed, accessed by push-based protocols, loaded in webmail and scrolled all concurrently at the same time.
-in Beta, we implemented CONDSTORE for IMAP allowing much quicker syncs for clients with large mailboxes, and consequently reducing load on the system.
-We've also made a couple front end changes to calendar loading, as well as management lists like Domains and Users taking a while to load.

Those are some quick off the top of my head changes that we've recently made relating to speed. There is still more we can improve. Indexing is something that we will look at but do to how large of a system it is, we couldn't weight MAPI, mailbox, other protocols, and all the other beta stuff at the same time until we get this beta finished.


EDIT: To specifically address this thread. For now, rather than requesting a new thread since this is technically resolved, I will just take it back to Known until we have the chance to take a look at this again and a bit harder. We may find some more aspects to speed up.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
echoDreamz Replied
In our environment specifically, we've had zero complaints about search speed, infact, the opposite, especially since mid Jan or so, the beta builds have been awesome for webmail clients, they report very quick performance across the board.
0
Webio Replied
I will repeat myself from various topics:

Search form above message list should be configurable by mail server admin. By configurable I'm thinking about some checkboxes which will enable/disable searching in message sender, receiver, title, message body etc.

Example:

Using advanced form where I've provided variables:

1. FROM some email

found messages right after pressing search button.

Providing the same keyword in field above message list performed search operation in few minutes. Seriously for most people looking for messages they are interested in finding messages from some sender and not a keyword inside body of those messages. For this kind of search operation advanced search form is there.

EDIT: I'm using 7242 build.
1
Giorgio Borra Replied
Hi Matt,

thanks for reply.

I have installed SM17 in current build 7242. 
For me the problem is the search when in a mailbox there are many folders and subfolders.
Please Matt can you try to create for example 10 subfolders in Inbox on a mailbox with at least 10 GB occupied space, and move many mail from inbox to that folders. Than do the same on Sent Items, create 10 folders and move many mail from sent items to that folders. After that reindex and try to search something in inbox. 
In my case, and from my customers with other servers and SM installations, the slowness is present when there are subfolders in Inbox and in Sent Items. 
Search in SM is very fast when there aren't subfolders.

Thanks

Giorgio


0
Alain Néris Replied
Mat Petty, As I wrote above, I am actually a SM15 user (after a short SM16 experience, 2 years ago). I have no doubt that SM17 (and the latest release of it) is better than SM16. I appreciate your efforts in designing and programming the best SmarterMail. 

Thanks for your detailed explanations. I learn a lot specially regarding your multithreading system avoiding readings and writings from and to the file system. I came up with almost the same strategies to face almost the same problems in my own developments. I do not know how you manage file attachements (large and frequent files such as jpg, pdf, and so on...) ? I had to deal with this problem in my image bank indexes, and finally I chose to read only once the image(.jpg) at the creation process of my galleries (to get image properties in my case), to store needed informations of the image in xml files and xml collections to be used later. But this may have nothing to do with your system.
5
Matt Petty Replied
Employee Post
Alain Néris, We store all raw content of an email in a grp file then we keep things like ID's, change numbers, last modified times, etc in the CFG file. Our code selectively reads and writes only specific parts of this file as it's being used. This allows us to dump a lot of data into a single file but still read and write it very quickly. Another method used by some databases, I was reading about RavenDB yesterday. How they store large binary objects is that they allow you to associate binary blobs to objects in the DB. When you read from their DB you get metadata about the binary data and it gives you a way to specifically pull the binary data when needed while still keeping it with the DB object.

Going more into our delayed-writers. Since we have a memory version of most files we can immediately update the values in memory so the rest of SmarterMail see's the change then about 250ms to 2 seconds (different for some files) later we commit the file to disk. We are also told by the system if the process needs to shutdown, allowing us to commit these files quickly to disk. This does bring up an interesting case though, like what happens in this 250ms to 2 second window where power could get cut to the server and whatever was changed is lost, this only affects json files though. GRP and CFG are pretty rugged to the system randomly shutting down, this is stuff we've improved lately, how the system reacts to a unexpected shutdown...


Giorgio Borra, I have a large account that I can split into many folders. I'll give that a test at some point today. Based on this test I will create a task on our board to have it looked at. We are in a period right now where we have effectively halted large changes in favor of stability and finishing up what's left in MAPI and fixing reported bugs community and tickets.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Giorgio Borra Replied
Hi Matt, did you take the test?
Giorgio Borra

Reply to Thread