We've also experienced the same slower response times with the current search. I even opened a support ticket, but the only suggestion was to reindex the affected accounts. What I've found through my testing is that search results will initially take up to 50 seconds or more to return. If you then run additional searches for other words response times will eventually improve to 15 seconds or less by the 3rd or 4th subsequent search.

I've also found the search response times are better (within a few seconds) immediately after an account has finished reindexing.

We also have a similar setup, Windows 2012R2 on a VM with 6 CPU, 12 GB RAM, and 10k SAS storage.

Also experienced the same issue and I addressed this in a post about general performance issues with the latest version and it was suggested that I was wrong, and that smartermail 17 is the fastest best version available.....

The faster version of SmarterMail were 15.x

Version 17.x is the slower I ever had (for search element)

SM team, is there a way to manually delete index ? 

I notice that reidex process is very fast...just 4-5 seconds for reindex a 15,3 GB mailbox.

Thanks

Giorgio

We are currently on version 15 and are trying to hold off the upgrade for as long as is possible.

Surprised to see there is no response from ST team regarding this issue.

We will have a minor release this week (non-MAPI related) that has a number of indexing and GRP fixes in it. This should resolve some oddities with indexing which, in turn, should speed up searching and advanced search. 

Hi Derek,

I have installed new version 7236. 

The same search now take 30 seconds.

Good job, we have gain time. But is not enough. 

It would be very appreciate if the search were almost instantaneous as with Outlook.

Version 15.x was very very fast, a few seconds for a simple search.

Can you insert a time counter or a progress bar during the search ?

Giorgio

For me, SM search on a 5GB mailbox with many many many thousands of emails takes ~7 seconds. Pretty quick.

Any update on improving the search and advanced search speeds via webmail?

Not too sure what you've changed on your server Christopher, but ours is dog slow.

Latest version of SM Build 7242 - 38GB Mailbox with mails dating back a decade.

Simple search takes around a minute, may be more. The search indicator status stops and then a little while later the results appear.

Some suggestions I've presented here:

https://portal.smartertools.com/community/a91846/search-form-above-message-list-where-it-is-looking-provided-keywords-recipient-sender.aspx 

IMHO 99% of the time users are looking for certain sender emails or names so allowing mail admins to search form to look only in email addresses FROM fields would speed things up very much

You should all also check that your searches, if successful, show all results that should show. I'm finding that many searches won't find emails after a certain date. For me, it's September 19th, or thereabouts.

We, too, upgraded from 15 to 17, and are also finding the simple search has suffered.

+1 for experiencing issues with search. 🙁

That said, the recent update did fix the webmail simple searches.

Installed version 7242, now search is fast. 3 seconds for a simple word.

Good Job ST !

Giorgio

Our search has always been quick, though we do use a quite large SSD-based CacheCade for our SM RAID.

Hi,

We upgrade to v17 this weekend, and we have a client with 80GB mailbox and the index its externally slow and dont find most of the emails.

Regards,

 

We`ve found that at least for our users with very large mailboxes (30-50+ GB), simply issuing an re-index for that mailbox was not sufficient - users would still not be able to find some emails. What we did in order force a real, full re-index, was to delete all of the existing indexes manually and then force the re-index. After this, all emails would come up in a search. We did this a while back and do not remember the exact steps we did (if we stopped the SM Service, etc), so please do not do this unless you know what you are doing and unless you can find what are the exact correct steps.

However, even with the mailbox completely indexed, there is still an issue that I wrote again in the past: normal, in-folder search is much, much slower than Advanced Search. 

We love the Advanced Search and do not want it modified in any way in terms of performance (except if this can be further improved :) ). However, it would be good if the Normal Searched was at least as fast as the Advanced one.

Hi, we have create 20 folders on the inbox and for each folders about 10 subfolders.

We have move many mail into these (folders and subfolders) and reindex.

The mailbox have 16 GB data.

The result is a very slow search also in subfolders with 30 mail. If I search a simple word it takes about 20-30 seconds.

It seems that when I click on a single folder SM search in all the folders and subfolders present in inbox.

ST can you investigate ?

In another mailbox 15 GB, only inbox folder, the search is almost instantaneous.

Thanks

Giorgio Borra

Yes we experience the sluggishness as well. It is extremely slow.

Also in the advanced search we can't do anything with the result except open one by one we need more functionality once you get the results. Only delete is available. We need the "more action"  button (the one with the 3 dots) in the main result window that would apply when multiple items are selected. Like forwarding, marking as read. Flagging, etc...

Lets keep positive that ST look to this issue in the future, We have a few client very unhappy with this, asking why an upgrade make the webmail worst.

Most of our clients and the people from my team are "happy" with SM15 (after a short but worrying trial of SM16, 2 years ago). From time to time, I think about V17, but, for now, I don't see the point of it.

According to my opinion, SM15 webmail is still a very effective professional "desktop" webmail. A popular webmail like yahoo webmail has nothing to envy, etc... And we do not need a fully responsive design interface for a "desktop" webmail.

I have a question for ST. Would it be possible to develop SM15 software and what would be the limits ?

ST is not going to keep developing on a now almost 4 year old product. Makes no sense. Focus development time and man hours on the current product and the future. You dont see the point in upgrading to a newer version, but want them to continue developing and improving v15, which they have... It's called SM17.

echoDreamz. 

Of course, I understand development time and man hours. It's the same for me in my own company and for everybody. But 4 years later and 2 releases ahead, there are still issues with slowness... Is it because of the technology ? About .NET, from 1.1 to 4.X... I know it quite well for my developments. And I know it is not so easy. Performance traps may occur. Is it because of file formats ? 

I just would like to understand. I often read the posts here waiting for The moment to upgrade.

Alain Néris, Maybe you could iterate more on specifically what kind of slowness are you seeing? We have made some changes that relate to speed on a system. Here's some.

-Internally we've reduced the number of locking between threads which allows many aspects of a user's account to be concurrently accessed, reducing the amount of spinners a user may see.
-We have a new internal system for handling all of our data, this allows us to more easily make file changes and control behavior consistently. (grp's-cfg's, stats, indexing are an exception to this)
-With this system we can now in many cases completely cache files internally in memory so that we don't need to read them from the file system.
-We implemented changes that allow many changes at the same time (or in a short period of time) to be rolled into ONE write to the disk, delayed cache writers.
-With the beta we are introduced a message caching mechanism allowing us to intake, move, and read thousands of messages in a matter of seconds. While messages are in this state they can be modified, indexed, accessed by push-based protocols, loaded in webmail and scrolled all concurrently at the same time.
-in Beta, we implemented CONDSTORE for IMAP allowing much quicker syncs for clients with large mailboxes, and consequently reducing load on the system.

-We've also made a couple front end changes to calendar loading, as well as management lists like Domains and Users taking a while to load.

Those are some quick off the top of my head changes that we've recently made relating to speed. There is still more we can improve. Indexing is something that we will look at but do to how large of a system it is, we couldn't weight MAPI, mailbox, other protocols, and all the other beta stuff at the same time until we get this beta finished.


EDIT: To specifically address this thread. For now, rather than requesting a new thread since this is technically resolved, I will just take it back to Known until we have the chance to take a look at this again and a bit harder. We may find some more aspects to speed up.

In our environment specifically, we've had zero complaints about search speed, infact, the opposite, especially since mid Jan or so, the beta builds have been awesome for webmail clients, they report very quick performance across the board.

I will repeat myself from various topics:

Search form above message list should be configurable by mail server admin. By configurable I'm thinking about some checkboxes which will enable/disable searching in message sender, receiver, title, message body etc.

Example:

Using advanced form where I've provided variables:

1. FROM some email

found messages right after pressing search button.

Providing the same keyword in field above message list performed search operation in few minutes. Seriously for most people looking for messages they are interested in finding messages from some sender and not a keyword inside body of those messages. For this kind of search operation advanced search form is there.

EDIT: I'm using 7242 build.

Hi Matt,

thanks for reply.

I have installed SM17 in current build 7242. 

Please Matt can you try to create for example 10 subfolders in Inbox on a mailbox with at least 10 GB occupied space, and move many mail from inbox to that folders. Than do the same on Sent Items, create 10 folders and move many mail from sent items to that folders. After that reindex and try to search something in inbox. 

In my case, and from my customers with other servers and SM installations, the slowness is present when there are subfolders in Inbox and in Sent Items. 

Thanks

Giorgio

Mat Petty, As I wrote above, I am actually a SM15 user (after a short SM16 experience, 2 years ago). I have no doubt that SM17 (and the latest release of it) is better than SM16. I appreciate your efforts in designing and programming the best SmarterMail. 

Thanks for your detailed explanations. I learn a lot specially regarding your multithreading system avoiding readings and writings from and to the file system. I came up with almost the same strategies to face almost the same problems in my own developments. I do not know how you manage file attachements (large and frequent files such as jpg, pdf, and so on...) ? I had to deal with this problem in my image bank indexes, and finally I chose to read only once the image(.jpg) at the creation process of my galleries (to get image properties in my case), to store needed informations of the image in xml files and xml collections to be used later. But this may have nothing to do with your system.

Alain Néris, We store all raw content of an email in a grp file then we keep things like ID's, change numbers, last modified times, etc in the CFG file. Our code selectively reads and writes only specific parts of this file as it's being used. This allows us to dump a lot of data into a single file but still read and write it very quickly. Another method used by some databases, I was reading about RavenDB yesterday. How they store large binary objects is that they allow you to associate binary blobs to objects in the DB. When you read from their DB you get metadata about the binary data and it gives you a way to specifically pull the binary data when needed while still keeping it with the DB object.

Going more into our delayed-writers. Since we have a memory version of most files we can immediately update the values in memory so the rest of SmarterMail see's the change then about 250ms to 2 seconds (different for some files) later we commit the file to disk. We are also told by the system if the process needs to shutdown, allowing us to commit these files quickly to disk. This does bring up an interesting case though, like what happens in this 250ms to 2 second window where power could get cut to the server and whatever was changed is lost, this only affects json files though. GRP and CFG are pretty rugged to the system randomly shutting down, this is stuff we've improved lately, how the system reacts to a unexpected shutdown...


Giorgio Borra, I have a large account that I can split into many folders. I'll give that a test at some point today. Based on this test I will create a task on our board to have it looked at. We are in a period right now where we have effectively halted large changes in favor of stability and finishing up what's left in MAPI and fixing reported bugs community and tickets.

Hi Matt, 

I hope you are all well. 

In this period we have many users who work remotely and use the SmarterMail web interface. Several companies where SmarterMail is installed complain about slow searches of emails. Is it possible to investigate the problem as I have already reported to you?

Thank you

Giorgio Borra

The new BETA includes some more improvements for search performance.

We are also reviewing some potential, extensive changes we may be able to make to improve the search performance. Those changes are currently under review.

This weekend just gone, we upgraded from the latest release of the previous SmarterMail 16 to the latest public SmarterMail 17 release (build 7242 as at now).

We start getting complaints from users that search has "stopped working" (actually working, but might take 60 seconds or more to get results that used to be fairly prompt in v16).

The SmarterMail indexing is marked as complete, and the users are on the new IndexV2 format already.

Imagine: Google can search complex queries across an index of the WHOLE internet, in milliseconds.  SmarterMail v17 won't let us look up a full-text search index for a measly few GB of mail within a minute.

What are we best to do?  The configuration has changed to JSON : is there still an easy way to revert back to SmarterMail v16 so our users can function?  Or is there a beta or new release that addresses the appalling degradation in search in v17?   Or is it a matter of legacy configuration that comes through during the upgrade, and there is something we can delete, etc, to get the search to use a fast index?

Please help!!  Users rely on search to get through their 1000s of emails.

My honest advice is to revert to Smartermail version 16 if you have backups - we've been unable to resolve the issues with the slow responses on the latest version of smartermail and these have been on going for months.

Hi ST, 

in the past 2 months we received many complaints about slowness in webmail search of SmarterMail v.17

Many our customers are using webmail instead of Outlook, because they are working from home in smart-working.

Do you have a solutions ?

Why the previous v.15 was almost instantaneous in search ?

I Hope you can help us because I think there are many users in the world with this problem.

Giorgio Borra

We have no new version for a long time, I wrote already that SM should release new versions with a note that the MAPI still beta and shouldn't be open for clients. There are alot of small bugs that are corrected in the beta version that not are in stable version.

that's exactly what ST has done... read the Production Beta post, please...

@Sergio... they did... did you read the beta post?

You do not need to enable MAPI. We've only had enabled enabled for our main domain and a few tech's and their personal domains.

Can anyone who has installed a later "Production Beta" (e.g. 74xx or higher), comment on whether the web search is faster in the beta version than it is in the [ "Current Build" Build 7242 November 1, 2019 ] which is still marked as current as at May 2020?  Or are SmarterTools personnel/developers able comment?  We really need to resolve this for our users.  Is the beta faster, or is web search still a major issue?

I can confirm the slowliness even in the latest Production Beta, but it's better than in 7242.

Ok thanks for the clarification, can I use my keys or I need to ask for a beta key?

Use your own key.

The BETA key is needed to test MAPI if you won't/can't purchase it

Use your own key.

The BETA key is needed to test MAPI if you won't/can't purchase it

I agree, the webmail search speed is abysmal, and I've already lost a large customer because of it -- just a couple months after upgrading to "Build 7242 (Oct 30, 2019)". I'm unlikely to lose another customer because of it, but I *am* bringing on another small town hall with about 35 staff. It would not look good if they moved to my service and the webmail search was unusable.

CRITICAL IMPORTANCE!

And yes, this search think make a big deference for one of our customers to. Probably they will leave to Google.

Maybe I'm repeating myself but can someone from you who is still using SM v15 check if search form is searching also inside message content?

I'm asking because like I wrote in many places under similar discussions IMHO users are looking for messages sent by certain email address or which have certain title and thats all. Looking for messages containing something is less needed (still IMHO).

I just performed on my mailbox search using advanced search where I've selected searching inside mail and used From attribute, Title attribute, selected that any condition must be fullfilled (so keyword should be inside From or Title or both of them) and provided keyword. Messages containing provided keyword popped right away and thats great because it should work this way. When I provided the same keyword on quick search form above ONLY Inbox section above message list search form finished search after I don't know maybe 100-150 seconds (I was writing this post during this time).

Seriously I'm proposing this for long time. If SmarterTools don't want to make this as default then PLEASE allow SmarterMail administrator to select where keyword should be looked for when user is using quick search form. This would save me some tickets from my customers which complain about search speed. Even if client would start to complain that this search form is not finding keywords inside email contents then I could just point him direction to advanced search and thats all.

So bottom line: search looking keyword without message content which takes 1-2 seconds vs search with looking inside message content which takes 100-150 seconds makes at least for me easy choice.

Webio 

I am still using SM v15. Search form is searching also inside message content.

We have upgraded to the latest (as at today) beta, # 7447 May 22, 2020 and can confirm the slow web search is still there in this latest beta version.

Running procmon64.exe, we see:

Within the first half second, it runs some file activity against all files in this user's IndexV2 folder (in this case 91 files, 1.4GB of index)

** Then the CPU for MailService.exe goes to 100% on one core for a period of 25 to 90 seconds, with no visible disk I/O **

Then it reads all the applicable *.grp files near instantaneously.

So, I imagine there is a serious bug in MailService code such that:

* Results matching search query returned very fast from Lucene index

* BUG: where SmarterMail performs some sustained CPU iterative operation, possibly on items returned from index or it is locked in some endless loop or has met a race condition.

* Then actual results are read from *.grp file and summary returned to user

It is sooo painful, this is awful. 

Hi SmarterTools, 

today I lost a customer. 

They chose to migrate email to Office365 with another company, and this is truly unpleasant.

Now is very urgent to fix webmail slow search in the current build (nov.2019...)

We can't wait for the beta version to go into production. 

If you can't (or don't wont) to fix the current version, we will be forced to migrate our customers to other products.

Please let us know something.

Giorgio Borra

In my case, I have a 2 gig mailbox (keep things fairly clean).  Searching my mailbox takes about 9 seconds the first time and then takes 3 to 4 seconds in subsequent searches because it is in memory.  

The machine is a VM with average hardware for the number domains and users it hosts so it is comparable to most of our customers environments.

These tests were performed with our BETA that has some improvements but is the same underlying systems.

Bottom line, the better the disk I/o, CPU, and the amount of memory you have on your machine will dramatically change the results.  If you want more accounts to be in memory to have better results, add more memory to your machine.

If you believe your hardware isn't an issue, open a ticket and we will try to help you out.

So, I guess you're saying V.17 requires more resources than V.15, which had a very quick search capability.

Seems a little backwards. I would think that each version would get faster, stronger, more lean. Maybe the new filing / indexing system needs a review?

Hi Tim,

2 GB its a very small mailbox. let you mailbox grow a bit more and you start feeling this and another problems, like the Active sync complete resync.

It was after the upgrade of v16 to 17 that the search slow down, and we start receive complain few days after the upgrade. Same hardware, same customers.

I know that your focus is MAPI but there a few stuff that need attention, one of them is this search issue.

Regards,

Replying to Tim's note above, and confirming it is not a hardware issue.

The problem is not with memory or disk i/o.  For our scenario as investigated with procmon above, our server is running with 50% memory free, and during the search being 'locked-up' on CPU there is no disk i/o.  The indexes are only accessed in the first second.

In version 16, the search was near instant.

In version 17, it takes over one and a half minutes for the same search. 

Same hardware, same mailbox size.

Version 17 search is slow on our system and is vastly different to version 16.

Our users reported the search is broken immediately after upgrading. 

People will not wait one and a half minutes staring at a screen, waiting, for every single email they look for. 

Please contact support.  We would like to evaluate.

Yes thank you Tim - we opened a ticket earlier in the week, and so hopefully they find something or it gets escalated to your dev team and resolved. We'd like to see this all working for everyone :)

Indexing and Searching have been dramatically improved in the next release.  Mailboxes will need to re-index and there will be some additional CPU when this build is installed.  But when that is done, most searches will be under a second and we will use less CPU and less Memory no matter the size mailbox.  The first search will be slightly slower but follow-up searches which will be almost instant.

That is AWESOME. Thank you so much for fixing that!

Thank you Tim, it is absolutely wonderful ! 

We wait this for long time. 

When will be available for download ?

Giorgio

Did you guys add the ability to 

Hi all,

I can confirm that the latest improvements to Search are simply amazing - both normal Search and Advanced Search are almost instantaneous even for 40GB+ mailboxes! We`ve struggled with this for a long time and the latest version is simply like a breath of air after being too long submerged underwater :)

Now, the next step would be to maybe have just 1 place you do your searches - the box where you now perform the normal search, but maybe add a small slider next to that to switch between "search in this folder" or "search in all mailbox / all folders" - this would be much more intuitive and easier for everyone to use. Sometimes people ask me how they can search in the whole mailbox as they do not know there are 2 kinds of search.

However, the main takeaway is that the search system has been overhauled and is now very fast.

Thank you for the hard work!

Do we have a release date for this upgrade? It's been over a month since it was announced.

Already released.

Yeah, the indexing changes were published in the July 10th release (7496): 

That made searching across the board infinitely faster. So if you upgrade to that release, or any release after that, you'll have that efficiency change.