¶ JOIN SEARCH
From version 6.16 of JOIN Case & Document, we use an external search engine. Below, an explanation is given about this search engine (based on Elastic Search) and all search functions are explained. The search function “global search” has been discontinued from version 6.16, while at the same time the full-text search function (under the more menu) has been replaced by the new search function “search in all”.
¶ Elastic Search
JOIN Case & Document makes optional use of an external search engine since version 6.16. This search engine is based on Elastic Search. This is an open source project in which a search algorithm has been developed that is comparable to the powerful search engine of Google. The algorithm is characterized by the fact that it combines various parameters (and “boosts” certain search terms) and can thus provide much better feedback to the end user in the correct order (relevance of search results). In addition, Elastic Search is enormously scalable, which guarantees performance at a very high level with large amounts of searchable data.
JOIN Zaak & Document also uses this algorithm for full text search (“search in all”). All relevant information of registrations and files (file content), combined with the authorizations of users and user groups, is stored together in an index optimized for search. The search algorithm uses this index and tries to return search results that are as relevant to the end user as possible.
The new index that is used for searching within JOIN (and which is therefore optimized for search) also has, next to a faster and more relevant display of search results in the full-text search screen, advantages for searching in lists (books, work list, cases and case documents). Think about:
- books with large numbers of registrations no longer require restrictions: sequence number blocks and other filter restrictions are no longer necessary
- the case warehouse always shows the most recently created cases, regardless of the case type they are registered as. Lists load very quickly.
- No case and document type needs to be entered for case documents. You can search freely for any term.
NB: If your organization does not use JOIN Search, the interface is identical. Only the way of searching and the speed of searching can deviate from what is described here.
NOTE: from version 19.4.627 we use Elastic version 7.3. The manual for updating can be found here: JOIN - Search function update Elastic 19.4.627. This manual assumes an installation of Elastic Search on the JOIN application server. If you want to install Elastic on a separate server, we ask you to hire a technical consultant from Decos.
¶ search in everything: full text search
An important search function in JOIN is “search in everything”. This is a function that is comparable to Google and searches the entire system (JOIN Case & Document). The main features of this search function are:
- search from 1 window
- simple and clear display of search results with standard filters
- quick search function
nb. please note that the search function performs significantly less well when you are not using the external search engine. For more information about the external search engine, please contact Decos.
¶ Search in everything: the search box
You will find the search box at the top right of the screen. This search box is always available, even if you are in the work inventory or dashboard.
If you click on the search box, JOIN will display your 5 most recent search queries. This list is always up to date. You can select one of these searches or enter your own search query. This search can contain one or more search terms. Keep in mind that the search is always standard with AND (so that the result narrows for multiple search terms). The search terms do not necessarily have to be complete words or be exact in some other way, but it is not possible to use other operators.
It is also worth mentioning that it is possible to search for parts of words. The minimum letter combination is 3 and since version 6.22 the maximum is adjustable (up to 100 characters). This can be set by the JOIN administrator in your environment.
When the search is completed, the result screen opens. The search algorithm determines the order of the search results. Relevance is determined by combining several criteria. Some examples of these so-called “boosters” in the search algorithm:
- where does the word occur. If the word is in the description of, for example, a case, it will receive a higher score than if it is in another field
- when was the document or case registered. Recent registrations receive a higher score
- how often does the word occur and (with multiple keywords in 1 assignment) what is the distance between the words. If the words are close together, the relevant registration will automatically receive a higher score.
The search result is shown in the overview page. You will notice that this happens very quickly (milliseconds) and Elastic’s search algorithm ensures that the most relevant information is shown at the top of the screen as much as possible. The relevance is determined on the basis of, for example, date (last change) and the place where the terms were found (in the description or, for example, the file)
In the search result screen, various filter options are available by default when using JOIN Search:
- A date filter to be able to filter on all registrations and files that have been modified in a certain period.
- A filter on the work list to search for specific actions (active, completed, etc.) that the user has access to
- A filter on files, so that the result is displayed specifically within files
- Flexible filters: depending on the search result, the objects and books that contain the result are displayed. Here you can further filter per book or per object to show the correct result.
From the result screen you can directly open the case, file or address. The file will be opened immediately, the file will be shown as a pop-up, etc. After closing the pop-up you will immediately return to the result screen.
Search in everything: the search options
Several search combinations are possible in the search box:
Search for a single word or part of a word: This will show results that match this word, as well as results where the search is part of another word. If you search for ‘bike’, we will show results with bike, balance bike, bike bell or biked. If you search for part of a word, for example ‘bik’, you have to make sure that the keyword is not smaller than 3 characters. Depending on the settings in the search engine (check with your application manager what the so-called “ngram” settings are) you can search for parts of words between a minimum of 3 and a maximum of 20 characters. See also the section “config editor” at the bottom of this page.
Multi-word search: You can also search for multiple words. For example JOIN Search. The results of this search are all registrations where in one of the fields (e.g. description or the content of a PDF file) this combination of words is found. We also show results with, for example, JOIN introduces Search or Search JOIN. Both words are present, the order or position in the relevant field has no restrictive influence on this.
Exact search: finally, you can also search exactly for certain words or combinations. You do this by placing the search between double brackets. For example, if you search for “JOIN Search”, we will only return results that exactly match this. JOIN introduces Search is then excluded from this. You can also combine exact searches. For example, if you search for “David Bronsveld” “Joost Went”, we will only show results in which both exact combinations occur in 1 field of a registration. Registrations with only “Joost Went” are excluded.
¶ search in lists: filter
In JOIN Case & Document you can search in books (also called lists). In each book you will find a list, where you can apply a search (filter) per column. If desired, you can combine multiple filters at the same time (search on different columns) or apply multiple filters within the same column. Here you can use so-called “operators” which are explained later in this wiki.
When you enter a value in the column and press [ enter ], the filter is applied and the search is performed.
You do not necessarily have to search for complete words, but you can also enter part of the word (a letter combination). For example, if you enter “eco”, you will get the results “ecology” and “decos” as search results. Keep in mind that regular text fields must be searched for at least 3 characters. If you search for fewer characters, the search engine will only show results that exactly match this. This restriction does not apply to date and numeric fields (as well as the sequence number). With the “clear filter conditions” button you can clear your filters with 1 action.
With JOIN Search you do not have to use AND when searching, it is automatically added “under water”. In addition, with JOIN Search you benefit from searching for words with accents. For example, if you are looking for the word “René” but search for “Rene” instead of “René”, the new search engine will show all results with or without é. Without an external search engine, only the results that exactly match the search query are shown.
¶ Search in lists: filter options
Filter options can be used in both full-text search and filtering in lists. In the full text search screen (where all data (including content of documents) can be searched) this is an exact search (search between “”), while in the lists (books and work stock) an extensive list of so-called operators is available.
Below are all operators that you can use when filtering in the lists (books). This allows you to make comparisons, set date ranges, etc. For more information or help, you can always contact the JOIN specialist of your organization. There are some minor differences between JOIN Search and the old database search. The differences are listed below.
Search methods
| Input | Example | Result |
|---|---|---|
| AND | Paul AND Veger | Both words are in the result: Paul Veger |
| OR | Paul OR Veger | One of the words is found in the result: Paul Hendriks / Bert Veger |
| AND NOT* | Paul AND NOT Veger | First word is found in the result, the second isn’t: Paul Hendriks |
| Value only | Piet | Everything where Piet is found: Pietersen / de heer Piet Jansen |
| Date | 1-3-2014 | March the first 2014 |
| Simple comparison | > Decos | All values alphabetically after Decos: Decos Information Solutions / Ded… |
| Number | > 5 | All numbers higher than 5: 6 / 7 / 8 |
| Number | >= 5 | All numbers equal to and higher than 5: 5 / 6 / 7 / 8 |
| Number | <> 5 | All numbers smaller than 5 or bigger than 5 : 1 / 2 / 3 / 4 / 6 / 7 / etc |
| Number | < 3 OR > 6 | Any number smaller than 3 or gibber than 6: 1 / 2 / 7 / 8 / etc |
| Date | > 1-3-2014 | Any date after March 1st 2014: June 12th 2014 / August 30th 2014 |
| Date | Date() | Current date (today) |
| Date | >Date() - 30 | The past 30 days |
| Date | > 1-3-2014 AND <1-6-2014 | All dates between March 1st 2014 and June 1st 2014 |
| Date | 01-2014 | All registrations with a date between january 1st and january 31st, 2014 |
| IS NULL | Applied in filter bar | The specific filtered field is empty in all search results |
| IS NOT NULL | Applied in filter bar | The specific filtered field is not empty in all search results |
| Exact value | =“JOIN” | Everything where just the word “JOIN” is visible. This can also be combined with other words. It’s also possible to put an entire sentence between brackets (like “JOIN Search”) or to comnine exact search queries (“JOIN Search” “Case system”). |
- The NOT function cannot be used in lists when using JOIN Search
- Exact search with JOIN Search works without the = sign. Search the database with the = sign. This function can also be used in full text search.
You can also provide even more complex searches by combining AND and OR. Keep in mind that AND takes precedence over OR.
The search query “Paul AND Veger OR Taalman” is executed as “(Paul AND Veger) OR Taalman” and thus to results containing Taalman, or both Paul and Veger.
¶ Search in lists: advanced options
In addition to searching in the list, JOIN also offers an advanced search for related information. For example, advanced search can search for the following information:
- extensive content / notes
- linked addresses
- linked documents
- linked dossiers
- linked meetings
- (workflow) activities
- linked objects / projects / contact moments (depending on your configuration)
You can find the advanced search function via the button at the top right of the list. Enter one or more filters per object and press OK. The search can also be combined with filters in the list of the book where you want to perform your search (for example, when you are looking for a case of a certain date, where the linked person is called “Jones”. You can also use this here. of the extensive filter options (AND OR etc.).
If you’re missing data to look up, you can request this from the functional manager of JOIN, as this can be altered in JOIN.
.png)
Advanced search does not yet work in combination with JOIN Search.
¶ Search in lists: filter in cases
If you work with the JOIN case system (based on JOIN Case types), you can search for cases in an improved way from version 6.16. All the functions mentioned above for searching through lists also apply for searching for cases. In addition, when opening the “cases” list, the last 10, 20 or 50 cases registered in the case system are displayed. This is regardless of the case type in which the case is registered, but of course including the case types for which you as a user have at least read rights.
NB: If you do not use the external search engine, the above does not apply.
nb. This function only works in conjunction with JOIN Search (Elastic) when the “filter bar behavior” setting in the config editor is set to Automatic or Always.
¶ Search in lists: filter in case documents
If you work with the JOIN case system (based on JOIN Case types), you can search through all case documents easily and quickly via the “case documents” section, regardless of the case type and document type they are registered under.
If you use JOIN Search, there are no search restrictions, such as pre-selecting a case type or a document type.
As with cases, the last 10, 20 or 50 (depending on your settings) case documents are shown that were added to the case system last. Regardless of the case type and document type to which they belong, but of course the case is taken into account. - and document types to which you as a user have rights. You will only see protected documents if you have explicitly been given access to them.
NB: If you do not use the external search engine, the above does not apply.
NB: This function only works in conjunction with JOIN Search (Elastic) when the “filter bar behavior” setting in the config editor is set to Automatic or Always.
¶ Configuration: JOIN Config Editor settings
The following settings are available in the JOIN Config Editor:
| institution | Description |
|---|---|
| Date range filter field | in this field you can set to which date the date range (in the full text search) responds. Here you can choose between the “date last modified” (item_timestamp) or the “creation date” (item_created) of, for example, the case or document |
| Elastic Search index name | The name of the index for Elastic Search |
| Elastic Search password | The password to login to Elastic Search API. This only works in combination with an Xpack license |
| Elastic Search Server URL | The Server URL where Elastic is running. Elastic can be installed on a different server than the JOIN Server (such as a central Elastic Server that other systems also use |
| Elastic Search username | The username required to login to Elastic Search API. This only works in combination with an Xpack license |
| Enable Ngram analyzing | This activates the Ngram analyzer. With Ngram it becomes possible to search on parts of words in both full text search and in list search. Keep in mind that the size of the search index increases when using Ngram. Later in this list you will find the settings for the minimum and maximum number of characters for searching by parts of words |
| External Search engine | The external search engine used. Currently only Elastic Search is used |
| External Search Engine ServiceTCP Channel | The TCP channel for the external search engine with which to communicate with the JOIN External Search service. |
| Fields to exclude from indexing | When certain fields may not be displayed (or searched for), this field can be excluded from indexing. These fields are therefore not available in the external search index. Consider, for example, the BSN (Social Service Number) field |
| Filter bar behavior for external search engines * | In this field you set whether and how displaying and filtering in lists uses the external search engine |
| Max number of items in aggregations | in the full text search screen, the search results list can be further refined by using filters on the left side of the screen. This setting determines how many filters can be shown per item type (for example, different case types under the “cases” section. |
| Maximum Ngram analyzer length | When searching for words (in both lists and in the full text search), parts of words can be searched. The technique used here is called Ngram. If this setting is set to 20, you can search for parts of words up to words with a maximum length of 20 characters. |
| Minimum Ngram analyzer length | the minimum number of characters to search for when searching for (parts of) words. |
| Reindexing start time | time (Night time is advised) at which the search index is updated. For example, authorization changes are always updated at night and not in real time |
| Reindexing end time | end time of the JOIN Search indexing process |
| Service call time out (seconds) | When set to 30 seconds, the request to the external search index service is canceled. |
¶ Configuration: filter bar behavior
JOIN has many lists:
- the case book with all sub-case types
- the contract or invoice book
- the work list
- etc. etc.
In the past, various filter restrictions had to be used to maintain the performance of these lists (both for displaying and filtering them). That is a thing of the past with the introduction of JOIN Search. The setting “filter bar behavior” determines whether JOIN Search is used in the relevant list (book / work list). Because this can also lead to a difference in functionality in some cases, a summary is given below:
| Sr. no. | list | setting “Never” | setting “Automatic” | setting “Always” | |
|---|---|---|---|---|---|
| 1. | work list | the work list retrieves its information from the database | the work list retrieves its information from the database | the work list (get list and filter) retrieves its information from the search index | |
| 2. | normal book <10000 records | the book retrieves its information from the database | both filtering and retrieving the list uses the database | the book (list retrieval and filtering) retrieves its information from the search index | |
| 3. | normal book >10000 | the book retrieves its information from the database | the book (list retrieval and filtering) retrieves its information from the search index | the book (list retrieval and filtering) retrieves its information from the search index. | |
| Performance restrictions are needed to maintain performance (“order by sequence number, limited search indexes”) | |||||
| 4. | hierarchical book (such as “cases”) | both the main book (cases) and the subbook (case type) retrieve their information from the database. | from the “top” level (eg cases) information is retrieved from the search index. The last 10,20 or 50 registrations created (regardless of the sub-book (case type) are displayed immediately | The search index is used both at the main level (main book “cases” and at the sub level (case type). | |
| At the top level, the content is only shown after a filter command | filtering is done based on the search index. There are no performance restrictions | There are no performance restrictions and lists are displayed directly at every level | |||
| The content is shown directly at sub-level | if you select a subbook (eg a specific case type), information is retrieved from the database. Filtering is done based on the database. | ||||
| 5. | Case documents | both the main book (cases) and the subbook (case type) retrieve their information from the database. | The search index is used both at the main level (main book “cases” and at the sub level (case type). | The search index is used both at the main level (main book “cases” and at the sub level (case type). | |
| at top level, the content is only shown after a filter assignment in combination with the mandatory choice for case type / document type | There are no performance restrictions and lists are displayed directly at every level | There are no performance restrictions and lists are displayed directly at every level | |||
| At the sub-level, the content is only shown after the mandatory combination of case type / document type |
¶ additional comments:
- There is a slight difference in the filter functions between searching the database and searching the external search index. Consider, for example, not being able to use NOT in the external search engine or not having to enter AND when using the external search engine.
- The default setting is AUTOMATIC. As of version 6.26, this setting will no longer be available and there is only a choice between “on” (ALWAYS) and “off” (NEVER)
- The external search engine service uses the JOIN Background service. If it is not running, it is also not possible to show or filter data from the search index.
¶ Support: Troubleshooting
Below is an overview of common error messages and the corresponding solution:
| Sr. No. | error message | Solution |
|---|---|---|
| 1. | The search index does not match the current software version. Ask an administrator to rebuild the index | Check whether the Decos JOIN services and Elastic Search background service are all started |
| 2. | Elastic service does not start | Sometimes ElasticSearch will not start. In the log (… DECOS \ apps \ Elastic \ elasticsearch-7.3.1 \ logs \ elasticsearch-service-x64-stdoutXXXX-XX-XX.log( there is a message like this: 2020-03-05 11:27:52 Commons Daemon procrun stdout initialized ERROR: Temporary file directory [C:\Users\MIGRAT~1\AppData\Local\Temp\5\elasticsearch] does not exist or is not accessible. Even though ElasticSearch runs under the Local Machine account, Elastic wants this temp folder to exist. By creating the subfolder and subsubfolder mentioned in the logging, you solve the problem after which Elastic can be started. In this example the folder 5 and the subfolder elasticsearch |
| 3. | The registration cannot be saved in the Search index because there is not enough storage space on the server | If there is less than 10% disk space available on the disk to which Elastic Search (ES) saves its index, ES may shoot into a so-called soft lock and refuse to run anymore. First, disk space must be created and then the following steps can be performed, to remove the soft lock, follow the steps below the table. |
- On the server, go to kibana.bat. This can be found in E:\DECOS\Apps\Elastic\kibana-7.X.X-windows-x86_64\bin (depending on the disk of installation and version and build number Elastic). Run these as Administrator. This may take a while.
- The version number in this example is old. Check with support for the latest supported version of Elastic.
- When everything is running, start the browser and go to http://localhost:5601
- Go to dev tools on the left (see image)
-
- To check if ES has really locked: first enter get decos_itemindex/_settings where other code is now. Then select the line of code and click the play button. On the right side you will get a list of lines, in it is the line: readonly_allow_delete: true;
If this line is missing, this is an indication that a soft lock may not be present. Thus, this line may have been deleted by ES.
With the following script we ensure that the lock opens (after pasting the text below, also select everything again and click the play button):
PUT decos_itemindex/_settings
`{
“index”: {
"blocks": {
"read_only_allow_delete": "false"
}
}
}`
If sufficient disk space is freed and the ES service is restarted, ES itself may realize, that sufficient disk space is free again and restart. The bulk indexer may need to be run, to add previously entered items and documents to the index again. This can be done with a date option, when the failure occurred, to shorten the processing time.