Elasticsearch search engine¶
Elasticsearch is an open-source, distributed, Java-based search engine that responds to queries in real-time and is easily scalable in reaction to changing processing needs.
For a detailed description of advanced settings that you might require in a specific production environment, see the documentation provided by Elastic, beginning with the Set up Elasticsearch section.
To proceed you need to be familiar with how indexing, filtering and queries work.
Step 1: Download and install Elasticsearch¶
Install Elasticsearch on your server. For example, use the following Docker command:
Ibexa DXP supports Elasticsearch in version 7.16.2 or higher.
Step 2: Verify that the Elasticsearch instance is up¶
To make sure that the Elasticsearch instance operates properly, access the instance (for example, with
If Elasticsearch operates properly, an object with cluster details is displayed. It should be similar to the following example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Step 3: Set the default search engine¶
Set the following environment variable (for example, in the
Step 4: Configure the search engine¶
Ibexa DXP comes pre-configured to work with an Elasticsearch cluster that uses default settings, and you can use this initial setup for testing purposes.
However, to effectively search through actual data, you must provide specific settings.
All configuration is made in the
All the settings, their order and meaning, correspond to the settings that are described in the Elasticsearch documentation.
First, decide how Ibexa DXP connects to Elasticsearch and configure other connection settings. For more information, see Configuring connections.
Then, define a field type mappings template that instructs Elasticsearch to interpret Ibexa DXP fields as specific types. For more information, see Configuring field type mappings.
Step 5: Push the templates¶
For each of your defined connections, push the templates to the Elasticsearch engine by running the following command:
You can modify the behavior of the command with a number of switches. Use the
-h switch to display a complete list of available options.
Step 6: Reindex the database¶
After creating index templates, run the following command to reindex your data:
Risks of premature indexing
Do not reindex your data before you create index templates. Otherwise Elasticsearch attempts to use its dynamic field mapping feature to create type mappings automatically.
When you configure the Elasticsearch integration, you must first configure the connections. You either connect to a cluster of Elasticsearch nodes or the Elasticsearch Cloud.
Define the connection settings under the
connections key. First, set a name of the connection:
1 2 3
Then decide whether to add a cluster that you administer and manage yourself, or use a cloud solution from Elastic, as well as configure additional parameters. You can then decide how the cluster handles communication with individual nodes, and configure the security settings.
A default connection
If you define more than one connection, for example, to create a separate connection for each repository, you must select the one that Ibexa DXP should use with the following setting:
1 2 3
A cluster consists of a number of nodes. You might start with just one node and add more nodes if you need more processing power.
When you configure a node, set the following parameters:
host- An IP address or domain name of the host. The default value is
port- A port to connect to. The default value is
9200. If you have several Elasticsearch instances that run on the same host, and want to make them distinct, you can change the default number.
scheme- A protocol to be used to access the node. Default value is
path- By default, path is not used. The default value is
null. If you have several Elasticsearch instances that run on the same host, and want to make them distinct, you can define a path for each instance.
pass- Credentials, if needed to log in to the host. Default values are
List the addresses of cluster nodes under the
1 2 3 4 5 6
There are several ways that you can use to pass host parameters. The easiest one is to pass them as a string:
You can also pass the host configuration as an object that lists parameter-value pairs, for example, when your authentication settings contain special characters.
Cluster connection configuration should have the following structure:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Multi-node cluster behavior¶
When you configure a cluster-based connection, and the cluster consists of many nodes, you can choose strategies that govern how the cluster reacts to changing operating conditions, or how workload is distributed among the nodes.
With this setting you decide how a list of hosts that form a cluster is managed.
The list of active hosts tends to change in time, due to different reasons, such as connectivity
issues, host malfunction, or the fact that you add new hosts to the cluster to increase
By default, the
StaticNoPingConnectionPool setting is used.
You can change the default setting with the following key:
1 2 3
For more information and a list of available choices, see Connection pool.
Load tests recommendation
If you change the connection pool setting, it is recommended that you to perform load tests to check whether the change does not negatively impact the performance of your environment.
When the cluster consists of many hosts, the
connection_selector setting decides what strategy
is used to pick a node to send query requests to.
By default, the
RoundRobinSelector setting is used.
If you prefer a different strategy, or have created your own, custom strategy, you can change the default setting with the following key:
1 2 3
For more information and a list of available choices, see Selectors.
Number of retries¶
retries setting configures the number of attempts that Ibexa DXP makes to connect
to the nodes of the cluster before it throws an exception.
null is used, which means that the number of retries equals to the number of nodes in the cluster.
1 2 3
Depending on the connection pool that you select, Ibexa DXP's reaction to reaching the maximum number of retries might differ.
For more information, see Set retries.
As an alternative to using your own cluster, you can use Elasticsearch Cloud, a commercial SaaS solution. With Elasticsearch Cloud you do not have to build or manage your own Elasticsearch cluster. Also, you do all the configuration and administration in a graphical user interface.
To connect to a cloud solution with Ibexa DXP, you must set the
elastic_cloud_id parameter by
providing an alphanumerical ID string that you obtain from the cloud's user interface, for example:
With the ID set, you must configure authentication to be able to access the remote environment.
Elasticsearch instances support
api_key authentication methods.
You select authentication type and configure the settings under the
authentication key. By default, authentication is disabled:
1 2 3 4
If you connect to Elasticsearch hosts outside of your local network, you might also need to configure SSL encryption.
If your Elasticsearch server is protected by HTTP authentication, you must provide Ibexa DXP with the credentials. When using basic authentication, you must pass the following parameters:
1 2 3 4 5
1 2 3 4 5 6 7 8
API key authentication¶
If your Elasticsearch cluster is protected by API keys, you must provide the key and secret in authentication configuration to
connect Ibexa DXP with the cluster. With API key authentication you can define different
authorization levels, such as
index, etc.. Such approach
proves useful if the cluster is available to the public.
For more information, see Create API key.
When using API key authentication, you must pass the following parameters to authenticate access to the cluster:
1 2 3 4 5
1 2 3 4 5 6 7 8
When you need to protect your communication with the Elasticsearch server, you can use SSL encryption. When configuring SSL for your internal infrastructure, you can use your own client certificates signed by a public CA. Configure SSL by passing the path-passwords pairs for both the certificate and the certificate key.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
If you do not have a client certificate signed by public certificate authority,
but you have a self-signed CA certificate generated by
elasticsearch-certutil or another tool (for example for development purposes),
use the following
1 2 3 4 5 6 7 8
If you configure both
cert entries, the
ca_cert parameter takes precedence over the
After you have configured SSL, you can still disable it, for example when the certificates
expire, or you are migrating to a new set of certificates.
To do this, pass the following setting under the
For more information, see Elasticsearch: SSL Encyption.
In a staging environment, you can log messages about the status of communication with Elasticsearch. You can then use Symfony Profiler to review the logs.
By default, debugging is disabled. To enable debugging, you can toggle either of the following two settings:
1 2 3 4
debuglogs basic information about a request, such as request status and time.
tracelogs additional information, such as steps to reproduce an exact copy of a query.
Make sure that you disable debugging in a production environment.
Field Type mapping templates¶
Before you can re-index the Ibexa DXP data, so that Elasticsearch can search through its contents, you must define an index template. Templates instruct Elasticsearch to recognize Ibexa DXP Fields as specific data types, based on, for example, a field name. They help you prevent Elasticsearch from using the dynamic field mapping feature to create type mappings automatically. You can create several field type mapping templates for each index, for example, to define settings that are specific for different languages. When you establish a relationship between a field mapping template and a connection, you can apply several templates, too.
Defining a template¶
To define a field mapping template, you must provide a number of settings under the
The structure of the template is as follows:
1 2 3 4 5 6 7 8 9 10
Set a unique name for the template and configure the following keys:
patterns- A list of wildcards that Elasticsearch uses to match the field mapping template to an index. Index names use the following pattern:
By default, repository name is set to
default, however, in the context of an Ibexa DXP instance, there can be several repositories with different names. Document type can be either
location. In a language code, hyphens are replaced with underscores, and all characters must be lowercase. An index name can therefore look like this:
You can use the
patternssetting when your data contains content in different languages. You can create index templates with settings that apply to a specific language only, for example, to eliminate stop words from the index, or help divide concatenations. You use patterns to identify index templates that contain settings specific for a given language:
1 2 3 4 5 6
settings- Settings under this key control all aspects related to an index. For more information and a list of available settings, see Elasticsearch documentation.
For example, you can define settings that convert text into a format that is optimized for search, like a normalizer that changes a case of all phrases in the index:
1 2 3 4 5 6 7 8 9 10 11 12 13
mappings- Settings under this key define mapping for fields in the index. For more information about mappings, see Elasticsearch documentation.
When you create a custom index template, with settings for your own field and document types, make sure that it contains mappings for all searchable fields that are available in Ibexa DXP. For an example of default configuration with a list of searchable fields. To see the default configuration, go to
vendor/ibexa/elasticsearch/src/bundle/Resources/config/and open the
Fine-tuning the search results¶
Your search results can be adjusted by configuring additional parameters. For a list of available mapping parameters and their usage, see Elasticsearch documentation.
For example, you can apply a mapping parameter, in this case, a normalizer, to a specific
mapping under the
1 2 3 4 5 6 7 8 9 10 11 12 13 14
You can also set a boosting factor for a specific field.
Boosting increases the relevance of hits, for example making keywords from the title more relevant than the ones from other places of the document.
Set the boosting factor under the
1 2 3 4 5 6 7 8 9 10
You can even copy contents of existing fields, process them and then paste into another field, which than can be queried:
1 2 3 4 5 6 7 8 9 10 11 12
Binding templates with connections¶
Once you have created the field mapping template(s), you must establish a relationship between the templates and a connection. You do this by adding the "index_templates" key to a connection definition.
If your configuration file contains several connection definitions, you can reuse the same template for different connections. If you have several index templates, you can apply different combinations of templates to different connections.
1 2 3 4 5
For more information about how Elasticsearch handles settings and mappings from multiple templates that match the same index, see Elasticsearch documentation.
To learn how you can create document field mappers, custom Search Criteria, custom Sort Clauses and Aggregations, see Search extensibility.