Integrations / Platforms / Magento 1 / Indexing
Sep. 26, 2019

Indexing

In order to provide fast and relevant search, we restructure your data in a special way through a process called Indexing. The extension does this for you. It runs through all of your data - products, categories, and pages - and creates indexable objects out of it. These objects are then uploaded to our servers, either automatically via the extension’s queue, or manually via the Magento console or the command line. Once pushed to our servers, the objects go through an indexing process that transforms them into searchable data.

To learn about the indexing process, have a look at the documentation.

If you are having any issues with your data, indexes, or queue, please check our troubleshooting guide.

The extension automatically keeps all your data up to date to provide the best search experience for your users. To do this, we provide two indexing mechanisms in Magento:

  • Section reindex - An entire section of the catalog (products, categories, pages) is pushed to our servers and reindexed.
  • Single item reindex - A single resource (product, category, page) is pushed to our servers and reindexed. This happens when a resource is updated.

By default, the indexing operations run synchronously. This means the Magento administrator has to wait until the indexing process is finished before continuing. Since this is inconvenient, and can cause unexpected issues, we created the indexing queue. This will process all index operations in the background and has some fail-safes built in.

Indexing products

It is essential for e-commerce businesses to have exact and up-to-date product data in the search. To accommodate as many use-cases as possible, we provide a number of configurations for search and indexing operations.

Indexable products

To prevent too many products from being indexed, and therefore to lower the number of indexing operations performed, we only index products that will actually show up in the webshop. This results in a set of requirements a product has to meet before we index the product.

We only index products that are:

  • Visible - either in the catalog, the search, or both
  • Enabled
  • Not deleted
  • In-stock - unless the Magento settings tell us to show out-of-stock products too

If there’s ever a missing product in your index, make sure the product meets all these requirements.

More information about troubleshooting for missing data can be found here.

Searchable attributes

It’s possible to configure which attributes should be searched when users type their query. To configure the list of searchable attributes, navigate to the products tab through System > Configuration > Algolia Search.

In the products tab, it’s possible to configure per attribute if it is searchable, retrievable or ordered. By default, all attributes are set to be searched as unordered. In general, this is better for the relevance of the search and we don’t recommend changing it without a specific reason.

Read the dedicated documentation to learn more about the difference between ordered and unordered search.

Another column in the configuration screen allows for the indexing of an empty value. Most of the time, this will be set to ‘No’ to prevent empty values from being indexed. This comes in handy when using the attribute for faceting. If empty values are indexed, Magento will automatically convert it to a ‘No’ value, which will then show up in the filters.

Default searchable attributes

Some attributes are indexed regardless of what is specified in the configuration. These attributes are not all searchable, but can be used for filtering, sorting, customizing the ranking, and building the results page.

The attributes that are always indexed:

name The product’s name
url The product’s URL
visibility_search The product’s visibility in the search
visibility_catalog The product’s visibility in the catalog
categories The product’s categories, formatted as a tree path
categories_without_path The product’s categories, without the tree path
thumbnail_url The product’s thumbnail image
image_url The product’s main image
in_stock The product’s stock availability
price The product’s price
type_id The product’s type (simple, configurable, bundled, etc.)

Facets

Facets are the attributes that will be used as filters on the results page. Common facets include price, color, categories, and brand. However, this will not work for every store. The facets have to be tuned to the products being sold and the way the end user searches for these products.

There are a couple of things that need to be specified for each facet:

  • The attribute
  • The label - this will be displayed above the filter
  • The type of facet

Configuration of facets

By default, we provide facets on the price, categories and color attributes.

Any numeric attribute (like price) will be shown as a slider.

Attributes that are specified as facets are automatically indexed as retrievable but not searchable. There’s no need to specify them in the Searchable Attributes configuration.

Sorting strategies

Sorting is only available on the InstantSearch results page.

When searching for products, users may expect multiple ways to sort the result set. For example, they might want to sort by relevance, popularity, price, or date.

The default sorting strategy when searching is sorting by relevance. Any other sorting strategy needs to be defined in the Sort Settings. For each strategy, you’ll need to define an attribute, sort order (ascending or descending), and a label.

Configuration of sorting strategies

By default, there are three sorting strategies:

  1. From lowest price to highest price
  2. From highest price to lowest price
  3. From newest to oldest

Each sorting strategy will create a new index, which will increase the amount of records. More information can be found here.

Attributes that are configured in a sorting strategy are automatically indexed as retrievable but not searchable. There’s no need to specify them in the Searchable Attributes configuration.

Index Settings

Through the Magento dashboard, the following settings for an index can be configured:

Additional index settings can be managed in the Algolia dashboard. It’s also possible to modify the settings programmatically by hooking into the algolia_products_index_before_set_settings event provided by the extension. A list of events provided by the extension can be found here.

Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.

Indexing Categories

To keep the number of records and indexing operations as low as possible, we only index categories that are actually active. This behavior can be changed through the settings.

Show categories that are not included in the navigation menu configuration

If set to ‘Yes’, all categories will be shown in the autocomplete search and InstantSearch results page.

Searchable Attributes

It’s possible to configure which attributes should be searched when users type their query. To configure the list of searchable attributes, navigate to the Category configuration through System > Configuration > Algolia Search > Categories.

It’s possible to configure per attribute if it is searchable, retrievable or ordered. By default, all attributes are set to be searched as unordered. In general, this is better for the relevance of the search and we don’t recommend changing it without a specific reason.

Read the dedicated documentation to learn more about the difference between ordered and unordered search.

Default searchable category attributes

Some attributes are indexed, regardless of what is specified in the configuration. These attributes are not all searchable but can be used for filtering, sorting, customizing the ranking, and building the results page.

The attributes that are always indexed:

name The category’s name
url The category’s URL
path The category’s path (parent categories)
level The category’s level in the category tree
include_in_menu The category’s visibility in the menu
_tags Filled automatically by the extension
popularity The category’s popularity
product_count The category’s amount of products

Index Settings

Through the Magento dashboard, the following setting for an index can be configured.

Additional index settings can be managed in the Algolia dashboard. It’s also possible to modify the settings programmatically by hooking into the algolia_categories_index_before_set_settings event provided by the extension. A list of events provided by the extension can be found here.

Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.

Indexing Pages

CMS Pages will be automatically indexed by our extension, allowing the users to search for pages in the autocomplete menu. By default, all active pages are indexed.

The settings provide an option to exclude certain pages, like error pages, so they don’t show up in the search results.

Configuration of excluded pages

The indexing of pages can be disabled altogether by navigating to the Additional Sections configuration, as shown below.

Configuration of additional pages

Searchable Attributes

For pages, it’s not possible to configure the searchable attributes through the admin interface. However, it is possible to change them programmatically by hooking into the algolia_after_create_page_object event provided by the extension. A list of events provided by the extension can be found here.

Default searchable page attributes

These attributes are indexed by default, and are not all searchable (some are). They can be used for filtering, sorting, customizing the ranking, and building the results page.

The attributes that are always indexed:

name The page’s name
url The page’s URL
slug The page’s slug
content The page’s content

Since records for our engine have to be smaller than 10 kilobytes, any page which is longer than 10,000 characters in content will not be indexed. In this case, only the page’s name would be searchable.

Read more here about the engine’s record limit.

Index settings

The following settings will always be sent to configure the index, and cannot be changed through the admin interface:

Additional index settings can be managed in the Algolia dashboard. It’s also possible to modify the settings programmatically by hooking into the algolia_pages_index_before_set_settings event provided by the extension. A list of events provided by the extension can be found here.

Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.

Indexing Suggestions

Every query that is being executed on the Magento installation is stored by Magento in the database. Magento automatically stores the query, the number of results, and the number of searches in the catalogsearch_query table, without any involvement from our extension.

Only back-end searches are stored by Magento. Search-as-you-type searches and instant search queries are not stored.

Our extension offers the possibility to index queries that are performed regularly on the Magento installation. In the settings, it’s possible to filter relevant queries (for example by the minimum number of results, minimum popularity, etc.). The resulting queries can be pushed into the suggestions index, providing an autocomplete on the most relevant queries for the Magento installation.

Configuration of suggestions

To ensure that the data in the suggestion index is good and relevant, the data in the catalogsearch_query table must be relevant as well. This can be achieved by enabling back-end search with our extension, by turning on the Enable Search and Make SEO Request settings in the Magento Administration.

With these options enabled, searches in the back-end will be processed by our extension. Since the data in catalogsearch_query will be updated as well, the queries in this table will become more relevant over time.

By default, suggestions are not indexed. When enabling the indexing of suggestions, a manual reindex needs to be triggered. Another way to start the indexing of suggestions is by adding a recurring job to the cron table:

1
1 * * * * php -f /absolute/path/to/magento/shell/indexer.php -- -reindex search_indexer_suggest

Searchable Attributes

For suggestions, it’s not possible to configure the searchable attributes through the admin interface. However, it is possible to change them programmatically by hooking into the algolia_after_create_suggestion_object event provided by the extension. A list of events provided by the extension can be found here.

Default searchable query attributes

These attributes are indexed by default, and are not all searchable (some are). They can be used for filtering, sorting, customizing the ranking, and building the results page.

The attributes that are always indexed:

query The query’s value
number_of_results The query’s number of results
popularity The query’s number of searches
updated_at The query’s last update timestamp

Index Settings

The following settings will always be sent to configure the index, and cannot be changed through the admin interface:

Additional index settings can be managed in the Algolia Dashboard. It’s also possible to modify the settings programmatically by hooking into the algolia_suggestions_index_before_set_settings event provided by the extension. A list of events provided by the extension can be found here.

Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.

Indexing Additional Sections

The autocomplete menu offers the possibility to display other sections from attributes, like colors and brands for example.

For this feature to work, the instant search page must be enabled

The attributes used for the additional sections have to be set to be attributes for faceting.

Configuration of additional sections

Searchable attributes

For Additional Settings, it’s not possible to configure the searchable attributes through the admin interface. However, it is possible to change them programmatically by hooking into the algolia_additional_section_items_before_index event provided by the extension. A list of events provided by the extension can be found here.

Default searchable attributes

These attributes are indexed by default, and are not all searchable (some are). They can be used for filtering, sorting, customizing the ranking, and building the results page.

The attributes that are always indexed:

value The attributes’ value (e.g. Red, XL, Nike, etc.)

Index settings

The following settings will always be sent to configure the index, and cannot be changed through the admin interface:

Additional index settings can be managed in the Algolia dashboard. It’s also possible to modify the settings programmatically by hooking into the algolia_additional_sections_index_before_set_settings event provided by the extension. A list of events provided by the extension can be found here.

Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.

Did you find this page helpful?