Guides / Scaling / Multi-cluster management (MCM)

Implementing MCM

On this page

Multi-cluster management is deprecated and will be sunset. If you have issues with your Algolia infrastructure due to large volumes of data, contact the Algolia support team.

When your data no longer fits on a single machine, you have to add new clusters. Multi-Cluster Management (MCM) simplifies this process by letting you distribute and manage your data across several machines.

For example, a music streaming application lets users create public and private playlists. With MCM, the number of playlists can grow without fearing they might exceed your current cluster’s size limit. You also get dedicated user access out of the box.

Splitting data across multiple clusters

Assigning user data to a cluster

Without MCM, an application is bound to a single cluster. It means that splitting data across several clusters requires you to assign data to a specific application ID, so you can manually maintain and orchestrate the distribution by yourself.

With MCM, all clusters use the same application ID. MCM keeps a mapping on each cluster to route requests to the correct cluster. All you need to do is assign a cluster to a user with the assignUserId method.

Copy 
1
2
3
4

var response = await client.AssignUserIdAsync(
  "user42",
  new AssignUserIdParams { Cluster = "d4242-eu" }
);

Adding private user data

Imagine that you want to index your users’ private playlists.

Copy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

const playlists = [
  {
    user: 'user42',
    name: 'My peaceful playlist',
    tracks: [
      // ...
    ],
    createdAt: 1500000181
  },
  {
    user: 'user4242',
    name: 'My workout playlist',
    tracks: [
      // ...
    ],
    createdAt: 1500040452
  }
];

Without MCM, every record needs a userID attribute to tag the right user. Additionally, you need to set the userID attribute as a filter at query time.

Copy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class SetSettingsThenSaveObjects
{
  private static readonly List<Dictionary<string, object>> playlists = []; // Your records

  private static string GetAppIdFor(string user)
  {
    return ""; // Implement your own logic here
  }

  private static string GetIndexingApiKeyFor(string user)
  {
    return ""; // Implement your own logic here
  }

  async Task Main(string[] args)
  {
    foreach (var playlist in playlists)
    {
      // Fetch from your own data storage and with your own code
      // the associated application ID and API key for this user
      var appId = GetAppIdFor((playlist.GetValueOrDefault("user", "") as string)!);
      var apiKey = GetIndexingApiKeyFor((playlist.GetValueOrDefault("user", "") as string)!);

      try
      {
        var client = new SearchClient(new SearchConfig(appId, apiKey));
        var settings = new IndexSettings { AttributesForFaceting = ["searchable(playlistName)"] };
        await client.SetSettingsAsync("<YOUR_INDEX_NAME>", settings);

        await client.SaveObjectsAsync("<YOUR_INDEX_NAME>", playlists);
      }
      catch (Exception e)
      {
        Console.WriteLine(e.Message);
      }
    }
  }
}

With MCM, all records are automatically tagged using the X-Algolia-User-ID header you send at query time. The engine automatically adds an extra attribute __userID__ inside records to identify the userID associated with each record.

Copy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class SetHeaderUserIDThenSaveObjects
{
  private static readonly List<Dictionary<string, object>> playlists = []; // Your records

  async Task Main(string[] args)
  {
    var client = new SearchClient(new SearchConfig("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY"));
    foreach (var playlist in playlists)
    {
      try
      {
        var playlistUserID = playlist.GetValueOrDefault("userID", "") as string;
        await client.SaveObjectsAsync(
          "<YOUR_INDEX_NAME>",
          playlists,
          false,
          1000,
          new RequestOptionBuilder().AddExtraHeader("X-Algolia-User-ID", playlistUserID).Build()
        );
      }
      catch (Exception e)
      {
        Console.WriteLine(e.Message);
      }
    }
  }
}

For simplicity’s sake, the preceding snippets add records one by one. For better performance, you should group your records by userID first, then batch them.

Adding public data

If you decide you want to index public playlists as well.

Copy 
1
2
3
4
5
6
7
8
9
10

const playlists = [
  {
    user: 'public',
    name: 'Hot 100 Billboard Charts',
    tracks: [
      // ...
    ],
    createdAt: 1500240452,
  },
];

Without MCM, you need to tag every public record with a particular value (such as “public”) for the userID attribute. Then, you need to filter on that value to search for public records.

Copy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class SaveObjectsMcm
{
  private readonly List<Dictionary<string, object>> playlists = []; // Your records

  private static List<Dictionary<string, string>> GetAllAppIdConfigurations()
  {
    return
    [ /* A list of your MCM AppID/ApiKey pairs */
    ];
  }

  async Task Main(string[] args)
  {
    // Fetch from your own data storage and with your own code
    // the list of application IDs and API keys to target each cluster
    var configurations = GetAllAppIdConfigurations();

    // Send the records to each cluster
    foreach (var config in configurations)
    {
      try
      {
        var client = new SearchClient(
          new SearchConfig(
            config.GetValueOrDefault("appID", ""),
            config.GetValueOrDefault("apiKey", "")
          )
        );
        await client.SaveObjectsAsync("<YOUR_INDEX_NAME>", playlists);
      }
      catch (Exception e)
      {
        Console.WriteLine(e.Message);
      }
    }
  }
}

With MCM, you can use the userID value * to flag records as public, allowing all users to see them. Public records are automatically replicated on all clusters to avoid network latency during search.

Copy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class SaveObjectsPublicUser
{
  private static readonly List<Dictionary<string, object>> playlists = []; // Your records

  async Task Main(string[] args)
  {
    var client = new SearchClient(new SearchConfig("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY"));
    await client.SaveObjectsAsync(
      "<YOUR_INDEX_NAME>",
      playlists,
      false,
      1000,
      new RequestOptionBuilder().AddExtraHeader("X-Algolia-User-ID", "*").Build()
    );
  }
}

Searching the data

Without MCM, you need to manually handle the filtering logic to filter on private data for a specific user and public data.

Copy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class McmSearchWithout
{
  private static string GetAppIdFor(string user)
  {
    return ""; // Implement your own logic here
  }

  private static string GetIndexingApiKeyFor(string user)
  {
    return ""; // Implement your own logic here
  }

  async Task Main(string[] args)
  {
    // Fetch from your own data storage and with your own code
    // the associated application ID and API key for this user
    var appId = GetAppIdFor("user42");
    var apiKey = GetIndexingApiKeyFor("user42");

    var client = new SearchClient(new SearchConfig(appId, apiKey));
    var searchParams = new SearchParams(
      new SearchParamsObject
      {
        Query = "<YOUR_SEARCH_QUERY>",
        FacetFilters = new FacetFilters(
          [new FacetFilters("user:user42"), new FacetFilters("user:public")]
        ),
      }
    );

    await client.SearchSingleIndexAsync<Hit>("<YOUR_INDEX_NAME>", searchParams);
  }
}

With MCM, all the engine needs is the userID (with the X-Algolia-User-ID header) that the query targets, so it can route the request to the right cluster. It automatically adds the right filters to retrieve both the user’s data and public data.

Copy 
1
2
3
4
5

var response = await client.SearchSingleIndexAsync<Hit>(
  "ALGOLIA_INDEX_NAME",
  new SearchParams(new SearchParamsObject { Query = "peace" }),
  new RequestOptionBuilder().AddExtraHeader("X-Algolia-User-ID", "user42").Build()
);

Accessing secured data

You can’t rely on filters to select what data to return to a specific user because users can change these parameters. If all you have to pick specific data is query time filters, anyone can alter these to access the data of another user.

To properly restrict access of a user to only their data (and the public data), you need to use secured API keys.

Without MCM, you need to set the filters to authorize manually.

Copy 
1
2
3
4

var response = client.GenerateSecuredApiKey(
  "YourSearchOnlyApiKey",
  new SecuredApiKeyRestrictions { Filters = "user:user42 AND user:public" }
);

With MCM, you still need to generate secured API keys, but all you have to do is provide the userToken. It automatically adds the right filters to authorize.

Copy 
1
2
3
4

var response = client.GenerateSecuredApiKey(
  "YourSearchOnlyApiKey",
  new SecuredApiKeyRestrictions { UserToken = "user42" }
);

With MCM, you always need to provide the X-Algolia-User-ID header, even when the query contains the userToken parameter. The header is required to route the requests efficiently.

Index configuration and API keys

When performing actions that have global impact on all clusters, such as using addApiKey or setSettings, you don’t need to provide the X-Algolia-User-ID header.

Did you find this page helpful?