Tools / Crawler / Linking Google Analytics

Linking Google Analytics

The content on this page is deprecated, instead use the new interface.

You can use Analytics to enrich the records you extract from a website. By boosting search results based on their popularity (or another metric tracked by Google Analytics), you can improve the relevance of your website’s search.

With a little bit of configuration, your crawler can fetch Google Analytics metrics automatically on each crawl.

Here’s a brief overview of the required steps:

  1. Generate a service account on Google Cloud Platform.
  2. Have the administrator of your Google Analytics account provide read access to the email address of that service account.
  3. Edit your crawler’s configuration, so that it has:
    • The credentials of the service account
    • The identifier of the Google Analytics view you want to fetch metrics from
  4. Verify that your crawler can connect to your Google Analytics view.
  5. Edit your crawler’s recordExtractor so that it integrates the metrics retrieved from Google Analytics into the output records (make sure this works as expected).

Create a service account

First, create a service account on Google Cloud Platform.

  1. Create (or select an existing) project from console.developers.google.com.
  2. Activate the Analytics Reporting API in that project.
  3. In the Credentials section, create a new Service account. You can skip the optional steps.
  4. Open the Service account and click on Add key -> Create new key. Select JSON, and download the resulting JSON file (you need this file in step 3).

Ga guide service account It’s recommended that you include the following information in the name of your service account:

  • The name of the crawled website
  • The word “google-analytics” or “GA”
  • The word “crawler”

Grant access to Google Analytics data

How an administrator of your Google Analytics account can give read access to the service account created in the last step.

Steps for an administrator:

  1. Log in to Google Analytics.
  2. Select Account, Property, and then the View that has the analytics for the website you are crawling.
  3. Go to the Admin tab.
  4. In the View panel on the right side of the screen, click on View User Management.
  5. Click the + button, then click “Add users”.
  6. Paste the email address of the service account generated in the previous step.
  7. In the Permissions panel, make sure that only the “Read & Analyze” permission is enabled.
  8. Click the “Add” button to confirm.

Ga guide grant access

  1. Return to the Admin tab of your Google Analytics View and click on View Settings.
  2. Copy the “View ID” number and send it to the owner of your crawler account.

Ga guide view id

Update your crawler’s configuration

In this step, edit the configuration of your crawler to integrate the credentials of your service account, and the “View ID” you received in the previous step.

  1. Go to your Crawler dashboard, select your crawler, and click on the Editor tab.
  2. Specify the following properties in your crawler’s externalDataSources property:
    • client_email: the email address of the service account you created in step 1.
    • private_key: the private key in the JSON file you downloaded in step 1.
    • viewIds: the “View ID” provided by the administrator of the Google Analytics account in step 2
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    
    {
      // ...
      externalDataSources: [
        {
          dataSourceId: 'myAnalytics',
          type: 'googleanalytics',
          metrics: ['ga:uniquePageViews'],
          startDate: '30daysAgo',
          endDate: 'today',
          credentials: {
            type: 'service_account',
            client_email: 'dummy@my-project.iam.gserviceaccount.com',
            private_key: 'YOUR_PRIVATE_KEY',
            viewIds: [YOUR_VIEW_ID],
          },
        },
      ],
    };
    
  3. Save your changes.

After saving your crawler’s configuration, your Google Analytics metrics will be fetched whenever you crawl your website. If an error occurs while fetching your analytics, it should be reported less than one minute after the crawling process starts.

Test if the Crawler can connect to the Google Analytics view

Before editing your crawler’s recordExtractor to leverage the metrics from the linked Google Analytics view, make sure:

  • The credentials you specified in the previous step are correct
  • The credentials give your crawler access to the view’s metrics.

These checks are performed when refreshing the external data, which makes your crawler fetch your linked metrics from Google Analytics. To refresh external data:

  1. Open the External Data tab of your crawler.
  2. Click on the Fetch data button.

If you get a “Failed to refresh external data” error message within 30 seconds, your crawler couldn’t connect to your Google Analytics view. In this case, check the error message on the Overview page.

Otherwise, the connection to Google Analytics was successful. If the URLs match your crawler configuration, you should see matching Google Analytics data.

Integrate analytics into records

In this step, you edit your recordExtractor so that it integrates metrics from Google Analytics into the records it produces.

  1. Go to Crawler Admin, select your crawler, and click the Editor tab.
  2. Read metric values from the external data source you’ve added beforehand, and store them as attributes for your resulting records. If Google Analytics has data for the current page, the associated metrics should be present in the dataSources parameter of your recordExtractor:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    
    {
      // ...
      recordExtractor: ({ url, dataSources }) => {
        // "myAnalytics" must match the `dataSourceId` defined in `externalDataSources`
        const pageviews = dataSources.myAnalytics['ga:uniquePageViews'];
    
        return [
          {
            objectID: url.href,
            pageviews,
          },
        ];
      },
    };
    

    For a complete example, check the recordExtractor of the sample Google Analytics crawler configuration.

  3. In the Test a URL field of the configuration editor (which you can find in the Admin > Settings tab), type the URL of a page with analytics attached to it.
  4. Click on Run test.
  5. When the test completes, click the External data tab. You should see the analytics data extracted from Google Analytics for that page.

If this doesn’t work as expected, try adding a trailing / to your URL, or test with another URL.

Did you find this page helpful?