Search by Algolia
What is end-to-end AI search?
ai

What is end-to-end AI search?

Simplicity is critical for search engines, but building one that enables that simplicity is complex.  Over the last 20+ years ...

Abhijit Mehta

Director of Product Management

Comparing AI search solutions in a crowded market landscape
ai

Comparing AI search solutions in a crowded market landscape

Many new AI-powered search solutions have been released this year, and each promises to provide great results, but as ...

Andy Jones

Marketing Campaign Production Manager

What is B2B ecommerce? Everything you need to know
e-commerce

What is B2B ecommerce? Everything you need to know

When you think of “customer experience,” what comes to mind? People, right? Specifically, consumers. Retail customers. That’s easy; the ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is ecommerce merchandising? Key components and best practices
e-commerce

What is ecommerce merchandising? Key components and best practices

A potential customer is about to land on the home page of your ecommerce platform, curious to see what cool ...

Catherine Dee

Search and Discovery writer

AI-powered search: From keywords to conversations
ai

AI-powered search: From keywords to conversations

By now, everyone’s had the opportunity to experiment with AI tools like ChatGPT or Midjourney and ponder their inner ...

Chris Stevenson

Director, Product Marketing

Vector vs Keyword Search: Why You Should Care
ai

Vector vs Keyword Search: Why You Should Care

Search has been around for a while, to the point that it is now considered a standard requirement in many ...

Nicolas Fiorini

Senior Machine Learning Engineer

What is AI-powered site search?
ai

What is AI-powered site search?

With the advent of artificial intelligence (AI) technologies enabling services such as Alexa, Google search, and self-driving cars, the ...

John Stewart

VP Corporate Marketing

What is a B2B marketplace?
e-commerce

What is a B2B marketplace?

It’s no secret that B2B (business-to-business) transactions have largely migrated online. According to Gartner, by 2025, 80 ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

3 strategies for B2B ecommerce growth: key takeaways from B2B Online - Chicago
e-commerce

3 strategies for B2B ecommerce growth: key takeaways from B2B Online - Chicago

Twice a year, B2B Online brings together industry leaders to discuss the trends affecting the B2B ecommerce industry. At the ...

Elena Moravec

Director of Product Marketing & Strategy

Deconstructing smart digital merchandising
e-commerce

Deconstructing smart digital merchandising

This is Part 2 of a series that dives into the transformational journey made by digital merchandising to drive positive ...

Benoit Reulier
Reshma Iyer

Benoit Reulier &

Reshma Iyer

The death of traditional shopping: How AI-powered conversational commerce changes everything
ai

The death of traditional shopping: How AI-powered conversational commerce changes everything

Get ready for the ride: online shopping is about to be completely upended by AI. Over the past few years ...

Aayush Iyer

Director, User Experience & UI Platform

What is B2C ecommerce? Models, examples, and definitions
e-commerce

What is B2C ecommerce? Models, examples, and definitions

Remember life before online shopping? When you had to actually leave the house for a brick-and-mortar store to ...

Catherine Dee

Search and Discovery writer

What are marketplace platforms and software? Why are they important?
e-commerce

What are marketplace platforms and software? Why are they important?

If you imagine pushing a virtual shopping cart down the aisles of an online store, or browsing items in an ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is an online marketplace?
e-commerce

What is an online marketplace?

Remember the world before the convenience of online commerce? Before the pandemic, before the proliferation of ecommerce sites, when the ...

Catherine Dee

Search and Discovery writer

10 ways AI is transforming ecommerce
e-commerce

10 ways AI is transforming ecommerce

Artificial intelligence (AI) is no longer just the stuff of scary futuristic movies; it’s recently burst into the headlines ...

Catherine Dee

Search and Discovery writer

AI as a Service (AIaaS) in the era of "buy not build"
ai

AI as a Service (AIaaS) in the era of "buy not build"

Imagine you are the CTO of a company that has just undergone a massive decade long digital transformation. You’ve ...

Sean Mullaney

CTO @Algolia

By the numbers: the ROI of keyword and AI site search for digital commerce
product

By the numbers: the ROI of keyword and AI site search for digital commerce

Did you know that the tiny search bar at the top of many ecommerce sites can offer an outsized return ...

Jon Silvers

Director, Digital Marketing

Using pre-trained AI algorithms to solve the cold start problem
ai

Using pre-trained AI algorithms to solve the cold start problem

Artificial intelligence (AI) has quickly moved from hot topic to everyday life. Now, ecommerce businesses are beginning to clearly see ...

Etienne Martin

VP of Product

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

We are Hacker News readers and probably just like you, there is not a day that goes by we don’t use it. It is a little like checking the weather app of the tech world. Long story short, Hacker News is awesome, and we wanted to add our two cents to make it even greater to use.

Indeed, here is our problem: how do we instantly access the old posts we wish we had saved?

Powering a new Hacker News search engine

Up until now we’ve been using hnsearch.com, maintained for years by the great folks at Octopart. I hope we speak on behalf of the HN community here, we are all grateful for the work they put in hnsearch.com and they inspired us to pursue their effort.

Back in September 2013, we created a “homemade Hacker News crawler” and built a search engine with the data we could get. It was not perfect but somehow, it did the job fine.

Now part of the Ycombinator W14 batch, we have a direct access to the data and it has allowed us to provide instant search for the entire content of Hacker News, 1.2 million articles, 5.2 million comments as of today. See for yourself right here: hn.algolia.com

Here is how we did it

  • Hacker News API access

    • YC provides us a private API access to fetch batches of 1000 items (an item being a comment or a post). Every two minutes, we update our database with the latest 1000 items. Last 48,000 items are refreshed every hour to keep the number of votes and comments up to date.
# Yep, that's a Lisp API :)
EXPORT_REGEXP = %r{^((d+) (story|comment|poll|pollopt) "(.+)" (d+) (?:nil|"(.*)") (?:nil|"(.+)") (?:nil|"(.*)") (?:nil|-?(d+)) (?:nil|(([d ]+))) (?:nil|(d+)))$}
  • Thumbnails generation

    • We use wkhtmltoimage to render the URLs and generate the associated thumbnails. Playing with connection timeouts and JavaScript infinite loops was a pleasure:
(timeout 60 xvfb-run --auto-servernum --server-args="-screen 0, 1024x768x24" 
wkhtmltoimage-amd64 --height 768 --use-xserver--javascript-delay 30000 "$URL" "$FILE" || 
timeout 60 xvfb-run --auto-servernum --server-args="-screen 0, 1024x768x24" 
wkhtmltoimage-amd64 --height 768 --use-xserver --disable-javascript "$URL" "$FILE") && 
convert "$FILE" -resize '100!x100' "$FILE"
  • Thumbnails storage

    • Thumbnails are resized and stored on a S3 bucket.
AWS::S3::S3Object.store("#{id}.png", open(temp_file), 'hnsearch', access: :public_read)
  • Thumbnails distribution

    • We configured a CloudFront instance targeting the S3 bucket to distribute thumbnails with low latency and high data transfer speed. We followed Amazon’s associated developer guide.
  • Indexing

    • We used the “algoliasearch-rails” gem and a standard (Ruby on Rails) MySQL-backed ActiveRecord setup. Indexing is performed automatically as soon as new items are added to the database, providing a near-realtime experience.
  • Configuration

class Item < ActiveRecord::Base
  include AlgoliaSearch

  algoliasearch per_environment: true do
    # the list of attributes sent to Algolia's API
    attribute :created_at, :title, :url, :author, :points, :story_text, :comment_text, :author, :num_comments, :story_id, :story_title, :story_url
    attribute :created_at_i do
      created_at.to_i
    end

    # The order of the attributes sets their respective importance.
    # `title` is more important than `{story,comment}_text`, `{story,comment}_text` more than `url`, `url` more than `author`
    # btw, do not take into account position to avoid first word match boost
    attributesToIndex ['unordered(title)', 'unordered(story_text)', 'unordered(comment_text)', 'unordered(url)', 'author', 'created_at_i']

    # add tags used for filtering
    tags do
      [item_type, "author_#{author}", "story_#{story_id}"]
    end

    # Custom ranking allows to automatically sort the results by a custom criteria
    # in this case, a decreasing sort of the number of HN points and comments.
    customRanking ['desc(points)', 'desc(num_comments)']

    # controls the way results are sorted sorting on the following 4 criteria (one after another)
    # I removed the 'exact' match critera (improve 1-words query relevance, doesn't fit HNSearch needs)
    ranking ['typo', 'proximity', 'attribute', 'custom']

    # google+, $1.5M raises, C#: we love you
    separatorsToIndex '+#$'
  end

  def story_text
    item_type_cd != Item.comment ? text : nil
  end

  def story_title
    comment? && story ? story.title : nil
  end

  def story_url
    comment? && story ? story.url : nil
  end

  def comment_text
    comment? ? text : nil
  end

  def comment?
    item_type_cd == Item.comment
  end

  def num_comments
    item_type_cd == Item.story ? story_comments.count : nil
  end
end
  • Search

    • Queries are sent directly to our API via the javascript client, the javascript code uses a public API-Key that can only perform queries.

Seeking feedback from the community

There is still room for improvement and we would love to know how you are searching for news on HN. What is important for you? Are you searching by date, by upvote, by comment or by user? All together maybe?

We would love to have your feedback! Don’t hesitate to checkout the code: We open-sourced it.

Special thanks to the Octopart and YC teams for making this experience possible!

Give it a try now: hn.algolia.com

About the author
Sylvain Utard

VP of Engineering

linkedintwitter

Recommended Articles

Powered byAlgolia Algolia Recommend

New experimental version of Hacker News Search built with Algolia
product

kevin

Creating search relevance for ecommerce and media with great content
product

Peter Villani

Sr. Tech & Business Writer

Algolia's top 10 tips to achieve highly relevant search results
product

Julien Lemoine

Co-founder & former CTO at Algolia