What is natural language search?

Dustin Coates

Search has changed a lot since the early days of the Internet. Users now expect fast and personalized results when exploring a website or search engine, and they’re less likely to experiment with numerous different keywords just to find what they’re looking for. With the rise of new computing technologies, websites are beginning to offer a more natural search experience by providing innovative ways to explore content — primarily through natural language search. Search is quickly becoming a two-way conversation.

What is natural language search?

Natural language search allows users to speak or type into a device using their everyday language rather than keywords. Users can use full sentences in their native language as if they are speaking to another human, leaving the computer to transform the query into something it can understand.

Natural language search vs. keyword search

Thanks to Google and other search engines, users have become accustomed to using keyword searches. But keyword searches are not an intuitive way for users to ask questions, and users are actually pretty bad at using them to find what they need. They force users to strip out question words and other connective language to form literal text strings that the search engine can use to query data. It also may require effort on the part of the business to mine intent from keyword searches.

While keyword search systems typically do allow for some form of compound questions, they often force users to manually construct complex search structure. For instance, rather than asking a simple question such as “What’s a vegetarian recipe with tomatoes and cheese?”, you’d be expected to search for something more like ”vegetarian Recipe” tomato cheese.

With the rise of digital voice assistants such as Siri and Alexa, however, people are becoming accustomed to having conversations with their devices in full and grammatically complex sentences. The effect is that many users now form queries like questions over different devices and platforms. Users are becoming accustomed to using natural language to get information and expect fast results. Therefore, it is essential that search systems of all types can begin to accept natural language searches.

History of natural language search

Although advances in computer science and computation speed have enabled breakthroughs in natural language search, attempts at implementing these systems actually go back to the early days of the internet and web.

In 1993, the MIT Artificial Intelligence Lab developed the START Natural Language Question Answering System. While it wasn’t technically an Internet search engine, the START system allowed users to search an online encyclopedia of information using full natural language sentences.

A few years later in 1996, Ask Jeeves was launched. This was the first search engine that allowed users to explore the web through natural language. It turned out, however, that Jeeves was a bit ahead of his time. Pretty soon thereafter, Google launched a keyword search engine and quickly built a powerful system with impressive relevance scoring that easily beat out the results of its competitors.

Nearly two decades later, Google and other search engines started to realize the value of natural language search and further develop the experience that Ask Jeeves was trying to provide.

How natural language search works

Natural language search uses an advanced computer science technique called natural language processing (NLP). This process uses vast amounts of data to run statistical and machine learning models to infer meaning in complex grammatical sentences. This has become much more feasible over the last decade as internet companies collect more and more data. Computing power is growing at exponential rates to allow for processing this data.

The power of natural language comes from the ability to not only parse questions, but also to break down meaning in compound and contextual-based sentences. For example, if a customer asked an e-commerce store “What size t-shirts do you have for my kids?”, the search system can determine that the customer is looking for t-shirts in the kids category and wants to know what sizes are in stock. If the store has past purchase and search history on this customer, it may even be able to determine the optimal size of clothes and preferred styles.

No longer is natural language search simply a tool for obtaining basic facts, like the weather, from a personal assistant. More and more, consumers are beginning their shopping and brand exploration journeys directly through voice assistants or searching by voice on mobile. It is essential, therefore, that companies ensure they are optimizing their technologies and sales funnels to ensure that these consumers are able to engage with them in conversational language.

Four tips for designing a natural language search-friendly site

When optimizing the site for natural language search, many sites overly focus on SEO and fail to prioritize the user experience. In the end, however, the goal of natural language search is to provide customers with a helpful, intuitive, and engaging interface to explore the site. Here are some design tips that keep the user experience in mind:

1. Design a voice search engine that reduces the haystack

Search systems should take advantage of all information and context that they have available. User profiles and past searches, for instance, can help provide valuable information about what a user may want. This is particularly useful if a voice query is a bit vague, as the search engine can infer meaning based on context. Furthermore, setting up filters to segment indexed data by predefined categories can help to refine searches to provide users more relevant results.

2. Do research and understand how users perform conversational searches

While natural language processing tools are powerful for understanding general meaning, most businesses will find that there are nuances in their industries or domains that need to be fine-tuned. Reviewing and regularly analyzing user searches can help expose these trends in searches so that the models can be optimized accordingly.

3. Test the site’s content for ranking with natural search queries

In addition to reviewing search accuracy, it is important that the actual website content is built in such a way that the natural language search engines can correctly match the queries. Try running common search queries to see how the content is being ranked and gradually tune the content to see how it affects the results.

4. Use everyday language in site content and answer customer needs

Using conversational language in site content will help ensure that users questions and needs are being answered. For common questions that may not be answered directly in the content, it can be useful to add answers to these questions in a FAQ section or dedicated page so that users can still locate the answers.

In summary, designing a natural language search friendly site involves using data to provide context to searches, fine tuning search algorithms and filters to the specific business domain, and structuring site content to fit well with conversational search patterns. These processes will help customers get used to transitioning to a more conversational experience with your website.