Python Script to search web using Google Custom Search API

In this article, we have developed a Python Script to search web using Google Custom Search API. This will strengthen your implementation skill.

Table of contents:

1. Introduction
2. Login to the Google Cloud Console
3. Create a new Project
4. Enable the custom search and create new credential
5. Create your custom search engine
6. Write the Python script

Introduction

Python is an interpreted programming language like JavaScript, PHP , bash and Ruby. Such languages are termed scripting languages, because their programs are executed line by line by a software called an interpreter. Programs in such languages are known as Scripts. Rather than a Python program, one can say; a Python script, similarly a bash script, PHP script and so on. In this article, we are going to write a script which works as a search engine.

A search engine is a software system designed to carry out web searches (like Google). We are talking about software system, but how can a simple script perform the task of a whole system?, right, this is when our main actor comes in --Google custom search API.

An API, Application Programming Interface is simply a software which enables communication between two distinct softwares. Yeah, as simple as that, Bobo from London and Hans from France, both want to have business together but unfortunately they can't understand each other, Youmbi who understands both French and English comes in to help them, then Youmbi is the API. Now what is the Google custom search API?, as the name implies, API for Google custom search.

We have another actor, the Google search engine. Actually, we are writing a Python script to search for articles in OpenGenus IQ. Who performs the search? Google search engine, who asks for the articles?, our Python script executed by our interpreter and who enables communication between the two of them?, the Google custom search API, then what happens exactly?

1. The scripts prompts us to enter the name or author of an article.
2. After entering the above information, the scripts passes them to the API
3. The API then asks the Google search engine to search what is given.
4. The Google search engine returns the results to the API, who then hands them over to our Python script.
5. Our Python script will then try its best to present us the results.

The script should not necessarily be written in Python, other languages like JavaScript, PHP, bash support this API, as well as compiled languages like C++, Java, C# etc. The problem now is: how do we get to use the API?.

Any individual no matter his programming level can use a given amount of APIs from Google as far as he has a Google account. Assuming you have one, then lets follow the various steps to obtain the services of the Google custom search API.

1. Login to the Google Cloud Console.

In your web browser, search "Google cloud console", then select the appropriate result. Its usually written Google cloud platform.

Sign in using your email address (@gmail.com).

2. Create a new Project.

To use any Google API, you need to have a project, you can create a new one in this console. Fill the necessary information required for your project then create it.

3. Enable the custom search and create new credential.

Go to the library section and type 'custom search api' in the search bar. Select the custom search api and enable it.

Click on the link to create a new credential and select API key.

Restrict this key to the Google custom search API. Do this by clicking on the radio button 'restrict key', a dropdown box will appear then select custom search API and save. Remember to copy this key, you will use it in the code, you can obtain it by clicking on SHOWKEY.

4. Create your custom search engine.

The next step is to create your custom search engine. To do this you will need to go to google programmable search engine , sign in by clicking on get started. Click on new custom search engine. Fill the name of the search engine and the websites you want to include in the search results, then click on create. In this case we have iq.opengenus.org

You will need to copy the ID of this search engine, it's going to be included in our script. Click on the search engine name to obtain it.

5. Write the Python script.

For your python script to communicate with your custom search engine through custom search API, you need to install the google-api-python-client library.

$ pip install google-api-client

Now we have every thing we need in order to use the custom search api.
After the installation you can now open your any text editor and code.

#google api modules
from googleapiclient.discovery import build

#credentials
api_key = "AIzaSyAHC4H1K1IqbjfPHtaMwFc_qO6nlNq3siw"
search_engine_id = "d63a9f69eed704db9"


resource = build("customsearch", "v1" , developerKey = api_key).cse()

#request creation

print("------------Search any article from name, author or tag in opengenus IQ--------------\n\n")
request = resource.list(q = input("article name, author or tag:") , cx = search_engine_id)

#request execution
result = request.execute()

print("search results (", result["searchInformation"]["searchTime"], ")")

print ("total" , len (result['items']), "results fetched")

for item in result['items']:
    print("___________________________________________")
    print("Title:", item["title"]) #title
    print("Link:", item["link"])
    print("Description:", item["snippet"])
    print("___________________________________________\n")

from googleapiclient.discovery import build

We import the build functions from our installed library.

resource = build("customsearch", "v1" , developerKey = api_key).cse()
The build function constructs a resource object to interact with the api. In our case its the "v1" version 1 of the "customsearch", custom search api with the developer key, developerKey.

request = resource.list(q = input("article name, author or tag:") , cx = search_engine_id)

We prepare a request with query q, which is the name or author of the article we want to search, cx, our search engine id.

result = request.execute()

The result of the execution returns a dictionary which contains the information about the search. Our main interest is the items key, which contains link to our articles. Finally we display the result in a for loop.
Running the script and entering q parameter as "converting vectors to strings" gives this;

There are more API available in Google cloud console, chose one of your choice to include in your personal project.