Searching tweets using the standard Search API of Twitter

Saatwik Bisaria -

Do not miss this exclusive book on Binary Tree Problems. Get it now for free.

Reading time: 35 minutes | Coding time: 10 minutes

Standard search API is a part of Twitter API which is used to send search queries to the twitter server. It searches the desired query in the twitter server and returns the tweet data and its various parameters.

It takes parameters in JSON format and returns the response data in JSON format. There are various parameters which can be used in a get request to filter out the desied result. The Search API is not meant to be an exhaustive source of Tweets. Not all Tweets will be indexed or made available via the search interface.

Resource URL

https://api.twitter.com/1.1/search/tweets.json

Characteristics of Twitter Search API

  • The twitter API uses JSON data format for returning and receiving the data.
  • The twitter API is HTTP-based (over SSL) API meaning we can use get method to retrieve data from twitter,post method to send requests to the twitter server and search method to search the twitter posts.
  • The twitter API limits the number of requests that can be sent to the twitter server per access token or twitter account.This is called twitter rate limit.If you encounter twitter rate limit exceeded error it means that Twitter rejected consecutive attempts to access its API under your Twitter account.The rate limit is different for different methods of the API.
  • The methods of twitter API accepts various parameters which are used to cusotmize the requests according to needs.
  • There are twitter API libraries for almost all programming languages.

Read the Documentation of twitter API from here

Getting Twitter API keys

To start with, we will need to have a Twitter developer account and obtain credentials (i.e. API key, API secret, Access token and Access token secret) on the to access the Twitter API, following these steps:

  1. Create a Twitter developer account https://developer.twitter.com/
  2. Go to https://developer.twitter.com/en/apps and log in with your Twitter user account.
  3. Click “Create an app”
  4. Fill out the form, and click “Create”
  5. A pop up window will appear for reviewing Developer Terms. Click the “Create” button again.
  6. In the next page, click on “Keys and Access Tokens” tab, and copy your “API key” and “API secret” from the Consumer API keys section.
  7. Scroll down to Access token & access token secret section and click “Create”. Then copy your “Access token” and “Access token secret.

Standard Search Operators

To use Twitter Search effectively, We use the Standard search operators for filter operations. The query can have operators that modify its behavior to filter the twittter response results. We need to specify the operators in the 'q' parameter
Various operators available in standard search API are -

  • watching now - It returns the tweets containing both “watching” and “now”. This is the default operator.
  • “Opengenus foundation” - It returns the tweets containing the exact phrase “Opengenus foundation”.
  • python OR java - It returns the tweets containing either “python” or “java” (or both).
  • AI -machine - It returns the tweets containing “AI” but not “machine”.
  • from:opengenus - It returns the tweets sent from Twitter account “opengenus”.
  • list:NASA/astronauts-in-space-now - It returns the tweets sent from a Twitter account in the NASA list astronauts-in-space-now
  • to:NASA - It returns the tweets Tweet authored in reply to Twitter account “NASA”.
  • @NASA - It returns the tweets mentioning Twitter account “NASA”.
  • marvel filter:safe - It returns the tweets containing “marvel” with Tweets marked as potentially sensitive removed.
  • marvel filter:media - It returns the tweets containing “marvel” and an image or video.
  • marvel -filter:retweets - It returns the tweets containing "marvel”, filtering out retweets
  • marvel filter:native_video - It returns the tweets containing “marvel” and an uploaded video, Amplify video, Periscope, or Vine.
  • marvel filter:periscope - It returns the tweets containing “marvel” and a Periscope video URL.
  • marvel filter:vine - It returns the tweets containing “marvel” and a Vine.
  • marvel filter:images - It returns the tweets containing "marvel” and links identified as photos, including third parties such as Instagram.
  • marvel filter:twimg - It returns the tweets containing “marvel” and a pic.twitter.com link representing one or more photos.
  • marvel filter:links - It returns the tweets containing “hilarious” and linking to URL.
  • marvel url:amazon - It returns the tweets containing “puppy” and a URL with the word “amazon” anywhere within it.
  • hulk since:2015-12-21 - It returns the tweets containing “superhero” and sent since date “2015-12-21” (year-month-day).
  • marvel until:2015-12-21 - It returns the tweets containing “puppy” and sent before the date “2015-12-21”.
  • movie -scary :) - It returns the tweets containing “movie”, but not “scary”, and with a positive attitude.
  • flight :( - It returns the tweets containing “flight” and with a negative attitude.
  • traffic ? - It returns the tweets containing “traffic” and asking a question.

What is Requests package?

The requests module allows you to send HTTP requests using Python.
The HTTP request returns a Response Object with all the response data (content, encoding, status, etc). We will be using the post method of requests library to send a request to tweet.

Install requests package using command:

pip install requests

Get Requests

The get() method sends a GET request to the specified url.It is used to retrieve data from the server in JSON format.

Syntax

requests.get(url, params={key: value}, args)

Python Code And Explanation

After importing the dependencies, first we want to create variables(consumer_key and consumer_secret) that will authenticate with Twitter.You will find all the required variables in your developer's account dashboard and we can copy and paste each of them as strings.

#importing all dependencies
import numpy as np
import tweepy
import requests
import base64

#Define your keys from the developer portal
consumer_key = 'XXXXXXXXXXXXXXXXXXXXXX'
consumer_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

The twitter API requires a single key that is a string of a base64 encoded version of the two keys separated by a colon so we will encode the consumer keys into base64 which is the usable form.

#Reformat the keys and encode them
key_secret = '{}:{}'.format(consumer_key, consumer_secret_key).encode('ascii')
#Transform from bytes to bytes that can be printed
b64_encoded_key = base64.b64encode(key_secret)
#Transform from bytes back into Unicode
b64_encoded_key = b64_encoded_key.decode('ascii')

Now, We will use requests package of python to post an authentication request using twitter authentication resource URL to the twitter server and store the post response in a variable. We To check and make sure that the request worked , We will print the status code of the request response. If the status code printed is 200 then the request worked successfully.

base_url = 'https://api.twitter.com/'
auth_url = '{}oauth2/token'.format(base_url)
auth_headers = {
    'Authorization': 'Basic {}'.format(b64_encoded_key),
    'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
}
auth_data = {
    'grant_type': 'client_credentials'
}
auth_resp = requests.post(auth_url, headers=auth_headers, data=auth_data)
print(auth_resp.status_code)
access_token = auth_resp.json()['access_token']

Now, we have to send a get request to the twitter server which will return the search results on twitter in JSON format according to the parameters specified. This request requires authentications so we will pass access token into the headers of this request.

Now we will assign the parameters required for this request in JSON format. The parameter of q specifies the search query which is a string we want to search in the twitter server.In this case, we have used the standard search operator tp specify that we want tweets mentioning NASA twitter handle.

Rest of the parameters are optional. If we want to specify the location of the tweet we can use geocode to specify the latitude,longitude,radius of the tweet. lang parameter restricts tweets to the given language, given by an ISO 639-1 code.

Now, we will send a get request to the twitter server and store the response. We will print the status code of the request response. If the status code printed is 200 then the request worked successfully.

search_headers = {
    'Authorization': 'Bearer {}'.format(access_token)    
}

search_params = {
    'q': '@NASA',
    'lang': 'eu',
    'result_type': 'mixed'
}

search_url = 'https://api.twitter.com/1.1/search/tweets.json'  
search_resp = requests.get(search_url, headers=search_headers, params=search_params)

Now , we will save the response in json format and print it. The response contains the searched tweets and its various parameters in JSON format.

for i in range(0,10):
  print('tweet number',i+1,'=',search_data['statuses'][i])

Now we shall parse through the JSON response to print the searched tweets.The response contains various parameters of the tweets returned
We get the output as the searched tweet:

JSON reponse for one tweet

tweet number 1 ={ 
   'created_at':'Tue Dec 31 19:08:29 +0000 2019',
   'id':1212088152992702469,
   'id_str':'1212088152992702469',
   'text':'RT @Albousidi_Omar: #الكسوف_الحلقي في #عُمان\n.\n#AnnularEclipse in #oman \n #AnnularSolarEclipse\n.\n.\nby: @Albousidi_Omar \n.\n.\n\n@NASA \n@Arabic…',
   'truncated':False,
   'entities':{ 
      'hashtags':[ 
         { 
            'text':'الكسوف_الحلقي',
            'indices':[ 
               20,
               34
            ]
         },
         { 
            'text':'عُمان',
            'indices':[ 
               38,
               44
            ]
         },
         { 
            'text':'AnnularEclipse',
            'indices':[ 
               47,
               62
            ]
         },
         { 
            'text':'oman',
            'indices':[ 
               66,
               71
            ]
         },
         { 
            'text':'AnnularSolarEclipse',
            'indices':[ 
               74,
               94
            ]
         }
      ],
      'symbols':[ 

      ],
      'user_mentions':[ 
         { 
            'screen_name':'Albousidi_Omar',
            'name':'Omar Albusaidi',
            'id':1484092538,
            'id_str':'1484092538',
            'indices':[ 
               3,
               18
            ]
         },
         { 
            'screen_name':'Albousidi_Omar',
            'name':'Omar Albusaidi',
            'id':1484092538,
            'id_str':'1484092538',
            'indices':[ 
               103,
               118
            ]
         },
         { 
            'screen_name':'NASA',
            'name':'NASA',
            'id':11348282,
            'id_str':'11348282',
            'indices':[ 
               125,
               130
            ]
         }
      ],
      'urls':[ 

      ]
   },
   'metadata':{ 
      'iso_language_code':'eu',
      'result_type':'recent'
   },
   'source':'<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>',
   'in_reply_to_status_id':None,
   'in_reply_to_status_id_str':None,
   'in_reply_to_user_id':None,
   'in_reply_to_user_id_str':None,
   'in_reply_to_screen_name':None,
   'user':{ 
      'id':1640324840,
      'id_str':'1640324840',
      'name':'Meshal Al Meshal',
      'screen_name':'meshal6362',
      'location':'Riyadh-nghaمسشفى الحرس الوطني ',
      'description':'\u200fpharmacistصيدلي\n\nsnapchat: meshal6362',
      'url':None,
      'entities':{ 
         'description':{ 
            'urls':[ 

            ]
         }
      },
      'protected':False,
      'followers_count':1601,
      'friends_count':4964,
      'listed_count':4,
      'created_at':'Fri Aug 02 12:11:38 +0000 2013',
      'favourites_count':25028,
      'utc_offset':None,
      'time_zone':None,
      'geo_enabled':True,
      'verified':False,
      'statuses_count':23252,
      'lang':None,
      'contributors_enabled':False,
      'is_translator':False,
      'is_translation_enabled':False,
      'profile_background_color':'C0DEED',
      'profile_background_image_url':'http://abs.twimg.com/images/themes/theme1/bg.png',
      'profile_background_image_url_https':'https://abs.twimg.com/images/themes/theme1/bg.png',
      'profile_background_tile':False,
      'profile_image_url':'http://pbs.twimg.com/profile_images/378800000790473435/6a33af7af08bf430f226029c610e8cc1_normal.jpeg',
      'profile_image_url_https':'https://pbs.twimg.com/profile_images/378800000790473435/6a33af7af08bf430f226029c610e8cc1_normal.jpeg',
      'profile_banner_url':'https://pbs.twimg.com/profile_banners/1640324840/1383897564',
      'profile_link_color':'1DA1F2',
      'profile_sidebar_border_color':'C0DEED',
      'profile_sidebar_fill_color':'DDEEF6',
      'profile_text_color':'333333',
      'profile_use_background_image':True,
      'has_extended_profile':True,
      'default_profile':True,
      'default_profile_image':False,
      'following':None,
      'follow_request_sent':None,
      'notifications':None,
      'translator_type':'none'
   },
   'geo':None,
   'coordinates':None,
   'place':None,
   'contributors':None,
   'retweeted_status':{ 
      'created_at':'Thu Dec 26 05:13:13 +0000 2019',
      'id':1210066009627013121,
      'id_str':'1210066009627013121',
      'text':'#الكسوف_الحلقي في #عُمان\n.\n#AnnularEclipse in #oman \n #AnnularSolarEclipse\n.\n.\nby: @Albousidi_Omar \n.\n.\n\n@NASA… https://t.co/diYwR9Sddp',
      'truncated':True,
      'entities':{ 
         'hashtags':[ 
            { 
               'text':'الكسوف_الحلقي',
               'indices':[ 
                  0,
                  14
               ]
            },
            { 
               'text':'عُمان',
               'indices':[ 
                  18,
                  24
               ]
            },
            { 
               'text':'AnnularEclipse',
               'indices':[ 
                  27,
                  42
               ]
            },
            { 
               'text':'oman',
               'indices':[ 
                  46,
                  51
               ]
            },
            { 
               'text':'AnnularSolarEclipse',
               'indices':[ 
                  54,
                  74
               ]
            }
         ],
         'symbols':[ 

         ],
         'user_mentions':[ 
            { 
               'screen_name':'Albousidi_Omar',
               'name':'Omar Albusaidi',
               'id':1484092538,
               'id_str':'1484092538',
               'indices':[ 
                  83,
                  98
               ]
            },
            { 
               'screen_name':'NASA',
               'name':'NASA',
               'id':11348282,
               'id_str':'11348282',
               'indices':[ 
                  105,
                  110
               ]
            }
         ],
         'urls':[ 
            { 
               'url':'https://t.co/diYwR9Sddp',
               'expanded_url':'https://twitter.com/i/web/status/1210066009627013121',
               'display_url':'twitter.com/i/web/status/1…',
               'indices':[ 
                  112,
                  135
               ]
            }
         ]
      },
      'metadata':{ 
         'iso_language_code':'eu',
         'result_type':'recent'
      },
      'source':'<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
      'in_reply_to_status_id':None,
      'in_reply_to_status_id_str':None,
      'in_reply_to_user_id':None,
      'in_reply_to_user_id_str':None,
      'in_reply_to_screen_name':None,
      'user':{ 
         'id':1484092538,
         'id_str':'1484092538',
         'name':'Omar Albusaidi',
         'screen_name':'Albousidi_Omar',
         'location':'',
         'description':'مصمم جرافيك - مصور فوتوغرافي - أحب الإبداع أينما كان، فكن مبدعا يحبك الجميع',
         'url':'https://t.co/jExUbxF8OH',
         'entities':{ 
            'url':{ 
               'urls':[ 
                  { 
                     'url':'https://t.co/jExUbxF8OH',
                     'expanded_url':'http://www.omaroman.com',
                     'display_url':'omaroman.com',
                     'indices':[ 
                        0,
                        23
                     ]
                  }
               ]
            },
            'description':{ 
               'urls':[ 

               ]
            }
         },
         'protected':False,
         'followers_count':1105,
         'friends_count':183,
         'listed_count':3,
         'created_at':'Wed Jun 05 04:53:56 +0000 2013',
         'favourites_count':492,
         'utc_offset':None,
         'time_zone':None,
         'geo_enabled':False,
         'verified':False,
         'statuses_count':510,
         'lang':None,
         'contributors_enabled':False,
         'is_translator':False,
         'is_translation_enabled':False,
         'profile_background_color':'131516',
         'profile_background_image_url':'http://abs.twimg.com/images/themes/theme14/bg.gif',
         'profile_background_image_url_https':'https://abs.twimg.com/images/themes/theme14/bg.gif',
         'profile_background_tile':True,
         'profile_image_url':'http://pbs.twimg.com/profile_images/815983131622707200/FaiIx19N_normal.jpg',
         'profile_image_url_https':'https://pbs.twimg.com/profile_images/815983131622707200/FaiIx19N_normal.jpg',
         'profile_banner_url':'https://pbs.twimg.com/profile_banners/1484092538/1483380515',
         'profile_link_color':'009999',
         'profile_sidebar_border_color':'EEEEEE',
         'profile_sidebar_fill_color':'EFEFEF',
         'profile_text_color':'333333',
         'profile_use_background_image':True,
         'has_extended_profile':False,
         'default_profile':False,
         'default_profile_image':False,
         'following':None,
         'follow_request_sent':None,
         'notifications':None,
         'translator_type':'none'
      },
      'geo':None,
      'coordinates':None,
      'place':None,
      'contributors':None,
      'is_quote_status':False,
      'retweet_count':158,
      'favorite_count':464,
      'favorited':False,
      'retweeted':False,
      'possibly_sensitive':False,
      'lang':'eu'
   },
   'is_quote_status':False,
   'retweet_count':158,
   'favorite_count':0,
   'favorited':False,
   'retweeted':False,
   'lang':'eu'
}

With this, you will have the complete idea of using the Search API of Twitter and search tweets with filters.

Sign up for FREE 3 months of Amazon Music. YOU MUST NOT MISS.