A treasure map to Twitter Data via Tweepy

Hi there! Twitter data is a true scrumptious source for anyone wants to develop something great, perform data analysis, test SOTA deep learning models or learn data science. For the above mentioned reasons, I’ve been using Twitter API for quite some time. In this article, I wanted to provide a comprehensive guide for people who wants to start with Twitter data. You may enjoy this article even if you’re already into twitter API because I will also talk about the recent policies for applying twitter dev account etc.

This article presents these to the readers:

We all use twitter all day, every day and as technology people, we can’t avoid ourselves from thinking “what would happen if I reach the slightest piece of it”, the motivation here is to analyze something from daily life and gut to apply your technical skills to a real-life problem. So let's motivate ourselves to solve a problem. The problem will be explained in this article will be fetching all tweets by new york times for the purposes of analyzing the celebrities in the headlines of the news. You may of-course come up with a more interesting problem and develop a brighter motivation to solve it.

Recipe to set up a developer account.

8. In the next text box where they ask about the usage of the scope of your API. Write down the appropriate option such as yes I will just use the Tweet functionality.

9. Some individuals or organizations may display the tweets of their official accounts on their websites. If you plan to add such a feature into your website you should elaborate on the usage of twitter data in the “Do you plan to display or aggregate data about Twitter content outside of Twitter” section.

10. In the next section, you will be asked if you share your analysis or, any of the accomplishments gained based on Twitter data with a government entity. Answer this question with an open intention.

11. So much for the application in the next step, you will review your application and expected to read and accept the term of use. Then you can send your application.

If your answers were clear and enough to evoke the impression that you actually have an idea about what you gonna do with Twitter data or at least you are willing to learn, Twitter approves developer accounts within a day normally. If you follow the hints in this article I can't imagine a reason why you wouldn't be approved.

When you are approved you will be notified with an email and you will be able to create an app. You will be expected to create an APP for each of your specific analyses and tasks you perform via Twitter. The APP will have access tokens that you will use them to authenticate in the Twitter API. In the APP creation process, you will be asked for the intent of usage so I suggest you to copy the answers in the application form so you wouldn't have to explain yourself all over again.

So if the above explanation was clear, we can come up with the pseudo-code of our method such as:

Of course, there must be some additions like exception handling and setting up a timer to avoid rate limit errors. We’ll get there soon. Now its time to talk about fetching the tweets in upper rows.

This is how cursors work:

See, the cursor way is much simpler and shorter in lines of code.

But since we cannot place the access straight into the code, I suggest to create a credentials.json and create a dictionary that holds them. Here:

After you created the credentials file you can load it and used them in your code, without exposing them:

Now, authenticate and return the API object

2. Getting id of a user using the Twitter screen name

3. Getting the timeline ids with the cursor

Non-commercial Twitter APIs allows to be requested for 15 minutes, and for the next 15 minutes, it expects you to show some manner and live it be. I'm having fun with the manner part but the 15 minutes part still holds. So I added some lines to serve as a stopwatch and some more lines to put my code into sleep for 15 minutes. Below you’ll also see some try-except blocks to avoid the process from quitting when there is Rate Limit Exceed error just in case. You may elaborate/alter the entire process.

4. Getting the tweets affiliated with the ids

Now, we have all the tweet ids so we can fetch the extended statues finally. Here:

Here is the end of this article, comment if you have any suggestions. Let me know if you want to learn more about how to store and analyze these tweets.

Related posts:

The Egyptian Pyramids were built thousands of years ago by the ancient Egyptians as tombs for their pharaohs and their consorts. These massive structures were constructed using millions of limestone…

This post was written by Jiří Sedláček — a tooling expert on the GraalVM team and author of VisualVM. GraalVM — a high-performance polyglot virtual machine from Oracle Labs — enables programmers to…

Liu Hui Ying was a normal detective in the eyes of the others, one who can solve any case that was handed to her. But what they don’t know is that behind her success are three ghosts! Born in the…