Scenario

For the final project you will use the Twitter API to download tweets and store them in a MongoDB database. You will develop reports on these tweets and then create an interactive map showing statistics about the tweets.

Final Project: Data Analytics With Twitter

Objectives

• Create an interactive map based on the Twitter data

Twitter and Python

1. Twitter was developed in 2006 as an application for users to write small messages for followers to read. It has developed into one of the most popular social media applications available today. Twitter is used by many companies for data mining purposes. They provide access to their cloud-based web service. You will need an authentication API key that we will obtain using the developer account you created in an earlier step in the project. You can use this developer account to do your own data mining for future projects.

5. In the next slide, what best describes you – choose Student.

10. You will need all four of these keys to authenticate to the Twitter API.

11. Open the Azure Lab environment. In the C: drive, there is a folder called Files. In Files, you will find Final_Project. In this folder are several Python programs. We will use these in the final project. If you want to do the final project on your own computer, then finalprojectstart.py, finalprojectMapStart.py, finalproject_keys.py, finalproject_statecodes.py, and TweetStream.py are available for download.

access_token_secret = ''

mapquest_key = 'gkQWpPFx9HpjPvCjEhr52sWi8Mzz2QWa'

14. Open Spyder again and finalprojectstart.py. Most of the code is written for you, but you will need to add your own code. First let’s discuss the purpose of this code. This Python program will connect to twitter using the API keys you provided. The program will read in a csv file into a pandas data frame, connect to MongoDB, connect to the Twitter API, then read the tweets (in json format) into the MongoDB Project database and the twitter_feed collection.

The CSV file will contain the Twitter handle, Twitter id, and state data. You can feel free to use any Twitter accounts you want. The example is national parks and the nationalparks.csv file contains this information. As long as you have those three fields, you can use any account you want. You may want to create your program with museums, NFL teams, NBA teams, MLB teams, hockey teams, state governments, and so forth—any Twitter accounts you want. Use the handle to search for the twitterID and put it in the same format as nationalparks.csv.

15. Once the program is finished and you see the prompt, the data is now populated in MongoDB.

MongoDB Reports

It’s important to know the content of the JSON documents.

4. You will be creating at least five queries in MongoDB as social media analytics on the Twitter data. The first four queries are guided below and the fifth query can be any query you choose. The video will show examples on how to write these queries. In order to speed up querying, we will create an index on every key in the twitter_feed collection. To do this, use the command (note the $** simply means create an index on all fields and subfields in a document).

Query 4: Let’s analyze the text of the tweet. Depending on your example Twitter feed, there are a variety of keywords you may want to search for. If you are looking at sports Twitter feeds, you may want to look at injury reports or trading. If you are looking at government, you may want to review legislation. For museums, you may be concerned about open admission. For this example we will search for texts containing the word picture.

Query 5: Develop your own query.

Interactive Map

5. You will need to create the mongo_client and set the database to Project.

6. Next, create two lists: one named tweets and one named park_locations.

Deliverable

1. Create a report about your data mining with Twitter. The report should have the following sections.

   5. Reports Query 1–5: Show the purpose of the query, the code you used to run it, the result, and your analysis. Your analysis can be a simple explanation of what you see or feel free to extrapolate on your result.

   6. Visualization: Include the html map with your final project deliverable. Explain how it visually illustrates the data from the database.