{"id":1350,"date":"2019-02-15T10:32:05","date_gmt":"2019-02-15T07:32:05","guid":{"rendered":"http:\/\/kusuaks7\/?p=955"},"modified":"2023-08-09T10:11:43","modified_gmt":"2023-08-09T10:11:43","slug":"using-scrapy-to-build-your-own-dataset","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/using-scrapy-to-build-your-own-dataset\/","title":{"rendered":"Using Scrapy to Build your Own Dataset"},"content":{"rendered":"

Ready to learn Big Data? Browse Big Data Training and Certification Courses<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

When I first started working in industry, one of the things I quickly realized is sometimes you have to gather, organize, and clean your own data. For this tutorial, we will gather data from a crowdfunding website called\u00a0FundRazr<\/a>. Like many websites, the site has its own structure, form, and has tons of accessible useful data, but it is hard to get data from the site as it doesn\u2019t have a structured API. In result, we will web scrape the site to get that unstructured website data and put into an ordered form to build our own dataset.<\/p>\n

In order to scrape the website, we will use\u00a0Scrapy<\/a>. In short, Scrapy is a framework built to build web scrapers more easily and relieve the pain of maintaining them. Basically, it allows you to focus on the data extraction using CSS selectors and choosing XPath expressions and less on the intricate internals of how spiders are supposed to work. This blog post goes a little beyond the great\u00a0official tutorial from the scrapy documentation<\/a>\u00a0in the hopes that if you need to scrape something a bit harder, you can do it on your own. With that, lets get started. If you get lost, I recommend opening the\u00a0video<\/a>\u00a0in a separate tab.<\/p>\n

Getting Started (Prerequisites)<\/strong><\/h4>\n

If you already have anaconda and google chrome (or Firefox), skip to Creating a New Scrapy Project.<\/p>\n

1.<\/strong>\u00a0Install Anaconda (Python) on your operating system. You can either download anaconda from the official site and install on your own or you can follow these anaconda installation tutorials below.<\/p>\n

2.<\/strong>\u00a0Install Scrapy (anaconda comes with it, but just in case). You can also install on your terminal (mac\/linux) or command line (windows). You can type the following:<\/p>\n

conda install -c conda-forge scrapy<\/pre>\n
3. Make sure you have Google chrome or Firefox. In this tutorial, I am using Google Chrome. If you don\u2019t have google chrome, you can install it here using this\u00a0link<\/a>.<\/p>\n
Creating a New Scrapy\u00a0project<\/h4>\n
1.Open a terminal (mac\/linux) or command line (windows). Navigate to a desired folder (see the image below if you need help) and type<\/p>\n
scrapy startproject fundrazr<\/pre>\n
<\/canvas><\/figure>\n
scrapy startproject fundrazr<\/p>\n
This makes a fundrazr directory with the following contents:<\/p>\n
<\/canvas><\/figure>\n
fundrazr project directory<\/p>\n
Finding good start URLs using Inspect on Google Chrome (or\u00a0Firefox)<\/strong><\/h4>\n
In the spider framework,\u00a0start_urls\u00a0<\/strong>is a list of URLs where the spider will begin to crawl from, when no particular URLs are specified. We will use each element in the\u00a0start_urls<\/strong>\u00a0list as a means to get individual campaign links.<\/p>\n
1. The image below shows that based on the category you choose, you get a different start url. The highlighted part in black are the possible categories of fundrazrs to scrape.<\/p>\n
<\/canvas><\/figure>\n
\n
Finding a good first start_url<\/p>\n
For this tutorial, the first in the list\u00a0start_urls<\/strong>\u00a0is:<\/p>\n
https:\/\/fundrazr.com\/find?category=Health<\/a><\/p>\n
2. This part is about getting additional elements to put in the\u00a0start_urls<\/strong>\u00a0list. We are finding out how to go to the next page so we can get additional urls to put in\u00a0start_urls<\/strong>.<\/p>\n
<\/canvas><\/figure>\n
Getting additional elements to put in\u00a0start_urls<\/strong>\u00a0list by inspecting Next\u00a0button<\/p>\n
The second start url is:\u00a0https:\/\/fundrazr.com\/find?category=Health&page=2<\/a><\/p>\n
The code below will be used in the code for the spider later in the tutorial. All it does is make a list of start_urls. The variable npages is just how many additional pages (after the first page) we want to get campaign links from.<\/p>\n
Scrapy Shell for finding Individual Campaign\u00a0Links<\/strong><\/h4>\n
The best way to learn how to extract data with Scrapy is using the Scrapy shell. We will use XPaths which can be used to select elements from HTML documents.<\/p>\n
The first thing we will try and get the xpaths for are the individual campaign links. First we do inspect to see roughly where the campaigns are in the HTML.<\/p>\n
<\/canvas><\/figure>\n
Finding links to individual campaigns<\/p>\n
We will use XPath to extract the part enclosed in the red rectangle below.<\/p>\n
<\/figure>\n
The enclosed part is a partial url we will\u00a0isolate<\/p>\n
In terminal type (mac\/linux):<\/p>\n
scrapy shell 'https:\/\/fundrazr.com\/find?category=Health'<\/pre>\n
In command line type (windows):<\/p>\n
scrapy shell \u201chttps:\/\/fundrazr.com\/find?category=Health<\/a>\"<\/pre>\n
Type the following into scrapy shell (to help understand the code, please see the video):<\/p>\n
response.xpath(“\/\/h2[contains(<\/span>@class<\/span><\/a>, ‘title headline-font’)]\/a[contains(<\/span>@class<\/span><\/a>, ‘campaign-link’)]\/\/@href”).extract()<\/span><\/span><\/p>\n
<\/canvas><\/figure>\n
There is a good chance you will get different partial urls as websites update over\u00a0time<\/p>\n
The code below is for getting all the campaign links for a given start url (more on this later in the First Spider section)<\/p>\n
Exit Scrapy Shell by typing\u00a0exit()<\/strong>.<\/p>\n
<\/canvas><\/figure>\n
exit scrapy \u00a0shell<\/p>\n
Inspecting Individual Campaigns<\/strong><\/h4>\n
While we should previously worked on understanding the structure of where individual campaigns links are, this section goes over where things are on individual campaigns.<\/p>\n
    \n
  1. Next we go to an individual campaign page (see link below) to scrape (I should note that some of these campaigns are difficult to view)<\/li>\n<\/ol>\n
    https:\/\/fundrazr.com\/savemyarm<\/a><\/p>\n
    2. Using the same inspect process as before, we inspect the title on the page<\/p>\n
    <\/canvas><\/figure>\n
    Inspect Campaign\u00a0Title<\/p>\n
    3. Now we are going to use scrapy shell again, but this time with an individual campaign. We do this because we want to find out how individual campaigns are formatted (including finding out how to extract the title from the webpage).<\/p>\n
    In terminal type (mac\/linux):<\/p>\n
    scrapy shell 'https:\/\/fundrazr.com\/savemyarm<\/a>'<\/pre>\n
    In command line type (windows):<\/p>\n
    scrapy shell \u201chttps:\/\/fundrazr.com\/savemyarm<\/a>\"<\/pre>\n
    The code to get the campaign title is<\/p>\n
    response.xpath(\"\/\/div[contains(@id<\/a>, 'campaign-title')]\/descendant::text()\").extract()[0]<\/pre>\n
    <\/figure>\n
    4. We can do the same for the other parts of the page.<\/p>\n
    amount Raised:<\/p>\n
    response.xpath(“\/\/span[contains(<\/span>@class<\/span><\/a>,’stat’)]\/span[contains(<\/span>@class<\/span><\/a>, ‘amount-raised’)]\/descendant::text()”).extract()<\/span><\/span><\/p>\n
    goal:<\/p>\n
    response.xpath(“\/\/div[contains(<\/span>@class<\/span><\/a>, ‘stats-primary with-goal’)]\/\/span[contains(<\/span>@class<\/span><\/a>, ‘stats-label hidden-phone’)]\/text()”).extract()<\/span><\/span><\/p>\n
    currency type:<\/p>\n
    response.xpath(\"\/\/div[contains(@class<\/a>, 'stats-primary with-goal')]\/@title\").extract()<\/pre>\n
    campaign end date:<\/p>\n
    response.xpath(“\/\/div[contains(<\/span>@id<\/span><\/a>, ‘campaign-stats’)]\/\/span[contains(<\/span>@class<\/span><\/a>,’stats-label hidden-phone’)]\/span[<\/span>@class<\/span><\/a>=’nowrap’]\/text()”).extract()<\/span><\/span><\/p>\n
    number of contributors:<\/p>\n
    response.xpath(“\/\/div[contains(<\/span>@class<\/span><\/a>, ‘stats-secondary with-goal’)]\/\/span[contains(<\/span>@class<\/span><\/a>, ‘donation-count stat’)]\/text()”).extract()<\/span><\/span><\/p>\n
    story:<\/p>\n
    response.xpath(\"\/\/div[contains(@id<\/a>, 'full-story')]\/descendant::text()\").extract()<\/pre>\n
    url:<\/p>\n
    response.xpath(\"\/\/meta[@property<\/a>='og:url']\/@content\").extract()<\/pre>\n
    5. Exit scrapy shell by typing:<\/p>\n
    exit()<\/pre>\n
    Items<\/h4>\n
    The main goal in scraping is to extract structured data from unstructured sources, typically, web pages. Scrapy spiders can return the extracted data as Python dicts. While convenient and familiar, Python dicts lack structure: it is easy to make a typo in a field name or return inconsistent data, especially in a larger project with many spiders (almost word for word copied from the great scrapy official documentation!).<\/p>\n
    <\/canvas><\/figure>\n
    \n
    File we will be modifying<\/p>\n
    The code for items.py is\u00a0here<\/a>.<\/p>\n
    Save it under the fundrazr\/fundrazr directory (overwrite the original items.py file).<\/p>\n
    The item class (basically how we store our data before outputting it) used in this tutorial looks like this.<\/p>\n
    <\/canvas><\/figure>\n
    items.py code<\/p>\n
    The Spider<\/strong><\/h4>\n
    Spiders are classes that you define and that Scrapy uses to scrape information from a website (or a group of websites). The code for our spider is below.<\/p>\n
    <\/canvas><\/figure>\n
    Fundrazr Scrapy\u00a0Code<\/p>\n
    Download the code\u00a0here<\/a>.<\/p>\n
    Save it in a file named\u00a0fundrazr_scrape.py<\/strong>\u00a0under the fundrazr\/spiders directory.<\/p>\n
    The current project should now have the following contents:<\/p>\n
    <\/canvas><\/figure>\n
    File we will be creating\/adding<\/p>\n
    Running the\u00a0Spider<\/strong><\/h4>\n
      \n
    1. Go to the fundrazr\/fundrazr directory and type:<\/li>\n<\/ol>\n
      scrapy crawl my_scraper -o MonthDay_Year.csv<\/pre>\n
      <\/canvas><\/figure>\n
      scrapy crawl my_scraper -o MonthDay_Year.csv<\/p>\n
      2. The data should be outputted in the fundrazr\/fundrazr directory.<\/p>\n
      <\/canvas><\/figure>\n
      Data Output\u00a0Location<\/p>\n
      Our data<\/strong><\/h4>\n
        \n
      1. The data outputted in this tutorial should look roughly like the image below. The individual campaigns scraped will vary as the website is constantly updated. Also it is possible there will be spaces between each individual campaign as excel is interpreting the csv file.<\/li>\n<\/ol>\n
        <\/canvas><\/figure>\n
        The data should\u00a0roughly\u00a0<\/strong>be in this\u00a0format.<\/p>\n
        2. If you want to download a larger file (it was made by changing npages = 2 to npages = 450 and adding download_delay\u00a0=<\/strong>\u00a02), you can download a bigger file with roughly 6000 campaigns scraped by downloading the file from my\u00a0github<\/a>. The file is called MiniMorningScrape.csv (it is a large file).<\/p>\n
        <\/canvas><\/figure>\n
        Roughly 6000 Campaigns Scraped<\/p>\n
        Closing Thoughts<\/h4>\n
        Creating a dataset can be a lot of work and is often an overlooked part of learning data science. One thing we didn\u2019t go over is that while we scraped a lot of data, we still haven\u2019t cleaned the data enough to do analysis. That is for another blog post though. If you have any questions or thoughts on the tutorial, feel free to reach out in the comments below,\u00a0youtube video page<\/a>, or through\u00a0Twitter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
        Scrapy is a framework built to build web scrapers more easily and relieve the pain of maintaining them. Basically, it allows you to focus on the data extraction using CSS selectors and choosing XPath expressions and less on the intricate internals of how spiders are supposed to work. If you need to scrape something a bit harder, you can do it on your own. With that, let’s get started. <\/p>\n","protected":false},"author":312,"featured_media":3061,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[1985],"class_list":["post-1350","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":1985,"user_id":312,"is_guest":0,"slug":"michael-galarnyk","display_name":"Michael Galarnyk","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Galarnyk","first_name":"Michael","job_title":"","description":"Michael Galarnyk is a Data Scientist at The Scripps Research Institute."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1350","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/312"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=1350"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1350\/revisions"}],"predecessor-version":[{"id":30112,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/1350\/revisions\/30112"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3061"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=1350"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=1350"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=1350"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1350"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}