8 Web Scraping in R

https://learn.datacamp.com/courses/web-scraping-in-r

Main functions and concepts covered in this BP chapter:

rvest
html_element()
html_elements()
html_attr()
html_attrs()
html_text()
html_table()
Types vs Classes vs IDs
XPATH
text()
position()
count()
httr

Packages used in this chapter:

## Load all packages used in this chapter
library(tidyverse) #includes dplyr, ggplot2, and other common packages

## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.6
## ✔ forcats   1.0.1     ✔ stringr   1.6.0
## ✔ ggplot2   4.0.1     ✔ tibble    3.3.0
## ✔ lubridate 1.9.4     ✔ tidyr     1.3.1
## ✔ purrr     1.2.0     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(rvest)

## 
## Attaching package: 'rvest'
## 
## The following object is masked from 'package:readr':
## 
##     guess_encoding

library(xml2)
library(httr)

Datasets used in this chapter:

## Load datasets used in this chapter
## For this chapter, they're all coming from the web (or copy/pasting in html code)

Tips for what HTML to use for this chapter:

Remember the purpose is to learn how to do the things they walk you through in DC. So you need to find a way to include working examples of what you learn (working example means an example that actually works that demonstrates the functionality). I’ll explain a few options here.

DataCamp may or may not show you the raw HTML code that it has you use. If it does display the raw HTML, copy it and store it in a string variable. For example, when I wrote this, the first question included this:

html_excerpt_raw <- '
<html> 
  <body> 
    <h1>Web scraping is cool</h1>
    <p>It involves writing code – be it R or Python.</p>
    <p><a href="https://datacamp.com">DataCamp</a> 
        has courses on it.</p>
  </body> 
</html>'

So just copy/paste all that, just like I did above, and you can now use html_excerpt_raw.

Sometimes it just loads it in a variable. For example, when I did it, it had a variable pre-stored named list_raw_html. It didn’t show me the contents of the variable. But if I just put list_raw_html on a line by itself and clicked Run Code (or ctrl-Enter), I could then copy/paste the output from the bottom and create the variable. For example:

list_raw_html <- "\n<html>\n  <body>\n    <ol>\n      <li>Learn HTML</li>\n      <li>Learn CSS</li>\n      <li>Learn R</li>\n      <li>Scrape everything!*</li>\n    </ol>\n    <small>*Do it responsibly!</small>\n  </body>\n</html>"

# Read in the corresponding HTML string
list_html <- read_html(list_raw_html)
# Extract the ol node
ol_node <- list_html %>% 
    html_element('ol')
# Extract and print all the children from ol_node
ol_node %>% 
    html_children()

## {xml_nodeset (4)}
## [1] <li>Learn HTML</li>
## [2] <li>Learn CSS</li>
## [3] <li>Learn R</li>
## [4] <li>Scrape everything!*</li>

You can also apply what you learn in DataCamp to an actual website instead of (or in addition to) their HTML. You can also use websites designed for people to learn how to web scrape, for example, https://www.scrapethissite.com.

8.1 Introduction to HTML and Web Scraping

Take a look at the following excerpt from a website:

Web Scraping is cool It involves writing code – be it R or Python.

DataCamp has courses on it.

The corresponding HTML is provided to you in the html_excerpt_raw variable. In this exercise you’ll use rvest to read it into R, so you can work with it.

As you are working with the rvest package during this entire course, it will always have been preloaded for you with the library(rvest) command.

html_excerpt <- read_html(html_excerpt_raw)
html_excerpt

## {html_document}
## <html>
## [1] <body> \n    <h1>Web scraping is cool</h1>\n    <p>It involves writing co ...

xml_structure(html_excerpt)

## <html>
##   <body>
##     {text}
##     <h1>
##       {text}
##     {text}
##     <p>
##       {text}
##     {text}
##     <p>
##       <a [href]>
##         {text}
##       {text}
##     {text}

Given the following ordered list (using the ol element), which is stored as list_raw_html:

Learn HTML Learn CSS Learn R Scrape everything! Do it responsibly!

list_html <- read_html(list_raw_html)

ol_node <- list_html %>%
    html_element('ol')

ol_node %>% 
  html_children()

## {xml_nodeset (4)}
## [1] <li>Learn HTML</li>
## [2] <li>Learn CSS</li>
## [3] <li>Learn R</li>
## [4] <li>Scrape everything!*</li>

Parse hyperlinks into a data frame Have a look at the following ul list of “helpful links”.

It consists of three li elements that in turn contain a elements with the links:

Helpful links Wikipedia Dictionary Search Engine Compiled with help from Google.

In this exercise, you’ll parse these links into an R data frame by selecting only a elements that are within li elements.

PS: You’ll use tibble(), a function from the Tidyverse, for that. tibble() is basically a trimmed down version of data.frame(), which you certainly already know. Just like data.frame(), you specify column names and data as pairs of column names and values, like so: my_tibble <- tibble( column_name_1 = value_1, column_name_2 = value_2, ... )

hyperlink_raw_html <- "\n<html>\n  <body>\n    <h3>Helpful links</h3>\n    <ul>\n      <li><a href=\"https://wikipedia.org\">Wikipedia</a></li>\n      <li><a href=\"https://dictionary.com\">Dictionary</a></li>\n      <li><a href=\"https://duckduckgo.com\">Search Engine</a></li>\n    </ul>\n    <small>\n      Compiled with help from <a href=\"https://google.com\">Google</a>.\n    </small>\n  </body>\n</html>"

links <- hyperlink_raw_html %>% 
  read_html() %>% 
  html_elements('li a')

domain_value = links %>% html_attr('href')
name_value = links %>% html_text()

link_df <- tibble(
  domain = domain_value,
  name = name_value
)

link_df

## # A tibble: 3 × 2
##   domain                 name         
##   <chr>                  <chr>        
## 1 https://wikipedia.org  Wikipedia    
## 2 https://dictionary.com Dictionary   
## 3 https://duckduckgo.com Search Engine

A few questions later though, it had a variable mountains_html. When I put it on its own line and ran it, it displayed the output, but not all of it; each row of the output ended with a “…”. But when I ran paste(mountains_html), it displayed all of it. I then copy/pasted all of that into a code chunk in my BP, stored it in mountains_html_raw, and then was able to do mountains_html <- read_html(mountains_html_raw) and the rest worked. For example:

For such cases, html_table() has an extra argument you can use to correctly parse the table, as shown in the video. Missing cells are automatically recognized and replaced with NA values.

mountains_html_raw <- "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n<html>\n<head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head>\n<body> \n  <table id=\"clean\">\n<tr>\n<th>Mountain</th>\n      <th>Height [m]</th>\n      <th>First ascent</th>\n      <th>Country</th>\n    </tr>\n<tr>\n<td>Mount Everest</td>\n      <td>8848</td>\n      <td>1953</td>\n      <td>Nepal, China</td>\n    </tr>\n<tr>\n<td>K2</td>\n      <td>8611</td>\n      <td>1954</td>\n      <td>Pakistan, China</td>\n    </tr>\n<tr>\n<td>Kanchenjunga</td>\n      <td>8586</td>\n      <td>1955</td>\n      <td>Nepal, India</td>\n    </tr>\n</table>\n<table id=\"dirty\">\n<tr>\n<td>Mountain </td>\n      <td>Height [m]</td>\n      <td>First ascent</td>\n      <td>Country</td>\n    </tr>\n<tr>\n<td>Mount Everest</td>\n      <td>8848</td>\n      <td>1953</td>\n    </tr>\n<tr>\n<td>K2</td>\n      <td>8611</td>\n      <td>1954</td>\n      <td>Pakistan, China</td>\n    </tr>\n<tr>\n<td>Kanchenjunga</td>\n      <td>8586</td>\n      <td>1955</td>\n      <td>Nepal, India</td>\n    </tr>\n</table>\n</body>\n</html>\n"


mountains_html <- read_html(mountains_html_raw)

# Extract the "dirty" table into a data frame
mountains <- mountains_html %>% 
  html_element("table#dirty") %>% 
  html_table(header = TRUE)

mountains

## # A tibble: 3 × 4
##   Mountain      `Height [m]` `First ascent` Country        
##   <chr>                <int>          <int> <chr>          
## 1 Mount Everest         8848           1953 <NA>           
## 2 K2                    8611           1954 Pakistan, China
## 3 Kanchenjunga          8586           1955 Nepal, India

8.2 Navigation and Selection with CSS

As you have seen in the video, CSS can be used to style a web page. In the most basic form, this happens via type selectors, where styles are defined for and applied to all HTML elements of a certain type. In turn, you can also use type selectors to scrape pages for specific HTML elements.

As demonstrated in the video, you can also combine multiple type selectors via a comma, i.e. with html_elements("type1, type2"). This selects all elements that have type1 or type2.

Have a look at the following HTML:

languages_raw_html <- "<html> 
  <body> 
    <div>Python is perfect for programming.</div>
    <p>Still, R might be better suited for data analysis.</p>
    <small>(And has prettier charts, too.)</small>
  </body> 
</html>"

languages_html2 <- read_html(languages_raw_html)

languages_html2 %>%
    html_elements('div, p') %>%
    html_text()

## [1] "Python is perfect for programming."                
## [2] "Still, R might be better suited for data analysis."

As you know, IDs should be unique across a web page. If you can make sure this is the case, it can reduce the complexity of your scraping selectors drastically.

Here’s the structure of an HTML page you might encounter in the wild:

structured_html_raw <- "<html>
  <body>
    <div id = 'first'>
      <h1 class = 'big'>Joe Biden</h1>
      <p class = 'first blue'>Democrat</p>
      <p class = 'second blue'>Male</p>
    </div>
    <div id = 'second'>...</div>
    <div id = 'third'>
      <h1 class = 'big'>Donald Trump</h1>
      <p class = 'first red'>Republican</p>
      <p class = 'second red'>Male</p>
    </div>
  </body>
</html>"

structured_html <- read_html(structured_html_raw)

structured_html %>% 
  html_elements('#first')

## {xml_nodeset (1)}
## [1] <div id="first">\n      <h1 class="big">Joe Biden</h1>\n      <p class="f ...

In this exercise, your job is to select the last p node within the div.

As you learned in the video, pseudo-classes can help you whenever you don’t have other means of selecting a specific node of page, e.g., through an ID selector or a unique class.

nested_raw <- "<html>
  <body>
    <div>
      <p class = 'text'>A sophisticated text [...]</p>
      <p class = 'text'>Another paragraph following [...]</p>
      <p class = 'text'>Author: T.G.</p>
    </div>
    <p>Copyright: DC</p>
  </body>
</html>"

nested_html <- read_html(nested_raw)
# select that last p in both groups
nested_html %>% 
  html_elements('p:last-child')

## {xml_nodeset (2)}
## [1] <p class="text">Author: T.G.</p>
## [2] <p>Copyright: DC</p>

nested_html %>% 
  html_elements('p.text:last-child')

## {xml_nodeset (1)}
## [1] <p class="text">Author: T.G.</p>

# only selects last p in the text class

By now, you surely know how to select elements by type, class, or ID. However, there are cases where these selectors won’t work, for example, if you only want to extract direct descendants of the top ul element. For that, you will use the child combinator (>) introduced in the video.

languages_raw2 <- "<ul id = 'languages'>
    <li>SQL</li>
    <ul>    
      <li>Databases</li>
      <li>Query Language</li>
    </ul>
    <li>R</li>
    <ul>
      <li>Collection</li>
      <li>Analysis</li>
      <li>Visualization</li>
    </ul>
    <li>Python</li>
  </ul>"


languages_html <- read_html(languages_raw2)

# all li elements in the nested list

languages_html %>% 
  html_elements('li') %>% 
  html_text()

## [1] "SQL"            "Databases"      "Query Language" "R"             
## [5] "Collection"     "Analysis"       "Visualization"  "Python"

# direct descendants only
languages_html %>% 
  html_elements('ul#languages > li') %>% 
  html_text()

## [1] "SQL"    "R"      "Python"

8.3 Advanced Selection with XPATH

For this chapter, this code looks a bit more like real life. Your goal is to extract the precipitation reading from this weather station. Unfortunately, it can’t be directly referenced through an ID.

weather_raw <- "<html>
  <body>
    <div id = 'first'>
      <h1 class = 'big'>Berlin Weather Station</h1>
      <p class = 'first'>Temperature: 20°C</p>
      <p class = 'second'>Humidity: 45%</p>
    </div>
    <div id = 'second'>...</div>
    <div id = 'third'>
      <p class = 'first'>Sunshine: 5hrs</p>
      <p class = 'second'>Precipitation: 0mm</p>
    </div>
  </body>
</html>"

weather_html <- read_html(weather_raw)

# Select all p elements
weather_html %>%
    html_elements(xpath = '//p')

## {xml_nodeset (4)}
## [1] <p class="first">Temperature: 20°C</p>
## [2] <p class="second">Humidity: 45%</p>
## [3] <p class="first">Sunshine: 5hrs</p>
## [4] <p class="second">Precipitation: 0mm</p>

# Select p elements with the second class
weather_html %>%
    html_elements(xpath = '//p[@class = "second"]')

## {xml_nodeset (2)}
## [1] <p class="second">Humidity: 45%</p>
## [2] <p class="second">Precipitation: 0mm</p>

# Select p elements that are children of "#third"
weather_html %>%
    html_elements(xpath = '//*[@id = "third"]/p')

## {xml_nodeset (2)}
## [1] <p class="first">Sunshine: 5hrs</p>
## [2] <p class="second">Precipitation: 0mm</p>

# Select p elements with class "second" that are children of "#third"
weather_html %>%
    html_elements(xpath = '//*[@id = "third"]/p[@class = "second"]')

## {xml_nodeset (1)}
## [1] <p class="second">Precipitation: 0mm</p>

weather_raw2 <- "<html>
  <body>
    <div id = 'first'>
      <h1 class = 'big'>Berlin Weather Station</h1>
      <p class = 'first'>Temperature: 20°C</p>
      <p class = 'second'>Humidity: 45%</p>
    </div>
    <div id = 'second'>...</div>
    <div id = 'third'>
      <p class = 'first'>Sunshine: 5hrs</p>
      <p class = 'second'>Precipitation: 0mm</p>
      <p class = 'third'>Snowfall: 0mm</p>
    </div>
  </body>
</html>"

weather_html2 <- read_html(weather_raw2)

With XPATH, something that’s not possible with CSS can be done: selecting elements based on the properties of their descendants. For this, predicates may be used.

# Select all divs
weather_html2 %>% 
  html_elements(xpath = '//div')

## {xml_nodeset (3)}
## [1] <div id="first">\n      <h1 class="big">Berlin Weather Station</h1>\n     ...
## [2] <div id="second">...</div>
## [3] <div id="third">\n      <p class="first">Sunshine: 5hrs</p>\n      <p cla ...

# Select all divs with p descendants
weather_html2 %>% 
  html_elements(xpath = '//div[p]')

## {xml_nodeset (2)}
## [1] <div id="first">\n      <h1 class="big">Berlin Weather Station</h1>\n     ...
## [2] <div id="third">\n      <p class="first">Sunshine: 5hrs</p>\n      <p cla ...

# Select all divs with p descendants having the "third" class
weather_html2 %>% 
  html_elements(xpath = '//div[p[@class = "third"]]')

## {xml_nodeset (1)}
## [1] <div id="third">\n      <p class="first">Sunshine: 5hrs</p>\n      <p cla ...

roles_raw <- "<table>
 <tr>
  <th>Actor</th>
  <th>Role</th>
 </tr>
 <tr>
  <td class = 'actor'>Jayden Carpenter</td>
  <td class = 'role'><em>Mickey Mouse</em> (Voice)</td>
 </tr>
</table>"

roles_html <- read_html(roles_raw)

# Extract the data frame from the table using a known function from rvest
roles <- roles_html %>% 
  html_element(xpath = "//table") %>% 
  html_table()
# Print the contents of the role data frame
roles

## # A tibble: 1 × 2
##   Actor            Role                
##   <chr>            <chr>               
## 1 Jayden Carpenter Mickey Mouse (Voice)

First extract the actors and roles from the table using XPATH.

# Extract the actors in the cells having class "actor"
actors <- roles_html %>% 
  html_elements(xpath = '//table//td[@class = "actor"]') %>%
  html_text()
actors

## [1] "Jayden Carpenter"

# Extract the roles in the cells having class "role"
roles <- roles_html %>% 
  html_elements(xpath = '//table//td[@class = "role"]/em') %>% 
  html_text()
roles

## [1] "Mickey Mouse"

8.4 Scraping Best Practices

Here’s some rvest code that I used to find out the elevation of a beautiful place where I recently spent my vacation.

# Get the HTML document from Wikipedia
wikipedia_page <- read_html('https://en.wikipedia.org/wiki/Varigotti')
# Parse the document and extract the elevation from it
wikipedia_page %>% 
  html_elements('table tr:nth-child(9) > td') %>% 
  html_text()

## [1] "Finale Ligure"

As you have learned in the video, read_html() actually issues an HTTP GET request if provided with a URL, like in this case.

The goal of this exercise is to replicate the same query without read_html(), but with httr methods instead.

Note: Usually rvest does the job, but if you want to customize requests like you’ll be shown later in this chapter, you’ll need to know the httr way.

For a little repetition, you’ll also translate the CSS selector used in html_elements() into an XPATH query.

# Get the HTML document from Wikipedia using httr
wikipedia_response <- GET('https://en.wikipedia.org/wiki/Varigotti')
# Parse the response into an HTML doc
wikipedia_page <- content(wikipedia_response)

# Extract the elevation with XPATH
wikipedia_page %>% 
    html_elements(xpath = '//table//tr[position() = 9]/td') %>% 
    html_text()

## [1] "Finale Ligure"

As you’ve seen in the video, a fundamental part of the HTTP system are status codes: They tell you if everything is okay or if there is a problem with your request.

It is good practice to always check the status code of a response before you start working with the downloaded page. For this, you can use the status_code() function from the httr() package. It takes as an argument a response object that results from a request method.

Now let’s assume you’re trying to scrape the same page as before, but somehow you got the URL wrong (Varigott instead of Varigotti).

response <- GET('https://en.wikipedia.org/wiki/Varigott')
# Print status code of inexistent page
status_code(response)

## [1] 404

# Access https://httpbin.org/headers with httr
response <- GET('https://httpbin.org/headers')
# Print its content
content(response)

## $headers
## $headers$Accept
## [1] "application/json, text/xml, application/xml, */*"
## 
## $headers$`Accept-Encoding`
## [1] "deflate, gzip"
## 
## $headers$Host
## [1] "httpbin.org"
## 
## $headers$`User-Agent`
## [1] "libcurl/8.14.1 r-curl/7.0.0 httr/1.4.7"
## 
## $headers$`X-Amzn-Trace-Id`
## [1] "Root=1-69ae1852-7d7bea0413d797d9086cf2fe"

ctually, there’s also a httpbin.org address that only returns the current user agent (https://httpbin.org/user-agent). You’ll use this for the current exercise, where you’ll manipulate your own user agent to turn it into something meaningful (for the owners of the website you’re scraping, that is).

As you saw in the video, there are two ways of customizing your user agent when using httr for fetching web resources:

Locally, i.e. as an argument to the current request method. Globally via set_config()

# Pass a custom user agent to a GET query to the mentioned URL
response <- GET('https://httpbin.org/user-agent', user_agent("A request from a DataCamp course on scraping"))
# Print the response content
content(response)

## $`user-agent`
## [1] "A request from a DataCamp course on scraping"

# Globally set the user agent to "A request from a DataCamp course on scraping"
set_config(add_headers(`User-Agent` = "A request from a DataCamp course on scraping"))
# Pass a custom user agent to a GET query to the mentioned URL
response <- GET("https://httpbin.org/user-agent")
# Print the response content
content(response)

## $`user-agent`
## [1] "A request from a DataCamp course on scraping"