Chapter 2 Getting Started with Data in R
Before we can start exploring data in R, there are some key concepts to understand first:
- What are R and RStudio?
- How do I code in R?
- What are R packages?
We’ll introduce these concepts in upcoming Sections 2.1-2.3. If you are already somewhat familiar with these concepts, feel free to skip to Section 2.4 where we’ll introduce our first data set: all domestic flights departing a New York City airport in 2013. This is a dataset we will explore in depth in this book.
2.1 What are R and RStudio?
For much of this book, we will assume that you are using R via RStudio. First time users often confuse the two. At its simplest:
- R is like a car’s engine.
- RStudio is like a car’s dashboard.
|R: Engine||RStudio: Dashboard|
More precisely, R is a programming language that runs computations while RStudio is an integrated development environment (IDE) that provides an interface by adding many convenient features and tools. So the way of having access to a speedometer, rearview mirrors, and a navigation system makes driving much easier, using RStudio’s interface makes using R much easier as well.
2.1.1 Installing R and RStudio
Note about RStudio Server: If your instructor has provided you with a link and access to RStudio Server, then you can skip this section. We do recommend though after a few months of working on the RStudio Server that you return to these instructions.
You will first need to download and install both R and RStudio (Desktop version) on your computer.
- You must do this first: Download and install R.
- Click on the download link corresponding to your computer’s operating system.
- You must do this second: Download and install RStudio.
- Scroll down to “Installers for Supported Platforms”
- Click on the download link corresponding to your computer’s operating system.
2.1.2 Using R via RStudio
Recall our car analogy from above. Much as we don’t drive a car by interacting directly with the engine but rather by interacting with elements on the car’s dashboard, we won’t be using R directly but rather we will use RStudio’s interface. After you install R and RStudio on your computer, you’ll have two new programs AKA applications you can open. We will always work in RStudio and not R. In other words:
|R: Do not open this||RStudio: Open this|
After you open RStudio, you should see the following:
Note the three panes, which are three panels dividing the screen: The Console pane, the Files pane, and the Environment pane. Over the course of this chapter, you’ll come to learn what purpose each of these panes serve.
2.2 How do I code in R?
Now that you’re set up with R and RStudio, you are probably asking yourself “OK. Now how do I use R?” The first thing to note as that unlike other statistical software programs like Excel, STATA, or SAS that provide point and click interfaces, R is an interpreted language, meaning you have to enter in R commands written in R code. In other words, you have to code/program in R. Note that we’ll use the terms “coding” and “programming” interchangeably in this book.
While it is not required to be a seasoned coder/computer programmer to use R, there is still a set of basic programming concepts that R users need to understand. Consequently, while this book is not a book on programming, you will still learn just enough of these basic programming concepts needed to explore and analyze data effectively.
2.2.1 Basic programming concepts and terminology
We now introduce some basic programming concepts and terminology that you’ll learn as you go. Note that in this book, we will use a different font to distinguish regular font from
It is important to note that while these tutorials serve as excellent introductions, a single pass through them is insufficient for long-term learning and retention. The ultimate tools for long-term learning and retention are “learning by doing” and repetition, something we will have you do over the course of the entire book and we encourage this process as much as possible as you learn any new skill.
- Console pane: where you enter in commands
- Objects: where values are saved, how to assign values to objects.
- Data types: integers, doubles/numerics, logicals, characters.
- Vectors: a series of values. These are created using the
c()stands for “combine” or “concatenate”. For example:
c(6, 11, 13, 31, 90, 92).
- Factors: Categorical data (as opposed to numerical data) are represented in R as
- Data frames: They are analogous to rectangular spreadsheets: they are representations of datasets in R where the rows correspond observations and the columns correspond to variables that describe the observations. We will revisit this later in Section 2.4.
- Testing for equality in R using
=which is typically used for assignment). Ex:
2 + 1 == 3compares
2 + 1to
3and is correct R syntax, while
2 + 1 = 3is not and is incorrect R syntax.
- Boolean algebra:
TRUE/FALSEstatements and mathematical operators such as
<=(less than or equal), and
!=(not equal to).
- Logical operators:
|representing “or”. Ex:
(2 + 1 == 3) & (2 + 1 == 4)returns
(2 + 1 == 3) | (2 + 1 == 4)returns
- Testing for equality in R using
- Concept of functions: they take in inputs (called arguments) and return outputs.
- You either manually specify a function’s arguments or use the function’s defaults.
This list is by no means an exhaustive list of all the programming concepts and terminology needed to become a savvy R user; such a list would be so large it wouldn’t be very useful, especially for novices. Rather, we feel this is the bare minimum you need to know before you get started; the rest we feel you can learn as you go. Remember that your knowledge of all of these concepts will build as you get better and better at “speaking R” and getting used to its syntax.
2.2.2 Errors, warnings, and messages
One slightly confusing part of R is how it reports errors, warnings, and messages. The default theme in RStudio colors errors, warnings, and messages in red, which makes them seem like you did something wrong. However, seeing red text in the console is not always bad.
R will show red text in the console in three different situations:
- Errors: When the red text is a legitimate error, it will be prefaced with “Error in…” and try to explain what went wrong. Generally when there’s an error, the code will not run. For example, as shown in Subsection 2.3.3 below if you see
Error in ggplot(...) : could not find function "ggplot", it means that the
ggplot()function is not accessible because the package was not loaded with
library(ggplot2), and thus you cannot use it.
- Warnings: When the red text is a warning, it will be prefaced with “Warning:” and try to explain why there’s a warning. Generally your code will still work, but with some caveats. For example, you see in Chapter 3 if you plot a scatterplot and one of the rows in your data frame is missing a value, you will see this warning:
Warning: Removed 1 rows containing missing values (geom_point). R will still make the scatterplot with all the remaining values, but it’s warning you that one of the points isn’t there.
- Messages: When the red text doesn’t start with either “Error” or “Warning”, it’s just a friendly message. You’ll see these messages when you load some packages like the
dplyrpackage in Subsection 2.3.2 below, or when you read data saved in spreadsheet files with
read_csv()as you’ll see in Chapter 5. These are helpful diagnostic messages and they don’t stop your code from working.
Remember, when you see red text in the console, don’t panic. It doesn’t necessarily mean anything is wrong.
- If the text starts with “Error”, figure out what’s causing it. Think of errors as a red traffic light: something is wrong!
- If the text starts with “Warning”, figure out if it’s something to worry about. For instance, if you get a warning about missing values in a scatterplot and you know there are missing values, you’re fine. If that’s surprising, look at your data and see what’s missing. Think of warnings as a yellow traffic light: everything is working fine, but watch out/pay attention.
- Otherwise the text is just a message. Read it, wave back at R, and thank it for talking to you. Think of messages as a green traffic light: everything is working fine.
2.2.3 Tips on learning to code
Learning to code/program is very much like learning a foreign language, it can be very daunting and frustrating at first. Such frustrations are very common and it is very normal to feel discouraged as you learn. However just as with learning a foreign language, if you put in the effort and are not afraid to make mistakes, anybody can learn.
Here are a few useful tips to keep in mind as you learn to program:
- Remember that computers are not actually that smart: You may think your computer or smartphone are “smart,” but really people spent a lot of time and energy designing them to appear “smart.” Rather you have to tell a computer everything it needs to do. Furthermore the instructions you give your computer can’t have any mistakes in them, nor can they be ambiguous in any way.
- Take the “copy, paste, and tweak” approach: Especially when learning your first programming language, it is often much easier to taking existing code that you know works and modify it to suit your ends, rather than trying to write new code from scratch. We call this the copy, paste, and tweak approach. So early on, we suggest not trying to write code from memory, but rather take existing examples we have provided you, then copy, paste, and tweak them to suit your goals. Don’t be afraid to play around!
- The best way to learn to code is by doing: Rather than learning to code for its own sake, we feel that learning to code goes much smoother when you have a goal in mind or when you are working on a particular project, like analyzing data that you are interested in.
- Practice is key: Just as the only method to improving your foreign language skills is through practice, practice, and practice; so also the only method to improving your coding is through practice, practice, and practice. Don’t worry however; we’ll give you plenty of opportunities to do so!
2.3 What are R packages?
Another point of confusion with many new R users is the idea of an R package. R packages extend the functionality of R by providing additional functions, data, and documentation. They are written by a world-wide community of R users and can be downloaded for free from the internet. For example, among the many packages we will use in this book are:
ggplot2package for data visualization in Chapter 3.
dplyrpackage for data wrangling in Chapter 4.
moderndivepackage that accompanies this book.
inferpackage for “tidy” and transparent statistical inference in Chapters 9, 10, and 11.
A good analogy for R packages is they are like apps you can download onto a mobile phone:
|R: A new phone||R Packages: Apps you can download|
So R is like a new mobile phone: while it has a certain amount of features when you use it for the first time, it doesn’t have everything. R packages are like the apps you can download onto your phone from Apple’s App Store or Android’s Google Play.
Let’s continue this analogy by considering the Instagram app for editing and sharing pictures. Say you have purchased a new phone and you would like to share a recent photo you have taken on Instagram. You need to:
- Install the app: Since your phone is new and does not include the Instagram app, you need to download the app from either the App Store or Google Play. You do this once and you’re set. You might do this again in the future any time there is an update to the app.
- Open the app: After you’ve installed Instagram, you need to open the app.
Once Instagram is open on your phone, you can then proceed to share your photo with your friends and family. The process is very similar for using an R package. You need to:
- Install the package: This is like installing an app on your phone. Most packages are not installed by default when you install R and RStudio. Thus if you want to use a package for the first time, you need to install it first. Once you’ve installed a package, you likely won’t install it again unless you want to update it to a newer version.
- “Load” the package: “Loading” a package is like opening an app on your phone. Packages are not “loaded” by default when you start RStudio on your computer; you need to “load” each package you want to use every time you start RStudio.
Let’s now show you how to perform these two steps for the
ggplot2 package for data visualization.
2.3.1 Package installation
Note about RStudio Server: If your instructor has provided you with a link and access to RStudio Server, you probably will not need to install packages, as they have likely been pre-installed for you by your instructor. That being said, it is still a good idea to know this process for later on when you are not using RStudio Server, but rather RStudio Desktop on your own computer.
There are two ways to install an R package. For example, to install the
- Easy way: In the Files pane of RStudio:
- Click on the “Packages” tab
- Click on “Install”
- Type the name of the package under “Packages (separate multiple with space or comma):” In this case, type
- Click “Install”
- Slightly harder way: An alternative but slightly less convenient way to install a package is by typing
install.packages("ggplot2")in the Console pane of RStudio and hitting enter. Note you must include the quotation marks.
Much like an app on your phone, you only have to install a package once. However, if you want to update an already installed package to a newer version, you need to re-install it by repeating the above steps.
knitrpackages. This will install the earlier mentioned
nycflights13package containing data on all domestic flights leaving a NYC airport in 2013, and the
knitrpackage for writing reports in R.
2.3.2 Package loading
Recall that after you’ve installed a package, you need to “load” it, in other words open it. We do this by using the
library() command. For example, to load the
ggplot2 package, run the following code in the Console pane. What do we mean by “run the following code”? Either type or copy & paste the following code into the Console pane and then hit the enter key.
If after running the above code, a blinking cursor returns next to the
> “prompt” sign, it means you were successful and the
ggplot2 package is now loaded and ready to use. If however, you get a red “error message” that reads…
Error in library(ggplot2) : there is no package called ‘ggplot2’
… it means that you didn’t successfully install it. In that case, go back to the previous subsection “Package installation” and install it.
knitrpackages as well by repeating the above steps.
2.3.3 Package use
One extremely common mistake new R users make when wanting to use particular packages is they forget to “load” them first by using the
library() command we just saw. Remember: you have to load each package you want to use every time you start RStudio. If you don’t first “load” a package, but attempt to use one of its features, you’ll see an error message similar to:
Error: could not find function
R is telling you that you are trying to use a function in a package that has not yet been “loaded.” Almost all new users forget do this when starting out, and it is a little annoying to get used. However, you’ll remember with practice.
2.4 Explore your first datasets
Let’s put everything we’ve learned so far into practice and start exploring some real data! Data comes to us in a variety of formats, from pictures to text to numbers. Throughout this book, we’ll focus on datasets that are saved in “spreadsheet”-type format; this is probably the most common way data are collected and saved in many fields. Remember from Subsection 2.2.1 that these “spreadsheet”-type datasets are called data frames in R; we will focus on working with data saved as data frames throughout this book.
Let’s first load all the packages needed for this chapter, assuming you’ve already installed them. Read Section 2.3 for information on how to install and load R packages if you haven’t already.
library(nycflights13) library(dplyr) library(knitr)
At the beginning of all subsequent chapters in this text, we’ll always have a list of packages that you should have installed and loaded to work with that chapter’s R code.
Many of us have flown on airplanes or know someone who has. Air travel has become an ever-present aspect in many people’s lives. If you live in or are visiting a relatively large city and you walk around that city’s airport, you see gates showing flight information from many different airlines. And you will frequently see that some flights are delayed because of a variety of conditions. Are there ways that we can avoid having to deal with these flight delays?
We’d all like to arrive at our destinations on time whenever possible. (Unless you secretly love hanging out at airports. If you are one of these people, pretend for the moment that you are very much anticipating being at your final destination.) Throughout this book, we’re going to analyze data related to flights contained in the
nycflights13 package (Wickham 2018). Specifically, this package contains five data sets saved in five separate data frames with information about all domestic flights departing from New York City in 2013. These include Newark Liberty International (EWR), John F. Kennedy International (JFK), and LaGuardia (LGA) airports:
flights: Information on all 336,776 flights
airlines: A table matching airline names and their two letter IATA airline codes (also known as carrier codes) for 16 airline companies
planes: Information about each of 3,322 physical aircraft used.
weather: Hourly meteorological data for each of the three NYC airports. This data frame has 26,115 rows, roughly corresponding to the 365 \(\times\) 24 \(\times\) 3 = 26,280 possible hourly measurements one can observe at three locations over the course of a year.
airports: Airport names, codes, and locations for 1,458 destination airports.
flights data frame
We will begin by exploring the
flights data frame that is included in the
nycflights13 package and getting an idea of its structure. Run the following code in your console (either by typing it or cutting & pasting it): it loads in the
flights dataset into your Console. Note depending on the size of your monitor, the output may vary slightly.
# A tibble: 336,776 x 19 year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time <int> <int> <int> <int> <int> <dbl> <int> <int> 1 2013 1 1 517 515 2 830 819 2 2013 1 1 533 529 4 850 830 3 2013 1 1 542 540 2 923 850 4 2013 1 1 544 545 -1 1004 1022 5 2013 1 1 554 600 -6 812 837 6 2013 1 1 554 558 -4 740 728 7 2013 1 1 555 600 -5 913 854 8 2013 1 1 557 600 -3 709 723 9 2013 1 1 557 600 -3 838 846 10 2013 1 1 558 600 -2 753 745 # … with 336,766 more rows, and 11 more variables: arr_delay <dbl>, # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>, # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
Let’s unpack this output:
A tibble: 336,776 x 19: A
tibbleis a kind of data frame used in R. This particular data frame has
19columns corresponding to 19 variables describing each observation
year month day dep_time sched_dep_time dep_delay arr_timeare different columns, in other words variables, of this data frame.
- We then have the first 10 rows of observations corresponding to 10 flights.
... with 336,766 more rows, and 11 more variables:indicating to us that 336,766 more rows of data and 11 more variables could not fit in this screen.
Unfortunately, this output does not allow us to explore the data very well. Let’s look at different tools to explore data frames.
2.4.3 Exploring data frames
Among the many ways of getting a feel for the data contained in a data frame such as
flights, we present three functions that take as their “argument”, in other words their input, the data frame in question. We also include a fourth method for exploring one particular column of a data frame:
- Using the
View()function built for use in RStudio. We will use this the most.
- Using the
glimpse()function, which is included in the
- Using the
kable()function, which is included in the
- Using the
$operator to view a single variable in a data frame.
View(flights) in your Console in RStudio, either by typing it or cutting & pasting it into the Console pane, and explore this data frame in the resulting pop-up viewer. You should get into the habit of always
Viewing any data frames that come your way. Note the capital “V” in
View. R is case-sensitive so you’ll receive an error is you run
view(flights) instead of
(LC2.3) What does any ONE row in this
flights dataset refer to?
- A. Data on an airline
- B. Data on a flight
- C. Data on an airport
- D. Data on multiple flights
View(flights), we see the different variables listed in the columns and we see that there are different types of variables. Some of the variables like
arr_delay are what we will call quantitative variables. These variables are numerical in nature. Other variables here are categorical.
Note that if you look in the leftmost column of the
View(flights) output, you will see a column of numbers. These are the row numbers of the dataset. If you glance across a row with the same number, say row 5, you can get an idea of what each row corresponds to. In other words, this will allow you to identify what object is being referred to in a given row. This is often called the observational unit. The observational unit in this example is an individual flight departing New York City in 2013. You can identify the observational unit by determining what “thing” is being measured or described by each of the variables. We’ll talk more about observational units in Section 2.4.4 on identification and measurement variables below.
The second way to explore a data frame is using the
glimpse() function included in the
dplyr package. Thus, you can only use the
glimpse() function after you’ve loaded the
dplyr package. This function provides us with an alternative method for exploring a data frame than the
Observations: 336,776 Variables: 19 $ year <int> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013, … $ month <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, … $ day <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, … $ dep_time <int> 517, 533, 542, 544, 554, 554, 555, 557, 557, 558, 558,… $ sched_dep_time <int> 515, 529, 540, 545, 600, 558, 600, 600, 600, 600, 600,… $ dep_delay <dbl> 2, 4, 2, -1, -6, -4, -5, -3, -3, -2, -2, -2, -2, -2, -… $ arr_time <int> 830, 850, 923, 1004, 812, 740, 913, 709, 838, 753, 849… $ sched_arr_time <int> 819, 830, 850, 1022, 837, 728, 854, 723, 846, 745, 851… $ arr_delay <dbl> 11, 20, 33, -18, -25, 12, 19, -14, -8, 8, -2, -3, 7, -… $ carrier <chr> "UA", "UA", "AA", "B6", "DL", "UA", "B6", "EV", "B6", … $ flight <int> 1545, 1714, 1141, 725, 461, 1696, 507, 5708, 79, 301, … $ tailnum <chr> "N14228", "N24211", "N619AA", "N804JB", "N668DN", "N39… $ origin <chr> "EWR", "LGA", "JFK", "JFK", "LGA", "EWR", "EWR", "LGA"… $ dest <chr> "IAH", "IAH", "MIA", "BQN", "ATL", "ORD", "FLL", "IAD"… $ air_time <dbl> 227, 227, 160, 183, 116, 150, 158, 53, 140, 138, 149, … $ distance <dbl> 1400, 1416, 1089, 1576, 762, 719, 1065, 229, 944, 733,… $ hour <dbl> 5, 5, 5, 5, 6, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 6, 6, … $ minute <dbl> 15, 29, 40, 45, 0, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 59, … $ time_hour <dttm> 2013-01-01 05:00:00, 2013-01-01 05:00:00, 2013-01-01 …
We see that
glimpse() will give you the first few entries of each variable in a row after the variable. In addition, the data type (see Subsection 2.2.1) of the variable is given immediately after each variable’s name inside
< >. Here,
dbl refer to “integer” and “double”, which are computer coding terminology for quantitative/numerical variables. In contrast,
chr refers to “character”, which is computer terminology for text data. Text data, such as the
origin of a flight, are categorical variables. The
time_hour variable is an example of one more type of data type:
dttm. As you may suspect, this variable corresponds to a specific date and time of day. However, we won’t work with dates in this class and leave it to a more advanced book on data science.
(LC2.4) What are some examples in this dataset of categorical variables? What makes them different than quantitative variables?
The final way to explore the entirety of a data frame is using the
kable() function from the
knitr package. Let’s explore the different carrier codes for all the airlines in our dataset two ways. Run both of these lines of code in your Console:
At first glance, it may not appear that there is much difference in the outputs. However when using tools for document production such as R Markdown, the latter code produces output that is much more legible and reader-friendly.
$ operator allows us to explore a single variable within a data frame. For example, run the following in your console
We used the
$ operator to extract only the
name variable and return it as a vector of length 16. We will only be occasionally exploring data frames using this operator, instead favoring the
2.4.4 Identification & measurement variables
There is a subtle difference between the kinds of variables that you will encounter in data frames: identification variables and measurement variables. For example, let’s explore the
airports data frame by showing the output of
Observations: 1,458 Variables: 8 $ faa <chr> "04G", "06A", "06C", "06N", "09J", "0A9", "0G6", "0G7", "0P2", … $ name <chr> "Lansdowne Airport", "Moton Field Municipal Airport", "Schaumbu… $ lat <dbl> 41.1, 32.5, 42.0, 41.4, 31.1, 36.4, 41.5, 42.9, 39.8, 48.1, 39.… $ lon <dbl> -80.6, -85.7, -88.1, -74.4, -81.4, -82.2, -84.5, -76.8, -76.6, … $ alt <int> 1044, 264, 801, 523, 11, 1593, 730, 492, 1000, 108, 409, 875, 1… $ tz <dbl> -5, -6, -6, -5, -5, -5, -5, -5, -5, -8, -5, -6, -5, -5, -5, -5,… $ dst <chr> "A", "A", "A", "A", "A", "A", "A", "A", "U", "A", "A", "U", "A"… $ tzone <chr> "America/New_York", "America/Chicago", "America/Chicago", "Amer…
name are what we will call identification variables: variables that uniquely identify each observational unit. They are mainly used to provide a unique name to each observational unit i.e. row, thereby allowing us to uniquely identify them.
faa gives the unique code provided by the FAA for that airport, while the
name variable gives the longer more natural name of the airport. The remaining variables (
tzone) are often called measurement or characteristic variables: variables that describe properties of each observational unit, in other words each observation in each row. For example,
long describe the latitude and longitude of each airport.
Furthermore, sometimes a single variable might not be enough to uniquely identify each observational unit: combinations of variables might be needed. While it is not an absolute rule, for organizational purposes it is considered good practice to have your identification variables in the left-most columns of your data frame.
(LC2.5) What properties of the observational unit do each of
tzone describe for the
airports data frame? Note that you may want to use
?airports to get more information.
(LC2.6) Provide the names of variables in a data frame with at least three variables in which one of them is an identification variable and the other two are not. In other words, create your own tidy data frame that matches these conditions.
2.4.5 Help files
Another nice feature of R is the help system. You can get help in R by entering a
? before the name of a function or data frame in question and you will be presented with a page showing the documentation. For example, let’s look at the help file for the
flights data frame:
A help file should pop-up in the Help pane of RStudio. If you have questions about a function or data frame included in an R package, you should get in the habit of consulting the help file right away.
We’ve given you what we feel are the most essential concepts to know before you can start exploring data in R. Is this chapter exhaustive? Absolutely not. To try to include everything in this chapter would make the chapter so large it wouldn’t be useful!
2.5.1 Additional resources
If you are completely new to the world of coding, R, and RStudio and feel you could benefit from a more detailed introduction, we suggest you check out ModernDive co-author Chester Ismay’s Getting used to R, RStudio, and R Markdown short book (Ismay 2016), which includes screencast recordings that you can follow along and pause as you learn. Furthermore, there is an introduction to R Markdown, a tool used for reproducible research in R.
2.5.2 What’s to come?
As we stated earlier however, the best way to learn R is to learn by doing. We now start the “data science” portion of the book in Chapter 3 with what we feel is the most important tool in a data scientist’s toolbox: data visualization. We will continue to explore the data included in the
nycflights13 package through data visualization. We’ll see that data visualization is a powerful tool to add to our toolbox for data exploring that provides additional insight to what the
glimpse() functions can provide.