--- title: "Lecture 5A: Tidy Data and the Tidyverse" subtitle: "STAT 545 - Fall 2026" format: live-revealjs: scrollable: true slide-number: true smaller: true menu: side: right width: wide webr: packages: - dplyr - tidyverse - knitr - RColorBrewer - ggplot2 persist: true editor: visual execute: echo: true warning: false engine: webr --- {{< include ./_extensions/r-wasm/live/_knitr.qmd >}} ```{r, include = F} library(tidyverse) ``` ## Today's Goals - recognize whether a given data set is ‘tidy’ or ‘untidy’ for their analysis - understand why ‘tidy’ data can be useful - reshape a data set between ‘long’ and ‘wide’ formats, using `tidyr::pivot_longer()` and `tidyr::pivot_wider()` - understand how to grapple with explicit missing values created by pivoting Quick links: - [Lecture Video](https://www.youtube.com/watch?v=pO-dgdfMRZw) ## Required Packages Install (if necessary) and load in the following packages: ```{webr} library(tidyverse) ``` ## The Tidyverse - a "meta"-package that contains a bunch of packages related to tidy data! - includes `dplyr`, `ggplot2`, `tidyr` (today's topic) and more ## What is "Tidy Data"? **Tidy data is defined as data where** - **each variables form a column,** - **each observation forms a row, and** - **each cell is a single measurement** What constitutes a variable, observation, and measurement will depend on the goal of your analysis! When we have tidy data, it makes our lives easier. ## What is "Tidy Data"? ![Artwork by Allison Horst (\@allisonhorst)](images/tidydata_2.jpg){fig-align="center"} ## Why Tidy Data? ![Artwork by Allison Horst (\@allisonhorst)](images/tidydata_3.jpg) ## Untidy Real Data Example | Owner | Pet 1 | Pet 2 | |-----------|------------------------------|-----------------| | Asfar | Cosmo (Orange Cat) | Zuko (Grey Dog) | | Grace | Dorito (Brown Chipmunk) | | | Zifu | Booger (Black and White Dog) | | | Flora | Mimi (White Cat) | | | Yvette | Mia (Beige and White Cat) | | | Alexandra | Bambi (Black and White Dog) | | | Erick | Maisy (Brown Cat) | | | Solana | Suki (White and Brown Cat) | | | Danielle | Enzo (Black Cat) | | | Kisa | Rio (Multicoloured Parrot) | | ## Tidy Data Example Instead, a tidy version of this data set may look like this: | Pet Name | Type | Owner | Colour | |----------|----------|-----------|-----------------| | Cosmo | Cat | Asfar | Orange | | Zuko | Dog | Asfar | Grey | | Dorito | Chipmunk | Grace | Brown | | Booger | Dog | Zifu | Black and White | | Mimi | Cat | Flora | White | | Mia | Cat | Yvette | Beige and White | | Bambi | Dog | Alexandra | Black and White | | Maisy | Cat | Erick | Brown | | Enzo | Cat | Danielle | Black | | Rio | Parrot | Kisa | Multicoloured | ## Discussion #1: Drinks The `fivethirtyeight` R package contains a dataset called `drinks`. This dataset was compiled as part of a [FiveThirtyEight article](https://fivethirtyeight.com/features/dear-mona-followup-where-do-people-drink-the-most-beer-wine-and-spirits/) that explored (among other things) which countries consumes the most alcohol. Let's look at a subset of the data: ```{webr, message = F, warning = F} # install.packages("fivethirtyeight") #uncomment if not installed library(fivethirtyeight) drinks_tbl1 <- as_tibble(drinks) %>% select(-total_litres_of_pure_alcohol) #remove total liters variable head(drinks_tbl1) #view the first few rows of the data ``` If we wanted to compare consumption across countries by drink type, is the data tidy? ::: {#editable-box contenteditable="true" style="border: 0.1px solid black; padding: 8px; font-size: 18px;"} Notes: ::: ## Discussion #2: Drinks The following graphic was made from the same `drinks` dataset. ![](/images/drinks.png){fig-align="center"} Without transforming the data (on previous slide), would we be able to plot this? ::: {contenteditable="true" style="border: 0.1px solid black; padding: 8px; font-size: 18px;"} Notes: ::: ## Basic Pivoting Once you have figured out what’s tidy for you, you may come to realize that your data is *not* tidy. As we have discussed, it will typically save you time and frustration to tidy it before moving on in your analysis. Very often this will involve using “pivoting” type functions. For example, the `tidyr` package in the `tidyverse` has two main pivoting functions: 1. `pivot_longer()` makes datasets *longer*: it moves some information in the columns into new rows, thereby *increasing* the number of *rows* of the dataset. 2. `pivot_wider()` makes datasets *wider*: it moves some information in the rows into new columns, thereby *decreasing* the number of rows of the dataset. ## **Pivoting Wider** Here is an example of (artificial) data that is currently in "long" format, looking grades before and after active learning was incorporated into their math and science lectures: ```{webr} #| context: setup grades_long <- tibble( student_id = c(1, 1, 1, 1, 2, 2,2, 2, 3, 3, 3, 3), time = c("Before", "After", "Before", "After", "Before", "After", "Before", "After", "Before", "After", "Before", "After"), subject = c("Math", "Math", "Science", "Science", "Math", "Math", "Science","Science", "Math", "Math", "Science", "Science"), score = c(85, 86, 92, 92, 78, 82, 88, 89, 95, 94, 90, 90) ) head(grades_long) ``` Sometimes, we require data involving time to have row for each observation, and one column for each time point (i.e., wide data). While this isn't our "standard" definition of tidy, sometimes this is what is required for *longitudinal data* (i.e., data involving measures repeated over time). Nevertheless, let's try to pivot this table! ## Pivoting Wider `pivot_wider` needs three pieces of information: - What is a set of columns that uniquely identifies each observation? Put their names in the `id_cols` argument. - Where should the names for the new columns come from? Put the name of the column you want to take the new variable names from in the `names_from` argument. - What values should the new columns contain? Put the name of the columns you want to take the values from to `values_from` in the `values_from` argument. Note that if you don’t specify an `id_cols` argument, `pivot_wider` will assume that you want it to be every column except those in `names_from` and `values_from`. Further, any columns not included in `id_cols`, `names_from`, and `values_from` (e.g., `ep_id`) will simply be dropped. ## Pivoting Wider: Grades Example Recall the structure of `grades_long`: ```{webr} head(grades_long, 4) #print first 4 rows ``` Can you pivot this table wider so that we have a columns "Scores_Before" and "Scores_After" for each student_id and subject? ```{webr} grades_wide <- grades_long %>% #assign new variable pivot_wider(names_from= ..., #get column names from ... values_from= ..., #fill in values from ... names_prefix = "Scores_") #add "scores_" to new column names head(grades_wide) ``` ### ## **Pivoting Longer** Here is a snippet of WHO data on the number of tuberculosis cases in different years in different countries. ```{webr} who_wide <- tibble(country = c("Afghanistan", "Brazil", "China"), `1999` = c(745, 37737, 212258), `2000` = c(2666, 80488, 213766)) who_wide ``` If we wanted to compare tuberculosis cases over time by country (e.g. by plotting the year on the x-axis and case count on the y-axis with a line for each country), then this format is not tidy. there should be one observation per unit within each population (country-years). In this case, we do not observe units within each country-year, so each observation is a country-year. The variables then fall into place: the country and year labels, and the case counts. Tidy Format: variables (the year, the country, and the case counts) on the columns. There are 6 rows, one for each unique country-year combination. In this example, the tidy format is *longer*. That means to produce it using `who_wide`, we need to lengthen it by moving some information in the column names (the info about the measurement year) into new rows. ## Pivoting Longer `pivot_longer` needs three pieces of information: - Which are the columns that we want to expand into more rows? Put their names in the `cols` argument. - We want to save the information in the names of those columns as values in new column(s) of our dataset. What should we name these new column(s)? This is the `names_to` argument. - We also want to preserve the information in the *values* of those columns - so we should save them as values in a new column of our dataset. What should we name it? This is the `values_to` argument. ## Pivoting Longer: WHO Example ```{webr} who_long <- who_wide %>% pivot_longer(cols = c(..., ...), #pivot these columns names_to = "...", #name new column "year" values_to = "...") #use ... as the values of the column who_long ``` Now we've got tidy data! ## Pivoting Longer: Grades Example Challenge yourself to transform `grades_wide` back to `grades_long`. Rename your new dataset `grades_long_pivoted` and compare the two long forms. ```{webr} grades_long_pivoted <- grades_wide %>% ... head(grades_long_pivoted) head(grades_long) ``` ## Pivoting Longer with Multiple Variables: WHO Example #### **Example: Column Names Contain Multiple Variable Values** Here’s a more realistic (but still simplified) look at the WHO Tuberculosis data. ```{webr} who_demo <- who2 %>% select(country, year, starts_with("sp")) %>% rename_with(function(x) str_remove(x, pattern="sp_"), starts_with("sp")) %>% filter(year %in% c(1999, 2000)) %>% filter(country %in% c("Afghanistan", "Brazil", "China")) head(who_demo) ``` This time, cases are broken down by gender (`f` or `m` only considered in this study) and by age range (`014` (0 - 14), `1524` (15 - 24), `2534` (25 - 34), and so on) Suppose now that we are interested in comparing tuberculosis rates over time across (potentially) gender, age, and country. Then the most granular population we are trying to describe is a country, gender, age, and year combination, and like in the last example, we have no measured sub-units within that population, so an observation is a unique combination of country, gender, age, and year. (What a mouthful!) ## Pivoting Longer with Multiple Variables: WHO Example Variables of interest: country, year, gender, age range, and case count. Values for gender and age range are currently located in the column names of `who_demo`, and values for case count are currently spread across multiple columns. So to tidy `who_demo` up, we need to use `pivot_longer()` to move the info in the columns into new rows. ```{webr} who_demo %>% pivot_longer(cols = !(country:year), # all columns aside from country to year names_to = c("gender", "age"), #new columns named age and gender names_sep = "_",#current gender and age are a single variable separated by _ values_to = "cases") #use the cases column for the values ``` Phew! ## Separating and Uniting Sometimes data has multiple pieces of information stored in a single column. To separate them into two distinct columns, we can use the `separate()` function from the `tidyverse` (technically, a package within tidyverse called `tidyr`. `separate():` separates information within a column. It involves three arguments: - `col`: specifies the column we want to separate - `into`: specifies the names of the new columns. - `sep`: specifies where we want to cut. The default is pretty clever - it separates at any non-alphanumeric value. (How this is accomplished involves *regular expressions*, which are very useful when working with character data. We will learn more about regular expressions in STAT 545B. ) ## Separating and Uniting Conversely, `unite()` combines information from two or more columns into one. Involves: - `col`: name of the new column - `...`: the names of columns to be united - `sep`: specifies the separator to use between values (i.e., "-") ## Separating and Uniting: WHO Example The `tidyr` package has a function for gluing columns together (`unite`) and for cutting columns apart (`separate`). Why might this help us tidy? Here is another snippet of WHO Tuberculosis data. ```{webr} table3 <- tibble(country = c("Afghanistan", "Afghanistan", "Brazil", "Brazil", "China", "China"), year = c(1999, 2000, 1999, 2000, 1999, 2000), rate = c("745/19987071", "2666/20595360", "37737/172006362", "80488/174504898", "212258/1272915272", "213766/1280428583")) table3 ``` The `rate` column contains the values of two variables: case counts and population counts. We would like to snip it apart at the “/” character to create two columns: ```{webr} table3_sep <- table3 %>% separate(col = ... , #column we wish to separate into = c( ..., ...)) #names of new columns table3_sep ``` ## Untidy Data As we’ve seen, tidy data is often very helpful. But there are also times when *un*tidy data is good. Here are a few reasons: - The format that lends itself best to fast computations might not be tidy. [Case Study: Tidy Genomics](http://varianceexplained.org/r/tidy-genomics/). - Untidy data is often easier for humans to interpret and edit. See [Untidy Data: The Unreasonable Effectiveness of Tables](https://arxiv.org/pdf/2106.15005.pdf). - We could lose important information about the context in which the data was collected by cleaning and tidying raw data. This can have important ethical implications; see Chapter 5 of the book [“Data Feminism”](https://data-feminism.mitpress.mit.edu/) by Catherine D’Ignazio and Lauren F. Klein. In summary, tidiness is a very useful concept, and tidying data is often useful. But we should remember that absolutes are few and far between in data science and statistics. Just because tidying data is *often* useful, doesn’t mean it’s *always* useful. ## Worksheet A4 We will spend the remainder of this class working through Worksheet A4. ## To Save These Slides With Your Code To Print to PDF (with your code!), do the following: - Open the in-browser print dialog `CMD/CTRL + P` or Right click \> Print - Change the Destination setting to Save as PDF. - Change the Layout to Landscape. - Change the Margins to None. Enable the Background graphics option. - Click Save 🎉 In all cases, ensure the code chunks don't get cut off (i.e., a line is too long). You will **not** be able to edit these from the PDF directly (though you could open the slides again and copy-and-paste to re-run the code!)