---
title: "Text wrangling and pattern matching with regex (regular expressions)"
subtitle: "Cornell R User Group"
date: "April 17, 2025"
author: "Tyler Wilcox"
format: html
---

```{r packages}
library(stringr)
library(dplyr)
library(tidyr)
```

## Regular Expressions

- Available in nearly every programming language, including R
- Useful way to automatically find, edit, or use patterns in any kind of text-based data
- Base R already supports them: `grep`, `regexpr`, 
- I recommend the [`stringr` package](https://stringr.tidyverse.org/index.html) for a more consistent set of tools that use regex and work well with data frames and can be easily used in a "tidy" workflow if you like [`dplyr`](https://dplyr.tidyverse.org)

### Simple Exact Matches

Find any occurrence of the two-letter sequence "an"

```{r}
x <- c("apple", "banana", "pear")
str_view(x, "an")
```

#### Replacement

If you can find it, you can replace it

```{r}
x <- c("tyler<AT>notadomain.com", "matt<AT>notadomain.com")
str_view(x, "<AT>")
str_replace(x, "<AT>", "@")
```

### Start and End of a String --- Anchors

#### End of a string `$`

The `$` lets you look for patterns at the *end* of a string.
Here, we can find all words that end with "ing"

```{r}
head(words)

str_view(words, pattern = "ing$")
```

You could omit the suffix "ing", especially useful when stemming words in text analysis.

```{r}
str_replace(words, pattern = "ing$", replacement = "-") |> 
    str_view(pattern = "-$")
```

#### Start of a string `^`

The complement of `$` is `^` --- `^` will locate patterns at the start of a string

Find all words that start with "al"

```{r}
str_view(words, pattern = "^al")
```

Putting `^` and `$` together, we can look for strings that start and end with patterns.

Here, we can find any words that start with "m" and end with "ing". Notice the use of `\\w+` to match any letters in between.

```{r}
str_view(words, pattern = "^m\\w+ing$")
```

### Fancier matches

#### "Word" characters `\w`

We just saw the `\\w` expression. This matches any letters, numbers, and an underscore ("_").

```{r}
x <- c("a.1", "a2", "b-4_")
str_extract_all(x, "\\w")
```

**Note**: In `R`, regex characters that are written with a single backslash such as `\w` need an extra backslash `\` since `R` treats the backslash as a special character. So if you find regex elsewhere, remember that a single backslash in most regex usually needs an extra `\`

The opposite of `\\w` is `\\W`, which matches anything that is *not* a letter, number, or underscore

```{r}
x <- c("a.1", "a2", "b-4_")
str_extract_all(x, "\\W")

# Compare with `\\w`
str_extract_all(x, "\\w")
```

There are many expressions, I recommend looking at the documentation in R to see your options

```{r}
?base::regex
```

#### Alphanumeric `[:alphanum:]`

One I use regularly is `[:alphanum:]` to match numbers and letters --- like `\\w`, but does not include underscores

```{r}
x <- c("a.1", "a2", "b-4_")
str_extract_all(x, "[:alnum:]")

# Compare with using `\\w`
str_extract_all(x, "\\w")
```

#### Character Classes `[abcdefg1357]`

You can set up matches for custom lists of characters by including inside a pair of square brackets `[]`.

Here, we can look for any occurences of "b" or "2"

```{r}
x <- c("a.1", "a2", "b-4_", "c2z")

str_view(x, "[b2]")
```

#### Negating a Character Class `[^abc135]`

If you want to match things that are *not* in your list of characters, precede that list with a `^`.

```{r}
x <- c("a.1", "a2", "b-4_", "c2z")

# Find "b" or "2"
str_view(x, "[b2]")

# Find anything EXCEPT "b" or "2"
str_view(x, "[^b2]")
```

#### Escaping Special Characters

Be careful if you try to match characters that are used as special expressions. You need to *escape* these characters with two backslashes `\\` for them to be treated as a literal character, not a special expression. For example, to match a period ".", we need to be escape it because `.` is an expression to match any single character.

```{r}
x <- c("a.1", "a2", "b-4_")

# This will match every single character!
str_view(x, pattern = ".")

# This will match only the period character
str_view(x, pattern = "\\.")
```

Other special characters that would need to be escaped include `. \ | ( ) [ { ^ $ * + ?`

We won't cover what all of these do, but to find literal matches for these characters, they need to be escaped, e.g., `\\?` to find a question mark in your text.

### Capture groups

One really useful thing regex will let you do is find complex patterns, and then extract a subset of text within those matched patterns.

For example, say I want to handle a list of emails but only am interested in extracting the handles, not the domain name information from the email addresses.

```{r}
x <- c("tyler@notadomain.com", "matt@gmeal.com", "erika@cornwall.edu")
```

I can use a capture group `()` to keep anything inside the parentheses

```{r}
str_match(x, pattern = "^(.+)@(.+\\.\\w+)$")
```

We could name the output columns for easier use by starting each capture group with `?<my_group_label>` followed by the regex for the match we are interested in. Here, we will label the extracted user handles and email domains.

```{r}
str_match(x, pattern = "^(?<handle>.+)@(?<domain>.+\\.\\w+)$")
```

### Multiple matches

#### At least one match `+`

The `+` operator I used just now lets you look for multiple matches for any expression that immediately precedes the `+`.

If I want to find all of the instances of "an" in our fruits example, we can use the `+`

```{r}
x <- c("apple", "banana", "pear")

# Without the `+` operator, we only match once
str_match(x, "an")

# With the `+` operator, we match multiple times in succession
x <- c("apple", "banana", "pear")
str_view(x, "(an)+")

# A subtle difference if we use `[]` instead of `()`
str_view(x, "[an]+")
```

#### No match or many matches `*`

The `*` operator I used just now lets you look for multiple matches in sequence or no matches at all.

If I want to find all of the instances of "an" in our fruits example, we can use the `+`

```{r}
x <- c("apple", "banana", "pear")

# With the `*` operator, we match one or more times in succession or none at all
x <- c("apple", "banana", "pear")
str_view(x, "(an)*")

# A subtle difference if we use `[]` instead of `()`
str_view(x, "[an]*")
```

#### Exact number of matches

`{n}` will match *exactly* n times

```{r}
x <- c("apple", "banana", "pear")
str_view(x, "(an){1}") # Match "an"
str_view(x, "(an){2}") # Match "anan"
```

`{n,}` will match n times *or more*

```{r}
str_view(x, "[an]{1,}") # Match at least one "a" or "n"
str_view(x, "[an]{2,}") # Match at least two in a row of "a" or "n"
```

You can also look for *at most* m matches: `{,m}`
Or *between* n and m matches: `{n,m}`

### Works in Data Frames, Too

This works with dataframes, so you can apply a regex operation to an entire column

```{r}
dat <- tibble(word = words,
              index = seq_along(word))

dat |> filter(str_detect(word, pattern = "ing$"))

# Find any word with "d" or "dd"
dat |> filter(str_detect(word, pattern = "d+"))
```

## Places I Find Regex Useful

### [`tidyr` package](https://dplyr.tidyverse.org)

- `separate_wider_regex()` lets you split up columns using regex

```{r}
df <- tibble(var = c("race_1", "race_2", "age_bucket_1", "age_bucket_2"))
df
```
1. Specify the pieces you want to split up into separate columns with a named regex
2. Unnamed regex will match in between but does not get kept as a column!

```{r}
# Here we split `var` into two columns: `name` and `number`
df %>% separate_wider_regex(var, c(var1 = ".*", "_", var2 = ".*"))
```

This is hard to split up otherwise, but we used the `*` to match zero or more single characters up to the first underscore and then, crucially, zero or more characters *after* the first underscore. This last piece let us handle rows where there was a single underscore in some cases and two underscores in others.

### Data Cleaning

## Learn More

- **R for Data Science** ebook chapter: [14.3 Matching patterns with regular expressions](https://r4ds.had.co.nz/strings.html#matching-patterns-with-regular-expressions)

- `stringr` [vignette](https://stringr.tidyverse.org/articles/regular-expressions.html)

- *Mastering Regular Expressions* [book](https://www.oreilly.com/library/view/mastering-regular-expressions/0596528124/) by Jeffrey Friedl

- regex "playground": <https://regex101.com>