Text wrangling and pattern matching with regex (regular expressions)

Cornell R User Group

Author

Tyler Wilcox

Published

April 17, 2025

library(stringr)
library(dplyr)

Attaching package: 'dplyr'
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union
library(tidyr)

Regular Expressions

  • Available in nearly every programming language, including R
  • Useful way to automatically find, edit, or use patterns in any kind of text-based data
  • Base R already supports them: grep, regexpr,
  • I recommend the stringr package for a more consistent set of tools that use regex and work well with data frames and can be easily used in a “tidy” workflow if you like dplyr

Simple Exact Matches

Find any occurrence of the two-letter sequence “an”

x <- c("apple", "banana", "pear")
str_view(x, "an")
[2] │ b<an><an>a

Replacement

If you can find it, you can replace it

x <- c("tyler<AT>notadomain.com", "matt<AT>notadomain.com")
str_view(x, "<AT>")
[1] │ tyler<<AT>>notadomain.com
[2] │ matt<<AT>>notadomain.com
str_replace(x, "<AT>", "@")
[1] "tyler@notadomain.com" "matt@notadomain.com"

Start and End of a String — Anchors

End of a string $

The $ lets you look for patterns at the end of a string. Here, we can find all words that end with “ing”

head(words)
[1] "a"        "able"     "about"    "absolute" "accept"   "account" 
str_view(words, pattern = "ing$")
[113] │ br<ing>
[251] │ dur<ing>
[280] │ even<ing>
[448] │ k<ing>
[512] │ mean<ing>
[533] │ morn<ing>
[709] │ r<ing>
[765] │ s<ing>
[860] │ th<ing>

You could omit the suffix “ing”, especially useful when stemming words in text analysis.

str_replace(words, pattern = "ing$", replacement = "-") |> 
    str_view(pattern = "-$")
[113] │ br<->
[251] │ dur<->
[280] │ even<->
[448] │ k<->
[512] │ mean<->
[533] │ morn<->
[709] │ r<->
[765] │ s<->
[860] │ th<->

Start of a string ^

The complement of $ is ^ — ^ will locate patterns at the start of a string

Find all words that start with “al”

str_view(words, pattern = "^al")
[27] │ <al>l
[28] │ <al>low
[29] │ <al>most
[30] │ <al>ong
[31] │ <al>ready
[32] │ <al>right
[33] │ <al>so
[34] │ <al>though
[35] │ <al>ways

Putting ^ and $ together, we can look for strings that start and end with patterns.

Here, we can find any words that start with “m” and end with “ing”. Notice the use of \\w+ to match any letters in between.

str_view(words, pattern = "^m\\w+ing$")
[512] │ <meaning>
[533] │ <morning>

Fancier matches

“Word” characters \w

We just saw the \\w expression. This matches any letters, numbers, and an underscore (“_“).

x <- c("a.1", "a2", "b-4_")
str_extract_all(x, "\\w")
[[1]]
[1] "a" "1"

[[2]]
[1] "a" "2"

[[3]]
[1] "b" "4" "_"

Note: In R, regex characters that are written with a single backslash such as \w need an extra backslash \ since R treats the backslash as a special character. So if you find regex elsewhere, remember that a single backslash in most regex usually needs an extra \

The opposite of \\w is \\W, which matches anything that is not a letter, number, or underscore

x <- c("a.1", "a2", "b-4_")
str_extract_all(x, "\\W")
[[1]]
[1] "."

[[2]]
character(0)

[[3]]
[1] "-"
# Compare with `\\w`
str_extract_all(x, "\\w")
[[1]]
[1] "a" "1"

[[2]]
[1] "a" "2"

[[3]]
[1] "b" "4" "_"

There are many expressions, I recommend looking at the documentation in R to see your options

?base::regex

Alphanumeric [:alphanum:]

One I use regularly is [:alphanum:] to match numbers and letters — like \\w, but does not include underscores

x <- c("a.1", "a2", "b-4_")
str_extract_all(x, "[:alnum:]")
[[1]]
[1] "a" "1"

[[2]]
[1] "a" "2"

[[3]]
[1] "b" "4"
# Compare with using `\\w`
str_extract_all(x, "\\w")
[[1]]
[1] "a" "1"

[[2]]
[1] "a" "2"

[[3]]
[1] "b" "4" "_"

Character Classes [abcdefg1357]

You can set up matches for custom lists of characters by including inside a pair of square brackets [].

Here, we can look for any occurences of “b” or “2”

x <- c("a.1", "a2", "b-4_", "c2z")

str_view(x, "[b2]")
[2] │ a<2>
[3] │ <b>-4_
[4] │ c<2>z

Negating a Character Class [^abc135]

If you want to match things that are not in your list of characters, precede that list with a ^.

x <- c("a.1", "a2", "b-4_", "c2z")

# Find "b" or "2"
str_view(x, "[b2]")
[2] │ a<2>
[3] │ <b>-4_
[4] │ c<2>z
# Find anything EXCEPT "b" or "2"
str_view(x, "[^b2]")
[1] │ <a><.><1>
[2] │ <a>2
[3] │ b<-><4><_>
[4] │ <c>2<z>

Escaping Special Characters

Be careful if you try to match characters that are used as special expressions. You need to escape these characters with two backslashes \\ for them to be treated as a literal character, not a special expression. For example, to match a period “.”, we need to be escape it because . is an expression to match any single character.

x <- c("a.1", "a2", "b-4_")

# This will match every single character!
str_view(x, pattern = ".")
[1] │ <a><.><1>
[2] │ <a><2>
[3] │ <b><-><4><_>
# This will match only the period character
str_view(x, pattern = "\\.")
[1] │ a<.>1

Other special characters that would need to be escaped include . \ | ( ) [ { ^ $ * + ?

We won’t cover what all of these do, but to find literal matches for these characters, they need to be escaped, e.g., \\? to find a question mark in your text.

Capture groups

One really useful thing regex will let you do is find complex patterns, and then extract a subset of text within those matched patterns.

For example, say I want to handle a list of emails but only am interested in extracting the handles, not the domain name information from the email addresses.

x <- c("tyler@notadomain.com", "matt@gmeal.com", "erika@cornwall.edu")

I can use a capture group () to keep anything inside the parentheses

str_match(x, pattern = "^(.+)@(.+\\.\\w+)$")
     [,1]                   [,2]    [,3]            
[1,] "tyler@notadomain.com" "tyler" "notadomain.com"
[2,] "matt@gmeal.com"       "matt"  "gmeal.com"     
[3,] "erika@cornwall.edu"   "erika" "cornwall.edu"

We could name the output columns for easier use by starting each capture group with ?<my_group_label> followed by the regex for the match we are interested in. Here, we will label the extracted user handles and email domains.

str_match(x, pattern = "^(?<handle>.+)@(?<domain>.+\\.\\w+)$")
                            handle  domain          
[1,] "tyler@notadomain.com" "tyler" "notadomain.com"
[2,] "matt@gmeal.com"       "matt"  "gmeal.com"     
[3,] "erika@cornwall.edu"   "erika" "cornwall.edu"

Multiple matches

At least one match +

The + operator I used just now lets you look for multiple matches for any expression that immediately precedes the +.

If I want to find all of the instances of “an” in our fruits example, we can use the +

x <- c("apple", "banana", "pear")

# Without the `+` operator, we only match once
str_match(x, "an")
     [,1]
[1,] NA  
[2,] "an"
[3,] NA  
# With the `+` operator, we match multiple times in succession
x <- c("apple", "banana", "pear")
str_view(x, "(an)+")
[2] │ b<anan>a
# A subtle difference if we use `[]` instead of `()`
str_view(x, "[an]+")
[1] │ <a>pple
[2] │ b<anana>
[3] │ pe<a>r

No match or many matches *

The * operator I used just now lets you look for multiple matches in sequence or no matches at all.

If I want to find all of the instances of “an” in our fruits example, we can use the +

x <- c("apple", "banana", "pear")

# With the `*` operator, we match one or more times in succession or none at all
x <- c("apple", "banana", "pear")
str_view(x, "(an)*")
[1] │ <>a<>p<>p<>l<>e<>
[2] │ <>b<anan><>a<>
[3] │ <>p<>e<>a<>r<>
# A subtle difference if we use `[]` instead of `()`
str_view(x, "[an]*")
[1] │ <a><>p<>p<>l<>e<>
[2] │ <>b<anana><>
[3] │ <>p<>e<a><>r<>

Exact number of matches

{n} will match exactly n times

x <- c("apple", "banana", "pear")
str_view(x, "(an){1}") # Match "an"
[2] │ b<an><an>a
str_view(x, "(an){2}") # Match "anan"
[2] │ b<anan>a

{n,} will match n times or more

str_view(x, "[an]{1,}") # Match at least one "a" or "n"
[1] │ <a>pple
[2] │ b<anana>
[3] │ pe<a>r
str_view(x, "[an]{2,}") # Match at least two in a row of "a" or "n"
[2] │ b<anana>

You can also look for at most m matches: {,m} Or between n and m matches: {n,m}

Works in Data Frames, Too

This works with dataframes, so you can apply a regex operation to an entire column

dat <- tibble(word = words,
              index = seq_along(word))

dat |> filter(str_detect(word, pattern = "ing$"))
# A tibble: 9 × 2
  word    index
  <chr>   <int>
1 bring     113
2 during    251
3 evening   280
4 king      448
5 meaning   512
6 morning   533
7 ring      709
8 sing      765
9 thing     860
# Find any word with "d" or "dd"
dat |> filter(str_detect(word, pattern = "d+"))
# A tibble: 172 × 2
   word      index
   <chr>     <int>
 1 add          12
 2 address      13
 3 admit        14
 4 advertise    15
 5 afford       17
 6 already      31
 7 and          38
 8 around       52
 9 attend       60
10 bad          68
# ℹ 162 more rows

Places I Find Regex Useful

tidyr package

  • separate_wider_regex() lets you split up columns using regex
df <- tibble(var = c("race_1", "race_2", "age_bucket_1", "age_bucket_2"))
df
# A tibble: 4 × 1
  var         
  <chr>       
1 race_1      
2 race_2      
3 age_bucket_1
4 age_bucket_2
  1. Specify the pieces you want to split up into separate columns with a named regex
  2. Unnamed regex will match in between but does not get kept as a column!
# Here we split `var` into two columns: `name` and `number`
df %>% separate_wider_regex(var, c(var1 = ".*", "_", var2 = ".*"))
# A tibble: 4 × 2
  var1       var2 
  <chr>      <chr>
1 race       1    
2 race       2    
3 age_bucket 1    
4 age_bucket 2

This is hard to split up otherwise, but we used the * to match zero or more single characters up to the first underscore and then, crucially, zero or more characters after the first underscore. This last piece let us handle rows where there was a single underscore in some cases and two underscores in others.

Data Cleaning

Learn More