What's a Regular Expression?

I. Introduction

Today Farsight Security announced DNSDB 2.0 Flexible Search for DNSDB API. Flexible Search offers powerful new search capabilities that enhance DNSDB API, and which make it possible to easily do the DNSDB searches you’ve always wished you could make.

Early Adopter Access will be available on August 19th, 2020 and General Availability is scheduled for October 20th, 2020. If you’re interested in applying for Early Adopter access, please contact [email protected].

Flexible Search will be bundled at no charge for paid DNSDB API customers (and customers given access to DNSDB API under a grant from Farsight), but will NOT be included as part of DNSDB Community Edition, the free, entry-level version of our flagship solution.

II. Search Syntax Options

Flexible Search is a “finding aid” that supplements and enhances (but does not replace) Standard DNSDB API.

Flexible Search offers users three search syntax modes in DNSDB Scout, and two otherwise.

• Keyword Search: easily search for a brand name or domain name — just type in a word or string of characters to match.
• Keyword Search is meant to provide an easy starting point for novice searchers. [Available In DNSDB Scout Only]
• Regular Expressions: Regular expressions are the industry-standard way of expressing search patterns. Regular expressions support simple keyword searches, but also gives you the most power when you want or need to begin making more-complex pattern searches.
• This article is meant to give you the chance to learn a little about regular expressions in general now, before Flexible Search is actually made available for your use.
• Globbing: Globbing is the other pattern matching option that will be available in DNSDB 2.0. We offer globbing as an option for those who may prefer it, but please note that it is simultaneously:
  • More syntactically complex when it comes to doing basic keyword searches, and
  • Much more limited when it comes to supporting non-trivial pattern matching.
  If you are nonetheless interested in it, see our new blogpost, “What is Globbing?. Most users who aren’t familiar with either regular expressions or globbing should focus on learning to work with regular expressions, as described in this article, instead.

The goal of this article is to give you an introduction to regular expressions (“regexes”) for those who find themselves wondering “what the heck ARE these ‘regex’ things that some ‘techies’ keep talking about?”

At its most basic, a regular expression (or “regex”) is just a string that describes a pattern to be matched.

For example, imagine a program scanning lines in one or more files, looking for lines that contain the regular expression pattern of interest. When it finds a line with that pattern, it prints that line out. Simple as that sounds, regexes can be extremely powerful and useful. Regexes are routinely used in the cybersecurity world by:

• Analysts searching logs and other large data files
• Data scientists massaging input data files so they can be ingested into machine learning models
• Developers validating input fields and so on.

In a comparatively short article like this one, we can only “scratch the surface” when it comes to all the features and capabilities of regular expressions, but we hope that even this short introduction will still serve to pique your interest in regular expressions and motivate you to learn more about them. If that happens, there are a number of regular expression books you should check out, including O’Reilly’s:

• Michael Fitzgerald’s Introducing Regular Expressions
• Jeffrey E. F. Friedl’s Mastering Regular Expressions (3rd Edition)
• Goyvaerts and Levithan’s Regular Expressions Cookbook (2nd Edition)
• Tony Stubblebine’s Regular Expression Pocket Reference (2nd Edition)

III. Sample Data Set

To help illustrate how regular expressions work, we’ve created a small sample data file with fifty-two assorted domain names called

domains.txt

(see Appendix I). We’re going to use that file as data for our examples.

IV. The Tool We’ll Use To Demonstrate Regular Expression Matching: GNU egrep

While some of you may participate in our Early Adopter Program, most of you won’t have access to DNSDB Flexible Search until October 20th, 2020. Therefore, we’ve couched the following discussion in terms of the commonly available

GNU egrep

command. That command will treat regular expressions the same way that DNSDB 2.0 Flexible Search regular expressions will work, allowing eager people to get some time in learning and practicing before Flexible Search goes to General Availability status.

Once you get access to Flexible Search, using regular expressions will be as simple as plugging them into the Find field in DNSDB Scout, or using the

--regex

qualifier to the

dnsdbflex

command-line client.

The Unix “

grep

” command name is a “portmanteaux” word. It was built from parts of the words in the phrase “globally search for a regular expression and print.”

It is a staple command-line utility on Unix systems (and on Unix-like operating systems such as Linux and Mac OS X) and should exist (in some form or another) on virtually every Unix or Unix-like system.

egrep

is an enhanced version of

grep

. It’s what we’re going to use for the examples shown in this article.

On the latest version of Mac OSX (aka Catalina), the system-provided

egrep

appears to (still) be:

$ /usr/bin/egrep --version
egrep (BSD grep) 2.5.1-FreeBSD


The 2.5.1 version of

egrep

is known to have bugs, bugs which have been fixed in later versions. Unfortunately, because the later versions use a different open source license, Mac OSX has not updated to one of the later version(s) of

egrep

where that bug has been corrected. The bug is serious enough that it visibly impacts the results you may get for even relatively simple queries.

Thus, we normally prefer to use, and recommend that you use, the GNU version of

egrep

. If

GNU egrep

is installed on your system (and used by default), you should see something like:

$ egrep --version
grep (GNU grep) 3.3
​[etc]


If

GNU egrep

is not installed, you may be able to install

GNU egrep

via your operating system’s package manager. For example, on a Mac using homebrew you can install

GNU grep

by saying:

$ brew install grep


You can also download and install

GNU egrep

from source.

V. Regular Expression “Building Block” Characters

Regular expressions are just strings (sometimes quite cryptic-looking strings, but still, just strings). We’ll normally put regular expressions inside single quote marks. Each regex gets built using a combination of:

• “Literal” characters (e.g., uppercase and lowercase letters, numbers, and some symbols)
• “Meta” characters – there are many symbols that serve as a shorthand for special things such as “match any one character here.” The meta characters that normally do special things in at least some circumstances are as follows:

Character Name Special Thing This Symbol Means
\ backslash "Escapes" the character after this one
. dot Match any one character here
* star Repeat the previous zero or more times
^ caret Match start of line
$ dollar sign Match end of line
? question mark Optional (match zero or one time only)
+ plus sign Matches one or more time
| vertical bar Logical or (match either)
{ and } curly braces Repetition count {min}, {min, max}, or {,max}
( and ) parentheses Define logical subexpression ("create grouping")
[ left square bracket Define character class


If you want to literally match those metacharacters, prefix them with a backslash. [Note: Some versions of

egrep

may attempt to guess if a metacharacter “should” be treated as a metacharacter or as the literal character. That is risky, however, so we generally urge you to explicitly indicate if you want a metacharacter to be treated as a literal.]

• “Character classes:” These can be of two types: shorthand character classes, and bracketed character classes.

Shorthand character classes (such as

\w

,

\d

,

\s

) are used in someregular expression implementations, but will not be available in DNSDB’sFlexible Search regex implementation.

Bracketed character classes are either predefined character classes that look like the following (this is not an exhaustive list of these):

[[:alpha:]] Any upper or lower case alphabetic character
[[:digit:]] Any digit from 0 to 9
[[:alnum:]] Any alphanumeric character
[[:xdigit:]] Any hexadecimal digit (e.g., 0-9 plus A-F or a-f)


or classes that the user defines, such as

[aeiouy] Matches any vowel (or pseudo-vowel, in the case of "y")
[^aeiouy] Matches any NON-vowel (including other letters, numbers, symbols, etc.)


Note that MOST metacharacters lose their special meanings within square brackets (a notable exception is the caret symbol, as just shown in the [

^aeiou

] example).

VI. The Simplest of Regular Expression: Matching A Literal Substring

Let’s use a regex to find lines from our sample data file that contain the literal string “off”. We’ll run the egrep command from a Terminal window on our Mac:

$ egrep 'off' domains.txt
coffee.com
office.com
office365.com


This is a pretty straight-forward command: it takes a regular expression (in this case the literal string off, in single quote marks) and looks for matching lines in the specified file (

domains.txt

). Three “hits” are found:

coffee.com

,

office.com

and

office365.com

. Those get printed out when we run that command.

While in this case we just looked for a short three-character string, we could have looked for a single character, many characters, or even multiple words. (Just be sure to enclose the literal string to be matched in single quote marks if the string includes spaces!)

If we wanted to find lines that DON’T contain the string ‘off’, we can use the

egrep -v

option to find lines that DON’T match the specified pattern:

$ egrep -v 'off' domains.txt


all lines EXCEPT coffee.com, office.com and office365.com get output here

VII. Case (In)sensitivity

Regular expressions are case sensitive by default (so if we’d looked for ‘OFF’ instead of ‘off’, we wouldn’t have found any matches).

If we want

egrep

to do case Insensitive matches, we can add the

-i

option to our

egrep

command:

$ egrep -i 'OFF' domains.txt
coffee.com
office.com
office365.com


VIII. Figuring Out WHAT’S Matching

Another handy option to

egrep

is the

--color

option. It highlights the text that matches the regular expression we supplied:

$ egrep --color 'off' domains.txt
coffee.com
office.com
office365.com


We don’t urgently need this option to understand such a simple match, but when regexes get more complex – or we make a mistake constructing our regex – highlighting the text that matched a regex can really come in handy as a debugging tool.

IX. Matching EITHER of Two Literal Substrings

Let’s do another literal substring regex.

What if we want to find lines that have EITHER the literal substring ‘go’ OR the literal substring ‘off’?

GNU egrep

can help use do that with the vertical bar (or “pipe”) meta character.

$ egrep --color 'go|off' domains.txt
coffee.com
duckduckgo.com
eugene-or.gov
google.com
house.gov
office.com
office365.com
oregonstate.edu
senate.gov
supremecourt.gov
uoregon.edu
whitehouse.gov


Note that the vertical bar (“pipe”) characters is a metacharacter – it does NOT need to be physically part of the string text we’re matching.

If helpful or necessary, you can also use parentheses to set off the limits of an alternating match. For example:

$ egrep --color 'e(go|ug)' domains.txt
eugene-or.gov
oregonstate.edu
uoregon.edu


That pattern matches all records that have

ego

or

eug

in them.

X. The Dot

Up until this part of the article, we’ve been matching literal strings. That’s cool and useful, but the real power of regular expressions comes when we begin to work with wildcards – in this case literally the dot (“.”) character. Dot is a metacharacter that matches any single character.

$ egrep --color 'g.p' domains.txt
blogspot.com


If we have two dots in a row, that matches any two characters:

$ egrep --color 'r..e' domains.txt
lclark.edu
marines.mil
supremecourt.gov


and we could also search for any three characters in a row, any four characters in a row, etc.

Note that if we want to match an ACTUAL dot (and dots are obviously VERY common in domain names), we need to ask to match an escaped (“backslashed”) dot:

$ egrep --color '\.k12\.' domains.txt
bethel.k12.or.us
cal.k12.or.us
springfield.k12.or.us


If we didn’t remember to escape those “real dots,” specifying an unescaped dot might coincidentally match real dots, but they’d also match any OTHER single character in that spot, too.

XI. The Power of “dot star”

If you think dot was cool, wait until you learn about dot star (‘.*’) – it’s VERY cool!

• dot stands for “match any character”
• star stands for “repeat the previous match zero or more times”

If we had a regular expression that was simply ‘.*’ it would match all lines.

Therefore, most matches that contain ‘.*’ also include other specific patterns to match. For example, let’s find lines that have a

b

, then zero or more other characters, then a

c

:

$ egrep --color 'b.*c' domains.txt
bing.com
blogspot.com
crabcake.com
ebay.com
facebook.com
github.com
youtube.com


If we didn’t have the star metacharacter to give us flexibility here, we’d have to write a much “clunkier” regex with all possible patterns of zero or more dots in between the two letters of interest:

egrep '(bc|b.c|b..c|b...c|b....c|b.....c|b......c|b.......c|b........c)' domains.txt
same output as the previous example omitted here


Yuck! And just imagine how ugly that expression would get if one of the domain names in the file happened to be a long name with a

b

near the start and a

c

twenty or thirty characters later! Truly, the “magic of dot star” is a huge convenience when it comes to writing some regular expressions.

XII. A Note About “dot star” Matches: “Greed Is Good”

When

GNU egrep

finds matches, sometimes there are different options that might work. For example, if you asked to match

'^st.*o'

there are three ways it could match one line from our sample data:

stackoverflow.com OR stackoverflow.com OR stackoverflow.com

All three of those matches start with “st” and end with “o”, right? But which one of those will

GNU egrep

return by default?

The answer is that

GNU egrep

agrees with the fictional character Gordon Gekko, played by Kirk Douglas in the 1987 movie “Wall Street,” who became (in)famous for saying “Greed is good.”

By default, wildcards in

grep

will always try to match as much as possible while still satisfying the requested pattern. So in this example, it will match as shown in the last of the possible result,

stackoverflow.com

.

XIII. Matching A Single Character That’s Part of an Enumerated Set of Characters

We’ve seen how dot matches any SINGLE character, and dot star matches any ZERO OR MORE characters. But what if want to match a single character from just an enumerated set of characters? For example, what if we want to match:

• the character b
• zero or more other characters
• at least one vowel (a, e, i, o, u, or

y

It turns out that regular expressions can help us do this as well, using square brackets (as introduced in Section 4, above) to define a character set:

$ egrep --color 'b.*[aeiouy].*c' domains.txt
bing.com
blogspot.com
crabcake.com
ebay.com
facebook.com
youtube.com


If you’re referring to a contiguous range of characters, rather than a short, enumerated list of characters, you can take advantage of the dash character to avoid having to type a long list:

• [a-z] (lowercase letters)
• [a-zA-Z] (uppercase and lowercase letters)
• [a-zA-Z0-9] (uppercase and lowercase letters plus digits)

If you want to put a literal caret (^) in a list of characters, you can, just don’t put it first (if you do, it will be interpreted as meaning “take the complement of the characters that follow).

If you want to include a literal right square bracket (]) in a list of characters, you can, you just must use it as the FIRST character in the list of characters.

If you want to put a literal dash (-) in a list of characters in square brackets, put it LAST.  

XIV. Repetition Factors (“Counts”)

You can also use “repetition factors” or “counts” to ask for multiples of patterns. For example, if you wanted to find names from our sample file that had two successive vowels, you could write:

$ egrep --color '[aeiouy]{2}' domains.txt
coffee.com
ebay.com
eou.edu
eugene-or.gov
facebook.com
freedom.com
geoduck.com
google.com
house.gov
oit.edu
paypal.com
reed.edu
sou.edu
springfield.k12.or.us
supremecourt.gov
uoregon.edu
whitehouse.gov
wikipedia.org
wou.edu
yahoo.com
youtube.com


In addition to asking for exactly a specific value, you can also specify a repetition range, such as:

• {2,5} (meaning match between two and five times)
• {3,} (meaning match if present at least 3 times)
• ? (can appear zero or one time only – same as saying {0,1}
• * (can appear zero or more times – same as saying {0,}
• + (can appear one or more times – same as saying {1,}

For example:

$ egrep --color 'ube?\.com$' domains.txt
github.com
youtube.com


In this case, the “e” was optional, which is why youtube.com AND github.com successfully matched.

XV. Anchors

The patterns that we’ve been matching have all been “floating” patterns. Those patterns can potentially match suitable text seen anywhere in lines they’re scanning. But what if we only want to match a particular pattern at the start of a line, or at the end of a line? Those type of searches are called “anchored searches,” and we can use special metacharacters to limit our results:

^ (the caret symbol) "At the start of the line"
$ (the dollar sign) "At the end of the line"


For example, let’s find the domains in the file that are ‘

.edu

‘ domains:

$ egrep --color '\.edu$' domains.txt
4j.lane.edu
eou.edu
lanecc.edu
lclark.edu
oit.edu
oregonstate.edu
pdx.edu
reed.edu
sou.edu
uoregon.edu
willamette.edu


Important note: In DNSDB 2.0 Flexible Search, domain names in RRnames (and some Rdata) are written with a “formal ending dot.” Literal dots are also escaped with a backslash. That means that the domain name

wou.edu

would be written in regular expression format as:

wou\.edu\.$

If that’s the case for the stuff you’re matching against the anchored search would need to be written:

$ egrep --color '\.edu\.$' domains.txt


rather than just

$ egrep --color '\.edu$' domains.txt


Or as another example, let’s find the domains that begin with an

s

:

$ egrep --color '^s' domains.txt
senate.gov
sou.edu
springfield.k12.or.us
stackoverflow.com
supremecourt.gov


XVI. Specialty Versions of grep

You may also want to know that there are some “specialty” versions of grep, such as:

• agrep: “approximate GREP for fast fuzzy string searching.”
• cidrgrep: “A grep-like tool used to filter IP addresses against one or more CIDR network patterns.” (see also grepcidr)
• ripgrep: an extremely fast grep implementation that also supports searching non-UTF8 files, searching compressed files, and much more.

XVII. Conclusion

You’ve now had a bit of a whirlwind introduction to regular expressions. If you want to learn more, check out the books mentioned in the introduction, or consider trying one of the online interactive regular expression tutorials.

Regular expressions may feel a bit like they’re “brain teasers” or puzzles from The New York Times puzzle page, but if you tackle them with the right attitude, you may find they’re exceptionally powerful and sort of fun, too!

Acknowledgements

The author would like to acknowledge valuable reviews and commentary from colleagues, including (in alphabetical order) Chris Mikkelson, Jeremy Reed, Chuq Von Rospach, Stephen Watt, and Eric Ziegast. Any remaining issues or errors are solely the responsibility of the author. 

APPENDIX 1. DOMAINS USED IN THIS ARTICLE’S EXAMPLES

$ cat domains.txt
4j.lane.edu
af.mil
amazon.com
apple.com
army.mil
bethel.k12.or.us
bing.com
blogspot.com
cal.k12.or.us
coffee.com
crabcake.com
duckduckgo.com
ebay.com
eou.edu
eugene-or.gov
facebook.com
freedom.com
geoduck.com
github.com
google.com
house.gov
instagram.com
lanecc.edu
lclark.edu
linkedin.com
live.com
marines.mil
microsoft.com
msn.com
navy.mil
netflix.com
office.com
office365.com
oit.edu
oregonstate.edu
paypal.com
pdx.edu
reddit.com
reed.edu
senate.gov
sou.edu
springfield.k12.or.us
stackoverflow.com
supremecourt.gov
twitter.com
uoregon.edu
whitehouse.gov
wikipedia.org
willamette.edu
wou.edu
yahoo.com
youtube.com


ENDNOTE

More on validating input fields: An example of input validation might be a rule that says “the employee salary field can only contain numbers, commas, a decimal point and/or a dollar sign.” “$83,412.15” would pass that validation definition but “$7K/month” would not. More carefully-defined validation rules might be used to screen out typos/data entry errors such as “8$3,412.15” or “$83,,412.15” or “$83,412.155” Validation rules might also be used to identify likely-out-of-range-values such as “$8341215.

Joe St Sauver Ph.D. is a Distinguished Scientist and Director of Research with Farsight Security®, Inc..