4.9. Regular expressions 101

Regular expressions are a powerful (and fairly standardized) way of searching, replacing, and parsing text with complex patterns of characters. If you’ve used regular expressions in other languages (like Perl), you should skip this section and just read the summary of the re module to get an overview of the available functions and their arguments.

Strings have methods for searching (index, find, and count), replacing (replace), and parsing (split), but they are limited to the simplest of cases. The search methods look for a single, hard-coded substring, and they are always case-sensitive; to do case-insensitive searches of a string s, you must call s.lower() or s.upper() and make sure your search strings are the appropriate case to match. The replace and split methods have the same limitations. You should use them if you can (they’re fast and easy to read), but for anything more complex, you’ll have to move up to regular expressions.

Example 4.19. Matching at the end of a string

This series of examples was inspired by a real-life problem I had in my day job, scrubbing and standardizing street addresses exported from a legacy system before importing them into a newer system. (See, I don’t just make this stuff up; it’s actually useful.)

>>> s = '100 NORTH MAIN ROAD'
>>> s.replace('ROAD', 'RD.')               1
>>> s = '100 NORTH BROAD ROAD'
>>> s.replace('ROAD', 'RD.')               2
'100 NORTH BRD. RD.'
>>> s[:-4] + s[-4:].replace('ROAD', 'RD.') 3
>>> import re                              4
>>> re.sub('ROAD$', 'RD.', s)              5 6
1 My goal is to standardize a street address so that 'ROAD' is always abbreviated as 'RD.'. At first glance, I thought this was simple enough that I could just use the string method replace. After all, all the data was already uppercase, so case mismatches would not be a problem. And the search string, 'ROAD', was a constant. And in this deceptively simple example, s.replace does indeed work.
2 Life, unfortunately, is full of counterexamples, and I quickly discovered this one. The problem here is that 'ROAD' appears twice in the address, once as part of the street name 'BROAD' and once as its own word. The replace method sees these two occurrences and blindly replaces both of them; meanwhile, I see my addresses getting destroyed.
3 To solve the problem of addresses with more than one 'ROAD' substring, we could resort to something like this: only search and replace 'ROAD' in the last 4 characters of the address (s[-4:]), and leave the string along (s[:-4]). But you can see that this is already getting unweildy. For example, the pattern is dependent on the length of the string we’re replacing (if we were replacing 'STREET' with 'ST.', we would need to use s[:-6] and s[-6:].replace(...)). Would you like to come back in six months and debug this? I know I wouldn’t.
4 It’s time to move up to regular expressions. In Python, all functionality related to regular expressions is contained in the re module.
5 Take a look at the first parameter: 'ROAD$'. This is a very simple regular expression which matches 'ROAD' only when it occurs at the end of a string. The $ means “end of the string”. (There is a corresponding character, the caret ^, which means “beginning of the string”.)
6 Using the re.sub function, we search the string s for the regular expression 'ROAD$' and replace it with 'RD.'. This matches the ROAD at the end of the string s, but does not match the ROAD that’s part of the word BROAD, because that’s in the middle of s.

Example 4.20. Matching whole words

>>> s = '100 BROAD'
>>> re.sub('ROAD$', 'RD.', s)     1
'100 BRD.'
>>> re.sub('\\bROAD$', 'RD.', s)  2
'100 BROAD'
>>> re.sub(r'\bROAD$', 'RD.', s)  3
'100 BROAD'
>>> s = '100 BROAD ROAD APT. 3'
>>> re.sub(r'\bROAD$', 'RD.', s)  4
>>> re.sub(r'\bROAD\b', 'RD.', s) 5
'100 BROAD RD. APT 3'
1 Continuing with my story of scrubbing addresses, I soon discovered that the previous example, matching 'ROAD' at the end of the address, was not good enough, because not all addresses included a street designation at all; some just ended with the street name. Most of the time, I got away with it, but if the street name was 'BROAD', then the regular expression would match 'ROAD' at the end of the string as part of the word 'BROAD', which is not what I wanted.
2 What I really wanted was to match 'ROAD' when it was at the end of the string and it was its own whole word, not a part of some larger word. To express this in a regular expression, you use \b, which means “a word boundary must occur right here”. In Python, this is complicated by the fact that the '\' character in a string must itself be escaped. (This is sometimes referred to as the backslash plague, and it is one reason why regular expressions are easier in Perl than in Python. On the down side, Perl mixes regular expressions with other syntax, so if you have a bug, it may be hard to tell whether it’s a bug in syntax or a bug in your regular expression.)
3 To work around the backslash plague, you can use what is called a raw string, by prefixing the '...' with the letter r. This tells Python that nothing in this string should be escaped; '\t' is a tab character, but r'\t' is really the backslash character \ followed by the letter t. I recommend always using raw strings when dealing with regular expressions, otherwise things get too confusing too quickly (and regular expressions get confusing quickly enough all by themselves).
4 *sigh* Unfortunately, I soon found more cases that contradicted my logic. In this case, the street address contained the word 'ROAD' as a whole word by itself, but it wasn’t at the end, because the address had an apartment number after the street designation. Because 'ROAD' isn’t at the very end of the string, it doesn’t match, so the entire call to re.sub ends up replacing nothing at all, and we get the original string back, which is not what we want.
5 To solve this problem, I removed the $ character and added another \b. Now the regular expression reads “match 'ROAD' when it’s a whole word by itself anywhere in the string”, whether at the end, the beginning, or somewhere in the middle.

This is just the tiniest tip of the iceberg of what regular expressions can do. They are extremely powerful, and there are entire books devoted to them. They are not the correct solution for every problem. You should learn enough about them to know when they are appropriate, and when they will simply cause more problems than they solve.


Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

--Jamie Zawinski, in comp.lang.emacs  

Further reading