You are here: Home ‣ Dive Into Python 3 ‣ 여기 있어요 :
Difficulty level: ♦♦♦♢♢
❝ 어떤 사람들은 어느 문제에 직면했을 때 이렇게 생각합니다. '흠, 이제 정규식을 써야할구먼'. 이제 그들의 문제는 두 개가 되버립니다. ❞
— Jamie Zawinski
대용량의 텍스트 덩어리에서 원하는 작은 부분을 찾아내는 일은 결코 쉽지 않습니다. 물론 파이썬의 string 클래스는 index(), find(), split(), count(), replace()와 같은 메쏘드를 통해 원하는 문자열을 검색할 수 있도록 지원하고 있지만, 그 기능이 아주 제한적입니다. 가령 index() 메쏘드는 전체 문자열 내에서 원하는 부분만을 찾을 수 있지만, 이게 또 대 소문자를 구분해 버리기 때문에 만약 대소문자 구분없이 검색하고 싶은 경우엔 lower() 나 upper() 와 같은 메소드를 함께 써줘야 하는 복잡한 코드가 되버리는거죠. 이런 귀찮은 상황은 replace() 나 split() 같은 함수를 사용할때도 피할 수 없고요.
만약 string 클래스의 메쏘드들만이라도 충분한 상황이라면, 그렇게 하시면 됩니다. 사용하기 간단하고 읽기도 쉽고 빠르거든요. 하지만 만약 여러분 코드가 이런 여러가지 string 클래스 메쏘드를 불러서 쓸 뿐 아니라 if 문으로 범벅이 된 채, 문자열을 쪼개고 (split()), 다시 합치고 ( join()) 등의 일을 반복한다면 어떨까요. 이때는 다른 방법에도 눈길을 한번 주시는 편이 나을겁니다.
정규식은 복잡한 문자 패턴을 이용해 문자열을 검색하고 치환하며 해석할수 있는 강력하고 표준화된 방법입니다. 정규식의 문법이 일반 프로그래밍 언어보다 엄격하기 때문에, 처음 보기에는 조금 이상하게 느껴지실 수도 있습니다. 하지만, 그 문법에 차차 익숙해지시다보면, 제한된 문자열 함수들과 if문을 이용해 만들어진 스파게티 코드보다는 훨씬 더 읽기 쉽습니다. 정규식 안에 내장 주석을 다는 다양한 방법도 있으니까, 이를 이용하여 깔끔하게 문서화 할수도 있고요 .
☞I 만약 Perl, JavaScript, PHP 같은 언어를 통해 미리 정규식을 사용해본 경험이 있는 분이라면, 파이썬의 정규식 또한 어렵지 않게 익히실 수 있을겁니다. 파이썬의
re모듈안을 살펴 보면 어떤 함수를 사용해야 하는지, 인자는 어떤식으로 구성해야 하는지도 알 수 있습니다.
⁂
여기서 사용된 예제는 제가 실제로 업무중 해결해야 했던 문제 가운데 발췌해 왔습니다. 몇 년 전 저는 오래된 시스템에서 사용자 주소를 추출하여 새 시스템으로 이전해야 했는데, 그때 겪었던 문제를 바탕으로 재구성했습니다.
>>> s = '100 NORTH MAIN ROAD'
>>> s.replace('ROAD', 'RD.') ①
'100 NORTH MAIN RD.'
>>> s = '100 NORTH BROAD ROAD'
>>> s.replace('ROAD', 'RD.') ②
'100 NORTH BRD. RD.'
>>> s[:-4] + s[-4:].replace('ROAD', 'RD.') ③
'100 NORTH BROAD RD.'
>>> import re ④
>>> re.sub('ROAD$', 'RD.', s) ⑤
'100 NORTH BROAD RD.'
'ROAD' 라는 문자열은 약어인 'RD.' 로 표기해야 했죠. 얼핏 보기에 그리 어려워보이지 않아서, string 클래스의 메쏘드인 replace()를 쓰면 되겠구나 라고 생각 했습니다. 게다가 모든 문자열이 대문자로 입력되있던 터라 대소문자 구분도 필요없었구요. 그래서 s.replace() 코드를 실행시켰더니 정말 제대로 동작하더라구요.
'BROAD' 같은 주소의 경우, 'ROAD' 라는 문자열이 두번이나 등장하게 되더라고요. 이런 경우 replace() 메쏘드가 두번 적용되 버리니까, 제 순진한 방법은 금방 통하지 않게 되버렸습니다.
'ROAD'가 두번 나타나는 경우를 위해 이렇게 한번 생각해 봤습니다. 문자열을 바꿀 때 주소 문자열 맨 뒤에 나오는 단어에서 마지막 4 글자에만 변환을 적용시키는 겁니다. s[-4:] 와 같은 코드를 작성하면 되지 않을까요? 흠. 그런데 다시 한번 생각해보니 이것도 썩 좋은 방법은 아닌것 같더라고요. 만약 'STREET' 를 'ST.'로 변경하는 경우엔 4 글자가 아니라 6 글자를 바꿔야 하는거잖아요? 나중에 버그가 생길게 뻔해보이는 아이디어 더군요.
re 모듈 안에 포함되있습니다.
'ROAD$')의 의미는 'ROAD' 라는 문자열을 찾는 정규표현식입니다. $ 표시와 함께 사용되어 문자열의 맨 뒤에 나오는 경우만을 찾도록 규정하고 있습니다. 카렛(caret) 이라고 부르는 ^ 기호는 문자열 맨 앞에 나오는 경우를 의미하고요. 예제코드는 re.sub() 함수를 이용하여, string s 에 정규표현식 'ROAD$' 를 적용하여 검색한 후, 반환되는 결과값을 'RD.'로 바꾸는 동작을 수행합니다. 사용된 정규 표현식은 string s 의 맨 뒤에 나오는 ROAD만을 타겟으로 하지, BROAD와 같이 string 중간에 나타나는 단어의 일부분인 경우에는 매치되지 않는다고 간주합니다.
주소 문자열을 단일화 하려는 저의 노력은 계속됩니다. 문자열 끝에서 'ROAD'를 찾아 변환시키는 접근법이 통하지 않는 경우를 발견했기 때문입니다. 희한하게도, 어떤 주소는 맨 뒤에 'ROAD' 를 붙이지 않은 채 도로명만 떡하니 나타나는 경우도 있더라고요. 한 술 더떠 그 도로명이 'BROAD' 인 경우도 있었는데, 그랬더니 아주 우스꽝스런 일이 생기더군요.
>>> s = '100 BROAD'
>>> re.sub('ROAD$', 'RD.', s)
'100 BRD.'
>>> re.sub('\\bROAD$', 'RD.', s) ①
'100 BROAD'
>>> re.sub(r'\bROAD$', 'RD.', s) ②
'100 BROAD'
>>> s = '100 BROAD ROAD APT. 3'
>>> re.sub(r'\bROAD$', 'RD.', s) ③
'100 BROAD ROAD APT. 3'
>>> re.sub(r'\bROAD\b', 'RD.', s) ④
'100 BROAD RD. APT 3'
'ROAD'라는 문자열을 찾아 바꾸는 것이었습니다. 그리고 이 문자열은 어느 다른 문자열의 일부분이면 안되는 거였죠. 정규표현식에서 이런 요구사항을 표현하려면 \b를 사용합니다. 이 기호의 의미는 "단어가 바로 여기서부터 시작되거나 끝나야 한다, 단어의 경계 (boundary)가 바로 여깁니다"" 라고 알려주는 역할을 합니다.
파이썬에서는 '\' 문자가 문자열안에 포함되면 escape 문자로 인식됩니다. 이걸 어려운 말로 backslash plague라고 하는데요. 이런 특징 때문에 파이썬에서 정규표현식 쓰기가 좀 까다롭긴 하죠.
펄에서는 이렇게 해주지 않아도 되기 때문에, 파이썬보다 편리합니다만, 때로는 펄의 이런 특징이 독이 되는 경우도 있습니다. 오류가 발생했을때 이 버그가 펄의 잘못된 문법으로 인해 생긴 것인지, 아니면 정규표현식을 잘못 작성해서 생긴 것인지 분간하기 어려울 때가 있으니까요. 세상 일 모두가 장단점이 있죠.
r 문자를 붙여주면 끝입니다. 이 문자를 붙여주면, 파이썬에게 "이 문자열은 아무것도 escape 처리하지 마셈" 이라고 알려주는 것과 같습니다. 일례로, '\t'은 탭 문자를 의미하지만, r'\t' 는 백 슬래쉬 문자 \에 t문자를 연속해서 사용한 것에 지나지 않습니다. 파이썬에서 정규표현식을 사용할 때는 반드시 raw string을 사용하시기를 권장합니다. 안그러면 코드 보기 정말 헷갈리거든요.
re.sub() 함수는 결과적으로 아무 일도 안하더라 이 말씀입니다.
$ 기호를 빼버리고 대신 \b 기호를 한번 더 사용해서 'ROAD' 라는 온전한 문자열 하나를 찾되, 그 문자열이 전체 문자열 어디에 있던지 상관없이 문자열 치환이 일어나도록 변경했습니다. 야호. 이제야 비로소 발뻗고 편히 쉴 수 있게 됬습니다.
⁂
You’ve most likely seen Roman numerals, even if you didn’t recognize them. You may have seen them in copyrights of old movies and television shows (“Copyright MCMXLVI” instead of “Copyright 1946”), or on the dedication walls of libraries or universities (“established MDCCCLXXXVIII” instead of “established 1888”). You may also have seen them in outlines and bibliographical references. It’s a system of representing numbers that really does date back to the ancient Roman empire (hence the name).
In Roman numerals, there are seven characters that are repeated and combined in various ways to represent numbers.
I = 1
V = 5
X = 10
L = 50
C = 100
D = 500
M = 1000
The following are some general rules for constructing Roman numerals:
I is 1, II is 2, and III is 3. VI is 6 (literally, “5 and 1”), VII is 7, and VIII is 8.
I, X, C, and M) can be repeated up to three times. At 4, you need to subtract from the next highest fives character. You can not represent 4 as IIII; instead, it is represented as IV (“1 less than 5”). 40 is written as XL (“10 less than 50”), 41 as XLI, 42 as XLII, 43 as XLIII, and then 44 as XLIV (“10 less than 50, then 1 less than 5”).
9, you need to subtract from the next highest tens character: 8 is VIII, but 9 is IX (“1 less than 10”), not VIIII (since the I character can not be repeated four times). 90 is XC, 900 is CM.
10 is always represented as X, never as VV. 100 is always C, never LL.
DC is 600; CD is a completely different number (400, “100 less than 500”). CI is 101; IC is not even a valid Roman numeral (because you can not subtract 1 directly from 100; you would need to write it as XCIX, “10 less than 100, then 1 less than 10”).
What would it take to validate that an arbitrary string is a valid Roman numeral? Let’s take it one digit at a time. Since Roman numerals are always written highest to lowest, let’s start with the highest: the thousands place. For numbers 1000 and higher, the thousands are represented by a series of M characters.
>>> import re >>> pattern = '^M?M?M?$' ① >>> re.search(pattern, 'M') ② <_sre.SRE_Match object at 0106FB58> >>> re.search(pattern, 'MM') ③ <_sre.SRE_Match object at 0106C290> >>> re.search(pattern, 'MMM') ④ <_sre.SRE_Match object at 0106AA38> >>> re.search(pattern, 'MMMM') ⑤ >>> re.search(pattern, '') ⑥ <_sre.SRE_Match object at 0106F4A8>
^ matches what follows only at the beginning of the string. If this were not specified, the pattern would match no matter where the M characters were, which is not what you want. You want to make sure that the M characters, if they’re there, are at the beginning of the string. M? optionally matches a single M character. Since this is repeated three times, you’re matching anywhere from zero to three M characters in a row. And $ matches the end of the string. When combined with the ^ character at the beginning, this means that the pattern must match the entire string, with no other characters before or after the M characters.
re module is the search() function, that takes a regular expression (pattern) and a string ('M') to try to match against the regular expression. If a match is found, search() returns an object which has various methods to describe the match; if no match is found, search() returns None, the Python null value. All you care about at the moment is whether the pattern matches, which you can tell by just looking at the return value of search(). 'M' matches this regular expression, because the first optional M matches and the second and third optional M characters are ignored.
'MM' matches because the first and second optional M characters match and the third M is ignored.
'MMM' matches because all three M characters match.
'MMMM' does not match. All three M characters match, but then the regular expression insists on the string ending (because of the $ character), and the string doesn’t end yet (because of the fourth M). So search() returns None.
M characters are optional.
The hundreds place is more difficult than the thousands, because there are several mutually exclusive ways it could be expressed, depending on its value.
100 = C
200 = CC
300 = CCC
400 = CD
500 = D
600 = DC
700 = DCC
800 = DCCC
900 = CM
So there are four possible patterns:
CM
CD
C characters (zero if the hundreds place is 0)
D, followed by zero to three C characters
The last two patterns can be combined:
D, followed by zero to three C characters
This example shows how to validate the hundreds place of a Roman numeral.
>>> import re >>> pattern = '^M?M?M?(CM|CD|D?C?C?C?)$' ① >>> re.search(pattern, 'MCM') ② <_sre.SRE_Match object at 01070390> >>> re.search(pattern, 'MD') ③ <_sre.SRE_Match object at 01073A50> >>> re.search(pattern, 'MMMCCC') ④ <_sre.SRE_Match object at 010748A8> >>> re.search(pattern, 'MCMC') ⑤ >>> re.search(pattern, '') ⑥ <_sre.SRE_Match object at 01071D98>
^), then the thousands place (M?M?M?). Then it has the new part, in parentheses, which defines a set of three mutually exclusive patterns, separated by vertical bars: CM, CD, and D?C?C?C? (which is an optional D followed by zero to three optional C characters). The regular expression parser checks for each of these patterns in order (from left to right), takes the first one that matches, and ignores the rest.
'MCM' matches because the first M matches, the second and third M characters are ignored, and the CM matches (so the CD and D?C?C?C? patterns are never even considered). MCM is the Roman numeral representation of 1900.
'MD' matches because the first M matches, the second and third M characters are ignored, and the D?C?C?C? pattern matches D (each of the three C characters are optional and are ignored). MD is the Roman numeral representation of 1500.
'MMMCCC' matches because all three M characters match, and the D?C?C?C? pattern matches CCC (the D is optional and is ignored). MMMCCC is the Roman numeral representation of 3300.
'MCMC' does not match. The first M matches, the second and third M characters are ignored, and the CM matches, but then the $ does not match because you’re not at the end of the string yet (you still have an unmatched C character). The C does not match as part of the D?C?C?C? pattern, because the mutually exclusive CM pattern has already matched.
M characters are optional and ignored, and the empty string matches the D?C?C?C? pattern where all the characters are optional and ignored.
Whew! See how quickly regular expressions can get nasty? And you’ve only covered the thousands and hundreds places of Roman numerals. But if you followed all that, the tens and ones places are easy, because they’re exactly the same pattern. But let’s look at another way to express the pattern.
⁂
{n,m} SyntaxIn the previous section, you were dealing with a pattern where the same character could be repeated up to three times. There is another way to express this in regular expressions, which some people find more readable. First look at the method we already used in the previous example.
>>> import re >>> pattern = '^M?M?M?$' >>> re.search(pattern, 'M') ① <_sre.SRE_Match object at 0x008EE090> >>> re.search(pattern, 'MM') ② <_sre.SRE_Match object at 0x008EEB48> >>> re.search(pattern, 'MMM') ③ <_sre.SRE_Match object at 0x008EE090> >>> re.search(pattern, 'MMMM') ④ >>>
M, but not the second and third M (but that’s okay because they’re optional), and then the end of the string.
M, but not the third M (but that’s okay because it’s optional), and then the end of the string.
M, and then the end of the string.
M, but then does not match the end of the string (because there is still one unmatched M), so the pattern does not match and returns None.
>>> pattern = '^M{0,3}$' ①
>>> re.search(pattern, 'M') ②
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'MM') ③
<_sre.SRE_Match object at 0x008EE090>
>>> re.search(pattern, 'MMM') ④
<_sre.SRE_Match object at 0x008EEDA8>
>>> re.search(pattern, 'MMMM') ⑤
>>>
M characters, then the end of the string.” The 0 and 3 can be any numbers; if you want to match at least one but no more than three M characters, you could say M{1,3}.
M out of a possible three, then the end of the string.
M out of a possible three, then the end of the string.
M out of a possible three, then the end of the string.
M out of a possible three, but then does not match the end of the string. The regular expression allows for up to only three M characters before the end of the string, but you have four, so the pattern does not match and returns None.
Now let’s expand the Roman numeral regular expression to cover the tens and ones place. This example shows the check for tens.
>>> pattern = '^M?M?M?(CM|CD|D?C?C?C?)(XC|XL|L?X?X?X?)$' >>> re.search(pattern, 'MCMXL') ① <_sre.SRE_Match object at 0x008EEB48> >>> re.search(pattern, 'MCML') ② <_sre.SRE_Match object at 0x008EEB48> >>> re.search(pattern, 'MCMLX') ③ <_sre.SRE_Match object at 0x008EEB48> >>> re.search(pattern, 'MCMLXXX') ④ <_sre.SRE_Match object at 0x008EEB48> >>> re.search(pattern, 'MCMLXXXX') ⑤ >>>
M, then CM, then XL, then the end of the string. Remember, the (A|B|C) syntax means “match exactly one of A, B, or C”. You match XL, so you ignore the XC and L?X?X?X? choices, and then move on to the end of the string. MCMXL is the Roman numeral representation of 1940.
M, then CM, then L?X?X?X?. Of the L?X?X?X?, it matches the L and skips all three optional X characters. Then you move to the end of the string. MCML is the Roman numeral representation of 1950.
M, then CM, then the optional L and the first optional X, skips the second and third optional X, then the end of the string. MCMLX is the Roman numeral representation of 1960.
M, then CM, then the optional L and all three optional X characters, then the end of the string. MCMLXXX is the Roman numeral representation of 1980.
M, then CM, then the optional L and all three optional X characters, then fails to match the end of the string because there is still one more X unaccounted for. So the entire pattern fails to match, and returns None. MCMLXXXX is not a valid Roman numeral.
The expression for the ones place follows the same pattern. I’ll spare you the details and show you the end result.
>>> pattern = '^M?M?M?(CM|CD|D?C?C?C?)(XC|XL|L?X?X?X?)(IX|IV|V?I?I?I?)$'
So what does that look like using this alternate {n,m} syntax? This example shows the new syntax.
>>> pattern = '^M{0,3}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$'
>>> re.search(pattern, 'MDLV') ①
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'MMDCLXVI') ②
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'MMMDCCCLXXXVIII') ③
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'I') ④
<_sre.SRE_Match object at 0x008EEB48>
M characters, then D?C{0,3}. Of that, it matches the optional D and zero of three possible C characters. Moving on, it matches L?X{0,3} by matching the optional L and zero of three possible X characters. Then it matches V?I{0,3} by matching the optional V and zero of three possible I characters, and finally the end of the string. MDLV is the Roman numeral representation of 1555.
M characters, then the D?C{0,3} with a D and one of three possible C characters; then L?X{0,3} with an L and one of three possible X characters; then V?I{0,3} with a V and one of three possible I characters; then the end of the string. MMDCLXVI is the Roman numeral representation of 2666.
M characters, then D?C{0,3} with a D and three out of three C characters; then L?X{0,3} with an L and three out of three X characters; then V?I{0,3} with a V and three out of three I characters; then the end of the string. MMMDCCCLXXXVIII is the Roman numeral representation of 3888, and it’s the longest Roman numeral you can write without extended syntax.
M, then matches D?C{0,3} by skipping the optional D and matching zero out of three C, then matches L?X{0,3} by skipping the optional L and matching zero out of three X, then matches V?I{0,3} by skipping the optional V and matching one out of three I. Then the end of the string. Whoa.
If you followed all that and understood it on the first try, you’re doing better than I did. Now imagine trying to understand someone else’s regular expressions, in the middle of a critical function of a large program. Or even imagine coming back to your own regular expressions a few months later. I’ve done it, and it’s not a pretty sight.
Now let’s explore an alternate syntax that can help keep your expressions maintainable.
⁂
So far you’ve just been dealing with what I’ll call “compact” regular expressions. As you’ve seen, they are difficult to read, and even if you figure out what one does, that’s no guarantee that you’ll be able to understand it six months later. What you really need is inline documentation.
Python allows you to do this with something called verbose regular expressions. A verbose regular expression is different from a compact regular expression in two ways:
# character and goes until the end of the line. In this case it’s a comment within a multi-line string instead of within your source code, but it works the same way.
This will be more clear with an example. Let’s revisit the compact regular expression you’ve been working with, and make it a verbose regular expression. This example shows how.
>>> pattern = '''
^ # beginning of string
M{0,3} # thousands - 0 to 3 Ms
(CM|CD|D?C{0,3}) # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 Cs),
# or 500-800 (D, followed by 0 to 3 Cs)
(XC|XL|L?X{0,3}) # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 Xs),
# or 50-80 (L, followed by 0 to 3 Xs)
(IX|IV|V?I{0,3}) # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 Is),
# or 5-8 (V, followed by 0 to 3 Is)
$ # end of string
'''
>>> re.search(pattern, 'M', re.VERBOSE) ①
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'MCMLXXXIX', re.VERBOSE) ②
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'MMMDCCCLXXXVIII', re.VERBOSE) ③
<_sre.SRE_Match object at 0x008EEB48>
>>> re.search(pattern, 'M') ④
re.VERBOSE is a constant defined in the re module that signals that the pattern should be treated as a verbose regular expression. As you can see, this pattern has quite a bit of whitespace (all of which is ignored), and several comments (all of which are ignored). Once you ignore the whitespace and the comments, this is exactly the same regular expression as you saw in the previous section, but it’s a lot more readable.
M, then CM, then L and three of a possible three X, then IX, then the end of the string.
M, then D and three of a possible three C, then L and three of a possible three X, then V and three of a possible three I, then the end of the string.
re.VERBOSE flag, so the re.search function is treating the pattern as a compact regular expression, with significant whitespace and literal hash marks. Python can’t auto-detect whether a regular expression is verbose or not. Python assumes every regular expression is compact unless you explicitly state that it is verbose.
⁂
So far you’ve concentrated on matching whole patterns. Either the pattern matches, or it doesn’t. But regular expressions are much more powerful than that. When a regular expression does match, you can pick out specific pieces of it. You can find out what matched where.
This example came from another real-world problem I encountered, again from a previous day job. The problem: parsing an American phone number. The client wanted to be able to enter the number free-form (in a single field), but then wanted to store the area code, trunk, number, and optionally an extension separately in the company’s database. I scoured the Web and found many examples of regular expressions that purported to do this, but none of them were permissive enough.
Here are the phone numbers I needed to be able to accept:
800-555-1212
800 555 1212
800.555.1212
(800) 555-1212
1-800-555-1212
800-555-1212-1234
800-555-1212x1234
800-555-1212 ext. 1234
work 1-(800) 555.1212 #1234
Quite a variety! In each of these cases, I need to know that the area code was 800, the trunk was 555, and the rest of the phone number was 1212. For those with an extension, I need to know that the extension was 1234.
Let’s work through developing a solution for phone number parsing. This example shows the first step.
>>> phonePattern = re.compile(r'^(\d{3})-(\d{3})-(\d{4})$') ①
>>> phonePattern.search('800-555-1212').groups() ②
('800', '555', '1212')
>>> phonePattern.search('800-555-1212-1234') ③
>>> phonePattern.search('800-555-1212-1234').groups() ④
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'groups'
(\d{3}). What’s \d{3}? Well, \d means “any numeric digit” (0 through 9). The {3} means “match exactly three numeric digits”; it’s a variation on the {n,m} syntax you saw earlier. Putting it all in parentheses means “match exactly three numeric digits, and then remember them as a group that I can ask for later”. Then match a literal hyphen. Then match another group of exactly three digits. Then another literal hyphen. Then another group of exactly four digits. Then match the end of the string.
groups() method on the object that the search() method returns. It will return a tuple of however many groups were defined in the regular expression. In this case, you defined three groups, one with three digits, one with three digits, and one with four digits.
search() and groups() methods in production code. If the search() method returns no matches, it returns None, not a regular expression match object. Calling None.groups() raises a perfectly obvious exception: None doesn’t have a groups() method. (Of course, it’s slightly less obvious when you get this exception from deep within your code. Yes, I speak from experience here.)
>>> phonePattern = re.compile(r'^(\d{3})-(\d{3})-(\d{4})-(\d+)$') ①
>>> phonePattern.search('800-555-1212-1234').groups() ②
('800', '555', '1212', '1234')
>>> phonePattern.search('800 555 1212 1234') ③
>>>
>>> phonePattern.search('800-555-1212') ④
>>>
groups() method now returns a tuple of four elements, since the regular expression now defines four groups to remember.
The next example shows the regular expression to handle separators between the different parts of the phone number.
>>> phonePattern = re.compile(r'^(\d{3})\D+(\d{3})\D+(\d{4})\D+(\d+)$') ①
>>> phonePattern.search('800 555 1212 1234').groups() ②
('800', '555', '1212', '1234')
>>> phonePattern.search('800-555-1212-1234').groups() ③
('800', '555', '1212', '1234')
>>> phonePattern.search('80055512121234') ④
>>>
>>> phonePattern.search('800-555-1212') ⑤
>>>
\D+. What the heck is that? Well, \D matches any character except a numeric digit, and + means “1 or more”. So \D+ matches one or more characters that are not digits. This is what you’re using instead of a literal hyphen, to try to match different separators.
\D+ instead of - means you can now match phone numbers where the parts are separated by spaces instead of hyphens.
The next example shows the regular expression for handling phone numbers without separators.
>>> phonePattern = re.compile(r'^(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$') ①
>>> phonePattern.search('80055512121234').groups() ②
('800', '555', '1212', '1234')
>>> phonePattern.search('800.555.1212 x1234').groups() ③
('800', '555', '1212', '1234')
>>> phonePattern.search('800-555-1212').groups() ④
('800', '555', '1212', '')
>>> phonePattern.search('(800)5551212 x1234') ⑤
>>>
+ to *. Instead of \D+ between the parts of the phone number, you now match on \D*. Remember that + means “1 or more”? Well, * means “zero or more”. So now you should be able to parse phone numbers even when there is no separator character at all.
800), then zero non-numeric characters, then a remembered group of three digits (555), then zero non-numeric characters, then a remembered group of four digits (1212), then zero non-numeric characters, then a remembered group of an arbitrary number of digits (1234), then the end of the string.
x before the extension.
groups() method still returns a tuple of four elements, but the fourth element is just an empty string.
The next example shows how to handle leading characters in phone numbers.
>>> phonePattern = re.compile(r'^\D*(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$') ①
>>> phonePattern.search('(800)5551212 ext. 1234').groups() ②
('800', '555', '1212', '1234')
>>> phonePattern.search('800-555-1212').groups() ③
('800', '555', '1212', '')
>>> phonePattern.search('work 1-(800) 555.1212 #1234') ④
>>>
\D*, zero or more non-numeric characters, before the first remembered group (the area code). Notice that you’re not remembering these non-numeric characters (they’re not in parentheses). If you find them, you’ll just skip over them and then start remembering the area code whenever you get to it.
\D* after the first remembered group.)
800), then one non-numeric character (the hyphen), then a remembered group of three digits (555), then one non-numeric character (the hyphen), then a remembered group of four digits (1212), then zero non-numeric characters, then a remembered group of zero digits, then the end of the string.
1 before the area code, but you assumed that all the leading characters before the area code were non-numeric characters (\D*). Aargh.
Let’s back up for a second. So far the regular expressions have all matched from the beginning of the string. But now you see that there may be an indeterminate amount of stuff at the beginning of the string that you want to ignore. Rather than trying to match it all just so you can skip over it, let’s take a different approach: don’t explicitly match the beginning of the string at all. This approach is shown in the next example.
>>> phonePattern = re.compile(r'(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$') ①
>>> phonePattern.search('work 1-(800) 555.1212 #1234').groups() ②
('800', '555', '1212', '1234')
>>> phonePattern.search('800-555-1212').groups() ③
('800', '555', '1212', '')
>>> phonePattern.search('80055512121234').groups() ④
('800', '555', '1212', '1234')
^ in this regular expression. You are not matching the beginning of the string anymore. There’s nothing that says you need to match the entire input with your regular expression. The regular expression engine will do the hard work of figuring out where the input string starts to match, and go from there.
See how quickly a regular expression can get out of control? Take a quick glance at any of the previous iterations. Can you tell the difference between one and the next?
While you still understand the final answer (and it is the final answer; if you’ve discovered a case it doesn’t handle, I don’t want to know about it), let’s write it out as a verbose regular expression, before you forget why you made the choices you made.
>>> phonePattern = re.compile(r'''
# don't match beginning of string, number can start anywhere
(\d{3}) # area code is 3 digits (e.g. '800')
\D* # optional separator is any number of non-digits
(\d{3}) # trunk is 3 digits (e.g. '555')
\D* # optional separator
(\d{4}) # rest of number is 4 digits (e.g. '1212')
\D* # optional separator
(\d*) # extension is optional and can be any number of digits
$ # end of string
''', re.VERBOSE)
>>> phonePattern.search('work 1-(800) 555.1212 #1234').groups() ①
('800', '555', '1212', '1234')
>>> phonePattern.search('800-555-1212') ②
('800', '555', '1212', '')
⁂
This is just the tiniest tip of the iceberg of what regular expressions can do. In other words, even though you’re completely overwhelmed by them now, believe me, you ain’t seen nothing yet.
You should now be familiar with the following techniques:
^ matches the beginning of a string.
$ matches the end of a string.
\b matches a word boundary.
\d matches any numeric digit.
\D matches any non-numeric character.
x? matches an optional x character (in other words, it matches an x zero or one times).
x* matches x zero or more times.
x+ matches x one or more times.
x{n,m} matches an x character at least n times, but not more than m times.
(a|b|c) matches exactly one of a, b or c.
(x) in general is a remembered group. You can get the value of what matched by using the groups() method of the object returned by re.search.
Regular expressions are extremely powerful, but they are not the correct solution for every problem. You should learn enough about them to know when they are appropriate, when they will solve your problems, and when they will cause more problems than they solve.
© 2001–11 Mark Pilgrim