Python 2 Unicode Tutorial

By Xah Lee. Date: . Last updated: .

Python 2 source code encoding

Python 2, string and unicode

If your string contain non-ASCII Characters , such as ♥ (U+2665: BLACK HEART SUIT) , then you must prefix your string with u , e.g. u"I ♥ Python".

The u-prefix makes the string a Unicode datatype. Without the u-prefix, string is just byte sequence.

The r and u can be combined, like this: ur"I ♥ Python"

# -*- coding: utf-8 -*-
# python 2

import sys
print( sys.version)

# unicode string starts with u
aa = u"I ♥ U"

print aa.encode('utf-8')
# I ♥ U

Sometimes when you print Unicode strings, you may get a error like this:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in position 16: ordinal not in range(128).

The solution is to use the .encode() or .decode() method.

# -*- coding: utf-8 -*-
# python 2

myStr = u'α'

# Bad. This is a error.
print 'Greek alpha: ', myStr

# Good
print 'Greek alpha: ', myStr.encode('utf-8')

Python 2, unicode in regex

When using regex on Unicode string, and you want the word patterns {\w, \W} and boundary patterns {\b, \B}, dependent on the Unicode character properties, you need to add the Unicode flag re.U when calling regex functions.

# -*- coding: utf-8 -*-
# python 2

# example showing the difference of using re.U regex flag

import re

rr = re.findall(r"\w+", u"♥αβγ!", re.U)

if rr:
    print rr
else:
    print "no match"

# prints [u'\u03b1\u03b2\u03b3']

# if re.U is not used, it prints “no match” because the \w+ pattern for “word” only consider ASCII letters

See: Python: Regex Flags .

Find replace unicode char in string

# -*- coding: utf-8 -*-
# python 2

# example of finding all unicode char in a string

import re

ss = u"i♥NY 😸"

# find all unicode chars
myResult = re.findall(u"[^\u0000-\u007e]+", ss)

if myResult:
    print myResult  # [u'\u2665', u'\U0001f638']
else:
    print "no match"