Why Chinese Word Segmentation Matters: Techniques, Challenges, and Python Demo

Joke

There is a joke as follows:

Before a blind date, the matchmaker told me the date was "honest and not talkative," but when I arrived I realized it actually meant "old person, few truthful words."

The same phrase "人老实话不多" can be segmented differently, leading to completely different meanings.

To understand why, we need to discuss the concept of word segmentation.

What Is Word Segmentation?

Word segmentation means breaking a continuous string of characters into independent lexical units. In English, spaces naturally separate words, but Chinese lacks such explicit delimiters, making segmentation a crucial step in Chinese natural language processing.

"南京市长江大桥" is a classic ambiguous example because it can be segmented in two ways:

南京市 长江大桥 – "Nanjing City" + "Yangtze River Bridge" (city name + bridge name).

南京 市长 江大桥 – "Nanjing" + "Mayor" + "Jiang Daqiao" (city name + job title + personal name).

Both segmentations are grammatically valid, but the first interpretation is far more common in real contexts. This example highlights the challenge of Chinese word segmentation, where the same character sequence can correspond to different word sequences, emphasizing the importance of contextual information and large corpora.

Because of this diversity, word segmentation is an interesting and challenging task in Chinese text processing.

Methods and Techniques for Chinese Word Segmentation

Dictionary‑based segmentation: relies on a large lexicon and uses maximum or minimum matching algorithms.

Statistical segmentation: learns word boundaries from massive corpora, e.g., using Hidden Markov Models (HMM) with states B (begin), M (middle), E (end), S (single) and the Viterbi algorithm, or Conditional Random Fields (CRF).

Deep‑learning segmentation: employs neural networks such as LSTM or BERT to perform end‑to‑end segmentation without intermediate state representations.

Python Implementation Example

Below is a simple dictionary‑based segmentation implementation:

<code>def simple_seg(text, dictionary):
    # initialize result list
    result = []
    # initialize index
    index = 0
    # get text length
    length = len(text)
    # longest word length in dictionary
    max_length = max(map(len, dictionary))
    while index < length:
        # try to match the longest word
        for size in range(max_length, 0, -1):
            word = text[index:index+size]
            if word in dictionary:
                break
        result.append(word)
        index += size
    return result

# dictionary
dictionary = ["人老", "实话", "人", "老实", "话不多"]
text = "人老实话不多"
print(simple_seg(text, dictionary))
</code>

We also introduce the jieba library, the most popular Chinese word segmentation tool in Python. It supports three modes: precise, full, and search engine, and allows custom dictionaries for higher accuracy.

Basic usage of jieba:

<code>import jieba

text = "人老实话不多"
seg_list = jieba.cut(text, cut_all=False)  # precise mode
print(" ".join(seg_list))
</code>

Running the code yields: "人 老实话 不 多".

Joke Continues

Reference:

[1] 光头刘长发. (2023, 04 14). 老实---网段改编. 知乎专栏. https://zhuanlan.zhihu.com/p/621935425