Normalization¶

The normalization of Urdu text is necessary to make it useful for the machine learning tasks. In the normalize module, the very basic problems faced when working with Urdu data are handled with ease and efficiency. All the problems and how normalize module handles them are listed below.

This modules fixes the problem of correct encodings for the Urdu characters as well as replace Arabic characters with correct Urdu characters. This module brings all the characters in the specified unicode range (0600-06FF) for Urdu language.

It also fixes the problem of joining of different Urdu words. By joining we mean that when space between two Urdu words is removed, they must not make a new word. Their rendering must not change and even after the removal of space they should look the same.

You can use the library to normalize the Urdu text for correct unicode characters. By normalization we mean to end the confusion between Urdu and Arabic characters, to replace two words with one word keeping in mind the context they are used in. Like the character ‘ﺁ’ and ‘ﺂ’ are to be replaced by ‘آ’. All this is done using regular expressions.

The normalization of Urdu text is necessary to make it useful for the machine learning tasks. This module provides the following functionality:

Normalizing Single Characters

Normalizing Combine Characters

Removal of Diacritics from Urdu Text

Replace all digits with Urdu and vice versa English

urduhack.normalization.normalize(text: str) → str[source]¶

To normalize some text, all you need to do pass Urdu text. It will return a str with normalized characters both single and combined, proper spaces after digits and punctuations and diacritics removed.

Parameters:	text (str) – `Urdu` text
Returns:	Normalized `Urdu` text
Return type:	str
Raises:	`TypeError` – If text param is not not str Type.

Examples

>>> from urduhack import normalize
>>> _text = "اَباُوگل پاکستان ﻤﯿﮟ 20 سال ﺳﮯ ، وسائل کی کوئی کمی نہیں ﮨﮯ۔"
>>> normalized_text = normalize(_text)
>>> # The text now contains proper spaces after digits and punctuations,
>>> # normalized characters and no diacritics!
>>> normalized_text
اباوگل پاکستان ﻤﯿﮟ 20 سال ﺳﮯ ، وسائل کی کوئی کمی نہیں ﮨﮯ۔

urduhack.normalization.normalize_characters(text: str) → str[source]¶

The most important module in the UrduHack is the character module, defined in the module with the same name. You can use this module separately to normalize a piece of text to a proper specified Urdu range (0600-06FF). To get an understanding of how this module works, one needs to understand unicode. Every character has a unicode. You can search for any character unicode from any language you will find it. No two characters can have the same unicode. This module works with reference to the unicodes. Now as urdu language has its roots in Arabic, Parsian and Turkish. So we have to deal with all those characters and convert them to a normal urdu character. To get a bit more of what the above explanation means is.:

>>> all_fes = ['ﻑ', 'ﻒ', 'ﻓ', 'ﻔ', ]
>>> urdu_fe = 'ف'

All the characters in all_fes are same but they come from different languages and they all have different unicodes. Now as computers deal with numbers, same character appearing in more than one place in a different language will have a different unicode and that will create confusion which will create problems in understanding the context of the data. character module will eliminate this problem by replacing all the characters in all_fes by urdu_fe.

This provides the functionality to replace wrong arabic characters with correct urdu characters and fixed the combine|join characters issue.

Replace urdu text characters with correct unicode characters.

Parameters:	text (str) – `Urdu` text
Returns:	Returns a `str` object containing normalized text.
Return type:	str

Examples

>>> from urduhack.normalization import normalize_characters
>>> # Text containing characters from Arabic Unicode block
>>> _text = "مجھ کو جو توڑا ﮔیا تھا"
>>> normalized_text = normalize_characters(_text)
>>> # Normalized text - Arabic characters are now replaced with Urdu characters
>>> normalized_text
مجھ کو جو توڑا گیا تھا

urduhack.normalization.normalize_combine_characters(text: str) → str[source]¶

To normalize combine characters with single character unicode text, use the normalize_combine_characters() function in the character module.

Replace combine|join urdu characters with single unicode character

Parameters:	text (str) – `Urdu` text
Returns:	Returns a `str` object containing normalized text.
Return type:	str

Examples

>>> from urduhack.normalization import normalize_combine_characters
>>> # In the following string, Alif ('ا') and Hamza ('ٔ ') are separate characters
>>> _text = "جرأت"
>>> normalized_text = normalize_combine_characters(_text)
>>> # Now Alif and Hamza are replaced by a Single Urdu Unicode Character!
>>> normalized_text
جرأت

urduhack.normalization.remove_diacritics(text: str) → str[source]¶

Remove urdu diacritics from text. It is an important step in pre-processing of the Urdu data. This function returns a String object which contains the original text minus Urdu diacritics.

Parameters:	text (str) – `Urdu` text
Returns:	Returns a `str` object containing normalized text.
Return type:	str

Examples

>>> from urduhack.normalization import remove_diacritics
>>> _text = "شیرِ پنجاب"
>>> normalized_text = remove_diacritics(_text)
>>> normalized_text
شیر پنجاب

urduhack.normalization.replace_digits(text: str, with_english: bool = True) → str[source]¶

Replace urdu digits with English digits and vice versa

Parameters:	text (str) – Urdu text string with_english (bool) – Boolean to convert digits from one language to other
Returns:	Text string with replaced digits