Unicode normalization is a process that aligns the representation of strings treated as the same character into a consistent form. The Korean character 가 can also be expressed as a single complete character or as a combination of initial and medial consonants. Although they may look similar on screen, the number of positions they occupy within a Python string can differ.
This difference becomes an issue when connecting detection and replacement in personal information processing. If the start and end positions found in a normalized string are applied directly to the original pre-normalization text, parts of the accurately detected identifier may remain. The first thing to check is whether the two stages use the same string, rather than just the detection threshold.
Presidio Korean Regression Test explained entity types and correct spans, and 4-Stage Structure of Personal Information Processing described policies and processes after detection. This article validates the contract that delivers strings and positions together between these stages using a synthetic example. Actual personal information or business logs were not used.
This is a conceptual image generated by AI. It does not represent actual screens or measurement results.
1. There’s No Guarantee That One Character 가 Occupies One Position
A code point is a number that identifies a character in Unicode. The first part of the two strings below becomes the same 가 when normalized.
composed = "uac00" # Ga: U+AC00
decomposed = "u1100u1161" # G + A: U+1100, U+1161The first has one code point, the second has two. The decomposed Jamo here are not arbitrary examples like ㄱ·ㅏ (compatibility Jamo), but ᄀ·ᅡ used for Hangul composition in Unicode. Hangul Examples in Unicode Normalization Standard
NFC is a form that combines possible combinations after canonical decomposition, and NFD is a canonically decomposed form. In this example, it becomes 가 with NFC and 가 with NFD. Both conversions can be executed with Python’s unicodedata.normalize(). Python unicodedata
import unicodedata
original = "u1100u1161 DEMO-AB12!"
normalized = unicodedata.normalize("NFC", original)
print(len(original), len(normalized))The actual output was 13 12. DEMO-AB12 is a practice identifier created for this article and does not replicate actual account or identifier formats. The reason for the overall length decreasing by one is the Hangul composition before the identifier.
len() counts the number of code points in this Python example. It should not be assumed to be the same unit as the number of visible character clusters on screen or UTF-8 bytes. The issue of external clients using UTF-16 positions requires additional verification, but this test is limited to the NFC·NFD difference within Python strings.
2. Detected Identifiers Remain the Same, But Masking Positions Change
Presidio’s analysis results include items like entity_type, start, and end. The Anonymizer receives both the text to be processed and the analysis results. In this structure, it’s crucial which text the positions refer to. Presidio Anonymizer Official Example
Below are the actual results of finding the same identifier in each string. The end position is not included, similar to Python slicing.
| Input Representation | Total Length | Identifier Position [start:end] |
Value Sliced from that Input |
|---|---|---|---|
u1100u1161 DEMO-AB12! |
13 | [3:12] |
DEMO-AB12 |
가 DEMO-AB12! created with NFC |
12 | [2:11] |
DEMO-AB12 |
Both detections are correct. The error occurs when applying the NFC result [2:11] to the first string. In the original text, that span is " DEMO-AB1". It includes the preceding space and misses the last digit 2 of the identifier.
wrong = original[:2] + "<DEMO>" + original[11:]The actual result was "u1100u1161. Conversely, if [3:12] found in NFD is used on the NFC string, it selects "EMO-AB12!". The problem is not only in the direction of length reduction but also in the direction of length increase, creating the same type of error.
This is an application integration error mixing coordinates from different texts. This result alone cannot claim that Presidio itself has a Unicode bug. This verification reproduced the issue by separating the position contract using standard library regular expressions and Python slicing to find fixed identifiers, without running an installed Presidio or NLP model.
Figure: Contract for passing strings and detection positions together. A synthetic example reconstructed based on official documents. Official Basis 1 · Official Basis 2
3. If Using Normalized Input, Apply Replacement to That Input
If there’s no need to preserve the original text representation, a simpler design is to first create a single normalized string and pass it identically to both detection and replacement.
원문 → NFC 문자열 생성 → 그 문자열에서 탐지 → 그 문자열에 치환The core of this synthetic code is also that order.
text = unicodedata.normalize("NFC", original)
detected = detect_demo(text)
safe = apply_verified(text, detected)detect_demo() returns the identifier’s position along with the analyzed string. apply_verified() first compares whether the passed string is the same as the analyzed string.
def apply_verified(text, detection):
if text != detection.analyzed_text:
raise ValueError("analyzed and replacement text differ")
return replace_spans(text, detection.spans)The actual result of this path was 가 , and the path that inserted the pre-normalization string was rejected before replacement. This comparison is an application check created in the example and does not mean that Presidio performs the same check by default.
If multiple processes or caches are involved, the version, normalization method, and index unit of the analysis input can be stored linked to the result. Within the same request, using the analyzed text object throughout reduces errors. Even if hashes are used as linking identifiers, the original text is not unnecessarily left in logs. Hashes of short sensitive values are also not automatically anonymized.
The sequence for actually connecting Presidio is as follows. The code below describes the official API usage pattern and was not executed in this standard library verification.
text = unicodedata.normalize("NFC", original)
found = analyzer.analyze(text=text, language="ko")
result = anonymizer.anonymize(text=text, analyzer_results=found)Replacement Order and Position Unit Also Included in Contract
Even if the same string is passed, if multiple spans are replaced from the front, the length may change with the first replacement, altering the positions of subsequent spans. This is why replace_spans() in the attached code processes spans from largest original position first. For non-overlapping spans like in the example, replacing from the back preserves the coordinates of the preceding spans. This order does not resolve the priority of overlapping entities; overlapping handling requires a separate policy.
If inputs and results are passed between different programs, the contract should specify at least four things: normalization form, version of the analyzed text, unit of position, and whether the end position is inclusive. This article uses Python code point basis and non-inclusive end positions. It does not conclude that UTF-16 positions from JavaScript, etc., can be used as-is.
Verification cases should not end with only strings of different lengths. You can add checks for whether inputs with the same number of code points but different content are rejected, whether all occurrences of an identifier disappear, and whether surrounding spaces and punctuation are preserved. These are regression cases to add to actual integration and do not mean that the original 8 checks verified all cases.
4. If Original Text Must Be Preserved, a Correspondence Relationship Is Needed
If the original text representation must be maintained, the choice differs. If it can be analyzed as-is and processed in the original text, the positions in the two stages will match. This ASCII practice identifier could be found regardless of the preceding Hangul representation, so this path was also verified. This does not mean that all actual recognizers process NFD input with the same quality.
If detection is done with the normalized version and replacement with the original, the correspondence between the normalized span and the original span must be managed separately. Adding one number to the start position as a correction is not a general solution.
원문: 가 DEMO-AB12 가 DEMO-AB12!
NFC: 가 DEMO-AB12 가 DEMO-AB12!The actual identifier positions were [3:12], [16:25] in the original text, and [2:11], [14:23] in NFC. Before the first identifier, one code point was reduced, and before the second, two were reduced. The difference in the two start positions is 1 and 2, respectively. Adding +1 to all results would still make the second span incorrect.
The author’s recommendation is to first decide if original text preservation is essential, and if so, to verify the span correspondence between the pre-processed result and the original text. A general-purpose correspondence algorithm that handles composition, decomposition, and reordering of combining characters was not implemented in this code. It should also not be assumed that concatenating the independently normalized results of each character in a string is always the same as normalizing the entire string.
5. Check Detection Accuracy and Final Replacement Results Together
The attached verify_unicode_offsets.py uses only Python standard libraries. The execution method is as follows:
python3 verify_unicode_offsets.py --output results.jsonExecuted on Python 3.9.6, Unicode database 13.0.0, and all 8 tests passed. Here, a test that reproduced incorrect replacement passing means that the expected residual characters actually appeared in the error path. It does not mean that the path is safe.
The tests verify length differences, NFC coordinate correctness, residual characters when applied to different strings, reverse direction errors, correct replacement of the same string, rejection of inconsistent input, cumulative position differences for repeated identifiers, and preservation of original text representation, respectively. Detailed numerical values and strings can be found in the accompanying results.json.
In actual integration regression tests, it’s better not to just check detection type, start, and end. Also check whether the expected value appears when each span is sliced from the analyzed input, and whether the synthetic identifier completely disappears from the replacement result while surrounding non-target characters are preserved. Since it’s possible to incorrectly pass different strings with the same number of code points, length comparison alone should not substitute for input identity.
This verification is not a measurement of Korean personal information detection rates, the overall operation of the Presidio engine, or position conversion for other language clients. What was confirmed is narrower: position values belong to the analyzed string, and if the string changes due to normalization, that relationship must be re-aligned.
Leave a Reply