Deduplicate Then Sort: A 12-Pipeline Order Test

Question: Does trim → dedupe → sort produce a different list than sort → dedupe on the same messy input?
Samples: 12 fixtures
Generated via core: 11
Manual / limits: 1
Published: 2026-07-18
Modified: 2026-07-18
Author: Dionis Ceban
Method: Twelve fixtures share one corpus plus focused mini-lists. Each pipeline runs through the shared core (trimText, removeDuplicateLines, sortLines) with explicit options. One observation fixture documents numbering-before-dedupe.

Why this test matters

Invisible spaces and case variants create false uniqueness. Sorting before normalizing locks those variants into separate sort keys.

Tools under test

Result summary

11 of 12 fixtures were executed through assets/js/tools-src/core.js. 1 entries are labeled manual review or limitation without inventing a transform result.

Full test matrix (12 samples)

ID	Case	Input	Operation	Observed	Status
list-dedupe-only-01	Deduplicate only (case-insensitive, trim)	`Apple apple Banana banana Cherry Item 2 Item 10 Item 1 Café Cafe Apple`	removeDuplicateLines	`Apple Banana Cherry Item 2 Item 10 Item 1 Café Cafe`	Pass
list-trim-dedupe-02	Trim → deduplicate	`Apple apple Banana banana Cherry Item 2 Item 10 Item 1 Café Cafe Apple`	trimText → removeDuplicateLines	`Apple Banana Cherry Item 2 Item 10 Item 1 Café Cafe`	Pass
list-sort-dedupe-03	Sort → deduplicate (no trim)	`Apple apple Banana banana Cherry Item 2 Item 10 Item 1 Café Cafe Apple`	sortLines → removeDuplicateLines	`Apple apple Banana Cafe Café Cherry Item 1 Item 10 Item 2`	Pass
list-trim-dedupe-sort-04	Trim → deduplicate → natural sort	`Apple apple Banana banana Cherry Item 2 Item 10 Item 1 Café Cafe Apple`	trimText → removeDuplicateLines → sortLines	`Apple Banana Cafe Café Cherry Item 1 Item 10 Item 2`	Pass
list-case-sensitive-05	Case-sensitive duplicates	`Alpha alpha ALPHA Alpha`	removeDuplicateLines	`Alpha alpha ALPHA`	Pass
list-case-insensitive-06	Case-insensitive duplicates	`Alpha alpha ALPHA Alpha`	removeDuplicateLines	`Alpha`	Pass
list-unicode-accent-07	Accent vs ASCII lookalike	`Café Cafe CAFÉ`	removeDuplicateLines	`Café Cafe`	Pass
list-numeric-sort-08	Numeric/natural sort vs lexical	`Item 10 Item 2 Item 1`	sortLines	`Item 1 Item 10 Item 2`	Pass
list-lexical-sort-09	Lexical sort of numbered items	`Item 10 Item 2 Item 1`	sortLines	`Item 1 Item 10 Item 2`	Pass
list-empty-lines-10	Empty and whitespace-only lines	`Keep Keep`	trimText → removeDuplicateLines	`Keep`	Pass
list-number-before-11	Numbering before deduplication	`1. Apple 2. Apple 3. Banana`	Observation (no auto transform)	If you number lines first (1. Apple, 2. Apple), dedupe will not merge them. Prefer dedupe before numbering.	Manual review
list-number-after-12	Numbering after deduplication	`Apple Apple Banana`	removeDuplicateLines → identity	`Apple Banana`	Pass

Edge cases and failures

Café vs Cafe stay distinct under ASCII comparison; Item 10 sorts before Item 2 unless numeric sort is enabled.

Conclusion

Normalize (trim) before dedupe when whitespace creates clones; choose case sensitivity deliberately; use numeric sort for numbered labels; number lines after uniqueness is settled.

Limitations

Accent folding / Unicode canonical equivalence is not applied during dedupe.
Natural sort follows the tool’s numeric option, not a full CLDR collation.
Number Lines is not executed inside these pipelines; numbering order is discussed as process guidance.

Reproducibility

Use the published list-corpus.txt and fixtures.json. Re-run npm run guides:evidence. Compare outputs character-for-character to expected-results.json.