PRISM

Context

PRISM is a Python package designed for string similarity matching¹. It provides a single API to compare strings using several independent algorithms, such as character distance metrics, token comparison, phonetic indexing, and semantic embeddings.

Challenge

String matching algorithms are used to identify similarities or differences between texts. However, different algorithms are suited to different types of text variations. For example, typographic errors (such as spelling mistakes) are typically detected by character-level edit distances. Vocabulary variations (such as synonyms like “car” and “automobile”) cannot be identified by character matching and instead require semantic representation models. Different algorithms are often implemented in separate libraries with varying input and output formats, requiring custom integration code to compare their results.

Approach

PRISM provides a unified framework built around a central Matcher class and a registry of similarity methods. Every method implements a standard interface defining a compute(a, b) function that returns a similarity score as a float value between 0.0 (entirely different) and 1.0 (identical).

The methods are grouped into six categories:

• Edit distance (calculates character-level modifications): Levenshtein, Damerau-Levenshtein, Hamming.
• Sequence similarity (analyzes character patterns and ordering): Jaro-Winkler, SequenceMatcher.
• Token-based (splits strings into individual words or sub-units called tokens before comparison): token sort, token set, partial ratio.
• Phonetic (converts words into codes based on English pronunciation): Soundex, Metaphone, Double Metaphone, Match Rating Codex.
• TF-IDF (Term Frequency-Inverse Document Frequency, which weights words based on their relative importance across documents): TF-IDF vectorisation combined with cosine similarity.
• Embedding (uses numerical vectors representing semantic meaning to compare texts): semantic similarity using the Jina Embeddings v3 model².

Heavy dependencies such as scikit-learn (for TF-IDF), fastembed (for embeddings), and matplotlib (for visualization) are loaded lazily when the respective method is first called. This prevents unnecessary memory overhead at startup.

Features

• Unified API: All registered methods are accessed via the compare(a, b) method of the Matcher class.
• Method filtering: Users can instantiate the Matcher with a subset of specific methods or category filters.
• Preprocessing pipeline: Input strings can be normalized using chained preprocessing functions (such as lowercase conversion, accent removal, or punctuation stripping) prior to similarity evaluation.
• Batch comparison: The compare_batch() method processes multiple pairs of strings in parallel using a pool of threads (ThreadPoolExecutor).
• Custom registry: New similarity metrics can be registered by decorating custom classes with the @register decorator.

Outcome

PRISM provides a structured environment to evaluate and compare multiple string similarity metrics simultaneously. By decoupling the algorithms from their source libraries and unifying their interface, the library enables developers to select, chain, and execute appropriate matching strategies based on their specific text data characteristics.