PRISM: LLM-Guided Semantic Clustering for High-Precision Topics
| Authors | Connor Douglas et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.03180 |
| Download | |
| Categories | cs.LG, cs.CL, cs.IR, cs.SI |
Abstract
In this paper, we propose Precision-Informed Semantic Modeling (PRISM), a structured topic modeling framework combining the benefits of rich representations captured by LLMs with the low cost and interpretability of latent semantic clustering methods. PRISM fine-tunes a sentence encoding model using a sparse set of LLM- provided labels on samples drawn from some corpus of interest. We segment this embedding space with thresholded clustering, yielding clusters that separate closely related topics within some narrow domain. Across multiple corpora, PRISM improves topic separability over state-of-the-art local topic models and even over clustering on large, frontier embedding models while requiring only a small number of LLM queries to train. This work contributes to several research streams by providing (i) a student-teacher pipeline to distill sparse LLM supervision into a lightweight model for topic discovery; (ii) an analysis of the efficacy of sampling strategies to improve local geometry for cluster separability; and (iii) an effective approach for web-scale text analysis, enabling researchers and practitioners to track nuanced claims and subtopics online with an interpretable, locally deployable framework.
Engineering Breakdown
Plain English
PRISM is a topic modeling framework that combines large language models (LLMs) with semantic clustering to identify and separate closely related topics in specialized domains. Instead of relying on expensive, frontier-scale embedding models or traditional latent semantic approaches, PRISM fine-tunes a sentence encoder using a small number of LLM-generated labels as training signals. The method achieves better topic separability than existing local topic models and even outperforms clustering on large pre-trained models, while requiring significantly fewer LLM queries during training. This makes it practical for production systems that need both precision and cost efficiency when discovering topics within narrow, domain-specific corpora.
Core Technical Contribution
The core innovation is a hybrid architecture that leverages LLMs as a labeling oracle rather than as the primary inference engine. PRISM decouples the expensive LLM component (used only for sparse, targeted labeling during training) from the cheap inference component (efficient semantic clustering on fine-tuned embeddings). The key insight is that you don't need to run large models at inference time if you intelligently seed a smaller embedding space with high-quality LLM guidance during training. This stands apart from prior work that either uses full LLM inference at test time or ignores LLM capabilities entirely, instead positioning LLMs as a data annotation tool that improves downstream model quality.
How It Works
The pipeline operates in three stages: (1) sample selection and LLM annotation—PRISM draws a strategic subset of samples from the target corpus and queries an LLM to assign semantic labels that capture fine-grained topic distinctions relevant to that domain; (2) embedding space fine-tuning—the sparse LLM-annotated samples are used to train a sentence encoder (likely via contrastive learning or supervised fine-tuning) such that the learned embedding space reflects the LLM's semantic judgments; (3) thresholded clustering—the entire corpus is embedded using the fine-tuned encoder, then clustered using distance-based methods with carefully tuned thresholds that separate closely related topics while avoiding false splits. The thresholded clustering step is crucial because it avoids the brittleness of fixed cluster counts and allows the model to discover natural topic boundaries determined by the embedding geometry.
Production Impact
Adopting PRISM would fundamentally change how teams approach topic discovery in production systems. First, it reduces operational cost: instead of running inference through GPT-4 or Claude on every document, you pay for LLM queries only once during model training (for a small labeled set), then use a lightweight embedding model at serving time—this could reduce inference cost by 100-1000x compared to full LLM pipelines. Second, it improves latency and reliability: embedding-based inference is sub-millisecond and deterministic, whereas LLM inference is variable and can involve rate-limiting or API failures. Third, the fine-tuned embedding model becomes portable—it can be deployed on-device, in edge environments, or behind simple REST endpoints without dependency on external APIs. The trade-off is that you need domain-specific labeled data and must spend engineering effort on threshold tuning; for broad, general-purpose topic modeling, off-the-shelf embeddings might suffice.
Limitations and When Not to Use This
PRISM assumes that a small number of strategically chosen LLM labels can adequately capture the semantic structure of a domain, which may fail for corpora with emergent topics or concepts that weren't represented in the training samples. The method's performance is sensitive to the quality of LLM labels and the sampling strategy used to select training samples—poor label quality or unrepresentative sampling could propagate errors through the fine-tuning process. Thresholded clustering requires manual threshold tuning or validation on a held-out set, adding engineering overhead compared to fixed-cluster approaches; there's no principled way to set thresholds a priori without domain knowledge. The paper doesn't address how to handle very large corpora where even the fine-tuning stage becomes expensive, or how performance degrades in low-resource scenarios where LLM access is extremely limited.
Research Context
PRISM builds on a long tradition of combining neural embeddings with semantic clustering (e.g., deep clustering, semantic-aware topic models) while incorporating recent advances in LLM-as-labeler paradigms seen in in-context learning and weak supervision literature. It implicitly acknowledges that frontier LLMs have become strong semantic judges but are too expensive for inference at scale, positioning itself as a practical synthesis of LLM quality and classical model efficiency—similar to how knowledge distillation transfers large model capabilities to smaller ones. The work likely benchmarks against standard baselines like BERTopic, traditional LDA variants, and direct clustering on embeddings from models like Sentence-BERT or OpenAI's embedding models. This contributes to a growing research direction around hybrid human-AI and model-model collaboration, where expensive models guide cheaper ones, rather than replacing them entirely.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::