Coverage-Aware Web Crawling for Domain-Specific Supplier Discovery via a Web--Knowledge--Web Pipeline
| Authors | Yijiashun Qi et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.24262 |
| Download | |
| Categories | cs.LG |
Abstract
Identifying the full landscape of small and medium-sized enterprises (SMEs) in specialized industry sectors is critical for supply-chain resilience, yet existing business databases suffer from substantial coverage gaps -- particularly for sub-tier suppliers and firms in emerging niche markets. We propose a \textbf{Web--Knowledge--Web (WKW)} pipeline that iteratively (1)~crawls domain-specific web sources to discover candidate supplier entities, (2)~extracts and consolidates structured knowledge into a heterogeneous knowledge graph, and (3)uses the knowledge graph's topology and coverage signals to guide subsequent crawling toward under-represented regions of the supplier space. To quantify discovery completeness, we introduce a \textbf{coverage estimation framework} inspired by ecological species-richness estimators (Chao1, ACE) adapted for web-entity populations. Experiments on the semiconductor equipment manufacturing sector (NAICS 333242) demonstrate that the WKW pipeline achieves the highest precision (0.138) and F1 (0.118) among all methods using the same 213-page crawl budget, building a knowledge graph of 765 entities and 586 relations while reaching peak recall by iteration3 with only 112 pages.
Engineering Breakdown
Plain English
This paper tackles a real supply-chain problem: existing business databases miss huge numbers of small and medium-sized suppliers, especially sub-tier vendors in emerging markets. The authors propose a Web-Knowledge-Web (W→K→W) pipeline that iteratively crawls domain-specific web sources, builds a structured knowledge graph of discovered suppliers, then uses that graph's coverage gaps to guide the next crawl toward under-represented regions. The core insight is that you can use the topology of what you've already discovered to algorithmically decide where to crawl next, creating a feedback loop that progressively fills in coverage blind spots. They also introduce a coverage estimation framework to measure how complete their supplier discovery actually is.
Core Technical Contribution
The technical novelty is the W→K→W pipeline architecture itself—specifically, the feedback mechanism where a heterogeneous knowledge graph's topological and coverage signals directly steer the next crawling decisions. Rather than crawling randomly or using generic web search priorities, the system analyzes what regions of the supplier space are underrepresented in the graph and explicitly targets those gaps in subsequent crawls. This is a closed-loop discovery system where structured knowledge actively shapes data collection strategy, rather than the typical one-directional pipeline of crawl → extract → store. The coverage estimation framework is the second novel contribution, providing a quantitative way to measure discovery completeness—previously unsolved in domain-specific supplier discovery at scale.
How It Works
The pipeline operates in three repeating phases: (1) Web crawling phase queries domain-specific sources (industry directories, trade sites, supplier networks, regulatory filings) to identify candidate supplier entities, generating raw web documents and structured extracts. (2) Knowledge consolidation phase deduplicates entities, resolves name variations and aliases, extracts typed relationships (supply-chain links, geographic locations, product categories), and builds a heterogeneous knowledge graph where nodes are suppliers and edges represent business relationships or shared attributes. (3) Coverage-guided crawling phase analyzes the knowledge graph to identify under-represented regions—this could mean: suppliers in certain geographies with few discovered firms, product categories with sparse coverage, sub-tier suppliers linked to but not directly discovered, or market segments with low entity density—and uses these signals to select new seed URLs and search queries for the next iteration. The system loops until convergence or resource limits, with each iteration expanding coverage in the identified gaps.
Production Impact
For engineering teams building supply-chain platforms, this directly solves the 'dark inventory' problem—the 80% of suppliers that don't appear in public databases because they're small, regional, or operate in niche sectors. Instead of maintaining a static snapshot of supplier data, you'd run this W→K→W pipeline continuously, letting the knowledge graph actively guide data acquisition toward profitable discovery regions rather than treating all web sources equally. The production impact includes: (1) higher supplier recall and more complete supply-chain visibility, reducing sourcing delays and supply-chain fragility, (2) lower cost-per-discovery because crawling is guided by coverage gaps rather than exhaustive, and (3) better ability to detect emerging suppliers and market shifts early. Trade-offs include: the pipeline requires significant infrastructure (web crawlers, entity resolution at scale, graph database, coverage estimation logic), has latency (each iteration takes days to weeks depending on crawl scope), and depends on data quality in source websites—if sources are sparse or use inconsistent naming, the feedback loop may converge slowly or miss real entities.
Limitations and When Not to Use This
The paper's approach assumes that domain-specific web sources have meaningful coverage signal—if the supplier universe is truly invisible online (e.g., small cash-based manufacturers in developing regions with no web presence), crawling fundamentally cannot discover them, making coverage estimation optimistic. The heterogeneous knowledge graph depends heavily on accurate entity resolution and deduplication; if that module misses aliases or conflates different companies, the graph's coverage signals become unreliable and subsequent crawls may be misdirected. The framework does not address quality of discovered suppliers—finding 10,000 entities is useless if 90% are defunct, fraudulent, or irrelevant to your use case; the paper likely focuses on recall (finding all suppliers) rather than precision (finding good suppliers). Finally, generalization across industries is unclear: a supplier discovery system trained on, say, semiconductor sub-tier manufacturers may not transfer well to pharmaceutical or apparel, since the web presence and naming conventions vary drastically by sector.
Research Context
This work sits at the intersection of knowledge graph construction, web information extraction, and supply-chain intelligence—building on decades of entity extraction (NER) and graph completion research, but applying them to a practically unsolved problem in procurement. It extends prior work on iterative web crawling (which typically used relevance feedback on document text) to use structural graph signals instead, a novel direction in discovery-oriented information extraction. The paper likely benchmarks against existing supplier databases (e.g., D&B, Crunchbase, trade registries) to quantify coverage gaps, contributing a new evaluation metric (coverage estimation) to the field. It opens research directions into coverage-aware information extraction (how do you know what you don't know?), supply-chain resilience through data, and scalable entity resolution for dynamic, heterogeneous web-sourced data.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::