Sign In
Chapters · #27

Building an Asynchronous Asset Inventory

Reading Progress45%

Chapter 27: Building an Asynchronous Asset Inventory

As the platform scales, manually tracking third-party dependencies and infrastructure components becomes an impossible task. To solve this, I designed a dual-stream Asset Inventory and Vulnerability Scanner engine that operates asynchronously without bloating the core application.

Instead of embedding heavy security scanning tools directly into the main Django backend, I created an isolated, offline microservice (scanner/) built on FastAPI. This service utilizes the official osv-scanner binary to parse application lockfiles (like requirements.txt or package-lock.json) against a locally mounted OSV database, ensuring no sensitive manifests are transmitted over the public internet. Concurrently, it leverages a cpe-guesser to normalize raw infrastructure strings (e.g., "nginx 1.21") into standardized CPE 2.3 formats.

The ingestion pipeline in the core backend (backend/telemetry/vulnerability_ledger.py) exposes a unified /api/telemetry/technology endpoint capable of receiving both infrastructure signatures and application dependency manifests. To achieve maximum throughput, the backend processes these payloads in batches using the high-performance Polars DataFrame library. Once the raw telemetry is normalized, the backend securely delegates all scanning logic to the isolated scanner microservice. This microservice dynamically queries the National Vulnerability Database (NVD) REST API and OSV.dev REST API to extract exact CVEs and CVSS metrics, seamlessly bridging the gap between hardware/infrastructure reporting and modern application lockfile scanning.

Public web surfaces add a third, evidence-bounded discovery stream. For every verified ValidatedSite, a scheduled enrich_web_technologies task calls Firecrawl only for that account's approved public origins. Firecrawl renders JavaScript and returns a structured list of observable technologies—frameworks, CMS products, web servers, analytics providers, CDNs, and exposed version strings—together with the page evidence and a confidence score. DEML treats these results as untrusted observations rather than vulnerability facts: names are normalized, exact duplicates collapse inside the account boundary, and only evidence-bearing candidates enter the existing CPE 2.3 and CVE comparison path. Private, loopback, link-local, metadata-service, credential-bearing, and cross-domain targets are rejected before any request is submitted. The connector is disabled unless its server-side secret and explicit feature flag are present; no Firecrawl credential reaches a browser or FORJD projection.

The transactional result is materialized in the dedicated web_technology_observations table. Each row records the native account UUID, verified site, source URL, technology name and version, confidence, evidence, matched CPE, CVE identifiers, and first/last-seen timestamps. This preserves provenance and makes the dashboard read path independent of Firecrawl availability. A scan failure never deletes the last known inventory and never manufactures a vulnerability. Successful scans forward a compact versioned summary to FORJD through the BFF adapters, so FORJD can project it without coupling Firecrawl to the UI. Tenant0 follows the same loop through its verified public domains; customer rows remain scoped to their owning UserProfile.account_id.

Scheduled task: enrich_web_technologies
  → verified public ValidatedSite origins
  → Firecrawl structured technology evidence
  → normalize/deduplicate technology + version
  → FORJD scanner surfaces (CPE 2.3 + NVD/OSV CVE enrichment)
  → PostgreSQL WebTechnologyObservation; summary forwarded to FORJD via BFF adapter
  → FORJD projection / security dashboard

Finally, enriched vulnerability evidence is persisted through FORJD scanner / threat surfaces (tenant-scoped tables and projections) and surfaced on the Security dashboard via Django BFF adapters. Critical findings may also be summarized in DEML-owned operational views for operator alerting. DEML does not write ClickHouse JSONEachRow / MergeTree volumes; that warehouse pattern is retired (Chapter 34).