Apollo Lead Scraper

What it is

Apollo Lead Scraper is a local, CLI-first MVP for extracting people or company records from an Apollo search that an authenticated user is permitted to access. It is intentionally the sourcing layer: it captures records, turns them into stable business data, and prepares them for spreadsheet use or a controlled downstream handoff.

The problem and business value

Lead lists are useful only when they are consistent, recoverable, and easy to move into the next business workflow. This project separates Apollo-specific browser and payload handling from provider-agnostic normalization, deduplication, checkpointing, export, and delivery concerns. That makes a search run easier to repeat, recover after interruption, and integrate without treating Apollo’s changing private browser payloads as a stable public API.

The business value is operational rather than a claimed growth result: less manual list preparation, cleaner records, and a safer path from permitted research to a usable lead batch. The repository does not contain verified adoption, revenue, or customer outcome data.

What it does today

• Opens a supplied Apollo people or company search through Playwright using a locally saved authenticated session.
• Captures relevant XHR/fetch responses, parses permissive person/company shapes with Zod-backed schemas, and falls back to visible table rows when structured records are unavailable.
• Handles pagination, configurable pacing, rate-limit detection, exponential backoff, progress reporting, and graceful interruption.
• Normalizes names, email addresses, domains, LinkedIn URLs, locations, company fields, and source metadata before deduplicating leads or companies.
• Writes spreadsheet-compatible CSV and optional JSON output, with page-level checkpoints that can be resumed by job ID.
• Supports multiple Apollo accounts through explicit account selection; it does not rotate accounts automatically.

Integrations and data flow

The main flow is:

authenticated Apollo session → Playwright browser → sanitized network interception → defensive response parser / DOM fallback → normalization → deduplication → local checkpoint and ingestion ledger → CSV/JSON export

The local ingestion ledger canonicalizes equivalent search URLs, caches completed queries, deduplicates lead identities across batches and separate searches, and stages page-level batches before delivery. An optional HTTP client can send normalized lead batches to an OpportunityOS ingestion endpoint with an idempotency key and an optional bearer credential. Pending batches can be retried with deliver-pending without opening Apollo or consuming Apollo quota.

The OpportunityOS connector and its request behavior are implemented and covered with a fake transport in the test suite. A real endpoint, provider account authorization, acknowledgement contract, and production delivery path have not been verified in this repository.

AI or automation workflow

There is no AI model or AI-generated content in this repository. The automation is deterministic browser and data-processing work: a human logs into Apollo, chooses the search and limit, and starts the run; the system then captures, parses, normalizes, deduplicates, checkpoints, exports, and optionally delivers the resulting batches. It is not an autonomous outreach workflow and does not send email or social messages.

Lead, email, notification, or collaboration flow

Lead records may include email and email-status fields when Apollo exposes them, and those fields are included in normalized exports. The optional lead flow stages normalized batches locally and sends them only when an OpportunityOS endpoint is configured; a batch is marked delivered only after the downstream sink acknowledges it. There is no email sender, notification service, owner-replied chat, or collaboration UI in this project.

Security/privacy boundaries

• Apollo authentication is manual. The CLI does not ask for or persist an Apollo username or password; Playwright session state is stored locally and ignored by Git.
• Lead data, checkpoints, ingestion state, exports, logs, and browser session state are local by default. Treat those runtime directories as sensitive operational data.
• Debug network logging records sanitized request/response metadata rather than credentials or full request payloads. Common authorization, cookie, token, password, and secret fields are redacted in logs.
• The scraper does not bypass authentication, credits, access controls, CAPTCHA, payment mechanisms, or rate limits, and it does not rotate proxies.
• No telemetry or external analytics are implemented. Data leaves the machine only through the explicitly configured downstream sink.
• The public portfolio brief describes behavior and boundaries only; it contains no session state, environment values, raw logs, private URLs, or lead data.

What is coming next

The next step is controlled live validation against an authenticated Apollo account: confirm the current response shapes, pagination controls, rate-limit behavior, and field availability without recording sensitive search values. The optional OpportunityOS path then needs a real endpoint, authorization, acknowledgement behavior, and production failure/retry testing.

Longer-term direction is to expose the reusable ScraperService through a REST API or worker, add another source through the existing registry boundary, and evaluate richer enrichment or scoring. A database-backed ingestion store would be needed for multi-worker deployment. Enrichment, intelligence, outreach, queues, and a web dashboard are intentionally outside the current MVP until extraction reliability and maintenance cost are validated.

Current activity or evidence from the project

The repository currently contains one initial Git commit dated 2026-08-26. Local verification on that revision passes the strict TypeScript build, 19 automated tests across 8 test files, ESLint, and the Prettier format check. The tests cover URL validation, defensive response parsing, account/session-path bookkeeping, normalization, deduplication, CSV generation, service orchestration, query caching, staged delivery retry, and idempotency-key handling.

The offline suite uses fixtures, temporary directories, and fake sources or transports. It does not claim a successful live Apollo scrape or production OpportunityOS delivery; the research notes explicitly leave live request/response observations and rate-limit observations pending.