How to Aggregate 30+ Job Boards Into One Recruiting Feed Without an ATS Integration

Key Takeaways

Why doesn't an ATS cover niche job boards? Greenhouse, Lever, and Ashby integrate with a handful of high-volume boards (LinkedIn, Indeed) because that's where the integration ROI is obvious. Niche boards — Y Combinator's Work at a Startup, We Work Remotely, Naukri, or any single company's Workday-hosted careers page — have no shared API standard, so ATS vendors skip them. The candidates are still there; your pipeline just doesn't see them.

What does a normalized job record look like across five different sources? A common schema needs at minimum: title, company, location, salary range, currency, source board, and a canonical URL. Everything else — equity, tags, seniority level — is source-specific enrichment you attach after normalization, not before.

How do you avoid pulling the same job twice from overlapping boards? Dedupe on a composite key (normalized company name + normalized title + location), not on URL — the same posting frequently appears on the company's own Workday page and on an aggregator like RemoteOK with different URLs.

The Problem With ATS-Only Sourcing

If you're sourcing candidates or building a market map of who's hiring for a role, your applicant tracking system is not your data source — it's your destination. It only knows about the boards someone configured an integration for. Meanwhile, a huge share of real postings live on boards with no ATS relationship at all: Y Combinator's job board alone lists 3,500+ jobs across 800+ funded startups, most of which never get syndicated anywhere else.

The fix isn't a bigger ATS contract. It's treating job data the way you'd treat any other structured web data: pull it directly from each source, normalize it into one schema, and let your sourcing tool or spreadsheet read from that instead of five open tabs.

Picking Your Board Mix

Every board scraper returns a different shape of data because every board's underlying markup is different. Before normalizing anything, it helps to know what each source is actually strong at:

Source	Best for	Salary data	Notes
YC Jobs Scraper	Early-stage startup roles	Yes — min/max/currency + equity range	Enriches with founder names and company metadata automatically
We Work Remotely Scraper	Fully remote roles	Sometimes — free-text bracket	includeDetails fetches full JD + apply URL
Hiring.cafe Scraper	Cross-ATS aggregation (2.8M+ postings, 46 ATS platforms)	Structured min/max/currency when transparent	onlyTransparentSalaries filters out the noise
Workday Jobs Scraper	Any single company on Workday (Nvidia, Salesforce, etc.)	Structured, parsed from the posting	Seeded by company career-site URL, not keyword search
RemoteOK Jobs Scraper	Tagged remote tech roles	Structured min/max	Tag-based filtering (python, react, backend)

For most recruiting use cases, a mix of three or four is enough: one broad aggregator (Hiring.cafe), one for the specific company or companies you're targeting (Workday), and one or two niche boards that match your candidate pool (YC for startups, WWR or RemoteOK for remote-first roles).

Normalizing Wildly Different Schemas

Here's the actual problem: none of these five actors return the same field names. A salary range on Hiring.cafe is salaryMin/salaryMax. On YC it's salaryMin/salaryMax too — but nested equity is a separate free-text field. On Workday it's buried inside a salary.rawText string next to structured min/max. Raw output, side by side:

// yc-jobs-scraper
{ "title": "Software Engineer", "companyName": "Kabilah",
  "salaryMin": 125000, "salaryMax": 180000, "salaryCurrency": "USD",
  "location": "New York, NY, US / Remote (US)" }

// hiring-cafe-scraper
{ "title": "Senior Software Engineer, Backend", "companyName": "Acme Corp",
  "salaryMin": 160000.0, "salaryMax": 220000.0, "salaryCurrency": "USD",
  "workplaceCities": ["San Francisco", "New York"], "workplaceType": "Remote" }

// workday-jobs-scraper
{ "title": "Senior CPU Performance Architect",
  "employer": { "name": "2100 NVIDIA USA" },
  "salary": { "min": 184000, "max": 282000, "currency": "USD", "period": "year" },
  "location": { "text": "US, CA, Santa Clara" } }

None of that is usable as-is for a unified feed. Write one small normalizer per source that maps into a shared shape, then run everything through it:

function normalize(record, source) {
  switch (source) {
    case "yc":
      return {
        title: record.title,
        company: record.companyName,
        location: record.location,
        salaryMin: record.salaryMin,
        salaryMax: record.salaryMax,
        currency: record.salaryCurrency,
        url: record.url,
        source: "yc",
      };
    case "hiringcafe":
      return {
        title: record.title,
        company: record.companyName,
        location: (record.workplaceCities || []).join(", ") || record.workplaceType,
        salaryMin: record.salaryMin,
        salaryMax: record.salaryMax,
        currency: record.salaryCurrency,
        url: record.url,
        source: "hiringcafe",
      };
    case "workday":
      return {
        title: record.title,
        company: record.employer?.name,
        location: record.location?.text,
        salaryMin: record.salary?.min,
        salaryMax: record.salary?.max,
        currency: record.salary?.currency,
        url: record.applyUrl,
        source: "workday",
      };
  }
}

Run every source's output through its normalizer, concatenate the results, and you have one flat array with a consistent shape — this is what you feed into a spreadsheet, a sourcing CRM, or a downstream LLM step for scoring fit against a candidate profile.

Deduplicating Across Sources

Overlap is common — a startup's own Workday posting frequently gets re-syndicated to RemoteOK or Hiring.cafe verbatim. Don't dedupe on URL; different sources give the same job different URLs. Dedupe on a normalized composite key instead:

function dedupeKey(job) {
  return [
    job.company?.toLowerCase().trim(),
    job.title?.toLowerCase().replace(/[^a-z0-9]/g, ""),
    job.location?.toLowerCase().split(",")[0]?.trim(),
  ].join("::");
}

If you'd rather not hand-roll this, run the combined dataset through the Data Deduplicator actor — it merges and deduplicates by any field combination without you writing the matching logic yourself, which matters once you're combining five sources instead of two.

Scheduling and Freshness

Job feeds go stale fast — a role open today can be filled in a week. Run each source actor on a schedule (daily is usually enough; hourly only if you're sourcing for high-velocity roles) and only re-normalize and dedupe the delta, not the full history each time. Most of these actors expose a maxItems/maxResults cap specifically so a scheduled daily run stays cheap.

Where This Goes Next

Once you have a normalized, deduped feed, the natural next step is turning posted salary ranges into an actual benchmarking dataset instead of a one-off pull — see Building a Salary Benchmarking Dataset From Public Job Postings for that. If you want to browse the rest of the job-board actors beyond the five here — Naukri, Bayt, Xing, Dice, Arbeitsagentur, and two dozen more — the full list is on the ParseBird homepage.