Running the Stepstone API on a schedule without it dying

blackfalcondata

Extracting fields is not the hard part of a Stepstone API. The challenge is getting a response worth extracting from, twice in a row, on a schedule—and fixing failures before a human notices.

Here is what our own Stepstone Actor handles, what a run costs, and where it gives up.

Doing it yourself

Building your own scraper makes sense when you need one snapshot, or when the data is simple enough that a broken run costs you nothing. The work involved is less simple. These specifics come from what our own scraper has to handle.

The front door. When measured on 2026-08-22, Stepstone served a plain request normally (302, API Gateway). Reaching the page is not the problem. Pagination, parsing and keeping both working are.

Getting past the gate. A 403 from Stepstone does not always mean refusal. It often means the session cookie went stale. The fix is to obtain a new one and repeat the request, not back off.

Telling a challenge from a block. Stepstone uses the same status for "prove you are a browser" and "go away". The body reveals the difference. A client that ignores it wastes its retry budget on the second case.

Rate limits. Stepstone rate-limits per-source rather than per-account. Exponential backoff makes a run slow; leaving it out makes the run fail.

Proxies. One IP requesting a thousand Stepstone pages looks like a pattern, not traffic. Rotation makes that volume unremarkable—and dominates the bill.

Pagination. A Stepstone pagination loop needs to distinguish "no more results" from "we stopped being served results". Both produce an empty page.

Silent breakage. A Stepstone redesign rarely breaks a scraper loudly. One selector fails, a field returns empty, the run reports success, and the first warning is a report that looks thin.

The real cost is upkeep. The first version takes an afternoon. Keeping it alive takes an hour every few weeks, forever. Failures stay quiet: you discover them when a report is empty, not when the scraper errors.

This is a measured run, not an estimate:

25 records in 24.9 seconds
cost: $0.03 — about $1.12 per 1,000 records

The shortcut

If you do not want to own that work, the same result takes one call:

```bash curl -X POST "https://api.apify.com/v2/acts/blackfalcondata~stepstone-de-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"query": "..."}' ```

Or use Python:

```python from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN") run = client.actor("blackfalcondata~stepstone-de-scraper").call(run_input={"query": "..."})

for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ```

What comes back

Each record contains 81 fields, grouped into categories such as:

company — company, companyId, companyLogo, companyUrl, companyWebsite, companyEmployees, +4 more
is — isSponsored, isTopJob, isPartnershipJob, isAnonymous, isHighlighted, isRepost
salary — salaryText, salaryMin, salaryMax, salaryCurrency, salaryPeriod

What people use it for

Benchmarking. Pricing and salary fields support comparisons across the whole set instead of an eyeballed page of results.
Reputation monitoring. Ratings and review text reveal movement over time that one page view cannot show.
Lead lists. Company-level fields narrow the data to the segment you care about before anyone opens a CRM.
Change detection. Timestamps let you diff runs and act on new entries instead of re-reading everything.
Geographic analysis. Location fields enable market-by-market comparisons without another data source.

What it will not do

Nothing here gets you past a Stepstone login. Public pages only, whoever writes the scraper.
A working scraper today is not a working scraper in six weeks; the failure mode is silence, not an exception.
Proxy cost scales with volume and is the line item people forget when they estimate the DIY route.

Questions people ask

Is scraping Stepstone legal?

Reading publicly visible pages is generally treated differently from accessing data behind a login, and courts in the US have repeatedly declined to treat public scraping as unauthorised access. That is not legal advice, and a site's terms of service are a separate question from the law. Anything behind a login is out of scope either way.

Why am I getting 403s when the page loads fine in my browser?

The check is not about the URL. A plain request to Stepstone from our probe on 2026-08-22 was answered by no bot-protection header — 0 of 3 attempts were refused. Your browser passes because it carries a TLS fingerprint, a cookie and a header set that a bare HTTP client does not.

Do I need proxies?

For anything beyond a small sample, yes. Our own input exposes `proxyConfiguration` because a single origin IP is the easiest thing in the world to rate-limit.

How often will it break?

Expect to touch it whenever Stepstone ships a redesign—and expect the failure to go unnoticed for a while. Extraction fails quietly, returning empty fields rather than errors. Budget an hour every few weeks rather than a fix-and-forget afternoon.

Is it cheaper to build it myself?

Only if your time is free. The measured run below cost $0.03 for 25 records; the equivalent DIY cost is proxy spend plus the maintenance hour, and the maintenance hour recurs.

Try it

To avoid owning the upkeep, use our Stepstone API Actor on Apify. It already handles everything above. New accounts get $5 of free platform credit each month, which covers a real run of this size several times over.

Disclosure: we build and maintain this Actor, and the link above is an affiliate link.

Running the Stepstone API on a schedule without it dying

Running the Stepstone API on a schedule without it dying

Doing it yourself

The shortcut

What comes back

What people use it for

What it will not do

Questions people ask

Try it

Further reading

Report Page