How to Scrape LinkedIn Profiles, Companies, and Jobs with an API

LinkedIn data looks consistent in a browser, but the underlying entities are not interchangeable. A person has experience and education, a company has industries and locations, a post has changing engagement metrics, and a job has employment and application fields.

This tutorial builds a maintainable public-data workflow with four SocQ LinkedIn APIs: profiles, companies, posts, and jobs. It covers URL preparation, asynchronous tasks, normalized records, storage, validation, refresh schedules, and responsible use.

Quick answer: Send public LinkedIn URLs to the matching POST /v1/linkedin/{resource} endpoint, store the returned task ID, poll until completion, then validate and upsert the normalized records. Do not send a profile URL to the jobs endpoint or assume that a missing public field is an empty value.

Official LinkedIn API or a Public Data API?

LinkedIn's official APIs are the right choice when a member authorizes your application, when you have access to an approved partner product, or when the workflow involves publishing or another permissioned action. Official access uses OAuth, and many permissions and partner programs require explicit approval. Profile results are also subject to member privacy settings and contractual storage rules.

The SocQ endpoints in this tutorial accept known public URLs and perform read-only collection. They do not authenticate as a LinkedIn member, search all of LinkedIn, publish content, or grant rights to use the returned data. Choose the access model before choosing the endpoint: official APIs for authorized account actions, a supported public-data API for narrowly scoped public reads.

Choose the Correct LinkedIn Endpoint

SocQ currently exposes four URL-based read endpoints:

These are public read operations. Use LinkedIn's official products when you need member authorization, publishing, ads, or another permissioned workflow.

Normalize and Route Input URLs

Store the submitted URL as provenance, but canonicalize a copy before routing:

1. Require HTTPS and a LinkedIn hostname.
2. Remove tracking query parameters.
3. Normalize locale and mobile host variants where safe.
4. Classify the path by entity type.
5. Deduplicate canonical URLs within the batch.
6. Reject unsupported paths before spending a request.

A simple router can map /in/ to profiles, /company/ to companies, known post patterns to posts, and /jobs/view/ to jobs. Keep ambiguous URLs in a review queue rather than guessing.

Submit a Profile Collection Task

curl -X POST "https://api.socq.ai/v1/linkedin/profiles" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://www.linkedin.com/in/satyanadella"
    ]
  }'

Use the same request shape for the other resources:

curl -X POST "https://api.socq.ai/v1/linkedin/companies" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://www.linkedin.com/company/microsoft/"]}'

curl -X POST "https://api.socq.ai/v1/linkedin/jobs" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://www.linkedin.com/jobs/view/software-engineer-at-microsoft-4429527118"]}'

The response creates an asynchronous task. Persist its ID next to your internal batch ID and inputs.

Poll the Task without Creating Duplicates

curl "https://api.socq.ai/v1/tasks/$TASK_ID?limit=100" \
  -H "Authorization: Bearer $SOCQ_API_KEY"

Treat queued and running as expected states. Poll with exponential backoff and a maximum delay. For a successful task, process data.results.items; when has_more is true, continue with next_cursor.

Your job table should record:

• Internal batch ID and SocQ task ID.
• Resource type and canonical input URLs.
• Submission, last poll, and completion timestamps.
• Terminal status and normalized error category.
• Last committed result cursor.

Do not resubmit merely because a worker restarted. Resume from persisted task state first.

Understand the Four Schemas

All records share normalized identity fields such as id, platform, resource, type, url, created_at, collected_at, and extra. Resource-specific fields remain explicit.

A profile includes name, description, avatar_url, visible follower and connection metrics, plus optional role, company, location, experience, education, and skills in extra.

A company adds website, logo, visible follower and employee counts, and optional industry, size, headquarters, founding, locations, and specialties.

A post includes text, normalized author, visible like/comment/repost metrics, media, publication time, and optional post type or hashtags.

A job uses name for the title, text for the description, author for the hiring company, and extra for location, seniority, employment type, industries, application URL, Easy Apply status, and salary when public.

Missing data must remain null or absent. A hidden follower count is not zero, an unknown salary is not unpaid, and an unavailable timestamp is not the collection time.

Python Worker for Mixed LinkedIn URLs

import time
import requests

BASE = "https://api.socq.ai/v1"
HEADERS = {
    "Authorization": f"Bearer {SOCQ_API_KEY}",
    "Content-Type": "application/json",
}

def collect(resource, urls):
    created = requests.post(
        f"{BASE}/linkedin/{resource}",
        headers=HEADERS,
        json={"urls": urls},
        timeout=30,
    )
    created.raise_for_status()
    task_id = created.json()["data"]["task_id"]

    delay = 2
    while True:
        response = requests.get(
            f"{BASE}/tasks/{task_id}",
            headers=HEADERS,
            timeout=30,
        )
        response.raise_for_status()
        task = response.json()["data"]

        if task["status"] == "succeeded":
            return task["results"]["items"]
        if task["status"] in {"failed", "cancelled"}:
            raise RuntimeError(task)

        time.sleep(delay)
        delay = min(delay * 1.5, 15)

profiles = collect("profiles", [
    "https://www.linkedin.com/in/satyanadella"
])

In production, add retries only for transient failures, persist the task before polling, and paginate results instead of assuming one response contains the full batch.

Model Source Records and Snapshots Separately

Profiles and companies change slowly; post metrics and job availability can change quickly. Use two layers:

linkedin_entity
  platform_id
  resource
  canonical_url
  stable public fields
  first_seen_at
  last_seen_at

linkedin_snapshot
  platform_id
  collected_at
  visible metrics
  mutable public fields
  source_task_id

Upsert the entity by resource + id when an ID exists. Otherwise use a canonical URL fallback and keep a merge path for later ID discovery. Append snapshots only after a successful collection and schema validation.

For jobs, keep first_seen_at, last_seen_at, and a locally inferred availability state. A failed refresh does not prove that a job closed.

Validate Before Loading into Production

At minimum, verify:

• platform equals linkedin.
• resource matches the endpoint used.
• Returned URLs use an expected public LinkedIn pattern.
• IDs and URLs do not collide across incompatible resource types.
• Counts are non-negative when present.
• created_at and collected_at parse as timestamps.
• Arrays and nested objects have the expected type.
• Every record can be traced to an input and task.

Quarantine invalid records rather than silently coercing them. Schema drift is easier to diagnose when you retain the raw response, validation error, provider version, and collection time.

Choose a Refresh Schedule by Entity

Do not scrape every entity at the same frequency:

Use business need, consent, contractual limits, and data minimization to reduce unnecessary collection.

Responsible LinkedIn Data Use

Collect only public fields needed for a legitimate purpose. Do not bypass authentication, private profiles, access controls, rate limits, or technical restrictions. Avoid sensitive inference and automated decisions that could materially affect a person without appropriate safeguards.

LinkedIn data can be personal data. Establish a lawful basis where required, define retention and deletion procedures, secure API keys and stored records, honor applicable rights, and review LinkedIn's current terms and local law with qualified counsel.

Common Failure Modes

The endpoint returns no item

Confirm the URL is public, canonical, still available, and routed to the correct endpoint. Preserve the input-level error rather than turning it into an empty record.

A field disappeared

Public visibility and page rendering change. Keep fields nullable and compare collection timestamps before diagnosing parser drift.

The same person appears twice

Canonicalize URLs and prefer a stable public platform ID. Maintain an alias table instead of deleting one record without evidence.

Metrics do not match the browser

Counts are collection-time snapshots and may be rounded, hidden, cached, or updated between requests. Compare timestamps and field definitions.

FAQ

Is scraping LinkedIn legal?

There is no universal answer. It depends on jurisdiction, data, access method, purpose, contract, and safeguards. This tutorial is not legal advice.

Can the API search all LinkedIn profiles or jobs?

The current four SocQ endpoints are URL-based collection endpoints. They do not promise exhaustive LinkedIn search.

Can I scrape private profiles?

No. Do not bypass privacy or access controls.

Should I store the raw response?

Yes, with appropriate security and retention. It helps audit transformations and diagnose schema changes.

Which key should I use for deduplication?

Prefer resource + id; fall back to a normalized canonical URL when the public ID is unavailable.