How to Search and Collect X/Twitter Posts with an API

X still looks like a timeline in the browser. The public-data contract is narrower: a username is not a post URL, a quote is not a retweet, a reply is not a quote, and a deleted or protected post should fail cleanly instead of becoming an empty record.

This tutorial builds a production workflow with the SocQ X APIs. It covers official versus public access, endpoint routing, input hygiene, asynchronous tasks, field meaning, refresh cadence, and responsible use.

Quick answer: Send usernames, x.com or twitter.com status URLs, or a search query to POST /v1/x/{resource}, store the task ID, and poll /v1/tasks/{id}. Do not collapse quotes, retweets, and replies into one table without a resource discriminator.

Official X API or a Public Data API?

The official X API is paid and permissioned. It is the correct product when you need to publish, send Direct Messages, manage ads, follow or like as a user, or consume streams and other account-authorized surfaces. Access and prices change; treat current developer documentation as the source of truth for write and partner products.

SocQ is a public-read layer for supported profiles, known posts, user post windows, search, conversation edges, and trends. It does not authenticate as an X user and does not replace official posting or ads. For pricing and vendor framing, read Twitter API alternatives, Twitter scraper APIs, and TwitterAPI.io alternatives. Choose the access model before choosing an endpoint.

Choose the Correct X Endpoint

Also available:

• post-retweeters for public status URLs.
• followers-list and following-list for usernames, with results_limit from 200 through 2000 in steps of 200.
• trends for numeric WOEIDs.

Quotes, retweets, and replies are different resources. A quote has its own post text. A retweeter list is a set of profiles. A reply is a post that belongs to a conversation. Do not upsert them as if they were the same object.

Normalize Usernames, URLs, and Queries

Keep the submitted value, then clean a copy:

1. Accept usernames with or without @, then store one canonical username.
2. Accept both x.com and twitter.com hosts. Require a numeric /status/{id} path for post-shaped endpoints.
3. Drop tracking query parameters. Do not require a specific mobile or locale host if the status ID is intact.
4. Deduplicate usernames case-insensitively and posts by status ID.
5. Reject empty queries and usernames before submission.
6. Keep search expressions in the provenance row so analysts can explain why a post entered the dataset.

user-posts and search accept results_limit from 20 through 2000 in steps of 20. Search sort_by is latest or top. post-replies also accepts sort_by of relevance, latest, or likes.

Submit a Collection Task

Collect profiles:

curl -X POST "https://api.socq.ai/v1/x/profiles" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"usernames":["openai","@github"]}'

Collect known posts from either hostname:

curl -X POST "https://api.socq.ai/v1/x/posts" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://x.com/X/status/2073395918011789670"]}'

Collect a user post window:

curl -X POST "https://api.socq.ai/v1/x/user-posts" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"usernames":["openai","github"],"results_limit":40}'

Search public posts:

curl -X POST "https://api.socq.ai/v1/x/search" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"AI lang:en","results_limit":100,"sort_by":"latest"}'

Collect replies and quotes as separate jobs:

curl -X POST "https://api.socq.ai/v1/x/post-replies" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://x.com/X/status/2073395918011789670"],"results_limit":20,"sort_by":"latest"}'

curl -X POST "https://api.socq.ai/v1/x/post-quotes" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://x.com/X/status/2073395918011789670"],"results_limit":20}'

Persist the returned task ID with the resource name and canonical inputs.

Poll the Task without Creating Duplicates

curl "https://api.socq.ai/v1/tasks/$TASK_ID?limit=100" \
  -H "Authorization: Bearer $SOCQ_API_KEY"

Treat queued and running as normal. Use exponential backoff. When the task succeeds, read data.results.items and follow next_cursor while has_more is true.

A robust worker should persist:

• Batch ID and SocQ task ID.
• Resource and canonical usernames, URLs, or query.
• Submission and completion times.
• Terminal status and error category.
• Last processed result cursor.

A worker restart is not a reason to submit the same search again.

Understand the Returned Fields

Records share id, platform, resource, type, url, created_at, collected_at, and extra.

A profile includes username, name, description, avatar_url, cover_url, is_verified, is_private, and visible metrics. Protected accounts are not a public-data target.

A post includes text, language, author, visible metrics, media, hashtags, and mentions when public. Use id as the preferred entity key and the status URL as provenance.

Replies and quotes reuse the post shape. Retweeters reuse the profile shape. Trends are location-scoped discovery rows, not posts.

Deleted and private posts should fail at the input or task boundary. Do not insert a blank post, invent engagement zeros, or reuse collected_at as created_at.

Search results are ranked discovery. latest and top are different orderings of a bounded window, not a complete historical archive.

Python Worker for Profiles, Posts, and Search

import time
import requests

BASE = "https://api.socq.ai/v1"
HEADERS = {
    "Authorization": f"Bearer {SOCQ_API_KEY}",
    "Content-Type": "application/json",
}

def collect(resource, payload):
    created = requests.post(
        f"{BASE}/x/{resource}",
        headers=HEADERS,
        json=payload,
        timeout=30,
    )
    created.raise_for_status()
    task_id = created.json()["data"]["task_id"]

    delay = 2
    while True:
        response = requests.get(
            f"{BASE}/tasks/{task_id}",
            headers=HEADERS,
            timeout=30,
        )
        response.raise_for_status()
        task = response.json()["data"]

        if task["status"] == "succeeded":
            return task["results"]["items"]
        if task["status"] in {"failed", "cancelled"}:
            raise RuntimeError(task)

        time.sleep(delay)
        delay = min(delay * 1.5, 15)

profiles = collect("profiles", {"usernames": ["openai"]})
hits = collect("search", {
    "query": "AI lang:en",
    "results_limit": 40,
    "sort_by": "latest",
})

Add cursor pagination, bounded retries for timeouts and retriable 5xx responses, and idempotent upserts before this reaches production.

Model Posts and Metric Snapshots Separately

Post identity and post metrics change at different rates.

x_entity
  resource
  platform_id
  canonical_url_or_username
  text_or_description
  author_id
  created_at
  first_seen_at
  last_seen_at

x_snapshot
  resource
  platform_id
  collected_at
  likes_count
  replies_count
  reposts_count
  quotes_count
  source_task_id

Upsert by resource + id. Store quotes, replies, and source posts in the same post table only if resource remains a required column. Keep retweeters in a profile or edge table.

Do not compute quote growth from a retweeter snapshot. The metrics are not interchangeable.

Validate Before Storage

• platform must equal x.
• resource must match the endpoint.
• Status URLs must contain a numeric /status/ ID when the resource is post-shaped.
• Author usernames should match the submitted account when the resource is user-posts and the field is present.
• Counts must be non-negative numbers or null.
• Timestamps must parse.
• A failed private or deleted input must not become a successful empty post.

Keep invalid rows in quarantine with the raw task payload.

Choose a Refresh Schedule by Entity

Reduce collection when the business question does not need a fresher snapshot.

Common Failure Modes

A post URL returns nothing

Confirm the URL is public, still available, and contains /status/{id}. Deleted and protected posts should fail cleanly. Preserve the input-level error.

Search looks incomplete

results_limit caps the request. Ranking, indexing, and the latest versus top switch all change the page. Store the query and sort with the records.

Quotes were stored as retweets

They are different endpoints and different objects. Keep resource in the primary key.

Follower jobs reject the limit

Followers and following accept 200 through 2000 in increments of 200. User-posts and search use steps of 20.

Metrics disagree with the browser

Public counts are snapshots. They may be delayed, rounded, or hidden. Compare collected_at before blaming the parser.

The same user appears twice

Canonicalize usernames and prefer a stable public ID. Keep an alias table when an account renames.

Responsible X Data Use

Collect only public fields needed for a legitimate purpose. Do not bypass protected accounts, blocks, login walls, or other access controls. Avoid sensitive individual profiling and automated decisions that could materially affect a person without safeguards.

X data can be personal data. Establish a lawful basis where required, set retention and deletion rules, protect API keys and exports, and review X's current terms with qualified counsel. A public-data API does not grant a license to republish posts or media.

FAQ

Is scraping X or Twitter legal?

There is no universal answer. It depends on jurisdiction, data, method, purpose, contract, and safeguards. This tutorial is not legal advice.

Does this replace the official X API?

No. Official access remains necessary for publishing, DMs, ads, streams, and other permissioned actions. SocQ covers supported public reads.

Can I collect private or deleted posts?

No. Those inputs should fail cleanly. Do not bypass protection.

Are quotes the same as retweets?

No. Quotes are posts. Retweeters are profiles. Replies are a third resource.

Does search return a complete history?

No. It returns a bounded, ordered window for one query.

Which key should I use for deduplication?

Prefer resource + id. Fall back to a normalized status URL or username when the public ID is missing.