How to Scrape Twitter/X Community Posts with an API

Collect Twitter/X community discussions with SocQ and keep post text, authors, publication times and sources together. Turn recurring questions into topic ideas, with original posts attached to each finding for later review.

This tutorial uses SocQ X Community Posts to collect posts and X Community to query the description and rules. The examples produce source-preserving JSON and a community-based CSV export.

Quick answer: Submit x.com/i/communities/{id} links to POST /v1/x/community-posts, save the task ID, and read all result pages after success. Query metadata through POST /v1/x/community using the same link. Attach the input community to each post to connect metadata and discussions.

Use a Public Community URL as Input

Use a URL in this shape, replacing it with the public community you want to research:

https://x.com/i/communities/1493446837214187523

The numeric segment is the community ID. Both community endpoints take a complete /i/communities/{numeric_id} URL.

If you only know the community name, find its public URL first and submit the link in urls.

Save the submitted URL, numeric community ID, publication time (published_at) and collection time (collected_at). Match x.com and supported twitter.com variants by community ID, and keep communities with different IDs separate even when their names match.

Collect Posts from the Twitter/X Community Feed

Set SOCQ_API_KEY and submit a task capped at 40 posts per community:

curl -X POST "https://api.socq.ai/v1/x/community-posts" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://x.com/i/communities/1493446837214187523"],
    "results_limit": 40
  }'

The response provides data.task_id. Save it, then query the existing task:

curl "https://api.socq.ai/v1/tasks/$TASK_ID?limit=100" \
  -H "Authorization: Bearer $SOCQ_API_KEY"

Wait between requests while the task is queued, running or retrying. A failed task exposes data.error_message. Once it is succeeded, the post records are in data.results.items.

Community Posts returns records in creation-time order. For a report sorted by likes or comments, sort the collected records locally and label the output as a sample ranking.

Understand Per-Community Limits

results_limit defaults to 100 and accepts multiples of 20 from 20 to 2000, such as 20, 40, 60 and 100.

The cap applies to each input community. Two URLs with results_limit=40 request up to 40 posts each; the saved deduplicated results determine the actual count.

The task-query limit sets how many saved records to read per page, and next_cursor identifies the next page. The submitted results_limit sets the collection cap.

Know Which Community Data You Are Getting

The metadata request is separate:

curl -X POST "https://api.socq.ai/v1/x/community" \
  -H "Authorization: Bearer $SOCQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://x.com/i/communities/1493446837214187523"]}'

Retrieve metadata through its own task ID. Member and moderator totals describe community scale, while a post's view count records views of that post. Active-author counts and unique reach require their own measurements.

Null means the source omitted a measurement; 0 is a returned count of zero. Preserve both values distinctly in your export.

Export Community Posts to JSON and CSV with Python

The following workflow uses Python's standard library. Save this task helper as socq_collect.py:

import json
import os
import time
from urllib.parse import urlencode
from urllib.request import Request, urlopen

BASE = "https://api.socq.ai/v1"
PENDING = {"queued", "running", "retrying"}

def api(method, path, payload=None):
    body = json.dumps(payload).encode("utf-8") if payload is not None else None
    request = Request(
        BASE + path, data=body, method=method,
        headers={
            "Authorization": "Bearer " + os.environ["SOCQ_API_KEY"],
            "Content-Type": "application/json",
        },
    )
    with urlopen(request, timeout=60) as response:
        result = json.load(response)
    if result.get("code") != 200:
        raise RuntimeError(result)
    return result["data"]

def collect(endpoint, payload):
    task_id = api("POST", "/" + endpoint, payload)["task_id"]
    print("Task:", task_id, flush=True)
    path = "/tasks/" + task_id
    deadline = time.monotonic() + 600
    data = api("GET", path + "?limit=100")
    while data["status"] in PENDING:
        if time.monotonic() >= deadline:
            raise TimeoutError("Resume GET " + path + "; do not resubmit automatically")
        time.sleep(2)
        data = api("GET", path + "?limit=100")
    if data["status"] != "succeeded":
        raise RuntimeError(data.get("error_message") or data["status"])

    items, seen_cursors = [], set()
    while True:
        if time.monotonic() >= deadline:
            raise TimeoutError("Resume pagination for " + task_id)
        page = data["results"]
        items.extend(page["items"])
        if not page["has_more"]:
            return task_id, items
        cursor = page.get("next_cursor")
        if not cursor or cursor in seen_cursors:
            raise RuntimeError("Missing or repeated result cursor")
        seen_cursors.add(cursor)
        data = api("GET", path + "?" + urlencode({"limit": 100, "cursor": cursor}))
        if data["status"] != "succeeded":
            raise RuntimeError("Task is not ready for result pagination")

The helper prints each task ID, waits up to ten minutes and reads every saved result page. After a client timeout, retain the ID and resume querying that task.

Next, save export_communities.py beside it:

import csv
import json
import sys
from urllib.parse import urlparse
from socq_collect import collect

urls = list(dict.fromkeys(sys.argv[1:]))
if not urls:
    raise SystemExit("Usage: python3 export_communities.py COMMUNITY_URL [COMMUNITY_URL...]")

def cell(value):
    text = "" if value is None else str(value)
    return "'" + text if text.lstrip().startswith(("=", "+", "-", "@")) else text

fields = ["community_id", "community_url", "post_id", "post_url", "text",
          "author_id", "author_username", "published_at", "collected_at",
          "likes_count", "comments_count", "shares_count", "views_count"]
bundles, seen = [], set()
with open("community-posts.csv", "w", newline="", encoding="utf-8-sig") as output:
    writer = csv.DictWriter(output, fieldnames=fields)
    writer.writeheader()
    for url in urls:
        metadata_task, communities = collect("x/community", {"urls": [url]})
        posts_task, posts = collect("x/community-posts", {
            "urls": [url], "results_limit": 40,
        })
        bundles.append({
            "input_url": url, "metadata_task_id": metadata_task,
            "posts_task_id": posts_task, "communities": communities, "posts": posts,
        })
        # Save completed collections before processing another community.
        with open("community-results.json", "w", encoding="utf-8") as saved:
            json.dump(bundles, saved, ensure_ascii=False, indent=2)
        community_id = urlparse(url).path.rstrip("/").split("/")[-1]
        for post in posts:
            post_id = post.get("id")
            key = (community_id, post_id)
            if post_id and key in seen:
                continue
            if post_id:
                seen.add(key)
            author = post.get("author") or {}
            metrics = post.get("metrics") or {}
            row = {
                "community_id": community_id, "community_url": url,
                "post_id": post_id, "post_url": post.get("url"), "text": post.get("text"),
                "author_id": author.get("id"), "author_username": author.get("username"),
                "published_at": post.get("published_at"),
                "collected_at": post.get("collected_at"),
                **{name: metrics.get(name) for name in
                   ("likes_count", "comments_count", "shares_count", "views_count")},
            }
            writer.writerow({key: cell(value) for key, value in row.items()})

Run it with one or more public community URLs:

python3 export_communities.py \
  "https://x.com/i/communities/1493446837214187523"

You get two outputs:

• community-results.json preserves each input URL, both task IDs, community metadata and post records.
• community-posts.csv provides one row per returned post within each community, with source, author, timestamps and available metrics.

The example creates separate metadata and post tasks for each community, attaching the input community to its posts. This makes each record's collection source explicit. Use a batch for combined retrieval, or the single-community task pattern here when you need per-post source attribution.

Deduplication uses community_id + post_id. It merges repeated rows within a community and keeps observations of the same post from different communities separate. Records missing a post ID are retained individually.

The program updates the files after each community completes. If a later request fails, the files retain the previously completed communities. Before using the output in a report, check completed inputs and recorded errors.

Keep JSON as the Source for Spreadsheet Exports

CSV is convenient for reviewers, but it flattens the data. Nested rules, media and author fields remain available in JSON. Regenerate a different spreadsheet from that saved JSON when your columns change.

The example uses Python's standard-library CSV writer to handle quotes, commas and multiline text. UTF-8 with a byte-order mark helps common spreadsheet imports. Formula-like public text is escaped in CSV; JSON preserves the original content. Import long numeric IDs as text to avoid spreadsheet precision loss.

Use Community Discussions for Topic Research

For example, ask which onboarding questions recur in these collected discussions. Group them by question and retain the supporting post IDs for each category.

Read the community's description and rules first. Then review post text, publication time and source links together. Group recurring questions, product complaints or resource requests, and keep the supporting post IDs next to your labels.

Use the community-details workflow to save descriptions and rules, then run topic classification or summarization in your application.

Label a local likes or comments sort as a "ranking within this capture," with its collection time and record count. Mark a post absent from a later capture as "not observed in this capture"; query that post separately to investigate deletion.

Look Inside an Individual Discussion

To expand one discussion, submit the returned post URL to X Thread Context and read the target, earlier reply-chain posts and reply relationships.

For that next step, read the Twitter thread context tutorial. If you only need replies to a known post, use X Post Replies. If your starting point is an account instead of a community, use X User Posts.

Keep Original Posts Behind Your Content Decisions

SocQ community metadata and post endpoints connect community identification, discussion collection and topic review. The code retains each post's community source and exports JSON and CSV, so content teams can review questions, organize examples and prepare inputs for summaries.

Open Community Posts and collect your target community.

Sources and Further Reading

• SocQ Community Posts documentation — accepted links, per-community limits and standard post fields.
• SocQ Community documentation — metadata, rules and counts.

FAQ

Can I collect posts from multiple X communities?

Yes. Submit several community URLs in urls. The result cap applies to each URL. The Python export example uses separate tasks to preserve a clear input-community source for every post.

Does this return every post ever published in a community?

A task reads a selection of posts from the specified community, capped by results_limit. Its returned records define that capture's scope. For historical analysis, save separate captures and report the scope of each.

Can I export community members as well as posts?

Community returns metadata and member and moderator counts; Community Posts returns posts. Individual member directories are separate data and are not returned by these endpoints.

How is a community feed different from a user's timeline?

A community feed groups posts in a discussion space; a user timeline groups posts from an account. Use community links for Community Posts and usernames for User Posts.

Can I choose Top or Likes sorting?

The endpoint returns creation-time ordering. Sort the sample by likes or comments in your application to produce a metric ranking for that capture.

Does pagination collect new posts?

The cursor reads the next page saved by the original task. Submit a new task for a fresh observation, then handle overlapping records by post ID.

How do I collect posts after the community migration announcement?

X published a Communities-to-XChat migration announcement. Submit the target community URL to Community Posts and read the returned posts after success. Record empty results and failures separately, and keep collection times to distinguish saved history from each new query.