Distributed Task Queue
Push scraping tasks into named groups. Approved executors pull the next PENDING task with row-level SKIP LOCKED guarantees — no duplicates, no double work.
Distributed Task Scraping Platform
Tskgone orchestrates scraping across executors, schedules periodic crawls, and exposes a signed REST API so agents and pipelines can push tasks and pull structured data — without reinventing a job queue.
A job queue built for scraping
Tskgone ships the primitives every scraping team rebuilds — task groups, executor pools, progress heartbeats, retries, and typed schemas — as a single platform.
Periodic Scheduling
Declare interval-based crawls with PERIODIC task groups. Tskgone tracks lastExecutedAt, archives execution history, and re-queues on the next interval.
Signed REST API
Every API call is signed with HMAC-SHA256. Rotate project credentials on demand. Use OAuth 2.0, MCP, or project-level API keys — whichever your workflow needs.
Heartbeat Timeouts
Executors post progress heartbeats to extend their lease. Tasks that go silent are auto-failed by a background scanner and become retryable.
Typed Task Types
Each task type ships with a JSON schema for input and output. Front-ends and agents get autocomplete; bad payloads are rejected before a worker sees them.
File Uploads & Artifacts
Executors can upload screenshots and JSON blobs with magic-byte validation. Files are scoped to the project and retrievable via signed requests.
How it works
From zero to a running distributed scraper in three steps.
- 1
Create a project
Projects hold your task groups, executors, and generated HMAC credentials. Invite teammates as ADMIN or MEMBER.
- 2
Register executors
Workers register with a UUID and declare the task types they handle. Approve them once; they pull tasks on their own after that.
- 3
Push and pull tasks
POST tasks to the API, get outputs back via signed GET. Or let PERIODIC groups re-queue themselves on a cadence.
Frequently asked questions
What is Tskgone?
Tskgone is a distributed task scraping platform. You define scraping task groups, executors pull PENDING tasks with HMAC-authenticated calls, and Tskgone tracks progress, timeouts, and execution history.
How is Tskgone different from a generic job queue?
Tskgone is scraping-aware. It ships with typed task types (Reddit post details, subreddit posts, search, screenshots, user details), a PERIODIC group mode that re-runs crawls on a cadence, and per-task-type timeout tuning.
How do executors authenticate?
Every executor request is signed with X-API-Key, X-Timestamp, and X-Signature (HMAC-SHA256 of method + path + timestamp + body). A 5-minute clock skew window prevents replay attacks.
Does Tskgone support OAuth or MCP?
Yes. The /api/mcp endpoint exposes every project, task group, and task tool to MCP-compatible clients, authenticated via user API keys or OAuth 2.0 access tokens with per-scope gating.
What task types are available today?
Reddit post details, post screenshots, post visits, subreddit details, subreddit post lists, user details, generic fetch, and search. More platforms (X/Twitter and beyond) are on the roadmap.