Home · Free toolkit · 09

Data source onboarding wizard

Turning a data source into a tool that Agents can purchase per-use essentially involves four steps:Integrate, schedule crawls, split into billable units, and set prices. This wizard walks you through four steps to directly produce a manifest.json(Can be submitted to us for listing) and an integration checklist—each item on this checklist was added after we experienced a production outage.

Free · No login required Locally generated · Content is not uploaded Output · manifest.json + checklist

1What is a data source?

First, clarify where the data comes from.

Credentials are written only in your own configuration. During our collection, we use Accept-Encoding: identity Direct connection—avoids gzip decompression failures.

2Sampling and updates

Determine how often to crawl and how much to crawl per fetch.

Unstable fields requireContent hashDetermine whether it actually changed; otherwise, storing every fetch as new data will cause the snapshot database to explode.

3Split into billable tools

A single data source typically yields 3–5 tools: cheaper ones for traffic acquisition, more expensive ones for deep lookup.

4Pricing and submission

How to set free quotas without losses? Use Free quota configurator Run once.

ResultsWhat this data source can become

Output 1manifest.json (ready for submission to app stores)

0 rows 
—

Output 2Integration checklist

0 rows 
—
We’ve fallen into this pitfall

1 · gzip causes the crawler to receive garbled text.Our own crawler library sends explicitly Accept-Encoding: identity, and require timeout and 3xx redirects; otherwise, enabling compression on the remote side will cause parsing to fail outright.

2 · Scanning on-chain logs from scratch will fail.Public nodes require a token to check archive status; the correct approach isRoll back 2000 blocks from the chain headStart scanning: $400 per segment, 250ms interval between segments to avoid rate limiting; automatic failover across multiple nodes.

3 · A field name in the assertion is misspelled, causing the entire section to be classified as "volatile".We probe it by using p.ok actual field names p.lastOk, resulting in all entries consistently being abnormal. Field names must match exactly against the write-database code.

4 · Collection task failures are silent.The rentgpu data API already includes this. platforms_not_adapted: ["gpushare"] and similar"适配器失效但没人知道"的情况。必须留覆盖率指标并定期看。

5 · Do not store snapshots indefinitely.Each task retains only the 30 most recent snapshots; content hash determines whether content “actually changed”, otherwise storage costs grow faster than revenue.