One · Data source onboarding
Turning APIs, databases, CSV files and crawlers into billable tools — this is the step where mines get buried.
01Collector left gzip on; parsing exploded the moment the other side enabled compression
- At the time
- Calling their API returned a gzip byte stream; we parsed it as text and got a pile of garbled errors.
- The fix
- The collection layer always sends
Accept-Encoding: identity, never relying on the other side's defaults; add timeouts and retries.
02Scanning on-chain logs from block 0 always fails
- At the time
- Scanning on-chain data from the genesis block ran until timeout with no result.
- The fix
- Scan incrementally from the chain head back a fixed number of blocks, persist the cursor, and a rerun resumes where it stopped.
03402 headers sent, unreadable in the browser
- At the time
- Packet capture showed the payment headers, but in front-end JS they were undefined.
- The fix
- Custom response headers must be explicitly added to
Access-Control-Expose-Headers, otherwise the browser blocks them under its security policy.
04Wrong amount unit, rejected outright by the facilitator
- At the time
- We wrote 0.01 USDC as 1e16; the facilitator only returned "amount mismatch", giving no hint that the unit was the problem.
- The fix
- USDC has 6 decimals: 0.01 = 10000. Generate prices through one shared conversion function — never by hand.
Two · Launch and routing
Syntax passing does not mean the site opens — every pitfall in this section lives outside syntax.
05nginx -t all green, site unreachable from a redirect loop
- At the time
- The 301 on an exact location and try_files' internal redirect nested into each other; the browser reported "too many redirects".
- The fix
- After any change, fire a real request and check the status code and hop count, using
--max-redirs to cap redirects as the check—not syntax alone.
06Admin URL 301 → trailing slash → 403 blank page
- At the time
/admin was 301-redirected to /admin/, which is a real directory with no index file, so it returns a flat 403; only /admin.html gets through.- The fix
- Internal rewrite straight to the file:
rewrite ^/admin/?$ /admin.html last;, with the login protection layer unchanged.
07Using if for regex checks in location clobbered the capture group
- At the time
- After adding an if regex condition to the clean-URL rules,
$1 got reset, and/pricing.html was 301-redirected to the homepage.
- The fix
- Move the if out of that location, or switch to map; never mix if with capture groups inside a location.
08MCP endpoint behind a proxy got buffered, SSE froze
- At the time
- streamable-http runs over SSE, and nginx's default buffering held the stream back, so the client waited forever.
- The fix
proxy_buffering off; + proxy_cache off; + lengthen proxy_read_timeout。
Three · Caching and versions
The nastiest kind: the server is fine, the page keeps erroring.
09Changed CSS / JS, production "looks like it didn't apply" (hit three times)
- At the time
- nginx had set
expires 7d, on static assets, while the page references carried no version number.
- The fix
- Long caching demands a version stamp: bump the version every time an asset changes, and put it in the release checklist.
10Every server log is 200, yet the page says "can't read data"
- At the time
- The browser was running a script cached for seven days, crashed when rendering the new data format, and catch reported it all as "server unreachable".
- The fix
- Admin scripts get their own no-cache; fetching and rendering split into two try blocks; total failure falls back to localStorage.
11Some pages new, some pages old
- At the time
- One CSS file served many pages, and we only bumped the version on a few of them.
- The fix
- Search the site for that asset name and bump versions in bulk — don't hand-pick pages.
12Error messages that point you the wrong way
- At the time
- Render crash, network failure, expired key and timeout all surfaced as one catch with one message; the backend team searched for two days.
- The fix
- Four failures, four messages; 401/403 go straight to a re-login prompt without retry; on failure show cached data first.
Four · Monitoring and operations
Cron jobs and liveness scripts destroy data quietly when written wrong.
13876 hand-edited pages overwritten by one cron run
- At the time
- A statically generated site rebuilds via cron, so hand-edited output is guaranteed to be wiped.
- The fix
- Edit only the generator template → regenerate → publish; add a guard script watching the output's mtime.
14Probing an MCP endpoint with a JSON client marked SSE as "dead"
- At the time
- The liveness script parsed the response body; MCP returns an SSE stream, so a parse failure was read as death.
- The fix
- Judge only by HTTP status code: both 200 and 402 mean alive; never parse the response body.
15Probe field name misspelled, every entry stuck at "fluctuating"
- At the time
- wrote
p.ok when the actual field is p.lastOk, so the check always fell into the error branch.
- The fix
- Check field names against the code that writes to the database, never from memory; before launch, validate the logic with two known data points.
16Daily rollups by incremental addition double the PV each day
- At the time
- "Yesterday's value + today's increment" doubles the moment you rerun.
- The fix
- Recompute the whole day and overwrite, keeping it idempotent; any timestamp stringification must come after all aggregation.
Five · GEO and citability
If AI can't see you, even great data won't sell.
17Default robots.txt blocks every AI crawler, GEO drops to zero
- At the time
- We used the framework's default template, and mainstream AI crawlers were shut out.
- The fix
- Allow 40+ AI crawlers explicitly, one by one, and Disallow private paths such as the admin area separately.
18llms.txt written as marketing copy gets cited by no AI
- At the time
- We filled it with intros and adjectives — AI can't verify any of it, so it counts as unwritten.
- The fix
- Write verifiable facts only: price, network, endpoints, absolute paths, explicit settlement rules.
19External AI mentioned you, but got the rules wrong
- At the time
- AI reported the revenue share as 15% and settlement as T+7, while the "mention rate" metric was green.
- The fix
- Monitoring must track "accuracy of the facts" separately — mentioned but wrong equals not mentioned.
Six · Pricing and settlement
With money, one mistake is hard to win trust back from.
20Priced too low to ever reach the payout threshold in a year
- At the time
- Unit price set too low: monthly volume never crossed the $50 payout line, so the money just sat on the books.
- The fix
- Before listing, do the math: calls × unit price × share, work back to the minimum viable price, then check the payback period.
21Editing the data file at runtime, old in-memory values overwrite it
- At the time
- We edited the data file while the service was running, and the old values were written back before the restart.
- The fix
- Stop the service before touching data; reconcile only against on-chain Transfer events, tolerance ±$0.02.
22Free quota rate-limited by IP is useless against distributed Agent
- At the time
- An Agent comes from a different IP every time, so per-IP limits are no limits at all.
- The fix
- Tie the quota to Agent / key dimensions, add a daily circuit breaker, and treat the free quota as acquisition cost when computing ROI.
These 22 entries became 20 free tools
The tools under every pitfall are free, need no login, and compute only inside your browser. If you can run it yourself, use them; when you can't, we list, promote and settle on your behalf.