Training Crawler
GEO Recommendation: Optional to Block
Bytespider
Operated by Widely attributed to ByteDance · Widely attributed to ByteDance · Purpose not officially documented
A web crawler widely attributed to ByteDance. Its precise purpose and use for AI training have not been officially confirmed.
Owner
Widely attributed to ByteDance
Trigger
Automated crawl (reported)
Used for AI Training
Not officially confirmed
robots.txt control
Inconsistent, per third-party reports
Impact if blocked
A robots.txt rule may not reliably stop this crawler, per widespread third-party reports — server-level blocking is commonly recommended instead
How to identify
User-Agent: Bytespider
What it does
Bytespider is widely attributed to ByteDance. Its precise purpose — including whether it is used to gather AI training data — has not been officially confirmed by ByteDance.
Multiple independent third-party sources describe Bytespider as high-volume and, at times, not reliably respecting robots.txt directives — but this comes from third-party observation rather than ByteDance's own documentation, so ELNIQ presents it as reported, not confirmed.
Why this is optional to block
- No official ByteDance documentation confirms this crawler's purpose or AI-training use
- Multiple independent reports describe it as not always respecting robots.txt
- Consider server- or WAF-level blocking if reliable enforcement matters to you
Identification & robots.txt
User-Agent identifier
Bytespider
Verification
No official verification method documented.
Robots.txt guidance — to opt out
User-agent: Bytespider
Disallow: /