SEOFuxx Bots
SEOFuxx uses two bots: the backlink crawler collects public links for a link index. The analyzer fetches pages and sitemaps for account holders. Find out how to identify and control both.
Identifiable bot name
seofuxx-crawler
Full user agent
Mozilla/5.0 (compatible; seofuxx-crawler/1.2; +https://www.seofuxx.com/seofuxx-crawler)
Backlink crawler (seofuxx-crawler)
Current crawl scope
Minimum interval per host
Page limit per host and run
Permanently stored page content
Backlink crawler (seofuxx-crawler)
What the crawler does
SEOFuxx collects the data for its backlink index itself. The crawler visits websites, extracts links, and turns them into a searchable link graph.
Fetch public pages
The crawler visits publicly accessible HTML pages within the approved crawl scope. The first stage focuses on .de websites.
Extract links
Internal and external links, link attributes, and anchor text are extracted. New internal URLs may be added to the crawl queue.
Build a link graph
The discovered connections become part of our own backlink index, making referring pages and changes over time visible.
Backlink crawler (seofuxx-crawler)
Considerate and controllable
The crawler is designed to minimize load on individual websites and keep site owners in control.
robots.txt is respected
Rules for seofuxx-crawler and general rules are evaluated according to RFC 9309. Blocked paths are not visited.
Interval between requests
Requests to the same host are sequential and at least two seconds apart. A longer Crawl-delay takes precedence.
Fixed technical limits
Page count, response size, and request duration are limited. A global rate limit also caps the overall load.
Cautious error handling
If robots.txt cannot be fetched reliably because of a server error, the crawler treats the host as blocked as a precaution.
Backlink crawler (seofuxx-crawler)
What we store
We only store data required for the link graph and technical crawl management.
- Source and target URLs of discovered links
- Anchor text and link attributes such as rel
- Technical metadata such as status and fetch time
What we do not store permanently
Page content is processed in memory only to extract links and technical metadata.
- No permanent copy of HTML content
- No images, videos, or other page assets
- No content from blocked or non-public areas
Analysis requests (seofuxx-analyzer)
The analyzer checks pages, robots.txt and sitemaps on behalf of a SEOFuxx account holder. This includes audits, sitemap checks and sitemap crawls. It does not build a link index.
Full user agent
Mozilla/5.0 (compatible; seofuxx-analyzer/1.0; +https://www.seofuxx.com/seofuxx-crawler)
Page fetches and, currently, sitemap crawls use a browser identifier because some hosts block bot requests. The identifier above applies to other analysis requests.
robots.txt group for the bot identifier
For requests using the bot identifier, the seofuxx-analyzer or seofuxx group applies, otherwise *. These lines show how to exclude those requests. Analysis requests do not consistently check robots.txt themselves; page and sitemap requests using a browser identifier are outside this group.
User-agent: seofuxx-analyzer
Disallow: /Sitemap crawl behavior
A sitemap crawl fetches up to three URLs at once and pauses for 100 ms after each batch. On HTTP 429 or 503, it honors Retry-After and retries affected URLs once. These values do not apply to the backlink crawler.
Backlink crawler (seofuxx-crawler)
Control access with robots.txt
You can block individual paths or your entire website for the SEOFuxx Crawler. Add the lines shown to your robots.txt for a complete block.
robots.txt changes are honored on the next visit. To block a specific area, replace / with the relevant path.
Complete block
User-agent: seofuxx-crawler
Disallow: /Questions or an issue with one of our bots?
Send us the affected domain and a short description. We will review the activity and help you directly.
Is this content helpful?