The SEOFUXX Crawler
Our crawler collects publicly accessible links to build an independent backlink index of the German web. Learn how it works, what data it stores, and how you can control its access.
Identifiable bot name
seofuxx-crawler
Full user agent
Mozilla/5.0 (compatible; seofuxx-crawler/1.2; +https://www.seofuxx.com/seofuxx-crawler)
Current crawl scope
Minimum interval per host
Page limit per host and run
Permanently stored page content
What the crawler does
SEOFUXX collects the data for its backlink index itself. The crawler visits websites, extracts links, and turns them into a searchable link graph.
Fetch public pages
The crawler visits publicly accessible HTML pages within the approved crawl scope. The first stage focuses on .de websites.
Extract links
Internal and external links, link attributes, and anchor text are extracted. New internal URLs may be added to the crawl queue.
Build a link graph
The discovered connections become part of our own backlink index, making referring pages and changes over time visible.
Considerate and controllable
The crawler is designed to minimize load on individual websites and keep site owners in control.
robots.txt is respected
Rules for seofuxx-crawler and general rules are evaluated according to RFC 9309. Blocked paths are not visited.
Interval between requests
Requests to the same host are sequential and at least two seconds apart. A longer Crawl-delay takes precedence.
Fixed technical limits
Page count, response size, and request duration are limited. A global rate limit also caps the overall load.
Cautious error handling
If robots.txt cannot be fetched reliably because of a server error, the crawler treats the host as blocked as a precaution.
What we store
We only store data required for the link graph and technical crawl management.
- Source and target URLs of discovered links
- Anchor text and link attributes such as rel
- Technical metadata such as status and fetch time
What we do not store permanently
Page content is processed in memory only to extract links and technical metadata.
- No permanent copy of HTML content
- No images, videos, or other page assets
- No content from blocked or non-public areas
Control access with robots.txt
You can block individual paths or your entire website for the SEOFUXX Crawler. Add the lines shown to your robots.txt for a complete block.
robots.txt changes are honored on the next visit. To block a specific area, replace / with the relevant path.
Complete block
User-agent: seofuxx-crawler
Disallow: /Questions or an issue with our crawler?
Send us the affected domain and a short description. We will review the activity and help you directly.
Is this content helpful?