Bulk URL checking is essential when auditing sites with hundreds or thousands of pages. Manually checking every address is not feasible. This guide covers use cases, checking methods, and how to interpret the results.
Why bulk checking is needed
Auditing after a migration
After moving a site to a new domain or changing its CMS, you need to make sure all important URLs return 200 or a valid 301. 404 errors, broken redirects, and redirect chains can be identified through bulk checking.
Monitoring broken links
Internal and external links “break” over time: pages are deleted and URLs change. Regular checking identifies 404 before they accumulate. Read more about the impact of broken links in the article on broken links and SEO.
Checking redirects
Redirect chains (A→B→C→D) slow down loading and can lose link equity. Bulk checking shows the final URL and the length of the chain. See the recommendations in the redirect guide.
Validating a sitemap
URLs in a sitemap should return 200. Checking identifies pages that were deleted without updating the sitemap, as well as pages with errors.
Checking after bulk changes
Changes to the catalog structure, URL slug changes, and page consolidation should be followed by a bulk check to confirm everything is working correctly.
What to check
HTTP status code
The primary indicator. 200 means the page is available. 301/302 means a redirect. 404 means the page was not found. 500 means a server error. See the code reference in the HTTP status code guide.
Final URL after redirects
With 301/302, it is important to know where the redirect leads. The goal is one redirect to the target URL, with no chains.
Response time
Slow pages (> 3 seconds) affect UX and Core Web Vitals. Bulk checking can include a TTFB measurement.
Content availability
A page with a 200 code may be empty (soft 404). Checking the response size or for key elements can identify these cases.
Sources of URLs to check
Sitemap
An XML sitemap contains a list of URLs for indexing. Importing from a sitemap is a standard way to get a list of URLs to check.
Site crawl
A crawler follows links across the site and collects URLs. It shows the actual structure, including pages that are not in the sitemap.
List from a file
A CSV or TXT file containing URLs is a manually prepared list of pages to check. This is useful for checking a subset, such as product pages only.
Google Search Console
Export URLs from the “Coverage” or “Pages” reports to check what Google sees.
Internal links
Extract URLs from the site's internal links. This identifies broken internal links.
Bulk checking tools
reChecker Bulk Checker
Bulk URL checking on reChecker lets you upload a list of URLs (from a sitemap, a file, or an input field) and get status codes, final URLs after redirects, and response times. Results can be exported for analysis.
reChecker Broken Links
The reChecker broken link checker crawls pages and checks all links (internal and external). It identifies 404, redirects, and timeouts.
Command line (curl + xargs)
For small lists:
# Проверка списка URL из файла urls.txt
while read url; do
code=$(curl -o /dev/null -s -w "%{http_code}" -L "$url")
echo "$url $code"
done < urls.txtScripts (Python, Node.js)
For custom logic, use a script with asyncio/aiohttp (Python) or axios (Node.js) to make parallel requests with a rate limit.
Interpreting the results
Distribution by code
| Code | Share | Action |
|---|---|---|
| 200 | > 90% | Normal |
| 301 | 5–10% | Check target URLs |
| 404 | > 5% | Audit, redirect, or remove from sitemap |
| 500, 502, 503 | Any | Fix urgently |
Redirect chains
If A→B→C→D, shorten it to A→D. Each extra redirect adds latency and risks losing link equity.
Grouping errors
Group 404 by patterns: /old-category/, /product/?id=, /page?utm_*. Patterns can point to systemic causes (a deleted category, incorrect parameters).
How often to check
- After a migration: once, then again after a week
- Regular monitoring: once a month for medium-sized sites, once a week for large sites
- After bulk changes: immediately after implementation
Limitations and ethics
Rate limiting
Do not send thousands of requests per second to one site. This creates load and may trigger a block. Tools that pause between requests (100–500 ms) offer a reasonable compromise.
External sites
When checking external links, follow robots.txt and do not overload other people's servers.
Processing results
Exporting and reporting
Bulk check results are usually exported to CSV or Excel. Group them by status code, domain, and page type. Create a summary table for the report.
Prioritizing fixes
- Critical: 5xx errors, widespread 404 on important pages
- High priority: Redirect chains, 404 on pages with traffic
- Medium: Individual 404, redirects to non-optimal URLs
- Low: Minor inconsistencies
Integrating into the workflow
Include bulk checking in regular audits: after every major update and monthly for monitoring. Automate alerts for critical errors when the tool supports it.
Considerations for different types of sites
Online stores
Thousands of product pages. Sitemap checking is the primary source. Pay attention to 404 after catalog updates and redirects when product URL slugs change.
Informational sites
Fewer pages, but each one matters. Checking internal links identifies broken links in articles. Check regularly after publishing.
Corporate websites
A small number of pages. A complete check takes minutes. Focus on redirects after a redesign or structural changes.
Alternative checking methods
Server logs
Log analysis shows actual requests and response codes. It identifies pages accessed by users and bots that are not in the sitemap.
Google Search Console
The “Coverage” report shows indexed and excluded URLs. It does not replace bulk checking, but complements it by showing exactly what Google sees.
Site crawlers
Desktop crawlers are suitable for crawling an entire site and checking status codes, but they require installation and setup. For a quick check of a list of addresses without installing anything, use the reChecker Bulk Checker.
Bulk checking checklist
| Step | Action |
|---|---|
| 1. Collect URLs | Sitemap, crawl, or list |
| 2. Check | Run the bulk checker |
| 3. Analyze | Group by codes and identify patterns |
| 4. Prioritize | Critical 404, redirect chains |
| 5. Fix | Redirects, sitemap updates, link fixes |
| 6. Recheck | Confirm after making changes |
Input formats
Sitemap XML
Upload a sitemap URL—the tool will extract all URLs and check them. This is useful for a complete check of indexable pages.
CSV/TXT
A list with one URL per line. Suitable for checking a subset, exporting from other tools, or using a manually prepared list.
Paste from clipboard
Paste a list of URLs from the clipboard. Check quickly without uploading a file.
Results and reports
Report structure
Typical fields: URL, status code, final URL (after redirects), response time, and response size. Results can be grouped by code and exported to CSV for further analysis.
Visualization
Some tools display the distribution by code as charts, giving a quick overview of the scale of the problems.
Comparing with previous checks
Compare results across regular checks. New 404 or changes to redirect chains are signals to investigate.
CI/CD integration
To automate checks during deployment, use a script to check key URLs after rollout. If there are 5xx errors or critical 404, roll back or send an alert. Limit the number of URLs to keep the pipeline fast.
Related materials
See the code reference in the HTTP status code guide. Fixing 404 is covered in the 404 guide. The impact of broken links is covered in the article on broken links. Bulk checking and finding broken links are covered in the Bulk Checker and the broken link checker on reChecker.