There are several causes you might need to have to search out many of the URLs on a web site, but your correct objective will figure out Everything you’re trying to find. As an example, you might want to:
Establish every single indexed URL to analyze challenges like cannibalization or index bloat
Acquire present and historic URLs Google has seen, specifically for web site migrations
Uncover all 404 URLs to Get well from article-migration glitches
In Each individual circumstance, an individual Instrument received’t Supply you with anything you would like. Sad to say, Google Search Console isn’t exhaustive, plus a “site:example.com” search is limited and hard to extract knowledge from.
With this article, I’ll stroll you through some applications to build your URL record and right before deduplicating the information employing a spreadsheet or Jupyter Notebook, determined by your website’s dimensions.
Aged sitemaps and crawl exports
In case you’re searching for URLs that disappeared through the Stay site not too long ago, there’s a chance somebody in your team may have saved a sitemap file or a crawl export ahead of the alterations had been built. When you haven’t presently, check for these documents; they are able to generally provide what you need. But, in case you’re looking at this, you almost certainly did not get so lucky.
Archive.org
Archive.org
Archive.org is a useful Device for Search engine optimization responsibilities, funded by donations. For those who seek for a domain and select the “URLs” possibility, you may access approximately 10,000 stated URLs.
Nevertheless, There are many constraints:
URL limit: You may only retrieve as many as web designer kuala lumpur 10,000 URLs, that's insufficient for larger web pages.
Excellent: Several URLs may be malformed or reference source documents (e.g., pictures or scripts).
No export option: There isn’t a created-in way to export the list.
To bypass the lack of the export button, make use of a browser scraping plugin like Dataminer.io. Having said that, these limits necessarily mean Archive.org might not supply an entire Option for more substantial web sites. Also, Archive.org doesn’t indicate whether or not Google indexed a URL—but when Archive.org identified it, there’s a great probability Google did, too.
Moz Professional
Whilst you could possibly ordinarily make use of a backlink index to uncover external websites linking to you personally, these instruments also learn URLs on your site in the process.
How to utilize it:
Export your inbound hyperlinks in Moz Professional to get a swift and straightforward listing of concentrate on URLs from a site. Should you’re working with a huge Web page, consider using the Moz API to export information beyond what’s workable in Excel or Google Sheets.
It’s essential to Be aware that Moz Professional doesn’t verify if URLs are indexed or found out by Google. On the other hand, considering that most internet sites utilize precisely the same robots.txt procedures to Moz’s bots since they do to Google’s, this technique frequently works nicely as a proxy for Googlebot’s discoverability.
Google Look for Console
Google Lookup Console provides several important sources for setting up your list of URLs.
Inbound links reports:
Much like Moz Pro, the One-way links segment supplies exportable lists of focus on URLs. However, these exports are capped at 1,000 URLs Just about every. You could utilize filters for precise webpages, but considering that filters don’t apply into the export, you would possibly ought to depend on browser scraping instruments—restricted to five hundred filtered URLs at any given time. Not perfect.
Performance → Search engine results:
This export offers you a summary of webpages receiving look for impressions. While the export is proscribed, you can use Google Look for Console API for much larger datasets. In addition there are free of charge Google Sheets plugins that simplify pulling much more considerable information.
Indexing → Web pages report:
This portion provides exports filtered by situation form, nevertheless they're also constrained in scope.
Google Analytics
Google Analytics
The Engagement → Webpages and Screens default report in GA4 is an excellent resource for gathering URLs, having a generous Restrict of 100,000 URLs.
Better yet, you'll be able to utilize filters to produce different URL lists, proficiently surpassing the 100k Restrict. For instance, if you would like export only blog URLs, adhere to these measures:
Phase 1: Increase a phase to the report
Move 2: Simply click “Produce a new phase.”
Phase three: Outline the phase by using a narrower URL sample, for example URLs made up of /website/
Take note: URLs found in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide precious insights.
Server log data files
Server or CDN log data files are perhaps the ultimate tool at your disposal. These logs seize an exhaustive listing of every URL route queried by people, Googlebot, or other bots over the recorded period of time.
Factors:
Information sizing: Log information is usually substantial, so many web-sites only keep the final two months of knowledge.
Complexity: Examining log files might be complicated, but various tools are available to simplify the procedure.
Blend, and very good luck
When you finally’ve gathered URLs from all these sources, it’s time to mix them. If your website is sufficiently small, use Excel or, for bigger datasets, applications like Google Sheets or Jupyter Notebook. Make certain all URLs are consistently formatted, then deduplicate the list.
And voilà—you now have a comprehensive list of present, previous, and archived URLs. Excellent luck!