How to Get All Page URLs From a Website (6 Ways)
Getting every URL on a website is a different job from copying the links off a single page, and the method you pick decides whether you end up with a complete list or a misleading one.
This guide covers six ways to do it, what each one actually returns, and where each falls short. If you only need the links that appear on one page, that is a different task — see how to extract all links from a webpage instead.
Quick comparison
| Method | Complete? | Needs access? | Best for |
|---|---|---|---|
| XML sitemap | Usually, if maintained | No | Fastest first attempt on any site |
site: search |
No, indexed pages only | No | A rough sense of what Google knows |
| Crawler (Screaming Frog) | Yes, for linked pages | No | Auditing a site properly |
| Search Console | Indexed and discovered | Yes, you must own it | Your own site |
| Browser console | One page only | No | A quick grab, no tools |
| Server or CMS export | Yes | Yes | Migrations and rebuilds |
1. Start with the XML sitemap
Most sites publish a sitemap listing the URLs the owner wants indexed. It is the fastest route and needs no tools. Try these in order:
example.com/sitemap.xmlexample.com/sitemap_index.xmlexample.com/robots.txt— look for a line beginningSitemap:, which is where the real location is declared
Large sites usually split their sitemap into an index that points at several child sitemaps: one for posts, one for pages, one for images. Open each child file and collect the URLs from all of them, or you will only capture part of the site.
What it misses: a sitemap reflects what the owner chose to submit, not what exists. Pages deliberately excluded, orphaned pages, and anything marked noindex may be absent. A stale sitemap can also list URLs that now return 404.
Once you have the raw XML, paste it into the extract URLs tool to pull the addresses out of the markup and leave the tags behind.
2. Use the site: search operator
Searching site:example.com in Google returns pages from that domain that are currently indexed. Narrow it further with a path, such as site:example.com/blog/.
What it misses: quite a lot. The operator only shows indexed pages, Google caps how many results it will page through, and the reported result count is an estimate rather than a figure you should quote. Treat it as a sample, not an inventory.
3. Crawl the site with Screaming Frog
A crawler starts at the homepage and follows every internal link, which is the closest you get to how a search engine sees the site. Screaming Frog SEO Spider crawls up to 500 URLs on its free tier, which covers most small sites outright.
Enter the domain, let the crawl finish, then export the Internal HTML report. You get status codes, titles, and canonical tags alongside the URLs, which is why this is the method to use when you are auditing rather than just listing.
What it misses: orphaned pages that nothing links to. A crawler can only find what is reachable by following links, so pair it with the sitemap to catch both sides.
4. Export from Google Search Console
If the site is yours, Search Console is the most honest picture of what Google has actually found. Open Indexing > Pages to see indexed and non-indexed URLs with the reason each one was excluded, or use Performance and add the Pages dimension to export URLs that have earned impressions.
What it misses: pages Google has never discovered, and anything on a site you do not own. The UI export is also capped at 1,000 rows, so use the Search Console API for larger sites.
5. Grab one page’s links from the browser console
For a single page and nothing installed, open DevTools with F12 and run:
[...document.querySelectorAll('a')].map(a => a.href).join('\n')
That returns every link on the page currently loaded, including navigation and footer links.
What it misses: everything not on that one page. This is a per-page trick, not a site-wide one, and on a site rendered by JavaScript it only sees what has already loaded.
6. Export from the CMS or server
With server access, the underlying source is the most complete of all: a database query for published content, or the file listing for a static site. This is the route to use before a migration, when missing a URL means breaking it.
Cleaning the list afterwards
Every method above produces something messier than you want. Sitemaps arrive wrapped in XML, crawler exports carry extra columns, and the same URL often appears several times with different tracking parameters.
A practical order of operations:
- Pull the URLs out of the raw output with the extract URLs tool, which ignores surrounding markup and text.
- Strip tracking parameters using remove UTM parameters, so the same page stops appearing as several different URLs.
- Remove duplicates with remove duplicate URLs.
- Sort the list with sort URLs so related paths sit together and gaps are visible.
- Reduce to domains with trim URLs to root if you are working across many sites rather than within one.
Once the list is clean, open it in batches with the bulk URL opener to review pages side by side. Keep batches to 15-20 tabs; beyond that most browsers slow down noticeably.
Which method should you use?
For a site you do not own, start with the sitemap and fall back to a crawl when the sitemap is missing or clearly incomplete. For a site you do own, combine the crawl with Search Console: the crawl finds what is linked, Search Console reveals what Google discovered anyway, and the difference between the two lists is usually the interesting part.
Use site: only to sanity-check, never as your inventory. It is the quickest method and the least trustworthy, which is an easy combination to be misled by.
One Comment