Dynamic sitemaps with Next.js + React.js + next-sitemap
By Gabriel Messias da Rosa
React Next.js HTML CSS SEO TYPESCRIPT
Sitemap is a file that explains to search engines, the structure of a website, as well as its updates and the main pages of the website to be indexed. It is usually created and used in XML format.
Google Sitemaps is one of several services provided by Google to its users. With it, you can easily and quickly consult your website's index statistics, crawl speed, page rank and search statistics, which list the most searched items on Google that direct users to places on the web page.
If your site works with the Next.js and React.js framework, you may have noticed that Next works very well with SSR (Server Side Rendering) and dynamic routes. But how to make these dynamic routes part of the sitemap automatically? That's what we'll see today!
Basic configuration of next-sitemap
Assuming we have this site that needs a sitemap for google indexing.
We need to generate a sitemap with the pages, home, about and users-list, but the private-page cannot be indexed and each time a user is added to the list, the sitemap should automatically update and add the user's profile page to the sitemap.
For this we are going to use a library called “next-sitemap” (if you are using yarn use the yarn add next-sitemap command) so we can generate sitemaps according to our needs.
This library needs a .config.js file, essential for its functioning.
In the root folder of the project, create a file called “next-sitemap.config.js”
So add a default configuration just for your basic working
The next-sitemap will load environment variables from the .env file by default.
The next step is to add a new script to the package.json file called postbuild, which will run our file right after the build in vercel and also locally, always generating a new sitemap for each site build.
Then add the postbuild by referencing the run command of the “next-sitemap” library.
Your scripts will look like this:
Time to build the project and see what this configuration gives us.
Note that right after the build, we have a different output giving us more details about what the next-sitemap generated.
Inside the PUBLIC folder 3 new files were generated.
You'll see that all pages have been added to the sitemap, even the ones we don't want. Let's fix this!
Deleting routes from sitemap and robots.txt with next-sitemap
To solve this let's add more settings to our next-sitemap.config.js file.
Add the “exclude” option with its value being an array of strings and each string being a route you want to exclude from the sitemap.
As we are also generating a robots.txt, we must remove from its radar the route that should not be indexed. For that add one more option called robotsTxtOptions, its value will be an object with more options. Use the “policies” option, its value will be an array of objects, to enable all possible robots, configure the “userAgent” key with “*”, then configure “allow” with the value a string with the route you want the robots to identify, for the entire site use the project root route, which in this case is “/”. To disable a route just do the same by changing “allow” to “disallow”.
In the end you will get a result similar to this.
Build the project and you will see the changes in robots.txt and sitemap-0.xml
Dynamic sitemap with Next SSR + next-sitemap
Assuming that the user list is constantly changing, to avoid having to build the project for each new user that is registered, for example, let's make the user list sitemap dynamic.
For this, we need to create a route next, a folder inside the 'pages' folder of next. Name this folder as 'server-sitemap.xml', inside this folder create an index.
Here's the trick, inside the index file you should export by default a function called ServerSitemap, with no return.
Leave it as is, this function is just so that Next allows us to use getServerSideProps to our advantage.
With that being said, create a common getServerSideProps function. Inside it, fetch the api and use the same data that is used in the dynamic route that you want to make the sitemap more dynamic, in this case, the user id.
Parse the json response from the fetch and store that in a constant. As in the example:
After creating another constant called fields that can be typed with ISitemapField type, you can import this type from the next-sitemap library. Make the fields value a map of the users array returned by the API. The return of this map must be an object with at least the key loc (you can see all the options in ISitemapField) with the value being the url of the site with the user id concatenated, and another key called lastmod with its value being a new date formatted. As in the example below:
Then return the next-sitemap function called getServerSideSitemap with its first parameter being the context of getServerSideProps and its second parameter being the array of fields.