Generating dynamic sitemaps with Next.js + React.js + next-sitemap
Gabriel Messias da Rosa
17 meses atrás / 10 minutos de leituraReact Next.js HTML CSS SEO TYPESCRIPT
Sitemap is a file that explains to search engines, the structure of a website, as well as its updates and the main pages of the website to be indexed. It is usually created and used in XML format.
Google Sitemaps is one of several services provided by Google to its users. With it, you can easily and quickly consult your website's index statistics, crawl speed, page rank and search statistics, which list the most searched items on Google that direct users to places on the web page.
If your site works with the Next.js and React.js framework, you may have noticed that Next works very well with SSR (Server Side Rendering) and dynamic routes. But how to make these dynamic routes part of the sitemap automatically? That's what we'll see today!
Basic configuration of next-sitemap
Assuming we have this site that needs a sitemap for google indexing.
We need to generate a sitemap with the pages, home, about and users-list, but the private-page cannot be indexed and each time a user is added to the list, the sitemap should automatically update and add the user's profile page to the sitemap.
For this we are going to use a library called “next-sitemap” (if you are using yarn use the yarn add next-sitemap command) so we can generate sitemaps according to our needs.
This library needs a .config.js file, essential for its functioning.
In the root folder of the project, create a file called “next-sitemap.config.js”
So add a default configuration just for your basic working
/** @type {import('next-sitemap').IConfig} */
module.exports = {
siteUrl: process.env.SITE_URL || "<https://example.com>",
generateRobotsTxt: true, // (optional)
// ...other options
};
The next-sitemap will load environment variables from the .env file by default.
The next step is to add a new script to the package.json file called postbuild, which will run our file right after the build in vercel and also locally, always generating a new sitemap for each site build.
Then add the postbuild by referencing the run command of the “next-sitemap” library.
Your scripts will look like this:
"scripts": {
"dev": "next",
"build": "next build",
"start": "next start",
"type-check": "tsc"
"postbuild": "next-sitemap",
},
Time to build the project and see what this configuration gives us.
Note that right after the build, we have a different output giving us more details about what the next-sitemap generated.
Inside the PUBLIC folder 3 new files were generated.
With the project running locally (use the yarn dev or yarn start commands) check the url http://localhost:3000/sitemap-0.xml
You'll see that all pages have been added to the sitemap, even the ones we don't want. Let's fix this!
<urlset xmlns="<http://www.sitemaps.org/schemas/sitemap/0.9>" xmlns:news="<http://www.google.com/schemas/sitemap-news/0.9>" xmlns:xhtml="<http://www.w3.org/1999/xhtml>" xmlns:mobile="<http://www.google.com/schemas/sitemap-mobile/1.0>" xmlns:image="<http://www.google.com/schemas/sitemap-image/1.1>" xmlns:video="<http://www.google.com/schemas/sitemap-video/1.1>">
<url>
<loc><https://example.com></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/about></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/private-page></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/101></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/102></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/103></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/104></loc>
<lastmod>2022-09-20T11:13:12.523Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
</urlset>
Deleting routes from sitemap and robots.txt with next-sitemap
To solve this let's add more settings to our next-sitemap.config.js file.
Add the “exclude” option with its value being an array of strings and each string being a route you want to exclude from the sitemap.
/** @type {import('next-sitemap').IConfig} */ module.exports = {
siteUrl: process.env.SITE_URL || "<https://example.com>",
generateRobotsTxt: true,
exclude: ["/private-page"],
// ...other options
};
As we are also generating a robots.txt, we must remove from its radar the route that should not be indexed. For that add one more option called robotsTxtOptions, its value will be an object with more options. Use the “policies” option, its value will be an array of objects, to enable all possible robots, configure the “userAgent” key with “*”, then configure “allow” with the value a string with the route you want the robots to identify, for the entire site use the project root route, which in this case is “/”. To disable a route just do the same by changing “allow” to “disallow”.
In the end you will get a result similar to this.
/** @type {import('next-sitemap').IConfig} */ module.exports = {
siteUrl: process.env.SITE_URL || "<https://example.com>",
generateRobotsTxt: true,
exclude: ["/private-page"],
robotsTxtOptions: {
policies: [
{ userAgent: "*", disallow: "/private-page" },
{ userAgent: "*", allow: "/" },
],
},
};
Build the project and you will see the changes in robots.txt and sitemap-0.xml
robots.txt
# *
User-agent: *
Disallow: /private-page
# *
User-agent: *
Allow: /
# Host
Host: <https://example.com>
# Sitemaps
Sitemap: <https://example.com/sitemap.xml>
sitemap-0.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="<http://www.sitemaps.org/schemas/sitemap/0.9>" xmlns:news="<http://www.google.com/schemas/sitemap-news/0.9>" xmlns:xhtml="<http://www.w3.org/1999/xhtml>" xmlns:mobile="<http://www.google.com/schemas/sitemap-mobile/1.0>" xmlns:image="<http://www.google.com/schemas/sitemap-image/1.1>" xmlns:video="<http://www.google.com/schemas/sitemap-video/1.1>">
<url>
<loc><https://example.com></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/about></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/101></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/102></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/103></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc><https://example.com/users/104></loc>
<lastmod>2022-09-20T11:36:08.228Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
</urlset>
Dynamic sitemap with Next SSR + next-sitemap
Assuming that the user list is constantly changing, to avoid having to build the project for each new user that is registered, for example, let's make the user list sitemap dynamic.
For this, we need to create a route next, a folder inside the 'pages' folder of next. Name this folder as 'server-sitemap.xml', inside this folder create an index.
Here's the trick, inside the index file you should export by default a function called ServerSitemap, with no return.
const ServerSitemap = () => {};
export default ServerSitemap;
Leave it as is, this function is just so that Next allows us to use getServerSideProps to our advantage.
With that being said, create a common getServerSideProps function. Inside it, fetch the api and use the same data that is used in the dynamic route that you want to make the sitemap more dynamic, in this case, the user id.
Parse the json response from the fetch and store that in a constant. As in the example:
import { GetServerSideProps } from "next";
export const getServerSideProps: GetServerSideProps = async (context) => {
const response = await fetch("<http://localhost:3000/api/users>");
const users = await response.json();
console.log("users", users);
};
const ServerSitemap = () => {};
export default ServerSitemap;
After creating another constant called fields that can be typed with ISitemapField[] type, you can import this type from the next-sitemap library. Make the fields value a map of the users array returned by the API. The return of this map must be an object with at least the key loc (you can see all the options in ISitemapField) with the value being the url of the site with the user id concatenated, and another key called lastmod with its value being a new date formatted. As in the example below:
import { GetServerSideProps } from "next";
import { ISitemapField } from "next-sitemap";
import { UserType } from "../../src/types";
export const getServerSideProps: GetServerSideProps = async (context) => {
const response = await fetch("<http://localhost:3000/api/users>");
const users: any[] = await response.json();
console.log("users", users);
const fields: ISitemapField[] = users.map((user: UserType) => ({
loc: `https://example.com/users/${user.id}`,
lastmod: new Date().toISOString(),
}));
};
const ServerSitemap = () => {};
export default ServerSitemap;
Then return the next-sitemap function called getServerSideSitemap with its first parameter being the context of getServerSideProps and its second parameter being the array of fields.
After all, you should have something like this:
import { GetServerSideProps } from "next";
import { getServerSideSitemap, ISitemapField } from "next-sitemap";
import { UserType } from "../../src/types";
export const getServerSideProps: GetServerSideProps = async (context) => {
const response = await fetch("<http://localhost:3000/api/users>");
const users: any[] = await response.json();
console.log("users", users);
const fields: ISitemapField[] = users.map((user: UserType) => ({
loc: `https://example.com/users/${user.id}`,
lastmod: new Date().toISOString(),
}));
return getServerSideSitemap(context, fields);
};
const ServerSitemap = () => {};
export default ServerSitemap;
This function will basically generate an updated sitemap whenever the sitemap route is accessed, in this case the route is https://example.com/server-sitemap.xml.
Checking the */server-sitemap.xml route we notice that it behaves like a common sitemap:
Also note that we have our users mock with only two users, another 2 commented out and they correspond to the generated sitemap.
If we add more users to our mock “API”, we notice the sitemap change as soon as we update the server-sitemap.xml route
User mockups:
XML with sitemap updated with new users dynamically:
This is an excellent way to streamline frequently changing sitemaps.
We use all the technologies mentioned here at 80 Lines, including for this post.
You can see the code in this repository https://github.com/GabrielMessiasdaRosa/dynamic-sitemap.
References