Avoid external API's rate limit during static generation

[elibenton]

Hey folks!

First time Next user here. I'm trying to build an election information site using the Google Civic Info API. I'd love to use static generation, because the API is only updated every couple of months, usually around election time.

I'd like to create a unique, static route for every The API is uses the Open Civic Data Division ID's (OCD-ID) to map every "political geography" within the country (think "Los Angeles County, California or Michigan's 6th Congressional District"). The ID's use a recursive structure such as below:

ocd-division/country:<country_code>(/<type>:<type_id>)*

So here's where I'm running into issues

Because the API can return a single recursive result, I can search single state, say California, and get every division from the largest all the way down to the smallest. I then want to use that single JSON object to make my static paths and static props. I'll use the OCD-ID's as the path and I'll provide the elected officials as the props.

Everything works fine in development, but then when I try to build the site, things quickly break down because the API hits a 100 request per second rate limit and begins returning 429 errors. This causes a pre-render error, because props that should be available no longer are.

It seems that the API is being called on every page build, even though I only want to call the API once and then traverse that object, using only a tiny fragment for each page. Here is the code below:

const QUERY =
	'https://www.googleapis.com/' +
	'civicinfo/v2/' +
	'representatives/' +
	'ocd-division%2Fcountry%3Aus%2Fstate%3Aca?' +
	'recursive=true&' +
	`key=${process.env.CIVIC_INFO_API_KEY}`


export async function getStaticPaths() {
	const res = await fetch(QUERY)
	const data = await res.json()
	const paths = Object.keys(data.divisions).map(div => ({
		params: { division: div.split('/') }
	}))
	return { paths, fallback: true }
}


export async function getStaticProps({ params }) {
	const res = await fetch(QUERY)
	const rawData = await res.json()
	const matchingDivision = Object.entries(rawData.divisions).find(
		div => div[0] === params.division.join('/')
	)
	const data = {
		divisionName: matchingDivision[1].name,
		divisionID: matchingDivision[0],
		divisionOffices: matchingDivision[1].officeIndices.map(index => ({
			office: rawData.offices[index],
			officials: rawData.offices[index].officialIndices.map(index => rawData.officials[index])
		}))
	}
	return { props: { data } }
}

Here's a common output from next build:

Everything works fine with server rendering, but it seems like this could be a textbook case for static generation. Can anyone help show me where I've gone wrong?

Please let me know if I can provide any other info. Thanks in advance for all help!

[ricokahler]

I have some next.js libs that might help in this scenario. They haven't been battle-tested yet but I'd love to get them there.

• https://github.com/ricokahler/next-plugin-preval
• https://github.com/ricokahler/next-plugin-query-cache (experimental)

[localjo]

Thanks! The next-plugin-query-cache worked for me. I was able to use the queryFetch drop-in replacement for fetch (in the documentation) and plug it in to googleapis like this:

google.options({
  fetchImplementation: queryFetch,
});

Working great to avoid rate limits with the Google Sheets API.

[reslea]

next-plugin-preval didnt work for me, as in current stable version 1.2 runs multiple times per build issue, it was a try to fix this in 1.3.0-alpha.1, but even in this version the issue is still exists.

And I was unable to use next-plugin-query-cache as I'm not controlling the fetch which is done by CMS client library.
hope this feedback will save someone a bit of time :)

[oldo]

Also coming across this issue with Contentful's API rate limit of 7 requests per second.

[p0wl]

We ran into this now and used the RetryLink to work around it during build time:

const link = from([
 	new RetryLink({
 		attempts: (count, operation, error) => {
 			const shouldRetry =
 				count < 5 &&
 				error.statusCode === 429 &&
 				error.response?.headers?.get('X-Contentful-RateLimit-Reset');

 			return shouldRetry;
 		},
 		delay: (count, operation, error) => {
 			const delay =
 				Number(error.response?.headers?.get('X-Contentful-RateLimit-Reset')) *
 				1000;
 			console.info('CONTENTFUL-RATELIMIT: retry in ', delay, 'ms');
 			return delay;
 		},
 	}),
 	createHttpLink({
 		uri: `${process.env.NEXT_PUBLIC_CMS_MARKETING_SPACE_URL}${process.env.NEXT_PUBLIC_CONTENTFUL_API_URL_POSTFIX}`,
 		credentials: 'same-origin',
 		headers: {
 			authorization: `Bearer ${API_TOKEN}`,
 		},
 	}),
 ]);

const client = new ApolloClient({
	cache: new InMemoryCache({
		possibleTypes: generatedIntrospection.possibleTypes,
		typePolicies,
	}),
	link,
...
});

[saadq]

I was facing a similar issue where doing a static build of 200+ pages that used the GitHub API would hit some secondary rate limits. I resolved this issue by simply adding a 500ms delay before making API calls during the build process. The IS_BUILD environment variable is used to determine whether or not we are in the build process.

export function avoidRateLimit(delay = 500) {
  if (!process.env.IS_BUILD) {
    return
  }

  return new Promise((resolve) => {
    setTimeout(resolve, delay)
  })
}

So anywhere I made API calls, I could just add it as the first line:

async function getReadme() {
  await avoidRateLimit()
  ...
}

async function getRepoDetails() {
  await avoidRateLimit()
  ...
}

And in your package.json, you can update your "build" script to be like this:

"build": "IS_BUILD=true next build"

And this let the builds go through. Not very elegant but it works

[Livy-17]

Thank you so much this works well!

[saadq]

Updated to use IS_BUILD=true env variable instead of using NODE_ENV=production

[spidgorny]

Thank you for your idea. This is a little more dynamic:

let lastFetch = new Date()

export async function avoidRateLimit() {
	if (process.env.NEXT_PHASE === 'phase-production-build') {
		let sinceLastFetch = new Date().getTime() - lastFetch.getTime()
		if (sinceLastFetch < 5000) {
			await sleepMs(5000)
		}
		lastFetch = new Date()
	}
}

To properly fix that issue, we should catch 429 error and only sleep after that, then repeat the request.

[konstantinbrazhnik]

Hi all,

There is a little know NextJS configuration that controls multithreading at build time. I found no mention of it in NextJS's documentation and only stumbled across it perusing the source code.

Without any configuration, the NextJS build will attempt to do some sort of multithreading, which results in multiple simultaneous HTTP requests at build time. I don't know what the default configuration is here and how it depends on the build server. In my case, we were using Netlify, so I imagine Netlify's builders also had a play in this.

Regardless, the following NextJS configuration in your next.config.js will force the build process to single-thread when it is cycling through the getStaticProps and getStaticPaths requests:

...
    experimental: {
        // This is experimental but can
        // be enabled to allow parallel threads
        // with nextjs automatic static generation
        workerThreads: false,
        cpus: 1
    },
...

After finding this in the code, I also found mention of this functionality here, so I will give credit where credit is due: https://docs.uniform.dev/sitecore/deploy/how-tos/how-to-control-nextjs-threads/

[heijmerikx]

Thank you for digging through the sourcecode and sharing your solution @konstantinbrazhnik !
I stumbled upon this info just because I had a suspicion the randomly failing Netlify builds where due to the headless CMS rate limiting. This solves it consistently for me. 🎉

[konstantinbrazhnik]

@elibenton I would be curious to know if you attempted to alter the workerThreads configuration to solve your API rate limit issue. It seems to me like this is the most direct solution to this issue that fixes it from a configurational standpoint rather than by deploying additional workaround code.

If it works for you as it has to some of the other people in this thread, perhaps marking this discussion as "Answered" would help others find this solution quicker.

[MattRey]

Brilliant! Thanks so much @konstantinbrazhnik - solved my build issues when using Wordpress api 🙏

[khaled-sakhai]

@konstantinbrazhnik Thank you so much, this solved my issue, i was getting a 429 Status code(too many requests because the API im consuming has some limiting access) the build kept failing, the build process worked and the app was super fast :D

[nmitic]

@jussivirtanen same here, and this one seems to help. I only got this issues when deploying with webhook to main branch on prod environment. When deploying to preview env of Vercel it was never an issue.

[smakman]

@konstantinbrazhnik you are a lifesaver!

[khaled-sakhai]

This is a quicker way to solve the issue, it adds a delay of 12 seconds between each api call:

By default, Next.js sets the NODE_ENV environment variable to "production" when you run the next build command to create a production build. In development mode, NODE_ENV is set to "development".

export async function getStaticProps(context){
.....
// Delay only in production environment
if (process.env.NODE_ENV == "production") {
await new Promise((resolve) => setTimeout(resolve, 12000));
}
......
}

[reslea]

Not a solution.
It only postpones every request for 12s but does nothing with multiple request per single second (or any other kind of period), so will not help avoiding rate limits.

[khaled-sakhai]

It really depends on what you're trying to do,
it has worked for me because the api i have has a limit of 2 requests per second and i only needed to call it 125 times.

[ottoo]

NextJS 15 now has options to help with this:

https://nextjs.org/blog/next-15#advanced-static-generation-control-experimental