Caching for server side toggles

[Meemaw]

Describe the feature request

It seems the current recommended approaches to get toggles with SSR/GSSP both use an un-cached fetch to the API. Since the definitions rarely change, I think it doesn't make much sense to fetch definitions for every request, but instead keep an unleash client instance around and re-fetch definitions every n seconds (in the background).

With this, we can then just do the toggles evaluation for each request.

Background

No response

Solution suggestions

Its fairly easy to implement something like this in user-land, but I think it would make sense to document/build this into the package itself for ease of usage.

[Tymek]

What we have right now

Server-side

App Router

You can do it with App Router server-side fetch cache, example here:
https://github.com/Unleash/unleash-client-nextjs/blob/main/example/src/app/app-page/page.tsx#L22

  const definitions = await getDefinitions({
    fetchOptions: {
      next: { revalidate: 15 },
    },
  });

Docs: https://nextjs.org/docs/app/building-your-application/data-fetching#caching-data

Pages Router

You can have an API endpoint that will fetch definitions, with cache-control (Docs). Then use this URL as source of definitions. As in middleware example.

Client-side

Unleash Edge

If you need to scale right now, we have Unleash Edge. It's written in Rust, crazy-performant implementation of a concept similar to Unleash Proxy. We have clients experimenting with setting it up on CloudFlare Edge Workers. It doesn't work on Vercel yet.

[Tymek]

Plans

Client-side

I'll add an example of how to implement API route "proxying" for frontend API.

[Meemaw]

@Tymek thanks for the reply. I'm particularly interested in the Pages Router. While cache-control would help with the load on BE, its still a network roundtrip, which will add at least 50ms latency to the SSR render time (assuming its cached on CDN).

I was thinking more of a in-memory cache which would be instant and add no overhead at all.

[Tymek]

You can use caching on SSR. (docs). Downside is that if you are returning definitions from there to the client, you will expose your configuration. You can filter toggles you're interested in I guess. Will including this in the library help? I'm hesitant because resolving feature toggles on frontend (running server-client on frontend) isn't something Unleash supported before Next.js SDK.

Correct me if I'm wrong, but I don't think it's possible to store something in-memory in a Serverless/Edge environment. I looked into using KV store, but definitions are too big for that.

[Meemaw]

You can use caching on SSR. (docs). Downside is that if you are returning definitions from there to the client, you will expose your configuration. You can filter toggles you're interested in I guess. Will including this in the library help? I'm hesitant because resolving feature toggles on frontend (running server-client on frontend) isn't something Unleash supported before Next.js SDK.

Correct me if I'm wrong, but I don't think it's possible to store something in-memory in a Serverless/Edge environment. I looked into using KV store, but definitions are too big for that.

By in memory I don't mean anything fansy, just storing things in a local variable. Not using Serveless/Edge, but I'm pretty sure you can do that. That being said, this would be more for the pages router, where you have a node server running for a long time.

[Tymek]

I don't think that will run as expected on Cloudflare/Vercel etc. I'll experiment with it later this week. And again, if you have any user-land implementations, sharing it will help :)

Previous Next.js SDK experiment was using this approach (archived https://github.com/Unleash/next-unleash). I'll look into bringing it back.

[Meemaw]

I don't think that will run as expected on Cloudflare/Vercel etc. I'll experiment with it later this week. And again, if you have any user-land implementations, sharing it will help :)

Well, we are using unleash-client for now, but wanted to migrate to @unleash/nextjs. We are using a Unleash client instance, which will just fetch definitions every n seconds in the background. The client then exposes unleash.getFeatureToggleDefinitions() method which just returns these definitions from memory.

Something similar to this could work for us:

let definitions: Promise<ClientFeaturesResponse> | undefined;

export const getDefinitions = async (): Promise<ClientFeaturesResponse> => {
  if (!definitions) {
    definitions = loadDefinitions();
    setInterval(async () => {
      const updated = await loadDefinitions();
      definitions = Promise.resolve(updated);
    }, 15000);
  }
  return definitions;
};

[Tymek]

Ok. Thank you for explaining 👍🏻
I'm in favor of wrapping cache(getDefinitions(...), ttl) in a functional style instead of OOP instances. It's flexible. An example use case with long-running Next.js server is definitely something I'd like to cover soon.

[mltsy]

I had similar thoughts to @Meemaw when trying to switch to this package for the first time just recently. I guess the biggest question I have is: why doesn't @unleash/nextjs use the Node SDK (unleash-client) on the server side? It seems like that already has all of this stuff worked out, and this Next package could just pass in the applicable environment variables and wrap it up in a nice bow for the user, providing a long-lived server-side client for cases where we're not using serverless edge functions (which is the case for me).

I suppose maybe that wouldn't have made sense if the original motivation of this package was to support Serverless/Edge, but it's certainly worth supporting as an additional use-case, considering how simple it ought to be to just wrap up unleash-client (and since this package already requires unleash-client anyway).

[Tymek]

Next.js doesn't use Node SDK, because of incompatibility in handing "Host" and "IP" - deprecated strategies, that cause errors when imported into client-side code. Goal is to have both SDKs unified. I'll check if I can patch Node SDK to work in the browser.

[yosiawan]

@mltsy in your PR (#66) you mentioned that the flagsClient is non-syncing does this means we still create multiple instances of Unleash in our app using this method?

const flags = flagsClient(toggles); // instantiates a static (non-syncing) unleash-proxy-client

Anyone know how to maintain a single instance in an SSR setup? It seems my app is causing a timeout in my Unleash server due to having multiple instances

[mltsy]

I'm not using unleash anymore, so take this with a grain of salt, but my recollection is that flagsClient will create a client, which does not, by default, contact the server. In the example given, getDefinitions is what sends a single request to the server to fetch the flag definitions. flagsClient takes whatever "toggles" you pass in, and stores them in memory for evaluation later. So if you are creating multiple of them, yes you will have multiple clients, but the unleash server will only be contacted when you call getDefinitions (unless you tell the client to start syncing with the server - I don't recall how that is done)