Poll an API in the Background

The Events API is in beta. Event shapes, limits, and behavior may change between releases.

Schedules are available on puter.com. A self-hosted Puter server answers events_disabled.

Some services can't tell your app when something changes, so your app has to ask them every so often. For example, importing new orders from a store, syncing a user's bookmarks from another service, or pulling the latest readings from a sensor. On other platforms you'd run a cron job on a server that calls the API for every user.

In Puter, each user gets a schedule that runs your handler in your app's events worker, every 15 minutes in this recipe, even while your app is closed. The handler asks the API for what changed since its last run, and saves it in the user's key-value store. Run Code on a Schedule covers handlers, permission and creating one schedule per user. This recipe adds the two things polling needs, a cursor and a way to finish a large batch.

The examples call an API that returns the changes after a cursor, a page at a time:

GET https://api.example.com/changes?after=<cursor>
→ { "items": [{ "id": "o-1", ... }, ...], "next": "<cursor>", "hasMore": true }

Most APIs that list changes work like this, with a cursor, a page token or a timestamp.

Publish the Handler

The handler keeps its cursor in the user's key-value store, so each run picks up where the last one stopped. Publish it as the app's developer, from a deploy script or your browser console:

const app = await puter.apps.get('order-sync');    // your app's name

async function syncChanges ({ ctx, user, fetch }) {
    const cursor = (await user.kv.get('sync:cursor')) ?? '';
    const res = await fetch(`${ctx.apiUrl}/changes?after=${encodeURIComponent(cursor)}`, {
        headers: { Authorization: `Bearer ${ctx.apiKey}` },
    });
    if (!res.ok) return;

    const page = await res.json();
    for (const item of page.items) {
        await user.kv.set(`order:${item.id}`, item);
    }
    await user.kv.set('sync:cursor', page.next);
}

await puter.events.handlers.publish('syncChanges', syncChanges, { appUid: app.uid });

The handler gets ctx, the context the schedule was created with, and fetch, which reaches outside services from the events worker. A few details:

Each item is saved under its own id, so saving it twice is harmless. That matters because a run can happen more than once.
The cursor is saved after the items. If a run stops partway, the next run fetches the same page again instead of skipping it.
When the API answers with an error, the handler returns without moving the cursor, and the next run asks again. A handler that throws counts as a failed run, and five failed runs in a row pause the schedule until you publish the handler again, so an outage at the other service would pause every user's sync.

Create the Schedule

In the app, after the user connects their account on the other service, create the schedule with the API's address and the user's key:

await puter.perms.request('backgroundEvents');

await puter.events.onSchedule({
    every: '15m',
    handlerName: 'syncChanges',
    context: { apiUrl: 'https://api.example.com', apiKey: userApiKey },
});

Create it once per user. Run Code on a Schedule shows how to check listSchedules() first.

The context is stored with the schedule and can't be changed. When the user's key changes, remove the schedule with puter.events.removeSchedule() and create a new one. The listSchedules() method shows the names of the context keys, never their values.

The schedule runs every 15 minutes from when it's created, which is as often as a free plan allows. A paid plan can run one every minute. See Rate Limits and Quotas.

Finish a Large Batch

A run has 30 seconds. After a long break, for example on the first sync, the API may have more pages than one run can fetch. The handler keeps fetching while there's time, and when pages are left over it schedules one extra run a minute later, with user.events.onSchedule() and at:

async function syncChanges ({ ctx, user, fetch }) {
    const stopAt = Date.now() + 20 * 1000;
    let cursor = (await user.kv.get('sync:cursor')) ?? '';
    let hasMore = true;

    while (hasMore && Date.now() < stopAt) {
        let res;
        try {
            res = await fetch(`${ctx.apiUrl}/changes?after=${encodeURIComponent(cursor)}&limit=100`, {
                headers: { Authorization: `Bearer ${ctx.apiKey}` },
                signal: AbortSignal.timeout(5000),
            });
        } catch {
            return;    // too slow or unreachable, so the next run asks again
        }
        if (!res.ok) return;

        const page = await res.json();
        if (page.items.length > 0) {
            await user.kv.set(page.items.map((item) => ({ key: `order:${item.id}`, value: item })));
        }
        cursor = page.next;
        hasMore = page.hasMore;
        await user.kv.set('sync:cursor', cursor);
    }

    if (hasMore && !ctx.followUp) {
        await user.events.onSchedule({
            at: Date.now() + 60 * 1000,
            handlerName: 'syncChanges',
            context: { ...ctx, followUp: true },
        });
    }
}

A few details:

The loop stops starting new pages after 20 seconds. Each page is at most 100 items (limit, if the API takes one), its request gives up after 5 seconds, and its items are saved in one call, so the last page started still fits inside the 30. Without those bounds, one slow request or a long run of writes could use up the 30 seconds before the cursor is saved, and every run would fail at the same page.
The extra run is a one-time schedule with the same handler and the same context, plus followUp: true. It continues from the saved cursor and ends on its own after it runs.
An extra run doesn't schedule another one. Whatever it leaves over, the next 15-minute run picks up. A schedule created from a handler counts as one step deeper in a handler chain, so extra runs that each scheduled the next would stop after a few steps.
The extra run and a regular run can overlap. Both start from the saved cursor, so they can fetch the same page twice, and whichever saves last wins, even when the other got further. A cursor that moves back only means pages are fetched again. An item the slower run saves can be older than the one it replaces until a later run fetches it again. If that matters, keep the API's updatedAt on each item and skip writing one older than what's stored. The key-value store has no compare-and-set, so the handler has to tolerate the rare overlap rather than prevent it.

Publish the handler again after the change. A deploy script runs in a new process each time, so its SDK hasn't seen what's published. Call handlers.list() before publishing changed code, or publish() rejects with events_handler_conflict:

await puter.events.handlers.list({ appUid: app.uid });
await puter.events.handlers.publish('syncChanges', syncChanges, { appUid: app.uid });

Full Code

The publish script runs as the app's developer, from a deploy script or your browser console:

const app = await puter.apps.get('order-sync');    // your app's name

async function syncChanges ({ ctx, user, fetch }) {
    const stopAt = Date.now() + 20 * 1000;
    let cursor = (await user.kv.get('sync:cursor')) ?? '';
    let hasMore = true;

    while (hasMore && Date.now() < stopAt) {
        let res;
        try {
            res = await fetch(`${ctx.apiUrl}/changes?after=${encodeURIComponent(cursor)}&limit=100`, {
                headers: { Authorization: `Bearer ${ctx.apiKey}` },
                signal: AbortSignal.timeout(5000),
            });
        } catch {
            return;    // too slow or unreachable, so the next run asks again
        }
        if (!res.ok) return;

        const page = await res.json();
        if (page.items.length > 0) {
            await user.kv.set(page.items.map((item) => ({ key: `order:${item.id}`, value: item })));
        }
        cursor = page.next;
        hasMore = page.hasMore;
        await user.kv.set('sync:cursor', cursor);
    }

    if (hasMore && !ctx.followUp) {
        await user.events.onSchedule({
            at: Date.now() + 60 * 1000,
            handlerName: 'syncChanges',
            context: { ...ctx, followUp: true },
        });
    }
}

await puter.events.handlers.list({ appUid: app.uid });
await puter.events.handlers.publish('syncChanges', syncChanges, { appUid: app.uid });

The app code runs in your app, signed in as the user, once they've connected the other service. Here userApiKey is the key your app got for them:

const allowed = await puter.perms.request('backgroundEvents');
if (allowed) {
    await puter.events.onSchedule({
        every: '15m',
        handlerName: 'syncChanges',
        context: { apiUrl: 'https://api.example.com', apiKey: userApiKey },
    });
}

← All recipes