Skip to main content

Self-hosted workers

A self-hosted worker is a process you run on your own machines. It connects outbound to FluidGrids, takes work from one or more task queues, and runs it: the workflow tasks of the durable functions on those queues, and the user activities they call. Use a worker when an activity needs your network, your files, a library the managed sandbox does not offer, or hardware such as a GPU.

The worker only opens an outbound WebSocket connection, so it needs no inbound port, VPN or public address.

Workers or runners?

A worker runs durable-function code: workflow tasks and your own activity functions. To run individual canvas nodes on your machine instead, use a runner.

Workflow code still runs in the same isolate as on the managed queue, from the exact bundle you pushed, so a run behaves the same wherever its tasks run.

1. Create a queue and a worker token​

fluidgrids queues create payments
fluidgrids workers tokens create payments-prod --queue payments # the token is shown once

In the app, the Task queues page has the same actions: New queue, then New token under Worker tokens. A worker token (fgr_…) lets a worker serve only the queues it names. Copy it when it is shown: it cannot be shown again. Keep it in your secret store and give it to the worker in the FLUIDGRIDS_WORKER_TOKEN environment variable, never in source code.

fluidgrids workers tokens list # names and prefixes, never the tokens
fluidgrids workers tokens rotate <id> # a new token; workers on the old one disconnect
fluidgrids workers tokens delete <id>

2. Implement the activities​

Declare a user activity where the workflow can import it, and implement it in a module the worker loads. Workflow code never imports the implementation.

activities/declarations.ts
import { declareActivity } from '@fluidgrids/workflows';

export const charge = declareActivity<{ amount: number }, { id: string }>('billing.charge', {
taskQueue: 'payments',
});
activities/index.ts
import { implementActivity } from '@fluidgrids/workflows';
import { charge } from './declarations';

export const chargeImpl = implementActivity(charge, async (ctx, args) => {
await ctx.heartbeat({ step: 'charging' });
const result = await myPaymentClient.charge(args.amount);
return { id: result.id };
});

The workflow calls it with ctx.activity, which takes the same retry, timeout and summary options as ctx.run:

const receipt = await ctx.activity(charge, { args: { amount: 42 }, startToCloseTimeout: '30s' });

3. Run the worker​

From a project, the CLI serves the bundle that fluidgrids push wrote and the activities a module exports:

export FLUIDGRIDS_WORKER_TOKEN=fgr_… # from your secret store
fluidgrids worker run --queue payments --activities activities/index.ts
OptionMeaning
--queue <names>The queues to serve, comma separated. name@dev, name@staging or name@prod serves one environment
--environment <env>Serve only this environment of every queue named without one
--activities <module>The activities to serve: a .ts or .js module, or a Python module name
--no-workflowsServe activities only
--buildBuild the bundle now instead of using the one push wrote
--max-concurrent-activities <n>Activity slots (default 100)
--max-concurrent-workflow-tasks <n>Workflow task slots (default 4)
--max-activities-per-second <n>This worker's activity start rate
--metrics-port <port>Serve Prometheus metrics on 127.0.0.1:<port>
--identity <name>The name the worker reports (default hostname@pid)
--token-env <VAR>Read the token from another variable

SIGTERM or Ctrl-C drains the worker: it stops taking work, gives running work time to finish, and hands back whatever is left so another worker can pick it up.

While you develop​

fluidgrids dev builds the project, creates a temporary queue and token, and runs a worker for it until you press Ctrl-C, then removes them again (--keep keeps them). It prints the queue name, so you can start runs on it:

fluidgrids dev
fluidgrids runs start order --task-queue dev-1a2b3c4d

Running a worker from code​

The fluidgrids worker run command wraps the SDK's Worker. Use it directly to embed a worker in your own service.

import { Worker } from '@fluidgrids/workflows/worker';
import { chargeImpl } from './activities';

const worker = await Worker.create({
taskQueue: 'payments',
workflowBundles: [{ path: '.fluidgrids/build/bundle.mjs' }], // what `fluidgrids push` wrote
activities: [chargeImpl],
maxConcurrentActivities: 16,
shutdownGraceTime: '30s',
metrics: { prometheus: { port: 9464 } },
});
await worker.run();

In Python, install the worker extra (pip install 'fluidgrids-workflows[worker]'):

import asyncio

from fluidgrids_workflows.durable import define_activity
from fluidgrids_workflows.worker import Worker


@define_activity("billing.charge", task_queue="payments")
async def charge(ctx, args):
await ctx.heartbeat({"step": "charging"})
return {"id": "ch_1"}


async def main() -> None:
worker = Worker("payments", workflow_bundles=[".fluidgrids/build/bundle.pyz"], activities=[charge])
await worker.run()


asyncio.run(main())

TypeScript and Python workers speak the same protocol, so both can serve one queue. A worker runs the workflow bundles of its own language; activities can be served from either.

What an activity can do​

An activity gets a context with:

  • info: the activity id (the same on every attempt, so use it as an idempotency key), the attempt number, the deadline and the last heartbeat details.
  • heartbeat(details): report progress. A heartbeat's answer also tells the activity when it has been cancelled or paused; retries resume from the last details.
  • cancellationSignal: aborted when the activity is cancelled.
  • connection(name): a credential from a FluidGrids connection, only for connections the activity declares and the worker token allows (see below).
  • completeAsync(): end the attempt without a result and get the activity's task token, so another system completes the activity later (see Asynchronous completion).
  • log and metrics.

The engine owns retries and timeouts. An attempt that runs past its deadline is stopped and retried by the activity's retry policy.

Local activities​

ctx.localActivity(declaration, options) runs a declared user activity inside the workflow task, on the same worker, without a round trip through the queue. Use it for short calls (the task has a 30-second budget for them). Local activities run at least once, so make them idempotent; they have no heartbeats, no asynchronous completion and no connections, and they need a self-hosted queue.

Connections for workers​

A worker never sees a credential unless you allow it. A workspace admin chooses, per worker token, which connections its user activities may use:

fluidgrids workers tokens allow-connections <tokenId> crm billing-api

In the app this is Connections on the token's row. A workspace admin must also allow self-hosted workers on the connection itself, on its page under Connections. OAuth connections release access tokens, never refresh tokens; API-key connections release the key. Every release is recorded in the audit log.

Environments​

To keep staging workers away from production work of the same queue:

  • start the worker with --environment staging (or environment in the SDK, or FLUIDGRIDS_WORKER_ENVIRONMENT), or name the queue as payments@staging;
  • issue the worker token for payments@staging. The token then binds every worker that uses it to staging, whatever the worker itself says.

Watching workers​

fluidgrids workers list --queue payments # connected and recently ended workers
fluidgrids workers list --queue payments --stale # workers that stopped answering
fluidgrids queues describe payments # backlog, rates and pollers

The Task queues page shows the same per queue: its workers with their slots, the workflow and activity backlog, and the age of the oldest task. A worker that stops answering is shown as stale, and its work goes back to the queue for another worker.

Workers export Prometheus metrics (task slots, task latencies, sticky cache, reconnects), and each workflow task and activity attempt is a span in the run's OpenTelemetry trace when the worker has OpenTelemetry installed.

Your plan sets how many workers a workspace can have connected at once; a worker over the limit is refused when it connects.

Next steps​