Your First Actor
Your First Actor
This lab walks the core Substrate objects by hand, with nothing but kubectl and kubectl ate:
| Object | Where it lives | Tool |
|---|---|---|
WorkerPool | Kubernetes CRD, in a namespace | kubectl apply / kubectl get workerpool |
SandboxConfig | Kubernetes CRD, cluster-scoped | kubectl get sandboxconfig |
| Atespace | Substrate API (PostgreSQL), global | kubectl ate … atespace |
ActorTemplate | Substrate API, in an atespace | kubectl ate … actor-template |
| Actor | Substrate API, in an atespace | kubectl ate … actor |
| Worker | Substrate API record, one per worker Pod | kubectl ate get workers |
Capacity (WorkerPools) is Kubernetes. Everything that changes often (templates, Actors, which Actor sits on which worker) lives in Substrate’s own API, because it changes too fast for etcd.
Status: Commands checked against the Substrate v1.0.0 source. Not yet run end to end on a cluster.
Prerequisites
- Install Agent Substrate (Standalone).
- A terminal in the root of the Substrate
v1.0.0clone, withATE_INSTALL_KIND=trueandKIND_CLUSTER_NAMEexported.
Step 1: Deploy the counter demo
The counter is a small Go HTTP server. Every request increments three counters:
- one in process memory
- one in a file on a durable volume
- one on the root filesystem
It replies with all three. That makes it ideal for watching state survive a suspend and resume.
go run ./cmd/ate-setup deploy demo counterThe installer:
- Builds the counter image with
koand pushes it to the local registry. - Applies the
ate-demo-counternamespace and thecounterWorkerPool (3 workers), and waits for the workers to roll out. - Creates the
ate-demo-counteratespace. - Creates the
counterActorTemplate withkubectl ate create actor-template. - Waits for the template’s golden snapshot.
Step 2: Look at the capacity (Kubernetes)
kubectl get workerpool counter -n ate-demo-counter -o yaml
kubectl get pods -n ate-demo-counterThe WorkerPool (apiVersion: ate.dev/v1) becomes a Deployment of three warm worker Pods. Each Pod runs ateworker-gvisor, the in-Pod coordinator that drives the gVisor sandbox. The pool’s spec.sandboxClasses is [{name: gvisor}], so its workers run gVisor sandboxes with the binaries from the class default SandboxConfig, gvisor-default.
The pool’s spec.template.resources.limits (1 CPU, 1 GiB) is each worker’s capacity: the budget the Actors placed on it share. The worker keeps 64 MiB of the memory for itself, so it reports 960 MiB.
Substrate’s view of the same Pods:
kubectl ate get workersThe ACTORS column shows 0/1000 for each: no Actors placed, against the default limit of 1000 Actors per worker. The CPU(MILLICORES) and MEMORY columns show allocated/capacity: 0/1000 and 0/960Mi.
Step 3: Look at the template (Substrate API)
kubectl get crd actortemplates.ate.devError from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "actortemplates.ate.dev" not foundThat’s expected. The ActorTemplate still exists. Only its Kubernetes CRD was removed. Substrate moved ActorTemplates (along with atespaces, Actors, and Workers) out of etcd and into its own API, backed by PostgreSQL, because they change faster than the Kubernetes API is built for. Same object, different home, different CLI. Ask Substrate for it:
kubectl ate get actor-templates -a ate-demo-counter
kubectl ate get actor-template counter -a ate-demo-counter -o yamlThe table shows GOLDEN TAG. It’s empty while the golden snapshot is still building, then holds the name of the snapshot tag that every new Actor of this template restores from.
The parts of the manifest that matter:
| Field | Meaning |
|---|---|
metadata.atespace / metadata.name | The template’s identity. Templates live in an atespace |
workerSelector.matchLabels | Which WorkerPools may run it, matched against the WorkerPool’s labels (workload: counter) |
containers[].wakeupProbe | GET /readyz on port 80. Substrate waits for it before it snapshots or hands the Actor traffic |
resources.limits | The sandbox size, cpuMillicores: 1000 and memoryBytes: 536870912 (512 MiB). Also used to place the Actor on a worker with enough capacity |
snapshotConfig.preferredFidelity | SNAPSHOT_FIDELITY_MEMORY: snapshots hold process memory, rootfs changes, and volumes |
volumes[].durableDir | A per-Actor directory that survives suspend and resume |
There’s no snapshotConfig.storageLocation, so snapshots go under ate-api-server’s --snapshot-location. On kind, that’s gs://ate-snapshots/, the in-cluster RustFS bucket.
There’s no sandbox class either. The WorkerPool decides the sandbox class, and a template picks its pools through workerSelector. This one matches the counter pool, which runs gVisor.
Templates are immutable. There’s no update: a new version is a new template.
Step 4: Write your own template
Create your own atespace and a template in it, reusing the counter image the installer built. First grab the resolved image reference:
export COUNTER_IMAGE=$(kubectl ate get actor-template counter -a ate-demo-counter -o json \
| jq -r '.containers[0].image')
echo "${COUNTER_IMAGE}"kubectl ate create atespace first-actorThe manifest is the protojson form of ateapipb.ActorTemplate (the same shape -o yaml prints), not a Kubernetes object. Unknown fields are rejected.
kubectl ate create actor-template -f - <<EOF
metadata:
atespace: first-actor
name: my-counter
workerSelector:
matchLabels:
workload: counter
containers:
- name: counter
image: ${COUNTER_IMAGE}
command:
- /ko-app/counter
wakeupProbe:
httpGet:
path: /readyz
port: 80
volumeMounts:
- name: data
mountPath: /home/counter
resources:
limits:
cpuMillicores: 1000
memoryBytes: 536870912
snapshotConfig:
preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
volumes:
- name: data
durableDir: {}
EOFCreating the template triggers a golden snapshot. Substrate boots the workload once, waits for its wakeup probe, and checkpoints it. Every Actor of this template then starts by restoring that snapshot instead of cold-booting.
Run this until GOLDEN TAG has a value:
kubectl ate get actor-template my-counter -a first-actorIf ERROR shows ERROR, read the message with -o yaml.
Step 5: Create an Actor
kubectl ate create actor counter-1 -a first-actor --template my-counter
kubectl ate get actor counter-1 -a first-actorThe Actor is ACTOR_STATE_SUSPENDED with no WORKER POD. It’s only a record in PostgreSQL and uses no compute yet.
kubectl get actor counter-1 doesn’t work. Actors have never been Kubernetes objects.
Step 6: Wake it with traffic
The router only accepts mTLS from allowlisted clients. In a separate terminal, start a proxy to the Actor. kubectl ate proxy actor gets a client certificate for the atenet-client identity, listens on 127.0.0.1:8000, and sends everything to the Actor’s port 80 through the router:
kubectl ate proxy actor counter-1 -a first-actorSend a request:
curl -X POST http://localhost:8000preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1The proxy adds the routing key for you: the ate-target-actor: first-actor/counter-1 header.
The router saw a suspended Actor, asked ate-api-server to resume it onto a free worker, waited for it to be ready, and then forwarded the request.
kubectl ate get actor counter-1 -a first-actor
kubectl ate get workersThe Actor is now ACTOR_STATE_RUNNING, with a WORKER POD and WORKER IPS. In get workers, one worker now shows ACTORS 1/1000, CPU(MILLICORES) 1000/1000, and MEMORY 512Mi/960Mi: the template’s limits are now reserved on that worker.
Send a few more requests and watch all three counters climb.
Step 7: Suspend and resume
kubectl ate suspend actor counter-1 -a first-actor
kubectl ate get actor counter-1 -a first-actor
kubectl ate get workersSubstrate checkpointed the sandbox (memory, rootfs changes, and the durable volume) and freed the worker. By default, suspend returns once the snapshot is on the worker’s node, and uploads it to snapshot storage in the background. Pass --mode=sync to wait for the upload.
The proxy from Step 6 is still running. Send another request through it:
curl -X POST http://localhost:8000All three counters continue from where they stopped. The memory count surviving is the interesting part: the process wasn’t restarted, it was restored. Check which worker it landed on. It may be a different Pod than before:
kubectl ate get actor counter-1 -a first-actorStep 8: Read the Actor’s logs
Actor logs follow the Actor across workers:
kubectl ate logs actor counter-1 -a first-actor
kubectl ate logs actor counter-1 -a first-actor --source=lifecycle--source=lifecycle shows only the lifecycle events (resume, suspend, and so on).
Clean up
Stop the proxy with Ctrl-C. Deleting a running Actor stops its workload and frees its worker first, so there’s no need to suspend it:
kubectl ate delete actor counter-1 -a first-actor
kubectl ate delete actor-template my-counter -a first-actor
kubectl ate delete atespace first-actorDeleting the template also deletes its golden Actor and golden snapshot. An atespace can only be deleted once it’s empty.
Keep the counter demo for the next labs. To remove it later:
go run ./cmd/ate-setup delete demo counter