Managing Feature Flags¶
This guide explains how to add, modify, and use Boolean feature flags in SyRF, including non-production runtime overrides shared by browser and API consumers.
Overview¶
The feature flags system uses env-mapping.yaml as the single source of truth for all environment configuration, including feature flags. The generator reads from this unified schema and produces:
- TypeScript interfaces (
FeatureFlags,FeatureFlagsExternalConfig) - Default values (
FEATURE_FLAG_DEFAULTS) - NgRx selectors for each flag
- Config parser (
parseFeatureFlags()) for runtime parsing - Updates to the web chart's Helm
values.yamlfor deployment configuration
The NGINX envsubst template (env.template.generated.json), appConfig.default.generated.json, the config TypeScript files and _env-blocks.tpl are written by a separate generator, pnpm run generate:env-blocks (from src/charts/syrf-common). generate:runtime-catalog writes RuntimeFeatureFlagCatalog.generated.cs. See Work with generated files.
Key benefits:
- Adding a new feature flag requires only editing
env-mapping.yamland running the generator - Uses unified
SYRF__prefix for all environment variables (not legacySPA__) - The API receives the same deployed defaults as the web app, so cross-stack gates such as
bulkPdfUploadstay consistent - Non-production administrators can temporarily override catalogued Boolean feature flags where runtime changes are supported, without replacing the deployed defaults
┌─────────────────────────────────────────────────────────────┐
│ src/charts/syrf-common/env-mapping.yaml │
│ (Single Source of Truth) │
│ │
│ featureFlags: │
│ services: [api] # Shared web/API deployed defaults │
│ envVars: │
│ - name: SYRF__FeatureFlags__MyNewFeature │
│ valuePath: featureFlags.myNewFeature │
│ default: "false" │
│ web: { tsName: myNewFeature, tsType: boolean } │
│ featureFlag: true │
│ displayName: "My new feature" │
│ category: uiFeatures │
│ description: "Enable my new feature" │
└─────────────────────────────────┬────────────────────────────┘
│
pnpm run generate:flags
│
┌───────────┴────────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ TypeScript Files │ │ Helm values.yaml │
│ │ │ (web .chart) │
│ - types.ts │ │ featureFlags: │
│ - constants.ts │ │ myNewFeature: │
│ - selectors.ts │ │ false │
│ - parser.ts │ │ │
└─────────────────────┘ └─────────────────────┘
File Locations¶
| File | Purpose |
|---|---|
src/charts/syrf-common/env-mapping.yaml |
Schema definition (edit this!) |
src/services/web/scripts/generate-feature-flags.ts |
Generator script |
src/services/web/src/app/generated/feature-flags.types.ts |
Generated TypeScript interfaces (types only, no JS output) |
src/services/web/src/app/generated/feature-flags.constants.ts |
Generated runtime constants (as const) |
src/services/web/src/app/generated/feature-flags.selectors.ts |
Generated NgRx selectors |
src/services/web/src/app/generated/feature-flags.parser.ts |
Generated config parser |
src/services/web/src/assets/data/env.template.generated.json |
envsubst template, written by generate:env-blocks (not generate:flags) |
src/services/web/.chart/values.yaml |
Auto-updated Helm values |
Flag lifecycle requirements¶
A feature flag switches a feature on and off. It does not migrate data. Chris's rule (2026-10-17):
"Flags are used to turn features on and off, not necessarily to auto migrate databases back and forth. So whilst the flag is present and flippable both the database and app code should work well with either on or off state and should be happy going back and forth. That should be an implementation requirement when adding the feature flag in the first place. (The extra infrastructure for and to support a specific flag can be removed at a later stage when the flag is retired.)"
So while a flag exists, the code and the database must work with it on or off, and survive being switched back and forth any number of times, including with data created in the other state. The author of the flag builds and tests that when the flag is added; it is not left to be discovered during a rollback.
This section is the canonical text. The agent checklist in
.claude/rules/feature-flags.md
summarises it, and ADR-023
records the migration side. If they differ, this section wins.
Environments, deployments, migrations and flags¶
Four terms have been getting confused. Use them in this sense in pull requests, runbooks and ADRs.
- Environment: local, preview, staging or production. Each has its own database and its own flag values.
- Deployment: puts a build into an environment. A promotion deploys staging's build to production; a rollback deploys the previous build. Flag values are configuration: changing one is a separate decision from deploying a build, even when a deployment-managed flag needs a rollout to change.
- Migration: part of a build. It runs before that build is deployed (ADR-023). While the previous build may still be serving, a migration only adds things, and it must suit every flag state of both builds. Flags never run, skip or reverse a migration, and no migration reads a flag.
- Flag: a switch that a running build reads, per environment. The only allowed link with migrations is one-way: a feature may refuse to switch on, failing closed, until its migrations have run (its readiness check).
For example, deletionLifecycle needs the deletion indexes, which are migrations 0005 to 0008
(ADR-024). With the flag on but those migrations
missing, the search-delete routes answer a typed 503 instead of recording a deletion that nothing could
advance. That is the allowed direction: the missing migration keeps the feature closed. Turning the flag on or
off never applies or reverses those migrations, and applying them does not turn the flag on.
Requirements when a flag is added¶
- Both states work. On is the new behaviour. Off is the previous behaviour, or the feature shown as
unavailable; never broken. Off is not only a rollback state: for a flag that production has not turned
on, it is production's behaviour, which is what keeps
mainpromotable. - Round trips are safe, any number of times. Data created while the flag is on is handled correctly while it is off (kept, ignored, or finished by work that is not flag-gated), and the reverse. Work already accepted keeps converging after the flag goes off: a kill switch stops new exposure, it does not strand what was already started.
- Flags read by several services. The browser, the API and Project Management may read the same flag, and a rollout reaches them at different times. Every combination of per-service values that a rollout can produce is safe, or the feature refuses to start until the values agree. The dependency catalog (see Non-Production Runtime Overrides) is one way to enforce this.
- Tests. An on, then off, then on test, with data created in each state.
- Retirement. The dead path is deleted. Anything that existed only for the flag (code, collections,
indexes, data) is removed in a later clean-up, by a migration if indexes or data are involved, once no
deployable build and no flag state needs it. Until then it stays, so that switching the flag back is
always possible.
pnpm feature:settledoes the mechanical part (see Settling a feature).
The pull request template has a checkbox for requirements 1, 2 and 4.
Adding a New Feature Flag¶
Adding a flag includes meeting the flag lifecycle requirements: both states work, round trips are safe and the on, off, on test exists. The steps below cover only the wiring.
Step 1: Scaffold the feature and its flag¶
Two commands, from the repository root:
pnpm feature:new <feature-id> --type <Release|Operational|KillSwitch|Experiment|Permission> --owner "<team>" \
[--permission <SignedIn|Project...Policy|Application...Policy>] [--flag <key>] [--hosts api,project-management] [--title "Sentence case label"] [--description "..."] [--category <category>]
pnpm run validate:generated
--permission is required with --type Release (see Feature fields). For example, pnpm feature:new demo-x --type Release --owner "API maintainers" --permission ProjectEditMembershipsPolicy --flag demoX. The command:
- Adds one entry to the
featureFlagssection ofsrc/charts/syrf-common/env-mapping.yaml:SYRF__FeatureFlags__<Key>, Helm valuefeatureFlags.<key>(default"false"), the web name anddisplayName, andruntimePolicywith the next freeorderandactivationRequirement: immediate.--flagdefaults to the feature ID in camelCase,--titleto the ID in sentence case, and--categorytouiFeatures. - Adds one schema 2 feature to
src/charts/syrf-common/feature-definitions.yaml:lifecycle: Developing,grain: Global, the flag inflags, a single-flag capability and the type's defaults (see Feature fields). AReleasegetstopicIds: [<feature-id>]: its user guide is Required, sovalidate:registryreports "guidance missing" for that topic until content exists. Every other type is exempt with a reason and has no topics. - With
--hosts api,project-management, also adds the Project Management half inbackendFeatureFlagsand adeploymentManagedpin, because Project Management never reads runtime overrides (see Carrying a web flag to Project Management). - Runs the generators that read those two files (
pnpm generateruns all of them, in order) and the registry validator, then prints the remaining steps. If a generator throws, it restores every file it could have changed (both sources, the checksum manifest and those generators' outputs) and reports that nothing changed.
It refuses, and writes nothing, when the feature ID, the flag key, its environment variable, its web name or its Helm value already exists, or when the key is a removed flag (a removed key is never reused). It writes no consumer code, content, What's New entry, walkthrough or cluster-gitops value. Those are the remaining steps it prints:
- Consumer code: the web reads
select<Key>(Step 2 below); .NET readsRuntimeFeatureFlagKeys.<Key>, which is generated. A .NET consumer also needs its property inFeatureFlags.csand its entry in the consumer manifest. - Catalog digest: a new catalogued flag changes the catalog digest that every stored runtime policy identity
carries. Update
PinnedCatalogDigestinRuntimeCatalogDigestPinTeststo the value the failing test prints. It is the only pin on the catalog's membership. - Content and What's New (a
Release): a non-Draft guide page that declares the topic in its front matter (syrf.topics, see Guide topics), and aruntime:entry for the feature indocs/content/whats-new.yaml, before the feature leavesDeveloping(an announced release Apply refuses without it:WhatsNewEntryMissing). See Runtime What's New entries. - Walkthrough: optional; a
walkthroughs.yamlentry plusdata-tour-idanchors andcommunication.walkthroughId. See Walkthroughs. - cluster-gitops:
featureFlags.<key>per environment (Step 3 below).
This is what feature:new writes, for reference or for a flag that needs a different section:
featureFlags:
services: [api] # Shared web/API deployed defaults
displayName: "Feature Flags"
envVars:
# ... existing flags ...
- name: SYRF__FeatureFlags__MyNewFeature
runtimePolicy:
key: myNewFeature
order: 54
category: uiFeatures
description: "Enable my new feature"
activationRequirement: immediate
valuePath: featureFlags.myNewFeature
default: "false"
web:
tsName: myNewFeature
tsType: boolean
featureFlag: true
displayName: "My new feature"
category: uiFeatures
description: "Enable my new feature"
displayName is required for every featureFlag: true entry: write a short sentence-case
label. The generator emits it as FEATURE_FLAG_DISPLAY_NAMES in feature-flags.constants.ts,
and the admin runtime feature flags page (/admin/features/flags) shows it as the row name.
Generation fails with the list of offending entries if any feature flag lacks a displayName.
runtimePolicy is the runtime catalog entry: stable key and order, category, description, activation requirement
and dependencies. The runtime key must match web.tsName when that browser alias is present.
Application-settings-only catalog entries belong in featureFlags.catalogOnlyFlags and require
configurationReason; they do not create new Helm values. Dependency entries must name known runtime or deployment
keys, and cycles are rejected. Catalog membership does not confer runtime eligibility; do not add arbitrary Boolean
operational or security settings.
After a hand edit of env-mapping.yaml or feature-definitions.yaml, run pnpm generate and then
pnpm run validate:generated. pnpm generate runs, in order: generate:env-blocks (env template, appConfig
defaults, config TS files, _env-blocks.tpl), generate:flags (feature-flag TypeScript and the web
.chart/values.yaml), generate:runtime-catalog (RuntimeFeatureFlagCatalog.generated.cs and
RuntimeFeatureFlagKeys.generated.cs), generate:feature-definitions, the Dockerfiles, and the NSwag client only
when swagger.json changed. Then it runs the registry validator. pnpm generate <name> runs only the named
generators. Do not hand-edit the generated outputs or add definitions or key constants to the handwritten
RuntimeFeatureFlagCatalog.cs facade.
Step 2: Use the Flag in Components¶
Import and use the generated selector:
import { selectMyNewFeature } from '@app/generated/feature-flags.selectors';
@Component({
selector: 'app-my-component',
template: `
@if (myNewFeatureEnabled()) {
<div>New feature content!</div>
}
`
})
export class MyComponent {
private store = inject(Store);
// Using signals (recommended)
myNewFeatureEnabled = this.store.selectSignal(selectMyNewFeature);
// Or using observables
myNewFeature$ = this.store.select(selectMyNewFeature);
}
Step 3: Configure for Deployment¶
The Helm values.yaml is auto-updated. To enable in a specific environment, update cluster-gitops:
# cluster-gitops/syrf/environments/staging/web/values.yaml
featureFlags:
myNewFeature: true # Enable in staging
Env var precedence and boolean defaults¶
The chart renders one env entry per name. Two rules keep that true:
- Explicit
env:overrides generated flag env vars. If an environment's rawenv:map (incluster-gitops) sets a name that the generated block also emits, for exampleSYRF__FeatureFlags__BulkStudyUpdateRequireCurrentExecutionVersion: "true", the generated entry is skipped and only the explicit value is rendered. This is done insyrf-common.deployment.dotnet(syrf-common.env.withoutExplicit), so it applies to api, project-management, quartz and identity. Prefer the flag's Helm value (featureFlags.<name>) over a rawenv:entry; the override exists so a duplicate key can never be rendered, not as the normal way to set a flag. - An explicit
falseis honoured when the default is"true". Helm'sdefaulttreatsfalseas empty, so a plain| default "true"can never renderfalse. For anyenvVarsentry whosedefault: "true", the generator emitsinclude "syrf-common.boolOrDefault", which keeps an explicit bool and falls back to the default only when the value is unset. Do not hand-write| default "true"on a boolean in a chart template.
A chart value that has no env-mapping.yaml entry renders nothing. Do not add one to a service chart's
values.yaml as documentation; an application setting that is read from appsettings (for example
FeatureFlags:MaxInProgressSessions) is not configured by the chart.
Carrying a web flag to Project Management¶
The featureFlags section renders only into the API. When Project Management also reads a web flag,
declare the same variable and valuePath in the backendFeatureFlags section too (and keep the
web: block only on the featureFlags entry), as bulkPdfUpload and reviewEligibilityPolicy do. One
featureFlags.<name> Helm value then drives the browser, the API and Project Management. A flag that is
read by Project Management but declared only in featureFlags is silently off there (#4292).
Project Management does not read the API's runtime override store, so a runtime override changes the API
and the browser but never Project Management. A flag read by both services must therefore be
deployment-managed: the API pins its effective value to the deployed value, rejects a runtime change
with 400 Bad Request and shows it as deployment-managed on the Feature flags admin page. An
override saved before a flag was pinned has no effect and can still be cleared. reviewEligibilityPolicy
is deployment-managed for this reason (it keeps its pageReload activation, which now only describes how
a changed deployed value reaches an open browser). Change it in GitOps for every consuming service. To pin
another flag, add it with its constraint and refusal text to deploymentManaged in
src/charts/syrf-common/feature-definitions.yaml and regenerate (see Feature registry).
Carrying a web flag to Identity¶
Do not add identity to the featureFlags section's services: that would render every
catalogued web flag into the Identity pod. When Identity needs one web flag, add a narrow
Identity-only section that maps the same Helm value, as identityThemeFlags does for
themeToggle (account portal slice 3,
shared theme preference):
identityThemeFlags:
services: [identity]
envVars:
- name: SYRF__FeatureFlags__ThemeToggle
valuePath: featureFlags.themeToggle
default: "false"
Identity binds it on IdentityFeatureFlags at start-up and never reads the API's runtime override
store, so the Identity half is deployment-only: a runtime override on the Feature flags admin page
changes the app, not Identity. Set the value in the Identity service's environment values in
cluster-gitops as well as the web and API values.
Reading a flag in .NET¶
API and Project Management code reads a runtime flag through one interface, IFeatureDecisions
(SyRF.Mongo.Common.RuntimeFeatureFlags, spec 6R.3 slice R3c). Inject it and use the decision's value:
public sealed class ExampleController(IFeatureDecisions decisions) : ControllerBase
{
[HttpPost]
public IActionResult Start() =>
decisions.Decide(RuntimeFeatureFlagKeys.ProportionalStudyAllocation).Value ? Ok() : NotFound();
}
- Never pass a default.
Valueis always a decision you may act on. When the policy cannot decide (not yet read, stale, unavailable, no reviewed contract), it is the safe value from the catalog's consumer contract, which is false for every reviewed contract today;ReasonandErrorCodesay why. The safe value belongs to the catalog, never to the caller. - Decisions are synchronous and never throw for any reader state, so they fit route selection, authorization handlers and admission checks.
- Several keys at once:
DecideAll(keys)answers every key from one capture of each policy source, so a capability that needs all (or any) of several flags never mixes two revisions.AllEnabledandAnyEnabledanswerAllOfandAnyOf. FeatureFlagsis deployed configuration, bound at start-up. Read it for start-up or domain settings, not for a decision that a runtime override or stop must change.
Which path decides a key depends only on the host's configuration, exactly as before R3c:
| Host configuration | Keys | Decided by |
|---|---|---|
RuntimeFeatureFlags:SharedReaderMode=Membership (API) |
disableMembership |
The legacy-format shared reader (60 s maximum age) |
SharedReaderMode=StatisticsServing (PM) |
statistics serving and its three consumer selectors | The legacy-format shared reader, with this host's FeatureFlags as fallbacks while it is current |
RuntimeFeatureFlags:Mode=Shared |
keys with a consumer contract | The bound authority (contract maximum age; stops force the key off; production confines runtime policy to eligible keys) |
| Anything else | every other key | The host's compatibility values: the API's runtime provider (legacy overrides, deployment pins, AuthorityPreserved), or Project Management's FeatureFlags |
Start-up refuses Mode=Shared together with an enrolled shared reader, so no key has two paths. Production
with ProductionPolicyMode=Legacy resolves to Legacy mode. The reasons are Evaluated, Stopped,
DeploymentControlled, Cached, Legacy, AuthorityPreserved, Stale, Unavailable and Unreviewed.
PolicyVersion is an opaque string, comparable only between decisions from the same source in one process.
Do not read IRuntimeFeatureFlagProvider, SharedRuntimeFeatureFlagReader or RuntimePolicyAuthorityEvaluator
directly: the consumer inventory guard fails CI on a new direct read. In tests,
mock IFeatureDecisions (Decide returns a FeatureDecision), or build RuntimeFeatureDecisions over a real
reader or evaluator when the test is about the path. The vocabulary follows OpenFeature, but SyRF does not use
the OpenFeature SDK: it would lose the catalog-owned safe value, synchronous decisions and single-capture
multi-key reads (spec 6R.3). A later adapter could expose IFeatureDecisions as an OpenFeature provider.
Flag decision (R3c): not flagged. Parity tests in SyRF.Mongo.Common.Tests, SyRF.API.Endpoint.Tests and
SyRF.ProjectManagement.Endpoint.Tests prove the same value as the replaced call sites for every catalog key and
every reader state; a flag would keep both paths alive.
A flag with a dependency and a deployment-managed halt: notificationEmail¶
notificationEmail (feature notification-email, #3942,
ADR-026 decision 13) shows three patterns together:
- A declared dependency on a pinned flag. It depends on
notificationInbox, which is deployment-managed (deploymentManaged). BecausenotificationInboxis a catalogued key, the dependency is declaredruntimeManageable: true(aruntimeManageable: falsedependency is only for a non-catalog backend key such asdeletionLifecycle, which has a value resolver; a catalogued key there stops the API at start-up). The pin still refuses a runtime change tonotificationInboxitself. - Both flags from one capture. Its single gate,
INotificationEmailAdmission, readsDecideAll([notificationEmail, notificationInbox]).AllEnabled, so the two are never read from different revisions. The route gate (FeatureGated) decidesnotificationEmailalone; the in-action check adds the dependency. - A deployment-managed halt beside the flag.
NotificationEmailDelivery:Paused(chart valuenotificationEmailDelivery.paused, defaulttrue; unset also means paused) stops sending without touching the flag or any data: paused, the worker keeps discovering saved items but claims and sends nothing; lifting it sends each waiting item once with its send-time checks. So switchingnotificationEmailon sends nothing until an environment's values lift the pause. Lifting it is an owner decision per environment (G-NOTIF); outside production the email transport must be Mailpit (emailTransport.provider: smtp), and production's is SES. Rendered withdigso an explicitfalsesurvives (a plaindefaultwould turn it back intotrue);notification_email_delivery_pause_test.yamlin the API chart pins all four cases.
Off, the preference routes answer 404, /inbox/email-settings falls through to not-found and the worker reads no
preference or ledger row; every row is kept, so on, off, on loses nothing (NotificationEmailWorkerTests).
Settling a feature¶
A Released feature settles when it has been on everywhere for longer than its settle period and nobody needs
"off" any more: its flag and the "off" branches go, and the feature becomes a permanent part of the product. The
clean-up PR is titled "remove flag <key> for <feature-id>".
pnpm feature:settle <feature-id>
pnpm run validate:generated # after deleting the references it prints
feature:settle does the mechanical part:
- Removes every
env-mapping.yamldeclaration of the feature's flags (each section's entry for the sameSYRF__FeatureFlags__<Key>, and a section left empty), and anycatalogOnlyFlagsentry. - Sets the feature's registry entry to
lifecycle: Settled. Itsflagsandcapabilitystay as history. - Adds the keys to the removed flags (
removedFlagsunderfeatureFlagsinenv-mapping.yaml, generated asRuntimeFeatureFlagKeys.RemovedFlags). Only catalogued keys are evaluated, so a stored runtime override for a removed key is ignored;RuntimeFeatureFlagCatalogTestschecks that for every removed flag. A removed key is never reused: the generator andfeature:newrefuse it. - Drops the flag's
deploymentManagedpin, if any. - Regenerates, then prints every reference still to delete: the consumer-manifest reads, and a search of tracked
files (generated files excluded) for the key, its configuration property, its
select<Key>selector and its environment variable. If a generator throws, it restores every file it could have changed and reports that nothing changed.
It refuses, and writes nothing, a feature that is not Released, and a flag that a flag group still names (group IDs
are versioned contracts, so remove the group in its own reviewed change first).
In the same PR, delete the "off" branches and their tests, the FeatureFlags.cs properties and the consumer-manifest
entries, and update the catalog digest pin as for a new flag. The registry validator refuses a
Settled feature whose key is still catalogued or is not a removed flag.
Ready to settle¶
Nobody types a review-by date (D21). Each environment's admin area works out which features are ready from its own
records, and nothing time-based ever fails a build. A feature is listed as Ready to settle when it is a Release
that is not Settled or Withdrawn, no emergency stop is in force on its flags, and it has been live for everyone
here for longer than its settle period (registry settleDays, 30 days for a Release unless the registry sets it).
Two sources (D43 = B). The read takes "went live for everyone" from both:
- Release records. The latest GeneralRelease record of this binding that is confirmed, or imported with
Dateprecision. It counts from the later of its confirmation and its lastRestoredevent, and gives no date while its last stop event (PausedorEmergencyStopped) has no restore after it.MonthandUnknownimports give no date: settling needs one. - The flag change history. The current policy lineage's audit receipts (
runtimeFeatureFlagAudit), walked back from the current revision: the date is the commit of the earliest revision from which every revision up to now had all the feature's flags on for everyone (saved value on, no stop, no targeting rules). A feature switched on by hand, with no release record, therefore still appears. If it was already on when the lineage began, the date is when the lineage began; if more than 500 changes have happened since, the date is the oldest one read. Both are later than the truth, never earlier. In production, flags the deployment controls are not recorded in the policy, so the history says nothing about them and the release record decides.
The later date wins. When the sources disagree, the later date is used, so a feature is never called ready sooner
than both allow, and the feature page notes the disagreement with both dates (dates more than an hour apart, or a day
against a Date-precision import, which stores the start of its day; a confirmation always lands seconds after its
commit). If the release record says live but the flags are not on for
everyone now, the feature is not listed and the page says so.
Where it shows (6G AD3, behind RuntimeFeatureReleases:Enabled): an Overview attention item per ready feature
("live for everyone for N days — ready to settle") and a "N features ready to settle" line on the summary card; a
"Ready to settle" lifecycle filter, status chip and banner on Features; and on the feature's Overview tab the
lifecycle step, a "Settling this feature" card with the settle period, the live-since and settle dates, the source and
any disagreement note. The Create clean-up issue link is built in the browser: a GitHub new-issue URL
(https://github.com/camaradesuk/syrf/issues/new?title=…&body=…, under 8 KB, no credential) prefilled with the
environment, the feature, its flags and owner, the dates, "set lifecycle: Released if it is not yet",
pnpm feature:settle <id> and the ordering-safety checklist below. Settling stays a reviewed pull request.
The read. GET /api/feature-releases/ready-to-settle (InspectRuntimeFeaturePolicy or ManageFeatureReleases,
Shared mode, 404 while RuntimeFeatureReleases:Enabled is off) returns, per candidate, featureId, flags,
lifecycle, status (ready, notYet, stopped or notLive), settleDays, liveSinceUtc (the later date),
releaseRecordSinceUtc, flagHistorySinceUtc, generalReleaseId, disagreement, releaseRecordNote,
flagHistoryCoverage, readyOnUtc and stopped, with the page's asOfUtc from the API's clock. It is read-only and
uses existing indexes, so it needs no migration. One request makes, after the binding check, at most four bounded
queries: the policy document by _id; the binding's released GeneralRelease records through {environmentId, status}
(0012, at most 1,000); their stop and restore events through {releaseId, recordedAtUtc} (0014, at most 1,000); and
the lineage's audit receipts newest first through {storeEpoch, revision} (0010, at most 500). More records or events
than the bound fail the read (503) rather than list from a partial read. Times: release confirmations are MongoDB time
and audit receipts the writer's clock; "now" is the API's clock. Host clocks keep NTP time, which is irrelevant at the
scale of days.
Ordering safety. Settling makes "on" unconditional. Before merging, confirm that every environment serves "on"
for each flag: the deployed value in cluster-gitops and any runtime override on that environment's Feature flags
admin page. Then deploying the clean-up changes nothing anywhere. After it deploys, remove the featureFlags.<key>
values from cluster-gitops; a removed key renders nothing, so this order is safe (production values need Chris).
Non-Production Runtime Overrides¶
The deployed GitOps value remains the default and recovery fallback. In staging and preview environments, a real SyRF administrator can set or clear an override through the API:
PUT /api/runtime-feature-flags/bulkPdfUpload
Content-Type: application/json
{
"value": true,
"expectedRevision": 12,
"reason": "Exercise bulk PDF upload in preview 2943"
}
Use "value": null to clear an override and return to the deployed value. GET
/api/runtime-feature-flags returns the full authoritative snapshot, revision, deployed and
override values, and activation metadata. Mutation is disabled in production.
The Feature groups section offers bulk PDF upload, reviewer workspace and supported statistics
application-use controls, with separate configured and effective On/Off/Mixed states.
Group membership comes from the API snapshot, which serves the flagGroups of the
feature registry. Enable/Disable group writes all members and
its audit entry atomically at the current revision; additional dependencies still require
confirmation. Individual flags remain available below. Statistics calculation/maintenance,
project readiness and explicit initial builds remain in the statistics operations workflow.
A failed or timed-out response may have committed: the UI refreshes without replaying it.
Use precomputed screening statistics (statistics-consumers-v2) includes serving, pages,
Project Overview, screening history and SignalR statistics refreshes. Enabling requires
deployed serving, maintenance and screening; live refreshes also need
SignalR connectivity. Existing per-project enrollment, freshness and fallback still apply.
Disabling stops these consumers while calculation/maintenance stays on. The exports flag
has no implemented export/report consumer, so it is excluded. Stage Overview and stage
history remain individual because staging has annotation maintenance disabled. Reviewer
history remains individual because it requires membership-screening maintenance, which
is also disabled on staging. Maintenance and the seven
metric-family gates remain deployment-managed across write hosts and are not group members.
The old statistics-screening-v1 and statistics-consumers-v1 requests remain accepted
with their original members for already-open clients; new snapshots advertise only v2.
The Bulk PDF upload and processing activity preset (bulk-pdf-upload-v1) changes
bulkPdfUpload and studyManagementProcessing together. The Processing activity page
also shows other study jobs; use the individual flags to control these separately.
Deletion handling in API and Project Management, notifier cleanup authority and a ready
PDF processing pipeline remain deployment prerequisites. Missing deletion handling blocks
the whole group change. Without notifier authority, a saved upload override remains
effectively off in the browser. Follow the
notifier authority activation runbook
before enabling uploads. Disabling this preset hides upload and Processing entry points;
it does not cancel existing jobs or disable cleanup authority. The notifier cleanup authority
(bulkPdfNotifierAuthority.enabled, deployment-managed) follows the same lifecycle rule: off
pauses clean-up and refuses deletes with nothing lost, and switching it back on is checked
against the recorded receipt's binding to its environment root and cleanup authority
(ADR-017 amendment 2026-10-17).
Default is the environment's deployed baseline, a saved change (override) replaces it, and the pill shows the effective value after dependency normalization and domain constraints. These overrides are environment-wide, not project enrollment or permission grants. Statistics maintenance and family gates are shown as deployment-managed instead of offering working-looking runtime buttons. An old override can still be cleared to the deployed default. Global statistics serving cannot be enabled unless deployed serving is already enabled. Use the statistics pilot administration workflow for project readiness, builds and consumer presets.
If a change needs other runtime-manageable flags, the first PUT returns 422
Unprocessable Entity with code: "featureFlagDependenciesRequired", the dependency set, and
canApplyAutomatically: true. After showing that set to the administrator, repeat the PUT with
the exact "confirmedDependencies" key/value set and the latest snapshot revision. Keep the legacy
"includeDependencies" field false: an older API replica will then reject the retry instead of
applying dependencies it cannot bind to that exact plan. The current server rejects the retry if
another API generation resolves a different dependency plan, rather than applying changes the
administrator did not review. It then applies the requested flag and every confirmed dependent
change atomically in one revision and audit entry.
The snapshot's effective values are authoritative even when a flag has no override. Loaded flags with unsatisfied prerequisites are normalized Off in both API and browser. If a later mutation would reactivate another dormant configured flag without confirming it, that flag is pinned Off by an audited override and must be enabled explicitly in a separate reviewed change.
Deployment-managed dependencies are different: their 422 response has
canApplyAutomatically: false and cannot be bypassed with includeDependencies. For example,
bulkPdfUpload requires DeletionLifecycle=true in both API and Project Management. Roll out and
verify that shared GitOps value across every consuming service before enabling the runtime override.
During a rolling deployment, wait until all relevant workloads are healthy on the new generation;
the API's local readiness check is deliberately only a fail-closed final guard, not proof that a
different service has completed its rollout.
Each change atomically updates the normal current-state override document and appends a separate audit entry. Authorization always checks the real signed-in user's administrator group, even while that administrator is impersonating someone else. The audit entry records both the real and impersonated identities plus the required reason.
After a successful mutation, the API broadcasts only the new revision over SignalR. Browsers then fetch a complete snapshot; they also poll every five minutes as a slow fallback. The API refreshes its cached snapshot periodically so backend gates converge across pods.
There is no SignalR backplane, so a broadcast reaches only the connections of the pod that sends it. In
Legacy mode only the writing pod broadcasts, after a legacy override change. In
RuntimeFeatureFlags:Mode=Shared (6B.4 G1) every API pod hints its own connections when its authority
reader observes a new policy identity (RuntimePolicyBrowserHint, called from
RuntimePolicyAuthorityRefreshService after each 8–12 s refresh). That is the only hint path. The writing pod
does not hint at commit time, because its browsers' refetch is served from this pod's reader, which would
still hold the previous revision. Instead a successful commit, emergency stop or ReleaseStop restore wakes
that pod's own refresh (IRuntimePolicyRefreshNudge), and the hint follows as soon as the reader has the new
identity. If that early read still sees the old identity, nothing is marked as hinted and the next regular pass
hints. Nudges coalesce (several during one pass cause exactly one more pass), the refresh never runs twice at
once, and after a nudged pass the next wait is a full 8–12 s delay. The message is the existing
RuntimeFeatureFlagsRevision invalidation; browsers refetch the snapshot and never use the hinted value. A pod
hints each store epoch and revision at most once and never for a successful start-up read (if the start-up
read fails, the first identity it reads later is hinted); a new store epoch after a re-initialisation or rebind
is hinted even when its revision is lower. A hint failure, or a hub that does not answer within 2 s, is
logged and changes nothing else: the refresh and acknowledgement carry on, and the commit receipt is never
affected (the nudge neither blocks nor throws). Tabs without a live hub connection still rely on focus,
online, reconnect and the five-minute poll.
Flag decision: not separately flagged. The hint runs only under the existing Mode=Shared switch, which no
shared environment runs, fires once per observed revision per pod and writes nothing, so switching Shared mode
off and on again is safe.
bulkPdfUpload is applied to both the Angular gate and the API endpoint gate from that same snapshot.
Runtime-manageable Boolean flags use the same endpoint. Most changes
apply immediately after the next snapshot refresh. The snapshot marks integratedPdfViewer,
signalRActive, zonelessChangeDetection, errorTracking and logRocketEnabled with
activationRequirement: "pageReload" because their viewer, transport, change-detection, diagnostic,
telemetry or session-recording lifecycles are bound during page initialization. Their overrides are
persisted immediately and become active when the browser is reloaded; the initial snapshot is applied
before those lifecycle services start. reviewEligibilityPolicy is also marked pageReload (review
configuration screens read the eligibility mode when they load) but it is not runtime-manageable: it is
deployment-managed because Project Management reads it too (see
Carrying a web flag to Project Management). It keeps its
ReloadRequired(...) catalog entry because a changed deployed value still reaches an open browser only on
reload. The authoritative list is ReloadRequiredFlags in
src/services/api/SyRF.API.Endpoint.Tests/RuntimeFeatureFlagCatalogTests.cs, which pins every
ReloadRequired(...) entry in the catalog. GitOps remains the deployed default and production source of truth.
See the administration delivery slices for current propagation boundaries, proposed grouped controls and retirement review criteria.
This MVP deliberately does not include percentage or audience targeting, typed non-Boolean configuration, event sourcing, projections, replay, or a richer audit-history UI.
Production runtime policy, release records, targeting and announcements are planned, not implemented: see the runtime feature releases plan and ADR-025 (Proposed). This guide describes current behaviour only.
The Features and releases area¶
/admin/features (6G AD1) gathers the runtime admin pages behind one side menu. Each page and tab is guarded
by its OD-2 capability, read from GET /api/account/capabilities (claim administrator where PM authority is not
enforced), and the API still decides every request:
| Page | What it shows | Capability |
|---|---|---|
| Overview | "Needs your attention" (features ready to settle, with a Create clean-up issue link; servers that have not confirmed the latest settings; features under an emergency stop) and summary cards. With nothing to report it shows one line: Nothing needs your attention. A failed Ready to settle read is named ("Couldn't check features ready to settle"); while releases are switched off the settle items are simply absent. The Releases card links to the Releases page; release attention items are a follow-up. | inspectRuntimeFeaturePolicy |
| Features | The registry (FEATURE_DEFINITIONS) with type, owner, lifecycle, live state here and links to the same feature in the other environments (D24). The lifecycle filter adds Ready to settle (see Ready to settle), a ready feature shows that status and its days live, and Needs action includes it. Read-only. |
inspectRuntimeFeaturePolicy |
Feature /admin/features/<featureId>/{overview,targeting,settings,history} |
Registry facts, live flag values and stop state; for a Release, the "Settling this feature" card (status, dates, source, disagreement note, and when ready the steps and the clean-up issue link); Targeting shows and changes the rules of a Project or Personal feature (see The Targeting page; a Global feature reads "Global: not targetable"); Settings shows "set in code" from the registry and a read-only "set here" until 6F G; History lists the feature's releases here (the Releases table filtered to the feature, first page, no admission check). | inspectRuntimeFeaturePolicy |
| Flags | The Feature flags page below, unchanged. /admin/feature-flags redirects here. |
manageRuntimeFeatureFlags |
Releases /admin/features/releases and /admin/features/releases/<releaseId> |
This environment's release records and one page per release (6G AD2, read-only; see The Releases screen). | manageFeatureReleases |
| Environment | The fleet and flag delivery panels below, plus the environment binding. The compatibility report is read only by the deployment's machine client, so it is not shown. | inspectRuntimeFeaturePolicy |
| Content (6G AD4) | Content preview: pick a feature and an audience (reviewer, project administrator, signed-out visitor) and see what that audience would get once the release is confirmed: the What's New entry (rendered by the same card people see), each help section and the walkthrough steps. Read-only; content is written in the repository. Offered only while RuntimeFeatureReleases:Enabled is on; while it is off the menu has no Content item and /admin/features/content goes to not-found. |
inspectRuntimeFeaturePolicy (the API also accepts manageFeatureReleases) |
Not flagged: administrator-only navigation over existing reads; every release action stays refused while
RuntimeFeatureReleases:Enabled is off.
Gradual rollout and Preview and check (6F W3). Two pages under a feature's Targeting tab, offered there only
when the tab's own read of the key's targeting (GET /api/runtime-feature-flags/{key}/targeting) has answered: a
Gradual rollout link in the rollout step (for an administrator who may change the policy) and a Check a
project (or Check a person) link in "Who gets it":
/admin/features/<featureId>/targeting/rollout(changeRuntimeFeaturePolicy): the saved share, state and group; an amount entry from 0.01% to 100% in steps of 0.01% (quick choices or Custom), stored as basis points (0.01% is 1, 100% is 10,000); Pause / Resume and Turn rollout off / Turn rollout on. Nothing applies on click: each change goes into a draft,…/rollout/previewpreviews it withPOST …/{key}/targeting/preview(a lower amount lists the projects that lose it, by name where the server gives one, before anything is saved, with the server's rollout notes), and the save sendsPUT …/{key}/targetingwith the preview digest and a required reason. The key's exclusions and allows are sent back exactly as read, because a targeting save replaces every rule of the key. A paused rollout holds its amount, so the amount entry is locked until it is resumed; Pause wording follows D32 C (only projects that existed when it was paused keep getting it). A key with no rollout can start one over projects (Project features only: rollouts over people are not supported yet)./admin/features/<featureId>/targeting/check(inspectRuntimeFeaturePolicy), and the right-hand panel of the rollout preview: Check a project (a person, optionally in a project, on a Personal feature) callsPOST …/{key}/simulateand shows the answer and the step that decided it, under the policy in force and, on the preview, with the draft side by side, then every step in a table. Owner decision D59 = B adds lookups: the project field searches by name or ID as you type (GET …/targeting/projects, debounced 300 ms, from two characters; each option shows the name, then "Owner … · n members · n stages · n studies · Created d MMM yyyy · short ID" as #4519's lookup does, leaving out what the server could not give and naming an owner the server could not name from the user list, else "unknown person" with a short ID; an archived project says so), and on a Personal feature the person field searches the Admin Console user list by name or e-mail address (GET /api/investigators, ListUsers, read into the store at sign-in; without ListUsers the field says to paste the ID). A pasted ID still works. An entry that is neither chosen nor an ID is refused in the browser. A refusal or failure is headlined in plain words (Chris, 2026-10-26), with the server's own message and reason code under a collapsed "Details", as on the Targeting page: an ID that matches nothing or a deleted one reads "No project with that ID exists." (or "No person with that ID exists.") and "You entered", with no answer beside it; an unknown reason reads "The check could not be completed.". The rollout preview's and save's refusals follow the same pattern. Owner rule (Chris, #4519): the pages refer to projects and people by name, never by a whole GUID. A chosen entry shows its name in the field; a pasted ID is named through the search's exact-ID match or the user list (the answer never waits for it: the caption shows the short ID until the name arrives); the result's caption, the step table and the lowering preview show names, with a short ID (first eight characters) only as quiet secondary text, and a project the server cannot name reads "Unnamed project" with its short ID.
While RuntimeFeatureTargeting:Enabled is off the targeting read answers 404, so both routes go to not-found and the
Targeting tab offers neither link. Flag decision: no new flag; the pages sit behind the existing targeting switch and
the capability guards, and the API decides every request.
Content preview rules¶
GET /api/feature-releases/content-preview lists the features with a What's New entry or article in the current
verified server bundle, or a walkthrough in the registry; GET /api/feature-releases/content-preview/{featureId}?audience=reviewer|projectAdministrator|signedOut
answers for one feature and audience (400 for another audience, 404 for an unknown feature). Both are read-only,
need InspectRuntimeFeaturePolicy or ManageFeatureReleases, answer 404 feature_availability_disabled before
authorization while RuntimeFeatureReleases:Enabled is off, and read only the registry and the cached bundle. The
answer applies the disclosure rules with the release taken as confirmed and the feature as on:
- Who it is for is the registry
audience(6R.A):signedInandprojectMembersreach reviewers and project administrators,projectAdministratorsonly project administrators,applicationAdministrators(or no audience) neither. Signed out reaches nothing personal. - What's New, for a feature that announces: a reached audience sees every stage its entry covers, with "New to you" when the feature notifies its release audience; everyone, signed in or not, sees what the public projection (6G.6) shows: a Global feature's entry at general release when its topic has a public guide page.
- Help: public classes for everyone, Draft for nobody, restricted classes (Private pilot, Retired) only for a reached signed-in audience and never signed out. The preview shows the topic's section of the page (from its heading to the next heading of the same level), or the whole page for a whole-page topic.
- Walkthrough: only signed in, for an announcing feature that reaches the audience. The steps come from the web
build's
walkthroughs.generated.ts. - A quiet or Withdrawn feature has no entry and no walkthrough. Left-out content carries only its IDs and a reason, never its text.
The Releases screen¶
Where to see releases. Admin console → Features and releases → Releases (/admin/features/releases), or a
feature's History tab for that feature only. The page reads the release read routes
(6G AD2a, #4476); 6G AD2b (#4518) is the page itself. It is read-only: preparing, applying, pausing and stopping from
here come with the Apply page in a later PR, and emergency stops and restores stay on the Flags page.
- List. Newest first, 50 a page (Show older releases follows
nextBefore), with tabs Needs action (Prepared, waiting for servers, servers need action, paused or stopped, or an announcement rowFailed), All and Live (Active). Each row shows the feature, its type, owner and stage; the status (Prepared, Ready to apply, Can't apply yet, Applying, Confirmed live, Waiting for servers, Servers need action, Paused, or Stopped while an emergency stop is in force on the feature's flag); the milestones; and the announcement state per channel: What's New (MaterialiseWhatsNew), in-app notice (NotifyInApp) and walkthrough. Outbox states read Pending, Sent, Cancelled by stop (CapabilityStopped) or Failed; rows written before D60 A (26 Oct 2026) can also read Sent to test sink (staging) or Not sent: delivery off here (DeliveryDisabled, previews and local stacks), which stay as they were. With the stored cancel reason or error class shown verbatim beside the label. A Prepared release shows the registry's plan (Queued on apply or None planned); an applied release without a row says Not queued (a quiet Apply, or a feature that does not announce that way). Walkthroughs have no outbox row, so they read None planned or Not tracked yet; an Operational feature reads "Not announced: operational features are invisible to users". - Admission check. For each Prepared release on the page (at most 10, one at a time) the screen runs the
side-effect-free Apply preview (
POST /api/feature-releases/{releaseId}/apply/preview,ManageFeatureReleases) with the feature's default announcements, against the current policy identity, and shows Ready to apply or Can't apply yet with the refusal. The release page shows the refusal's reason code verbatim and the server's text for it (RuntimeReleaseAdmission.Message; a 422 refusal's ownmessagewhen the server sends one). Nothing is applied or recorded; the preview's reason reads "Admission check from the admin Releases screen (Apply preview only; nothing is applied)". - Release page. Status, the progress strip (Prepared, Applied, Confirmed live), milestones as exact times in Europe/London with the zone (requested, scheduled, apply requested, confirmed, recorded, how it was confirmed, precision), the release key, ID, stage, record version and pinned help bundle, the announcement rows (state, attempts, completed time, cancel reason, last error class, and "Delivered to: Test sink" only on a staging row the retired test sink completed before D60 A), and the event timeline newest first with each event's request reason in quotes, verbatim. An imported historical release has no record (its detail GET is 404), so its page shows the timeline only.
- States. Loading; empty ("No releases in
yet"); switched off (404 feature_availability_disabled: "Releases are switched off in"); refused (401/403: "You can't see releases here", with the server's title, such as "Release history requires Shared API authority mode.", quoted verbatim); not found; unavailable (503, with Try again).
Flag decision: not separately flagged. Every read sits behind RuntimeFeatureReleases:Enabled (off everywhere), and the
page only reads.
Using the Feature flags admin page¶
The page at /admin/features/flags (redesigned in #4385; /admin/feature-flags redirects there) is a summary
card above one list of flags.
Summary card. One line says whether the servers have the current settings: All servers up to
date (only when the server reports the fleet Active), Updating: 2 of 4 servers have the latest
settings while hosts adopt it, Only 2 of 4 servers have the latest settings after the one-minute
confirmation window, or Server status unavailable. A saved policy is not adoption, so a save never
reads "up to date" until the fleet does. The line re-checks every 10 seconds while servers are updating,
for up to two minutes. Legacy mode says server status is not tracked; a read-only environment says so
here once instead of on every row. In Shared mode a second line gives the emergency-stop state of the
membership control (Membership disable: stopped, not stopped, updating… or stop state
unknown), read from authority.stoppedKeys. Emergency stop… or Restore… opens a dialog (both
appear when the state is unknown or updating). Details is collapsed by default. It holds the mode,
policy revision, the last saved revision while the page catches up, the stop-state explanation, the
fleet host table (each host state in words, such as Up to date or Catching up), flag delivery status
and the explanatory notes. Its diagnostics load only when it is opened. A revision the page has saved
stays the floor until a snapshot reaches it, even if its receipt is dismissed or replaced, so a lagging
snapshot never reads "not stopped" straight after a stop. Policy identities compare GUIDs and digests
case-insensitively, as the snapshot guard does.
List. Search matches the key, name, description and category. Show filters to All, Changed from default (a saved override) or Can't change here. Each row has the name, a one-line description, an On/Off pill (marked changed, stopped or updating only when it is; a row at its default shows just the value), and a Change menu (Turn on, Turn off, Use default (On/Off); the current choice is disabled). Selecting the name expands the row to show the full description, key, default, saved change, reload requirement and dependencies. A deployment-managed flag shows a lock and its reason instead of a menu. In Legacy mode its menu offers only Use default, to clear an old override. Feature groups is a collapsed section below the list; each group has Turn all on and Turn all off in Legacy mode and shows why it is locked otherwise.
Change dialog (Shared). The dialog opens straight away and asks the server for a preview. It then states the effect in one sentence ("Off → On for everyone in this environment"), lists any other flags the preview changes, and offers an optional comment. The preview digest covers the audit reason, so editing the comment requests a fresh preview half a second after typing stops; the primary button is enabled only while the preview on screen matches the comment. Confirming commits exactly that previewed request and digest. Cancel, Escape or closing the dialog commits nothing. A dependency refusal, at preview or at commit, shows the needed flags in the same dialog; Review combined change previews again with the confirmed set and still needs confirmation. A revision conflict closes the dialog and reloads current values. When saved, the dialog closes, a short message says "Servers update within about a minute", and the row shows the new value marked updating until the API's reader catches up (the page re-reads quietly every 3 seconds, up to five times).
Outcome unknown. If the connection drops before the server answers, the dialog says it is not sure the change was saved and offers Check outcome and Retry unchanged request (the same mutation ID, so it cannot apply twice). Nothing is committed automatically. If the dialog is closed, a banner keeps both actions and the mutation ID on the page, and other changes stay blocked until the outcome is known. The stop has the same banner with Retry unchanged stop.
Reload recovery. Before a change, restore or stop is sent, the page saves a small record in the tab's session storage: the kind, key, mutation ID, the policy identity the request expects and (for a change or restore) the identity its preview proposed. It holds no value, reason, comment or preview digest, and it is removed once the outcome is known. If the page reloads first, it reads the current policy, then looks up the original mutation ID; both are reads, and nothing is previewed, committed or stopped again. A committed request shows as saved. If it was not committed and the policy has since moved on, an information banner says it wasn't saved and can't be saved now (every commit checks the identity it expects). Otherwise the banner offers Check again and Discard; ordinary changes are paused until one of them settles it, and the stop stays available while a change is unresolved. While the page's own earlier stop is unresolved, Emergency stop… is unavailable but stays focusable, and its tooltip says to check the earlier stop first. When a recovered restore is saved while a remembered stop is still unresolved, the page checks that stop again (reads only), so its own receipt settles it. A blocked session storage leaves the page as it was without this record. This path is not flagged: it is administrator-only, reads only, and does nothing unless a record exists.
Emergency stop and restore. The stop dialog says what the stop does and needs a reason. Stop re-reads the current policy and then sends the stop, so the dialog is the confirmation. Restore previews the release on the server, shows the value the control goes back to, and needs a reason before it can be confirmed; it is blocked while a stop request is unresolved.
Legacy mode uses the same list and a simple confirmation dialog with an optional comment, because the
legacy PUT has no preview. Dependencies the change needs are listed in that dialog. A revision conflict
reloads current values and requires a fresh action rather than silently overwriting another
administrator.
Schema Properties Reference¶
| Property | Required | Description |
|---|---|---|
name |
Yes | Environment variable name (SYRF__FeatureFlags__*) |
valuePath |
Yes | Path in Helm values.yaml (e.g., featureFlags.myNewFeature) |
default |
Yes | Default value as string ("true" or "false") |
web.tsName |
Yes | camelCase name used in TypeScript (e.g., myNewFeature) |
web.tsType |
Yes | TypeScript type (always boolean for feature flags) |
featureFlag |
Yes | Must be true to mark as feature flag |
displayName |
Yes | Sentence-case label shown on the admin runtime feature flags page (generation fails without it) |
description |
Yes | Human-readable description (used in JSDoc comments) |
category |
Yes | Grouping for organization (see categories below) |
adminOnly |
No | If true, flag requires admin UI to be visible |
Categories¶
Use existing categories for consistency:
| Category | Purpose |
|---|---|
projectSettings |
Project configuration features |
uiFeatures |
General UI features |
robFeatures |
Risk of Bias related features |
dataFeatures |
Data import/export features |
development |
Development/debugging features |
monitoring |
Observability features (Sentry, LogRocket) |
Admin-Only Flags¶
Flags marked adminOnly: true are automatically gated by the admin UI state:
- name: SYRF__FeatureFlags__ExampleAdminAction
valuePath: featureFlags.exampleAdminAction
default: "false"
web:
tsName: exampleAdminAction
tsType: boolean
featureFlag: true
category: robFeatures
description: "Example admin action (illustrative, not a deployed flag)"
adminOnly: true
Generated selector automatically combines with admin state:
export const selectExampleAdminAction = createSelector(
selectConfigState,
selectShowAdminUiOptions,
(configState: AppConfig, showAdminUi: boolean): boolean =>
configState.exampleAdminAction && showAdminUi,
);
Generator Commands¶
cd src/services/web
# Generate all files
pnpm run generate:flags
# Validate schema without writing (CI/pre-commit friendly)
pnpm run validate:flags
# Preview changes without writing files
pnpm run generate:flags -- --dry-run
# Show verbose output including Helm env vars
pnpm run generate:flags -- --verbose
Pre-commit Hooks¶
Two pre-commit hooks validate generated code when relevant files change:
1. validate-env-blocks (syrf-common)
Triggered by changes to:
src/charts/syrf-common/env-mapping.yamlsrc/charts/syrf-common/templates/_env-blocks.tpl
2. validate-feature-flags (web)
Triggered by changes to:
src/charts/syrf-common/env-mapping.yamlsrc/services/web/src/app/generated/feature-flags.*.tssrc/services/web/src/assets/data/env.template.generated.jsonsrc/services/web/.chart/values.yaml
If validation fails (files are stale), the commit is blocked until you run the appropriate generator.
CI Validation¶
The validate-generated-code job in pr-tests.yml runs both validators:
- Detects if schema or generated files changed
- Runs
pnpm run validate:env-blocksin syrf-common - Runs
pnpm run validate:flagsin web - Reports status in PR test summary
Why validate-only (not auto-generate)? Generated TypeScript could break the build. Developers should run the generator, review changes, test the build passes, then commit.
Note: The web chart's deployment.yaml env vars are generated by syrf-common/scripts/generate-env-blocks.ts, not the web generator. The web chart uses shared templates via {{ include "syrf-common.env.webConfig" . }} and {{ include "syrf-common.env.featureFlags" . }}.
The generators are idempotent - running them multiple times produces identical output. This ensures consistent validation in CI and pre-commit hooks.
How It Works at Runtime¶
- Build time: Helm templates reference
values.yamlpaths - Container startup: NGINX envsubst replaces
${SPA__*}placeholders inenv.template.json(the image build copiesenv.template.generated.jsonin under that name) - App bootstrap:
main.tsfetches config JSON files - Config parsing:
parseFeatureFlags()converts string values to booleans - State management: NgRx store holds parsed config
- Component access: Selectors provide typed access to flags
Helm values.yaml → deployment.yaml env vars
↓
Docker container
↓
NGINX envsubst → appConfig.env.json
↓
main.ts fetch()
↓
parseFeatureFlags() → AppConfig
↓
NgRx Store (config state)
↓
selectMyFeature selector
↓
Component signal/observable
Architecture: AppConfigService Integration¶
The AppConfigService cleanly separates generated code (feature flags) from manual code (derived values):
// app-config.service.ts
// AppConfig = FeatureFlags (generated) + AppSettings (manual)
export interface AppConfig extends FeatureFlags, AppSettings {}
// ExternalConfig = FeatureFlagsExternalConfig (generated) + ExternalConfigSettings (manual)
export interface ExternalConfig extends FeatureFlagsExternalConfig, ExternalConfigSettings {}
static loadConfig(externalConfig: ExternalConfig): void {
// Generated: parses all feature flags using schema defaults
const featureFlags = parseFeatureFlags(externalConfig);
// Manual: complex derived values (URLs, version strings, etc.)
const appSettings = buildAppSettings(externalConfig);
this.staticConfig = { ...featureFlags, ...appSettings };
}
Why Some Values Aren't Generated¶
Non-feature-flag configuration (AppSettings) requires manual logic that can't be generated:
| Property | Reason |
|---|---|
displayVersion |
Environment-specific formatting (30+ lines of logic) |
auth0Domain |
Computed from {tenant}.{region}.auth0.com |
apiUrl |
Derived from apiOrigin + '/api' |
protectedUrls |
String split + array append logic |
randomId |
Runtime-generated browser session ID |
File Structure¶
src/app/
├── generated/ # AUTO-GENERATED (don't edit!)
│ ├── feature-flags.types.ts # Interfaces only (no JS output)
│ ├── feature-flags.constants.ts # FEATURE_FLAG_DEFAULTS, env vars, helm paths
│ ├── feature-flags.selectors.ts # NgRx selectors
│ └── feature-flags.parser.ts # parseFeatureFlags()
│
└── core/services/config/
├── app-config.service.ts # Main service (uses generated code)
└── app-settings.interface.ts # Non-feature-flag types (manual)
Note: types.ts contains only TypeScript interfaces (erased at compile time). All runtime values (as const) are in constants.ts for clear separation.
Troubleshooting¶
Generated files out of sync¶
Validation fails in CI¶
The validate-generated-code job reports which files are stale:
This checks if values.yaml and the TypeScript files match the schema.
Pre-commit hook fails¶
# Run the generator to update files
pnpm run generate:flags
# Check what changed
git diff
# If TypeScript changed, test the build
pnpm run build
# Then stage the generated files
git add src/app/generated/ .chart/values.yaml
Flag not working in deployment¶
- Check Helm values in cluster-gitops
- Verify deployment has correct env var:
kubectl exec ... -- env | grep SYRF__FeatureFlags - Check browser dev tools → Network → fetch for config JSON
- Verify flag value in parsed config (browser console)
TypeScript build fails after generation¶
Generated code might reference types that don't exist yet. Check:
- Is
selectConfigStateexported from the config selectors? - Is
selectShowAdminUiOptionsavailable for admin-only flags? - Are NgRx imports correct in the generated selectors file?
Related Documentation¶
- Environment Variable Code Generation - The broader env-mapping system
- Environment Variables Reference - Complete env var listing
- How-To: Extend Env-Mapping Schema - For non-feature-flag config
- ADR-023: Pre-deploy database migrations - Migrations and flags are separate (amendment 2026-10-17)
- Operate runtime feature flags - Operator runbooks: binding, lineage import, enrolment, collector, coordinator, inventory, change, stop, restore, recovery and Legacy return (written, not executed)
- Staging rehearsal plan and production prerequisites - Not executed
- Isolated rehearsal checklist - How each runbook step is exercised in the local stack
Shared reader: first API/PM migration slice¶
The shared reader is an opt-in non-production migration seam. Its independent deployment setting is
RuntimeFeatureFlags:SharedReaderMode (SYRF__RuntimeFeatureFlags__SharedReaderMode in the host environment).
Configure it through the existing environment/GitOps configuration; it is deliberately not a runtime flag
that could disable its own provider. The default is Disabled, preserving existing behavior.
| Mode | API | Project Management |
|---|---|---|
Disabled |
Existing runtime provider and behavior | Existing startup/domain behavior |
Shadow |
Observe persisted global overrides without changing consumers | Observe the same persisted overrides without changing domain behavior |
Membership |
Use the shared snapshot for the existing disableMembership action-selector gate |
Continue shadow observation; no membership or reviewer domain control is changed |
StatisticsServing |
Invalid for this host | Use the shared materializedProjectStatisticsServing value as an additional stop for precomputed statistics serving |
StatisticsServing also uses shared overrides for the Pages, Stage Overview, SignalR and Exports
consumer selectors. These are reversible consumer requests, not evidence that any project is Fresh
or allowlisted. disableMembership gates the auditable disable endpoint; re-enabling an already-disabled member stays
available with the flag off. It does not remove membership authorization or rewrite persisted membership state. Each route decision captures one immutable
observation. Already admitted requests complete under the existing application protocol. All other API
consumers continue using their existing provider. In StatisticsServing mode PM keeps writes, family
gates, project allowlists and reviewer-capacity controls deployment-owned. Shared serving and consumer
values can stop precomputed reads, but cannot enable them when deployment serving or writes are off. The durable
statistics control and freshness checks still apply. An unavailable or stale observation stops new
materialized reads in PM; it does not stop maintenance or undo an admitted operation. This makes the
existing Use precomputed statistics admin switch meaningful in both API and PM on an enrolled
non-production deployment. Calculate and maintain statistics is still a separate, deployment-bound
control; its coordinated runtime transition remains follow-up work.
Both hosts reconcile the existing global Mongo override document every 15 seconds with majority reads and
a five-second read deadline. There are no change streams in this first slice. Complete frozen snapshots
replace previous snapshots, so resetting a flag removes the old override. Missing/malformed documents,
read failure, revision regression, conflicting content at an equal revision and observations older than
60 seconds produce an unknown/unavailable decision. The enrolled API route fails closed in those states;
the last observation remains available for diagnosis. A successful later reconciliation recovers. Cold start
with no persisted state does not synthesize an authoritative revision zero: first initialize the legacy
state through an ordinary reviewed flag edit while in Shadow mode, then compare before selecting Membership.
The admin flag page exposes GET /api/runtime-feature-flags/provider-status to real administrators only.
It distinguishes the independently fetched saved revision from the local observed revision, mode, state,
observation time and nullable evaluated value. Current means a recent successful local observation, not
fleet convergence. PM also exposes a limited read-only GET /api/runtime-feature-flags/reader-status
containing only local mode, observation state/revision/time and the effective statistics-serving decision;
the same Boolean policy is already public in the API snapshot. The admin page displays that result
separately from the API result. PM retains the runtime-feature-flag-reader diagnostic health entry.
Neither endpoint is a fleet acknowledgment. There is no fleet registry, and the UI explicitly reports
fleet state as unknown.
The legacy-global-overrides digest covers only sorted override content. It excludes deployed defaults,
catalog metadata and environment identity. Matching digests do not establish equal effective values
when hosts have different defaults. PM health includes its selected consumer's deployed default and
evaluated value. This is not the approved-baseline/epoch authority described by the broader architecture;
a document carrying the future storeEpoch field is rejected by this legacy reader until its migration
is implemented. Production and unknown/conflicting deployment identities reject opt-in modes at startup.
Before rollback to Disabled, compare the saved revision, the legacy provider's observed value and the
shared reader's value for disableMembership; wait for legacy convergence or write a reviewed forward
revision first. Do not use rollback to bypass an unavailable authority or restore a stopped capability.
For PM StatisticsServing, first make the deployment fallback agree with the reviewed runtime serving
decision. Otherwise reverting the reader to Disabled can reopen precomputed serving despite an Off
override. No deployment setting or runtime policy is changed merely by merging this implementation.
Consumer inventory guard¶
pnpm run validate:generated also checks the C# typed consumer inventory in
consumer-manifest.json. New or changed common
flag seams and reads through typed/inferred aliases require a reviewed classification, owner, scope and
activation boundary. Changes to _flags.SomeGate in an already classified file are detected. This guard
is conservative static evidence, not proof of every dynamic configuration lookup or external worker;
those remain explicit audit work before migration. Update the manifest after reviewing a change, never
regenerate it in CI to accept a new read automatically. In PR Tests, the Validate Generated Code lane runs for any change to
the manifest, the validate:generated scripts or a .cs file under src/libs/ or src/services/, so a
changed flag read cannot merge unclassified.
Since R3c the guard also enforces reading flags through IFeatureDecisions. A
production file that names IRuntimeFeatureFlagProvider, SharedRuntimeFeatureFlagReader or
RuntimePolicyAuthorityEvaluator in code, or an existing-runtime consumer that reads FeatureFlags, fails
unless the manifest's legacySeamAllowList names it with a kind and a reason. The kinds are implementation
(the decision paths and their registration), control-plane (diagnostics and reports),
deployed-configuration (a start-up value read beside runtime decisions) and pending-migration. An entry that
no longer needs to be there fails as stale, so the list only shrinks. The pending migrations are the five
domain callers of IRuntimeFeatureFlagProvider.IsEnabled (ReviewController, StageAllocationReadController,
StageWorkloadSharesController, AnnotationFoldTargets, SubmitAnnotationSessionService) and the four
statistics history controllers. The history controllers read FeatureFlags as the runtime provider mutates it:
moving them to IFeatureDecisions would decide materializedProjectStatisticsPages from the bounded authority
in Shared mode, which changes their staleness behaviour, so it needs its own review.
The legacy store never interprets or overwrites a document containing storeEpoch, including a malformed
epoch (owner decision D7=A, spec 6C.5). A host in Legacy mode (or production with
ProductionPolicyMode=Legacy) that finds one starts normally and serves the deployment-controlled values
for every key: the authority's saved overrides and stops are preserved but not applied. From that moment the
process is a read-only legacy reader for its lifetime: snapshots carry policyState:
DeploymentControlledAuthorityPreserved and mutationAllowed: false, legacy changes answer 409
runtime_policy_authority_preserved without writing, provider status says the runtime policy is preserved but
not applied, authority-status reports reason LegacyModeAuthorityPreserved, and the process marks its own
syrf-api writer-registry entry runtimePolicyReadOnly so it does not block a later lineage freeze. The
legacy-format shared reader (API membership, PM statistics) likewise serves deployed defaults in state
AuthorityPreserved. Returning the hosts to Shared resumes the unchanged authority at the same identity
(runbook 10 of Operate runtime feature flags). Older
images that predate this behaviour still refuse to start against an imported authority, and their registry
entries are always counted by the freeze. Stop/freeze older writers before lineage import.
Runtime-policy reader/writer boundary guard¶
Runtime-policy code has two sides. The reader side (snapshots, evaluation, catalog, host acknowledgements and the
host's own writer-registry entry) lives in SyRF.Mongo.Common/RuntimeFeatureFlags and RuntimeFeatureReleases;
every service ships it, and PM publishes its own heartbeat. The writer, administration and release side (the
mutation store, flag store, lineage import and its entry point, release intent, the environment-report and workload
stores, the writer freeze) lives in the API-only project src/services/api/SyRF.API.RuntimeAdministration, which
only SyRF.API.Endpoint references in production. Its types keep their SyRF.Mongo.Common.* namespaces. Logic both
sides need (receipt validation and digests, effective values, deployment-constraint and observation validation) is
reader-side, and the module reaches the reader's internals through InternalsVisibleTo.
RuntimePolicyBoundaryArchitectureTests in SyRF.ProjectManagement.Endpoint.Tests fails if:
- PM's assembly closure contains the module, or any type the module declares;
- a project other than those in its
ModuleReferencerslist references the module (the API, plus two test projects that use the writers for setup); - a PM production source, or a reader-side type, names a writer-side type (every type the module declares). The
ReaderReachInsexception list is empty; - a reader-side type writes MongoDB documents, indexes or collections (host acknowledgements excepted).
The source scans ignore comments but not string literals, and assume file-scoped namespaces (a test checks that).
To read a failure, look at the named consumer, project or assembly. Route the consumer through a reader-side type,
moving shared logic into a reader-side file, or move writer code into the module. A new test project that needs the
writers goes into ModuleReferencers with its reason. Design: spec 6R.3 (reader/writer module split).
Reviewed runtime-policy lineage import¶
The step-by-step operator procedure for this command, with preflight checks and refusals, is runbook 2 of Operate runtime feature flags.
PR #4347 provides an explicit operator command in the API image for the Increment B legacy-lineage transition. It is separate from schema migrations and normal startup. It does not activate runtime modes, enroll consumers, establish fleet readiness, or authorize a production operation. Coordinate its use with an authority-capable reader/writer rollout; the legacy store and shared legacy reader refuse epoch-bearing state. Do not use it as a standalone live-environment rollout.
The preview requires an existing legacy policy, a verified environment binding, and a reviewed baseline JSON file with these fields:
values: a Boolean value for every key in the generated shared runtime catalog; unknown keys and missing keys are rejected. Reconcile all consuming services' rendered/deployed inputs before approving this artifact. The command never chooses the API's current settings as the environment baseline.approval: the reviewed baseline decision/reference.provenance: the source configuration/report identities used to reconcile it. These are operator review evidence, not an assertion that a compatibility collector or fleet acknowledgement already exists.
Use the existing Mongo configuration and RuntimeEnvironment, plus the binding expectations used by the
migration tooling: SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT, ..._DATABASE, and ..._CLUSTER.
The expected cluster must come from a verified binding/status result. Never substitute a guessed target.
To produce a read-only preview, run the API image's normal entry point with:
SYRF_RUNTIME_POLICY_MODE=preview
SYRF_RUNTIME_POLICY_INPUT=/reviewed/baseline.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/lineage-preview.json
The output contains the source revision/digests and audit references, accepted baseline and target identity, new store epoch, mutation ID and preview digest. Review and retain this exact artifact. It uses Canonical Extended JSON to preserve BSON integer and identifier types. Editing or reformatting through a tool that changes BSON types invalidates it; generate and review a new preview instead.
Before apply, freeze deployments/autoscaling and stop all legacy policy writers. Hold that operational
freeze throughout the command. The command also requires an existing, well-formed syrf-api registry with
zero registered instances; missing registry data and stale/live registrations refuse. There is no bypass
switch. This registry check is evidence of stopped instances, not a Kubernetes deployment lock.
Apply the exact reviewed preview with:
SYRF_RUNTIME_POLICY_MODE=apply
SYRF_RUNTIME_POLICY_INPUT=/reviewed/lineage-preview.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/lineage-receipt.json
SYRF_RUNTIME_POLICY_ACTOR=<operator identity>
SYRF_RUNTIME_POLICY_WRITER_FREEZE_ACK=<reviewed maintenance/freeze reference>
Apply requires migration-owned audit indexes 0009 and 0010 with their expected definitions; it never
creates indexes. It verifies the binding and source again, preserves the numeric revision and exact
overrides, and commits the authority document and receipt together with snapshot reads and majority write
concern. The receipt references the old audit rows; those rows remain untouched. Missing state, a changed
source, an unknown legacy field or another lineage refuses rather than silently initializing/resetting.
Input and output paths must differ, and output must not already exist. If apply loses its response or cannot write its output artifact, retry the identical reviewed preview with a new output path. The persisted mutation receipt resolves the original result without another policy write. A different preview using the same mutation ID is refused. A receipt proves that import committed, not that any consumer has accepted the lineage or that the fleet is ready. Restore/rebind and later baseline changes require their own reviewed transitions; this command never rotates an existing lineage.
First-time initialisation (no legacy document)¶
Where no global flag document exists (production: legacy runtime mutation is disabled there, so there is
nothing to import), the same command takes SYRF_RUNTIME_POLICY_SOURCE=initialize (owner decision D8=A,
slice 6C.6). Unset or import, it is the import above, unchanged. Initialise creates revision 0 from the
deployed values: the FeatureFlags configuration the command itself renders, for every catalog key, with
no overrides and no stops, bound to the verified binding. Its input carries only approval and provenance.
The preview is deterministic (the store epoch and mutation ID are derived from the reviewed content), and apply
needs the reviewed digest, a new receipt path, the same deployed values and migration catalogue as the preview,
the writer freeze, and the receipt kind deployedValueInitialization. It refuses any existing global document
(legacy or authority), an unverified binding, registered writers, and migrations that are not Ready (the API
image's project-management catalogue, checked as SYRF_MIGRATION_MODE=verify does). An exact replay returns the
original receipt. Readers accept the new receipt kind only for revision 0 with no overrides and no stops. Steps,
expected output and every refusal: runbook 2, Initialise.
Read-only authority observation¶
PR #4348 adds an optional observer for the imported authority in API and Project Management. The default
RuntimeFeatureFlags:Mode=Legacy registers no refresh worker and makes no authority-source queries.
Shadow is read-only and requires matching non-production identities plus all three pinned application
configuration values: RuntimeFeatureFlags:Binding:Environment, ...:Database, and ...:Cluster.
Use the verified binding/status evidence for these values. These are deployment/application settings,
not live feature-flag controls; no Helm value is introduced by this slice. #4352 adds the explicit
non-production Shared consumer mode described below; production Shared mode remains refused.
Each process reconciles every 15 seconds with a five-second read deadline. It reads the current envelope, binding and current mutation receipt coherently, validates the receipt and all digests, then atomically publishes an immutable snapshot. Legacy or missing state is never Ready. After initialization, a missing/legacy document, changed environment/binding/store epoch, revision rollback or equal-revision/different identity quarantines that reader instance. There is no automatic rebind or quarantine reset; complete the reviewed recovery procedure before restarting. Ordinary read failure immediately removes decision availability. Snapshots expire after 60 seconds measured with a monotonic clock, so moving wall time backwards cannot extend trust. An operation can retain one immutable valid capture while later operations see the refreshed observation.
GET /api/runtime-feature-flags/authority-status is authenticated and requires the current
InspectRuntimeFeaturePolicy decision (legacy claims only in the existing Off/Shadow
authorization modes). It returns local-process state, identity, timestamps and reason, with revisions as
decimal strings. It returns no policy values, targeting lists or fleet-readiness claim, and does not query
authority on demand. A valid local snapshot does not mean every expected host has acknowledged it.
PR #4349 extends receipt validation to ordinary global authority mutations as well as lineage import. Existing feature consumers remain on their existing paths; this observer does not enroll them or make a live lineage import operationally safe by itself. #4352 adds local Shared admission and #4358 adds host acknowledgements; scoped compatibility reporting remains subsequent work.
Idempotent authority mutation primitive¶
PR #4349 adds RuntimePolicyMutationStore in SyRF.Mongo.Common. It is not registered as a replacement
for the legacy API store and exposes no additional public mutation endpoint. API wiring must preserve
current permission checks and introduce the reviewed Shared-mode admission protocol; constructing this
library service does not enroll consumers or enable production policy.
A request carries a mutation ID, complete expected policy identity, flag change and real/impersonated actor. The store requires a verified non-production binding, the existing migration-owned audit indexes, and a receipt-validated initialized authority. It reuses dependency planning and deployment compatibility checks, preserves the accepted baseline and unrelated overrides, and atomically commits the new envelope and immutable receipt with snapshot reads and majority writes. The deployment constraint guard is preserved and incremented. PR #4360 interprets receipt-verified schema-v1 deployment constraints: compatible changes proceed, while saved enabled keys outside the capability intersection refuse. Unknown or inconsistent constraints still refuse ordinary changes. Emergency stops remain independent of constraint evidence.
Concurrent editors at one identity produce one commit and one revision conflict. Retrying the identical
mutation ID returns its original receipt, even after subsequent changes; reusing that ID with different
payload or actor refuses. The receipt retains the complete original request and resulting envelope.
FindReceiptAsync resolves the outcome without another write. Unknown commit outcomes are not converted
into a success or a known-aborted conflict: retry the same request to resolve them. The read-only observer
accepts both import and ordinary mutation receipts, and rejects inconsistent receipt data. Receipt success
proves persistence only; fleet acceptance remains a separate coordinator responsibility.
Deployment constraint transactions¶
PR #4360 adds the bound ChangeDeploymentConstraintAsync shared-store primitive. A trusted coordinator
can record one active rollout, extend its expiry or release it with an exact reviewed constraint digest.
Every operation requires the complete current policy identity, a mutation ID, an unimpersonated actor and
reason. Record validates the serving catalogs, target catalog and global schema-v1 enabled-key capability
intersection against saved policy. A stopped feature retains its saved capability requirement.
Constraint operations increment the same constraintGuard as policy edits inside a snapshot/majority
transaction. They leave policy revision, baseline, values and policy digest unchanged. This invalidates old
previews without falsely indicating a policy change to hosts. An expired constraint keeps blocking
incompatible changes; expiry is never a release. Ordinary extend/release requires the original coordinator.
Break-glass is a distinct audited action whose caller must enforce its separate administrator permission.
Receipts make identical retries idempotent, including a retry of record after release, and detect corrupted
active constraint evidence. Emergency stop continues even when constraint evidence cannot be interpreted.
This is a transaction-layer increment, not a deployment endpoint. It inherits explicit non-production admission; Legacy remains default. No new collection, index, startup write, migration or environment setting is introduced. The next transport increment must authenticate environment-bound machine scopes, construct capability intersections from trusted reports and require collector evidence of rollout/rollback completion before normal release. The administrator break-glass capability, expired-constraint status in the fleet UI, and report/coordinator endpoints are still required. Never expose this primitive directly to policy editors.
Scoped compatibility reports¶
PR #4361 adds GET /api/environment/compatibility-reports/current, protected by the dedicated
environment:report:read machine scope. Reporting defaults off. An enabled API requires
RuntimeFeatureFlags:CompatibilityReport:Enabled=true, an explicit Provider (OpenIddict or Auth0),
ClientId and Audience in that same section, plus the existing exact RuntimeFeatureFlags:Binding
expectations. The report audience must match that provider's configured API validation audience, and the
runtime/active environment must agree with the binding. This does not select Shared policy mode: Legacy
can report that no accepted bound baseline exists without inventing one.
For staging Identity, the dedicated client is syrf-environment-report. Its only grant is client credentials
and its only scope is environment:report:read; configure EnvironmentReport:ClientSecret through the
normal secret owner to enable its registration. Missing, placeholder or weak secrets leave it disabled
and revoke managed registrations holding that scope, using the same lifecycle as the statistics evidence
client. No secret or environment value is supplied by this PR. Production Auth0 provisioning remains a
separate approved tenant change; the API accepts only the configured client-credentials subject, client ID,
audience and scope through the existing validated bearer handler. User/BFF principals, mixed scopes and
other clients cannot read a report. A report token cannot authenticate on ordinary API routes, even if it
also carries an end-user scope. Report permission grants no coordinator or policy-write permission.
Collection uses a snapshot transaction and verifies the current environment binding and policy receipt.
It projects only policy identity, required global enabled keys, stops, accepted Boolean baseline and host
self-observations. It excludes actor records, credentials, database names and targeting identities.
Host self-observations never establish trusted workload inventory or readiness. A stable report is
Complete only for a stated purpose whose required sources are all Current; see purposes and completeness.
PRs #4362–#4365 supply constraint, workload, desired-state and migration evidence; #4366
revalidates the sources and returns Unstable if they keep changing. Never treat an HTTP 200 or the presence of a policy as deployment clearance.
The GET retains an immutable sanitized report in runtimeEnvironmentReport, so it is marked as a side
effect for support impersonation. It requires existing migration-owned indexes 0019 and 0020; it never
creates them. Reports expire for checks 15 minutes after collection and become TTL-eligible after 30 days
(OD-20); top-level retention timestamps are BSON dates. Responses are private/no-store. Missing bindings,
indexes, inconsistent authority or unavailable evidence return a bounded 503 without internal details.
Turning reporting off denies collection and preserves previously retained reports until normal retention.
The endpoint does not mutate policy, enroll consumers, activate a feature or promote a deployment.
Report purposes and completeness¶
PR #4405 (slice 6A.0) lets a report earn Complete. The caller states what the report is for with the
optional query purpose:
purpose |
Use | Extra required sources |
|---|---|---|
policy-change |
Before a runtime policy edit | none |
promotion |
Before promoting a candidate build | gitops (desired provenance) |
legacy-return |
Before switching Runtime back to Legacy | none; legacyParity must be present |
| absent | Diagnosis only | the report is never Complete (purpose and requiredSources are null) |
Any other value, an empty value or a repeated purpose returns 400 ProblemDetails with code
environment_report_purpose_invalid before anything is collected. Values are case-sensitive.
Every purpose requires policy, baseline, deploymentConstraint, hostAcknowledgements, workloads
(the aggregate of workloads:api and workloads:project-management) and migrations. When the bound policy
touches a statistics key, domain:statistics and domain:statistics-control are required too. A key
touches statistics when its effective runtime value is on, or it has a saved override or an active stop.
Statistics keys are the reviewed consumer contracts that Project Management also consumes (today
materializedProjectStatisticsServing, Pages, StageOverview and SignalR). domain:statistics
is always Missing, so a report whose policy touches statistics can never be Complete; that is intended,
because statistics are not production-eligible. The report lists the set it applied in requiredSources.
The report is Complete only when all of these hold:
- a supported purpose is given;
RuntimeFeatureFlags:FleetInventoryModeisCollector(same setting andStaticdefault as fleet inspection). InStaticmode a purposed report addsfleetInventoryMode:StatictounknownOrUnsupportedand staysIncomplete, because configured counts are not observed inventory;- the sources were stable across captures (otherwise
Unstable, as before); - every required source appears exactly once with state
Current, and no required name is inunknownOrUnsupported; - the
migrationsprerequisite isReady(NotReadyorUnknownnever completes a report); - the report has not expired;
- for
legacy-return,legacyParityis present (it is null without a bound policy).
Sources that are not required, such as gitops for policy-change or domain:statistics when the policy
does not touch statistics, may be Missing or listed as unknown without blocking. A Current migrations
source only means verification ran, so completeness also requires every migration this build knows to be
applied with its checksum and declared indexes (Ready). Complete describes evidence, not permission:
the offline checker still decides.
Host heartbeats no longer make reports Unstable. The stability comparison ignores each host's
lastSeenAtUtc, and the hostAcknowledgements source digest is computed over the stored host rows
without lastSeenAtUtc and authorityCheckedAtUtc. Instance UID, process start, image, catalog digest,
acknowledged policy, readiness and state still count. The returned applied entries keep lastSeenAtUtc.
legacyParity (RT-15) is computed server-side for every baseline key that is production-eligible (today
only disableMembership). Each entry records legacyValue (the deployed FeatureFlags value, as the
evaluator computes it with stored policy ignored), runtimeValue (the evaluator's runtime decision: off
while a stop is active, otherwise override or baseline), stopActive and agrees. An entry agrees only
when no stop is active and the two values match; the section's agrees is true only when every entry agrees.
appliedMigrations lists the id, version and checksum of every migration the runner recorded as
applied, in version order, read in the same verification step (at most 10,000 rows; more refuses
collection). It never includes timestamps, actors or errors, and it is null unless the migrations
source is Current. It takes part in the stability comparison, so the offline checker can confirm that a
candidate's required migrations have been applied.
All four fields are additive and nullable; schemaVersion stays 1 and retained reports collected earlier
read back with them null. The capabilities digest now ends purpose-completeness-v1:legacy-parity-v1:applied-migrations-v1.
No flag is added: the route stays behind the default-off RuntimeFeatureFlags:CompatibilityReport:Enabled,
and collection still writes only the retained report.
Checking a candidate offline¶
scripts/runtime-compatibility-check.py judges a candidate build against a report saved from
GET /api/environment/compatibility-reports/current. It reads two local JSON files and prints one JSON object
(verdict, exitCode, reasons[] of {code, detail}, sorted by code then detail). It is offline only: no network,
credentials, database or cluster access, so it never collects or refreshes evidence and is not deployment clearance by itself.
python3 scripts/runtime-compatibility-check.py --report report.json --candidate candidate.json \
--environment staging --binding-epoch <guid> --purpose policy-change|promotion|legacy-return [--now ISO8601]
The candidate uses schema syrf.runtime-candidate/v1 (catalogDigest, optional compatibleCatalogDigests, catalogKeys,
policySchemaVersion, migrations[{id, version, checksum?, required}], images, targetMode Legacy|Runtime, and
optional catalogGrains, a map of catalog key to contract grain). A report's policy.targetedKeys[{key, grain}] (policy
schema 2, see Targeting) join the keys the candidate's catalog must
contain, and when the candidate gives catalogGrains, a targeted key whose grain differs (or is missing) refuses
targeted-key-grain-changed. A schema-1-only candidate (policySchemaVersion: 1) against a schema-2 report is already
outside N-1..N (policy-schema-unsupported), so rolling back below a schema-2 binary while targeting exists never passes:
remove every rule first (the document returns to schema 1).
--now defaults to the current UTC time. Fixtures live in scripts/testdata/runtime-compatibility/; run
python3 scripts/test-runtime-compatibility-check.py.
| Exit | Verdict | Meaning |
|---|---|---|
| 0 | pass |
No evidence gap and no refusal |
| 1 | refuse |
Binding mismatch (checked first, short-circuits), or a replacement-build, legacy-parity or constraint refusal |
| 2 | insufficient-evidence |
Incomplete, unstable, unsupported or expired evidence (takes precedence over refusals) |
| 3 | unusable-input |
Unreadable or invalid files/arguments (short-circuits) |
Reason codes: input-unreadable, input-invalid, candidate-schema-unsupported (3); binding-mismatch,
catalog-key-missing, catalog-digest-incompatible, targeted-key-grain-changed, migration-not-applied, image-change-hosts-not-current,
legacy-parity-stop-active, legacy-parity-differs, constraint-catalog-outside, constraint-key-outside,
constraint-schema-outside (1); report-schema-unsupported, report-expired, report-unstable, report-incomplete,
purpose-mismatch, required-source-unknown, required-sources-missing, policy-missing, migration-evidence-missing,
legacy-parity-missing, policy-schema-unsupported (policy schema must be N or N-1), constraint-action-required, constraint-state-unknown, constraint-expired, required-source-undeclared (2). Timestamps must carry a UTC offset, and a
missing argument is input-invalid (3), not argparse's usual exit 2.
The checker does not trust the report's own requiredSources alone: policy, baseline, deploymentConstraint,
hostAcknowledgements, workloads and migrations must always be declared and Current. --now exists for fixtures;
setting it earlier than the real time defeats the freshness and constraint-expiry checks, so never pass it for a real
decision. A host counts as current when it acknowledges the report's policy identity; its readiness and heartbeat age are
outside this checker (the server reports readiness as Unknown today, and fleet freshness belongs to the admission path).
Retrieve retained compatibility evidence¶
PR #4368 adds GET /api/environment/compatibility-reports/{reportId} for the same dedicated
machine reader and exact environment binding as collection. It uses the existing default-off
RuntimeFeatureFlags:CompatibilityReport:Enabled setting. Human Inspect access and a browsing UI remain
follow-up work; this route grants no additional token scope or policy authority.
The response preserves the original report ID, source observations, completeness and timestamps. A report
past its 15-minute validity may still be retrieved for diagnosis during its 30-day retention, but it is
historical evidence, never a fresh deployment check. Retrieval does not recollect sources, extend expiry,
write a replacement, create indexes or refresh TTL. Records past DeleteAfterUtc return 404 even before the
TTL sweep removes them. Missing IDs and records belonging to another binding also return 404; unreadable
binding or inconsistent stored metadata returns sanitized 503. Every response is private/no-store.
Switching reporting off denies retrieval while retaining saved records; switching it back on restores access under the same binding until retention expires. A different binding never gains access to the old reports.
Constraint evidence in compatibility reports¶
PR #4362 adds the optional deploymentConstraint projection and a deploymentConstraint source entry.
Both are captured in the report's policy snapshot. A bound authority with no active constraint returns a
null projection and a Current source with an explicit absence digest. Legacy/uninitialized policy keeps
constraint evidence Missing; null alone never proves that a deployment is unconstrained.
An active document must match its immutable constraint receipt and the current environment/binding/store lineage. The report exposes rollout ID, constraint digest, serving/target catalog digests, global schema-v1 capability intersection, policy revision at record and timestamps. Coordinator/actor identity, reason and receipt internals are excluded. Policy revision and the source's shared-guard revision are decimal strings. The source revision advances on record, extend and release even though policy identity does not change. Every policy edit also increments this shared guard; the source revision can advance with an unchanged constraint digest.
An expired but unreleased constraint remains present with state: ActionRequired and reason: Expired.
Collection never releases or extends it. Inconsistent receipts, unsupported schema/lineage, invalid guards
or malformed projection evidence refuse collection. A verified constraint is one required source among
several (see purposes and completeness); it is not permission to deploy.
This is additive evidence on the existing default-off, report-only machine route. It introduces no new
scope, index, migration, activation or deployment setting.
Workload collector publication¶
PR #4363 adds POST /api/environment/observations behind
RuntimeFeatureFlags:WorkloadCollector:Enabled (default false). Enabling it requires explicit Provider,
ClientId and Audience under that section plus the same verified runtime environment binding used by the
report service. environment:observations:write is a separate machine scope: it cannot read reports, read or
mutate policy, or use any other API route. Mixed-scope, session, user and impersonated credentials are refused.
The OpenIddict client is syrf-environment-collector; only an explicit strong EnvironmentCollector:ClientSecret
seeds it. No credential or collector is provisioned by this PR.
Each publication contains one complete API or project-management Deployment snapshot: environment/binding, observation ID, previous observation ID, Deployment UID/resource version, lossless generation strings, desired replica count, collection timestamp and up to 1,000 pods with UID, pinned image digest, readiness and termination status. The body is bounded to 512 KiB. The collector must retain its last accepted observation ID: new publications compare against it; exact retries reuse the same ID and payload. A timeout is unknown, so retry the same request. Conflicting replay or a superseded publication is refused, never silently rebased. Snapshots older than 30 seconds or in the future are refused; collector and API clocks must agree.
PR #4369 requires every pod to include explicit Boolean ready and terminating fields. Omission is
unknown evidence, not false: an authorized request missing either field returns 400 before publication.
Explicit false values remain valid. Existing observations written by the typed publisher already include
both fields and retain their exact replay digest; malformed stored payloads missing either field are refused
rather than silently interpreted. No stored observation is rewritten or deleted and no migration is needed.
The existing default-off collector setting continues to control publication.
Migration 0018 owns the age index. Publication verifies it and the environment binding, then updates only
runtimeWorkloadObservation. It neither reads policy nor removes host acknowledgements. Reports expose
collector evidence separately from host self-reports with per-service Current/Missing/Stale/Unknown source
states. A fresh observation does not mean every pod is ready; deployment generation, replica and readiness
checks still belong to the compatibility checker. Completeness additionally needs Collector inventory
mode and a purpose (see purposes and completeness);
4364–#4366 add desired GitOps, migration and cross-source evidence. Automatic host retirement and live Kubernetes collection are¶
follow-ups; no runtime mode, activation, migration execution or production access changes here.
Read-only workload capture¶
PR #4370 adds scripts/capture-runtime-workload.py, an explicitly invoked capture command that produces
one JSON request for the existing observations endpoint. It does not publish, provision credentials,
select a default cluster, change kube context, read Secret resources or activate a feature. The existing
collector endpoint remains default-off. Use a separately approved read-only Kubernetes identity with
get on the named Deployment and list on ReplicaSets and Pods in the selected namespace.
Example for an approved non-production context (values must match the reviewed environment binding):
python3 scripts/capture-runtime-workload.py \
--context "$REVIEWED_CONTEXT" --namespace "$REVIEWED_NAMESPACE" \
--service api --deployment syrf-api --container syrf-api \
--environment "$BOUND_ENVIRONMENT_ID" --binding-epoch "$BOUND_EPOCH" \
--expected-observation-id "$LAST_ACCEPTED_OBSERVATION_ID" \
--output api-observation.json
Omit --expected-observation-id only for a known first publication. The service/deployment/container mapping
is explicit operator input; the command never guesses it. The capture can be submitted immediately to
POST /api/environment/observations by the separately authenticated publisher. Retain the exact output
and observation ID for uncertain-outcome retries; do not change evidence under that ID. The command never
overwrites an existing output file or retries publication. Publication still enforces binding, expected
observation ID and 30-second freshness; a capture file is not a durable freshness guarantee.
Each capture reads Deployment, ReplicaSet and Pod JSON twice using only kubectl get. It follows controller
UIDs from Deployment to ReplicaSet to Pod, including old rollout ReplicaSets; labels alone cannot enroll a
foreign pod. It requires the exact named container, an explicit readiness condition and container Boolean,
and a repository-qualified @sha256: image ID (optionally prefixed docker-pullable://). Digest-only or
missing image IDs are refused rather than inferred from image tags. Missing observations, paginated lists,
duplicate UIDs, malformed ownership, changed selected resource versions and clock regression refuse capture.
A valid zero-replica observation is allowed but never establishes fleet readiness.
The observation time precedes the first read. All six reads and revalidation must finish within 20 seconds; each command has a five-second deadline. Responses larger than 8 MiB are refused, as is output above the endpoint's 512 KiB limit. Only the allowlisted observation fields are emitted; raw environment values, annotations, labels, container diagnostics and command stderr never enter the artifact. Matching reads establish observed agreement, not an atomic Kubernetes snapshot. The collector endpoint and later compatibility checks still must validate freshness, desired generation, replica count and readiness.
This slice supplies workload acquisition only. Scheduled publication/credential handling, desired Argo/GitOps acquisition, trusted retirement and rollout-completion checks remain ordered follow-ups. Tests use synthetic responses exclusively; no live cluster or environment is enrolled by this change.
Authenticated deployment coordinator¶
PR #4374 adds default-off machine-only PUT and DELETE /api/environment/deployment-constraints/{rolloutId}.
Enable only with reviewed RuntimeFeatureFlags:DeploymentCoordinator settings (Enabled, Provider,
ClientId, Audience) and the existing exact environment/database/cluster binding. The API authority must
also be Shared with the same binding. This does not relax the existing production Shared-mode refusal.
The separate scope is environment:deployments:coordinate; report, collector, human and session credentials
cannot use it, and coordinator tokens cannot use unrelated routes or the administrator break-glass path.
Native Identity seeds syrf-environment-coordinator only when EnvironmentCoordinator:ClientSecret is supplied
with at least 32 characters, using client credentials and this sole scope. An absent/short secret removes the
client's grant through the existing automation-client reconciliation. The API also accepts the explicitly
configured Auth0 machine provider; this code does not provision or change an Auth0 tenant.
PUT accepts Record or Extend; DELETE accepts Release. The body carries a mutation ID, complete expected policy
identity (revision as a canonical decimal string), action, reason and operation-specific constraint fields.
Coordinator clients and proxies must preserve the JSON body on DELETE as well as PUT. Always send action
explicitly: the current non-nullable wire enum defaults an omitted action to Record.
Record includes the immutable completion targets described below; Extend and Release use the reviewed
constraint digest. Release additionally selects Target or Rollback and always calls the verified store path.
The actor is derived from the configured authenticated provider/client, never from a request actor field.
Keep that client identity stable for an active rollout: only its owning coordinator may extend or release it.
Rollout IDs are one ASCII path segment, up to 128 letters/digits or -_.: characters.
Responses expose only mutation ID, rollout ID, action, resulting constraint digest (null after release) and original receipt time. Exact uncertain-outcome retries preserve the entire request. HTTP 409 refuses a changed policy/constraint or unproven completion; HTTP 503 or transport loss leaves the outcome unknown. Requests are bounded to 32 KiB and ten seconds; responses and authorization denials are no-store. No endpoint accepts BreakGlassRelease. Its separately authorized administrator adapter remains a follow-up.
The gate defaults off, preserves all saved constraint/audit data, and never runs a migration. Credential provisioning, workflow invocation and environment enrollment are separate approved operations; none occurs in this change. Tests use synthetic identities, a TestServer and isolated local test data only. The API chart does not expose a dedicated coordinator values block; reviewed chart/config wiring and environment enablement remain follow-ups. Adapter tests pin configuration and identity refusal, the server-derived owner and receipt mapping independently of controller fakes. Further transport improvements are explicit missing-action rejection and distinguishing corrupt stored-state FormatException from malformed request identity (the current controller maps either to 400); neither changes the store's fail-closed mutation behavior.
Verified deployment completion primitive¶
PR #4373 adds ChangeVerifiedDeploymentConstraintAsync for the later authenticated coordinator adapter.
Recording through this entry point requires immutable target and rollback expectations: one catalog digest
and one repository-qualified SHA-256 image per API/PM service. The target catalog must equal the constraint's
target; the rollback catalog must be in its serving set. Extension preserves these expectations unchanged.
Normal release explicitly selects Target or Rollback. Inside the existing binding/policy/guard transaction, the store reads current collector observations and host acknowledgements, requires collector-backed Active adoption under the configured positive counts, and matches every observed pod image and current catalog to the selected recorded expectations. Missing, stale, mixed-image, non-ready or unacknowledged evidence refuses release without changing the guard, constraint or audit. These expectations also enforce verification when the older internal primitive is called. Exact successful mutation replay uses the immutable receipt even if observations have since expired or disappeared; it never performs the operation a second time.
Old internal constraints/receipts without completion expectations remain readable/replayable, but cannot be released through the verified coordinator entry point. Its break-glass action is refused: that operation must remain on the separate administrator capability path. No HTTP coordinator or break-glass adapter is exposed by this slice. The original internal primitive retains its documented caller-authentication requirement.
No new flag is introduced for this internal safety boundary. Existing default-off runtime/collector modes are unchanged, no collections or indexes are migrated, and no live constraint is recorded or released. This proves observed target/rollback completion; candidate compatibility, domain readiness and live coordinator credentials/workflow integration remain separate requirements.
Collector-backed fleet inventory¶
PR #4372 adds deployment configuration RuntimeFeatureFlags:FleetInventoryMode=Static|Collector.
The default is Static, preserving the configured-count inspection path. Collector is an explicit opt-in
for environments with the trusted workload publisher; unknown values refuse inspection. No environment is
enrolled by this change. This configuration gates read-only fleet semantics, not policy decisions or migrations.
Collector mode reads authority, acknowledgements and workload observations in one bound database snapshot.
Both API and PM must have fresh observations (strictly younger than 30 seconds), matching binding and the
configured positive FleetExpectedReplicas counts. Deployment generation must be observed, every expected
pod must be present, ready and non-terminating, and UIDs must be unique across services. Missing evidence,
zero/surplus replicas, generation lag, stale/future timestamps and service disagreement never establish Active.
The existing host policy/catalog checks, autoscaler Unknown rule and statistics-domain Unknown rule still apply.
Retirement uses MongoDB's clock only (PR #4407). Every heartbeat write stores serverSeenAtUtc, stamped by
MongoDB itself ($$NOW), beside the host's own lastSeenAtUtc. The write keeps its filter, upsert, transaction
and late-write protection. Evaluation reads MongoDB's current time ($$NOW) in the same snapshot transaction.
A historical acknowledgement absent from the fresh service inventory is marked Retired in the returned view
only when MongoDB has not seen it rewritten for the quiet period, RuntimeCollectorFleet.RetirementQuietPeriod
(the 30-second freshness window plus 5 seconds, so 35 seconds). Hosts heartbeat every 8–12 seconds, so a
quiet row belongs to a stopped pod. A row written more recently may be a pod that started after the collector's
read: it leaves the fleet Unknown (CollectorHostObservationDisagrees). A host clock that runs behind or ahead
cannot change this, because both times come from MongoDB. A row whose start time follows its heartbeat is also
Unknown.
Rows written before serverSeenAtUtc existed, or by an image without it, carry only the host clock. Such a
legacy row retires only once its lastSeenAtUtc is at least LegacyRetirementAge (15 minutes) behind MongoDB's
time. The 15 minutes is an assumed bound on host clock skew, not a measured or verified one: a live legacy host
whose clock runs more than 15 minutes behind MongoDB could be retired. The bound applies only to legacy rows. It
stops mattering once every host runs an image that writes the server stamp, because every row then carries
MongoDB's own time. The first rollout away from images that do not stamp the field can therefore take 15 minutes
to reach Active. After that, a rollout reaches Active about 35 seconds after the old pods stop, once the new pods
have acknowledged.
MongoDB's time is read as the first statement of any transaction that may retire rows (fleet inspection, production admission and deployment completion). A snapshot transaction sees rows as of its first operation, so a slow transaction cannot make a live row look older than it was. Emergency stops never read it.
All rows remain stored and visible, including retired rows. No TTL, deletion or migration is introduced.
serverSeenAtUtc is an additive root field: older readers ignore it, and older rows parse without it. It is not
part of the stored payload or the API response. Switching back to Static uses every retained row again.
Static mode counts every retained acknowledgement, so it suits only hosts whose instance UIDs stay the same
(for example the local runtime-fleet harness). In Kubernetes every replaced pod has a new UID, so after any
rollout the old pods' rows keep a Static fleet out of Active (it reads Applying, then Degraded). Kubernetes
environments therefore need Collector mode before fleet Active can be relied on.
A pod present in the inventory but missing an acknowledgement still blocks Active.
The local runtime-fleet harness exercises this end to end against real API and project-management processes:
bash e2e/run-local.sh --runtime-fleet --scenario collector-scale-up. It restarts the APIs in Collector mode,
publishes a synthetic inventory through POST /api/environment/observations with a mock-OIDC machine token, and
asserts Active with every listed host current, Unknown (CollectorHostObservationDisagrees) while an unlisted
API is live, and Active again with that host Retired once it has been quiet for 35 seconds by MongoDB's clock.
What it covers and what it does not:
runtime fleet lane.
This confirms policy adoption against observed UID membership. It does not prove deployment target image compatibility, release a deployment constraint, or complete statistics domain readiness. Live collector setup, rollout target proof and coordinator integration remain required follow-ups.
Explicit workload publication¶
PR #4371 adds scripts/publish-runtime-workload.py to submit a saved capture once. Supply an already-issued
machine token in SYRF_RUNTIME_COLLECTOR_TOKEN; it must have the separate environment:observations:write
scope and match the API's configured client, provider, audience and environment binding. The command does not
obtain tokens, provision credentials, schedule collection or activate the default-off collector endpoint.
No new flag is needed for an explicitly invoked command; the endpoint's existing gate remains authoritative.
python3 scripts/publish-runtime-workload.py \
--api-origin "$REVIEWED_API_ORIGIN" --environment "$BOUND_ENVIRONMENT_ID" \
--binding-epoch "$BOUND_EPOCH" --service api --input api-observation.json
The API origin must be HTTPS (HTTP is accepted only for loopback test servers), with no user information, path, query or fragment. Redirects are refused. The command checks the saved capture's environment, binding epoch, service and observation ID before sending its exact bytes. The API still validates the complete contract, permissions, expected-observation CAS and initial freshness. No local age check prevents an exact retry of an already accepted observation after its initial admission window.
Exit 0 means a 200 receipt with the matching observation ID and a valid digest was received; stdout contains only those two receipt fields. Exit 1 means an HTTP 3xx/4xx refusal, except 408/425/429; exit 2 means local input refusal; exit 3 means the outcome is uncertain (408/425/429, transport loss, 5xx, or an invalid/missing receipt). Preserve the exact capture and replay it after investigating an uncertain outcome. The command never retries, rewrites the capture, prints credentials or copies remote error bodies into logs. A successful receipt confirms publication, not fleet readiness or rollout completion. Rotated/advanced observations can make an old retry conflict; do not replace evidence under an existing observation ID.
Environment proxy settings are explicitly ignored, so bearer tokens are sent directly to the reviewed origin. HTTP 408, 425 and 429 are uncertain outcomes (exit 3), as are transport failures and 5xx responses: retain and replay the exact capture. The ten-second urllib timeout is per socket operation, not a total elapsed deadline; an invocation under a scheduler needs its own process deadline. Terminating a publisher must also be treated as an unknown outcome.
Tests use mocks and a loopback HTTP stub only. Live publication, token provisioning, automatic scheduling, desired GitOps acquisition and trusted retirement/rollout checks remain separate follow-ups.
Desired deployment provenance¶
PR #4364 adds optional desired evidence to each collector observation: applicationUid, lossless
resourceVersion, a 40-character gitOpsRevision, chartTag and 64-character renderedSettingsDigest.
The collector must bind these to the same service and timestamp as its workload snapshot, using the earliest
observation time when collecting multiple sources. This is a collector obligation: desired has no independent
timestamp, so the API applies the workload timestamp and cannot attest the actual source-read time. It must derive rendered-settings digests from allowlisted
non-secret configuration; the API accepts digests, never raw settings or secret values.
Reports include per-service GitOps sources and project the aggregate desired revision/chart tags only when
both API and PM have Current provenance at the same GitOps revision. The aggregate settings digest is a
canonical hash of sorted service/digest pairs; each original digest stays available in the workload projection.
Different revisions yield Unknown, not a guessed winner. Missing or stale evidence leaves the desired snapshot
null. Optional evidence can be added and removed without invalidating stored observations; exact retries of
older requests without desired preserve their receipt digest.
This uses the existing default-off collector/report settings and scopes, with no new migration or rollout.
Reports can be Complete only in Collector mode (see purposes and completeness); migration verification
and cross-source stability checks are implemented in #4365–#4366.
PR #4365 supplies fresh migration evidence through the existing default-off compatibility-report route.
The API supplies its PmMigrations.All catalogue. Each request uses the migration runner's read-only
verification: every known migration must be applied with its expected checksum, and every declared index
must match. The migrations prerequisite is Ready or NotReady; a missing catalogue or unreadable
state produces Unknown. This does not use the five-minute schema health cache.
The migration source revision is the ordered catalogue digest; its evidence digest covers that catalogue and the verification result. Neither raw diagnostics nor migration actors/history are returned. Evidence is dated before verification starts and expires after 30 seconds, including time spent collecting the remaining report. Collection writes only the retained report, never migration state or indexes. A later request rechecks everything, so an index removed after a healthy result is detected immediately.
This proves readiness only against this binary's catalogue, not against a future candidate build. Reads
of migration records and index definitions are not atomic with the report's policy snapshot. Reports
are revalidated as described below; completeness follows purposes and completeness. The former crossSourceRevalidation missing-input reason is no longer emitted.
No migration, live collector, deployment or activation is performed by this slice.
PR #4366 revalidates report sources before retention. It collects an independent snapshot, recollects, and compares the policy/baseline, constraint, workload and desired evidence, host acknowledgement digest, migration verification result, prerequisite states and unknown reasons. Per-read timestamps and generated report IDs are excluded from comparison; stored observation timestamps (other than host heartbeats, since
4405) and freshness states are included.¶
A mismatch starts another comparison using the latest snapshot. Three comparisons (at most four captures)
are allowed under the existing request deadline. A stable result is then evaluated for its purpose
(purposes and completeness). Continuing changes return Unstable with crossSourceChanged; read failure or cancellation
cannot produce a stable result. Only the final report is retained with majority acknowledgement.
Each snapshot rechecks the original environment binding. Freshness is evaluated at the end of collection, so a source that ages out while other evidence is read cannot remain Current. This verifies agreement between observed source snapshots, not an atomic transaction spanning Mongo migration/index reads and external collectors. No report in this slice authorizes activation or deployment.
PR #4367 adds optional statisticsControl metadata from the durable global statistics singleton.
The domain adapter reads at most two rows in the report's own snapshot and rejects duplicate controls,
missing required fields, malformed data, unknown modes and negative epochs/versions. No repository
initialization, index creation or cache is involved. The output includes mode, active-reviewer-tracking
mode, whether a mode transition is pending, catalogue/storage/source/schema versions and lossless string
write/mode/client-invalidation epochs. It excludes document IDs, actor/audit fields, transition tokens and
project allowlists. Missing or invalid evidence is explicit in domain:statistics-control.
The control metadata participates in cross-source revalidation. domain:statistics remains unknown:
this record alone cannot establish deployed maintenance flags, fleet protocol compatibility, inventory
or per-project serving readiness, so a report whose policy touches statistics stays Incomplete. The existing default-off report setting and
machine permission boundary apply; collection does not initialize or transition statistics state.
Reviewed consumer evaluation¶
PR #4350 adds runtimePolicy.consumerContract metadata to the canonical YAML and includes it in the
catalog digest. The initial disableMembership (API admission) and materializedProjectStatisticsServing
(API/PM read admission) contracts are extended by #4352 with the existing pages, Stage Overview and SignalR
selectors. Other keys remain unreviewed; absence of metadata
never means safe or production-eligible. See the audited contract evidence.
GET /api/runtime-feature-flags/authority-evaluation/{key} requires the current InspectRuntimeFeaturePolicy
permission before catalog lookup or evaluation. In Shadow mode it evaluates one immutable, receipt-validated
local capture, with its accepted baseline and current deployment prerequisites. All five contracts expire at
30 seconds measured monotonically and tolerate no failed reads. Legacy mode reports Disabled. Unknown keys
return 404; known but unreviewed keys report UnreviewedConsumer; an unsupported host or unavailable/stale
policy returns no value. The declared safe value remains separate from that unknown decision.
Responses include local-process scope, catalog classification and a decimal-string revision when a fresh decision is available. ProductionEligible describes the reviewed catalog candidate, not permission to activate it, production readiness, or fleet acknowledgement. No database query happens on demand. This inspection route does not replace any domain gate or enable Shared mode by itself. Fleet coordinator admission remains required before a rollout can be declared Active.
The added catalog metadata changes CatalogDigest. Previously generated lineage previews must be regenerated and reviewed against this catalog; an old digest is rejected rather than silently reinterpreted. No live lineage import has run in this implementation programme. This change runs no data migration and changes no Helm/browser defaults.
Authority emergency-stop primitive¶
PR #4351 extends the bound shared writer with EmergencyStop and ReleaseStop actions for
the audited global controls with a false safe value (initially membership and serving; #4352 also audits
the three statistics selectors). These are library operations, not new public endpoints or permission grants. Legacy
remains the deployment default and production writes remain refused.
A stop records the key in killSwitches and advances policy identity without changing the accepted baseline
or saved overrides. The bounded evaluator applies the safe value before dependency normalization; ordinary
policy edits cannot clear the stop. Releasing it explicitly resumes the latest saved policy, subject to the
existing deployment and dependency checks. This does not remove saved memberships, statistics, or history,
and does not alter deployment-owned statistics maintenance gates.
Both actions require exact binding/lineage/revision, actor and reason, and atomically commit an immutable
receipt with policy state. Identical request replay resolves the original receipt, including after later
edits. Reusing its mutation ID for a different action refuses. Emergency stop bypasses deployment-constraint
and new-audit-index readiness gates (plan §6.4/Q1 and §14.5); the built-in unique audit _id and shared policy
transaction still serialize receipts. It does not bypass binding or authority integrity. ReleaseStop retains
the normal readiness and constraint gates. No index is created or migration run by either action.
Older authority readers that cannot understand non-empty stops refuse that policy. Before any live enrollment, deployment compatibility checks, the preview/commit UI and production-grade fleet admission (independently observed inventory, not only the host acknowledgements described below) must be complete. No emergency stop has been applied to a real environment.
Targeting (policy schema 2)¶
Project allow and exclude rules are writable, and open and close routes, behind a switch. 6F PR A (#4463) taught every API and Project Management binary to read and evaluate targeting rules; 6F PR B (#4472) adds the preview and commit routes behind
RuntimeFeatureTargeting:Enabled(default off in every environment) and the first targetable flag; 6F PR C (#4482) decides the pilot's route gate and project availability for the admitted project (see Route gates and availability). 6F PR F (#4488) adds project rollouts with stable cohorts (see Gradual rollout). 6F PR E (#4495) adds person, pair and "active members of a project" rules on Personal keys (see Personal and combined targeting). 6F PR H (#4496) adds per-rule switches (see Switching a rule off). 6F PR G (#4510) adds consent (join and leave a beta) and stricter-only environment feature settings (see Consent and environment settings). Rollouts over people are refused asnotYetSupportedorpersonalRuleOnProjectGrain. Plan: delivery plan §7.2.
The targetable flag. screeningKeywordHighlighting has a reviewed grain: Project consumer contract (API only,
productionEligible: false, new-request-admission) and owns the registry feature keyword-highlighting
(grain: Project, permission: ProjectDesignPolicy, whatsNew: quiet while it is a Developing pilot). The pilot choice is owner decision D33, answered A on
2026-10-24; retargeting means moving that contract and feature to another flag and regenerating. With no rule
saved it decides exactly as before in Legacy and Shadow mode; in Shared mode it moves from the API's compatibility values
to the bound authority, which gives the same value while the policy is fresh and the contract's safe value (off) when it is
stale or unavailable. Production stays deployment-controlled (non-Global keys are never production-eligible). Every
other key is Global and refuses rules (globalKey). The catalog generator accepts grain: Project | Personal on a
contract but never with productionEligible: true; registry check 9 requires a feature's grain to equal its flags'
contract grain, a non-Global feature's flags to all have contracts, and no deployment-managed flag to be targeted.
Switch on an isolated stack (never staging or production without an owner decision): set
RuntimeFeatureTargeting:Enabled=true for API and Project Management alike (chart value runtimeFeatureTargeting.enabled,
env SYRF__RuntimeFeatureTargeting__Enabled), with RuntimeFeatureFlags:Mode=Shared and a bound environment. Every
fresh API and PM host must advertise policy schema 2 (policySchemaVersions: [1, 2] on its acknowledgement, which every
binary since PR A does) before the first rule is accepted.
Preview, then save (capability ChangeRuntimeFeaturePolicy; both answer 404 and read nothing while the switch is off):
POST /api/runtime-feature-flags/screeningKeywordHighlighting/targeting/preview
{ "expectedIdentity": { …current policy identity… },
"targeting": { "rules": [
{ "id": "<new guid>", "kind": "allow", "label": "Wave 1 pilot", "subjects": [ { "type": "project", "id": "<project guid>" } ] },
{ "id": "<new guid>", "kind": "exclude", "label": "Held for data review", "subjects": [ { "type": "project", "id": "<project guid>" } ] } ] } }
PUT /api/runtime-feature-flags/screeningKeywordHighlighting/targeting
{ "mutationId": "<new guid>", "expectedIdentity": { … }, "targeting": { "rules": [ …the previewed rules… ] },
"previewDigest": "<from the preview>", "reason": "Pilot for the keyword-highlighting study" }
The rules are the key's complete replacement; "targeting": null (or no rules) clears them, and removing the last
rule anywhere writes schema 1 again with the v1 digest. Clearing a key that has no stored rules is an idempotent no-op: the
preview proposes the current identity, and the commit writes no revision and no receipt and returns the unchanged identity. The preview returns the rules before and after by kind, the
projects that gain and lose the key (counts, and up to 50 with their names), dependency notes, schema admission
(required, hosts supporting and total, Ready | Blocked) and the usual identities; it writes nothing. The commit goes
through the policy writer exactly as a policy change does: binding re-check, receipt chain, replay by mutation ID (retry
the same ID and body after an unknown outcome; another body under that ID is refused), the preview digest (domain
syrf-runtime-targeting-preview-v1, over the key, expected identity and rules, the candidate document, the constraint
guard and the deployed values), the deployment constraint, then schema-2 admission. Its receipt is an ordinary
globalPolicyMutation with request.action = "Targeting" and request.targeting = { key, rules } (null to clear), so
GET mutations/{mutationId} resolves it. Saved settings, stops and other keys' rules are untouched, and a later policy
change or stop keeps every stored rule.
Refusals are HTTP 422 { code: "runtime_targeting_refused", reason, message } and write nothing:
| Reason | When |
|---|---|
globalKey |
A rule on a key with no Project or Personal contract (clearing is allowed on any key, so rules a catalog change left incompatible can be removed) |
deploymentManaged |
The key is pinned to its deployed value across API and PM |
personalRuleOnProjectGrain |
A person or pair subject, or a rollout over people, on a Project key |
notYetSupported |
A rollout over people on a Personal key |
rolloutPaused |
A change to a paused rollout's amount, group or subject type (make it Active or turn it off first) |
invalidRule, duplicateSubject, limitExceeded |
Schema rules 2 to 6 (the message is the exact schema message; enabled on a rollout is invalidRule with a message pointing to its Off state, and a requirement, stop or default rule is invalidRule with a message naming exclude, allow and rollout), a document over 512 KiB, a rollout change over more than 100,000 projects, or a rollout over projects on a Personal key |
unknownSubject |
A named project does not exist or is deleted, or a person the key's stored rules do not already name is not an active Investigator (absent, deactivated, suspended while applicationSuspension is on, or pending deactivation); one generic reason |
parentOff |
An allow, or a rollout that is not Off, on a key whose required Global parent is off: turn the parent on or target it first; it is never widened |
schemaUnsupportedByFleet |
The first schema-2 write while any fresh host (or no host at all) does not advertise schema 2, or pausing a rollout while any fresh host (or no host at all) does not advertise rolloutPausedAt (6F.F2) |
A deployment constraint that allows only policy schema 1 refuses the write (runtime_policy_constraint), and a coordinator
may now record allowedCapabilities.policySchemaVersion: 2. Production refuses every targeting write, a clear included and
on any key, eligible or not: Legacy production has no policy administration, and Runtime production answers
runtime_policy_constraint ("This key is not eligible for production runtime policy.").
How the browser reads a targeted feature (6F W1)¶
The web app asks one service, FeatureGate (src/services/web/src/app/core/services/feature-gate.service.ts), whether a feature is on: isEnabled('<featureId>') returns a signal.
- A Global feature reads the runtime snapshot flag, exactly as before.
- A Project or Personal feature reads the
enabledvalue thatGET api/projects/{projectId}/features/availabilityreturns for the current person and project.enabledis absent (null) when no targeting is stored for the feature, and then the gate falls back to the runtime snapshot flag, so with no stored targeting the browser behaves as it did before the gate. - The availability client keeps one answer for the current (person, project) and clears it when either changes. A late response for the previous project is dropped, and until the new project's answer arrives the gate reads the snapshot flag, never the previous project's targeted value (RT-32, web half).
- A revoked allow, or a new exclusion, takes effect on the next availability fetch, which follows every accepted runtime revision (SignalR hint, fallback poll, focus or reconnection): the control then disappears without a reload.
- The gate hides controls only. The server's permission and route checks still decide what a person may do.
The keyword-highlighting pilot (keyword-highlighting, flag screeningKeywordHighlighting) uses it for the keyword lists in project settings, the review page's keyword toggle and the PDF viewer's highlighting. A new web consumer of a Project or Personal feature should inject FeatureGate rather than select the flag.
Switching a rule off¶
Owner decision D28 = A (6F PR H, spec 6F.9). An exclude or allow rule can be saved with "enabled": false (absent in a
request means true). The switch is an ordinary reviewed change: preview, reason, a new revision and a receipt, exactly as
for any other rule edit; send the same rule id with enabled flipped to switch it back. A switched-off rule keeps its place,
label and subjects, and:
- the evaluator skips it (the "Preview and check" trace shows
skippedfor the exclusions or allows step, naming the rule), so a disabled allow decides as if it were absent and a disabled exclusion stops excluding; - it counts for neither the required-enabled set (a key with only disabled allows does not need the capability on every
host) nor the preview's gains (re-enabling it previews the gain, disabling it previews the loss), and it does not trip
the
parentOffcheck; - the stored document always carries
enabledon exclude and allow rules (trueorfalse), as schema 2 already shows, so documents written before this change are unchanged and removing every rule still returns the document to schema 1; changing onlyenabledchanges the preview digest and the policy digest.
A rollout has no second control: its switch is its Off state. Requirements, stops and the environment default have no
switch; a commit that sends enabled on a rollout, or a rule of kind requirement, stop or default, is refused as
invalidRule (an enabled on a rollout points to its Off state; the other kinds are told that only exclude, allow and rollout rules exist). When RuntimeFeatureTargeting:Enabled is off the existing rules still apply (exclusions and stops stay in
force, allows and rollouts are dropped); a switched-off exclusion is skipped in that state too, so it never excludes.
The Targeting page (6F W2)¶
/admin/features/<featureId>/targeting (PR #4519, canvas board
Targeting) shows one Project or Personal feature's rules in the order SyRF checks them: requirements, stops,
exclusions, specific projects (or people), gradual rollout and the environment default. It needs
inspectRuntimeFeaturePolicy; adding, editing, switching, removing and saving need changeRuntimeFeaturePolicy, and an
inspector sees the same page without those controls. It reads the saved rules from
which answers { key, grain, identity, rules: { exclude, allow, rollout }, projects, consent }: the identity the rules were
read at (the next preview's expected identity), the key's grain, the stored rules exactly as written (switched-off rules
included), and the owning feature's consent (code, environment setting and reason, effective). It is behind
RuntimeFeatureTargeting:Enabled (off: a plain 404 before any capability check or read; the page then says targeting is
switched off), needs ChangeRuntimeFeaturePolicy or InspectRuntimeFeaturePolicy and Shared mode, and writes nothing. A
key this binary's catalog does not declare answers a 404 problem whose code is runtime_feature_flag_unknown (#4563),
so the page can tell it from the switch: it then says Rules not found (the server does not know the feature's flag,
for example while it runs an older version), never "switched off". Both capabilities, like ListUsers, require the
application Administrator role (ApplicationAuthorityPolicy), so the person IDs this read returns to an inspector reach
no one who cannot already list every user (pinned by RuntimeAdministrationCapabilityHttpTests).
projects names the projects the rules name only for a caller who may change the policy (the preview already returns names
under that capability); for an inspector the page has no names (see "Names, never bare IDs" below). This read returns
people as IDs only; the page names them from the Admin Console user list the people lookup below reads.
- Every change is a draft. Adding or editing a rule (a reason of at most 80 characters, shown with the rule; projects
and people found by name, e-mail address or ID: see the lookups below), switching an exclusion or allow on or off
(D28 A) and removing a rule change only the draft. The draft bar says what is not saved yet. Preview changes posts the whole draft to the targeting preview and shows the
changes, the projects that gain and lose the feature (with a "because" taken from the draft), people by name and e-mail
address, inactive people listed for removal, dependency and rollout notes, and schema admission (a
Blockedfleet disables saving). Save commits the previewed rules with the preview digest and a reason (proposed from the draft and editable; the proposal follows the draft, for example when a pasted project's name arrives, until you type in the field, and from then on your text is kept). - Refusals and conflicts. A refused preview or save is headlined in plain words for each known
reason(unknownSubject,duplicateSubject,limitExceeded,invalidRule,personalRuleOnProjectGrain,notYetSupported,rolloutPaused,parentOff,schemaUnsupportedByFleet,globalKey,deploymentManaged,runtime_policy_constraint; 401/403 as "You may not change these rules."). ForunknownSubjectthe page looks up every project of the draft by its exact ID and names the ones that no longer exist ("Example project C no longer exists.", or "A project in this draft no longer exists." with the short ID when it has no name), and people the user list does not have. The server'smessageandreasonstay available, verbatim, in a collapsed Details disclosure. Any 400 on a preview or save is a refusal too (headlined with the server's text, any GUID replaced by a name or short ID), and so is a 404 there: targeting was switched off since the rules were read ("Targeting has been switched off here since the rules were read, so they can't be changed now."). A refused save is never offered to be sent again unchanged (#4563). After a 404 the page also disables Save until you go back to the draft or the rules are read again, and a refused preview closes with "Your draft is kept, but nothing can be previewed or saved until targeting is switched on here again." instead of "Change the draft, then preview it again."; after a 400 Save stays available, so the draft or reason can be fixed. A revision conflict keeps the draft; Preview again reads the rules again and previews the same draft against the new identity. An unknown save outcome (a timeout, a 5xx, or a receipt that cannot be verified) can be checked (GET mutations/{mutationId}) or sent again with the same mutation ID and body. - Stops. The emergency stop stays an immediate action on the Flags page: the Stops step links there for a stoppable flag and says when a stop is in force. Where a per-project stop would be, the page says "Need it off for just one project? Add an exclusion for that project (saved with a reason). Use Stop for everyone only in an emergency." (D35 A).
- Consent. With an effective consent other than
none, "Choices projects have made" (or "people") shows the joined and left counts fromGET features/{featureId}/choices(counts only, D29/D30); the exclusions step notes how many left. - Rollout. The rollout is shown read-only, with its pause time when paused (D32 C: only projects that existed then keep getting it). The step links to the W3 rollout page, and "Who gets it" links to "Check a project" (the simulator).
- Names, never bare IDs (owner feedback on #4519: "Don't just refer to entities in UI by unfriendly Guid"). Rule rows
and chips, the summary, the draft bar, the rule editor's chips and sentence, the preview's tables, its proposed save
reason, refusals and errors name projects and people; rules still store IDs only. Projects take their names from the
targeting read, previews and the project lookup; a project the page knows only by ID (a pasted ID) is looked up by its
exact ID through the same lookup (one request per project, deduplicated, at most 50, four at a time). People are named
from the Admin Console user list (
ListUsers), as name with the e-mail address as secondary text. Where there is no name the page says so in plain words, with the first eight characters of the ID as de-emphasised secondary text for support: "Project no longer exists" (the read or a preview listed it without a name, or the exact-ID lookup found nothing: absent or deleted), "Unknown project" (no name available, for example for an inspector, or the lookup failed), "No longer has an account" (the user list is read and does not have them), "Unknown person" (noListUsers, or the list is not read). A project owner the lookup could not name is named from the user list, else "unknown person ()". Server text shown in running text has any GUID replaced by the name, else the short ID. - Project lookup (owner feedback on #4519). The rule editor's project field searches as you type (after a 300 ms
pause, from two characters) and offers up to 20 projects, each on two lines so two similarly named projects are not
confused: the name (with an Archived tag for an archived project, offered only for its exact pasted ID), then
"Owner
· members · stages · studies · Created · …", leaving out any part the server could not give. Choosing one adds it as a chip (for "one person in one project" it fills the project); hovering over or focusing a chip shows the same line, which screen readers also hear. Pasting one or more project IDs and pressing Add still works, and a single pasted ID the lookup finds shows its name and context. The "excluded by another rule" warning is unchanged. The lookup is
which answers [{ projectId, name, active, ownerId, ownerName, memberCount, stageCount, studyCount, createdAtUtc }]: a
project ID matches that project exactly (archived or not, so a pasted ID resolves), any other text matches active
project names case-insensitively as literal text, names starting with it first, then names containing it, at most 20.
Deleted projects never match. It is behind RuntimeFeatureTargeting:Enabled (off: a plain 404 before any capability
check or read), needs ChangeRuntimeFeaturePolicy (the editor's audience; claim-administrator only where PM authority
does not decide), refuses a query shorter than 2 or longer than 100 characters with a 400 before reading, and writes
nothing. pmProject has no name index, so a name query scans the collection, bounded by a 5-second server time limit
and the 20-result cap. Where each field comes from:
| Field | Source and cost |
|---|---|
memberCount |
Active members, computed in the same pmProject projection: the schema-1 Memberships (schema-0 Registrations) whose status history ends Active, or with no history whose IsActive is not false, as Project.Memberships decides. No extra read. |
stageCount |
The size of the project's Stages, in the same projection. |
createdAtUtc |
createdAtUtc (6F.F2) when the project has it, else CreationDate (every older project; stored as a UTC instant); null when neither exists. |
ownerId, ownerName |
The project's OwnerId, and the owner's name from one pmInvestigator read by _id for all the matches' owners (first name and surname, else the preferred name), under the same 5-second limit; ownerName is null when the record is missing or has no name, and every ownerName is null when that read fails or times out (logged as a warning; the matches are still returned, #4563); the editor then names the owner from the Admin Console user list, else shows "unknown person ( |
studyCount |
One aggregation over pmStudy for all the matches ($match on ProjectId $in the result IDs, $group by ProjectId): a covered scan of the ProjectId_1 index, about one index key per study of the listed projects, never one query per project. Same 5-second limit; if it fails or times out, every studyCount is null (logged as a warning) and the matches are still returned. Studies a search deletion moved out carry a holding project ID, so they are not counted. FEAT-024 materialized statistics are not used: they are dark and default-off, so most projects have no row. |
| - People lookup (owner feedback on #4519: "like the user impersonation lookup, where I can enter an email or a | |
| person's name"). On a Personal feature the people options ("Turn it on for specific people", "Keep it off for specific | |
| people") and the person field of "one person in one project" search as you type, the same way (after a 300 ms pause, | |
| from two characters), and offer up to 20 people as name plus e-mail address. Choosing one adds that person's ID as a chip | |
| (for "one person in one project" it fills the person); pasting one or more person IDs and pressing Add still works. | |
| The chips and the one-sentence summary show the names the lookup knows; the rule itself stores IDs only, never names or | |
| e-mail addresses. A person ID matches that person exactly; other text matches first names, surnames, preferred names, | |
| full names or any of the person's e-mail addresses case-insensitively as literal text, those starting with it first, | |
| then those containing it. |
It reuses the search behind Admin Console > User Impersonation: the user list GET /api/investigators
(ApplicationListUsersPolicy, decided as ListUsers), which the browser already reads at sign-in for anyone allowed
to list users, searched in the browser. No new endpoint. ListUsers and ChangeRuntimeFeaturePolicy both require the
application Administrator role (ApplicationAuthorityPolicy), a small trusted group who can already see every user,
so everyone who may change targeting can use it. The separate impersonation candidates route
(GET /api/impersonation/candidates) needs ImpersonateUsers, the support impersonator role, and is not used. Where the
account may not list users (for example while impersonating), the lookup says so and person IDs are pasted as before;
a failed read of the list says the search is unavailable. The list carries no deactivated flag, so the lookup can offer
a deleted account, as the User Impersonation page does.
Flag decision: not a new flag. The page, its read and the project lookup sit behind RuntimeFeatureTargeting:Enabled
(default off everywhere) like every targeting route, and the page itself is administrator-only navigation. The people
lookup adds no route: it searches, inside that page, the user list administrators already read.
Preview and check (6F.5). POST /api/runtime-feature-flags/{key}/simulate explains why one subject gets a decision
and what a draft would change. It needs InspectRuntimeFeaturePolicy or ChangeRuntimeFeaturePolicy and sits behind
RuntimeFeatureTargeting:Enabled: off, it answers a plain 404 before any capability check or read. It is read-only (it
grants and writes nothing, and a POST that only reads, so read-only impersonation admits it as it does the policy preview).
POST /api/runtime-feature-flags/screeningKeywordHighlighting/simulate
{ "subject": { "projectId": "<guid>", "personId": null },
"draft": { "rules": [ …the rules to try; an empty list clears the key… ] } } // "draft" is optional
The response is { key, grain, subject, current, draft, comparison }. current is the trace under the policy in force
and draft (null without a draft) the trace with the draft's rules replacing this key's rules; each trace is
{ value, reason, decidedByStep, ruleId, steps: [{ step, name, outcome, detail, ruleId?, ruleLabel?, bucket? }] } with the
seven steps in the displayed order (requirements, stops, exclusions, allows, rollout, default, baseline) and outcomes
pass, off, on, skipped or notApplicable. comparison is { valueChanged, changedStep }: the first step whose
outcome or rule differs. The admin "Check a project" panel (6F W3) shows "Now" and "With your draft" side by side. It is the same RuntimeTargetingEvaluator
and DecideTargeted path a request uses with its trace switched on, so the trace cannot disagree with the decision.
The project and person are subjects of the simulation, never the caller: the context is marked
FeatureContextOrigin.Simulation and the administrator's own identity and memberships are never read. Reads are bounded by
grain: a Project key reads the project once, for its creation time (the person is ignored, never read or checked); a
Personal key reads the person once and, when one is named, the project once for the membership (suspended or pending
deactivation leaves no person; an unreadable project or an inactive membership is not a member).
Unknown IDs are refused (owner decision D59 = B). Only application administrators can simulate, and they can already
list every project and person, so the route says when an ID matches nothing instead of answering as for a subject no rule
names. After the switch and capability checks, a project the read confirms does not exist or is deleted answers 422
runtime_targeting_refused with reason unknownProject and message "No project with that ID. It does not exist or has
been deleted."; on a Personal key a person the read confirms does not exist or whose account was deleted (deactivated)
answers reason unknownPerson, "No person with that ID. No account has it, or the account was deleted." (the person is
checked first). A suspended person, or one whose deletion is pending, exists and gets the untrusted-person answer. A read
that does not answer refuses nothing and keeps the fail-closed answer. Real IDs get the same answer as before.
A draft is checked against the schema rules (shapes, duplicates, limits; 422) but not the save-time checks that need the
database or grain (known projects, parent flags, deployment pins): preview a save for those. A Global key, production and a
non-Shared host answer 422 globalKey or simulationUnavailable.
Switch off. Stored rules are kept. Allows (and rollouts) stop applying, while exclusions and stops still apply, so switching off never widens access; on again, the rules apply as saved. Decisions are a function of the stored policy and the switch alone, so on, off, on gives the same decisions with policy written in either state. This holds for the route gates and availability below too: with rules stored and the switch off, an exclusion still closes the pilot's route and availability for its project, while an allowed project falls back to the saved default.
Gradual rollout¶
6F PR F (spec 6F.8, RT-36 and RT-37). A key's one rollout rule turns it on for a stable share of projects (on a
Project key; a rollout over people is refused: personalRuleOnProjectGrain on a Project key, notYetSupported on a
Personal key, where the population preview over people has not been built):
{ "id": "<new guid>", "kind": "rollout", "label": "Gradual", "subjectType": "project", "group": "screening-keywords",
"basisPoints": 2500, "state": "Active" }
- Cohorts. Algorithm
syrf-cohort-v1(RuntimeCohort): the fields["syrf-cohort-v1", environmentId, decimal(saltVersion), group, subjectType, subjectId], each as a big-endian 4-byte UTF-8 length then the UTF-8 bytes, are hashed with SHA-256; the bucket is the first 8 bytes as a big-endian unsigned integer, modulo 10,000; a project is in the rollout when its bucket is belowbasisPoints(decision reasonSplit, and the simulator's rollout step shows the bucket).environmentIdcomes from the environment binding, so each environment has its own cohort and a restored copy with a new binding epoch keeps it;saltVersionis 1. No random, clock, IP or e-mail input exists, and an architecture test pins thatRuntimeCohortreferences none. Rules in the samegroupshare one queue: a 10% feature's projects are inside a 20% feature's. Raising the amount only adds projects. - Vectors.
src/charts/syrf-common/cohort-vectors.json(produced once byscripts/cohort-vectors.py, Python standard library) is reproduced by C# (RuntimeCohortVectorTests), TypeScript (scripts/cohort.test.ts, inpnpm run validate:generated) and Python (python3 scripts/test-cohort-vectors.py, in the script-tests workflow). The vectors are append-only:scripts/cohort.test.tsrefuses a change to an existing vector against the git base; a new algorithm needs a new ID and a new file. - States.
Activeapplies the share.Off(Turn off) removes the rollout's projects (named allows keep theirs) and keeps the amount and group, soActiveagain restores the identical cohort.Pausedholds the amount (the writer refuses a change to its amount, group or subject type,rolloutPaused) and is exact (owner decision D32 = C, spec 6F.F2): only projects that existed when it was paused keep getting it. With the targeting switch off, rollouts (like allows) are not applied; exclusions and stops still are. - Exact Pause (6F.F2). When a rollout moves to
Paused(fromActive,Offor no rollout), the commit stampspausedAtUtcon the rollout rule with MongoDB's time, read in the commit transaction (the writer's other server-time mechanism; the preview cannot know that time, so its proposed rule shows none). A label change while paused keeps it; a submittedpausedAtUtcis ignored. Making the rolloutActiveagain, or turning itOff, clears it, and the field is written only while paused, so every other rollout keeps its stored shape and digest. While paused, a project inside the held share is in only if itscreatedAtUtc(reliable creation times) is absent, which every project created before creation times existed is, or earlier thanpausedAtUtc; a project created at or after the pause is out (decision reasonRolloutPaused, availabilityrolloutPaused) and resuming reaches it. Outside the share nothing changes. A rollout over people follows the same rule with the Investigator'screatedAtUtc, once person rollouts are written. The request context takes the project's creation time from the project read it already makes (the enforced project policy's per-request read where there is one, otherwise one read per request deciding a Project key), and the person's from the read that trusts them; a project read that does not answer counts as created after any pause (fail closed). The simulator's rollout step showspausedAtUtcand the creation time it compared (subjectCreatedAtUtc). A rollout paused before 6F.F2 has no pause time and keeps the old meaning (a new project inside the held share still joins) until it is resumed and paused again; the preview'srolloutNotessays so. - Pause and older binaries. A binary before 6F.F2 refuses
pausedAtUtc(schema rule 3), so the writer stamps a pause only while every fresh API and PM host advertises the policy featurerolloutPausedAton its acknowledgement (policyFeatures); otherwise the pause is refused withschemaUnsupportedByFleetand the preview'sschemaAdmissionisBlocked. Before rolling an environment back below this release, resume or turn off every paused rollout, or the older binary cannot read the policy. Production never stores targeting, so this concerns non-production environments only. - Preview of a change. When a change alters who is in the rollout (an amount raised or lowered, a group changed, turned
off or on), the preview reads the project population in its own snapshot transaction: a covered scan of the
pmProject_idindex (IDs only, at most 100,000, elselimitExceeded), each ID's bucket before and after, then_id $inreads that keep only the non-deleted projects whose membership differs.gainingandlosingthen list exactly the projects whose decision changes (counts, and up to 50 names the administrator can already list); lowering 25% to 5% lists exactly the projects with buckets 500 to 2,499 that no allow or default keeps on. The preview digest covers that list, and a commit recomputes it in its transaction, so the commit goes through only with the digest of a preview that named the same projects (a project created in the lowered range in between makes the preview stale). Pausing at the same amount reads nothing. While the stored or proposed rollout is paused with a pause time, the preview also reads the storedcreatedAtUtcof every project it decides (the population candidates and the projects the rules name), so resuming lists exactly the projects created during the pause that gain the key, and turning a paused rollout off never lists them as losing it.rolloutNotessays in plain words what the proposed state means (Pause: "only projects that existed when it was paused keep getting it"; Turn off; lowering). - Availability. A project outside an
Activerollout that the default leaves off reportsrolloutPending("Coming to you"; decision reasonRolloutPending); aPausedrollout's share cannot grow, so it is not pending. A project inside a paused rollout's held share but created after the pause reportsrolloutPaused(decision reasonRolloutPaused). A request with no project never matches a rollout and gets the context-free decision.
Personal and combined targeting¶
6F PR E (spec 6F.6, RT-33 to RT-35). A Personal key takes three subject types, in one canonical order: pair (a person
in one project), person (the person wherever they are) and project (meaning the project's active members). The
catalog has no Personal key yet: this is the code path a future reviewed grain: Personal contract uses.
{ "id": "<new guid>", "kind": "allow", "label": "Beta testers",
"subjects": [ { "type": "person", "id": "<investigator guid>" },
{ "type": "pair", "projectId": "<project guid>", "personId": "<investigator guid>" },
{ "type": "project", "id": "<project guid>" } ] }
- One person level (owner decision D31 = A). The person is the Investigator ID, which is what the API's
user_idclaim carries; there is no separate "user" level. Allows match pair, then person, then project (the order only picks the explanation). A pair or project allow needs the person to be an active member of that project, read live; an exclusion matches on its subject alone, so it never excludes less than it says. - Who the person is (RT-34).
FeatureTargetContextFactory.ClaimedPersonaccepts exactly one authenticated identity, exactly oneuser_idclaim that parses to a non-empty GUID, and no machine credential (gty=client-credentials, or anenvironment:*scope).FeatureTargetSubjectsthen confirms an active Investigator through the authority snapshot reader: missing, deactivated, suspended or pending deactivation, another ID, an unreadable or malformed record, or a read that throws all leave no confirmed person. Fails closed (owner decision D44 = B): for a Personal key, a request whose person cannot be confirmed (two identities or twouser_idclaims, no or a malformed claim, a machine token, an unknown, deactivated or suspended Investigator, or a failed read) decides Off with the decision reasonPersonUnconfirmed, never on by default: under default On, a person you excluded by name no longer gets the feature just because their identity could not be confirmed. The feature's own route answers the normal unrouted 404, and availability reports itunavailable/notReleasedwithenabled: false, so the feature is simply hidden (no new wire reason, nothing about the cause is disclosed). A stop still reportsStoppedfirst; the rule wins over allows, rollouts and the default. A signed-in request with nouser_idclaim at all counts as unconfirmed on a Personal key. Global and Project keys never read a person and decide exactly as before; so does background work (ForOperation), which never carries a person. The rule applies only where the policy stores targeting for the key and only whileRuntimeFeatureTargeting:Enabledis on; with the switch off nothing changes. A deployment pin that fixes a key's value still decides as the context-free value does. Under support impersonation, full or read-only, the impersonated person is decided once the real caller passes the same checks (6F.6-4). - Membership (RT-35). For a trusted person and an admitted project, one authority read of the project (uncached,
linearizable): the membership must exist and be active in a project that is not deleted. Removing a member, or
disabling the membership, changes availability (
enabled: false) and the gated route on the very next request, with no policy change. The route then answers the project policy's own refusal (a 403 like any non-member's), because the policy no longer admits that person; a member whom no rule reaches gets the unrouted 404. - Bounded reads. The person read happens only when a requested key is Personal grain, and the membership read only
for a trusted person with an admitted project; Global and Project requests read nothing new, and the reads happen only
where the policy stores targeting for the key. The route gate's phase 2, availability (with or without a project), and the
simulator all use the same
FeatureTargetSubjects, so they trust a person on the same terms. - Saving (owner decision D34 = A). Rules hold opaque person IDs only (no names or e-mails). A person the key's stored
rules do not already name must be an active Investigator (
unknownSubjectotherwise). A stored person who later becomes inactive stays in the rules, never matches, and is listed in the preview'sinactivePeoplefor removal; rules are never rewritten automatically, and the person does not block an unrelated change to the key. A rollout over people isnotYetSupported; a rollout over projects never fits a Personal key (invalidRule). - Preview. On a Personal key,
gainingandlosinglist projects whose active members gain or lose the key (decided for an active member no person or pair rule names);gainingPeopleandlosingPeoplelist{ personId, projectId }for the people that person rules (projectIdnull) and pair rules change, IDs only (a person's name needsListUsers), counts and up to 50 each;inactivePeoplelists the proposed rules' people who are no longer active. All three are empty on a Project key. - Not yet. A Personal feature cannot be released through a targeted release intent (
PersonalAudienceNotSupported). RT-33's "left an offered feature" row is decided by consent (6F PR G): U6's storedLeftisOptedOut(availabilityoptedOut), see Consent and environment settings. - Switch. As for project rules: off, person, pair and project allows stop applying while person and project exclusions still apply; on, off, on gives the same decisions with rules written in either state.
Tests: RuntimeTargetingPersonalDecisionTests (the RT-33 table through IFeatureDecisions), RuntimePolicyPersonalTargetingMutationTests
(saving, unknownSubject, inactive people) in SyRF.Mongo.Common.Tests; FeatureAvailabilityHttpTests.Personal.cs (RT-33,
RT-34, RT-35 and impersonation over HTTP) and FeatureTargetPersonTests in SyRF.API.Endpoint.Tests.
Consent and environment settings¶
6F PR G (spec 6F.7, owner decisions D29 = A and D30 = A). Behind RuntimeFeatureTargeting:Enabled (default off
everywhere); no migration, no start-up work, and with no consent configured every decision, gate and availability body is
exactly as before.
- The setting is code's. A registry feature declares
consent: none | offered | required(defaultnone).noneis an ordinary rollout that cannot be left.offered: the feature is on where targeting reaches the subject ("Turned on for you") and the subject may leave; leaving keeps it off even as the rollout grows.required: listed as "Open to join" and off until the subject joins. KillSwitch, Operational and Permission features, and every Global feature, are alwaysnone. A non-nonefeature needs release content: a non-Draftsyrf.topicsguide page for one of its topics, whose title is the beta's name (generated into the C#FeatureDefinition.Titleonly, never the public TypeScript). The registry validator (check 10) refuses each breach with an exact message. - Whose choice. A Project feature's choice belongs to the project and is made by its administrator
(
ProjectEditPolicy); a Personal feature's belongs to the person. The other subject's choice is ignored. - Evaluation.
requiredand not joined is a requirement (step 1, reasonConsentRequired); a storedLeftis an exclusion (step 3, after exclusion rules, reasonOptedOut). Leaving arequiredfeature therefore decidesConsentRequired, notOptedOut, by design: the Beta features page lists people who left a required beta under "Open to join" with Rejoin (availabilityconsentRequiredwithchoice: left). A stop dominates consent: a stopped key reportsStopped(availabilitystopped) even when consent is required and not given, never "Open to join". AJoinednever turns a feature on by itself: it only lifts therequiredgate where targeting already reaches the subject. The context-free decision of arequiredfeature is off (nobody has joined); the route gate's phase 1 still opens when the default is on, so a project that joined reaches the route. Consent applies with the switch on or off, so switching off can never widen access. A failed choice read keeps features with consent off (ConsentRequired) and makes availabilityDegraded. - "Open to join" only where a Join would count (#4572). A subject
that has made no choice on a
requiredfeature getsConsentRequiredonly where targeting would turn the feature on for it once joined (the decision core decides the later steps as if it had joined). Where targeting does not reach it, the decision is still Off and reports the step that keeps it off:RolloutPending/RolloutPausedfor a rollout that has not reached it (availabilityrolloutPending/rolloutPaused, "Coming soon"),Excludedfor an exclusion, andEvaluated(availabilitynotReleased, not listed) when nothing turns it on. So availability never offers a Join that the consent route would not record. For a feature over several keys, availability reportsconsentRequiredonly when the capability would be on with the keys held off for consent turned on; otherwise another key's reason. A stop still reportsStopped, and a storedLeftstill reportsConsentRequired(Rejoin is always recorded). Only the reported reason changed; every value is as before. - Environment settings: stricter only. The effective consent is the stricter of code and environment
(
none < offered < required). An environment tightens with a reason through preview and commit, like targeting (ChangeRuntimeFeaturePolicy, Shared mode, receipt and replay, schema-2 admission):
POST /api/runtime-feature-flags/features/keyword-highlighting/settings/preview
{ "expectedIdentity": { … }, "setting": { "consent": "required", "reason": "Pilot terms not yet agreed" } }
PUT /api/runtime-feature-flags/features/keyword-highlighting/settings
{ "mutationId": "<new guid>", "expectedIdentity": { … }, "setting": { … }, "previewDigest": "<from preview>", "reason": "…" }
A setting not stricter than the code's refuses 422 consentNotStricter (loosening needs a code change); a feature that
cannot take consent refuses consentNotAllowed, a Settled or Withdrawn one consentFeatureNotLive (it owns no keys, so the setting would do nothing), one without release content consentNeedsContent, a malformed one
invalidSetting. setting: null clears it (back to the code's setting). The setting is stored in the policy document's
featureSettings (schema 2). A deployment whose code tightens a setting applies at once with no write; the preview's
codeSettingChanged is true when a stored environment setting no longer tightens anything ("code setting changed in this
deployment"). Production refuses every setting write. Quieter announcements stay with D13 and 6G.
- Joining and leaving. PUT /api/account/features/{featureId}/consent (Personal) and
PUT /api/projects/{projectId}/features/{featureId}/consent (Project), body { "choice": "joined" | "left" }. Validation
is by format only (a 400 never depends on data); otherwise the answer is always 204, and a row is written only when the
caller has the feature's permission (and, for a project, passes ProjectEditPolicy; a reviewer's PUT writes nothing),
the effective consent is not none, and, for joined, targeting reaches the subject or a choice already exists. Off,
both answer a plain 404 before authorization.
- Support impersonation (owner decision on #4523). The account route
records a choice on the impersonated person's behalf under full impersonation: the older ("legacy") contract (M2b
impersonationReadOnlyEnforcement off, which has no read-only mode, so it counts as full: Join, Leave and Rejoin are
allowed and the row and log line carry both identities) or an M2b edit context. Read-only M2b
impersonation is refused 403 impersonation-read-only before the body is read, by the M2b contract and again by the
route's own RefusesReadOnlySupportImpersonation guard, and writes nothing. The project route refuses every
impersonation, full or read-only, with 403 consent_requires_own_session before the body is read: a project's choice
changes the project for every member. The other rules are unchanged: joined only where targeting reaches the
impersonated person, and the 204 that never says whether anything was written.
- Audit of who chose. Every row records the subject (subjectId), the identity the request ran as (actorId: the
person, the impersonated person under impersonation, or the project's administrator) and, always written:
realActorId (the signed-in person who made the request: under impersonation the administrator, from the M2b
ActorInvestigatorId or the principal's own user_id on the legacy contract; otherwise the same as actorId),
recordedUnderImpersonation (true only for a choice made under impersonation; a later choice of the person's own sets
it back to false) and realActorRecordedAtUtc (MongoDB time, equal to updatedAtUtc in the same write). All three are
additive: rows written before them have none and read exactly as before, and a build without them ignores them. A build
without them still rewrites a row's choice and updatedAtUtc and leaves the three as they were, so they describe the
current choice only while realActorRecordedAtUtc equals updatedAtUtc. No migration and no rewrite of stored rows. Each
recorded choice is also logged (Information, FeatureConsentController) with the choice, feature, subject type and ID,
ActorId, RealActorId and RecordedUnderImpersonation: IDs only, never a name or an email. Under M2b, the contract's own
pmImpersonationAudit change-admitted record for the route is written first, as for every edit-mode change.
- Store. runtimeFeatureConsent, _id = "{featureId}:{subjectType}:{subjectId}", read and written by _id only (see
MongoDB reference). Reads ignore rows of another binding epoch. A request reads
choices once (_id $in), only for features whose effective consent is not none, so Global and none features read
nothing (IFeatureDecisions.ConsentPlan). Background work uses RuntimeFeatureConsentReads.ForOperationAsync (the
project's choice only); no Project Management consumer of a Project key exists yet. Erasure on account deletion joins the
D27 / D34 follow-up.
- Availability. An entry for a feature with consent gains consent: { mode, choice, canChange } and title, and the
reasons consentRequired (Open to join, only where joining would turn the feature on, #4572) and optedOut (Rejoin);
rolloutPending comes from 6F PR F. Absent means no
consent. canChange is the caller for a Personal feature (under support impersonation too, on the person's behalf) and a
project administrator for a Project one, never under impersonation. A GET cannot tell read-only from edit mode, so
canChange says who may choose; read-only impersonation still has the PUT refused, and the web disables the buttons from
its own impersonation mode. Permission still comes first.
- Counts. GET /api/runtime-feature-flags/features/{featureId}/choices (InspectRuntimeFeaturePolicy) answers
{ subjectType, joined, left } for the captured binding: counts only, never IDs or names, one anchored _id range.
Tests: RuntimeTargetingConsentTests, RuntimeFeatureConsentStoreTests (including the audit fields and their rollback
behaviour) and the RT-33 U6 row in
RuntimeTargetingPersonalDecisionTests (SyRF.Mongo.Common.Tests); FeatureAvailabilityHttpTests.Consent.cs,
RuntimeTargetingAdministrationIntegrationTests.Consent.cs and the impersonation and catalogue tests
(SyRF.API.Endpoint.Tests); registry check 10 in scripts/feature-registry.test.ts.
Moving a beta to general availability¶
A required step (owner decision on #4523, Leave wording option A). A
beta's Leave confirmation tells the person that it "remains off for you until you rejoin or the feature moves from beta
testing to generally available". That is true only because a generally available feature takes no consent: with
effective consent none, no stored choice is read (IFeatureDecisions.ConsentPlan skips the feature), so a stored Left
no longer keeps it off and the feature follows its release like any other. So when a feature with consent becomes
generally available:
- In the registry PR that sets
lifecycle: Released(the registry's "live for everyone" state; see Lifecycle), setconsent: noneor delete the line. Registry check 10 refuses aReleasedorSettledfeature that keepsconsent: offeredorconsent: required, inpnpm run validate:generatedand in generation, with the messageconsent "<mode>" is not allowed for a <lifecycle> feature; a generally available feature is not a beta, so set consent: none …. - In every environment that tightened it, clear the environment setting (
setting: nullthrough the settings preview and commit above), because the effective consent is the stricter of code and environment. The settings route still accepts a setting for aReleasedfeature (onlySettledandWithdrawnrefuseconsentFeatureNotLive), so this step is checked by review, not by the server. Production refuses every setting write, so it never has one.
Stored runtimeFeatureConsent rows are kept (no migration, no clean-up job) and are simply not read while consent is
none; they join the D27 / D34 subject-deletion follow-up like every other choice. The Beta features pages stop listing
the feature, because an entry carries consent only while effective consent is not none.
The Beta features pages (6F W4)¶
The consent UI. It reads only the availability client (FeatureAvailabilityService, the single discovery source) and the
FeatureGate, and writes only through the generated FeatureConsentHttpService. It lists exactly the entries that carry
consent (effective consent not none); with the switch off, or no feature with consent, it renders nothing new.
- Where. My Account → Beta features (
/manage/beta-features) for Personal betas, and a "Beta features" section on a project's Settings → General page for Project betas. The account link appears only while the person's answer lists a beta (or while the page is open); the project section only for a project administrator (editin the permission report, the sameProjectEditPolicythe PUT checks) and only while the project's answer lists a Project-grain beta. - How an entry is grouped (
shared/beta-features/beta-features.model.ts,classifyBeta), in this order:stoppedshows Stopped (never "Open to join"), under "Using now" for a subject that is signed up (joined, or anofferedbeta not left), which can still leave, otherwise under "Coming soon" with no Join; aleftchoice (oroptedOut) is "Open to join" with Rejoin, for both modes;consentRequiredis "Open to join" with Join;rolloutPendingandrolloutPausedare "Coming soon"; on (the gate) is "Using now" ("Turned on for you" foroffered, "You're in" forrequired); off for another reason after joining (for example a person SyRF cannot confirm, D44 = B, reported asnotReleased) is "Using now", Off, and can be left. Anything else (excluded, unreleased, blocked by a prerequisite, with no choice) is not listed. - Join and Leave. The page sends
{ choice: "joined" | "left" }, then callsFeatureAvailabilityService.reread()(which resolves once that fetch has settled) and reports what the new answer says: "You joined …", "Nothing changed for …" when the server wrote nothing (the PUT always answers 204), or, for a Leave, the owner's wording (option A on #4523): "You left the '' beta. It remains off for you until you rejoin or the feature moves from beta testing to generally available." (the project section: "This project left the ' ' beta. It remains off for everyone in the project until you rejoin or the feature moves from beta testing to generally available."). The second half is true because general availability drops consent (Moving a beta to general availability). The page records one choice at a time, so while one beta's choice is saved and re-read, Join and Leave are disabled on every card (the saving card's button says "Saving…"), never only on the saving card. - One "on" rule. Whether a beta is on for this context is
betaIsOn(beta-features.model.ts): theFeatureGatefor a single-flag feature (the answer'senabled, else the runtime snapshot flag), otherwiseenabled, falling back toavailability. The page, the project section's visibility and the My Account link (hasListedBetas) all use it, so the link and section never appear for a page that would list nothing, or hide while it would list a beta. - Project confirmation. In the project section every Join, Rejoin and Leave first opens a Material dialog
(
ProjectBetaConfirmDialogComponent, M3): the title names the beta ("Leave "…" for the whole project?"), the text says the change affects the entire project and every member and may change how the project behaves for them, and the buttons are Cancel and "Join / Rejoin / Leave for the whole project". Only the action button sends the PUT; Cancel, Escape or a backdrop click send nothing. - Impersonation (the tab's
SupportImpersonationServicestate). On the account page, full impersonation (not-enforced, the M2b contract off, oredit; the older ("legacy") impersonation has no read-only mode and counts as full, an owner decision on #4523) keeps Join, Leave and Rejoin enabled: each button and each outcome says "(on behalf of)", with the name the impersonation banner shows ("the impersonated user" when the app has none), and the notice says the choice is made on that person's behalf and recorded as made by you while impersonating them. Read-only ( read-only, and every state that is not clearly full: checking, entering or leaving edit mode, refused, unavailable) keeps them disabled, with a notice saying why and that edit mode in the banner turns them on. The project section stays disabled under any impersonation, with a short note. A 403impersonation-read-onlyorconsent_requires_own_sessionshows the matching notice.consent.canChange: falsealso disables them. - Notices. "New to you" uses the 6E/6G rule (announcement delivered and not seen), and a beta with a runtime What's New
entry embeds the 6G
syrf-whats-new-entryrow, which records seen and hide. - Search and filters (account page): search matches the beta's name and its What's New text; the filter chips show one group at a time with counts.
Specs: beta-features.model.spec.ts, manage/beta-features/beta-features.component.spec.ts,
project-betas.component.spec.ts (including the confirmation), manage.component.spec.ts (the link) and the reread cases in
feature-availability.service.spec.ts. User guide: user-guide/account/beta-features.md.
Targeted route gates and availability¶
6F PR C (spec 6F.3 and 6F.4). Both act only where the policy stores targeting for the key a request needs
(IFeatureDecisions.HasStoredTargeting: RuntimeFeatureFlags:Mode=Shared, a Project or Personal contract for this host,
outside production, a fresh policy, and at least one stored rule), with RuntimeFeatureTargeting:Enabled on or off. With
the switch on, allows apply; off, only exclusions and stops do. With no stored targeting (always the case until the switch
has been on, because rules can only be written then), and in Legacy or Shadow mode or production, every gate and
availability answer is byte-identical to main before PR C, the pilot included (pinned by a golden recorded against
main's code, FeatureAvailability/Goldens/feature-targeting-switch-off.golden.txt; the stored-rules, switch-off answers
are feature-targeting-switch-off-stored-rules.golden.txt).
- Two-phase route gate.
[FeatureGated]on a Global key is unchanged: one decision during action selection, before authorization, so an off route is a plain 404 for everyone. On a Project or Personal key (today the pilot'sPUT api/projects/{projectId}/screening-keywords), phase 1 asks onlyMayBeOnForSomeTarget(key): the context-free value, or, where targeting is in force, whether requirements pass, no stop holds the key and an enabled allow (or anActive/Pausedrollout) the grain allows exists; with the switch off it is exactly the context-free value. Nothing about any project is decided before authorization. Phase 2,FeatureTargetGateFilter(an action filter, ordered before[ApiController]model validation), acts when the key has stored targeting: it decides the key for the routedprojectIdthat the project policy admitted and answers exactly an unrouted request's 404 (no body, no content type) when it is off, so a member of an excluded or untargeted project sees the route as absent. An absent or deleted project, and a caller the policy refuses, get the policy's own answer before phase 2 runs, exactly as any route of that policy; a non-member therefore gets the same refusal in a pilot project and any other, and learns nothing about the pilot. A test keeps every targeted gate on a{projectId}route behind a project policy. - Context.
FeatureTargetContextFactoryis the one place a request builds aFeatureTargetContext: the project only from the admitted route value (never a body or query), no target at all for a Global-only request, and no person or membership read for a Project key (every member gets one decision). A Personal key reads the trusted person and their membership (Personal and combined targeting), and a feature with consent its subject's stored choice (Consent and environment settings). Background or message work names its project withFeatureTargetContext.ForOperation(projectId), which never carries a person. - Project-aware availability.
GET api/projects/{projectId}/features/availabilitydecides Project and Personal features for that project in the same singleDecideAll. Each entry gainsenabled(the capability is on for this caller here and nothing holds it off), the browser's gate for a non-Global feature, separate fromavailability, which still needs a confirmed release: a pilot creates none, so a Developing pilot isenabled: truebutunavailable/notReleasedin its project. New reasons:excluded,prerequisiteMissingand (PR F)rolloutPending(anActiverollout has not reached this project yet: "Coming soon"). The route without a project omits Project-grain features (D25 A). ThecontextIddomain issyrf-feature-availability-v2and coversenabled. This applies whenever a permitted Project or Personal feature's key has stored targeting, switch on or off (so the response shape does not change when the switch is turned off); with no stored targeting,enabledis absent and the domain stays v1.
The policy document (runtimeFeatureFlagOverrides, _id: "global") has two schemas:
- Schema 1 is today's document, byte-for-byte, with the
syrf-runtime-global-policy-v1digest. It is still written whenever there is no targeting and no feature setting, so removing every rule returns the document to a shape every older binary reads. - Schema 2 adds
targeting(per catalog key, at most 20 rules:excludeandallowrules withenabled, a label and 1–200project,personorpairsubjects, and at most onerolloutwithsubjectType,group,basisPointsandstateActive|Paused|Off) andfeatureSettings(per registry feature,{ consent, reason }). It needs at least one of them non-empty, allows at most 2,000 subjects, and digests undersyrf-runtime-global-policy-v2in one canonical order (keys ordinal, rules by kind then ID, subjects by type then IDs), so array order never changes the digest. Subject IDs are lowercase GUIDs only.RuntimePolicyAuthority.Parserefuses anything else with an exact message (RuntimeTargeting.Messages); a schema-1 document keeps its old messages.
Decisions follow one order, RuntimeTargetingEvaluator (displayed order = evaluation order): requirements (authority
state, deployment configuration, a required parent off for the same subject), stops, exclusions, specific allows (pair,
then person, then project), gradual rollout, the saved setting, the baseline. Steps before the allows can only keep a
feature off; allows and rollouts can only turn it on. Grain is checked when deciding, not when reading: a Project key
reads project subjects only (every member of a project gets one decision), a Personal key reads pair, person and
project subjects (a project or pair allow means an active member), and a rule the grain forbids is ignored with reason
TargetingIncompatible, so a catalog change never takes the policy down. Global keys ignore targeting and decide
exactly as before (pinned against a frozen copy of main's results).
IFeatureDecisions.DecideAll(keys, in FeatureTargetContext) decides for one subject (ProjectId, the trusted
Investigator as PersonId, PersonIsActiveMember); the older Decide and DecideAll(keys) use the empty context.
Request contexts come from FeatureTargetContextFactory (above; the simulator builds a Simulation context for its
named subject with the same FeatureTargetSubjects reads), and allows and rollouts apply only while the switch is on. A key
whose value the deployment fixes (a pin forcing it on or off) decides exactly as the
context-free values do on the targeted path too, so stored rules never override a deployment pin (#4469). The preview's
gains and losses use the same deployment-aware decision (RuntimeTargetingDeployment.Decide). Consent and environment
feature settings follow owner decisions D29 and D30 (Consent and environment settings),
per-rule switches follow D28 (6F PR H), and the single "person" level follows owner decision D31 = A.
Compatibility: hosts advertise policySchemaVersions; a deployment constraint may allow policy schema 1 or 2 and refuses
a schema-2 policy it does not allow; keys with an enabled allow or an Active or Paused rollout join the required-enabled
set; the compatibility report gives the document's policy.schemaVersion and policy.targetedKeys. The anonymous browser
snapshot keeps schemaVersion: 1 and never carries targeting (a targeted key shows its context-free decision). An older
binary given a schema-2 document reports the authority Unavailable and serves safe values, which is why the writer
refuses a first schema-2 write until every fresh host advertises 2.
MVP rehearsal (targeting-mvp scenario)¶
The runtime-fleet harness's targeting-mvp scenario
(bash e2e/run-local.sh --runtime-fleet --scenario targeting-mvp, local stack only, run on Bramble; never part of all)
walks the 6F MVP boundary through two API and two Project Management processes in Mode=Shared. It creates projects P
and Q through the PM e2e setup API (the admin creates both; the reviewer is made a project Administrator of both; the
standard user belongs to neither), then: the targeting routes answer 404 while the switch is unset; with
RuntimeFeatureTargeting:Enabled=true on all four hosts (profile targeting, which also turns on
RuntimeFeatureReleases:Enabled on the APIs because project availability sits behind it) it previews and saves an allow
for P, checks the keyword route and availability for both members and the non-member on both APIs, runs the simulator
for the policy in force and drafts, applies an exclusion with the default On, clears every rule (schema 1 again) and
restarts all four hosts with the switch off (profile targeting-off) and on again. Fleet adoption is read from the
harness key's fleet view: fleet readiness compares policy identities only, and every acknowledgement carries the one
policy document's revision. When #4487 added the scenario the pilot's own /fleet always read Unknown
(DomainReadinessNotEstablished); since 6F PR I it reaches Active too (see Confirming activation). The run evidence for #4487 is kept with the programme's handover notes.
Non-production bound consumer mode¶
PR #4352 connects RuntimeFeatureFlags:Mode=Shared to API membership-disable admission and API/PM statistics
serving plus the existing pages, Stage Overview and SignalR selectors. It requires matching non-production
environment identities and the three pinned binding expectations above. SharedReaderMode must be Disabled;
legacy-format reader enrollment cannot be combined with the bound authority. No defaults, Helm deployment
values or real environments are changed by this PR.
Every audited decision reads the current immutable authority, expires at 30 seconds, and refuses admission on the first failed read. The API disable route becomes 404 while stopped/unavailable, while the enable route retains its permission checks and can restore an already-disabled member. API/PM statistics stop materialized serving/selectors; maintenance, family gates, pilot allowlists and durable lifecycle controls remain separate. Accepted baseline and overrides survive on/stop/restore. The PM selectors use the same normalized evaluator as the API instead of losing their existing runtime overrides during cutover.
The API refresh worker reads a bounded compatibility projection of initialized authority rather than the
legacy store. GET /api/runtime-feature-flags exposes that projection as read-only; no current authority
returns 503. Its legacy numeric revision contract is unchanged in this slice: lossless browser identity and
reconnect handling remain required follow-ons. Authenticated authority status/evaluation already expose
full identity with decimal-string revisions. The legacy PUT/group API refuses writes in Shared mode even
for administrators; the identity-aware permission/preview/receipt write path is still pending.
Only the audited keys bypass the API compatibility cache on each decision. Other API flag consumers keep their existing cached behavior and are not claimed as bounded runtime consumers. PM maintenance and non-enrolled selectors keep their deployment inputs. Legacy and Shadow modes keep their existing consumer paths. Shared mode is a per-host admission option, not a fleet readiness assertion: no Active status, production enrollment, coordinator clearance or rollout approval is implied. Host acknowledgements and the static expected inventory now exist (see the fleet section below); production-grade fleet admission, compatibility reporting and the production gate remain required.
Browser bootstrap and connection recovery¶
PR #4353 hands the exact successfully applied pre-bootstrap snapshot to RuntimeFeatureFlagsService.
The service seeds its revision before subscribing to background fetches, so an older response from another
API pod cannot overwrite the snapshot that selected the application's initial configuration. A successful
bootstrap also resolves the initial-snapshot wait without requiring a second request. If bootstrap failed,
the existing finite retry/fallback behavior remains.
The service refetches on successful first SignalR connection, explicit or automatic reconnection, browser online return and window focus return. The first-connection event is separate from the existing reconnect project-reload event; it does not force a project reload. Revision hints and the five-minute independent poll remain active, including when SignalR is disabled. A newer refresh trigger cancels the previous client fetch, and destruction removes the window listeners and subscriptions.
This repairs global snapshot recovery without changing permissions, saved review work, feature defaults or lifecycle teardown. It does not complete the browser contract: lossless revision transport, contextual identity/digest checks, authenticated/project-specific envelopes, and desired-versus-active lifecycle reporting remain required in later slices. No production or environment activation accompanies this change.
Browser safe fallback (browserRuntimeSafeFallback, 6B.4 G3)¶
If the bootstrap request for the runtime snapshot fails, the browser keeps the deployed values until a later
fetch succeeds. Under an emergency stop that is permissive: E2E deploys disableMembership On, so a tab whose
bootstrap failed shows "Disable membership" although the control is stopped. The server still refuses the action,
so this is about visibility, not security (RT-14, browser half).
The deployment setting browserRuntimeSafeFallback changes that. When it is true and the bootstrap request fails
(a network error, a non-OK status or an invalid snapshot), main.ts calls
applyRuntimeFeatureFlagBootstrapFailure(). That applies RUNTIME_SAFE_VALUES before the app starts, so the
store is seeded with them. The values hold until the first accepted snapshot, which then replaces them with the
server state. A key the snapshot omits falls back to the deployed value, because the deployed values are captured
before the safe values are applied (by design, but not covered by a test; the server always returns every catalogued
key). RuntimeFeatureFlagsService.safeFallbackActive() is true until then, including while a rejected snapshot leaves
the safe values in force. It is a diagnostic and test hook, not a UI contract: nothing renders it. A failed fetch after the first accepted
snapshot keeps that snapshot, as before. Nothing is stored, so each page load decides again. The console warning
says which fallback was used: using safe values … or using deployed defaults.
RUNTIME_SAFE_VALUES(src/app/generated/runtime-safe-values.generated.ts) is written bygenerate:runtime-catalog. It lists the safe value of every key that has a reviewedconsumerContractwithproductionEligible: true, in catalog order. Today that isdisableMembership: false. Statistics contracts are non-production and are not listed.generate-runtime-flag-catalog.test.tspins the list.- The setting is
SYRF__BrowserRuntimeSafeFallbackin thewebConfigsection ofenv-mapping.yaml. The Helm value isruntimeFeatureFlags.browserSafeFallbackin the web chart, defaultfalse, and the browser readsbrowserRuntimeSafeFallbackfromappConfig.env.json(AppSettings; only the string"true"turns it on). It is not a runtime catalog entry: it has no override, is not in the catalog digest and does not needcatalogOnlyFlags. It reaches only the browser.
Flag decision: flagged, by this deployment-static setting, false in every environment until a reviewed
rollout. It changes what every user sees when the bootstrap fails, so it needs a switch. A runtime flag cannot be
that switch, because it would be read before any snapshot exists. Off is exactly the old behaviour. Nothing is
persisted, so on, off and on again are all safe; runtime-feature-flags.service.spec.ts tests that sequence. The
E2E case is runtime-fleet-browser-recovery.spec.ts on the runtime-fleet lane, which sets the setting per tab
through an appConfig.env.json overlay.
Operation-specific administration capabilities¶
PR #4354 separates runtime administration at the existing application-authority gate. Provider status,
authority status and authority evaluation request InspectRuntimeFeaturePolicy; individual and group
policy mutations request ChangeRuntimeFeaturePolicy before reading policy or validating keys/bodies.
EmergencyStopRuntimeFeatures and ManageFeatureReleases are distinct registered capabilities for the
remaining stop and release routes. All four initially require the PM Administrator role (OD-2).
PR #4375 advances the policy version to application-authority.runtime-operations.2 and adds the
Administrator-only BreakGlassRuntimeDeploymentConstraint capability. Existing capability ordinals and the
legacy ManageRuntimeFeatureFlags name are preserved for older clients; the old aggregate is not a
fallback grant for these routes. Self-capabilities includes the new names without changing its string
DTO schema. Off and Shadow application-authority modes retain their existing claim behavior; Enforced
uses current PM authority, denies on unavailability, and authorizes the impersonated target rather than
the real actor. Successful mutations retain both identities in the audit record. Break-glass recovery is
the explicit exception: impersonation is refused before capability evaluation.
Flag decision: use the existing application-authority rollout modes; no new switch or role migration is needed for an operation split whose initial role mapping is unchanged. Production mutation refusal, Shared-mode legacy-write refusal, runtime defaults and saved review data are unchanged. Public bound preview/commit, stop routes, release management and browser presentation of separate capabilities remain follow-up integration work; registering a capability does not expose a new operation or activate a feature.
Bound emergency-stop API¶
PR #4355 adds POST /api/runtime-feature-flags/{key}/emergency-stop for explicit non-production Shared
API mode. It requires EmergencyStopRuntimeFeatures before policy access. Send mutationId, reason
and expectedIdentity copied from the authority observation, including environment, binding/store
epochs, catalog/baseline/policy digests and the decimal-string revision. The writer accepts only an
audited global safe-false consumer; it preserves saved baseline and override values and writes the
stop and immutable audit receipt atomically. Legacy and Shadow modes cannot write through this route.
A successful response contains mutationId and the committed identity; it means durably committed,
not Active or fleet-converged. Consumers observe the stop through their existing bounded refresh path.
A 409 runtime_policy_revision_conflict includes a decimal-string actualRevision: inspect the new
identity before deciding on a new mutation. A 503 runtime_policy_commit_unknown does not prove failure:
retry the same mutation ID and unchanged request, including reason and expected identity. An exact
replay returns its original receipt even after another administrator restores the policy, without
reapplying the old stop. Actor identity is part of idempotency; another actor cannot claim that receipt. A new mutation ID with the current expected identity records another revision even when the key is already stopped; idempotency is per mutation ID, not per key.
The request deadline is ten seconds; a caller that loses the response uses the same retry procedure.
Emergency stops bypass the deployment-constraint and new-audit-index gates as specified in the plan, but never bypass binding, lineage, receipt integrity or authorization. The API has no content or notification dependency and does not wait for a fleet acknowledgement. Restore remains the shared writer's tested ReleaseStop operation pending the reviewed public preview/commit workflow; this slice does not expose a restore route or enroll an environment. Production remains refused.
Reader corrections accompanying the route count time spent reading toward freshness, and treat malformed BSON/argument failures as unavailable so they cannot escape an observation refresh. The catalog digest includes descriptions and definitions: changing either invalidates an old imported authority. Until the catalog-evolution workflow exists, do not enroll or mutate against a changed catalog; regenerate/review an import preview before first enrollment. Never rewrite a stored digest to bypass validation.
Bound preview, commit and restore¶
PR #4356 adds POST /api/runtime-feature-flags/{key}/preview and
PUT /api/runtime-feature-flags/{key}/policy in explicit non-production Shared API mode. Both require
ChangeRuntimeFeaturePolicy before policy access. The additive /policy route has a typed, lossless
commit receipt; the legacy PUT keeps its existing response contract and remains refused in Shared mode.
Send mutationId, the complete decimal-revision expectedIdentity, nullable value, reason, optional
confirmedDependencies, and action (Change or ReleaseStop). Preview is read-only, including under
support impersonation: it writes no policy, receipt, index or lease. Dependency confirmation follows the
existing typed 422 response; submit the exact confirmed key/value set before accepting the final preview.
The response includes the proposed identity, saved override/baseline changes, before/proposed effective
values, requested/dependency/induced-pin causes, and previewDigest. FleetReadiness is explicitly Unknown
until expected hosts and acknowledgements are integrated. A preview is not a rollout approval.
Commit the same request plus previewDigest. The API requires that digest even locally. The store
recomputes it inside the same transaction that writes policy and receipt; it includes the request and
actor, exact authority identity, proposed policy, captured deployment flags and deployment-constraint
guard. Record, extend and release each increment that guard and invalidate outstanding previews,
even when the policy identity itself is unchanged. Changed requests or deployment inputs require a new preview; changed policy identity returns a
revision conflict. Staging also requires a preview at the shared-store boundary. Emergency stops retain
their independent capability/path and do not require preview, content, fleet or deployment-constraint
approval. A Change-policy request cannot smuggle an EmergencyStop action.
To restore, preview and commit ReleaseStop with null value and no dependency changes. Saved values
are preserved; effective values still obey deployed prerequisites. A no-op accepted under a new mutation
ID intentionally advances revision and audit; exact replay uses the original receipt. If a commit returns
503 or its response is lost, retry the same ID and unchanged request, including digest and actor context.
A missing receipt is not proof of abort while another host may still be committing. No environment is
enrolled by this change. Public UI integration, expected-host inventory, coordinator,
production admission, targeting and release/content integration remain required.
Resolving a lost mutation response¶
In explicit non-production Shared mode, an administrator with ChangeRuntimeFeaturePolicy can call
GET /api/runtime-feature-flags/mutations/{mutationId}. Committed returns the original
committedIdentity with its decimal-string revision, even after later changes or a stop restoration.
It confirms the durable receipt, not current policy or fleet adoption. Unknown has no committed identity
and does not prove an abort: retry the same mutation ID and unchanged original request. Do not mint a
replacement ID merely because lookup is Unknown. Storage failures return 503 instead of an outcome.
Outcome responses use Cache-Control: no-store so a previous Unknown result cannot be reused by a browser.
The lookup takes a read-only snapshot of the verified binding, current authority and receipts. It refuses foreign lineages, corrupt or future receipts, and missing current-authority evidence. It neither installs indexes nor runs migrations and remains available in the presence of a deployment constraint. Lineage-import IDs are not policy mutation IDs. Legacy and Shadow modes refuse this route. This adds no environment enrollment or production activation; the default mode remains Legacy.
Host acknowledgements and fleet inspection¶
In explicit non-production Shared mode, API and Project Management publish runtimePolicyHostAck after
startup reconciliation and each jittered 8–12 second poll. Reads retain the five-second deadline; an
acknowledgement write has a separate two-second deadline and cannot invalidate a valid local decision.
Each acknowledgement includes pod UID, process start identity/time, image version, catalog and full
policy identity, last successful authority check, state/error class, polling mode and last-seen time, and (since 6F PR A)
policySchemaVersions, the policy document schemas the binary reads ([1, 2]; a row without it means [1]).
Only checks younger than 30 seconds qualify as current. No user or project identifiers are included.
Publishing requires RuntimeFeatureFlags:InstanceUid (the pod UID, unique in the environment),
ImageVersion, positive ExpectedReplicas for that service, and explicit Autoscaled=false for a fixed
fleet. These are deployment metadata, not feature toggles. Missing UID/image/count produces no acknowledgement;
missing or invalid autoscaling metadata is treated as autoscaled. Migration 0011 must already have installed
the {service, lastSeenAtUtc} index. Publishing never creates indexes or runs migrations. Legacy and Shadow
never publish. A late write from an older process cannot replace a restarted process's record for the same UID.
The migration forward-rule guard explicitly enrolls this writer; its Mongo test pins the UID, service and
BSON date fields, idempotent replacement and absence of TTL. Retained reports have corresponding shape coverage for their environment, collection timestamp and TTL
expiry fields, and workload observations pin their service ID and BSON observation date with no TTL.
Collections without an enrolled writer remain catalogue-only.
GET /api/runtime-feature-flags/{key}/fleet requires InspectRuntimeFeaturePolicy, Shared mode and a
verified environment binding. Responses are not cached. The API needs explicit
RuntimeFeatureFlags:FleetExpectedReplicas:api and RuntimeFeatureFlags:FleetExpectedReplicas:project-management
counts matching each service's rendered ExpectedReplicas; no count is inferred from observed hosts.
For a synthetic two-API/two-PM exercise, both expected counts are 2 and each process has its own UID.
Missing or invalid expected inventory and any autoscaled host remain Unknown. A host's count disagreeing
with inventory requires action. Exact count, compatible full identity and fresh evidence from every host
are required for Active. Active means adoption of this policy, not that its value is On or that a user is authorized.
With incomplete acknowledgements the state is Saved/Applying during the 60-second confirmation window,
then Degraded. Per-host binding/catalog incompatibility is always reported; the aggregate prioritizes
Unknown when inventory is missing or autoscaled, otherwise incompatibility requires action immediately.
This slice can confirm the audited immediate membership control. Statistics controls remain Unknown until separate domain-readiness evidence is integrated. Rows have no TTL; missing or silent hosts never disappear from the evidence. Extra old pod UIDs also prevent Active. Trusted workload collection and explicit retirement of old UIDs, chart rendering of metadata/counts, production admission, the admin UI, metrics and the multi-replica 200-commit freshness exercise remain required before real enrollment. These trial timing bounds are not a measured production guarantee. The timestamp comparisons assume synchronized UTC clocks across the database and hosts; clock skew can make evidence Unknown. The 200-commit trial must measure skew and the worst-case poll/read/write margin before live enrollment. No live configuration is changed by this PR.
Rendering host metadata¶
The API and PM charts can set runtimePolicyHostMetadata.enabled: true to derive each host's four
acknowledgement metadata values from its own Deployment. The default is false and adds no variables.
UID comes from metadata.uid (not pod name), image version is the exact container image including a debug
suffix, expected replicas comes from replicaCount, and autoscaling comes from hpa.enabled. Raw env:
entries cannot override these observations when enabled, including case/colon aliases. The chart does not
select Shared mode or supply binding, migrations, or cross-service FleetExpectedReplicas values.
HPA remains explicitly autoscaled/Unknown. Scale-to-zero is allowed and renders zero, which does not publish a positive-count acknowledgement. Knative opt-in is refused pending trusted workload inventory. This is an opt-in deployment-metadata setting, not an additional runtime policy switch; the existing Legacy mode remains the default. No environment values or live deployments are changed. Trusted inventory must still reconcile cross-service desired counts and retire old pod UIDs before real enrollment.
Administrator deployment constraint recovery¶
PR #4375 exposes POST /api/environment/deployment-constraints/{rolloutId}/break-glass-release for a
stranded coordinator. This is a separate BreakGlassRuntimeDeploymentConstraint application capability,
assigned to Administrator; Change, Emergency stop and Manage releases are not fallback grants. Application
authority off/shadow uses the existing administrator claim decision; enforced mode uses current PM authority
and answers 503 when that authority is unavailable. A single authenticated human identity is required;
automation scopes, multiple authenticated identities and support impersonation are refused.
The request contains mutationId, the full expectedIdentity (decimal-string revision), a nonempty reason
(maximum 1,000 characters), and the reviewed expectedConstraintDigest. The bound store checks the exact
policy identity, active rollout and constraint digest, releases the constraint and writes the before-state,
real actor, reason and receipt in one majority-acknowledged transaction. Recovery does not assert rollout
completion, change feature policy, or enable a feature. It deliberately bypasses the owning coordinator and
completion-evidence requirements; it cannot cross an environment binding or authority lineage.
The flag decision is the existing deployment-only authority mode: this route is available only in explicit
non-production Shared API mode. Legacy and Shadow refuse it and retain all constraint/audit data.
It remains usable if the machine coordinator endpoint is disabled. No additional toggle or migration is
needed for this recovery path. Production Shared mode is still refused.
Requests are limited to 16 KiB and ten seconds; responses are private/no-store. A 409 requires inspection of current policy/constraint state. A 503 may mean a committed operation: retry the same mutation ID and unchanged request, including actor context. Exact replay returns the original timestamp without releasing a subsequent constraint. Do not substitute a new mutation ID merely because the response was lost.
The audit record is durable. Surfacing the last break-glass event in the fleet view and the administrator recovery UI remain ordered follow-ups; this slice provides the authenticated recovery operation only. Tests use synthetic replica-set data; no live constraint or production environment was changed.
Lossless browser snapshot ordering¶
PR #4376 adds revisionText, the invariant decimal representation of the snapshot's Int64 revision.
The numeric revision remains for existing clients. Bootstrap and the browser propagation service prefer
revisionText, validate a canonical nonnegative Int64, and compare using exact integers. A present malformed
string is refused even if the numeric field looks valid. Older servers remain readable only when their
numeric revision is a nonnegative JavaScript safe integer. Unsafe legacy numbers are never rounded into an
ordering decision.
An invalid bootstrap does not change deployed values. An invalid later response retains the last accepted snapshot, releases the initial-navigation wait and leaves future polls/reconnect fetches active. Adjacent revisions above 2^53 cannot overwrite each other in reverse order. SignalR remains an invalidation hint; its numeric payload is not used for ordering. No extra deployment flag is needed for this additive wire correction; runtime modes and feature decisions are unchanged, and no saved data is migrated. Full lineage/context identity, staleness UX and lossless administrator mutation requests remain follow-ups.
Production authority admission (default off)¶
PR #4377 introduces deployment-only RuntimeFeatureFlags:ProductionPolicyMode=Legacy|Runtime,
defaulting to Legacy. In production, Legacy resolves the authority provider to Legacy even when the
raw RuntimeFeatureFlags:Mode remains Shared. Returning to Runtime requires Shared mode, matching
production environment identities and explicit verified binding. The mode changes no saved policy,
baseline, receipt, audit or stop data. This supersedes the earlier blanket production Shared refusal;
it does not enroll a deployment or authorize activation.
Only reviewed production-eligible, global, false-safe contracts may use runtime decisions or mutations.
Currently this is disableMembership. Other flags retain normalized deployment values even if saved
runtime overrides exist or authority refresh fails. Statistics serving and maintenance remain deployment
controlled in production.
Ordinary enabling changes and stop releases that restore On require an exact preview digest and fresh
acknowledgements of the current policy from the configured fixed API/PM fleet, rechecked inside the
mutation transaction. Missing, stale, autoscaled or extra historical hosts refuse admission. Admission
follows RuntimeFeatureFlags:FleetInventoryMode, the same operator-supplied setting as fleet inspection,
which defaults to Static. Static admission counts every retained row, so it refuses after any Kubernetes
rollout until the old rows are removed. With Collector, admission reads the workload observations and
MongoDB's time in the same transaction and evaluates exactly as the inspect view does. Absent rows that
MongoDB has not seen for 35 seconds (15 minutes for legacy rows) are retired, so an enable or restore is
admitted about 35 seconds after a rollout's old pods stop. A live absent row, or missing or stale collector
evidence, still refuses. An unsupported mode value refuses. A passing preview does not establish adoption
of the candidate revision. Emergency stop remains independent of fleet readiness, in either mode, and
preserves saved overrides.
The code slice is tested with production-like binding identities in an isolated replica-set fixture. Production activation remains blocked on the B/C prerequisites: complete compatibility evidence, promotion preflight parity checks for returning to Legacy, recovery exercise, freshness trial and explicit operator approval. Pods start in Legacy without consulting authority parity; the promotion preflight must prevent an unsafe mode rollback. No chart, live binding, credentials or production state is changed here.
Browser lineage guard¶
PR #4378 adds an optional authority envelope to global Shared snapshots: schema version 1 and the
verified environment, binding/store epochs, exact decimal revision and catalog/baseline/policy digests.
Legacy snapshots carry no authority. No targeting rules or user/project identity are included.
The browser pins the first accepted provider mode and lineage for the page lifetime. A changed environment
or epoch, lost authority metadata, unsupported schema or conflicting equal revision stops further snapshot
application and exposes snapshotFault on the service. Current values and unsaved work are retained; no
forced reload occurs. Lower revisions are ignored; equal identities with identical values are idempotent.
Equal policy identities with different effective values also fault, covering deployment fallback drift.
A new page can bootstrap only from the server's verified reader; reconnect alone never acknowledges a new
lineage. Legacy-only revision behavior is preserved. SHA-256 digests accept the server's uppercase
hexadecimal representation and lowercase hex; case alone does not change a snapshot identity.
This is a global transport guard, not personalized authorization or a client security boundary. Server gates remain authoritative. Personalized request generations/context hashes and a user-facing recovery notice remain follow-ups; the service exposes the fault for that notice without destroying review state. No separate deployment flag is needed: authority metadata exists only in explicit Shared mode, and Legacy remains the default. No policy data, migrations or environment configuration change.
Feature registry (schema 2)¶
The generator also writes the feature inventory (docs/features/feature-inventory.md): counts by type and lifecycle, the complexity metrics (flags, evaluation seams, generators, commands and hand-edited files to add a flag, all computed from the sources), the features table, unclaimed flags and removed flags. It has no dates and no runtime state; "Ready to settle" is shown in each environment's admin page from its own history. Its diff in a PR is the reviewable complexity effect.
src/charts/syrf-common/feature-definitions.yaml is the one reviewed source for product features, the preset flag
groups and the deployment-managed pins. PR #4379 introduced it (schema 1); R1b of wave 6R (#4432) made it schema 2.
pnpm feature:new adds a feature (see Adding a New Feature Flag). After a hand edit,
run pnpm generate (or only generate:feature-definitions) and then pnpm run validate:generated. The generator validates the registry against the runtime flag catalog and writes:
| Output | What reads it |
|---|---|
SyRF.Mongo.Common/RuntimeFeatureReleases/FeatureDefinitions.generated.cs |
FeatureDefinitionCatalog.All (release preparation, 6D) |
src/services/web/src/app/generated/feature-definitions.generated.ts |
Read-only metadata (no web consumer yet) |
SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFeatureFlagGroups.generated.cs |
RuntimeFeatureFlagGroups.All and Find |
SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFeatureFlagPins.generated.cs |
ProjectStatisticsRuntimeActivation (pinned keys) |
src/services/web/src/assets/data/whats-new.json |
The home-page What's New feed (from docs/content/whats-new.yaml; see Update the What's New feed) |
CI checks source and output checksums and rejects omitted generators.
Feature fields¶
| Field | Required | Meaning |
|---|---|---|
featureId |
Yes | Stable, permanent ID (lower-case words joined by hyphens); never a flag key |
type |
Yes | Release, Operational, KillSwitch, Experiment or Permission. Sets the defaults below |
owner |
Yes | The team that owns the feature |
settleDays |
No | Overrides the type's settle period (whole days, 1 to 3650) |
grain |
Yes | Global, Project or Personal |
lifecycle |
Yes | Developing, Piloting, Released, Settled or Withdrawn (below) |
flags |
Yes | The flag keys this feature owns. A key belongs to at most one feature |
permission |
Release: yes | The one authorization policy using the feature requires: SignedIn, or an existing Project* or Application* policy name (below) |
audience |
No | Who the feature is for (6R.A): signedIn, projectMembers, projectAdministrators or applicationAdministrators. Defaults from permission (table below); an explicit value may only be narrower |
consent |
No | Whether people join or leave it (6F.7): none (default), offered or required. Always none for KillSwitch, Operational and Permission features and for a Global feature; a non-none feature needs a non-Draft guide page for one of its topics (its title names the beta). See Consent and environment settings |
compatibleWorkflowVersions |
Yes | Reviewed workflow versions (may be empty at any lifecycle; release admission refuses an empty list with WorkflowCompatibilityMissing) |
topicIds |
Yes | Permanent guidance topic IDs (guide pages declare them in their syrf.topics front matter) |
capability |
Yes | Typed Flag/AllOf/AnyOf expression over the feature's own flags |
communication |
No | Overrides of the type's communication defaults; every field is optional |
permission says who may use the feature, so the availability endpoint (Feature availability,
6E.2) checks permission first and reports a feature only to callers allowed to use it. The value is SignedIn (any active, signed-in user) or the name of an
existing ASP.NET policy that starts Project or Application, for example ProjectEditMembershipsPolicy (the policy on
the route that disables a membership). A Stage* policy needs a stage, so it is refused. A Release must declare it;
other types may omit it, and a feature without it is never reported as available (it fails closed). The registry
validator checks the shape (pnpm run validate:registry); FeatureDefinitionPermissionTests in SyRF.API.Endpoint.Tests
proves the name is a registered policy, so a misspelt name fails the build. pnpm feature:new --type Release requires
--permission <value> and refuses, writing nothing, without it. permission is generated into FeatureDefinition.Permission
and the TypeScript definitions; it does not enter the runtime catalog digest. A release Prepared before this field
existed must be prepared again, because Apply refuses a changed definition.
Any registry change to a feature (including adding or changing audience or permission) changes its generated
FeatureDefinition, so every release of that feature that was Prepared before the change must be prepared again, because
Apply refuses a changed definition (#4504). No shared environment has a
Prepared release today (RuntimeFeatureReleases:Enabled is off everywhere).
audience (6R.A). audience says who the feature is for, in words that What's New, walkthroughs, help topics and the
release preview can show ("for project administrators"). It is presentation metadata: no authorization, route-gate or
availability decision ever reads it (FeatureAudienceArchitectureTests fails if one does); permission and targeting stay the gates. Most
features need no audience line: it defaults from permission by the table below, which lives in one place
(PERMISSION_AUDIENCE in src/charts/syrf-common/scripts/generate-feature-definitions.ts). The default is
generated into FeatureDefinition.Audience and the TypeScript registry, and shown in the
feature inventory. The validator (pnpm run validate:registry) refuses, with an exact
message, an audience that is not one of the four values, that is wider than the permission's default (signedIn ⊃
projectMembers ⊃ projectAdministrators; applicationAdministrators only with an application-administrator
permission), or that has no permission to be checked against. A permission the table does
not cover is refused too, so a new policy forces a decision (and FeatureAudienceArchitectureTests fails when the API
registers a policy the table lacks). The default is classified from what each policy requires by default
(ResourceSecurity.json grants and the application and machine-credential policies), not from its name.
| Audience | Default for these permission values |
|---|---|
signedIn |
SignedIn; ProjectViewPolicy and ProjectRequestToJoinPolicy (granted to every signed-in user) |
projectMembers |
ProjectViewStudiesPolicy, ProjectLeavePolicy, ProjectSubmitPdfCorrectionsPolicy, ProjectViewScreeningProgressGraphPolicy, ProjectViewScreeningProgressGraphDecisionsPolicy (granted to every project member) |
projectAdministrators |
ProjectDesignPolicy, ProjectEditPolicy, ProjectBulkPdfUploadPolicy, ProjectApprovePdfCorrectionsPolicy, ProjectInvitePolicy, ProjectSetMembersStatusPolicy, ProjectBulkUpdateStudiesPolicy, ProjectImportSearchPolicy, ProjectViewSearchesPolicy, ProjectRemoveSearchPolicy, ProjectCalculateRobPolicy, ProjectEditMembershipsPolicy, ProjectViewMembershipsPolicy, ProjectDeleteMembershipsPolicy, ProjectExportDataPolicy (the project Administrators group), and the owner-only ProjectChangeOwnerPolicy, ProjectAssignPermissionsPolicy, ProjectDeletePolicy (the narrowest project audience there is) |
applicationAdministrators |
ApplicationSendEmailPolicy, ApplicationListUsersPolicy, ApplicationListEmailTemplatesPolicy, ApplicationModifyEmailTemplatesPolicy, ApplicationBatchAdminProjectsPolicy, ProjectStatisticsParityReadPolicy, ProjectStatisticsOperatorPolicy |
The labels on What's New, walkthroughs, help chips and the release preview, and the Audience column in the admin Features list, arrive with those screens.
communication fields: userGuide ({ kind: Required }, or { kind: Exempt, reason: "…" }), whatsNew
(announce or quiet), notify (releaseAudience, administrators or none), alertingOwner (defaults to
owner) and walkthroughId (a docs/content/walkthroughs.yaml ID; optional; checked both ways by the validator, see Walkthroughs).
The schema 1 guidance field is now communication.userGuide and is refused at the top level.
Defaults by type (owner decision, 2026-10-23). The generator resolves them, so the generated C# and TypeScript carry final values and no reader applies a default:
| Type | User guide | What's New | Notify | Settle period |
|---|---|---|---|---|
Release |
Required | announce |
releaseAudience |
30 days |
Operational, KillSwitch, Experiment, Permission |
Exempt ("<Type> control; no user-facing guidance") |
quiet |
none |
none |
A once-only quiet release of a Release feature is a release-time choice (6D PR A adds it to the Prepare and Apply
requests), not a registry edit.
Lifecycle. Developing, Piloting and Released describe a feature that still has a live flag. Settled
means the clean-up PR has merged: no flag and no branch remain, and the feature is a permanent part of the product.
Withdrawn means the feature was removed from the product. Both keep their former flags and capability keys as
history, and the validator refuses either if one of those keys is still in the runtime catalog. They replace the
schema 1 value Retired; in C#, FeatureLifecycle.Withdrawn keeps Retired's numeric value (3) and Settled is 4.
Release preparation accepts only a Developing or Piloting feature: it refuses a Released one (already live
for everyone) as well as Settled and Withdrawn. A Released or Settled feature is generally available and takes no
consent: setting lifecycle: Released on a beta also means setting consent: none (registry check 10; see
Moving a beta to general availability). (Release content's own publication class Retired
is a different concept and is unchanged.)
Workflow compatibility is checked at release admission, not on the lifecycle label (Chris, 2026-10-23).
lifecycle records the feature's real state, so a feature that shipped before the registry can be Released with an
empty compatibleWorkflowVersions. A release that goes through the process must have reviewed workflow compatibility
at 6D Apply admission. The content compiler still requires every content entry's workflowVersions to be a subset of
its feature's list.
Retirement is fact-based, with no dates in the registry. The registry holds the intent: the type and its settle
period. Each environment's own release history holds the fact: when the feature became available to everyone and
stayed unstopped. A Release feature is ready to settle once that has lasted longer than its settle period; the
admin page that lists it arrives later in 6R, and pnpm feature:settle does the clean-up. Nothing time-based ever fails a build. See §5A (6R.2, "Retirement is
fact-based") of the remaining delivery plan.
Flag groups and deployment-managed pins¶
flagGroups lists the preset groups served in the runtime snapshot (RuntimeFeatureFlagSnapshot.Groups) and
written by the legacy PUT api/runtime-feature-flags/groups/{key}. Each entry has key, name, description,
keys and an optional advertised (default true). The advertised order is the file order. A group with
advertised: false is a retained ID: an older browser snapshot may still submit it, so Find resolves it, but
All never lists it. Group IDs are versioned contracts: never change the keys of an existing ID. Add a new ID and
set the old one to advertised: false.
deploymentManaged lists the flags pinned to their deployed value because a service that never reads runtime
overrides consumes them (today reviewEligibilityPolicy). Each entry has key, constraint (shown on the admin
page) and refusal (the 400 message). The nine statistics maintenance gates stay in
ProjectStatisticsRuntimeActivation, where a parity test checks them against the domain family map.
None of these fields enter RuntimeFeatureFlagDefinition. Its fields feed the catalog digest that every stored
runtime policy identity carries, so RuntimeCatalogDigestPinTests pins the digest value; registry metadata goes on
FeatureDefinition only.
Complexity effect (R1b). Flags 55, caller evaluation seams 5 and generators 5 are unchanged. The hand-written
C# groups (RuntimeFeatureFlagGroups.cs data) and pins (PinnedKeys) are gone: a new group or pin is now one YAML
entry plus generate:feature-definitions, instead of a hand edit in C#. That removes 2 of the up-to-6 situational
hand edits for adding a flag. The cost is two more generated files, which validate:generated checks.
Feature registry validator¶
pnpm run validate:generated runs scripts/feature-registry.ts (alone: pnpm run validate:registry). It
fails on any broken cross-link and names the file and entry:
- Flag keys: every capability, owned
flagsanddeploymentManagedkey is in the runtime catalog; no key is owned by two features or pinned twice; a capability uses only its feature's ownflags. ASettledorWithdrawnfeature is the reverse: none of its keys may still be in the catalog. - Guidance: a feature whose user guide is Required (
communication.userGuide, or theReleasedefault) has a non-Draft guide page (syrf.topicsfront matter) for each topic once it leavesDeveloping(a Developing feature only warns). Each declaration'sfeatureId/topicIdresolves. - Links: each What's New
userGuidePathnames a publisheduser-guide/page by path (.htmlor.md) or front-matterpermalink:(a page with a permalink is reachable only there), and each optionalfeatureIdexists. Help topics resolve throughhelp-topics.generated.ts, which must match the pages' front matter.communication.walkthroughIdand the walkthrough'sfeatureIdpoint at each other; a walkthrough has 1-8 steps, bodies of at most 400 characters (no image or HTML), anchors that exist asdata-tour-id="…"in a web template, and no walkthrough for a feature whose help topics are all restricted (6G.4 W1). - Groups: members are catalogued and unique, and a group ID's keys never change compared with the base
commit:
HEADlocally (the pre-commit hook) andHEAD^1in GitHub Actions, which is the base branch tip of a pull request's merge ref (theValidate Generated Codejobs check out withfetch-depth: 2). Without a readable base it prints "group history unchecked". - Schemas:
env-mapping.yamlagainstenv-mapping.schema.json, and the generatedwhats-new.jsonagainstwhats-new.schema.json. - CI paths: every source it reads matches the PR Tests
Validate Generated Codechange regex and the pre-commitGENERATED_PATTERN, so a registry or guide edit cannot skip it. - Runtime What's New (6G.1): each
runtime:entry indocs/content/whats-new.yamlnames a feature whose resolvedcommunication.whatsNewisannounce, one of itstopicIdsand validstages, with at most one entry per feature and stage. Its body has no image, no raw HTML and onlyhttps://or user-guide links that resolve. A feature that announces needs an entry once it leavesDeveloping(a Developing feature only warns). - Consent (6F.7):
consentisnone,offeredorrequired; KillSwitch, Operational and Permission features are alwaysnone; a non-nonefeature is Project or Personal grain and has a non-Draft release-content entry (asyrf.topicsguide page for one of its topics); aReleasedorSettledfeature (generally available) is alwaysnone(Moving a beta to general availability); andFeatureDefinitions.generated.cscarries each consent-eligible feature's title as the pages give it now (else "run pnpm generate").
Feature IDs are independent of flag keys and permanent topic IDs. The initial definition is
project-membership-disable, owned by API maintainers, with the topic project-membership-status.
It is a Release that owns disableMembership and composes the disableMembership=true requirement.
Guidance is required (the Release default); the existing public
membership guide supplies background, but a content-manifest entry and reviewed workflow compatibility
are still needed before release. Its lifecycle is Developing (it is off everywhere), with no workflow versions;
this catalog invents no milestone date.
Every existing flag is registered (registered in #4438 and reviewed by the owner before merge): 30 features claim the 54 catalogued flags,
each key exactly once (FeatureDefinitionCatalogTests checks this, so a new flag needs its feature in the same
change). By type: 20 Release, 9 Operational, 1 KillSwitch. Each lifecycle is the real state from the deployed
values, all with empty workflow versions (see above): 9 Released, 9 Piloting, 12 Developing. The count of 12
includes the keyword-highlighting project-targeting pilot (6F.2), which keeps its own lifecycle and grain.
Released and Piloting Release features are Exempt from guidance ("shipped before guidance was required" or
"piloting before guidance was required") and quiet in What's New, because no guide page declares their topics and no
runtime What's New entry exists (registry checks 2 and 8). Developing ones keep the Required and announce defaults
with one topic named after the feature, so the validator warns until their content exists. Every Release declares
the permission its page or action needs (registry check 6). # review: comments mark the type, owner, permission
and grouping judgement calls. The file's header comment states the conventions.
Capability expressions are typed Flag/AllOf/AnyOf, bounded to eight nested levels and 64 nodes. Unknown
flags, flags the feature does not own, scripts, empty expressions, duplicate IDs, unknown fields (including a
reviewBy date) and malformed guidance or communication fail generation. Missing flag
values never satisfy even a false requirement. Matching flags is only capability composition: it neither
authorizes a user nor confirms availability, publication or a release. Permission-first availability,
content publication classes and the release writer remain subsequent contracts. Do not place private
article text, audience lists or secrets in these compiled metadata files.
Flag decision: not flagged. These are build-time sources, generation and validation; a flag cannot switch them.
The groups, pins, REST golden and catalog digest are unchanged, which proves runtime behaviour is unchanged, and
the new FeatureDefinition fields have no consumer until 6D PR A, which sits behind RuntimeFeatureReleases:Enabled.
Existing feature decisions, policy, saved data and migrations are unchanged.
Release lifecycle (walkthrough)¶
A release moves one registry feature's flag requirement into effect in one environment and records when that
happened. Everything below sits behind RuntimeFeatureReleases__Enabled=true (default false in every environment) and
Shared authority mode, so production, which runs Legacy policy mode, cannot reach it. The steps, each detailed in the
subsection it links to:
- Prepare (Prepared release intents). Preview, then prepare with that digest. The
record is
Prepared(version 1) with aPreparedevent. No policy changes. - Apply (Applying a release). Preview the Apply against current policy, then apply with
that digest. One transaction commits the policy, moves the record to
Applying(version 2), and writesPolicyCommitted,AnnouncementQueuedand the outbox rows. Admission can refuse it (422). Apply is idempotent by release ID. - Confirm (Confirming activation). The reconciler on each Shared API host confirms the
stage milestone once the fleet is
Activefor the committed policy. The record becomesActive, getsconfirmedAtUtc(MongoDB time) and oneActivationConfirmed. If the fleet is not Active within 60 s the record isDegraded, and it is still confirmed once later. - Announce (Dispatching release announcements). Behind
RuntimeFeatureReleases__OutboxDispatch__Enabled=true(default false). The in-app notice waits for confirmation, then the in-app notice handler completes it, and the What's New row is completed by its handler, the same way in production, staging, previews and local stacks (owner decision D60 A, 26 Oct 2026, which replaced OD-19's staging test sink and the preview and local refusal). Availability then discloses the entry and reports the notice delivered ("New to you") (6G.2). Neither handler sends email. - Stop and restore (Stop and restore recording). An emergency stop or a pause uses
the ordinary policy path, which never reads release data. The reconciler then records
EmergencyStoppedorPaused, cancels the unsent notice once, and moves the record toPaused. A restore recordsRestored, then the record returns toActivewith the same milestone and no second confirmation. - Import history (Importing past releases). A reviewed operator command, run outside
startup, records a release that happened before release records existed:
imported: true, noappliedblock and no outbox rows. The reconciler ignores these records, and availability counts them as released. Imported records are outside RT-22.
A beta's general availability adds a registry step: the PR that sets lifecycle: Released also sets consent: none
and each environment's consent setting is cleared, so stored Left choices stop applying (registry check 10; see
Moving a beta to general availability).
Crash recovery (RT-22). A crash at any step resumes without duplicates. A lost Prepare or Apply response is
resolved by retrying the unchanged request with the same mutation ID, which returns the original receipt. A crash
inside Apply writes nothing, and its retry applies once. A crash inside a reconcile writes nothing, and the next pass
confirms once (ObservedAfterRecovery after a gap of more than three periods). A dead reconciler's lease expires after
30 s of MongoDB time, and a resumed holder whose lease has gone gets LeaseLost and writes nothing (the tested path; the version compare-and-set behind it is not separately driven). A dispatcher that dies holding
a row lets the lease expire after 60 s, and the row is leased again. A re-delivery after the handler ran hits the
handler's deduplication key, so the row is Done once. A stop that commits between confirmation and dispatch cancels
the notice once and sends nothing. RuntimeReleaseCrashRecoveryTests (SyRF.Mongo.Common.Tests, replica set) proves
each boundary. The release-lifecycle scenario of the runtime-fleet harness
(bash e2e/run-local.sh --runtime-fleet --scenario release-lifecycle, local stack only, run on Bramble) runs the whole
lifecycle through real API and Project Management processes, and SIGKILLs the API that holds the reconcile lease. It is
never part of all. It needs a feature the registry can release: on main, project-membership-disable declares no
compatibleWorkflowVersions, so its Apply preview refuses with WorkflowCompatibilityMissing and the scenario stops there. Its
release profile mounts the server help bundle compiled from the checkout's docs/content
(RuntimeFeatureReleases:GuidanceBundlePath), so an announced Apply finds the feature's runtime What's New entry (6G.1).
Before any production release (written prerequisites only; none is done by 6D): a releasable registry entry
(reviewed compatibleWorkflowVersions, and guidance Required with a public article or a reviewed exemption); the
compiled guidance bundle, which every API image built since 6G.7 carries (no value to set); ProductionPolicyMode and
RuntimeFeatureFlags:Mode=Shared in production, with production fleet admission; migrations 0011–0022 applied
and bound; and the release switches set per environment in cluster-gitops. The What's New and in-app notice handlers
exist (6G.2); the screens that show their result are 6G PRs C and D.
Prepared release intents¶
PR #4380 adds a preparation-only slice behind RuntimeFeatureReleases__Enabled=true (default false)
and Shared authority mode. It supports features with one exact flag requirement whose lifecycle is
Developing or Piloting: a Global feature's single saved-value Set here, and a Project or Personal feature's
targeting change as described in Targeted releases. A Released, Settled or Withdrawn
feature is refused at preview and prepare with RuntimeReleaseAdmission.LifecycleNotReleasable ("Only a Developing
or Piloting feature can be prepared or applied."). Preparation
records the reviewed intent; it does not activate policy, verify guidance, schedule execution, publish
content or create a confirmed milestone date. The initial Developing definition remains unreleased.
Preview and preparation require ManageFeatureReleases; retained reads require
InspectRuntimeFeaturePolicy or ManageFeatureReleases. Each permission check precedes mode, request or data inspection:
POST /api/feature-releases/{featureId}/previewreturns the exact release preview digest and expected and proposed policy identities. Required guidance is explicitlyUnverified.POST /api/feature-releases/{featureId}/preparerequires that preview digest and saves the intent plus itsPreparedevent atomically. The server binds the authenticated actor and environment.GET /api/feature-releases/{releaseId}reads the retained preparation in the current environment binding. This read remains available in Shared mode when preparation is disabled.
ReleaseKey identifies a logical release within one environment and feature: a lowercase kebab-case
identifier beginning with a letter, at most 100 characters. A new attempt for the same release retains its
key; a genuinely separate release uses a new key and release ID. Migration 0021 adds the unique
{environmentId, featureId, releaseKey} index alongside the unchanged 0011–0020 migrations.
Preparation refuses missing or incompatible release indexes; flags never provision them.
The preview covers release metadata, actor, policy candidate and the definition snapshot. Preparation serializes against policy, constraint and lineage changes through the existing authority constraint guard, without advancing the policy revision or changing values. Every new prepare advances that shared constraint guard, which the preview digest includes, so any other outstanding preview becomes stale and must be previewed again (an exact retry of an already prepared request returns its receipt and does not advance the guard). The record and receipt share one transaction. An exact mutation retry returns its original receipt; changed requests cannot reuse a mutation ID. An unknown commit response requires retrying the unchanged request, not assigning another mutation ID. Foreign binding or store epochs are refused.
requestedAtUtc is server time. An optional scheduledForUtc is intent only; confirmedAtUtc remains
null and milestones remain empty. Turning preparation off retains records and events; turning it on
again can resolve the original receipt. Preparation and Apply accept only a Developing or Piloting
feature; any other lifecycle is refused with 422 LifecycleNotReleasable (see below). No deployment or
production enablement is included.
Applying a release¶
6D PR A adds Apply behind the same RuntimeFeatureReleases__Enabled=true (default false everywhere) and
Shared authority mode. Production runs Legacy policy mode, so Apply is not reachable there. Both routes need
ManageFeatureReleases, checked before mode, request or data:
POST /api/feature-releases/{releaseId}/apply/previewis side-effect free. Body:{ mutationId, expectedIdentity, reason, announcement, quietReason }. It returns the Apply preview digest (domainsyrf-release-apply-v1, over the stored preparation result, the definition snapshot, the policy preview of the fixed flag change against current policy, the request, the announcements, the guidance result and the matched What's New entry), the expected and proposed policy identities with the policy changes, the guidance result (guidanceState:Verified,ExemptorUnverified;guidanceBundleDigest, the verified bundle's digest;guidanceTopics, the feature's topics found in it), the admission result (admitted,admissionRefusal) andqueuedAnnouncements. The digest also binds the guidance result and, for an Apply that queuesMaterialiseWhatsNew, the matched runtime What's New entry (itsentryIdandrevision) and the verified bundle's digest (#4473), so a changed bundle or entry needs a fresh preview. It is marked read-only safe for support impersonation.POST /api/feature-releases/{releaseId}/applytakes the same body plus thatpreviewDigestand returns the applied release:status: "Applying",version: "2", the committed policy identity,requestedAtUtc(from preparation),committedAtUtc, the announcement choice andconfirmedAtUtc: null. Apply is a change for impersonation.
One MongoDB transaction re-reads the binding, the authority and its receipt chain; resolves a replay by mutation
ID; requires a Prepared record whose definition snapshot equals the current definition; checks the expected
identity and the preview digest; runs admission; commits the policy through the same path as a policy commit
(policy preview digest, production fleet admission, deployment constraint, audit receipt with _id = the mutation
ID); moves the record from version 1 to 2 (status: "Applying", an applied block and an unconfirmed
milestones.{stage} entry with precision: "Instant"); inserts the "{mutationId}:PolicyCommitted" event; and
inserts the announcement rows with an "{mutationId}:AnnouncementQueued" event. A failure anywhere before commit
writes none of them. The mutation-outcome lookup resolves an Apply mutation ID like any policy mutation.
Idempotent by release ID. An exact retry returns the original receipt. The same mutation ID with a different body is refused ("Release mutation ID was reused with a different request."), and so is a second Apply of an applied release ("Only a Prepared release can be applied; this release is Applying."). After an unknown outcome, retry the unchanged request.
Policy drift. The flag change and the definition snapshot are fixed at preparation. If other policy changes have moved the revision since, preview the Apply again against current policy and apply with that digest. A stale identity returns 409; a stale or missing digest returns 400 ("The reviewed apply preview is missing or stale. Preview the apply against current policy and retry."). A changed definition is refused: "The feature definition changed after preparation; prepare a new release."
Announcements come from the feature's resolved registry communication settings: whatsNew: announce queues a
MaterialiseWhatsNew row and notify other than none queues a NotifyInApp row, so a Release queues both
and every other type queues none. announcement: "Quiet" (enum value 1) with a required quietReason queues none;
the preview shows the choice, the digest binds it and the record and PolicyCommitted event keep it. A quiet
reason without Quiet is refused. Rows are Pending, keyed
{environmentId}:{bindingEpoch}:{releaseId}:{stage}:{releaseId}:{stage}:1:{kind} (unique by migration 0016),
with requiresConfirmation: true and the committed identity as guardPolicy. The outbox dispatcher sends them
(see Dispatching release announcements), and the release stays Applying until
the reconciler confirms activation (below).
Admission refuses with 422 { code: "runtime_release_admission", reason, message }, checked in this order:
| Reason | Refused when |
|---|---|
LifecycleNotReleasable |
the feature is not Developing or Piloting |
WorkflowCompatibilityMissing |
the feature has no reviewed compatibleWorkflowVersions |
GuidanceBundleMissing |
the user guide is Required and no compiled server bundle is readable: RuntimeFeatureReleases:GuidanceBundlePath names an absent or unreadable file, or it is unset (the default) and the API's release-guidance/ directory holds no single server-<sha256>.json (an image built without the 6G.7 step, or a local run) |
GuidanceBundleDigestMismatch |
the bundle does not verify: its file name is not server-<sha256>.json, the digest recomputed from its content differs from that name or from its embedded digest, or its format is not a compiled server bundle. Also, deliberately, when the release was prepared with a contentBundleDigest and the mounted bundle is any other bundle: such a release applies only with that exact bundle, because Apply must use the content that was reviewed. The pin applies to the guidance check and, for an Apply that queues MaterialiseWhatsNew, to the What's New check too, so a guidance-exempt feature that announces is pinned as well (#4473). After recompiling the content, prepare the release again |
GuidanceTopicMissing |
a topic in the feature's topicIds has no article for this feature in the bundle (or the feature names no topics) |
GuidancePublicationNotReleased |
a topic has no PublicPreview or PublicReleased article (Draft, PrivatePilot and Retired do not count) |
GuidanceWorkflowNotCovered |
a topic's public articles do not list every version in the feature's compatibleWorkflowVersions |
WhatsNewEntryMissing |
the Apply would queue MaterialiseWhatsNew (the feature announces and the Apply is not quiet) but the verified server bundle has no runtime What's New entry covering this feature and stage, or no bundle verifies. A quiet Apply with a reason (D13) needs no entry (6G.1) |
KeyStopped |
the release's flag is under an emergency stop at the current policy (the same stop test as confirmation and the dispatcher's guard; in production only an eligible key's stop counts). Restore the key with ReleaseStop first, then preview the Apply again: the restore moves the revision, so the earlier preview is stale anyway (#4456) |
IncompatibleHost |
any acknowledged host is incompatible (foreign catalog digest, binding or lineage) |
FleetActionRequired |
the fleet for the key reports Action required |
So project-membership-disable (Developing, no workflow versions, guidance Required) is refused today. The Apply
preview reports the same reason as admitted: false with admissionRefusal, and a refused Apply writes nothing. An
emergency stop never runs admission and never reads the bundle, so it commits while a release of its key is Prepared
or refused.
What's New admission (6G.1). An Apply that would queue MaterialiseWhatsNew reads the same verified server bundle,
before the session opens, and needs a runtime What's New entry (whatsNew in a schema 2 bundle, compiled from
docs/content/whats-new.yaml runtime:) whose featureId is the release's feature and whose stages include the
release's stage. Without one it is refused WhatsNewEntryMissing, at the Apply preview and at Apply, and nothing is
written. The same release applied with announcement: "Quiet" and a reason queues nothing and needs no entry, and a quiet
Apply of a guidance-exempt feature never reads the bundle. A schema 1 bundle (compiled before 6G.1) still verifies but has
no entries, so it admits only quiet or non-announcing releases. Flag decision: not separately flagged; this only tightens
the existing default-off RuntimeFeatureReleases:Enabled Apply.
Which entry admitted the Apply (#4473). The PolicyCommitted event has a whatsNew block beside guidance:
{ entryId, revision, bundleDigest, refusal: null } for an Apply that queued MaterialiseWhatsNew, and null for one
that queued none (quiet, or a feature that does not announce). It records the bundle digest even for a guidance-exempt
feature, whose guidance.bundleDigest is null. The same block, with its refusal, is part of the Apply preview digest.
The prepared-bundle pin (#4473 decision). A release prepared with a contentBundleDigest is admitted only against
that bundle for What's New as well as for guidance: one pin rule, so the event's whatsNew.bundleDigest is always the
bundle the release was prepared against when one was named. A mismatch refuses GuidanceBundleDigestMismatch; a quiet
Apply queues no What's New row, so it is not pinned by What's New. Leaving contentBundleDigest unset (the usual case)
pins nothing, and Apply records whichever verified bundle admitted it. The pin is about audit and consistency only:
availability always serves the current bundle's entry (see What's New and in-app notices),
so readers see edited text as soon as a new bundle is deployed, with its new whatsNew.revision.
Guidance admission (6D PR E, RT-26, decision D-Q6). When the feature's resolved communication.userGuide is
Required, Apply and its preview read the compiled server bundle that
pnpm run compile:release-content wrote (server-<sha256>.json, see
Audience-separated content artifacts) from the path in
RuntimeFeatureReleases:GuidanceBundlePath (environment variable SYRF__RuntimeFeatureReleases__GuidanceBundlePath)
or, when that is unset (the default), from the copy baked into the API image at /app/release-guidance/ (6G.7, below).
There is no separate expected-digest setting: the digest is the one in the content-addressed file name, re-verified by
recomputing the compiler's digest (SHA-256 of JSON.stringify of the payload without digest) from the parsed file, so
reformatting cannot hide an edit. The bundle is read and verified before Apply's MongoDB session opens, so file IO,
hashing and parsing never hold the transaction open; the transaction uses only that result. A release prepared with a contentBundleDigest verifies only against that bundle.
Every topic must have an article for the feature, published for the release's audience, and the articles must cover the
reviewed workflow versions. Admission checks topics and their publication class, not topic revisions: any article
revision in the verified bundle satisfies a topic. Pinning the reviewed revision belongs to 6G's protected revisions
(future work). A release reaches everyone (the Global audience is the only one until targeting, 6F), so
only PublicPreview and PublicReleased articles count. The file is read only for guidance-required features and
for Applies that would announce in What's New (6G.1). The last verified bundle is kept with the SHA-256 of its bytes
(each Apply still reads and hashes the file, but an unchanged file is not parsed again). It is one entry replaced by a
single reference write, so it is bounded without the racy clear it replaced (#4473); a second bundle loaded in the same
process only costs a re-parse. No failure throws: every
problem is one of the refusals above, so admission fails closed. Exempt features skip the check and record the
exemption. The PolicyCommitted event's guidance block records state, exemptionReason, bundleDigest and
topics. Flag decision: not separately flagged. It sits behind RuntimeFeatureReleases:Enabled and only makes
admission stricter; a switch that bypassed it would defeat OD-5.
Where the bundle comes from (6G.7, owner decision D38 A). The API image carries it: CI compiles the bundle from the
same commit and builds it into the image, so each API version carries the file made from its own code and content, and
there is no extra step or value to keep in step. In the reusable image build (.github/workflows/_docker-build.yml,
API job only), the step Compile release guidance bundle installs the root tooling from the lockfile and runs
A compiler error fails the job at that step with the compiler's own message. The API Dockerfile (generated from
docs/architecture/dependency-map.yaml, image_files) then copies .release-guidance/server/server-*.json read-only
(0444) to /app/release-guidance/; with no staged bundle the COPY fails, so an API image is never built without it.
Before anything is pushed, Check release guidance bundles in the image exports the image filesystem and requires
exactly one server bundle in the API image, at /app/release-guidance/<compiled name>, byte-identical to the compiled
file and read-only, and none in the web or user-guide image (pinned by
.github/scripts/test-api-release-guidance-image-contract.sh). With RuntimeFeatureReleases:GuidanceBundlePath unset,
the API reads the single server-<sha256>.json in release-guidance/ beside its binaries (/app/release-guidance/ in
the image); none, or more than one, is GuidanceBundleMissing. An explicitly set GuidanceBundlePath still wins (the
runtime-fleet harness and tests use it). Because the bundle is built from the content, the API's change paths include
docs/content/**, user-guide/**, feature-definitions.yaml and the compiler (detect-service-changes.sh,
.github/services.json): a content-only change rebuilds the API image. To build the image locally, compile into an
empty staging directory from the repository root first (a stale second bundle would make the API refuse both):
rm -rf .release-guidance && pnpm run compile:release-content .release-guidance/server "$(mktemp -d)", then
docker build -f src/services/api/SyRF.API.Endpoint/Dockerfile -t syrf-api .. The P12a Secret route is retired: the chart renders no bundle path,
Secret volume or mount, and the former runtimeFeatureReleases.guidanceBundle.secretName and fileName values are
ignored. The server bundle holds restricted Markdown, so the API image is as private as the bundle; images are pulled
from private GHCR packages. Replace the bundle file, never edit it in place: the bundle cache
stamps path, length and last-write time, so an in-place rewrite that keeps the length and time is not noticed (a new,
content-addressed file name always is). Apply refuses missing or incompatible release and outbox indexes (0012–0017, 0021)
before writing. Turning the switch off refuses new previews and Applies only: GET /api/feature-releases/{releaseId}
still reads the record, now with its current status and version, and turning it on again resolves the original
receipt. Flag decision: no new switch, because Prepare and Apply are one operator workflow behind
RuntimeFeatureReleases:Enabled.
Confirming activation¶
6D PR B adds the confirmation reconciler (RT-21). RuntimeReleaseReconcilerService runs in each API process only
where RuntimeFeatureFlags:Mode=Shared (the runtime-policy writer; decision D11). It is absent in Legacy mode,
in production while ProductionPolicyMode is Legacy, and in Project Management. Every 10 s ± 2 s it lists this
binding's applied releases (Applying, Degraded, ActionRequired, Active or Paused with an applied block,
never an imported one, through the {environmentId, status} index) and reconciles each one under its own lease:
- Lease.
reconcileLease: { token, owner, expiresAtUtc }on the record, taken withfindOneAndUpdateonly when absent or expired by MongoDB's own clock ($$NOW), for 30 s. The commit releases it. Leases avoid duplicate work; the version compare-and-set (with the lease token in the same filter) and the unique event index make confirmation exactly-once, so any number of API replicas is safe. - One transaction. It reads MongoDB's time first, then the binding, the authority and its receipt chain, then
the record. Every decision uses that MongoDB time and stored evidence only, never the API host's clock: the
fleet's freshness checks, the 60 s window and the recovery gap. So a skewed API clock, or which replica holds the
lease, changes neither a decision nor a confirmed instant (decision D12). Host acknowledgements still carry each
host's own clock, as before; a host whose clock is far from MongoDB's reads
Unknownand blocks confirmation. - Confirm when all of these hold: the status is
Applying,DegradedorActionRequired; the stage's milestone is unconfirmed; the feature's reviewed flag requirement (from the record's definition snapshot) holds for the current policy and its key is not stopped; the current store epoch equals the applied one with a revision at or above it; and the fleet for the key isActiveat the current policy (every expected hostCurrent; see Host acknowledgements and fleet inspection). It then setsmilestones.{stage}.confirmedAtUtcandrecordedAtUtcto the transaction's MongoDB time,confirmedPolicyto the current identity andsemanticstoObservedActivation, orObservedAfterRecoverywhen more than 30 s (three periods) passed since the last reconcile (or since the commit, if none ran): the instant is when MongoDB observed activation, never an earlier guess. The record becomesActiveand the event"{applyMutationId}/{releaseId}:ActivationConfirmed"recordsconfirmedBy(the pod'sRuntimeFeatureFlags:InstanceUid), the confirmed policy and theCurrenthosts. A confirmed milestone is never rewritten. - Degrade. An
Applyingrelease whose fleet is notActive60 s after its commit (or, after a restore, 60 s after the restore was recorded) becomesDegraded, orActionRequiredwhen the fleet reports Action required. No milestone or event is written, and reconciliation continues, so a laterActivefleet still confirms once. - Observation. For an unconfirmed release, every pass records
lastReconciledAtUtc(the heartbeat the recovery gap is measured from),lastFleetStateandfleetReason. A confirmedActiveorPausedrelease is evaluated without a lease and is written only when a new policy receipt exists or its fleet state or reason changes, so a stable release keeps its version and is not rewritten every pass. It needs no heartbeat: its milestone is never written again.
Only a key whose reviewed consumer contract decides at new-request-admission can reach a fleet Active:
disableMembership and, since 6F PR I, the targeting pilot screeningKeywordHighlighting. Such a key is decided at each
new request from the adopted policy, so identity adoption is activation. Any other key (the statistics keys, which
decide per read operation and also need domain readiness, and keys without a contract) reads Unknown
(DomainReadinessNotEstablished), so its release stays Applying and then Degraded, and never confirms. The fleet
compares policy identities only, never a flag's value; for a targeted release, "the capability holds" is judged per
subject by the reconciler (see Targeted releases).
GET /api/feature-releases/{releaseId} returns milestones (keyed by stage) and confirmedAtUtc from them.
A pass that cannot list releases (release indexes 0012–0015 or 0021, the audit indexes 0009–0010 or the
outbox index 0022 missing or incompatible, an unverified binding) logs one warning and writes nothing; a failed
release is logged and retried on the next pass. Flag decision: not separately flagged. It only converges releases
an operator already applied, which must continue when RuntimeFeatureReleases:Enabled goes off; it does nothing
without applied records and is absent in Legacy mode.
Stop and restore recording¶
6D PR D records stops and restores in the release's history (RT-24). The stop path does not change: an emergency
stop, a pause (Change to off) and a restore (ReleaseStop, or Change back on) still go through the ordinary
policy commit, which reads no release record, event, outbox row or document. An emergency stop therefore commits with
the release, event and outbox indexes dropped, the dispatcher failing and the reconciler stopped; the record catches
up on the next pass after they return (RuntimeReleaseStopRestoreTests, replica set).
In the same reconcile transaction, before confirming anything, the reconciler reads the policy receipts after the
record's applied.lastObservedRevision in {storeEpoch, revision} order (audit index 0010), at most 100 per pass,
and advances lastObservedRevision past them. For each receipt that changes whether the release's capability holds
(the same test as confirmation: the reviewed flag value and no stop on its key), it appends one event:
| Receipt | Event | Record status |
|---|---|---|
EmergencyStop that stops the capability |
EmergencyStopped |
Paused |
Any other change that stops it (a Change to off) |
Paused |
Paused |
ReleaseStop, or a Change, that makes it hold again |
Restored |
Applying, then Active once the fleet is Active |
A receipt that changes neither (an unrelated key, a second stop while stopped) appends nothing. Each event's ID is
"{receiptMutationId}/{releaseId}:{type}", so it is exactly-once by the 0013 index, and it records
sourceMutationId, sourceRevision, action, key, stage, occurredAtUtc (the receipt's own recordedAtUtc),
occurredPrecision: "Instant", provenance: "Observed", observedPolicy, observedBy and the status before and
after. A stop and its restore inside one reconcile period are both recorded, in revision order.
Milestones are never rewritten. A restored release whose milestone was already confirmed goes back to Active
once the fleet is Active for the restored policy, with no second ActivationConfirmed; its confirmedAtUtc,
requestedAtUtc, release ID and announcement ID stay byte-identical. A release stopped before it was confirmed goes
Paused rather than staying Applying, and is confirmed once, as usual, after its restore. Nothing is confirmed
while the history scan is behind (more than 100 receipts pending). A restore sets lastRestoredAtUtc, from which the
60 s confirmation window then runs.
History assumes the capability held at the Apply revision. That holds because Apply refuses a key under an emergency
stop (KeyStopped, #4456); before that refusal, a release applied
while its key was stopped was recorded as Paused at the next, unrelated policy receipt. If the dispatcher leases an unsent
notice between the reconciler's read and its cancellation, the reconciler's compare-and-set fails and the whole stop
recording rolls back; the next pass records it, so Paused can appear one reconcile period late.
Stop-time cancellation. In the transaction that records a stop, the reconciler cancels the release's unsent
in-app notices (Pending NotifyInApp rows, found through the {releaseId, state} index 0022) exactly as the
dispatcher's guard does: Cancelled, cancelReason: "CapabilityStopped", cancelledByMutationId and one
"{firstStopMutationId}/{releaseId}:AnnouncementCancelled" event, where the first stop is the first receipt since
Apply that stopped the capability. Both writers derive that same ID and insert the event only if it is absent, so a
notice is cancelled once with one event, whichever sees the stop first. A Leased row is left to the dispatcher
holding it, whose guard cancels it under the same ID. MaterialiseWhatsNew rows are not cancelled: the dispatcher
defers them while stopped. A restore never requeues a cancelled notice (D-Q5); a new notification needs a new
announcement revision. Flag decision: not flagged, because it is part of the reconciler (absent in Legacy mode) and
the stop path does not change.
Malformed notices (#4458). A Pending NotifyInApp row that
does not parse (no guardPolicy, a guardPolicy without a GUID storeEpoch and an integer revision, or anything the
dispatcher's row parse rejects) is poison. The stop-time cancellation skips it, so the stop event and Paused are still
recorded and the well-formed notices still cancelled. After that transaction commits, the reconciler marks the row
terminal in its own write: state: "Failed", lastErrorClass: "MalformedRow", failedAtUtc from $$NOW, with
attempts untouched. It logs the row ID at Error. Later passes do not see the row again. Rows are only written by
Apply, so this arises only from a bug or a manual edit. If the process dies between the commit and that write, or the
dispatcher leased the row meanwhile, the dispatcher's guard marks it the same way.
Importing past releases¶
6D PR F adds a reviewed operator command for releases that happened before release records existed (RT-27, plan
§8.5), RuntimeReleaseHistoryEntryPoint. Like the lineage command it runs
the API image's normal entry point before startup, never as part of startup, migrations or a flag. It is a written
operational prerequisite only: it has not been run in any shared environment, and the operator procedure is
runbook 11. Flag decision: not flagged,
because, like initialise, it does nothing unless an operator runs it.
The reviewed input is a JSON file with exactly these fields:
| Field | Value |
|---|---|
featureId |
a feature in the generated feature definition catalog (any lifecycle) |
releaseKey |
lowercase kebab-case, beginning with a letter, at most 100 characters |
stage |
LimitedRelease or GeneralRelease |
precision |
Date, Month or Unknown |
occurredOn |
yyyy-MM-dd for Date, yyyy-MM for Month, absent or null for Unknown; never in the future |
source |
where the date comes from, for example the whats-new.json entry (at most 1000 characters) |
reviewer |
who reviewed it (at most 200 characters) |
SYRF_RUNTIME_RELEASE_HISTORY_MODE=preview
SYRF_RUNTIME_RELEASE_HISTORY_INPUT=/reviewed/history-input.json
SYRF_RUNTIME_RELEASE_HISTORY_OUTPUT=/reviewed/history-preview.json
plus the Mongo configuration, RuntimeEnvironment and the SYRF_MIGRATION_BINDING_EXPECTED_* binding expectations,
as for the lineage command. The preview is read-only and deterministic: the release ID and the mutation ID are derived
from the binding, the feature and the release key, so the same input in the same binding always gives a
byte-identical plan. It prints the exact record and event documents and the previewDigest; apply fills only the
fields listed in its applyFills (recordedAtUtc, importedBy and previewDigest), which are null in the preview.
SYRF_RUNTIME_RELEASE_HISTORY_MODE=apply
SYRF_RUNTIME_RELEASE_HISTORY_INPUT=/reviewed/history-preview.json
SYRF_RUNTIME_RELEASE_HISTORY_OUTPUT=/reviewed/history-receipt.json
SYRF_RUNTIME_RELEASE_HISTORY_ACTOR=<operator identity>
Apply recomputes the plan from its input against the current binding and catalog and refuses unless the reviewed
digest still matches. It needs the migration-owned release indexes (0012–0015, 0021) and never creates one. In
one transaction it writes:
- the record:
status: "Active",version: 1,imported: true, noappliedblock, andmilestones.{stage}withsemantics: "Imported", theprecision, the reviewedoccurredOnliteral andconfirmedAtUtcset to the first instant (UTC) of that day or month, or null forUnknown; - one
"{mutationId}:HistoricalImported"event:provenance: "Imported",provenanceDetail: { source, reviewer },semantics: "Imported",occurredAtUtc,occurredPrecision,occurredOn,importedByand thepreviewDigest.
It writes no policy, no audit row and no outbox row: nobody is notified about a past release. The confirmation
reconciler lists only records with an applied block and without imported: true, and skips an imported record it
is asked to reconcile by ID, so it never leases or writes one. Apply and its preview
refuse an imported release ("An imported historical release is history only; it cannot be applied."), and
GET /api/feature-releases/{releaseId}, which reads a release's preparation, returns 404 for one. An imported key
also blocks a later preparation with the same key.
An exact replay of the same preview returns the original receipt and writes nothing, so after an unknown outcome retry the identical preview with a new output path. Refusals (exit 2, nothing written): an unknown feature ("Unknown feature: …"), a modified preview ("The reviewed import preview was modified …"), a stale one, for example after a rebind ("The reviewed import preview is stale: …"), a different import or a prepared release holding the same key ("A release with this feature and release key already exists …"), invalid input, a missing index or actor, an existing output path, and a combined migration or lineage mode.
Targeted releases¶
6F PR I (spec 6F.10) releases a Project or Personal feature to an audience instead of to everyone. It sits behind
RuntimeFeatureReleases__Enabled=true and RuntimeFeatureTargeting__Enabled=true (both default false in every
environment) and Shared authority mode. Production stays Legacy and refuses every targeting write, so it cannot be
reached there. Global features keep the single Set path above, unchanged. No migration and no start-up work.
Prepare. The preparation body (POST /api/feature-releases/{featureId}/preview and .../prepare) sets
policy.action to Targeting, leaves policy.value null and carries the key's complete replacement rules in
targeting: { rules }, in the targeting rule shape. The rules' enabled allow subjects are
the release's audience (a LimitedRelease to them). Every targeting save-time check applies, with its reason
(runtime_targeting_refused: personalRuleOnProjectGrain, unknownSubject, parentOff, notYetSupported, limits),
plus these release refusals (422 runtime_release_admission):
| Reason | When |
|---|---|
TargetingRequired |
A Project or Personal feature is prepared with a saved-value Set, which would reach every subject |
AudienceMissing |
The rules name no enabled allow subject (for example, exclusions only), or the feature's requirement is false (an allow can only turn a flag on) |
RolloutAudienceNotSupported |
The rules include a rollout (any amount, 10,000 basis points included): a targeted release's audience is its named allow subjects. Refused before the targeting checks, so the answer does not depend on whether rollouts can be saved (6F.8) |
RolloutPresent |
The key's current rules already hold a rollout, Active, Paused or Off (6F.8). A targeted release replaces the key's whole rule set, so it would drop the rollout; remove the rollout first. Refused at preview, prepare and Apply (a rollout added after preparation is refused at Apply with this reason, not TargetingChanged), before anything is written. Unlike the other Apply refusals, which come back as a 200 preview with Admitted: false and an AdmissionRefusal, Apply preview answers this one with a 422 (runtime_release_admission, the refusal's message), because the preview cannot be built with a rollout present; a release screen must show that message |
PersonalAudienceNotSupported |
A Personal feature (6F PR E): its rules may name people, but confirmation decides the audience as projects without a person, so it could never confirm. Save its rules through the targeting routes instead |
A Targeting action on a Global feature is refused (globalKey). The request stays within the 32 KiB release bound.
The record also stores targetingBase, the key's rules at preparation (null when it had none).
Apply. Apply rebuilds the rules from the Prepared receipt and commits them through the ordinary targeting
mutation path inside the Apply transaction: the globalPolicyMutation receipt carries request.action = "Targeting"
and request.targeting = { key, rules }, with the targeting preview digest, the project existence check, the
deployment constraint and schema-2 admission (schemaUnsupportedByFleet, a runtime_targeting_refused refusal from
the commit; the Apply preview does not report it). Every existing admission check still runs: lifecycle, workflow
compatibility, guidance, What's New, KeyStopped and the fleet (IncompatibleHost, FleetActionRequired). One more
refusal: TargetingChanged when the key's rules differ from targetingBase, because Apply replaces the key's rules and
would silently undo that change; prepare a new release against the current rules. An unrelated policy change since
preparation still needs only a fresh Apply preview. The record's applied block gains targeting: { key, rules }, the
rules Apply committed. Releasing an audience a tester already piloted through the targeting route works: the rules are
committed again as a new revision.
Confirm. The reconciler confirms a targeted release when, besides the usual conditions, every project of its
audience decides the flag on at the current policy (through the same deployment-aware evaluator requests use, as an
operation context: stops, exclusions, required parents and pins apply) and RuntimeFeatureTargeting:Enabled is on
in the reconciling API, so no milestone is confirmed while no host applies allows. The fleet must be Active (see
Confirming activation). The same per-audience test drives the stop and restore history and
the outbox guard: a later targeting write that keeps the audience on records nothing; one that excludes a project of
the audience records Paused; allowing it again records Restored. Those tests read the policy as stored, so turning
the switch off never writes release history.
Availability. available needs a confirmed release and enabled, so it holds exactly where the release's rules
turn the feature on: in a project of the audience, not elsewhere, with no per-project release matching. The release
block names the feature's release in every project.
GeneralRelease stays explicit (§8.1). No policy, however wide (a saved default On, or a rollout at 10,000 basis
points once 6F.8 lands), adds a GeneralRelease milestone; it needs its own intent, a separate record with its own
confirmation.
Personal audiences are refused (PersonalAudienceNotSupported) until releasing to people is built. RuntimeReleaseTargetedIntentTests
(SyRF.Mongo.Common.Tests, replica set) and FeatureAvailabilityHttpTests.TargetedRelease.cs (Prepare, Apply and
confirm an allow for P, then availability available in P and notReleased in Q) cover it.
Reading releases (admin Releases screen)¶
6G PR AD2 adds the read routes the admin Releases screen uses (the screen is AD2b). They are read-only, need InspectRuntimeFeaturePolicy
or ManageFeatureReleases (either one is enough; a caller with neither gets 403, and the check precedes any read), and
sit behind RuntimeFeatureReleases:Enabled: while it is false the list and events routes answer 404
feature_availability_disabled before authorization and read nothing. Every query is filtered to the current
environment binding (environmentId plus bindingEpoch), so a release, event or announcement row written under another
epoch or environment never appears. Nothing here writes, and there is no migration.
GET /api/feature-releases?featureId=&status=&before=lists this binding's releases, newest first (requestedAtUtcdescending, an imported release last, then release ID), at most 50.statusis a record status (Prepared,Applying,Active,Degraded,ActionRequired,Paused); an unknown one is 400. The response is{ releases, nextBefore }: passnextBeforeasbeforefor the next page (absent on the last page); abeforethat is not a release of this binding is 400. Each item carries the feature, release key, stage, status, version,imported, milestones,confirmedAtUtcand itsannouncements. An imported release lists here (withimported: trueand norequestedAtUtc) although its detail GET is 404. The query uses migration0012(environmentId,status), or0021(environmentId,featureId,releaseKey) whenfeatureIdis given, by hint, and sorts the binding's releases in memory; an explain test pins both plans to an index scan.GET /api/feature-releases/{releaseId}/eventsreturns{ releaseId, events, truncated }: the release's events in recorded order through0014, at most 200 (truncatedsays more exist). An event shows its type, recorded and occurred times, stage, provenance and, forPolicyCommitted, the Apply reason; no actor, request body or digest. A release outside this binding, or unknown, is 404.GET /api/feature-releases/{releaseId}gainsannouncements: [{ kind, state, attempts, lastErrorClass, cancelReason, completedAtUtc, testSink }], read from the release outbox through0022.testSinkis whether a finished delivery went to a test sink (null until something is delivered): false for every delivery since D60 A, true only on a staging row the retired test sink completed before then. While the switch is off the block is omitted and the outbox is not read; the rest of the detail is unchanged.
One malformed release row blocks the list. If a release row cannot be parsed (a FormatException from the prepared
record, or from the imported-record check), the whole list answers 503 runtime_policy_unavailable, as the detail GET
does for that release. Failing closed is intended: the list never silently skips a record. An operator who sees a
blocked list should look for a bad row in the release collection (filter by the current environmentId and
bindingEpoch), not for an outage.
Dispatching release announcements¶
6D PR C adds the release outbox dispatcher (RT-23; RT-25 as amended by D60 A) behind its own switch,
RuntimeFeatureReleases__OutboxDispatch__Enabled=true (RuntimeFeatureReleases:OutboxDispatch:Enabled, default false
everywhere). It is a plain configuration switch, like RuntimeFeatureReleases:Enabled, not a catalogued runtime flag.
The hosted service RuntimeReleaseOutboxDispatcherService is registered only on an API host in Shared authority mode
(the one writer host); Legacy (production today) and Shadow hosts never run it. Every 10 s ± 2 s it dispatches up to
20 due rows:
- Lease. One
findOneAndUpdateagainst MongoDB's clock takes aPendingrow, or aLeasedrow whose lease has expired, withnextAttemptAtUtc ≤ $$NOW, and setsstate: "Leased",lease: { token, owner, leasedAtUtc }andnextAttemptAtUtc = $$NOW + 60 s, so a holder that dies releases the row through the same0017scan. Only rows this host can route are leased; a kind with no handler staysPendingand visible. - Guard, in one snapshot transaction before any side effect, in this order:
| Check | Outcome |
|---|---|
The row does not parse (wrong schemaVersion, a deduplication key that does not match, a missing field) |
terminal Failed, lastErrorClass: "MalformedRow", logged at Error |
| The row's environment or binding epoch differs from the current binding (a restored copy) | Cancelled, cancelReason: "BindingChanged" |
An outbound kind (NotifyInApp) under a binding whose environment ID is not valid (fail closed; no valid binding reaches this since D60 A) |
Cancelled, "DeliveryDisabled" |
No guardPolicy, or one without a GUID storeEpoch and an integer revision |
terminal Failed, "MalformedRow", logged at Error |
NotifyInApp, and some policy revision since Apply stopped the release's capability (emergency stop or a change that turns it off) |
Cancelled, "CapabilityStopped", plus one "{stopMutationId}/{releaseId}:AnnouncementCancelled" event; it stays cancelled after a restore (decision D14) |
MaterialiseWhatsNew while the capability does not hold |
deferred ("CapabilityNotHeld"); What's New returns with its original date once restored |
The release is not Active, or its milestone is unconfirmed |
deferred ("AwaitingConfirmation") |
Deferral sets Pending, deferrals + 1 and nextAttemptAtUtc = $$NOW + backoff; it does not count as an attempt.
A malformed row is poison (#4458): failedAtUtc from $$NOW,
attempts untouched, the pass outcome Quarantined, and it is never leased again. The binding checks run between
the two malformed-row checks, so a restored copy's row is still cancelled BindingChanged.
3. Handle. Each kind reaches the one handler registered for it, the same in every valid environment (D60 A, 26 Oct
2026). The binding's stored announcementDelivery is not read, so a binding written before D60 with testSink
(staging) or disabled (previews, local stacks) delivers exactly as one written since, which stores production.
No re-bind or data change is needed. Handlers are idempotent by the deduplication key.
4. Complete by compare-and-set on the lease token: Done, completedAtUtc, attempts + 1 and result ({ handler,
testSink: false, announcementDelivery: "production" } in every environment, the shape production rows always had). A handler
failure, including the handler's own timeout or cancellation, sets attempts + 1, lastErrorClass and Pending
after the backoff; the 20th makes the row terminal Failed (failedAtUtc). A guard that cannot run (authority not
initialised, binding changed mid-pass, the release record missing) sets guardErrors + 1 and lastGuardErrorClass
and backs off without touching attempts, so no outage can make a row Failed. A malformed row is not a guard
error: the guard marks it MalformedRow (step 2). Only a shutdown of the pass itself leaves the
row leased until its lease expires. An operator requeue is a follow-up.
Readiness (the indexes and the binding) is checked once per pass, and each row's guard still re-reads the binding in its own transaction.
The backoff is 15 s, doubling per failure or deferral, capped at 1 hour. A stop is never delayed by any of this: the stop path does not read the outbox.
Handlers. The API registers one handler per kind (the dispatcher refuses two), the same in every environment:
| Handler | Kind | Effect |
|---|---|---|
RuntimeReleaseWhatsNewHandler (6G.2) |
MaterialiseWhatsNew |
none: completing the row is the record availability reads |
RuntimeReleaseInAppNoticeHandler (6G.2) |
NotifyInApp |
none: completing the row (result.testSink: false) is what availability reports as delivered |
Both are idempotent, so a repeated delivery changes nothing, and neither sends email or anything else outside the
environment's own database. So in every environment, once the release is confirmed, the in-app notice and What's New
end Done by these handlers (What's New is deferred while the capability does not hold; the notice is cancelled
CapabilityStopped if a stop came first). There is no projection store: the outbox rows are the record of what was
announced.
Before D60 A (26 Oct 2026; OD-19, superseded). Staging sent the in-app notice to RuntimeReleaseTestSinkHandler,
which recorded it in the staging-only runtimeFeatureReleaseTestSink collection (result.testSink: true, never reported
as delivered), and previews and local stacks cancelled it DeliveryDisabled. The handler and its routing are removed.
Rows they finished are kept unchanged (rows are never rewritten): a staging notice completed by the sink stays
undelivered and shows "Sent to test sink", and a cancelled one shows its reason. The runtimeFeatureReleaseTestSink
collection is no longer written or read; any rows in it stay until a later clean-up.
Retention. Done, Cancelled and Failed rows are never deleted: there is no TTL index, no deleteAfterUtc and no
clean-up job (ADR-023, forward only). Migration 0022 adds {releaseId, state} for the bounded per-release scan.
Dispatch refuses to run (logs and writes nothing) until 0009, 0010, 0013, 0016, 0017 and 0022 exist, and
refuses if the outbox has any TTL index.
Switching. Off, a pass reads and writes nothing, so rows stay Pending and kept. On, the rows are dispatched under
the same guards, so a notice for a release that was stopped meanwhile is cancelled rather than sent. Off, on, off, on
delivers nothing twice (RuntimeReleaseOutboxDispatchTests, replica set). Flag decision: flagged, because this is the
only 6D component with outbound side effects and it needs its own kill switch. D60 A added no flag: it changes only
non-production delivery (production's routing, results and responses are unchanged), it changes no stored data, and it
is rolled back by reverting the change.
Feature availability¶
6E.2 answers "which released features can this person use here" for discovery surfaces (What's New and walkthroughs,
6G and 6G.4). It is a discovery fact, not a gate: UI and route gates keep reading the flag snapshot and
IFeatureDecisions, and availability reads the same decisions, so there is no second evaluation path.
| Route | Admission |
|---|---|
GET /api/features/availability |
Signed in. Features whose permission is a Project* policy are omitted without being evaluated (D25 option A) |
GET /api/projects/{projectId}/features/availability |
ProjectViewPolicy, before the action runs (switch on), so an absent or deleted project answers exactly as every other ProjectView route (404, or 410 project-deleted for an active member) |
Both answer Cache-Control: private, no-store. The server, per request:
- Takes the registry's features that are not
Withdrawnand declare apermission. - Permission first. Authorises each distinct policy once with
IAuthorizationService(the existing handlers and gates;SignedInpasses for a signed-in caller). A feature the caller may not use is omitted, never reported with a reason, so the body is byte-identical to one from a registry without it. If a check finds the response already started (a deletion racing the request), the handler's answer stands. - Makes one
IFeatureDecisions.DecideAllover the union of the remaining features' flag keys. - Reads the released records of exactly those features (
RuntimeReleaseStateReader), only when aGlobalfeature that is notSettledis among them (Developing/Pilotinganswers depend on the record, and aReleasedfeature'sreleaseandannouncementfeed What's New and "New to you"). Legacy mode has no binding, so nothing counts as released there. Per feature the read takes the latest record in the captured authority binding (environment and binding epoch) with a confirmed stage milestone, or an importedActiverecord (Unknown precision included). GeneralRelease beats LimitedRelease, then the later confirmation. It is one bounded query (at most 1000 rows, projection only, majority read), cached per binding for 10 s, so a process issues at most one release query per 10 s. A record is announced only when it was applied with announcement modeDefaultand queued at least one kind (6G.2: a Default-mode release of a feature that is quiet with no notice announces nothing). In the same cached read, when an announced record is among them, one more query reads their outbox rows by_id $in(the deduplication keys, at most two per record; projectionstateandresult.testSink), so a process issues at most one outbox query per 10 s too. A failed outbox query keeps the released records, reports nothing as done or delivered, logs a Warning and makes the responseDegraded. Any failure of that query other than a cancelled caller (a Mongo error, a malformed record, an unexpected exception type) returns the reader'sFailure, and the failure is cached for the same 10 s, so a bad read never becomes a 500. The failure is logged once per period at Warning with the exception and its type. A response whose permitted features are allSettledor notGlobaldoes not call the reader, so it isCurrentwhether or not the read would have failed. - Evaluates, first matching row wins:
| Condition | availability |
reason |
|---|---|---|
Grain not Global (until 6F) |
unknown |
policyUnavailable |
Settled |
available |
null |
A decision is Stale |
unknown |
staleAuthority |
| Any other decision error | unknown |
policyUnavailable |
Capability false and a key Stopped |
unavailable |
stopped |
| Capability false | unavailable |
notReleased |
Capability true, Released |
available |
null |
Capability true, Developing/Piloting, released record |
available |
null |
Capability true, Developing/Piloting, no released record |
unavailable |
notReleased |
Capability true, Developing/Piloting, release read failed |
unknown |
policyUnavailable |
The response carries contextId (sha256: over the environment and binding, user, project and every field except
checkedAtUtc, including rehearsal, delivered and each whatsNew entry; an equality token, never recomputed by the
browser), policyVersion, rehearsal (retired, always false, below), freshness (checkedAtUtc, and state Current or Degraded
when a reported decision, the release read, the outbox read, the discovery state read or a needed bundle read failed)
and features ordered by featureId: featureId, availability, reason, release (stage, releasedAtUtc,
precision; null without a released record), announcement ({releaseId}:{stage}, revision 1, delivered, and
seen and dismissed from the caller's own discovery state below; null for a quiet, imported or unannounced release)
and whatsNew (below). It never carries a flag key, store epoch, digest or targeting (RT-60), pinned by the
FeatureAvailabilityResponse golden, FeatureAvailabilityContractTests and the endpoint catalogue's response-field
pin.
Switching. Behind RuntimeFeatureReleases:Enabled (default false; false in every shared environment): off, both
routes answer 404 { "code": "feature_availability_disabled" } and read nothing. The answer comes from
RuntimeFeatureReleasesSwitchMiddleware, which Program.cs runs after authentication and before UseAuthorization
(it reads the [RuntimeFeatureReleasesEnabled] endpoint metadata; ProgramMiddlewareOrderTests pins the order). So on
the project route ProjectViewPolicy never runs and the project is never loaded: a non-member, a member, an
administrator and an anonymous caller get the same disabled body. Two cases differ:
- A subject that subject admission refuses (with
applicationSuspensionon) still gets that refusal, because admission runs first. - The 401 challenge comes from authorization, which has not run. With the switch on, anonymous gets 401.
The attribute's resource filter is the backstop where the middleware is absent. It is read-only and stateless, so on,
off, on gives the same answers with any data (FeatureAvailabilityHttpTests). Flag decision: it shares the release
switch rather than adding a flag, because availability exists only to serve release discovery. Enabling it in an
environment also needs RuntimeFeatureFlags:Mode=Shared and a bound environment; production stays Legacy and off.
Seen and dismissed announcements¶
6E.3 remembers, per person, that they have seen or dismissed one announcement (OD-14), so 6G's "New to you" and 6G.4's walkthroughs do not show it again:
PUT /api/account/announcements/{announcementId}/state
{ "revision": 1, "seen": true, "dismissed": true }
- Validation is by format only:
{releaseId}:{LimitedRelease|GeneralRelease}(theannouncement.idavailability reports),revisionat least 1, and at least one ofseenanddismissedtrue. Anything else is 400, the same 400 whether or not the announcement exists. - It writes only when the announcement is one availability reports now: the latest released, non-quiet,
non-imported record of a feature in this authority binding (read from the same cached
RuntimeReleaseStateReaderread, no second evaluation path). An unknown, quiet, imported, superseded or foreign-binding announcement, and Legacy mode, write nothing. Either way the answer is 204, so the route never says whether a release exists. - A
revisionahead of the announcement's current one also gets 204 and writes nothing, so a client cannot record a future revision to hide its own later re-notification. - No permission or project check. The row is the caller's own preference, read back only through availability, which omits every feature the caller may not use before it joins the rows. A dismissal of a feature the caller cannot use is stored and never returned, and their availability body does not change.
- Storage:
runtimeFeatureDiscoveryState, one document per person and announcement,_id = "{userId:D}:{announcementId}"(MongoDB reference), written by one upsert by_idwith MongoDB$$NOWtimestamps. The same revision keeps the firstseenAtUtcanddismissedAtUtc(retries and races leave one row); a higher revision replaces both (re-notification); a lower one is ignored. There is no un-dismiss. No secondary index, so no migration. - Join: availability reads the caller's rows by
_id($inover exactly the announcements in its response).seenanddismissedare true only for a row stored at a revision at least the announcement's. A failed read reports both false andfreshness.state: Degraded. - Impersonation: the PUT is a change, so read-only support impersonation refuses it; under an edit context it
writes the target's row, never the actor's (
SupportImpersonationContractTests). - Switching: the same
RuntimeFeatureReleases:Enabledswitch. Off, the PUT answers 404feature_availability_disabledbefore reading the body; rows are kept and ignored, and apply again when it is on. - Account deletion keeps these rows today, like every other per-user store (D27, Chris chose A on 2026-10-23); erasure belongs to the OD-6 subject-deletion work.
Tests: FeatureAvailabilityHttpTests (RT-41 API half, no leak and no junk with row counts, the switch round trip),
RuntimeFeatureDiscoveryStoreTests (replica set: shape, races, revisions), FeatureAvailabilityEvaluatorTests (the
join) and the impersonation suites.
What's New and in-app notices¶
6G.2 discloses runtime What's New text through availability, the one disclosure gate: there is no second personalised endpoint. Each feature entry may carry:
"announcement": { "id": "<releaseId>:LimitedRelease", "revision": 1, "delivered": true, "seen": false, "dismissed": false },
"whatsNew": { "id": "project-membership-disable", "revision": "sha256:…", "title": "…", "body": "…Markdown…",
"topicId": "project-membership-status", "guidePath": "projects/membership/members-groups.html#disabling-and-re-enabling-a-membership" }
whatsNewis non-null only when all of these hold: the feature isavailableto the caller (so permission came first, and a stopped feature isunavailable); its release is announced withMaterialiseWhatsNewamong its queued kinds; that outbox row isDone; and the current verified bundle has a runtime entry covering the release's stage. A confirmed release whose row is stillPending(or deferred) shows nothing. A stop hides the entry at once; a restore shows it again with the originalrelease.releasedAtUtc. A missing or unverifiable bundle giveswhatsNew: nullandDegraded; a verified bundle without an entry for the stage givesnulland staysCurrent.revisionissha256:and the entry's hash from the current bundle; no route takes a revision, and deploying a new bundle changesrevisionandcontextIdon the next read. The body is Markdown, which the browser renders only throughMarkdownPipeinuntrustedmode (6G PR D).announcement.delivered("New to you") is true only when the release'sNotifyInApprow isDonewithresult.testSink: falseand the feature'snotifyisreleaseAudience(anadministratorsnotice belongs to the admin Releases screen, 6G.11). Since D60 A (26 Oct 2026) the in-app notice handler completes the row the same way in production, staging, previews and local stacks, so "New to you" works in all of them. A cancelled notice is neverDone, so after a stop and restore it stays false (D14). A staging row the retired test sink completed before D60 keepsresult.testSink: trueand stays undelivered.rehearsalis retired and alwaysfalse. Under OD-19 it was true outside production so the screens labelled runtime entries "Rehearsal"; D60 A (26 Oct 2026) made every environment deliver announcements as production does, so no environment is a rehearsal. The field stays so the response shape, production's bytes and production'scontextIdare unchanged (the hash input still carries"rehearsal":false); the browser no longer reads it, and removing it is a later clean-up. (Staging and preview context IDs change once, which only makes a browser re-read.)- Bundle cache. Availability reads the bundle through
RuntimeReleaseGuidanceBundleCache, keyed by path, length and last-write time and re-checked at most every 10 s; the file is read and hashed again only when one of them changed, so no request hashes the file. It is read only when some entry passes the tests before the bundle (so a broken bundle never degrades a response that could not show What's New). Bundle files are content-addressed, so a new bundle normally has a new path; an in-place rewrite that keeps the length and last-write time is not noticed. - Cost. One release query, one outbox query and at most one bundle stat per 10 s per process, however many
requests (
RuntimeReleaseStateReaderTestsandRuntimeReleaseGuidanceBundleTests, 50 concurrent callers).
Flag decision: not separately flagged. Reads sit behind RuntimeFeatureReleases:Enabled (404 and no reads when off)
and the handlers behind RuntimeFeatureReleases:OutboxDispatch:Enabled; both are default false everywhere. No
migration, no new collection (the reader only reads the outbox by _id), nothing at start-up. On, off, on gives the same
bodies with any data. Tests: FeatureAvailabilityHttpTests (a reviewer, a non-member and the no-project route get
bodies byte-identical to a registry without the feature, with the project administrator as the positive control; the
outbox, stop and restore sequence; the switch round trip), FeatureAvailabilityEvaluatorTests,
RuntimeReleaseStateReaderTests and RuntimeReleaseOutboxDispatchTests (production, staging, previews and local stacks
with the registered handlers, including bindings that still store the legacy testSink or disabled value).
What's New in the app (6E.4 and 6G.3)¶
The browser reads availability through one service, FeatureAvailabilityService
(src/services/web/src/app/core/services/feature-availability.service.ts, spec 6E §6E.4). Every discovery surface reads
only this service; none fetches availability, flags or release records itself, and a review of a new discovery surface
checks that.
- One answer per context. It holds the answer for the current context: the signed-in (or impersonated) user, and the
project in the URL (
/projects/{projectId}/…), if any. Inside a project it calls the project route; elsewhere it calls the no-project route. Signed out, it fetches nothing. - When it fetches. On a context change (sign-in, sign-out, impersonation, entering or leaving a project) and after each
runtime snapshot that
RuntimeFeatureFlagsServiceaccepts. That service already refetches on its slow fallback timer, a SignalR revision hint, reconnection and window focus or online, so availability adds no timer or listener of its own: one accepted revision hint gives one availability fetch. - Context safety. A context change clears the answer at once; a response for a context that is no longer current is dropped, so one project's or one person's entries never appear in another's.
- Failures. Only the switch-off 404 is an empty answer: the body
{ code: "feature_availability_disabled" }, or a 404 with an empty body (an API build without the route). Any other error keeps the last answer for the same context. That includes another status, any other body and the project route's absent-project 404, whose body is not JSON. A new context has no answer yet, so surfaces render nothing. - Seen and dismissed.
setAnnouncementState(id, revision, seen, dismissed)callsPUT api/account/announcements/{id}/stateand updates the local answer at once; if the write fails, it fetches again so the screen shows the server's state.
Two surfaces show runtime What's New entries, with the same rules: the feature is available here, the response carries
its whatsNew entry, and this person has not dismissed it.
- Home page What's New (no-project context). Runtime entries merge with the static feed in date order (the static
isoDateagainst the entry's Europe/London date) into one list of changes grouped by month: the two newest months open, older months collapsed. A runtime entry renders in the same format as a static change. Because of D25 A, features that need a project permission never appear here. - "New in this project" on the project overview (project context). Pending D36; built to recommendation A. A
compact card that renders nothing without an entry. It is one self-contained component
(
project/project-overview/project-whats-new-card/), so it can move with the owner's answer.
Each entry (shared/whats-new/whats-new-entry.component.ts) shows:
- The date. The confirmed milestone in Europe/London, in the static feed's style ("19th August 2026"). It is
formatted with
Intl.DateTimeFormat('en-GB', { timeZone: 'Europe/London' })and never depends on the reader's machine zone (RT-29):2026-08-18T23:30:00Zis "19th August 2026" (BST) and2026-12-01T23:30:00Zis "1st December 2026" (GMT). - "New to you" when
announcement.deliveredis true andseenis false, in every environment (D60 A). There is no "Rehearsal" label any more (OD-19's label, superseded by D60 A on 26 Oct 2026). - The body. It is rendered only through
MarkdownPipeinuntrustedmode, so no image, raw HTML or remote resource reaches the page. - Actions. "Read the guide" is a link (
guidePaththroughUserGuideUrlPipe). "Hide" is a button that recordsseenanddismissedfor this person only. Opening a row, following the guide link or following a link in the body recordsseen.
With releases switched off, or no entry, both surfaces render exactly what they rendered before. A component spec pins
the home page DOM against the DOM captured from main before this change.
Flag decision: not separately flagged. The server switch makes availability answer 404, so the browser shows only the
static feed and no card. No migration, nothing at start-up. Tests: feature-availability.service.spec.ts (6E.4-1,
6E.4-2), shared/whats-new/**/*.spec.ts (RT-29 under three machine zones with a positive control, the entry rules,
untrusted rendering, the actions), whats-new.component.spec.ts (the static-only DOM, merge order, no "Rehearsal", RT-41
rendering) and project-whats-new-card.component.spec.ts. On the runtime-fleet lane,
e2e/tests/runtime-fleet-browser-whats-new.spec.ts (bash e2e/run-local.sh --runtime-fleet --spec
runtime-fleet-browser-whats-new) runs the MVP boundary end to end: release switches on, the local binding left as
stored (it delivers the notice as production does, D60 A), Apply and confirmation, then the card for a project administrator, nothing for a
reviewer, a non-member or the home page, Hide for one person, a later joiner, and a stop and restore. It needs a
releasable registry entry, so on main it fails at the Apply preview with WorkflowCompatibilityMissing, like the
release-lifecycle scenario.
Public What's New for signed-out readers (6G.6)¶
GET /api/whats-new/public (anonymous, WhatsNewController, PublicWhatsNewProjection) answers the runtime What's New
entries that general availability has made public, the same for every caller:
{ "rehearsal": false,
"entries": [ { "id": "project-membership-disable", "revision": "sha256:…", "title": "…", "body": "…Markdown…",
"topicId": "project-membership-status", "guidePath": "projects/membership/members-groups.html#…",
"announcementId": "<releaseId>:GeneralRelease", "releasedAtUtc": "2026-11-12T23:30:00Z", "precision": "Instant",
"audience": "projectAdministrators" } ] }
audience is the feature's registry audience in camelCase (signedIn, projectMembers, projectAdministrators,
applicationAdministrators; null when the definition has none), and nothing else about the registry. It is presentation
only (owner decision D57 = B, 26 Oct 2026): a feature that is fully released with a public guide page is still announced
publicly, and an entry whose audience is projectAdministrators says so. The browser shows a "For project administrators"
chip beside the date on the home page What's New, for both the public entry and the signed-in copy of the same
announcement (the home page labels the signed-in copy from the public answer, because availability carries no audience).
There is no label for the other audiences, and none on the project "New in this project" card, which only project
administrators see. The field never decides access. It rides on the existing RuntimeFeatureReleases:Enabled switch, so
there is no new flag.
An entry appears only for a Global feature (not Settled or Withdrawn) whose latest released record in the
captured binding is a confirmed, announced GeneralRelease with its MaterialiseWhatsNew row Done, whose capability
holds now (one DecideAll over exactly those features' keys, through the availability truth table, so a stop or a
stale authority hides it and a restore brings it back with the same date), and whose current bundle entry covers
GeneralRelease with a public guide page (guidePath not null). It is dated by the GeneralRelease milestone. A
LimitedRelease-only, stopped or unreleased feature never appears, and the body has no clock, policy revision, flag key,
environment or binding, so it is byte-identical to one from a registry without those features. Any failed read gives
entries: [], never a guess: that includes a failed binding capture or a failed DecideAll, so the route never answers
500 (cancellation of the request itself still propagates). This is published content, not an anonymous availability view (D26 A).
rehearsal is retired and always false (D60 A, 26 Oct 2026): every environment delivers announcements as production
does, so the answer no longer tells an anonymous caller anything about the environment. The field stays so the shape and
production's bytes are unchanged.
With RuntimeFeatureReleases:Enabled off it answers 404 {"code":"feature_availability_disabled"} and reads nothing, as
the other runtime release routes do. On, it answers Cache-Control: public, max-age=60, so after a stop a shared or
browser cache may show the old entry for up to a minute (plus the 10 s release read).
In the browser, PublicWhatsNewService fetches it once and after each accepted runtime snapshot (a 404 is an empty
answer; any other error keeps the last answer). The home page What's New merges, in date order:
- signed out: the static feed and the public entries;
- signed in: the static feed, the personalised entries and the public entries, without any public entry whose
announcementIdthe no-project availability answer reports (whatever its availability or dismissal). So an entry the person can use shows once, with its own "Hide" and "New to you"; an entry they hid stays hidden; a feature the answer reports stopped is not shown from a cached public body. Public entries are held back for a signed-in reader until their availability answer has loaded, so a hidden entry never flashes; if availability fails, only the static feed shows (the conservative choice, since the dismissal cannot be checked).
Public entries carry no announcement state, so they have no "Hide" and no "New to you".
Flag decision: behind RuntimeFeatureReleases:Enabled (default false everywhere; the #4533 fixes use the same switch and add no flag); read-only, no migration, nothing at
start-up. Tests: FeatureAvailabilityHttpTests.PublicWhatsNew.cs (RT-68 over HTTP: general release publishes dated that
day, a limited release never does, stop and restore, the no-leak byte comparison with its positive control, one
DecideAll, the switch), EndpointAuthorizationCatalogTests (anonymous classification, pinned response fields),
public-whats-new.service.spec.ts, runtime-whats-new.spec.ts and whats-new.component.spec.ts (signed-out public
only, deduplication, RT-68, the switched-off DOM equal to the pre-6G DOM).
Shared administrator preview and commit¶
Current page layout: the review panel, retained-request panel and saved receipt described here are now presented in the change dialog, the outcome banner and Details; see Using the Feature flags admin page. The preview, digest, commit and retry rules below are unchanged.
PR #4381 connects individual controls on the existing Feature Flags page to the Shared policy API when its verified snapshot carries authority metadata. Legacy snapshots retain the existing mutation flow. Shared group controls remain unavailable because there is no shared group transaction endpoint.
In Shared mode the snapshot's mutationAllowed is false: it describes only the legacy PUT, which the
server refuses there. Each flag's runtimeManageable and managementReason describe the Shared preview and
commit routes instead, using the same deployment constraints as the legacy store (for example the statistics
maintenance gates). In production only eligible keys are manageable; the others read "This flag is
deployment-controlled in production." The page offers On, Off and Default when the snapshot carries
schema-1 authority and the flag is manageable, and shows the reason otherwise. The authority envelope also
lists stoppedKeys: the ordinal-sorted keys whose emergency stop is in force at that policy (production lists
only eligible keys). The field is additive within schema version 1; the browser's lineage guard accepts
envelopes without it and refuses one whose stoppedKeys is not a list of key strings.
A change first requests a server preview. The page shows exact before/proposed revisions, effective values, the proposed override (including Deployed default), induced changes and fleet readiness. Confirm and save commits that reviewed request with its preview digest and one mutation ID. Cancel before commit writes nothing. A dependency refusal, at preview or at commit, returns to the existing explicit dependency-confirmation flow; it never retries or commits automatically, and a new preview and confirmation are required.
A successful receipt is shown as Policy saved, without claiming fleet adoption. After a lost response, keep the page open: controls remain locked to the original request, and Check outcome reads its receipt. An absent receipt means Unknown, not aborted. Retry unchanged request preserves every field and mutation ID. While the outcome is unknown, the page shows the retained request (flag, current and proposed values, override and revisions) so you can see what a retry would write; there is no Confirm and save button. A refusal of that retry does not prove that the earlier attempt failed. The mutation ID stays visible for diagnosis. Do not start a new request to resolve an unknown outcome. If the page is reloaded or closed and reopened in the same tab, Reload recovery (in Using the Feature flags admin page) picks the request up again by its mutation ID.
The server still owns permissions, environment binding, eligibility, dependencies, full-identity checks and fleet admission. A known initial refusal requires a fresh snapshot and another reviewed preview; changes never fall back to legacy PUT in Shared mode. The existing Shared mode is the opt-in boundary; this slice adds no environment configuration, migration, production activation or automatic page reload. Emergency-stop controls, full fleet-status UI and the hermetic multi-replica freshness exercise remain follow-ups before the complete B/C operator workflow is claimed.
Audience-separated content artifacts¶
PR #4382 adds an offline content compiler. Since 6G.5 E1 it has no manifest: it derives its entries from the
syrf.topics front matter of user-guide/ and docs/content/protected/ pages (see
Guide topics). It validates
feature/topic references against the versioned feature definitions, explicit publication classes, workflow
references and asset audiences. It produces separate server and public JSON artifacts with SHA-256 content
identities. Source ordering does not affect those identities; article, asset or reader-class edits change
the article revision. Private-only edits leave public bytes and the public digest unchanged.
Output directories must be separate trees outside the source root. Server artifacts use owner-only file permissions and are refused in the repository's public guide/web asset trees, including symlink aliases. Keep server bundles private: they contain full restricted Markdown, audience IDs and assets (the API image carries one, 6G.7, so treat the image as private too). Only copy public artifacts into a future public renderer. The compiler does not publish, render or serve either artifact; API reader authorization and old-revision access checks are still required before protected content is delivered. Markdown rendering/sanitization and URL resolution belong to that renderer.
Every declaration names featureId, stable topicId and workflowVersions, plus an optional section: a heading
anchor on the page (as kramdown generates it), refused when the page has no such heading. publication defaults to
PublicReleased in user-guide/ and is required in docs/content/protected/. The entry's variantId (the page path),
title (the section heading or page title) and Markdown (the page without front matter) are derived. No page declares
assets yet, so assets and assetIds are empty. Supported classes are Draft, PrivatePilot, PublicPreview, PublicReleased and Retired. Draft/private entries
require an explicit documentation audienceId. Retired entries require a reviewed historicalPublication
class and retain its audience; retirement never makes a private pilot public automatically.
Publication evidence (6G.1, #4382). A declaration alone is not proof, so the compiler checks the evidence it can
see. A public class (PublicPreview, PublicReleased) is accepted only for a source inside user-guide/, which the
public site publishes (Public content must live in the public user guide ...); a restricted class is accepted only
outside it. A restricted entry's audienceId must equal its featureId, meaning "the people the feature is available
to", until 6F defines audiences. A Retired entry or asset whose historicalPublication is public is refused, because
nothing records which revisions were published (D40); Retired with PrivatePilot is still accepted. D40 was answered
B on 2026-10-24: environments will record which public revisions their guide site served (6G.P, below), but the
refusal stays until 6H consumes that record.
Server bundle schema 2 (6G.1). The server bundle is schemaVersion: 2: the schema 1 fields plus whatsNew, the
runtime What's New entries from docs/content/whats-new.yaml (runtime:), each
{ id, featureId, stages, title, body, topicId, guidePath, revision }. guidePath is the topic's public article page
and section (for example projects/membership/members-groups.html#disabling-and-re-enabling-a-membership), or null while
the topic has no public article; revision is the SHA-256 of the entry. The public bundle stays schemaVersion: 1 and
never carries whatsNew. The CLI reads docs/content/whats-new.yaml.
The API's reader accepts schema 1 and schema 2, each with its exact field list.
Publication manifest (6G.P P1, D40 B). Beside the public bundle the compiler writes publication.json:
{ schemaVersion: 1, entries: [{ featureId, topicId, variantId, publication, revision, path, anchor }], digest }. It
lists the public entries only (PublicPreview, PublicReleased), in bundle order, each with the same revision as its
bundle entry, the guide page it is served at (path, the page with .md replaced by .html) and its section (anchor,
or null). It carries no title or Markdown, and nothing new becomes public: the web bundle's help-topics.generated.ts
already ships every public topic's feature, page and section. digest is the compiler's usual SHA-256 of the payload
without digest, so the C# RuntimeReleaseGuidanceBundle.PayloadDigest verifies it (pinned by a golden digest in
compile-release-content.test.ts and RuntimeHelpPublicationManifestTests). The user-guide image serves it at
/syrf-publication.json: the reusable image build (_docker-build.yml, user-guide job only) runs Compile the
publication manifest (both bundles to job-private temp; only publication.json is staged, read-only, as
.help-publication/syrf-publication.json), the generated user-guide/Dockerfile copies it into the nginx stage outside
Jekyll, and Check the publication manifest in the guide image requires exactly one
/usr/share/nginx/html/syrf-publication.json, byte-identical to the compiled file and read-only, before anything is
pushed (pinned by .github/scripts/test-user-guide-publication-image-contract.sh; the existing guidance check still
refuses any server-*.json there). The compiler, guide-topics.ts, generate-feature-definitions.ts and
feature-definitions.yaml are user-guide change paths, so a change to them rebuilds the guide image. To build the image
locally, stage the manifest first (user-guide/README.md, "Building the production image locally").
"Served" will mean what the API's publication recorder (6G.P P2) observes: an anonymous GET of this manifest from the
environment's own guide origin with a valid digest, then each listed page answering 200 without a redirect and
containing its section anchor. Until P2 is merged and switched on, nothing reads the manifest. Flag decision: not
flagged, because the file is static, public-only and read by nothing until P2, which sits behind
RuntimeFeatureReleases:Enabled. Known gap: path ignores permalink: (eight guide pages use one, in whats-new/ and
data-export/; none declares a topic today), so a page with a permalink would fail P2's page check and go unrecorded rather than be
recorded wrongly.
Public output contains only explicitly public articles, their public search metadata and the public assets
they reference. Restricted articles/assets, audience IDs and unreferenced assets cannot leak through this
projection. Public articles cannot reference restricted assets; restricted assets must match the article's
audience. Restricted sources themselves are refused under user-guide/ and web public assets: filtering
one bundle cannot protect files already exposed by another public build. Keep future private sources in a
separate non-public source tree. Relative paths, symlink containment, file/bundle sizes and UTF-8 are checked.
The initial asset formats are PNG/JPEG/WebP; active-content formats require a later reviewed renderer.
The checked-in topic is declared in the front matter of the already-public membership guide
(user-guide/projects/membership/members-groups.md, section: disabling-and-re-enabling-a-membership). PublicReleased
describes that existing article, not the Developing feature's lifecycle. Its empty workflow list asserts no reviewed
workflow compatibility. No release date, guidance readiness, activation, announcement or new publication
is inferred from compilation. The compiler consumes no feature-flag values. Flag decision: no runtime
switch is needed for this explicit offline tool, which is not registered in an application or publishing
pipeline. Required guidance remains Unverified in release preparation; release Apply verifies the compiled server
bundle (see Applying a release). Protected delivery to readers is still a later integration (6G).
Help publication record (6G.P)¶
The API records which public help topic revisions this environment's own public guide site has served, so that a
later public archive of retired pages (6H) can rest on evidence rather than on a typed historicalPublication line
(D40 B, 2026-10-24). Nothing reads the record yet except tests and evidence runs; there is no route and no UI.
What "served" means. The API's anonymous HTTPS GET of its guide origin returned the publication manifest that the
guide image carries at /syrf-publication.json (built by the content compiler, 6G.P P1) with a valid digest and, for each
manifest entry, the page itself: 200, no redirect, a text/html content type, a body containing <html (any case, so a
soft 404 that answers 200 with plain text or a bare message is not a guide page) and id="{anchor}" when the entry has a
section. Bodies are decoded as UTF-8 (the guide is Jekyll output). The
guide site is the only thing that serves public help pages and is promoted separately from the API, so the API's own
bundle is never used as the source. One row per environment and topic variant revision, in runtimeHelpPublication
(see MongoDB reference): the
verified binding's environmentId, bindingEpoch and clusterFingerprint, the entry, the origin, first and last
observation times (MongoDB time), the SHA-256 of the HTML last served and lastGuideOrigin, the origin that served the
latest observation (guideOrigin stays the first origin; rows written before lastGuideOrigin existed lack it and still
read, with a null value).
Settings (API, read on every pass):
| Setting | Meaning |
|---|---|
RuntimeFeatureReleases:Enabled |
The switch (default false everywhere). Off at start-up, the recorder is not registered at all; switched off later, a pass reads, fetches and writes nothing. Rows stay |
RuntimeFeatureReleases:PublicationRecord:GuideOrigin |
The environment's public guide origin, scheme and host only, for example https://help.staging.syrf.org.uk. Unset or anything but an origin: the recorder idles (NotConfigured) |
RuntimeFeatureReleases:PublicationRecord:IntervalMinutes |
Minutes between passes, default 60, minimum 1, maximum 1440, each ± 10 %. The first pass waits a jittered 1-5 minutes (never during start-up) |
Environment variables use the usual SYRF__ prefix and double underscores, for example
SYRF__RuntimeFeatureReleases__PublicationRecord__GuideOrigin. Setting the origin per environment in cluster-gitops is an
operational step that has not been taken: production https://help.syrf.org.uk, staging
https://help.staging.syrf.org.uk, a preview or local stack its own guide service.
Guards.
- Registered only on the Shared-mode API writer (the reconciler's guard) with the switch on. Legacy mode, production in Legacy policy mode and every other host register nothing.
- Each pass verifies the environment binding (one read plus
hello) before any HTTP request; an unverified binding writes nothing (BindingUnverified). - Origin guard. A production binding records only
https://help.syrf.org.uk(the code constantRuntimeHelpPublicationEvidence.ProductionGuideOrigin); every other binding refuses that origin;httpis allowed only under alocal:orpreview:binding. So a staging API pointed at the production site, or production pointed elsewhere, records nothing (OriginRefused). - Whole-manifest refusal. Exact field lists,
schemaVersion: 1, a recomputed digest, at most 200 entries, every entryPublicPrevieworPublicReleased, every path a relative.htmlpath without.., scheme or query, and every anchor null or a kramdown ID. Any failure refuses the whole manifest (ManifestRefused), so nothing from it is recorded. - Bounded. Per replica and pass: one binding check, one manifest
GET, at most 200 pageGETs (each distinct page once), one unordered bulk upsert. The named client sends no cookies or credentials and never follows a redirect; each request has 10 s, the manifest 1 MiB and a page 4 MiB, and the pass 60 s (entries left when it ends are not recorded,BudgetExceeded). Replicas upsert the same rows idempotently. - Fail-soft. Every failure writes nothing for that pass or entry. Each reason is logged with its own detail when it first appears, again
at most once every 24 hours while it persists (so a recorder that fails every pass is not silent), and when it clears:
Disabled,NotConfigured,OriginRefused,BindingUnverified,ManifestMissing(an older guide image has no manifest),ManifestRefused,PageNotServed,BudgetExceeded,StoreUnavailable. The recorder never stops the host and adds nothing to readiness or health.
Production evidence. RuntimeHelpPublicationEvidence.IsProductionEvidence(row, binding) is true only for a
production row read under the reading host's verified production binding with the same epoch and cluster, the
production origin and a public class. Staging, preview and local rows never satisfy it, nor a local stack bound as
production against a local origin, and production rows copied into another database by a reseed fail there.
RuntimeHelpPublicationStore.FindAsync(environmentId, topicId, variantId) reads one environment's rows for a topic
variant by an anchored _id range; together they are the contract 6H will consume. One read returns at most 1000 rows and
the result carries a Truncated flag that is true when the cap was hit. The cap takes rows by _id (which ends in a
revision hash, not an age), so beyond 1000 rows both the subset and its order by first observation are approximate: treat a
truncated result as incomplete.
Reading the rows. On a local or Bramble stack (read-only):
db.runtimeHelpPublication.find({ _id: { $gte: "local:<stack>:project-membership-status:", $lt: "local:<stack>:project-membership-status;" } })
Flag decision. Behind the existing RuntimeFeatureReleases:Enabled (spec 6G.P, Q-P1 recommendation A, pending
Chris): the recorded topics exist for runtime releases and the record needs the same verified binding, so no new switch.
The switch is read through RuntimeHelpPublicationRecorder.SwitchedOn alone, so a dedicated switch (Q-P1 option B)
would change one method and the registration. GuideOrigin and IntervalMinutes are configuration, not switches.
Off means no reads, no HTTP and no writes; rows stay, and on again resumes the upserts, so round trips are safe with data
from either state. No migration. Production records nothing until it is bound, runs Shared mode and has the switch on.
Inspecting fleet adoption¶
Current page layout: there is no per-flag Inspect fleet button any more. The summary line reads the membership control's fleet (the only key with tracked adoption, which reports the whole policy), and the host table is in Details; see Using the Feature flags admin page.
PR #4383 adds Inspect fleet beside each flag when the administrator catalog carries Shared authority. It calls the existing inspect-capability endpoint and shows its exact policy revision, adoption state, expected replica counts, host states, reasons and last-seen times. A saved mutation is not evidence that hosts adopted it. Domain readiness remains unknown for controls the server cannot yet assess.
The response must match the selected flag and every field of the catalog policy identity. Refresh the catalog if a newer policy was committed. Selecting another flag or receiving a new catalog identity cancels the previous lookup. Checking again clears the previous observation immediately; failed or forbidden reads show Unknown. Observations expire after 15 seconds and require Check fleet to obtain fresh evidence. This is an on-demand inspection view, not a continuous convergence monitor. It changes no policy, enrollment, configuration or data. The existing Shared-mode opt-in is the flag boundary; Legacy has no fleet action. Emergency-stop controls, automatic convergence monitoring and the multi-replica exercise remain follow-ups.
Administrator emergency stop¶
Current page layout: Emergency stop… in the summary card opens a dialog that needs a reason; Stop prepares and sends the stop in one confirmed step. Check and unchanged retry are unchanged; see Using the Feature flags admin page.
PR #4384 exposes a separate Prepare emergency stop action for the audited global disableMembership
control in Shared mode. It fetches the current catalog identity within the same environment/binding/store
lineage, shows its exact revision and audit reason, and requires Confirm emergency stop before invoking
the dedicated stop endpoint. The audit reason uses the membership control's existing comment, or an explicit
administrator-page default. The server independently checks the emergency-stop capability and safe-state
contract. Stop forces the control's audited safe value (Off) without discarding saved overrides.
The stop flow is independent of ordinary preview/save state: an ordinary mutation with an unknown outcome does not disable it. An ambiguous stop keeps one immutable request and mutation ID. Check stop outcome reads that receipt; Retry unchanged stop sends the same request. Missing receipts and later refusals do not prove an earlier attempt aborted. After a reload or navigation in the same tab, Reload recovery (in Using the Feature flags admin page) looks the stop up again by its mutation ID and never sends it again. An initial known conflict/refusal can be prepared afresh.
Emergency stop saved confirms only the durable receipt and names the committed revision. Confirming reloads the catalog even if an ordinary mutation is unresolved, without replacing that original mutation. The API's policy reader can lag a commit by a few seconds; until the page shows the saved revision the receipt says so and offers Refresh. The receipt goes once dismissed or once a refresh reaches its revision. Inspect fleet adoption separately.
The section shows the current stop state (Stopped or Not stopped) after every catalog load,
including a page reload. It reads authority.stoppedKeys from the snapshot (added in #4381), so no extra
request or capability is needed. While a saved receipt is newer than the snapshot the section shows Stop
state updating instead of a stale answer. Prepare emergency stop is offered only when no stop is
known to be active. A server that omits stoppedKeys leaves the state unknown; the section says so and
keeps the stop available, and the server still refuses a stop that does not apply. The stop does not call release preparation, guidance or announcement services. Releasing a stop is a
separate action: see Restoring a stopped membership control.
Hermetic multi-replica timing evidence remains open.
Flag decision: the existing Shared authority opt-in and server capability are the gates. This adds no
production activation, deployment configuration, migration or audience enrollment.
Restoring a stopped membership control¶
Current page layout: Restore… in the summary card opens a dialog that shows the server preview's restored value and needs a reason; see Using the Feature flags admin page.
PR #4385 adds Preview restore (accessible name "Preview restore of membership control") to the
emergency-stop section of the Shared administrator page. It is offered while the section shows Stopped,
or when the server does not report the stop state; Prepare emergency stop is offered only when the control is not
known to be stopped. It uses the
existing ReleaseStop policy action with no override or dependency edits. The server preview must remove
the selected stop while preserving its saved override. Review the before/after effective values, saved
values and Emergency stop column (for example Stop released), then confirm. A control with no matching stop cannot be confirmed
through this restore flow.
Restoration uses the ordinary policy-change permission, preview digest, exact identity, deployment constraint and production-readiness checks. It does not use the emergency-stop bypass. Preparation and confirmation are blocked while the local emergency-stop request is preparing, confirming, committing, resolving or unknown. The original restore request is retained after ambiguous responses, with the same receipt lookup and unchanged retry as ordinary policy edits. A confirmed restore dismisses an older known stop receipt, refreshes the catalog and shows a Policy saved receipt with the committed revision rather than claiming adoption; inspect fleet adoption separately. The server preserves saved overrides and stop/restore data; the browser changes no migrations or release milestone dates. Existing Shared-mode opt-in remains the flag boundary. Production configuration remains unchanged. Hermetic convergence/stop/restore timing remains a follow-up; browser recovery after a reload is covered by Reload recovery (in Using the Feature flags admin page).