Dynamic Endpoint Configuration

The Octopus backend assembles its public GraphQL schema at runtime from a single document, endpoints.json. Every change to that file is versioned in MongoDB and goes through the Config UI so we can stitch together data coming from Tango, in-process streams, REST services, upstream GraphQL APIs, and a few carefully reviewed fallback helpers.

The loader in endpoint_config.loader turns the JSON description into resolvers and schema definitions:

  • Inline or external GraphQL type definitions are collected first.

  • Variables declared under vars are resolved against environment overrides and injected into upstream URLs and queries.

  • Query, mutation, and stream (subscription) entries become FastAPI/ASGI callables that delegate to REST clients, GraphQL transports, or (rarely) local Python helpers.

  • The resulting SDL is merged with the static widget registry schema before we hand everything to Ariadne.

Expected Document Blocks

Each top-level key in endpoints.json contributes a specific piece to the runtime schema.

types

Inline GraphQL type definitions. Either provide a string containing SDL or an object describing a simple type. These inline types are combined with custom_types.graphql at load time so exports stay portable.

vars

Key/value pairs that the loader exposes through endpoint_config.vars.get_resolved_vars(). Values default to the JSON entry but can be replaced by environment variables at runtime. Call substitute(...) in resolver builders to expand {VAR} placeholders before making upstream calls. Variables may be either string or list[string]. List variables are useful for endpoint groups (for example TANGO_DBS) consumed by frontend widget dropdowns.

queries, mutations, streams

Blocks containing named field specifications. Every entry can set:

  • method: one of GET, POST, PUT, GRAPHQL, or LOCAL (defaults to GET).

  • mapping: optional map of output fields to upstream fields. When present we generate a dedicated FooItem GraphQL type and coerce the response to that shape.

  • args: GraphQL argument declarations ({"device": "String!"} for example).

  • list: mark the field as returning a list. Subscriptions that set list use state-aware list streaming with smart filtering.

  • Transport specific properties (url, endpoint, query, function, root). For GRAPHQL, endpoint may be a single string or an array of strings. When an entry defines a GraphQL arg named endpoint (type String), that arg is treated as a selector for the upstream endpoint and is not forwarded as an upstream GraphQL variable.

Queries and mutations become top-level GraphQL fields; streams are exposed as subscriptions. When multiple services supply data we commonly combine them by calling a GraphQL operation for structure and filling in extra fields via REST requests before the response enters the GraphQL layer.

locals

⚠️ CRITICAL SECURITY WARNING ⚠️ Locals run with the same privileges as the backend. They should never be enabled in production.

Safe usage guidelines

  • Use locals only when no API exists yet.

  • Delete them immediately once an upstream service becomes available.

  • To hard-disable the feature at runtime:

    • Set ENABLE_ENDPOINT_LOCALS=false

    • Helm deployments: vars.ENABLE_ENDPOINT_LOCALS: "false"

When disabled:

  • The loader ignores the locals block.

  • LOCAL endpoints are never registered.

  • Arbitrary Python snippets cannot be stored through the Config UI.

How the /config/test-local/{name} route works

  • The endpoint is guarded the same way as every other mutating Config UI route.

  • In config_editor.py:320 it is registered with strict dependencies:

    • _require_cfg_session

    • _require_csrf

    • _require_same_origin

  • This means requests must:

    • Carry a valid cfgui_session cookie (set after a successful Config UI login).

    • Echo back the matching CSRF token.

  • Because the router is mounted under /config (app_factory.py:360-361), the path exposed to the UI is exactly as documented.

  • Browsers without those cookies/headers are rejected with 401/403.

✅ Practical usage

  • The UI can call it during development, but effectively only from an authenticated Config UI context.

  • If you need to exercise it manually (e.g. with curl or Postman):

    1. Log in at /config-ui first.

    2. Reuse the stored session + CSRF cookies in your request.

Resolver Backends

The spec supports three resolver families so we can aggregate information from multiple systems:

  • REST (GET, POST, PUT): endpoint_config.rest_support.build_rest_resolver expands {VARS} placeholders and formats $variable segments inside the URL using the GraphQL arguments. JSON bodies are sent for write methods. Any non-success status raises and surfaces through the GraphQL error extension.

  • GraphQL (GRAPHQL): endpoint_config.graphql_support.build_gql_resolver opens a client (HTTP or WebSocket) to the declared endpoint, forwards JWTs after re-signing them if required, and unwraps the desired root field. If multiple endpoints are configured and no selector is provided, the first endpoint is used. Callers can select by endpoint: "NAME_OR_URL" or by 1-based index (endpoint: "1", endpoint: "2", …). Empty list arguments short-circuit subscription traffic to avoid needless upstream load. The endpoint field itself can be either:

    • a concrete URL,

    • an inline list ([{A},{B}]),

    • or a variable key whose value is a list (for example endpoint: "TANGO_DBS"). This lets backend deployments define endpoint groups once and reuse them across multiple resolvers/widgets. GraphQL transport is coerced by operation kind, so the same endpoint group can back both HTTP queries/mutations and WS subscriptions. For list subscriptions (list: true), the backend also applies state-aware filtering to reduce duplicate events:

    • mergeList argument is auto-added to GraphQL subscription fields as Boolean = false (delta mode by default).

    • In delta mode (mergeList: false), backend emits only changed items from current frame.

    • In snapshot mode (mergeList: true), backend emits merged full state seen so far.

    • Change detection ignores timestamp; if only timestamp changed, event is dropped.

    • Item identity key is device/attribute.

    • If caller passes an empty list variable (for example fullNames: []), stream emits [] once and stops.

List Subscription Modes (mergeList)

When a stream entry uses method: GRAPHQL and list: true, schema field gets an optional mergeList argument:

subscription TangoAttrs($fullNames: [String!]!, $mergeList: Boolean = false) {
  tangoAttributes(fullNames: $fullNames, mergeList: $mergeList) {
    device
    attribute
    value
    timestamp
  }
}

Use modes based on consumer:

  • mergeList: false (default): low-churn delta stream for widgets/runtime consumers. Backend still keeps internal state to detect real changes, but emits only changed rows.

  • mergeList: true: merged full snapshot stream for inspection/debug workflows (for example GraphiQL comparisons), at the cost of larger payloads.

This design keeps frontend traffic smaller while preserving correctness for high-frequency upstream streams that spam timestamp-only updates.

  • Local helpers (LOCAL): resolved through endpoint_config.locals_runtime.import_any. They may reference code stored in locals or import real Python modules. Treat these as a development escape hatch only; prefer to expose the logic through REST or GraphQL services so it can be audited and rate limited independently of the backend.

All resolver factories accept a should_cancel coroutine so long-running operations respect cooperative cancellation when websockets or HTTP clients drop connections.

Security Around Endpoint Editing

Tight controls protect the configuration flow so only the Config UI can change active endpoints:

  • The FastAPI router in config_editor.py is mounted at /config and only exposes version management operations. There is no GraphQL mutation capable of editing endpoint documents.

  • Every route requires a password-authenticated Config UI session. Sessions are signed cookies (cfgui_session) protected with PBKDF2-derived secrets and a strict lifetime (CFG_UI_TTL_MIN).

  • All mutating operations additionally require both a matching cfgui_csrf cookie and X-CSRF-Token header and must originate from the same host. Missing or mismatched headers are rejected before the payload reaches MongoDB.

  • Login attempts are throttled with in-process counters (CFG_UI_RL_* knobs) scoped to each worker.

  • Incoming payloads go through utils.validation.validate_payload, which checks the JSON structure, ensures Python snippets parse, and prevents malformed variables from being stored.

  • Versions are written to MongoDB through config_store.save_new_version. The store keeps the full history, refuses to delete the active version, and records both the active and last-known-good versions for automatic recovery.

  • After a save or activation the loader invalidates its caches and triggers a graceful reload so the GraphQL schema reflects the new snapshot everywhere.

With these safeguards the only path to editing an endpoint is through the password-protected Config UI, using browser requests that satisfy our session, CSRF, same-origin, and rate-limiting checks. Operations initiated outside the UI lack the cookies/headers and are rejected with 401 or 403 responses.

Frontend Endpoint Sources

For widget configuration UIs, the backend exposes Query.endpointSources with:

  • types: map of widget schema field type -> variable group key (default: TANGO_DB -> TANGO_DBS, TANGO_WS -> TANGO_DBS__WS)

  • vars: map of variable group key -> list of endpoint URLs

Configure the type map with ENDPOINT_TYPE_BINDINGS as JSON or CSV: TANGO_DB=TANGO_DBS,TANGO_WS=TANGO_DBS__WS.

TANGO_DBS__WS is a derived alias generated from TANGO_DBS by converting http(s) URLs to ws(s) and /db to /socket.

Operational Guidance

  • Prefer REST or upstream GraphQL methods for long-term functionality. They make dependencies explicit, keep Python inside the services that own the data, and make load management easier.

  • Keep CONFIG_AUTO_ROLLBACK=true in production so the loader automatically falls back to the last known good version if startup fails.

  • Keep locals empty on production releases. When a temporary helper is needed during development, include a removal task before the change is promoted.

  • Use environment variables to adapt the same config file to multiple deployments: vars defaults let local developers work without secrets while production injects the correct URLs and credentials.

  • Document new endpoint entries inside the Config UI message field so the version history explains why data sources were added or changed.