Skip to content

Deployment: Basics

The Quick Start brought a whole system up with a couple of copy-paste commands. This page explains what was actually underneath that - what an Elastic MDS deployment is made of, and how you build and configure one for your own environment.

Audience: operators standing up a deployment (and architects who want the model). This page covers the deployment model and how you bring up and operate a deployment; production-grade topologies, networking modes, and scaling are covered in Deployment: Advanced.

What a deployment is made of

A deployment is a set of services you run - the same base images and the same deployment-config/, whether you run them conventionally as native host processes, under Docker, or under Kubernetes. Three key concepts:

  • Base images. MetaFluent assembles various combinations of software features into container images and publishes them to the GitHub Container Registry (ghcr.io). A base image is stock and unconfigured - it contains the software but knows nothing about your deployment: your hosts, your data sources, your tuning. It is not meant to be run as-is. The Image Catalog lists the available images and what each is for.
  • Deployment projects. A deployment project (deploy-mf-*) is a Git repository you clone. It holds a default configuration for a base image, along with the scripts to operate it; you adapt that configuration for your needs. This is the unit you operate - the deployment project is how a stock base image becomes a configured, runnable service.
  • Compositions. A composition describes how a set of services is wired together and brought up as a running system. The Docker Compose file behind the Demo entry in the Deployment Cookbook - the one the Quick Start used - is a Docker Compose composition. Compositions are worked templates for the deployment topologies - a starting point you adapt for development, evaluation, and single-host use; production at scale runs the same service catalog on Kubernetes.

Which base images - and therefore which deployment projects - you need depends on the topology you are building, and a topology is driven by a few key decisions: whether it needs to scale, whether it uses host networking, and how resilient it must be and at what granularity. Those decisions, and the production-grade (including Kubernetes) topologies that follow from them, are the subject of Deployment: Advanced.

Anatomy of a deployment project

Clone a deployment project and you get a small, predictable layout:

deploy-mf-<service>/
  deployment-config/
    deployment.properties     <- your primary configuration
    resources/                <- configuration resources; add or adapt files here
    overrides/                <- generated; do not edit
  scripts/
    start                     <- run the configured service
    stop                      <- stop it
    bake                      <- build a self-contained, configured image
    config-merge              <- reconcile your config with an upstream release
  README.md

The file you edit most often is deployment-config/deployment.properties - the settings that make the stock image your deployment. You may also add or adapt files under resources/, for configuration resources your deployment needs. The overrides/ directory is generated by the build - leave it alone; it is rewritten when you take an upstream update.

A base image ships with sensible defaults, and it is a deliberate goal that as much as possible you can run on stock configuration - adapting deployment.properties (and resources/) only where your environment genuinely differs. What the settings mean and how configuration works is the subject of Configuration: Basics; this page is about operating and packaging a deployment, not the configuration model itself.

Setting up a DNS entry for your API Gateway

Everything you do to monitor and control a running system goes through its API gateway, addressed throughout the documentation - and by built-in tools such as sim-ctl - by the convention name mf-api-gateway on port 9090. The Quick Start mapped that name to 127.0.0.1 in /etc/hosts because everything ran on one machine. A real deployment needs the name to resolve from wherever your clients and operators are.

The gateway listens on port 9090 on its host. Make mf-api-gateway resolve to that host's address in one of two ways:

  • A DNS A record (preferred). Add mf-api-gateway to your internal DNS, pointing at the gateway host's IP address. Every client and operator on the network then resolves it automatically, and every http://mf-api-gateway:9090/... command in these docs works unchanged from any machine.
  • Per-client /etc/hosts. Where you cannot (or need not) touch DNS, add the same mapping to /etc/hosts on each machine that talks to the gateway:
echo "<gateway-host-ip> mf-api-gateway" | sudo tee -a /etc/hosts

Either way, the point is that the convention name - not localhost, not a raw IP - is what the docs and tools expect, so using it keeps every command portable across machines and deployments. (A topology with more than one gateway is a Deployment: Advanced concern.)

Running a deployment

A deployment project runs three ways - conventionally as a native host process, under Docker, or under Kubernetes - the same base image and the same deployment-config/ in each case, only the launch mechanism differs. The Static MDS and Demo cookbook pages walk through all three for a concrete deployment; the conventional route is the installation-package launcher covered in Deployment: Without Docker. What follows is the deploy-mf-<service> / scripts/start workflow, one way to bring a service up under Docker.

Your MetaFluent GitHub organization

MetaFluent provides each customer a private GitHub organization named MetaFluent-Dist-NN; the number NN is given to you when your access is set up. Everyone on your team signs in with their own GitHub account and is invited by email. The organization holds everything you deploy with:

  • the deployment projects (deploy-mf-*), with their installation packages published as releases,
  • the SDKs (jms-sdk, jdbc-sdk),
  • the container images, in your own registry at ghcr.io/metafluent-dist-nn.

Each deployment project has two lines: master carries the latest release and stable carries the latest milestone. A default clone gives you the release line; to work from the latest milestone, check out stable. Releases are tagged RELEASE-v* and MILESTONE-v*.

The operator workflow is clone, configure, start. For each deployment project your topology calls for:

git clone git@github.com:MetaFluent-Dist-NN/deploy-<service>.git   # replace NN with your organization number
cd deploy-<service>

# Adapt deployment-config/ for your environment (deployment.properties, resources/)
scripts/start

scripts/start pulls the stock base image (the first time; log in with docker login ghcr.io using a personal access token with the read:packages scope) and runs it with your deployment configuration applied; the image itself stays stock. Your base images are in your organization's registry, ghcr.io/metafluent-dist-nn, so point scripts/start there once, in an optional scripts/start-extras file (see Setting up your own container registry for how this works):

# scripts/start-extras
DOCKER_IMAGE="ghcr.io/metafluent-dist-nn/mf-<service>:$version"

Stop it with:

scripts/stop

Your configuration lives in the deployment project and is applied each time you start - edit it, scripts/start again, and your changes take effect. Repeat for each deployment project in your topology.

MAMA feeds

mf-projector-mds, mf-projector-mama, and mf-mds-consolidated include the MAMA adapter, which reads OpenMAMA-bridged feeds (for example RDF/RMDS). The OpenMAMA libraries are licensed separately and supplied by you. The adapter needs two things:

  • Your OpenMAMA installation, as a directory with lib/ (libmama.so and your bridge) and config/ (mama.properties). The application looks for it at /app/mama:
  • Docker or Kubernetes: mount it at /app/mama, for example -v /opt/openmama:/app/mama:ro. With a deployment project, set METAFLUENT_MAMA_INSTALL=/opt/openmama and scripts/start mounts it for you; scripts/bake builds it into your own image instead.
  • Installation package: place it under app/mama/ - see MAMA feeds.
  • A bridge specification describing your bridges and feeds: in deployment-config/resources/mama-adapter/, copy mama-spec-sample.json.txt to mama-spec.json and edit it.

To run with ETA feeds only, turn the MAMA adapter off with METAFLUENT_MAMA_DISABLED=true. The cookbook examples do this, since they run against the ETA simulator.

Building a self-contained image

Running the stock image with your deployment configuration mounted as a volume is one way to operate. The deployment project also provides tools to build your own, fully configured and self-contained image. We call this baking a configuration.

Baking produces a derived image - the stock base image with your configuration built into it - as a single, self-contained artifact. Instead of a stock image plus a mounted configuration, you get one image that already is your configured service.

scripts/bake --tag <your-tag>

<your-tag> is a tag of your own choosing that identifies this build. scripts/start will then run the baked image rather than the stock base.

scripts/bake pulls the base image from the registry named by --base-registry. Use your organization's registry, or a registry of your own if you mirror the base images:

scripts/bake --tag <your-tag> --base-registry ghcr.io/metafluent-dist-nn

You would bake when you want an immutable, portable artifact - for example to promote one reviewed build unchanged through test and into production, or to run in an environment that should not fetch and configure at start time.

A baked image is yours. It goes to a registry you control (see below).

Running without Docker

You are not required to run Elastic MDS under Docker. A base image is just a filesystem plus a startup script, so it can be installed and run as a conventional, native application - and packaged (for example as an RPM) for a Docker-free environment. If that is how your organisation operates, see Deployment: Without Docker.

Setting up your own container registry

Baked images - and, if you prefer, mirrored copies of the stock base images - belong in a container registry you control. For many organisations this is not optional: banks and other regulated operators are reluctant to have any operational element depend on a service outside their control, so pulling from an internal registry rather than directly from ghcr.io at deploy time is often a requirement.

Any standard OCI container registry works. Common choices:

The pattern is the same in every case: authenticate Docker to your registry, then docker tag and docker push the image - one you built, or a mirror of a stock base - into it, under the same image names. scripts/start takes its image from DOCKER_IMAGE, which you set in an optional scripts/start-extras file - to your organization's registry (ghcr.io/metafluent-dist-nn) or to a registry of your own. scripts/start reads that file just before it runs the image, and the version variable holds the selected release or milestone tag:

# scripts/start-extras
DOCKER_IMAGE="registry.example.internal/metafluent/mf-<service>:$version"

Likewise, scripts/bake --base-registry registry.example.internal/metafluent bakes from the base images in your registry, and --registry names where the baked image goes.

We will expand this section with worked, per-vendor steps.

Staying current

Two things can change upstream, and they are separate:

  • A new base image - updated software. You pull the newer image; on its own this does not change anything else.
  • An updated deployment project - new or changed default configuration, resources, or scripts from upstream.

A new base image does not necessarily imply a new deployment project - often you simply run a newer image against the same project. This section is about the second case: taking an updated deployment project into your clone while preserving your own edits, both deployment.properties and anything you added or changed under resources/. Because the project ships defaults that you have adapted, a straight merge can collide. The project includes config-merge for exactly this:

scripts/config-merge --upgrade . --upstream <release-ref>

config-merge --upgrade fetches the upstream release and performs an interactive three-way merge into your project - preserving your edits while folding in new or changed upstream configuration, and refreshing the build-generated scripts/. It does the git fetch and applies the result for you; you review what it proposes. --upstream names the release to reconcile against. If you bake, re-bake afterwards.

Each deployment project ships the release notes for the version it delivers - what changed and any compatibility notes - in the project repository. Check them before taking an upgrade.

Where to go next