Adding an ELTMaestro server edge node (engine-only)¶

An edge node is a separate Linux server that runs workflows but holds no local state — no Postgres, no meta-service, no keystore. The master server dispatches work to it over SSH, and the node connects back to the central server's Postgres audit DB (sqlmaestro) for all batch/step/watermark state. Use edge nodes to scale execution horizontally or to run workflows close to a particular data source/network.

Same shape as the full INSTALL.md, but the OS prerequisite is just JDK 21 (no PostgreSQL), and you run install-engine.sh instead of install.sh. The engine-only installer ships in the same eltm-portable tarball.

How it works¶

  master / central server                         edge node
  ┌─────────────────────────┐                     ┌─────────────────────────┐
  │ Postgres audit DB  sshd  │  ← JDBC 5432 ──────│ engine (--slave)         │
  │  (sqlmaestro)            │  ─ SSH: run cmd ──▶│  sqlm.jar + JDBC + Spark │
  │ meta-service, WPF, etc.  │     + SFTP job xml  │  sshd, deploy dir        │
  │                          │  ◀─ SSH: log back ─│                          │
  └─────────────────────────┘                     └─────────────────────────┘

The engine batch run uses SSH to execute run commands on the node, not just to copy files — and it's bidirectional:

Master → node (dispatch). When a workflow targets a non-master server, the master looks the node up in t_maestro_servers (→ its SSH connection in connection2), SSH-executes . ~/.env_integrator; echo $MAESTRO_DEPLOYED_FILES_DIR to find the deploy dir, SFTPs the job XML(s) there, then SSH-executes the run command . ~/.env_integrator; java -jar $MAESTRO_JAR_FILE --slave --batch <xml> &. The --slave engine connects to the central audit DB via ABC* and runs the workflow — it never calls the meta-service.
Node → master (log return). When the slave finishes, it looks up the master (select server_ssh_connection_name from t_maestro_servers where is_master=true), SSHes back to the master, reads the master's $MAESTRO_CONSOLE_LOG_DIR, and SFTPs its console log there. This is wrapped in try/catch — if the reverse path isn't wired, the workflow still runs and the central audit DB still tracks all state (the slave writes it directly); only the console-log copy-back is skipped. Wire it (Step 5e) for log visibility on the master.

(Every non-Snowflake job runs as a Spark job, so the node ships with Spark + Hadoop.)

Conventions used below¶

maestro is the dedicated unprivileged user that owns the install and that the master's SSH connection logs in as — its ~/.env_integrator is what the master sources. Any account works; this doc uses maestro.
/data/ is the parent dir the tarball extracts into → /data/iserver/....
<VERSION> matches the tarball suffix (current release line 17.0.4).
<central_server> is the host/IP of the main ELTMaestro server (its Postgres).
<node_ip> is this edge node's address as the central server sees it.

Step 1 — prerequisites (root, once per host)¶

Only JDK 21 + the common CLI tools — no PostgreSQL on an edge node. Also make sure sshd is installed and running (the master dispatches over SSH).

RHEL / Rocky / Alma 8.10+ or 9.4+¶

sudo dnf install -y java-21-openjdk-devel
sudo dnf install -y curl tar gzip openssl hostname rsync openssh-server
sudo systemctl enable --now sshd

Java 21 is only in RHEL/Rocky/Alma 9.4+ (and 8.10+). On 9.3 or older the java-21-openjdk package is absent and the installer's preflight will stop.

Debian 12 / Ubuntu 22.04 / Ubuntu 24.04¶

sudo apt update
sudo apt install -y openjdk-21-jdk-headless curl ca-certificates tar gzip hostname openssl rsync openssh-server
sudo systemctl enable --now ssh

install-engine.sh's preflight finds Java under /usr/lib/jvm/ and verifies the tools; if anything's missing it prints the exact dnf/apt line. rsync is optional (quieter Spark startup).

Step 2 — create the dedicated user (root, once)¶

sudo useradd -m -s /bin/bash maestro
sudo mkdir -p /data
sudo chown maestro:maestro /data

Use the same user the master will SSH in as. install-engine.sh warns (but does not fail) if run as root.

Step 2.5 — download the release from S3 (as maestro)¶

su - maestro
cd /data
curl -fLO https://eltmaestro.s3.amazonaws.com/docker/eltm-portable-<VERSION>.tar.gz

Same tarball as the full install — see DOWNLOADS.md for exact URLs.

Step 3 — extract the tarball (as maestro)¶

cd /data
tar -xzf eltm-portable-<VERSION>.tar.gz
ls -d iserver        # -> /data/iserver/...

Step 4 — run install-engine.sh (as maestro)¶

/data/iserver/install-engine.sh \
    --abchost <central_server> \
    --abcport 5432 \
    --abcuser maestro \
    --abcpassword '<audit-db-password>'

(install-engine.sh is the convenience symlink to bin/install-engine.sh. --abchost is required; --abcport defaults to 5432, --abcuser to maestro; the password is prompted with read -s if you omit it on a terminal.) In order it:

Detects distro and runs the preflight with ENGINE_ONLY=1 — JDK 21 + tools, skips the PostgreSQL check (no local PG on an edge node).
Writes the central audit-DB settings (ABCHOST/ABCPORT/ABCUSER/ABCPASSWORD
ABCMASTER=false) into ~/.env_integrator.paths (mode 600 — it holds the password), then installs ~/.env_integrator.
Creates the engine dirs, incl. data/workflow (= $MAESTRO_DEPLOYED_FILES_DIR, where the master SCPs job XML).
Downloads Spark 4.1.1 + Hadoop 3.4.3 into tools/ (~1.3 GB, one-time).
Installs the integrator metadata (metadata/integrators/<jobType>/system.cfg, get_*, static_*, launcher.sh) into $PGDIR/metadata — the engine reads each target platform's config from here even on an edge node with no Postgres. Without it, Snowflake/MPP jobs emit SQL with unsubstituted $SYSTEM_* tokens (e.g. $SYSTEM_TOKEN_ENCLOSER) and DDL fails.
Asserts the audit JDBC driver library/JDBC/postgresql-42.7.11.jar, does a TCP reachability probe to the central DB, and prints the wiring summary.

It does not install a local Postgres, the meta-service, a keystore, or start anything. Expect ~5-10 min (the Spark/Hadoop downloads dominate).

Snowflake / MPP key-pair targets on an edge node. The engine jar (with the javax jaxb-api) and --add-opens ride this same tarball/env_integrator, and the metadata above is now installed — but key-pair Snowflake also needs, on this node: the private key file at the path the connection string's private_key_file= references, and an NTP-synced clock (skew ⇒ JWT token is invalid). See SNOWFLAKE-SLAVE-REMEDIATION.md for the per-node checklist + script.

Step 5 — wire the node into the cluster¶

Three things connect a node: (a) the master can SSH to it, (b) the node can reach the central Postgres, and (c) the node is registered in the audit DB.

5a. On the EDGE NODE — accept the master's SSH key¶

The master authenticates to the node with an SSH key (recommended) or password. For key auth, put the master's public key in the install user's authorized_keys:

# as maestro on the edge node
mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo '<master-ssh-public-key>' >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Confirm sshd is up (Step 1). Test from the master: ssh -i <master-private-key> maestro@<node_ip> 'echo ok'.

5b. On the CENTRAL SERVER — let the node reach Postgres¶

The edge engine connects to jdbc:postgresql://<central_server>:5432/sqlmaestro. On the central host (bare-metal /data/iserver/server/pgsql/data/, or inside the eltmaestro container for a Docker deploy):

# postgresql.conf
listen_addresses = '*'          # or the central server's IP

# pg_hba.conf  — allow this node + the maestro user to the sqlmaestro DB
host  sqlmaestro  maestro  <node_ip>/32  scram-sha-256

Reload Postgres (pg_ctl reload, or docker exec eltmaestro supervisorctl ...), and open the firewall:

sudo firewall-cmd --add-port=5432/tcp --permanent && sudo firewall-cmd --reload   # firewalld
# or: sudo ufw allow from <node_ip> to any port 5432                              # ufw

5c. Register the node + its SSH connection (in the audit DB)¶

The master finds a node by server_name in t_maestro_servers and its SSH creds in connection2. Easiest is the WPF admin client (it encrypts the SSH password and formats the key field for you); the equivalent SQL on the central sqlmaestro DB is:

-- 1) the SSH connection the master uses to reach the node (type 'H' = host/SSH)
INSERT INTO connection2
  (connection_nm, host_distinguished_nm, secondary_distinguished_nm,
   login_nm, password_encrypted_txt, port_num, connection_type_cd, database_instance_nm)
VALUES
  ('EDGE1_SSH', '<node_ip>', '<node_ip>',
   'maestro', '', 22, 'H', 'key:sftp:/home/maestro/.ssh/id_master');
--   database_instance_nm formats:
--     key auth  : 'key:sftp:/path/to/master_private_key'   (sftp transport)
--     password  : 'pass:sftp:'  + put base64(password) in password_encrypted_txt
--                 (the WPF stores Crypt/base64; e.g. `printf '%s' '<pw>' | base64`)

-- 2) register the node as a non-master execution server
INSERT INTO t_maestro_servers
  (server_id, server_name, server_address, server_maestro_port,
   server_ssh_connection_name, is_master)
VALUES
  ('EDGE1', 'EDGE1', '<node_ip>', 8181, 'EDGE1_SSH', false);

server_name (EDGE1 here) is the value you target a workflow at; the path on the node is discovered at run time from the node's own $MAESTRO_DEPLOYED_FILES_DIR.

5e. (Reverse path) Let the node SSH back to the master for log return¶

When a slave finishes it SSHes back to the master to copy its console log there (BatchThread finally block). This needs the master registered with is_master=true and an SSH connection the slave can use — the opposite direction of 5a/5c. Skippable: without it the workflow still runs and the central audit DB still records everything; only the console-log copy-back to the master is lost (the log stays in the node's $MAESTRO_CONSOLE_LOG_DIR).

On the MASTER server — run sshd and authorize the node's public key:

# as the master's install user
sudo systemctl enable --now sshd            # (sshd / ssh)
echo '<node-ssh-public-key>' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys

On the NODE — place the node→master private key (the slave's engine reads it from the path in the connection below): e.g. ~/.ssh/id_to_master (chmod 600).

In the central sqlmaestro DB — the master's SSH connection + the master row:

INSERT INTO connection2
  (connection_nm, host_distinguished_nm, secondary_distinguished_nm,
   login_nm, password_encrypted_txt, port_num, connection_type_cd, database_instance_nm)
VALUES
  ('MASTER_SSH', '<master_address_from_node>', '<master_address_from_node>',
   'maestro', '', 22, 'H', 'key:sftp:/home/maestro/.ssh/id_to_master');   -- path ON THE NODE

-- register the master itself (is_master=true) with its SSH connection
INSERT INTO t_maestro_servers
  (server_id, server_name, server_address, server_maestro_port,
   server_ssh_connection_name, is_master)
VALUES
  ('MASTER', 'MASTER', '<master_address>', 8181, 'MASTER_SSH', true);

The slave resolves MASTER_SSH by is_master=true, so exactly one master row must carry a valid server_ssh_connection_name. (For a Docker-deployed master, the eltmaestro container must itself run sshd for this to work.)

5d. Target a workflow at the node¶

Assign the workflow's execution server to the node's server_name (in the WPF / job config). On the next run the master takes the SSH branch instead of running locally: it SCPs the job XML(s) to the node and launches java -jar sqlm.jar --slave.

Step 6 — verify¶

# on the edge node — confirm the env the master will source
. ~/.env_integrator
echo "$ABCHOST $ABCPORT $ABCUSER $ABCMASTER"        # -> <central> 5432 maestro false
timeout 5 bash -c "</dev/tcp/$ABCHOST/$ABCPORT" && echo "central DB reachable (TCP)"
ls "$MAESTRO_JAR_FILE"; "$SPARK_HOME/bin/spark-submit" --version 2>&1 | grep -i version

Then dispatch a workflow targeting the node and confirm: the job XML lands in the node's $MAESTRO_DEPLOYED_FILES_DIR (/data/iserver/data/workflow), $MAESTRO_CONSOLE_LOG_DIR/<xml>.log shows the engine starting --slave, and a new batch_cycle_run row appears in the central sqlmaestro.

Notes¶

The central DB must be named sqlmaestro and be plain Postgres (the engine hardcodes the db name + org.postgresql.Driver). The audit JDBC driver postgresql-42.7.11.jar ships in the tarball under library/JDBC/.
The node runs nothing standing — keep sshd up and the deploy dir writable; the master launches the engine per dispatch.
Re-running install-engine.sh with new --abchost/creds rewrites ~/.env_integrator.paths (so updated values take effect) but leaves an existing ~/.env_integrator alone.
No local HDFS is started; Spark runs local and stages to local dirs.
For a Docker deploy of the central server, the node reaches Postgres via the central host's address only if 5432 is published from the eltmaestro container (or both are on the same Docker network).