Fixing a ZDM Job Stuck in RUNNING… then SUSPENDING

Hi everyone, hope you’re all doing well.

Today I want to share a war story from a ZDM migration cutover that turned in an unexpected issue caused by an human error.

The short story: I accidentally hit Ctrl+C at the wrong time during a zdmcli resume job. This caused the ZDM job to keep in EXECUTING status with no changes at all. I will show here what I did and how I fixed that, saving our cutover :-).

The Setup

We’re in the middle of migrating one of our client’s non-prod databases from EC2 on-premises to ExaCS on AWS (Multicloud), using ZDM 26.1 in Instant Deploy (Serverless) mode. The database in question was a 1.9 TB non-CDB Oracle 19c database migrating to Exa@AWS.

Job 63 had been running since October 23rd and had completed everything up through ZDM_CONFIGURE_DG_SRC, meaning the RMAN restore was done, the Data Guard was configured and in sync, and we were paused and ready for cutover. The job had been sitting at this pause point for almost two weeks, waiting for our maintenance window.

Cutover day arrives. Time to resume.

The Accidental Ctrl+C

I ran the resume command, intending to skip the ZDM switchover phase since we’d handle that part manually:

zdmcli resume job -jobid 63 -skip SWITCHOVER

And then, without thinking, I just hit CTRL+C.

The command was still initializing. The ZDM CLI client died. But in ZDM Serverless mode, the CLI is just a client that talks to an embedded Apache Derby database (that ZDM uses to control the job statuses, etc). The important question was: what state did it leave behind?

zdmcli query job -jobid 63 | grep "Current status"

Current status: EXECUTING

Not great. The job was showing as EXECUTING, which meant it had committed the state transition before doing any actual work. That’s what ZDM does on resume, it sets the status to EXECUTING first, then starts processing. My Ctrl+C killed the client after the commit went through.

I tried resuming again:

PRGO-4116 : failed to resume the job "63" as the job status currently is: "EXECUTING"

Right. Can’t resume what’s already executing, even if nothing is actually running.

The Suspend

On my mind I just thought: OK, if is in EXECUTING state, I will change it to SUSPEND, it was supposed to work, right?

zdmcli suspend job -jobid 63

Job execution will be suspended after phase "ZDM_CONFIGURE_DG_SRC".

That looked promising. But then:

Current status: SUSPENDING

And then, the job kept in SUSPENDING state, even after several minutes.

The Diagnostic

I am working with ZDM for a long time, so I am very familiar with the log path structure. I verified the log:

/u02/app/oracle/zdm/zdm_<MYDB>_63/derby.log

Looking through it, I found something interesting mixed in with the ZDM internals — a Derby error that had nothing to do with the migration itself:

Wed Nov 04 17:15:06 CST 2026 Failed Statement is: INSERT INTO IMGTYPE (IMGTYPENAME, ACE, BASETYPE,
 CREATEDATE, CREATOR, ...) VALUES (?, ?, ?, ?, ...) begin parameter #1: ORACLEGISOFTWARE 
ERROR 23505: The statement was aborted because it would have caused a duplicate key value in a unique 
or primary key constraint or unique index identified by 'SQL0000000086-f0e9813d-01a1-1350-4b83-00000e224840' 
defined on 'IMGTYPE'.

There it was. This error didn’t happen at the Ctrl+C moment — the job was already stuck in SUSPENDING by then. What the log showed was that a subsequent zdmcli invocation (likely the suspend command or a status query) had triggered Derby’s schema initialization routine, which tried to insert a record into IMGTYPE and collided with one already there from a previous run. The client crashed on that duplicate key, never completing the state transition it was sent to do.

The IMGTYPE error was a side effect — not the root cause. But it was the breadcrumb I needed: it confirmed that (1) ZDM was using an embedded Derby database to manage job state, and (2) that database was at a known path I could connect to directly.

Finding the Derby Repository

ZDM Serverless stores everything in an embedded Apache Derby database at:

To connect to it, I needed Derby’s JDBC client. ZDM bundles its own Java 11 JDK and Derby jars, but there are a few tips worth calling out before you try this yourself.

Tip #1 – Don’t use the System Java

The ExaCS nodes have Java 8 as the system default. The Derby jars bundled with ZDM 26.1 were compiled targeting Java 11 (class file version 55.0). If you try to run them with the system java, you’ll get:

UnsupportedClassVersionError: org/apache/derby/iapi/jdbc/AutoloadedDriver has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version only recognizes up to 52.0

Always use ZDM’s own JDK:

/acfs01/acfs/zdm/zdm26.1/zdmhome/jdk/bin/java -version

# openjdk version "11.x..."

Tip #2 — The driver class name

The Derby JDBC driver in ZDM is not the typical org.apache.derby.jdbc.EmbeddedDriver you’ll find in every Stack Overflow answer. The registered service class (from META-INF/services/java.sql.Driver inside derby.jar) is:

org.apache.derby.iapi.jdbc.AutoloadedDriver

Using JDBC 4.0 auto-discovery alone won’t work, you need an explicit Class.forName() call with this exact name.

Connecting and Reading the Job State

Claude wrote for me a small Java program to connect to the Derby repo and read the job state. First, let me list what tables exist, because nothing in ZDM’s docs tells you the schema:

import java.sql.*;
public class fixzdm {
  public static void main(String[] args) throws Exception {
    Class.forName("org.apache.derby.iapi.jdbc.AutoloadedDriver");
    Connection c = DriverManager.getConnection(
      "jdbc:derby:/acfs01/acfs/zdm/zdm26.1/zdmbase/derbyRepo");
    ResultSet r = c.getMetaData().getTables(null, null, "%", new String[]{"TABLE"});
    while(r.next())
      System.out.println(r.getString("TABLE_SCHEM") + "." + r.getString("TABLE_NAME"));
    r.close(); c.close();
  }
}

GHSUSER23.COLLECTION
GHSUSER23.CREDENTIALS
GHSUSER23.CREDENTIALS_ROLES
GHSUSER23.CUSTOMER
GHSUSER23.DATA_GUARD_CFG
GHSUSER23.DB
GHSUSER23.DB_TAG
GHSUSER23.DG_CFG_SITES
GHSUSER23.GHAUDIT
GHSUSER23.HOLDER
GHSUSER23.ID_GEN
GHSUSER23.IMAGE_ROLES
GHSUSER23.IMGS
GHSUSER23.IMGSRS
GHSUSER23.IMGSRS_IMAGES
GHSUSER23.IMGSRS_USERS
GHSUSER23.IMGTYPE              ← duplicate key error happened here
GHSUSER23.IMGTYPE_USERACTION
GHSUSER23.JOBSCHEDULER         ← job status lives here
GHSUSER23.JOBSCHEDULER_ROLES
GHSUSER23.MOVE
GHSUSER23.PARENT_CHILD_ROLE
GHSUSER23.ROLES
GHSUSER23.SITES
GHSUSER23.SITES_TAG
GHSUSER23.SPARE1
GHSUSER23.SPARE2
GHSUSER23.SWHOMES
GHSUSER23.TAG
GHSUSER23.USERACTION
GHSUSER23.USERS
GHSUSER23.USER_ROLES
GHSUSER23.WCOPY_ROLES

As you can see, the Derby schema is GHSUSER23 and the tables are displayed above.

Note: The schema name GHSUSER23 may differ across ZDM deployments. Before running any queries, verify yours by listing schemas: SELECT TABLE_SCHEM FROM SYS.SYSTABLES GROUP BY TABLE_SCHEM. Use whatever schema shows up for your ZDM tables — it may not be GHSUSER23 in your environment.

Querying job 63 directly:

ResultSet r = s.executeQuery(
  "SELECT JOB_IDENTIFIER, CURRENT_STATUS, IS_PAUSED, PHASE, PAUSE_AFTER_PHASE " +
  "FROM GHSUSER23.JOBSCHEDULER WHERE JOB_IDENTIFIER = 63");

JOB_IDENTIFIER: 63 CURRENT_STATUS: SUSPENDING IS_PAUSED: 0 PHASE: ZDM_CONFIGURE_DG_SRC PAUSE_AFTER: null

ResultSet r = s.executeQuery(

There it was. The SUSPENDING status committed and sitting in Derby, with IS_PAUSED=0 and no PAUSE_AFTER_PHASE. No process to move it forward.

Baselining Against a Healthy PAUSED Job

Before touching anything, I queried another job that I knew was correctly in PAUSED state: job 55 (for another DB), which had been parked at the same phase since its own maintenance window hadn’t happened yet:

JOB_IDENTIFIER: 55 CURRENT_STATUS: PAUSED IS_PAUSED: 1 PHASE: ZDM_CONFIGURE_DG_SRC PAUSE_AFTER: ZDM_CONFIGURE_DG_SRC

Three columns were different. That’s exactly what I needed to fix on job 63.

Applying the Fix

With the target values confirmed, I updated all three columns in a single transaction with explicit commit:

import java.sql.*;
public class fixzdm {
  public static void main(String[] args) throws Exception {
    Class.forName("org.apache.derby.iapi.jdbc.AutoloadedDriver");
    Connection c = DriverManager.getConnection(
      "jdbc:derby:/acfs01/acfs/zdm/zdm26.1/zdmbase/derbyRepo");
    c.setAutoCommit(false);
    Statement s = c.createStatement();
    int rows = s.executeUpdate(
      "UPDATE GHSUSER23.JOBSCHEDULER SET CURRENT_STATUS='PAUSED', IS_PAUSED=1, " +
      "PAUSE_AFTER_PHASE='ZDM_CONFIGURE_DG_SRC' WHERE JOB_IDENTIFIER=63");
    System.out.println("Rows updated: " + rows);
    c.commit();
    // verify
    ResultSet r = s.executeQuery(
      "SELECT JOB_IDENTIFIER, CURRENT_STATUS, IS_PAUSED, PAUSE_AFTER_PHASE " +
      "FROM GHSUSER23.JOBSCHEDULER WHERE JOB_IDENTIFIER=63");
    while(r.next())
      System.out.println("Status: " + r.getString("CURRENT_STATUS") +
        " IS_PAUSED=" + r.getString("IS_PAUSED") +
        " PAUSE_AFTER=" + r.getString("PAUSE_AFTER_PHASE"));
    r.close(); s.close(); c.close();
  }
}

Compile and run using ZDM’s own JDK, this is critical:

ZDMHOME=/acfs01/acfs/zdm/zdm26.1/zdmhome
CP=${ZDMHOME}/jlib/derby.jar:${ZDMHOME}/jlib/derbyshared.jar

${ZDMHOME}/jdk/bin/javac -cp ${CP} /tmp/fixzdm.java -d /tmp/
${ZDMHOME}/jdk/bin/java  -cp /tmp:${CP} fixzdm

Rows updated: 1 Status: PAUSED IS_PAUSED=1 PAUSE_AFTER=ZDM_CONFIGURE_DG_SRC Exception in thread "main" java.sql.SQLException: Cannot close a connection while a transaction is still active. → Harmless — the commit() already ran. ResultSet was still open at close() time.

Rows updated: 1. The commit ran successfully. The close() exception is a known Derby behavior when a ResultSet is open at connection close time, it has zero effect on the committed data.

Checking the Job Status again

With Derby updated, I queried the job through ZDM’s CLI to confirm it picked up the new state:

zdmcli query job -jobid 63 | grep "Current status"

Current status: PAUSED

And resumed:

zdmcli resume job -jobid 63 -skip SWITCHOVER

Job "63" resumed successfully.
Current status: PAUSED

The job picked up from ZDM_SWITCHOVER_SRC (skipped), ran through ZDM_POST_DATABASE_OPEN_TGT, ZDM_DATAPATCH_TGT, and completed. <MYDB> is now running on Exadata@AWS.

Key Takeaways

• In ZDM Serverless mode, the CLI is stateless. There is no persistent daemon, each command is a self-contained JVM that talks to embedded Derby and exits. A killed client leaves the last committed state in Derby with no recovery mechanism.
• When a job gets stuck in a transitional state (EXECUTING, SUSPENDING), the fix lives in table GHSUSER23.JOBSCHEDULER, specifically columns CURRENT_STATUS, IS_PAUSED, and PAUSE_AFTER_PHASE.
• Always use ZDM’s own Java 11 JDK (/acfs01/acfs/zdm/zdm26.1/zdmhome/jdk/bin/java) — not the system Java — to connect to Derby jars bundled with ZDM 26.1.
• The Derby driver class is org.apache.derby.iapi.jdbc.AutoloadedDriver — not org.apache.derby.jdbc.EmbeddedDriver.
• Before making any writes, baseline against a known-good job in the state you want. It eliminates guesswork and gives you the exact values to target.
• The derby.log file at the job’s work directory (e.g. /u02/app/oracle/zdm/zdm_<MYDB>_63/derby.log) is your best friend when things go sideways, it was what led me to the repository path in the first place.

Conclusion

An accidental CTRL+C in the wrong SSH session turned into deep-dive into ZDM internals, but it also gave me a much better understanding of how ZDM Serverless actually works under the hood. The fact that it uses an embedded Derby database is completely undocumented as far as I know, and knowing the schema structure opens up real diagnostic possibilities when things go wrong.

If you’ve hit something similar or have questions, drop them in the comments. And if you’re running ZDM migrations, be very careful with Ctrl+C during resume operations. 🙂

Peace,

Vinicius