Interview practice · Advanced

Production troubleshooting drills

Work from symptoms to evidence, diagnosis, correction, and verification.

Learn this topic first: Interview lab: design, troubleshoot and code →

6 explained questions · 5 practice MCQs

Questions and answers

1. The job completed normally but business rows are missing. What next? (Intermediate)

Separate normal program termination from business success. Check the input set, selected member/library, predicates, commit outcome, and whether a broad error handler suppressed failures. Compare run counts and expected control totals.

Trace one missing business key end to end. Confirm that the output is not simply being queried from a different environment. Repair the root cause and use a controlled replay keyed by durable state, rather than rerunning all work without duplicate protection.

2. CHAIN fails only in production. What do you compare? (Advanced)

Compare the exact key values and types, resolved file and member, access-path definition, select/omit rules, overrides, and job environment. Check for trailing blanks, scale differences, and status handling that reads stale fields.

Reproduce the specific key under the same access path and authority. Inspect concurrency and error status separately from a simple no-match result. Do not assume the production data is absent merely because a different session can find a row through another interface.

3. A batch slows dramatically after a release. How do you narrow it down? (Advanced)

Compare row volumes, query counts/plans, library resolution, lock waits, and external latency with the previous baseline. Identify whether queue wait or execution duration changed. Inspect new per-record SQL, index maintenance, or longer transaction boundaries.

Use a representative sample and preserve current evidence. Roll back a demonstrably harmful change when the release plan supports it, but keep investigating the cause. A reboot that temporarily warms or clears state does not establish why the regression occurred.

4. A service-program change works in new jobs but not old jobs. Why? (Advanced)

Long-lived jobs may retain activated code/resources or cached application state, while new jobs activate the newly deployed version. Also check whether consumers were bound to a different object or signature.

Compare qualified objects and job lifecycles. Use a coordinated recycle or activation-management procedure after compatibility checks. Avoid ending every subsystem as a first move; identify the affected application jobs and confirm their transaction/recovery state.

5. A deadlock happens once a day. How do you reproduce it? (Advanced)

Find the involved business keys, jobs, and lock acquisition order. Build a controlled two-worker test that pauses each transaction after acquiring its first resource, then requests the other resource in the observed order.

Use the reproduction to validate consistent ordering or shorter units of work. Add bounded retry handling only for classified retryable failures and verify that it repeats the entire safe unit. Keep diagnostic correlation so future contention can be distinguished from the original cycle.

6. The integration reports success but the partner has no record. What now? (Advanced)

Determine what success meant: local commit, accepted HTTP request, queue send, or confirmed partner business completion. Correlate the operation ID across the outbound ledger, transport response, and partner lookup.

Do not resend blindly. Reconcile whether the partner rejected, delayed, or completed the request under another identifier. Improve the status model so accepted and completed are distinct, and alert on aging pending operations. This avoids misleading users with a success flag at the wrong boundary.

Practice checkpoint

  1. 1. Does normal job termination prove every intended row was posted?
    1. No
    2. Yes, always
    3. Only for CLLE
    4. Only on an empty queue
  2. 2. Production CHAIN differs from a test SQL query. Compare:
    1. Only the query editor theme
    2. Keys, access paths, members, and resolved objects
    3. Only screen size
    4. Only source indentation
  3. 3. Old server jobs behave differently after deployment. Inspect:
    1. Only disk capacity
    2. Only scheduled day
    3. Activation and cached state
    4. Only the output queue name
  4. 4. What makes a deadlock test meaningful?
    1. Sleeping randomly once
    2. Only running a single transaction
    3. Changing every lock timeout
    4. Reproducing the actual resource acquisition order
  5. 5. What is the first step after ambiguous integration success?
    1. Correlate and reconcile the operation outcome
    2. Send the same payment with a new key
    3. Delete pending work
    4. Assume the partner failed
Show answer key and explanations

1. A — No Business completion needs explicit counts/outcomes and transaction verification.

2. B — Keys, access paths, members, and resolved objects The two operations may not address the same data or predicate semantics.

3. C — Activation and cached state Existing jobs can retain runtime state that new jobs do not have.

4. D — Reproducing the actual resource acquisition order A controlled interleaving exposes the circular dependency.

5. A — Correlate and reconcile the operation outcome Transport acceptance and business completion are different milestones.

IBM documentation and further reading