Atlas secondary stuck in RECOVERING after oplog rollover

Atlas secondary stuck in RECOVERING after oplog rollover

Hello, I'm 小兔大白糖. An Atlas dedicated replica set stayed yellow for more than a week: one member stuck in RECOVERING, the deploy stuck in UPDATING, while app traffic still worked — the real question is usually not “upgrade the tier again,” but whether a secondary has already fallen off the oplog window.

Oplog rollover means the capped oplog overwrote the resume point a lagging secondary needed. Incremental catch-up is then impossible; only an initial sync or node replacement works, and Atlas customers often need Support to clear the stuck change.

Mental model: treat the oplog as a fixed-length ring tape — new writes keep overwriting the oldest segment. The analogy covers “the window shrinks and resume points vanish”; it does not explain Atlas’s deploy state machine, or how slowly a secondary can apply entries.

This is a teaching postmortem for backend engineers running bulk writes on MongoDB Atlas replica sets, and for anyone who needs a crisp line between RECOVERING and what w=majority does not protect. Timestamps, sizes, and lag figures below are teaching-case readings. 中文版:Atlas 从节点 oplog 被卷掉后一直 RECOVERING。

Reading time: ~18 minutes

Contents


1. What is the one-line conclusion?

Conclusion: once a secondary falls off the oplog window, only a full rebuild helps — a tier upgrade does not replay overwritten history.

A large, sustained upsert workload made the primary append oplog entries faster than one secondary could apply them. Replication lag grew until that secondary’s required history was overwritten. Incremental catch-up became impossible. The node stayed in RECOVERING and needed a full initial sync.

Atlas would not finish a configuration deploy until all three members were healthy, so the cluster remained UPDATING (“0 of 3 servers complete”) for a long time — more than a week in the teaching case. The application kept serving traffic because the primary and the other secondary were healthy. Fault tolerance was reduced to “do not lose another node.”

Raising the tier (M10 → M20) does not repair the member that has already fallen off the oplog. It does lower the chance of the same failure under a future burst, because secondaries can apply faster and the oplog window is usually larger. Pacing bulk writes matters more than write concern alone.


2. What was the impact?

AreaWhat happened
AvailabilityReads and majority writes continued on the healthy primary and one healthy secondary.
Fault toleranceOne voting member was unusable. A second failure could lose majority.
Cluster changesDeploys, failovers-as-a-test, and tier changes were blocked or unsafe while state stayed UPDATING.
Data on the bad nodeIn the teaching case, frozen about 15 days behind the primary (syncedTo 2026-10-27 16:30 HKT). Not a client-visible data loss on the primary.
OperationsNo self-service “restart this member” or “force resync” in Atlas.

3. How does the timeline read?

Times below mix Atlas alert timestamps (UTC) and the lag reading (HKT). The rows are a teaching-case signal chain.

WhenSignal
2026-10-26REPLICATION_OPLOG_WINDOW_RUNNING_OUT (closed after the window recovered briefly).
2026-10-27 16:30 HKTUnhealthy secondary’s syncedTo stops here. Later readings never move past it.
2026-10-27Metric threshold alert, then closed.
2026-11-03HOST_DOWN on the replica set for about 9 minutes, then closed.
2026-11-04Oplog-window alert again.
2026-11-09UI still yellow / RECOVERING. API state UPDATING. No OPEN / TRACKING alerts on the panel. Disk about half full.
2026-11-10Another oplog-window alert. MongoDB patch level moved (8.0.36 → 8.0.37) while the cluster stayed UPDATING.
2026-11-11Lag check: healthy secondary 0s; unhealthy secondary ~354 hours behind, still RECOVERING.

4. How was it found?

  1. Atlas Overview stayed yellow: banner Deploying your changes failed; we will try again shortly, plus 0 of 3 servers complete (waiting for 3 servers to be healthy, configuring MongoDB).
  2. Member list: one PRIMARY and one SECONDARY later showed green; one member stayed RECOVERING.
  3. Early on, all three rows were yellow. That was the cluster-level UPDATING state, not three dead processes.
  4. A common misread is “scale M10 → M20 first.” Treat a tier change as a hypothesis until disk, lag, and alerts line up — not as the repair for a member already outside the window.

5. How to investigate?

5.1 What Atlas would and would not show

  • Cluster state from the Admin API: UPDATING, tier still M10, not paused.
  • Alert panel: no OPEN or TRACKING items at investigation time. The failure was a stuck workflow, not a live “disk full” or “host down” alarm.
  • Closed alerts told the story: oplog window pressure, then a short host-down, then oplog window pressure again.
  • UI ... menu had Test Resilience, pause, and terminate. It had no per-node restart and no resync.
  • Test Resilience only steps down the current primary. Atlas also refuses it unless every member is healthy and lag is under 10 seconds. It does not repair RECOVERING.

5.2 Replication evidence

Checked from a client that could see secondary replication info (teaching case):

MemberRole at read timesyncedToLag
node-aSECONDARY2026-11-11 (current)0s
node-bPRIMARY——
node-cRECOVERING2026-10-27 16:30 HKT~354 hours

On a healthy member, dbStats showed disk roughly 33 GB used of ~60 GB (~54%). This was not a full disk.

Oplog on the healthy primary still held on the order of 5–6 days of history when sampled. A secondary stuck on 2026-10-27 was already outside that window, so it could not resume by replaying oplog.

Automation on the reachable member still had a newer config version pending than the version it had successfully applied. That matches the UI retry loop: Atlas keeps trying to finish “configure MongoDB,” and each attempt waits for three healthy servers.

5.3 Business workload

The suspected producer is a multi-million-row upsert job (entity-shaped documents, teaching sketch; UpdateOne + upsert, batches sent with bulk_write). That pattern was not proven by a single slow-query log line as “the” command that killed the node. It is the kind of workload that fills an M10 oplog quickly:

  • every upsert is an oplog entry, including no-op-looking overwrites
  • a tight loop does not wait for the slowest secondary
  • M10 disk IOPS and CPU cap how fast a secondary can apply those entries

Other heavy insert/update/delete jobs would have the same effect. “Upsert” is not special; oplog bytes per second vs secondary apply rate is.

5.4 Ruled out

IdeaWhy it does not fit
Disk full~54% used on the healthy member.
All three nodes downPrimary and one secondary served traffic; lag 0 on the healthy secondary.
Restart from the Atlas UIControl does not exist for Atlas (it exists in Cloud Manager).
Test ResilienceRestarts the primary only, and only when the set is already healthy.
Tier upgrade as the immediate fixThe broken member is ~15 days behind the oplog window. A bigger box does not replay overwritten history.
Current CPU saturation as proofNo resource alerts still OPEN during the stuck period. Historical oplog-window alerts are the capacity clue, not a live requirement to scale before resync.

6. What is the root cause?

writes faster than a secondary can apply
        → replication lag grows
        → oplog is a fixed-size ring, so the oldest entries are overwritten
        → the lagging secondary no longer has a common oplog point
        → incremental replication is impossible
        → only initial sync (full rebuild) or a node replacement can catch up
        → that rebuild is slow or stuck
        → member stays RECOVERING
        → Atlas deploy waits for 3 healthy servers
        → cluster stays UPDATING and retries the failed change

Oplog rollover means overwrite, not “the file vanished.” New entries consume a capped collection. When the cap is reached, MongoDB removes the oldest entries. The oplog window is how much clock time those remaining entries cover. Fast writes shrink the window from “several days” toward “hours” even if the byte size stays the same.

Once node-c needed an entry older than the current window, no amount of waiting would let it tail the oplog. Atlas’s automatic recovery is an initial sync. If that sync never finishes, the UI stays on RECOVERING and the deploy stays at 0/3.

Root-cause chain: bulk writes outrun apply rate, oplog rollover, RECOVERING, stuck UPDATING
Root-cause chain: bulk writes outrun apply rate, oplog rollover, RECOVERING, stuck UPDATING · 点击查看 SVG 原图

7. What does the architecture look like?

Normal steady state: clients write to the primary; both secondaries tail the same oplog and stay inside the window.

Healthy replica set: primary appends oplog; both secondaries stay inside the window
Healthy replica set: primary appends oplog; both secondaries stay inside the window · 点击查看 SVG 原图

Broken state after rollover:

Broken state: one secondary outside the oplog window, stuck RECOVERING
Broken state: one secondary outside the oplog window, stuck RECOVERING · 点击查看 SVG 原图

8. What can you do — and what not?

ActionEffect
Keep using the cluster for normal app trafficOK while primary + one secondary stay up. Avoid another member loss.
Pause large bulk upserts until 3/3 are healthyDo this. More fast writes shrink the window and can knock out the remaining secondary.
Test ResilienceDo not run. Wrong tool, and prerequisites fail.
Pause / terminate the clusterDo not use as a repair. Terminate destroys the deployment.
Scale M10 → M20 nowDoes not restore overwritten oplog. Optional only after the set is healthy, as future headroom.
Ask Atlas Support to resync or replace the RECOVERING member and clear the stuck planRequired.

Support request (template)

Subject: Stuck UPDATING — one member RECOVERING and outside the oplog window

Cluster: <project name> / <cluster name>
Tier: M10 dedicated, single-region, MongoDB 8.0, 3-node replica set
State: UPDATING for more than a week.

UI:
- "Deploying your changes failed; we will try again shortly."
- "0 of 3 servers complete (waiting for 3 servers to be healthy, configuring MongoDB)."
- Members: PRIMARY healthy, one SECONDARY lag 0, one member RECOVERING.

Replication:
- Healthy secondary syncedTo "now", lag 0s.
- RECOVERING member syncedTo 2026-10-27 16:30 HKT, about 354 hours behind.
- Sampled oplog window on the healthy primary was about 5–6 days, so incremental catch-up is impossible.
- Disk on a healthy member ~33GB / ~60GB. Not disk-full.
- No OPEN alerts at time of writing.
- Closed alerts: REPLICATION_OPLOG_WINDOW_RUNNING_OUT (Oct 26, Nov 4, Nov 10), HOST_DOWN x3 on Nov 3.

Please:
1. Force a clean initial sync or replace the RECOVERING member.
2. Clear the stuck UPDATING / failed deploy so the cluster returns to IDLE with 3 healthy members.
3. Do not treat an M10→M20 upgrade as the fix for this member. A tier change does not replay overwritten oplog.

We will keep large bulk writers paused until the set is healthy.

9. What to do after Support restores the set?

  1. Confirm API state is idle (not UPDATING) and the UI shows primary + two secondaries, none RECOVERING.
  2. Confirm both secondaries report lag near 0 and syncedTo is current.
  3. Only then rerun bulk jobs, with the pacing in section 10.
  4. Decide separately whether M20 or a larger oplog is worth it for the next multi-million-row job. That is prevention, not the repair for a rolled-off member.

Tier upgrade helps later because:

  • more CPU, memory, and disk IOPS raise secondary apply rate
  • more disk usually means a larger default oplog, so the same lag is less likely to fall off the window

It is not a guarantee. A tight enough loop can still outrun M20. Pacing and lag checks are the actual control.


10. How should bulk writes be paced?

10.1 What w=2 does and does not do

WriteConcern(w=2) means “do not return until two data-bearing members have acknowledged this write” (primary + any one secondary), and wtimeout aborts the wait.

On a 3-node set that is not the same as “wait until every secondary has applied it.”

ConcernProtectsDoes not protect
w=1Primary applied the write. Fastest.Acknowledged data can still exist only on the primary.
w=2 or w=majority (same bar on 3 voting nodes)An ack means two members have it. Survives one node loss.The slower secondary. The fast secondary can satisfy w=2 while the other falls off the oplog.
wtimeoutThe client fails instead of hanging forever.The batch is then not durable at that concern. It must be retried or recorded as failed.

So the snippet that sets w=2 and BATCH_SIZE=2000 with no pause is a durability setting, not a replication-lag brake. If one secondary is healthy, bulk_write returns as soon as that secondary acks, and the sick secondary is ignored.

w=majority is the better default for application writes. Use it for the bulk job too, and add time between batches plus a lag gate.

10.2 Paced, idempotent upserts

Requirements:

  • stable filter (entity_id or another unique business key) so a retry does not duplicate rows
  • ordered=False so one bad document does not block the rest of the batch
  • inspect BulkWriteError; do not treat “no exception” as the only success check if you later split retries
  • pause between batches
  • back off on write-concern timeout instead of immediately sending the next 2,000
  • stop the job if lag or oplog window crosses a limit you cannot wait out

BATCH_SIZE=500 and PAUSE_SECONDS=0.5 are starting points for an M10, not universal constants. Raise batch size only after lag stays near zero. Multi-million-row jobs should be checkpointed so a restart resumes from the last successful key.

import time
from pymongo import MongoClient, UpdateOne
from pymongo.errors import AutoReconnect, BulkWriteError, NetworkTimeout
from pymongo.write_concern import WriteConcern
 
# Start conservative on a small dedicated tier. Increase only while
# secondary lag stays near 0 and the oplog window stays above 24h.
BATCH_SIZE = 500
PAUSE_SECONDS = 0.5
MAX_RETRIES = 5
 
client = MongoClient(
    "mongodb+srv://<user>:<password>@<cluster-host>/?retryWrites=true&w=majority",
    serverSelectionTimeoutMS=10000,
)
collection = client["<database>"].get_collection(
    "<collection>",
    write_concern=WriteConcern(w="majority", wtimeout=10000),
)
 
 
def flush(operations: list) -> None:
    """Write one batch. Back off if the set cannot acknowledge in time."""
    delay = PAUSE_SECONDS
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            collection.bulk_write(operations, ordered=False)
            time.sleep(PAUSE_SECONDS)
            return
        except BulkWriteError as exc:
            # Write-concern timeout: the batch may be on the primary already.
            # Upsert on a stable key makes a retry safe. Other write errors
            # (validation, etc.) should be inspected in exc.details and not
            # blindly retried forever.
            write_concern_error = (exc.details or {}).get("writeConcernError")
            if write_concern_error and attempt < MAX_RETRIES:
                time.sleep(delay)
                delay = min(delay * 2, 30)
                continue
            raise
        except (AutoReconnect, NetworkTimeout):
            if attempt == MAX_RETRIES:
                raise
            time.sleep(delay)
            delay = min(delay * 2, 30)
 
 
def iter_documents():
    """Yield source rows. Checkpoint the caller so a restart can resume."""
    raise NotImplementedError
 
 
pending = []
for item in iter_documents():
    pending.append(
        UpdateOne(
            {"entity_id": item["entity_id"]},
            {"$set": item},
            upsert=True,
        )
    )
    if len(pending) >= BATCH_SIZE:
        flush(pending)
        pending = []
 
if pending:
    flush(pending)
Paced bulk upsert flow: lag gate, majority, backoff, checkpoint
Paced bulk upsert flow: lag gate, majority, backoff, checkpoint · 点击查看 SVG 原图

10.3 Lag gate

The application user often cannot run replSetGetStatus. Two practical options:

  1. Operator check (Atlas → Metrics → Replication Lag and Oplog Window, or a monitoring user):
// on the primary, with a role that can read repl set status
db.adminCommand({ replSetGetStatus: 1 })
// compare each secondary member.optimeDate to the primary
// also: rs.printSecondaryReplicationInfo()
  1. Job rule: before each batch (or every N batches), if max secondary lag is above 30 seconds, or the oplog window is under 24 hours, sleep until it recovers or abort. Do not start a multi-million-row job when any member is RECOVERING.

Suggested budgets for this tier (tune with real lag graphs):

SignalContinueSlow downStop
Max secondary lagunder 5s5–30sover 30s, or any member not SECONDARY/PRIMARY
Oplog windowover 48h24–48hunder 24h
Member state1 primary + 2 secondaries—any RECOVERING / DOWN

10.4 Oplog size

On dedicated tiers, oplog size can be increased in cluster configuration. A larger oplog uses disk and lengthens the time a secondary may be offline before it must initial-sync. It does not make a secondary apply faster. Set it only with free disk headroom (in the teaching case the cluster was ~half full, so there was some room, not unlimited room). Confirm the current Atlas control name before changing it; the default is a percentage of disk.


11. Detection checklist for next time

  1. Overview banner: failed deploy, 0 of N, waiting to be healthy.
  2. Member states: do not treat “all rows yellow” as “all nodes dead” while the cluster is UPDATING.
  3. Replication lag per secondary, not only the primary operations graph.
  4. Oplog window vs the lagging member’s syncedTo. If lag time > window, only initial sync works.
  5. Disk, CPU, and IOPS at the time lag was growing, plus closed REPLICATION_OPLOG_WINDOW_RUNNING_OUT / HOST_DOWN alerts.
  6. Which job was writing (bulk upsert, backfill, migration), its batch size, and whether it slept.
  7. Only then choose: wait, resync via Support, or add capacity for the next run.
Detection tree: serving vs catch-up vs Support resync
Detection tree: serving vs catch-up vs Support resync · 点击查看 SVG 原图

12. What to remember?

  1. A green primary does not mean the replica set is healthy. Check every secondary’s lag and the oplog window.
  2. Falling off the oplog is a one-way door for incremental replication. The fix is initial sync or replacement, which Atlas customers cannot click.
  3. w=majority / w=2 protects acknowledged writes. It does not keep the slowest secondary inside the window.
  4. Bulk upserts need a rate limit: batch size, pause, backoff, and a lag/window gate.
  5. Scaling up reduces how often this happens. It does not repair a member whose resume point is already gone.
  6. While one member is RECOVERING, do not start another full-collection upsert. Protect the secondary that is still caught up.