Skip to contentAbdalla Emad Eldin— Dubai

Failure log

Every engineer has a list like this. Most keep it private, which is why interviews are full of projects that apparently went perfectly. These are real defects in my own work, what caused them, and the structural change that makes each one unable to happen again — not the promise to be more careful, which is not a fix.

  1. A backup check that could not fail

    Steadhold
    What broke
    The restore verification guarded a step by listing a file path with ls and matching a substring of the output. It passed every time, including when the file was missing.
    Why
    ls prints the path inside its own error message, so the substring matched whether or not the file existed. The check was structurally incapable of returning false.
    The fix
    Every verification step now has a test that deliberately breaks its input and asserts the step rejects it. A check that has never been observed failing is not treated as a check.
    Why it cannot recur
    The corruption test runs in CI. If a verifier stops being able to fail, the suite goes red.

  2. Two bugs that reported healthy databases as corrupt

    Steadhold
    What broke
    Backup verification failed on clusters that were fine. Twice, from different causes: a bash-ism passed to sh -c, and a flag that pg_amcheck does not accept.
    Why
    Both errors produced a non-zero exit code, and the verifier treated any non-zero exit as corruption. It could not tell “this database is damaged” from “I ran the command wrong”.
    The fix
    Tool failure and data failure are now separate outcomes. A command that cannot run raises an operational alert; only a check that ran and rejected the data reports corruption, and the report names which check caught it.
    Why it cannot recur
    A false corruption report is louder than a real one: it destroys trust in the only signal that says the backups are worth having.

  3. A restore that came back one second short

    Steadhold
    What broke
    Point-in-time restore formatted its target timestamp by trimming the milliseconds, which moved the requested instant backwards by up to a second.
    Why
    The restore still succeeded and still looked correct, so nothing failed. A customer restoring to the moment before a bad migration would have silently lost that second's writes, with no way to notice.
    The fix
    The timestamp is passed at full precision, and the restore test asserts on both sides of the boundary: a row written before the target must be present, and a row written after it must be absent.
    Why it cannot recur
    Correct-looking output is the most expensive kind of wrong. Any test for a boundary now asserts what must be missing, not only what must be there.

  4. Pausing a project quietly aged its only backup

    Steadhold
    What broke
    Pausing stopped the containers. Write-ahead log archiving stops with the container, so a paused project's most recent copy was whatever had been archived before the pause.
    Why
    Pausing reads as a safe, reversible action, so it was not treated as a data-durability event. For a free-tier project paused by the idle detector, nobody was watching at all.
    The fix
    Pause takes a backup after the checkpoint, confirms it landed, and refuses to stop the containers if either step fails. A paused project whose backup failed is worse than a running one, which is the opposite of the point of pausing.
    Why it cannot recur
    Every state transition that stops a process is now classified by what it does to durability, not by how reversible it looks.

  5. Deleted projects kept their backups forever

    Steadhold
    What broke
    Purging a project destroyed its volume and its rows, but the backup repository in object storage was never touched. Data a customer had asked to destroy stayed retained indefinitely, with nothing recording that it should not be.
    Why
    By purge time there is no container left to run the backup tool inside, and the cleanup had been written as something the node does.
    The fix
    The control plane deletes the storage prefix itself, then lists that prefix again and only records the destruction when the listing comes back empty.
    Why it cannot recur
    Deleting and assuming leaves objects retained forever behind a row that asserts they are gone. Deletion is not done until it has been read back.

  6. A repair job destroyed the data it was meant to fix

    Job pipeline
    What broke
    A cleanup job re-derived salary fields by parsing the text of each listing. Where the text was vague it overwrote structured salary values that had been captured correctly at the source.
    Why
    The job treated the rendered text as the source of truth when the structured field was the source of truth. A repair that reads a worse representation than the one it is repairing can only lose information.
    The fix
    Broken data is re-derived from its origin, never from a downstream rendering of itself, and a repair that would replace a structured value with a less certain one leaves the record alone and flags it instead.
    Why it cannot recur
    Repairs are reviewed like migrations: what is the source, what is being overwritten, and what happens to the rows where the source is ambiguous.

  7. A typographic ligature made a CV unreadable to software

    CV toolchain
    What broke
    The PDF renderer joined “fi” into a single glyph, so text extraction returned words like “first” and “fintech”. Applicant tracking systems read the file as gibberish in exactly the words that mattered.
    Why
    The document looked perfect to a human reader, and the defect only existed in the extracted text layer — which is the only layer the first reader, a machine, ever sees.
    The fix
    Ligatures are disabled in the print stylesheet, and the build now extracts the text back out of the finished PDF and checks it, rather than trusting how the page looks.
    Why it cannot recur
    Any artefact with a machine reader is now checked the way that reader will read it.

  8. A review gate caught two defects in the code written to satisfy it

    Steadhold
    What broke
    The first version of the dashboard was correct in every colour and wrong in every mechanic. Writing an interaction standard and running it as a twenty-question gate on every UI change found, on its first run, a command palette missing two actions the row menu already had, and a project list printing “Showing 20 of 20” while a second page existed.
    Why
    Interface quality decays silently, because nothing fails when a keyboard path is missing or a count is wrong. Without a written standard there is nothing for a review to be against.
    The fix
    The standard is a document, and the gate runs on every UI change rather than at the end. Three rules that decay quietly — no stylesheet naming a ramp step, no drop shadows, no reference to a token that does not exist — are enforced by tests.
    Why it cannot recur
    A gate that finds nothing on its first run is usually a gate that cannot find anything.