SOHAIB®

Available immediately

← Selected systems

AI evaluation · Terminal benchmark engineering

Engineering evaluations models cannot bluff.

I design execution-based evaluations for terminal-based coding agents—real engineering environments where success is decided by a deterministic grader, not by how convincing the answer sounds.

See selected work Discuss an evaluation
Measure
Execution, not eloquence
Environment
Terminal + real system state
Defense
Hidden cases + mutations
Output
Reproducible capability signal

01 / The standard

Did the agent solve the engineering problem?

Generated code can look excellent while restoring the wrong database state, leaking future data, trusting a broken journal, or hardcoding one visible fixture.

My work turns those distinctions into measurable outcomes. The benchmark owns the environment, the protected truth, and the scoring boundary. The model owns only its investigation and repair.

02 / Benchmark construction

From realistic failure to defensible signal.

Hardness is designed. Correctness is proved. Each layer exists to remove an easier explanation for success.

  1. 01

    Construct

    Anchor the task in a real engineering failure with enough ambiguity to require investigation—not transcription.

  2. 02

    Force

    Use cost models, baselines, and ablations to prove the intended reasoning is necessary before building the grader.

  3. 03

    Isolate

    Package dependencies, fixtures, state, and runtime boundaries so every agent receives the same problem.

  4. 04

    Verify

    Score observable behavior, persistent state, artifacts, and invariants rather than plausible-looking code.

  5. 05

    Attack

    Pressure-test shortcuts with mutations, hidden cases, no-op controls, path attacks, and regression checks.

  6. 06

    Analyze

    Review trajectories to find wrong assumptions, missed signals, shallow fixes, and weak self-verification.

03 / Selected evaluations

Problems that resist the first plausible answer.

Representative work across storage, systems, ML infrastructure, temporal data, and release engineering.

01

Storage systems · Crash recovery

Coherent—but stale.

A shadow-paged embedded key-value store recovered into a structurally valid database that represented the wrong transactional moment. Ordinary corruption checks passed; only reasoning about durability and logical state exposed the failure.

  • transaction durability
  • recovery semantics
  • persistent state
  • logical invariants

EvidenceIndependent model trajectories reproduced the failure. Submission passed external technical review.

02

Systems · Emulator repair

Three semantics, one correct repair.

A terminal benchmark hid interacting defects across DMA grant width, rotating interrupt priority, and axis modulus behavior. Visible evidence permitted a tempting partial repair; held-out cartridges forced the complete system model.

  • unfamiliar code
  • stateful execution
  • held-out behavior
  • sandbox boundaries

EvidenceFive partial baselines rejected. Mutation sweep caught 70/70 incorrect variants.

03

Machine learning · Feature engineering

Future data must stay in the future.

A fraud-feature task tested whether agents could derive historical signals without leaking future transactions. The verifier evolved to challenge global-statistic shortcuts, cross-run state, over-wide outputs, and solutions that looked predictive for the wrong reason.

  • temporal leakage
  • future invariance
  • AUC calibration
  • stateful shortcuts

EvidenceAccepted after verifier hardening, adversarial replay, and genuine difficulty calibration.

04

ML infrastructure · Recovery

Recover the last usable global commit.

A distributed checkpoint resharding task required validating coordinator and rank journals, transaction identity, epochs, logical ownership, physical ranges, aliases, and block hashes before selecting a recoverable commit.

  • transactional recovery
  • tensor topology
  • binary integrity
  • cross-rank consistency

EvidenceAccepted through pass, adversarial, quality, trial, and final evaluation gates.

05

Build & release · Reproducibility

Byte-for-byte means every byte.

A deterministic release-rebuild task covered timestamp clamping, UTF-8 path ordering, executable-mode normalization, archive ownership, symlinks, empty directories, and gzip metadata under a pinned Linux environment.

  • reproducible builds
  • filesystem metadata
  • path ordering
  • cross-platform parity

EvidenceVerifier review exposed missing discriminators, turning a green-looking result into a sharper coverage model.

04 / Deterministic grading

How correctness gets proved.

A grader should fail for the same reason the production system would fail—not because a solution differs stylistically from a reference patch.

  1. 01

    Behavior

    APIs, files, processes, databases, and state transitions must match the contract.

  2. 02

    Protected truth

    Held-out fixtures and expected state remain outside the submitted program's trust boundary.

  3. 03

    Isolation

    Unprivileged execution, path containment, controlled scratch space, and immutable evaluator assets.

  4. 04

    Controls

    A correct oracle must pass; no-op, partial, malicious, and hardcoded solutions must fail.

  5. 05

    Mutations

    Deliberately wrong implementations prove each important requirement changes the score.

  6. 06

    Reproducibility

    Pinned dependencies and architecture-aware runs separate platform drift from task evidence.

05 / Trajectory analysis

What model failures reveal.

A useful miss diagnoses capability. It shows where investigation, system modeling, or verification stopped being reliable.

01

Surface repair

The symptom disappears, but the model never discovers the state transition or invariant that caused it.

02

Weak verification

A patch is written and a convenient test passes; the original failure is never recreated end to end.

03

Sticky system model

An early architectural assumption survives even after logs, artifacts, or runtime behavior contradict it.

04

Visible-case optimization

The implementation fits public examples while violating the broader invariant those examples represent.

05

Over-editing

The agent changes more than the responsible layer, increasing regression risk and hiding the causal fix.

06 / Engineering context

Production engineering underneath the benchmark.

Five years across APIs, databases, infrastructure, asynchronous workflows, and AI applications make the tasks feel like engineering—not puzzles wearing code.

AI training & evaluation

Handshake AI

Software-engineering tasks · automated grading · model behavior

Agentic benchmark evaluation

Mindrift

Task construction · reproducible environment · multi-model failure analysis

Senior software engineering

Production systems

Backend architecture · databases · Docker · Linux · integrations
PythonFastAPILinuxDockerDatabasesOpenID FederationMachine learningMixed-CRS geospatial ETLBuild systemsReact / Next.jsAgentic AITool calling

AI labs · Evaluation teams · Technical platforms

Build an evaluation that measures the real thing.

Available for benchmark authoring, coding-agent evaluation, grader design, trajectory analysis, expert review, and AI engineering.

Discuss an evaluation

Continue exploring

Selected systems Contact