The research behind Lumis SDK
Lumis SDK is the open-source implementation companion to a peer-oriented research paper on agentic self-healing pipelines.
The paper
Much of this framework is the open-source implementation companion to "Agentic Self-Healing for Data & AI Pipelines: An Affordable Vendor-Agnostic Architecture using Open-Source Software" by Solomon Eshun, Dennis Murage, Sharleen Muoki, Chih-Chun Chen, Stephen Adjignon, Matteo Staar, and Oliver Angélil.
The paper compares existing ZeroOps, observability, and AIOps platforms for pipeline monitoring, root-cause analysis, and automated remediation, and finds the main gap is architectural rather than technological: the ingredients for self-healing pipelines already exist but are fragmented across vendor-specific platforms. It proposes an affordable, vendor-agnostic reference architecture — agentic recovery and incident response — combining monitoring, pipeline metadata, incident history, deterministic policy checks, AI-assisted diagnosis, approval workflows, and controlled remediation.
Read the published paper on arXiv ↗ or download the local PDF.
The two learning loops
The first loop writes every verified episode—symptoms, diagnosis, action, approval, outcome, verification, and final resolution—back into incident memory with provenance. Future incidents can retrieve this accumulated knowledge to ground diagnosis.
The second loop promotes recurring, verified diagnosis-and-remediation patterns from Layer 7 into candidate deterministic rules in Layer 4. In practice this should be a governed pipeline: configurable recurrence threshold, equivalence checks, provenance, human review, replay tests, and a version-controlled merge. Once approved, the known incident takes the deterministic path before model reasoning.
From paper to framework
| Paper concept | Where it lives in the SDK |
|---|---|
| Detect / triage / diagnose / plan / approve / remediate / verify / learn lifecycle | The guarded lifecycle contracts and run_guarded_lifecycle orchestrator. |
| Deterministic policy checks before AI | The legacy text and structured deterministic rule engines. |
| Bounded, provenance-carrying context | EvidenceService, the EvidenceProvider port, and redaction. |
| AI-assisted diagnosis under budgets | The optional ModelGateway behind an explicit ModelUsePolicy. |
| Approval workflows and controlled remediation | Playbooks, default-deny policy, evidence-linked proposals, and idempotent approvals—no core executor. |
| Incident history and organizational learning | Operational memory with explicit truth states, verification-aware learning, and governed deterministic-rule candidates. |
Authorship and maintenance
Lumis SDK is maintained by Solomon Eshun — the paper's lead author — together with the open-source contributors who join the project. The paper describes the technology-flexible reference architecture; the SDK is its Apache-2.0 Python implementation companion, and the two evolve together.