The platform
The components
The full inventory of an automation platform, grouped by job, with the tool that fills each slot in Datalumina OS.
The architecture page showed how the parts connect. This page lists them. Each row is one slot in the platform, what it is responsible for, and what fills it in Datalumina OS. Treat the right-hand column as one valid answer rather than a requirement. The slot matters more than the vendor, and most of these have three reasonable alternatives.
Intake
- Edge
- Firewall
- API
- Webhooks
- Validation
State
- Database
- Event ledger
Execution
- Queue
- Workers
- Workflow engine
- Scheduling
- Reliability
Integrations
- Service adapters
- Messaging
- Configuration
- Secrets
- AI layer
Operations
- Errors and traces
- Metrics and logs
- Dashboards
- Recovery
Delivery
- Deployment
- Pipeline
- Production controls
Intake
Everything enters here. This is the layer that checks and rejects.
| Component | Responsibility | In Datalumina OS |
|---|---|---|
| Edge | Accept public traffic, terminate TLS, and block what should never reach you. | Cloudflare for DNS and TLS, then Caddy inside the box |
| Firewall | Deny by default, and open only the ports that have a reason to be open. | Hetzner cloud firewall: 80 and 443 public, SSH limited to known IPs |
| IP allow list | Keep private services private, so leaked credentials are useless from elsewhere. | Supabase network restrictions, limited to the server's IP |
| API | Receive events, expose internal endpoints, and return as soon as the work is queued. | FastAPI, with health checks and task streaming |
| Webhooks | One verified endpoint per source system. | Stripe, Polar, Drip, Cal.com, Typefully, Senja, YouTube, Resend |
| Validation | Reject malformed input before it reaches a workflow. | Pydantic schemas on every payload |
State
Two jobs that usually share one database. One holds your business records, the other holds the history of what the system did.
| Component | Responsibility | In Datalumina OS |
|---|---|---|
| Database | Store business records and the data shared across integrations. | PostgreSQL on Supabase, with SQLAlchemy and Alembic |
| Event ledger | Keep the payload, status, attempts, errors, and result of every event. | Postgres event records with duplicate protection |
This is not a data lake. A lake stores data for analytics and machine learning. This platform moves operational work between systems and keeps the state needed to run that work. It can feed a lake, but that is one more connection.
Execution
Where the work actually happens, outside the request cycle.
| Component | Responsibility | In Datalumina OS |
|---|---|---|
| Queue | Buffer work and hand each job to whichever worker is free. | Redis with Celery |
| Workers | Run slow or failure-prone work in the background. | A standard lane plus a dedicated Claude and Codex lane |
| Workflow engine | Compose reusable nodes, route between steps, and pass context along. | Typed events, nodes, routers, and chains across 22 workflows |
| Scheduling | Start recurring work such as syncs, reports, and cleanup. | Celery Beat, currently 17 recurring jobs |
| Reliability | Make sure a job runs once, finishes, or comes back. | Atomic claiming, retries, timeouts, and stale-event recovery |
Integrations
The part that would otherwise get rewritten for every new automation.
| Component | Responsibility | In Datalumina OS |
|---|---|---|
| Service adapters | Hold the reusable client and business logic for each external API. | Stripe, Visma, Close, Airtable, Slack, LinkedIn, GA4, GitHub |
| Messaging | Send transactional email and run sequences without a separate tool. | Preferences, sequences, and scheduled sends |
| Configuration | Define environment-specific behavior without touching workflow code. | Typed Pydantic settings, validated at startup |
| Secrets | Load credentials at runtime, never from the repository. | 1Password references, injected at deploy |
| AI layer | Call a model when a step needs judgment. | Pydantic AI, Claude, Codex, OpenAI, Anthropic, AssemblyAI |
Operations
Automations fail. What separates a platform from a pile of scripts is what happens next.
| Component | Responsibility | In Datalumina OS |
|---|---|---|
| Error tracking and traces | Capture exceptions and inspect a single workflow or model run. | Sentry and Langfuse |
| Metrics and logs | Track system health, worker activity, and queue behavior. | Prometheus, Loki, and Promtail |
| Dashboards | See the state of the system without opening a terminal. | Grafana boards for the API, events, workers, containers, server, and alerts |
| Recovery | Inspect events, retry failures, and rescue work that got stuck. | Internal dashboard with retry controls |
| Engineering support | Make the platform safe to change. | Schemas, tests, replay tools, and runbooks |
Delivery
The same stack on your laptop and on the server, shipped the same way every time.
| Component | Responsibility | In Datalumina OS |
|---|---|---|
| Deployment | Package each service and run an identical stack locally and in production. | Docker Compose on Hetzner |
| Pipeline | Test changes and deploy approved versions repeatably. | GitHub Actions with a self-hosted runner on the server |
| Production controls | Keep a bad deploy from becoming an outage. | Automatic migrations, restart policies, and Slack notifications |