Beyond ETL: Building Modern, Agentic Data Pipelines

Traditional ETL (Extract, Transform, Load) processes have been the backbone of data warehousing for decades. They are reliable workhorses for moving structured data from point A to point B. But in today's fast-paced, API-driven world, the very definition of "data processing" has expanded far beyond the classic ETL script.

Modern applications demand more. We need to handle unstructured data, orchestrate complex sequences of microservice calls, run computationally intensive tasks, and enrich data in real-time from a dozen different sources. The rigid, batch-oriented nature of traditional ETL pipelines begins to crack under this pressure. They become brittle, hard to scale, and a nightmare to maintain.

What if we could move beyond ETL? What if we could treat any complex business process—from data transformation to service orchestration—as a single, scalable, on-demand task? This is the promise of agentic workflows, a new paradigm for building intelligent, developer-friendly data and processing pipelines.

The Cracks in Traditional Processing

If you've ever managed a complex data pipeline or background job system, these pain points might feel familiar:

• Operational Overhead: You're stuck managing and scaling server clusters, job queues, and monitoring tools. Your focus is more on infrastructure than on the business logic itself.
• Rigid Logic: A change in one part of a monolithic ETL script can have cascading, unintended consequences. Adapting to new data sources or business rules is slow and risky.
• Limited Scope: Traditional pipelines are good at moving data but struggle with more complex, stateful operations like orchestrating a sequence of API calls with retries and conditional logic.
• "It works on my machine": The logic is often tightly coupled to a specific environment, making it difficult to test, deploy, and scale reliably.

The core issue is that these systems weren't designed for a world where your business logic is an asset you want to execute flexibly, at scale, via a simple API.

Enter the Agentic Workflow: Intelligent Processing as Code

An agentic workflow reframes the problem. Instead of a single, monolithic script, you break your process down into smaller, independent, and intelligent "agents."

• An Agent is a self-contained unit of business logic, packaged as a container. It does one thing well—like enriching a user profile from an API, resizing an image, or running a financial calculation.
• A Workflow is the orchestrator. It defines how these agents are connected, how data flows between them, and how to handle errors, retries, and parallel execution.

This approach, powered by a platform like processing.services.do, allows you to define your most complex business logic as code and execute it with a simple API call.

Intelligent Processing On-Demand with processing.services.do

processing.services.do is an agentic platform designed to run your custom logic at scale. You define the "what" (your agents), and we handle the "how" (scalability, reliability, and orchestration).

Let's see how this transforms a complex task like data enrichment into a trivial API call. Instead of writing custom queue workers and orchestration logic, you simply run a pre-defined workflow.

import { Do } from '@do-sdk/core';

// Initialize the client
const processing = new Do('processing.services.do', {
  apiKey: 'your-api-key',
});

// Define and run a data enrichment workflow
const job = await processing.run({
  workflow: 'enrich-user-profile',
  payload: {
    userId: 'usr_12345',
    sources: ['clearbit', 'linkedin', 'internal_db'],
  },
  config: {
    priority: 'high',
    // Get notified when the long-running job is done
    onComplete: 'https://myservice.com/webhook/job-done',
  }
});

console.log(`Job started with ID: ${job.id}`);

Let's break down what's happening here:

1. Simple Invocation: We're triggering a complex process named enrich-user-profile with a simple processing.run() call.
2. Abstracted Complexity: The developer making this call doesn't need to know how the enrichment happens. The workflow could be calling three different agents in parallel, combining the results, and storing them in a database. That logic is encapsulated on the platform.
3. Built-in Asynchronicity: By providing an onComplete webhook, we've effortlessly handled a potentially long-running task. The platform will manage the job's lifecycle and notify our service when the results are ready, enabling robust, event-driven architectures.
4. Infinite Scalability: If you call this function once or a million times, processing.services.do automatically scales the underlying compute resources to meet the demand, ensuring consistent performance without any manual intervention.

From ETL to "Process Anything"

The agentic model unlocks use cases that are simply not practical with traditional tools.

• Microservice Orchestration: A "new user signup" workflow can be an agentic process. It orchestrates calls to your user service, billing provider, email service, and a Slack notification agent, with built-in retries and state management.
• Media Processing: Trigger a workflow to generate multiple watermarked video renditions in different resolutions for every new video upload. The platform parallelizes the work across hundreds or thousands of agents.
• Batch Financial Calculations: Run complex, multi-step Monte Carlo simulations as a single job, pulling data from multiple sources and delivering a final report to storage upon completion.

The paradigm shifts from "data pipeline" to "processing as a service." If you can containerize your logic, you can execute it at scale.

The Future is Agentic

Stop wrestling with brittle scripts and complex infrastructure. Modern applications require a modern approach to background processing—one that is developer-centric, infinitely scalable, and flexible enough to handle any business logic you throw at it.

By embracing agentic workflows with a platform like processing.services.do, you can transform your most complex processing challenges into simple, maintainable, and powerful API calls.

Ready to build your first agentic workflow? Discover processing.services.do and start processing anything, instantly.

Frequently Asked Questions

Q: What kind of processing can I perform with processing.services.do?

A: You can run virtually any custom logic. Common use cases include data transformation (ETL), batch processing, image/video rendering, financial calculations, and orchestrating sequences of microservice calls. If you can code it, we can process it.

Q: How do I define my processing logic?

A: You define your business logic as containerized agents. processing.services.do acts as the orchestrator, invoking your agents with the provided payload and managing the execution state, scalability, and error handling for you.

Q: Is the processing service scalable?

A: Yes. Our platform is engineered for high-throughput, parallel processing. It automatically scales compute resources based on your workload, ensuring your jobs are completed efficiently, whether you're running one task or a million.

Q: Can I run long-running tasks?

A: Absolutely. The platform is designed for both synchronous (quick) and asynchronous (long-running) jobs. For long jobs, you can provide a webhook URL to be notified upon completion, allowing you to build robust, event-driven systems.