Data pipelines are the plumbing of the modern data stack. They extract data from source systems (databases, APIs, SaaS applications, event streams), apply transformations (cleaning, filtering, aggregating, joining), and load the results into destination systems (data warehouses, lakes, analytics tools, AI training environments).
Pipelines range from simple ETL (Extract, Transform, Load) processes that move data on a schedule, to complex real-time streaming architectures that process millions of events per second. The choice of architecture depends on the latency requirements of the use case: batch pipelines are sufficient for nightly reporting; real-time pipelines are required for live personalization, fraud detection, or consent enforcement.
For data governance, pipelines represent one of the most significant compliance risk surfaces. Data flowing through a pipeline may change form, be joined with other datasets, be replicated across environments, and end up in systems far removed from the original collection point. Without governance controls embedded in the pipeline, including consent signal propagation and data classification tagging, pipelines can silently violate privacy and data use obligations at scale.