KI-Glossar

📖

Begriffe

Apache Hadoop MapReduce

Programming model and distributed implementation for processing large datasets on clusters, dividing processing into Map and Reduce phases. MapReduce is one of the first popular frameworks for large-scale batch processing.

📖

Begriffe

ETL (Extract, Transform, Load)

Data integration process consisting of extracting data from heterogeneous sources, transforming it according to defined business rules, then loading it into a target system. ETL pipelines are typically executed in batch to synchronize data.

📖

Begriffe

Job Scheduling

Automatic scheduling mechanism for batch processing tasks according to predefined schedules, dependencies, or event triggers. Modern schedulers manage parallelization, retries, and execution monitoring.

📖

Begriffe

Shuffling

Costly data redistribution operation between cluster nodes during grouping or aggregation phases in distributed processing. Shuffling often represents the main bottleneck in MapReduce and Spark jobs.

📖

Begriffe

HDFS

Distributed file system designed to store large files on standard machines with fault tolerance through replication. HDFS provides high-performance sequential access suitable for batch processing with MapReduce.

📖

Begriffe

YARN

Resource orchestrator for the Hadoop ecosystem, responsible for allocating CPU, memory, and storage to distributed applications. YARN enables concurrent execution of multiple processing frameworks on the same Hadoop cluster.

📖

Begriffe

Apache Sqoop

Bidirectional data transfer tool between Hadoop and relational databases, optimized for massive parallel imports/exports. Sqoop automatically generates the necessary MapReduce code to efficiently move data.

📖

Begriffe

Apache Hive

Data warehousing infrastructure built on Hadoop providing a SQL-like interface (HQL) for querying large volumes of data stored in HDFS. Hive translates queries into MapReduce jobs for batch execution.