AI 용어집
인공지능 완전 사전
Apache Hadoop MapReduce
Programming model and distributed implementation for processing large datasets on clusters, dividing processing into Map and Reduce phases. MapReduce is one of the first popular frameworks for large-scale batch processing.
ETL (Extract, Transform, Load)
Data integration process consisting of extracting data from heterogeneous sources, transforming it according to defined business rules, then loading it into a target system. ETL pipelines are typically executed in batch to synchronize data.
Job Scheduling
Automatic scheduling mechanism for batch processing tasks according to predefined schedules, dependencies, or event triggers. Modern schedulers manage parallelization, retries, and execution monitoring.
Shuffling
Costly data redistribution operation between cluster nodes during grouping or aggregation phases in distributed processing. Shuffling often represents the main bottleneck in MapReduce and Spark jobs.
HDFS
Distributed file system designed to store large files on standard machines with fault tolerance through replication. HDFS provides high-performance sequential access suitable for batch processing with MapReduce.
YARN
Resource orchestrator for the Hadoop ecosystem, responsible for allocating CPU, memory, and storage to distributed applications. YARN enables concurrent execution of multiple processing frameworks on the same Hadoop cluster.
Apache Sqoop
Bidirectional data transfer tool between Hadoop and relational databases, optimized for massive parallel imports/exports. Sqoop automatically generates the necessary MapReduce code to efficiently move data.
Apache Hive
Data warehousing infrastructure built on Hadoop providing a SQL-like interface (HQL) for querying large volumes of data stored in HDFS. Hive translates queries into MapReduce jobs for batch execution.