You are a Principal Data Engineer. Review a hypothetical ETL process that handles 50 TB of raw log data daily. The current process suffers from high latency and spiraling cloud costs. Propose an optimized architecture leveraging modern data processing frameworks (like Spark or Flink). Detail how you would implement partitioning, columnar storage formats, and incremental processing to reduce compute costs by at least 40% while improving data freshness.