大数据处理 - Big Data Processing

课程介绍 Course Introduction

学分：3 | 先修课：数据结构与算法、数据库系统 | 学期：第6学期

大数据处理课程聚焦海量数据的存储、计算与分析技术，主要内容涵盖Hadoop生态系统、MapReduce编程模型、Spark内存计算、分布式文件系统HDFS、NoSQL数据库与流处理框架Kafka、Flink等。课程强调分布式计算原理、数据分区与容错机制，训练学生设计可扩展的数据处理管道，应对TB至PB级别的数据挑战。

Big Data Processing focuses on storage, computation, and analysis of massive datasets. Topics cover the Hadoop ecosystem, the MapReduce programming model, Spark in-memory computing, the HDFS distributed file system, NoSQL databases, and stream processing frameworks such as Kafka and Flink. The course emphasizes distributed computing principles, data partitioning, and fault tolerance, training students to design scalable data processing pipelines for TB to PB-scale data challenges.

大作业 Final Project

作业标题：分布式日志分析系统

基于Hadoop与Spark搭建日志采集、清洗、分析与可视化的完整管道，处理GB级真实日志数据并生成统计报告，对比批处理与流处理的性能差异。

Build a complete log collection, cleaning, analysis, and visualization pipeline using Hadoop and Spark, process GB-scale real log data, generate statistical reports, and compare performance differences between batch and stream processing.