Senior Big Data Engineer
Responsibility:
● Designing and build scalable data pipelines to extract, transform, and load data from a variety of sources with specific time latency.
● Maintaining and optimizing existing data pipelines and automate data workflows such as data ingestion, aggregation, and ETL processing.
● Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
● Design and implement reliable, scalable, robust and extensible big data systems that support core products and business (e.g. growth analysis, money-laundering analysis and multi-dimensional analysis)
● Develop and implement techniques and analytics applications to transform raw data into meaningful information using programming languages and visualization software.
● Research and evaluate latest data related technologies and tools to keep our data platform updated and competitive
Requirements:
● Proficiency in developing robust data pipelines, including data collection and ETL(Extract, Transform, Load) processes and architecting data systems
● Proficiency in at least one programming languages such as Python, Java, or Scala.
● At least 3 year of experience in the Big Data technologies(Hadoop, M/R, Hive, Spark/PySpark, Presto, Flume, Kafka, Flink etc.)
● At least 1 year of experience in the AWS platform services (Glue, Athena, EMR, Redshift etc.)
● Experience designing and implementing various components of a data platform, including data ingestion, storage, data warehousing, data orchestration
● Experience in writing, analyzing and debugging SQL queries
● Passionate and deep understanding about technologies in the Data area
● Solid communication and collaboration skills
