My client is seeking a Senior Data Engineer to join their dynamic team. This role is crucial for enhancing their data infrastructure and analytics capabilities in a fast-paced and rapidly evolving environment. The ideal candidate will have 4+ years of hands-on experience and a strong command of data engineering principles and technologies.

Key Responsibilities

• Design, develop, and maintain robust data pipelines for real-time analytics and machine learning applications.
• Implement and optimize stream processing solutions using technologies such as Apache Flink or Spark.
• Work collaboratively with quants, data scientists and analysts to transform and validate datasets.
• Ensure efficient data storage and retrieval using AWS services (S3, ClickHouse) and SQL solutions.

Core Technical Skills

• Stream Processing: Expertise in Flink, Spark Streaming, PyFlink, Kafka, and AWS MSK.
• Programming: Proficient in Python (including libraries like Pandas, Polars, boto3, and pyarrow) and SQL (Flink/Spark SQL).
• Data Storage: Experience with S3, ClickHouse, Parquet, and JSON.
• Monitoring/Logging: Familiarity with Grafana for data visualization and Fluentd for logging.

Cloud & Infrastructure

• Proficient in AWS services such as EMR and Kinesis Data Analytics.
• Strong knowledge of containerization technologies (Docker, Kubernetes) and orchestration frameworks (Airflow, YARN).
• Familiarity with Infrastructure as Code (IaC) tools such as Terraform and Ansible is a plus.
• Understanding CI/CD practices for DevOps integration.

Qualifications

• Bachelors degree in Computer Science, Engineering, or related field (Master's preferred).
• 4+ years of experience as a data engineer or in a similar role.
• Strong problem-solving skills and ability to work under pressure in a fast-paced environment.