Hong Kong, Hong Kong SAR, Hong Kong
Data Engineer
Job Description:
About the role:
- You are pioneering and innovative and want to be part of the cutting-edge and disruptive crypto-currency world
- You are eager to learn new knowledge in both financial and technical fields
- You thrive in a non-hierarchical organization with a casual working environment
- You enjoy solving complex distributed systems challenges and optimizing streaming data pipelines
- You value comprehensive documentation and collaborative problem-solving
As a Data Engineer you will:
- Build real-time data pipelines using Apache Flink (PyFlink) to process high-volume logs through EC2 Vector MSK PyFlink S3 ClickHouse architecture
- Design stream processing systems with watermark strategies, window operations, exactly-once semantics, and state management for critical data
- Deploy and manage AWS infrastructure including Managed Flink, S3 data lakes, MSK, IAM, and CloudWatch monitoring with optimized partitioning strategies
- Optimize performance and troubleshoot SQL queries (Flink/ClickHouse), production issues, data skew, and build Grafana dashboards for pipeline monitoring
- Implement data quality frameworks with schema evolution, validation strategies, and translate business requirements into scalable data solutions
- Manage DevOps processes including JAR dependencies, Docker containerization, Kubernetes deployments, and comprehensive documentation
Qualifications:
- University degree in Computer Science, Software Engineering or related disciplines
- Apache Flink/PyFlink with watermarks, state management, and window operations
- Python (pandas, polars, boto3) and SQL (Flink SQL, ClickHouse)
- Kafka/AWS MSK and message streaming concepts
- AWS services: Managed Flink, S3, MSK, CloudWatch, IAM
- Docker, Kubernetes, and containerization fundamentals
- Grafana monitoring and production troubleshooting experience
Required Skills:
Data