ETL Cloud Data Engineer

Job Summary:

We are looking for a highly skilled Python / ETL Data Engineer to design, develop, and optimize data pipelines using AWS Glue, AWS Athena, AWS Data Pipeline, and other ETL tools. The ideal candidate will have strong expertise in Python, SQL, AWS, and data pipeline management, ensuring smooth data extraction, transformation, and loading (ETL) processes.

Key Responsibilities:

• Develop and maintain data pipelines using AWS Glue, AWS Athena, AWS Data Pipeline, and other ETL tools.
• Implement monitoring and early detection processes for pipeline issues such as missing or lagging data.
• Extract, transform, and load (ETL) data from various sources, including relational databases, non-relational databases, and flat files.
• Optimize and transform data to meet business requirements and load it into target data stores.
• Troubleshoot and optimize performance of data pipelines for scalability and efficiency.
• Collaborate with Data Scientists and Business Analysts to understand data requirements and implement solutions.
• Build and maintain REST APIs for data access and integration with external systems.
• Ensure data security and compliance best practices across all pipelines.
• Document and test ETL processes to maintain high-quality data flows.

Required Skills:

• Strong Python programming skills for data processing and ETL automation.
• Proficiency in SQL and database design (relational and non-relational databases).
• In-depth knowledge of AWS services, including AWS Glue, Redshift, S3, and EMR.
• Experience in building ETL pipelines on cloud-based environments (preferably AWS).
• Hands-on experience with REST API development and integration.
• Familiarity with data warehousing concepts and best practices.
• Understanding of data modeling, API testing, and troubleshooting.
• Experience with data governance, data security, and compliance frameworks.
• Exposure to big data technologies like Spark, Hadoop, or Kafka.
• Knowledge of containerization (Docker, Kubernetes) and DevOps practices.

Qualifications:

• Minimum 5 years of experience as a Data Engineer with ETL cloud experience, preferably AWS.
• AWS or Azure certification preferred.
• Strong Python programming skills and experience building APIs.
• Malaysian citizen or permanent resident.
• Oil & Gas industry experience is a plus.
• Degree in Computer Science, Engineering, or equivalent.