Lead Data Engineer

Responsibilities:

• Enhance existing data pipelines using industry-standard tools and practices.
• Redesign and implement pipelines leveraging AWS managed components.
• Set up monitoring systems to proactively detect and alert on pipeline issues.
• Investigate and resolve pipeline-related issues.
• Document solutions and operational guidelines for data pipelines.
• Develop unit and integration testing for pipeline components.

Requirements:

• Minimum 6 years of hands-on experience in data engineering, particularly designing and building ETL pipelines.
• Strong background in project execution and technical expertise.
• Proficiency in Python.
• Solid knowledge of SQL.
• Experience with shell scripting (Linux).
• Familiarity with orchestration tools such as Airflow or similar.
• Experience with AWS services including S3, Glue, EMR, Lambda, and Step Functions.
• Exposure to AWS Batch, Athena, file formats like Parquet, NoSQL databases, and tools like AWS OpenSearch.
• Ability to work onsite when requested by the project team.