Python Data Engineer - 8+ Years - Chennai - Immediate Joiners

Job Brief

We are looking for a skilled Python Data Engineer with expertise in data processing, ETL pipeline development, and big data technologies.

Responsibilities

• Design and build scalable ETL pipelines to ingest, transform, and process large datasets.
• Develop and optimize Python-based data processing scripts for batch and real-time workflows.
• Work with relational (SQL) and NoSQL databases for efficient data storage and retrieval.
• Utilize big data technologies like Apache Spark, PySpark, and Hadoop for distributed data processing.
• Implement data validation, cleansing, and quality checks to ensure accuracy and consistency.
• Work with cloud-based data services (Azure, AWS, or GCP) for scalable data solutions.
• Optimize query performance and data transformation logic for better efficiency.
• Collaborate with data scientists, analysts, and software engineers to enable data-driven insights.
• Implement best practices for data governance, security, and compliance.
• Monitor and troubleshoot ETL workflows, job failures, and data integrity issues.

Requirements

• 5+ years of experience in Data Engineering with strong expertise in Python.
• Proficiency in SQL (query optimization, indexing, partitioning, stored procedures).
• Experience with Apache Spark / PySpark for big data processing.
• Strong understanding of ETL development, data warehousing, and pipeline orchestration.
• Hands-on experience with cloud data services (AWS Redshift, Azure Data Factory, GCP BigQuery, etc.).
• Experience with workflow automation tools (Apache Airflow, Prefect, or Luigi).
• Knowledge of data modeling, schema design, and data lakes.
• Familiarity with Docker, Kubernetes, and CI/CD pipelines for deployment.