Sr. Data Engineer

Job Summary

The Senior Data Engineer is responsible for designing, building, and managing scalable data pipelines and data platform solutions within the Enterprise Data Platform (EDP) using Azure and Databricks. The role focuses on enabling reliable, governed, and high-performance data integration and transformation across multiple source systems to support analytics, reporting, and advanced data use cases.

Duties and Responsibilities

This position combines hands-on data engineering, platform optimization, and stakeholder coordination, ensuring the delivery of robust, secure, and business-aligned data solutions while supporting data governance and best practices.

Data Engineering & Pipeline Development

              Design, develop, and maintain scalable data pipelines using Azure Databricks, Azure Data Factory, and related services

              Implement ETL/ELT processes to ingest, transform, and load data into the Enterprise Data Platform

              Develop batch and near real-time data processing solutions

              Apply medallion architecture (bronze, silver, gold layers) for structured data processing

              Ensure data quality, consistency, and integrity across pipelines and datasets

Platform Operations & Optimization

              Monitor, maintain, and optimize data pipelines, workflows, and jobs in Databricks and Azure

              Troubleshoot pipeline failures, performance issues, and data inconsistencies

              Optimize Spark jobs, query performance, and storage utilization (e.g., Delta Lake optimization)

              Ensure high availability, scalability, and reliability of data services

              Maintain documentation and ensure compliance with data governance, security, and operational standards

Team Management

              Provide guidance to data engineers and developers on best practices in Azure and Databricks

              Review code, pipelines, and technical designs to ensure quality and standards compliance

              Support the team in resolving technical issues and improving delivery efficiency

              Promote reusable frameworks, templates, and automation practices

              Mentor team members on data engineering concepts and cloud-based architectures

Stakeholder Engagement

              Collaborate with business units, analytics teams, and platform engineers to gather and validate data requirements

              Translate business needs into scalable data models and pipelines

              Act as liaison between source system owners, data consumers, and platform teams

              Communicate pipeline status, issues, and improvements to stakeholders

              Support analytics and BI teams by ensuring availability of clean and reliable datasets

Technical Competencies and Skills

Data Engineering & Databricks, Azure Data Platform, Data Modeling & Performance Optimization, Data Governance & Security, DevOps & Automation

Education, Trainings and Licenses Required

Bachelor's degree in Computer Science, Information Technology