Senior Data Engineer

Job Purpose

As a Senior Data Engineer, you will lead the design and optimization of scalable data pipelines and modern data architectures to support business objectives. You will drive engineering best practices, mentor peers, and embed automation for testing and monitoring. Collaborating with cross-functional teams, you will translate business needs into reusable, cost-effective data products. You will stay abreast of emerging technologies to ensure the team remains at the forefront of the evolving data landscape.

Key Responsibilities

Data Pipeline Architecture and Implementation

• 

  

  Design and implement scalable data pipelines for batch and real-time workloads.
• Build robust, efficient, and maintainable pipelines using modern frameworks (e.g., Spark, Delta Lake, dbt, Airflow).

Design and Maintenance of Data Lakehouse Solutions

• Architect and maintain data lakehouse and data warehouse solutions.
• Develop well-structured, performant data models using dimensional/star schema design (Kimball methodology).

Optimization of Data Infrastructure

• Drive performance tuning of Spark jobs and SQL.
• Monitor and optimize storage and computing costs with workload profiling and KPI dashboards.

Engineering Standards and Mentorship

• Promote best engineering practices and lead code reviews.
• Mentor junior engineers and facilitate knowledge-sharing sessions.

Data Quality Assurance

• Implement automated testing, monitoring, and validation frameworks.
• Establish data validation layers and integration tests to ensure reliability.

Technology Exploration

• Evaluate emerging technologies (e.g., Iceberg, new ingestion tools).
• Recommend and integrate new tools where appropriate.

Cross-Functional Collaboration

• Work closely with BI, Data Science, and Business teams to align solutions with business KPIs and requirements.

Shared Team Responsibilities

• Develop reusable components, naming conventions, and documentation standards.
• Conduct regular design reviews and peer code reviews.
• Implement CI/CD pipelines for data jobs.
• Maintain shared knowledge bases and facilitate onboarding and continuous learning.

Key Skills & Experience

Education:

• Bachelors degree in Computer Science, Engineering, or similar.

• (Preferred) Masters degree in a related field.

Experience:

• 5+ years in data engineering or related fields.

Technical Skills:

• Data Factory, Python, Git, Databricks.

• SQL Server, Azure Platform.

• Power BI (data modeling and DAX development).

• Linux OS concepts.

• Data modeling and ETL (Kimball concepts).

Working Environment

• Office hours, multi-tasking environment.

• Interaction with stakeholders at various levels across teams.

• Use of modern data engineering tools and cloud services.