Job Openings Technical Lead, Data (Python/Spark)
About the job Technical Lead, Data (Python/Spark)
Note: This is a Remote position but only for candidates who are residing in Canada.
- Build and manage the data processing pipeline
- Ensure reliable processing of data, keeping error counts in check, add data validation checks, and ensure performance of data pipeline is good.
- Responsible for building the infrastructure for running the data pipeline
- Work with data ingestion and other feature teams to ensure the data pipeline runs smoothly.
Experience and Skills
- You are currently a technical lead managing a very large scale data processing pipeline.
- You are a hands-on technical leader
- You have 12+ years of professional software development experience
- You have 6+ years of experience with developing data applications in Python.
- You have a deep knowledge of building, scaling and managing a large scale real-time data pipeline (processing 100s of millions of events per day).
- You have a deep knowledge and experience working with two or more of the following.
- Large scale message queues like Kafka / Kinesis
- Real-time stream processing systems like Spark, Apache Flink, Kinesis Data Streams
- Have experience in Airflow
- You have experience with understanding the trade-offs between scalability, reliability, developer/ops overhead and infrastructure cost of building scalable data pipelines.
- You have expert knowledge of AWS. Should have built or managed a large scale data pipeline hosted within AWS.
- You have significant experience working with either SQL based databases (PostGres, etc) or MongoDB.
- You have the ability to define the roadmap and drive execution of the team.