Job Openings Technical Lead, Data (Python/Spark)

About the job Technical Lead, Data (Python/Spark)

Note: This is a Remote position but only for candidates who are residing in Canada.


  • Build and manage the data processing pipeline
  • Ensure reliable processing of data, keeping error counts in check, add data validation checks, and ensure performance of data pipeline is good.
  • Responsible for building the infrastructure for running the data pipeline
  • Work with data ingestion and other feature teams to ensure the data pipeline runs smoothly.

Experience and Skills

  • You are currently a technical lead managing a very large scale data processing pipeline.
  • You are a hands-on technical leader
  • You have 12+ years of professional software development experience
  • You have 6+ years of experience with developing data applications in Python.
  • You have a deep knowledge of building, scaling and managing a large scale real-time data pipeline (processing 100s of millions of events per day).
  • You have a deep knowledge and experience working with two or more of the following.
  • Large scale message queues like Kafka / Kinesis
  • Real-time stream processing systems like Spark, Apache Flink, Kinesis Data Streams
  • Have experience in Airflow
  • You have experience with understanding the trade-offs between scalability, reliability, developer/ops overhead and infrastructure cost of building scalable data pipelines.
  • You have expert knowledge of AWS. Should have built or managed a large scale data pipeline hosted within AWS.
  • You have significant experience working with either SQL based databases (PostGres, etc) or MongoDB.
  • You have the ability to define the roadmap and drive execution of the team.