Job Openings Data Engineer

About the job Data Engineer

Responsibilities:

  • Design, develop, and maintain scalable data pipelines for the efficient collection, storage, and processing of large datasets.
  • Develops and design best practices around data collection and maintain orchestration of data flows, for batch, near-real time data.
  • Develop and design best practices of CDC and SCD for existing systems to maintain data consistency
  • Participates in evaluation and deployment of big data tools, relevant to the development.
  • Ability to identify different platforms, techniques, and methodology used in big data platforms, and demonstrate for various system in the organization.
  • Optimize and tune data systems for performance and reliability.
  • Develop and implement data models and schemas to support business intelligence and analytics needs.
  • Monitor and troubleshoot data pipelines to ensure smooth and efficient operation.
  • Stay up-to-date with the latest advancements in data engineering and incorporate them into our data infrastructure.
  • Document data engineering architectures and follow best practices.

Qualifications:

  • Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.
  • 5+ years of experience as a Data Engineer, Data Architect, Big Data Administrator or similar role.
  • Strong programming skills in Python.
  • Experience with UNIX system administration using command line.
  • Experience with data processing frameworks such as Airflow, Airbyte, DBT, Hadoop, Spark, Hive, and Kafka,
  • Proficiency in SQL and experience with relational databases.
  • Knowledge of data modeling, ETL processes, and data warehousing solutions.
  • Knowledge of Structuring data in various schemas suitable for a data lake, warehouse, or marts.
  • Experience in implementing scalable, reliable, and efficient data storage solutions with PostgreSQL, MinIO.
  • Experience in enhancing the performance of big data applications, including optimizing system operations.
  • Excellent problem-solving skills and the ability to work independently or as part of a team.
  • Strong communication skills and the ability to explain complex technical concepts to non-technical audiences.