Associate Principal - Data Engineering (Confluent Kafka)

Location: Pune

No. of positions open: 1

Years of Experience: 11-15 Years

Prerequisite skills: 

• AWS Cloud services, Confluent Kafka, Oracle XStream CDC, Connector Frameworks, Scripting & Automation, Monitoring Tools

Job Description:

We are looking for an architect who has hands-on experience with the Confluent Kafka Oracle CDC XStream connector.

This is not a trivial integration. The source Oracle system produces about 5TB of archive logs, and we expect to stream data at a peak rate of around 10 Mbps. The architecture must account for best practices in:

1) Confluent-managed Kafka usage (scalability, fault tolerance, etc.)

2) Error handling and retries

3) Notification and alerting mechanisms

4) Source system optimizations, including:

a) Use of multiple outbound servers

b) SGA allocation and tuning

c) Monitoring of the Streams Pool.

Given the involvement of multiple moving partsOracle DB, Capture Process, Streams Queue, XStream Outbound Server, and the Kafka Connectorwe need someone who can provide architectural guidance to ensure a robust, performant, and maintainable setup.

Key Responsibilities

• Architectural Design: Define and validate the end-to-end architecture for streaming Oracle data into Kafka using XStream CDC. Ensure scalability, fault tolerance, and maintainability.
• Performance Optimization: Tune Oracle source systems including SGA allocation, Streams Pool monitoring, and outbound server configuration to support high-throughput data capture (e.g., 5TB archive logs, 10 Mbps peak rate)
• Connector Management: Oversee deployment and lifecycle of Kafka connectors, including automation of pause/resume operations during maintenance
• Error Handling & Monitoring: Implement robust error recovery, alerting mechanisms, and health checks for connector operations
• Stakeholder Collaboration: Work with MFG Directors, platform teams, and external consultants to align on architecture and operational readiness
  

Technical Skills Required

• Confluent Kafka: Deep understanding of Kafka internals, connector framework, and Confluent-managed Kafka features.
• Oracle XStream CDC: Hands-on experience with Oracle Streams, XStream Outbound Server, Logical Change Records (LCRs), and redo log mining
• Connector Frameworks: Familiarity with Debezium, Kafka Connect, and snapshot streaming configurations
• Scripting & Automation: Ability to automate connector operations using shell scripts and manage deployments across environments
• Monitoring Tools: Experience with alerting systems and performance dashboards for connector health and throughput

Preferred Experience

• Prior implementation of Oracle CDC with Kafka in production environments.
• Experience with Oracle 19c and understanding of deprecated LogMiner features
• Familiarity with Terex or similar manufacturing data ecosystems is a plus

Skills

Mandatory Skills : AWS Kinesis,Java,Kafka,Python,AWS Glue,AWS Lambda,AWS S3,Scala,Apache SparkStreaming,ANSI-SQL