Director of Engineering

We are seeking an Engineering Director with deep Data Engineering expertise to lead the design and delivery of a next-generation, AI-enabled data ingestion platform on AWS.

You will be #LI-hybrid based in Hyderabad and reporting to CTO.

This role owns the strategy, architecture, and execution for how data is ingested, transformed, modelled at scale — across both batch and real-time (streaming) pipelines.

Data engineering is the core of this role.

You will define the pipeline architecture, data models, and quality frameworks that everything else depends on, while modernising legacy and mainframe pipelines into cloud-native, agent-assisted systems.

This is a hands-on leadership role that blends data engineering, platform engineering, and AI/DevOps excellence, responsible for building high-performing teams and delivering reliable, scalable, and intelligent data systems.

Key Responsibilities

Data Engineering & Pipeline Architecture (Core)

• Own the end-to-end architecture for batch and streaming data ingestion — source onboarding, extraction, transformation, loading, and serving.
• Design robust, scalable data models and schemas (dimensional, normalized, and data-vault where appropriate) for a modern lakehouse.
• Build high-throughput, fault-tolerant pipelines using distributed processing (Apache Spark / EMR, Glue, Flink) with idempotency, replay, and backfill built in.
• Architect real-time ingestion and change-data-capture (CDC) using Kafka / MSK / Kinesis and streaming frameworks.
• Standardize open table and file formats (Apache Iceberg / Delta / Hudi, Parquet, Avro) and manage schema evolution and partitioning strategies.
• Own pipeline orchestration and dependency management (Airflow / MWAA, Step Functions, dbt) with clear SLAs for freshness and latency.
• Drive performance, scalability, and cost optimization across compute and storage (partition pruning, file compaction, resource tuning, FinOps).

Data Quality, Governance & Reliability

• Embed data quality, validation, lineage, and control frameworks directly into ingestion (Great Expectations / Deequ-style checks, data contracts).
• Define and enforce data contracts, cataloging, and metadata management (AWS Glue Data Catalog, Lake Formation).
• Ensure parity validation and controlled, phased migration from legacy and mainframe systems with zero data loss.
• Implement observability for data pipelines — freshness, volume, schema-drift, and anomaly detection — with SLOs/SLIs and error budgets.
• Ensure DevSecOps, encryption, access control, and compliance (PII handling, GDPR, data residency) across the platform.

Strategy & Engineering Leadership

• Define and execute the data-first, AI-enabled ingestion platform strategy aligned to business outcomes.
• Build and lead teams spanning Data Engineering, Platform/DevOps, and AI/ML.
• Establish engineering standards, reusable frameworks, operating models, and delivery roadmaps.
• Mentor senior and staff engineers; grow technical depth and set the bar for engineering craftsmanshi.

AI & Agentic Engineering

• Introduce AI-assisted engineering (Claude Code, Copilot) across the data SDLC to accelerate pipeline development and testing.
• Leverage AWS Bedrock / LLM platforms for ingestion intelligence — schema inference, mapping, validation, and anomaly detection.
• Drive AIOps and agent-driven automation for auto-remediation, intelligent alerting, and self-healing pipelines.

Platform, DevOps & Cloud Foundations

• Establish best-in-class CI/CD, GitOps, and release engineering for data pipelines and platform components.
• Standardize Infrastructure as Code (Terraform / CDK) and reusable, self-service platform building blocks.
• Drive event-driven and microservices-based architectures on AWS-native services (S3, Glue, Lambda, EKS, MSK/Kinesis).

Stakeholder & Delivery Management

• Partner with Product, Architecture, Data Governance, and Business teams on roadmaps and prioritization.
• Manage delivery, risk, and cross-program dependencies; communicate trade-offs clearly to senior stakeholders.

• 12+ years in software/data engineering, with 5+ years in engineering leadership roles.
• Proven experience leading data engineering, platform, or data-intensive organizations at scale.

Data Engineering (Core)

• Deep expertise designing and operating large-scale batch and streaming data pipelines in production.
• Dstributed data processing: Apache Spark (EMR/Glue), and Flink or Beam.
• Data modeling and warehousing/lakehouse design (Redshift, Snowflake, Databricks, Athena, or equivalent).
• Lakehouse table formats and file formats: Iceberg / Delta / Hudi, Parquet, Avro; schema evolution and partitioning.
• Orchestration and transformation: Airflow / MWAA, Step Functions, dbt.
• Streaming and CDC: Kafka / MSK, Kinesis, Debezium or equivalent.
• Expert-level SQL and Python (Scala/Java).
• Data quality, lineage, cataloging, and governance frameworks (Great Expectations/Deequ, Glue Catalog, Lake Formation).

AWS & Cloud Architecture

• Deep AWS expertise: S3, Glue, EMR, Lambda, MSK/Kinesis, Redshift, DynamoDB, EKS, VPC, IAM.
• Experience designing large-scale, multi-account cloud data platforms.

DevOps & Platform Engineering

• CI/CD (Jenkins, Harness, GitHub Actions); IaC (Terraform, CDK); GitOps (ArgoCD/Flux); Containers (Docker, Kubernetes/EKS).

AI & Modern Engineering

• Experience driving AI adoption in engineering/data workflows; familiarity with LLMs, AWS Bedrock, and AIOps platforms.

Observability, Reliability & Security

• Prometheus, Grafana, OpenTelemetry, CloudWatch; SRE principles and incident management.
• DevSecOps practices, IAM, encryption, and compliance frameworks (PII, GDPR, data residency).

Experian is a global data and technology company, powering opportunities for people and businesses around the world. We operate across a range of markets, from financial services to healthcare, automotive, agribusiness, insurance, and many more. Experian invests in people and new advanced technologies to unlock the power of data. We have an amazing team of 25,200 people in 32 countries.

Our uniqueness is that we celebrate yours. Experian's people first, inclusive and purpose driven culture is multi award-winning; World's Best Workplaces™ 2025 (Fortune Global Top 25), Great Place To Work™ in 26 countries to name a few.

Check out Experian Life on social or explore our Careers Site (experian.com/careers) to understand why.

Our uniqueness is that we celebrate yours. Experian's culture and people are important differentiators. We take our people agenda very seriously and focus on what matters; DEI, work/life balance, development, authenticity, collaboration, wellness, reward & recognition, volunteering... the list goes on. Experian's people first approach is award-winning; World's Best Workplaces™ 2024 (Fortune Global Top 25), Great Place To Work™ in 24 countries, and Glassdoor Best Places to Work 2024 to name a few. Check out Experian Life on social or our Careers Site and Glassdoor to understand why.

Experian is also proud to be an Equal Opportunity and Affirmative Action employer. If you have a disability or special need that requires accommodation, please let us know at the earliest opportunity.

Benefits

Experian care for employee's work life balance, health, safety and wellbeing. In support of this endeavor, we offer best-in-class family well-being benefits, enhanced medical benefits and paid time off.

This is a hybrid remote/in-office role.

Recruitment Fraud Awareness - Experian's recruitment process is conducted only through authorised channels. Recruitment communications will only be sent from an @experian.com email address. Experian will never ask candidates to make any payment as part of an application, interview, assessment, onboarding, or recruitment process. To apply for roles or verify opportunities, please visit experian.com/careers.

Experian Careers - Creating a better tomorrow together

Find out what its like to work for Experian by clicking here