Lead Data Engineer

Responsibilities

• Design, develop, and optimize complex batch and real-time data pipelines using Databricks, Spark, and related technologies.
• Build and maintain scalable lakehouse architectures and high-performance data models to support business driven outcomes
• Provide technical leadership and day-to-day coordination for a small offshore data engineering team.
• Define tasks, review deliverables, and ensure adherence to engineering standards and timelines.
• Lead the implementation of advanced ETL/ELT frameworks and reusable data engineering patterns.
• Solve complex data challenges including performance tuning, data skew, partitioning, and large-scale processing.
• Implement robust data quality, validation, and observability frameworks.
• Collaborate closely with API, Analytics, and Platform teams to deliver reliable, production-grade data assets.
• Drive best practices in coding standards, testing, and CI/CD for data pipelines.
• Troubleshoot and resolve production issues, ensuring high availability and reliability.
• Act as the primary liaison between onshore stakeholders and offshore delivery teams.
• Required Qualifications
• 8-12 years of experience in data engineering.
• Proven experience leading or coordinating offshore or distributed engineering teams.
• Deep hands-on expertise with Databricks, Spark, and distributed data processing.
• Advanced proficiency in Python and/or Scala, plus strong SQL skills.
• Strong experience with performance tuning and optimization of large-scale data pipelines.
• Experience with cloud data platforms (Azure strongly preferred).
• Solid understanding of data modeling, streaming architectures, and data lifecycle management.
• Experience implementing data quality, lineage, and governance controls.
• Preferred
• Experience with Kafka/Confluent and real-time streaming pipelines.
• Exposure to Snowflake or similar cloud data warehouses.
• Experience supporting AI/ML pipelines or feature engineering workflows.
• Background in regulated industries (e.g., healthcare).

Benefits

Additional Information

Job Description Role Summary We are seeking a highly technical Senior Data Engineer to design, build, and optimize scalable data pipelines and platforms. This role is primarily hands-on, with responsibility for leading and coordinating a small offshore data engineering team to deliver high-quality, performant data solutions.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at henryschein? Share your experience