Senior Data Engineer - Databricks

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks and PySpark.
• Develop end-to-end data workflows including data ingestion, transformation, and consumption.
• Optimize Spark jobs, cluster configurations, and pipeline performance.
• Manage and orchestrate workflows using Databricks Jobs and notebooks.
• Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks.
• Implement data quality checks, monitoring, and error handling mechanisms.
• Design and maintain Delta Lake tables with proper optimization strategies.
• Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions.
• Troubleshoot and resolve production data pipeline issues.
• Required Skills & Qualification
• Strong Data Engineering fundamentals including ETL/ELT pipeline design.
• Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning).
• Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration.
• Knowledge of Delta Lake features such as ACID transactions and schema evolution.
• Strong Python programming for data processing and automation.
• Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture).
• Strong SQL skills for data transformation and analysis.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Experience with Git version control and CI/CD practices.
• Minimum 8 years of experience in data engineering or related roles.
• 2-3 years of hands-on experience with Databricks.
• Experience building and maintaining production-grade data pipelines at scale.
• Proven experience modernizing legacy data pipelines.
• Experience working in Agile development environments.
• Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional

Additional Information

Job Description - We are looking for an experienced Senior Data Engineer - Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform. - The role requires strong expertise in PySpark, Databricks, and modern data engineering practices, along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at ELLIOTT MOSS CONSULTING PTE. LTD.? Share your experience