Information Technology - Lead Data Engineer

Responsibilities

• Be an effective distributed-system implementer in the following coreactivities:
• o Design and develop data engineering services and their ecosystem usingdistributed databases (relational, columnar, graph, in-memory); orchestration(Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis,Spark).
• oDesign and develop MLOps production pipelines; provide technical support todata scientists/ML engineers by getting their ML/DL models deployed at scaleand meeting SLAs on both cloud and on-premises GPU and CPU instances.
• o Design data models for mission-critical, high-volume, near-real-time/batchdata; build idempotent/atomic production data pipelines to make data ingestionmore fault tolerant.
• o Design and develop intuitive, highly automated, self-service data platformfunctions for business users.
• o Design, build, and operate scalable and reliable data pipelines on theDatabricks platform.
• Explore, evaluate and champion the introduction of next-generationtechnologies in the data-ingestion workflow. Participate in project planningand provide technical guidance on cloud architecture for data projects.

Requirements

• BS in Computer Science or other related discipline is required. Advanceddegrees in Computer Science (PhD, MS) are highly desirable.
• -5+ years of relevant industry experience in some or most of the followingtechnical areas:
• o Advanced programming skills in Python. Conversant with data structures andalgorithm design.
• o Experience in building data pipelines (including data collection,warehousing, processing, analysis, monitoring, and governance) usingopen-source data ingestion platforms.
• o Intermediate-level knowledge and experience with AWS cloud components andbest practices. Good understanding in deploying data stores such as S3,RedShift, Elasticache, PostgreSQL, and EMR.
• o Hands on experience with Databricks workspace, cluster management, AI Agentcapabilities, and job orchestration
• o Prior experience in modern software development is required (such as webfrontend UI, backend API microservices, understanding of CI/CD and Scrum/Kanbanagile development). Strong grasp on object-oriented or functional programming(using e.g. Python, Java, Scala, or C#).

Additional Information

Job Description The lead data engineer is a senior software developer with strong softwareengineering skills who is responsible for building custom open-source-baseddata ingestion and MLOps platforms. He/she has deep appreciation of thecomplexity of the data engineering process, such as the challenges of dataingestion involving large or near-real-time datasets, the maintenance of highdata quality, and the importance of automation for increasing pipelinerobustness and reducing the need for human intervention.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at SINGAPORE AIRLINES LIMITED? Share your experience