Senior Data Engineer AWS & GCP

About the role

SUMMARY: As a Data Engineer, you will be responsible for designing, developing, maintaining, and optimising a data pipeline infrastructure using a proprietary data platform , which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis. WHAT WILL YOU BE DOING: Data Engineering Ownership : End-to-end creation and upkeep of robust data workflows and components, leveraging tools such as Databricks. This includes designing early-stage prototypes and deploying large-scale data acquisition, handling, and storage strategies. ETL/ELT Workflow Management : Construct and refine data ingestion pipelines to seamlessly integrate varied data sources into the central platform. Ensure data consistency through validation, cleaning, and enrichment routines. Stakeholder Collaboration : Partner with commercial teams and data stakeholders to gather and refine specifications for analytical products and visual reporting needs. Architectural Design & Data Modelling : Engage with analysts and data experts to define efficient data models and architectural plans. Support dashboard and report development, while ensuring optimal data structuring for performance. Pipeline Performance & Reliability : Continuously assess and fine-tune data workflows, addressing system inefficiencies, integration problems, and data fidelity concerns. Quality Assurance in Data : Define expectations for data integrity and collaborate with QA to automate validation checks. Utilise monitoring tools to surface and track data quality metrics. Data Controls & Compliance : Apply internal governance standards and safeguard sensitive data through access rules, encryption protocols, and retention strategies, aligning with organisational security frameworks. Technical Enablement & Knowledge Sharing : Work closely with interdisciplinary teams to ensure data needs are met, while thoroughly documenting processes and solutions. Convey technical ideas in a way that's accessible across departments. Innovation & Process Evolution : Keep informed on developments within the data ecosystem. Advocate for and implement improvements in efficiency, tooling, and automation. Participate in internal knowledge communities. Agile Delivery Engagement : Contribute to delivery efforts by actively participating in sprint planning, stand-ups, and backlog refinement. Help define and deliver technical tasks, aligning work with CI/CD best practices. WHAT WE ARE LOOKING FOR: Approximately 4 years of experience in data engineering for mid-level roles 6 or more years of experience for senior-level data engineering positions Between 1 to 4 years of experience building data products using platforms such as Databricks, Spark, Cloudera, or HortonWorks Skilled in Python (especially PySpark), Scala, and SQL Experienced in designing and implementing scalable data pipelines for high-volume environments Solid understanding of ELT/ETL practices and data integration strategies Capable of writing robust production code with automated testing Familiar with CI/CD tools such as GitHub Actions and Jenkins for deploying code Hands-on experience with distributed data processing using Apache Spark Proficient with cloud services including AWS, Azure, or Google Cloud and tools like S3, Glue, Lambda, Redshift, and BigQuery Good knowledge of data modelling, relational databases, and SQL performance optimisation Strong analytical and problem-solving abilities, with attention to troubleshooting details Effective communicator with experience working in collaborative, cross-functional teams Basic understanding of machine learning concepts such as classification, regression, A/B testing, and experimental design AWESOME BUT NOT REQUIRED: Understanding of the UK media landscape, including over-the-top (OTT) and traditional broadcast advertising Familiarity with concepts and trends in digital advertising and marketing analytics Experience applying statistical approaches such as regression and classification, as well as designing and analysing A/B and other controlled experiments Awareness of modern data architecture methodologies including Data Mesh, enterprise-level data frameworks, and business intelligence architecture Solid grasp of data protection practices, regulatory compliance, and governance principles Exposure to a variety of data management domains such as metadata handling, data quality enforcement, master data systems, and governance tooling Hands-on experience with data visualisation platforms like Tableau, Looker, AWS QuickSight, and ThoughtSpot for building interactive reporting solutions WHAT'S IN IT FOR YOU? Be part of our collegial environment where responsibility and authority are shared equally amongst colleagues and help create our company culture A culture in which we don't criticise failure but ensure we learn from our mistakes An Agile environmen