Data Engineer

Responsibilities

• Unity operates in a fast-paced, high-trust environment. You will lead the design and implementation of systems that unlock the value of both structured and unstructured data.
• LLM-Powered Data Extraction & Structuring
• Design and deploy pipelines that use Large Language Models (LLMs) to extract, normalise, and structure data from unstructured sources (documents, PDFs, contracts, reports, emails, call transcripts, project artefacts).
• Implement Text2SQL, schema inference, entity extraction, classification, summarisation, and semantic enrichment workflows .
• Build automated ingestion frameworks using Databricks (Spark, Delta Lake, Auto Loader) to transform raw first-party data into analytics-ready and AI-consumable formats.
• Leverage Databricks capabilities (e.g. PySpark, Delta Live Tables, MLflow, Databricks AI/Vector Search) to operationalise AI-driven transformations at scale.
• Retrieval-Augmented Generation (RAG) & Knowledge Systems
• Architect and maintain RAG pipelines combining LLMs with:
• Structured data stored in Delta Lake (Databricks)
• Unstructured document repositories and object storage (e.g. S3 / ADLS / GCS)
• Vector databases or Databricks Vector Search
• Design embedding pipelines and semantic search layers integrated with lakehouse data models.
• Build internal AI-powered search and conversational assistants grounded in trusted enterprise data.
• Optimise relevance, grounding accuracy, latency, and response quality for interactive AI applications used by internal teams.
• Implement hybrid retrieval approaches combining Spark SQL / Delta queries with vector similarity search.
• AI-Native Data Platform Architecture
• Design scalable lakehouse architectures spanning ingestion, transformation, storage, vectorisation, and model interaction layers.
• Build and operate production-grade pipelines for:
• Structured analytics data in Delta Lake
• Semi-structured data (JSON, logs, event streams)
• Unstructured data (documents, transcripts, knowledge repositories)
• Embedding generation and indexing workflows
• Develop robust, analytics-ready data models and semantic layers using Delta Lake and Databricks SQL to ensure consistent, governed data consumption.
• Embed observability across both data and LLM pipelines (evaluation, hallucination detection, lineage tracking via Unity Catalog, usage analytics).
• LLMOps, Evaluation & Optimisation
• Evaluate GenAI systems using experimentation frameworks, offline benchmarks, and real-world user feedback.
• Improve response quality through prompt engineering, retrieval optimisation, grounding strategies, and agent orchestration.
• Implement reliable deployment and monitoring strategies for GenAI systems interacting with lakehouse data.
• Leverage MLflow and Databricks model serving for lifecycle management of models and pipelines.
• Own production rollouts of internally facing GenAI applications.
• Establish repeatable LLMOps patterns covering testing, performance optimisation, cost control, and governance.
• Data Quality, Governance & Compliance
• Impl

Additional Information

About Unity Advisory Unity Advisory is a new-generation professional services firm built for an AI-enabled world. We operate a lean, conflict-free, and client-centric model that integrates advanced technology and AI into every workstream. With no audit practice, we are free from traditional conflicts and legacy silos. This allows us to move faster, collaborate openly, and focus entirely on creating value for clients. Our flat structure and collaborative culture empower exceptional people to deliver exceptional work. We combine deep advisory expertise with cutting-edge data, AI, and commercial insight to help clients navigate complex challenges faster, smarter, and with greater clarity. At Unity, we are redefining how expert advisory is delivered-one innovative engagement at a time. Core Internal AI & Data Platform | Unity Advisory We are looking for a highly skilled, commercially aware Senior Data Engineer to join Unity Advisory in an internal-facing role focused on building and operating the firm's Core Internal AI & Data Platform , with a strong emphasis on LLM-powered data extraction, unstructured data processing, Retrieval-Augmented Generation (RAG), and AI-enabled knowledge systems . This is a senior, hands-on engineering role responsible for designing and productionising LLM-native data pipelines and retrieval architectures that transform fragmented, unstructured internal data into structured, trusted, AI-ready assets. A key component of this role is architecting and optimising our data platform around Databricks as a unified lakehouse and AI foundation , enabling seamless integration between large-scale data processing, structured and unstructured data, vector search, and GenAI applications. This role sits at the intersection of data engineering, GenAI systems, and modern cloud lakehouse platforms , with clear opportunity to shape standards, influence architecture, and scale a next-generation internal data capability.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at unity-advisory? Share your experience