Senior Architect, Databricks

Responsibilities

• Design data flow and processing pipelines for analysis using modern toolsets such as Spark on Scala, Storm, Flume, Sqoop
• Design data structures for ingestion and reporting, specific to use case and technology
• Collaborate with data science and engineering teams to productionize machine learning and Generative AI workloads within modern cloud ecosystems
• Establish best practices for AI data readiness, feature engineering, model monitoring, observability, and governance
• Lead technical discussions around AI-enabled modernization strategies including semantic layers, vector databases, AI agents, and knowledge retrieval systems
• Evaluate emerging AI technologies and recommend fit-for-purpose tooling aligned to client objectives, scalability, security, and compliance requirements
• Guide and coach consultants, share best practices, perform code reviews
• Coordinate activities with data source application owners to ensure integration with analytics platform and source systems
• Ensure that data integrity issues are addressed
• Liaise and attend meetings necessary to perform duties and aid business and technical development

Requirements

• 10+ years of professional experience and 3+ years of expeirence with Databricks
• 8+ years of prior consulting experience
• Very strong technical expertise in Databricks, including architecture, performance optimization, Delta Lake, Unity Catalog, workflow orchestration, security, governance, and cloud-native data engineering best practices
• Experience architecting and deploying AI/ML or Generative AI solutions within enterprise environments
• Hands-on experience with LLM ecosystems and frameworks such as OpenAI, Azure OpenAI, LangChain, Semantic Kernel, MLflow, or Databricks Mosaic AI
• Knowledge of Retrieval-Augmented Generation (RAG), vector databases, embeddings, prompt engineering, and AI orchestration patterns
• Familiarity with AI governance, model lifecycle management, responsible AI principles, and enterprise security considerations
• Experience integrating AI capabilities into data platforms, analytics solutions, or customer-facing applications
• Knowledge of standard methodologies, concepts, best practices, and procedures within a Big Data environment
• A commitment to inclusion and diversity, and openness to new ideas and perspectives
• Travel requirements up to 30 to 50%
• Hybrid role: 1 to 2 days a week in office
• Seattle or Washington, D.C.
• $199,100 - $223,400 USD
• Los Angeles
• $205,400 - $234,100 USD
• New York City or San Francisco
• $215,200 - $244,700 USD
• A location not listed above
• $183,800 - $212,800 USD
• Other consultancies talk at you.
• At West Monroe, we work with you.
• We're a global business and technology consulting firm passionate about creating measurable value for our clients, delivering real-world solutions.
• At West Monroe, we zero in on the heart of the opportunity, getting to results faster and preparing people for what's next.
• You'll feel the difference in how we work. We show up personally. We're right there in the room with you, co-creating through the challenges. With West Monroe, collaboration isn't a

Benefits

Additional Information

Are you ready to make an impact? West Monroe is seeking a Senior Architect to join our thriving Technology and Experience Practice. This is an excellent opportunity to work with clients developing Big Data strategy and roadmaps with emphasis on Databricks, while gaining exposure to a variety of industries including Healthcare, Financial Services, Insurance, and Utilities.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at westmonroe4? Share your experience