Strategic Technical Leadership: Drive the multi-year technical roadmap and architectural vision for search services, ensuring they are secure, cost-efficient, and highly scalable for petabyte-scale enterprise workloads.
Architectural Oversight: Design and oversee the implementation of large-scale OpenSearch clusters, ensuring low-latency indexing and query performance across multi-cloud environments.
Deep Platform Integration: Lead the deep integration of OpenSearch with core CDP components like Apache Iceberg, SDX, and Ozone to provide a unified, secure search experience.
Engineering Excellence: Define and champion organization-wide best practices for security (RBAC, TLS), observability, and enterprise-grade reliability.
Complex Problem Solving: Design solutions for extreme scale, multi-tenancy isolation, and complex hybrid/multi-cloud deployment models.
Mentorship & Influence: Mentor staff and senior engineers on complex technical designs and foster a culture of continuous improvement and innovation across the entire engineering organization.
Open Source Stewardship: Act as a primary influencer and liaison within the upstream OpenSearch community, aligning their roadmap with Cloudera's product strategy.
Technical Requirements
Search Expertise: 10+ years of experience (scaling from the 8+ years required for Staff level) working with OpenSearch or Elasticsearch in large-scale production environments.
Distributed Systems Mastery: Deep understanding of consensus algorithms, CAP theorem, replication, sharding, and fault-tolerant design patterns.
Mastery in Programming: Mastery of Java for core development and proficiency in Go or Python for advanced automation and tooling.
Infrastructure & Cloud: Extensive experience with Kubernetes (K8s) and container orchestration, including deploying search workloads on AWS (EKS/AOSS), Azure (AKS), or Google Cloud (GKE).
Systems Thinking: Demonstrated ability to drive architectural decisions with a focus on non-functional requirements such as high availability and security.
Requirements
Deep expertise in Lucene internals (e.g., segment merging, bitsets, and codecs)
Advanced knowledge of Vector Database capabilities for Generative AI (RAG) and retrieval-augmented analytics.
Significant history of contributions to open-source projects like the Apache Software Foundation or the OpenSearch Project
Why Cloudera?
What you can expect from us:
Generous PTO Policy
Support work life balance with Unplugged Days
Flexible WFH Policy
Mental & Physical Wellness programs
Phone and Internet Reimbursement program
Access to Continued Career Development
Comprehensive Benefits and Competitive Packages
Paid Volunteer Time
Employee Resource Groups
EEO/VEVRAA
#LI-NK1
Benefits
Vision insurancePaid time offFlexible schedule
Additional Information
Business Area:
Engineering
Seniority Level:
Director
Job Description:
At Cloudera, we empower people to transform complex data into clear and actionable insights. With as much data under management as the hyperscalers, we're the preferred data partner for the top companies in almost every industry. Powered by the relentless innovation of the open source community, Cloudera advances digital transformation for the world's largest enterprises.
As a Principal OpenSearch Engineer, you will be the foremost technical authority and visionary for the search heartbeat of the Cloudera Data Platform (CDP). You will not only drive the technical vision but also ensure the strategic evolution of a high-performance, scalable, and secure search infrastructure that powers mission-critical data discovery and analytics for global enterprises. You will solve the most complex, high-impact technical problems that span multiple organizational boundaries.
Cloudera · India