Senior Data Engineer (1043) - DataSF - Office of City Administrator

Requirements

• Education :
• Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.
• Substitution :
• Completion of the 1010 Information Systems Trainee Program may be substituted for the required degree.
• Verification:
• Note:
• Falsifying one's education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco.
• One-year full-time employment is considered equivalent to 2000 hours (2000 hours of qualifying work experience is based on a 40 hour work week). Any overtime hours that you work above forty (40) hours per week are not included in the calculation to determine full-time employment
• Applicants must meet minimum qualification requirements by the final filing date unless otherwise noted.
• Desirable Qualifications
• Technical Knowledge Hands-on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
• Demonstrated expertise in writing advanced, performant SQL, and using tools like dbt for SQL-based data transformation and modeling.
• Strong programming skills in Python (with libraries like pandas, PySpark) for data processing and automation.
• Proficiency with an Infrastructure as Code tool, with a preference for Terraform.
• Experience building and deploying data pipelines using orchestration tools like Azure Data Factory, Airflow, Dagster, or similar technologies.
• Deep

Benefits

Additional Information

DataSF is seeking a Senior Data Engineer with 3+ years of experience to join our growing team. Reporting to the Principal Data Engineer, you will be instrumental in designing, building, and maintaining the City's data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs. This is an exciting position for someone eager to apply advanced data engineering techniques to complex urban challenges, contributing directly to San Francisco's commitment to efficient, equitable , and ethical service delivery. Learn more about DataSF's recent work on our blog . If you are an entrepreneurial and passionate data enthusiast, join our team to improve government through good use of data! Essential duties include, but are not limited to, the following: Platform Administration : Manage our central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization. Administer our platform with a focus on data democratization and accessibility, while protecting privacy and security. Pipeline Developmen t: Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources. Design infrastructure to support both streaming and batch processes, and both structured and unstructured data sources. Infrastructure as Code (IaC) : Use Terraform to define, deploy, and manage data infrastructure, ensuring our pipelines are reproducible, version-controlled, and production-ready. Best Practices & Innovation : Champion and implement best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing. Think creatively to find new ways to improve our data platform's capabilities and efficiency. Provide guidance to department partners on data engineering best practices. Collaboration : Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders in diverse domains to understand data requirements and build solutions that meet their needs. Monitoring & Support : Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at Cityandcountyofsanfrancisco1? Share your experience