Azure Platform & Database Administrator

The Azure Platform & Database Administrator is responsible for the administration, reliability, security, and operational excellence of CAQH’s Azure data platforms, including Databricks, Azure SQL, the Enterprise Data Warehouse (EDW), and Azure Data Factory. This role will design and manage the technical foundations that support data engineering, analytics, and product delivery ensuring our platforms are stable, scalable, and aligned with CAQH’s modernization strategy.

The ideal candidate brings deep expertise in Azure cloud infrastructure, Databricks platform administration, SQL database management, and automated DevOps practices. This individual will own platform governance, manage schema/catalog structures, oversee data access and security, optimize performance, and partner closely with data engineers and product teams to enable smooth development and deployment workflows.

This is a full-time, remote, exempt position reporting to the Sr. Director, Data Engineering & Architecture.

Base Salary: $110,000 - $120,000 annually.

Specific Responsibilities:

Databricks Platform Administration

·        Manage and maintain all Databricks workspaces across development, test, and production environments.

·        Administer Unity Catalog / Hive Metastore, including schema design, permissions, external locations, mounts, and data governance configurations.

·        Manage clusters, cluster policies, job clusters, cluster performance, auto-scaling, and cost optimization.

·        Oversee Databricks Jobs, workflows, task orchestration, alerting, and SLAs.

·        Configure and maintain external data sources and connectors (ADLS Gen2, Delta tables, SQL MI, Snowflake, APIs).

·        Implement and maintain CI/CD pipelines for Databricks notebooks, SQL, and infrastructure using Azure DevOps/GitHub Actions.

·        Develop, maintain, and enforce platform security standards including RBAC, service principals, secrets, key vault integration, and audit logging.

Azure Platform & SQL Database Administration

·        Administer Azure SQL Managed Instance and other SQL environments supporting the CAQH EDW.

·        Perform database performance tuning, indexing, partitioning, storage management, and query optimization.

·        Oversee backup/restore operations, failover strategies, and high availability configurations.

·        Manage schema changes, deployments, and environment synchronization across dev, test, and prod.

·        Partner with the Data Engineering team on efficient ingestion and transformation patterns to SQL-based systems.

·        Monitor capacity, costs, and system health for all SQL and storage resources in Azure.

ADF & Integration Platform Operations

·        Administer Azure Data Factory including linked services, integration runtimes, pipelines, triggers, and parameterized workflows.

·        Implement enterprise standards for logging, error handling, retries, alerting, and operational monitoring.

·        Ensure secure configuration and compliance for all data movement operations, including PII/PHI safeguards.

Platform Governance & Standards

·        Establish and enforce standards for data access, identity management, secrets rotation, key management, and auditability.

·        Participate in Data Governance initiatives including metadata management, lineage tracking, and catalog standardization.

·        Build documentation for platform configurations, processes, and operational playbooks.

·        Evaluate new Azure features and provide recommendations for modernization and cost efficiency.

Operational Excellence & Support

·        Monitor platform reliability, SLAs, performance, and uptime across all Azure data services.

·        Troubleshoot and resolve platform-wide issues including job failures, performance bottlenecks, and connectivity disruptions.

·        Partner with InfoSec to ensure compliance with CAQH security policies and healthcare data regulations, including but not limited to HITRUST security access reviews.

·        Support agile development teams by managing environments, access, and deployment pipelines.

Supervisory Responsibilities:

This position does not have direct reports. However, the Sr. Data Engineer may collaborate with contractors or external vendors on specific initiatives and is encouraged to provide technical guidance to interns or newer team members when appropriate.

Skills: 

·        Deep experience administering Databricks (clusters, UC, jobs, external tables, secrets, governance).

·        Strong SQL database administration skills including performance tuning, HA/DR, and security.

·        Expertise with Azure cloud services including ADF, ADLS, Azure SQL MI/SQL Server, Key Vault, Monitor/Log Analytics.

·        Strong understanding of CI/CD and DevOps practices (Azure DevOps/GitHub Actions, YAML pipelines, infrastructure-as-code).

·        Experience supporting enterprise-scale data platforms and implementing platform usage standards.

·        Knowledge of data governance, metadata, lineage, RBAC, and PHI/PII compliance requirements.

·        Familiarity with Snowflake, Synapse, and Fabric concepts preferred.

·        Excellent communication skills in a remote, cross-functional environment.

·        Experience troubleshooting complex platform/service issues and improving operational resiliency.

Experience:

·        5–8 years of experience as an Azure Platform Administrator, Cloud Engineer, or Database Administrator.

·        Hands-on experience managing Databricks environments at scale.

·        Experience supporting EDW or analytics platforms in Azure or hybrid environments.

·        Experience building and managing CI/CD pipelines for data workloads.

·        Healthcare or regulated data experience strongly preferred.

Education:

·        Bachelor’s degree in Computer Science, Information Systems, Engineering, or equivalent experience.

·        Azure Administrator Associate, Azure Database Administrator, or Databricks Administrator certifications preferred.