Data Engineer

The Data Engineer will play a strategic role in designing, developing, and optimizing advanced data pipelines and infrastructure to support CAQH’s enterprise data modernization efforts. This role is critical to delivering scalable, secure, and high-performance data solutions that drive analytics, reporting, and AI/ML initiatives across the organization. This role owns complex data modeling, enrichment logic, and pipeline reliability across Databricks, Azure SQL, and downstream analytics and API consumers. The position requires deep domain expertise, strong cross-team collaboration, and the ability to translate ambiguous requirements into production-ready data solutions while guiding vendors and internal teams.

The Data Engineer is a full-time, remote, exempt position and reports to the Sr. Director, Data Engineering & Architecture.

Base Salary: $120,000 - $135,000

Data Engineering & Architecture

·        Design, build, and maintain complex ETL/ELT pipelines across Databricks, Azure SQL, and downstream gold-layer models supporting priority projects

·        Lead development and evolution of enriched data models, including field-level enrichment logic, recency rules, and provider-level enrichment flags.

·        Own data logic, including reconciliation between source and target data sources and resolution of duplication and data discrepancies.

·        Implement and refine medallion architecture patterns (bronze → gold), ensuring data quality, traceability, and performance at scale.

Data Quality, Governance & Reliability

·        Identify, document, and remediate systemic data quality issues, including null handling, soft deletes, authorization flags, and incorrect organizational mappings.

·        Define and operationalize rules for data in collaboration with product, governance, and engineering stakeholders.

·        Produce authoritative documentation (Confluence, mapping workbooks) to serve as a single source of truth for enrichment logic and data behavior.

·        Cross‑Functional & Vendor Collaboration

·        Act as a primary technical counterpart for vendors providing detailed queries, validation logic, and corrective guidance on upstream data issues.

·        Partner closely with product owners, architects, and application teams to ensure data models align with product defined use cases.

·        Support UAT and release readiness by preparing data, validating counts, and resolving last‑mile data defects under tight timelines

Technical Leadership

·        Provide hands-on technical leadership without formal direct reports, mentoring peers and guiding best practices in SQL, Databricks notebooks, and data modeling.

·        Influence architectural decisions related to Databricks compute, job scheduling, and environment promotion strategies

Skills: 

·        Advanced SQL expertise (complex joins, reconciliation, performance tuning).

·        Deep hands-on experience with Databricks, Delta Lake, and Azure SQL.

·        Strong data modeling skills for analytical, operational, and API‑driven use cases.

·        Proven ability to debug and stabilize messy, evolving enterprise data domains.

·        Excellent written and verbal communication, especially for explaining complex data behavior to non‑technical stakeholders.

·        Experience using Git, DevOps tools, and CI/CD pipelines for data engineering workflows.

Experience:

•               4–7 years of hands-on experience in a data engineering or analytics engineering role.

·        Demonstrated success leading data modernization or migration initiatives in cloud environments.

·        Prior experience working with healthcare or other regulated data environments is highly desirable.

·

Education:

·        Bachelor’s degree in Computer Science, Information Systems, Data Engineering, or a related field.

·        Azure Data Engineer Associate or related certification (preferred).

·        Coursework or certification in AI/ML (preferred but not required).