Job Description:
Mandatory Areas
Microsoft fabric
Databricks
Data pipelines
Data products
Kafka Azure event hub
Mandatory Skills:
Microsoft fabric Databricks Data pipelines Data products Kafka Azure event hub
Preferred Skills:
Apache airflow Azure data factory Medallion architecture ci/cd devops tms
CORE RESPONSIBILITIES:
Design and build production-grade data pipelines spanning source ingestion bronze landing silver transformation and gold-layer data product delivery within the squads domain scope.
Own pipeline KTLO (Keep the Lights On) for assigned data products including monitoring alerting incident response and ongoing reliability improvements.
Implement data ingestion patterns for assigned source systems including batch file ingestion API-based ingestion and event-driven streaming (Kafka Azure Event Hub) depending on squad scope.
Apply medallion architecture (bronze silver gold) and Fabric IQ certification standards consistently across all data product builds.
Collaborate with System Analysts to implement field-level transformations business rule logic and data quality checks as specified in product requirements documentation.
Participate in and contribute to pipeline design reviews ensuring solutions align with the organizations Databricks and Fabric engineering standards.
Support the migration and deprecation of legacy platforms including SCOpsBI SQL Server and SAP boundary systems following the organizations extract validate rebuild cutover and decommission pattern. Write and maintain comprehensive pipeline documentation including data lineage transformation logic SLA definitions and dependency maps. Contribute to the organizations DevOps and engineering reliability practices including CI/CD pipeline setup testing frameworks and incident runbooks.
Mentor and technically guide junior and mid-level data engineers within the squad.
REQUIRED QUALIFICATIONS:
7-10 years of data engineering experience with a strong track record delivering production data pipelines in large enterprise environments.
Expert proficiency in PySpark or Spark SQL for large-scale data transformation on distributed compute platforms.
Hands-on experience with Databricks including Delta Lake Unity Catalog and workflow orchestration.
Experience with Microsoft Fabric or Azure Synapse Analytics; familiarity with Fabric IQ and OneLake is a plus.
Proficiency with pipeline orchestration tools such as Azure Data Factory Databricks Workflows or Apache Airflow.
Solid SQL skills for data validation transformation logic and ad-hoc source system analysis.
Experience building and maintaining ingestion pipelines from enterprise operational systems such as ERP WMS TMS or comparable platforms.
Strong understanding of data quality frameworks including implementing checks alerting on anomalies and maintaining SLA-compliant pipeline health.
Experience with DevOps practices for data pipelines including version control (Git) CI/CD and automated testing.
Ability to operate independently on complex technical problems with minimal oversight while maintaining clear technical documentation.
...Wider Group is urgently hiring one CDL-A Team for a priority dedicated lane . This is our highest-paying team position and will be filled immediately. Pay : ~$0.87$0.90 CPM (roundtrip miles). ~$4,500 Weekly Guaranteed. ~ Consistent, dedicated miles....
...physical, emotional, and social aspects of well-being to promote holistic healing. We are currently seeking a PRN, Inpatient Psychiatric Nurse Practitioner to join our skilled and patient focused team. For more information, please contact Eva Fassauer at ***email...
...Job Title: K-8 Teacher (English) Program: K-8 Reports to: Co-Principals FLSA Status: Exempt Compensation: $68,723 - $10... ...Requirements Advanced fluency in English (reading, writing, and speaking) with a strong set of communication and interpersonal skills....
Job Title: Choir Director (Grades 512) (the Director) Department: High School, Middle School Reports to: Section Leader Classification: Part-Time Faculty Position, 20262027 School Year Compensation: Depends on Qualifications and Experience How to...
The Interventional Urology Principal Electrical Engineer is responsible for leading, defining, and evaluating new product development tasks as they pertain to active implantable medical devices, specifically the electrical domain. This role will serve as a mentor, technical...