Senior Azure Databricks Engineer(ID: 4002)
STAFIDE
Role description
As a Senior Azure Databricks Engineer, you will:
- Design, develop, and maintain scalable and reliable data processing solutions using Azure Databricks.
- Build and manage robust batch and streaming data pipelines within Databricks environments.
- Develop and optimize data transformation and processing solutions using Python, PySpark, and SQL.
- Design and optimize data models to support scalable processing, performance, and reliability.
- Manage multiple parallel data processing workflows and shared data sources efficiently.
- Implement and maintain CI/CD pipelines using Azure DevOps and YAML-based configurations.
- Apply Infrastructure as Code (IaC) using ARM/Bicep for deployment and infrastructure automation.
- Monitor, troubleshoot, and optimize data processing workloads and Databricks environments.
- Collaborate with cross-functional engineering and business teams to deliver reliable and maintainable data solutions.
- Contribute to Agile development practices and continuously improve engineering standards, system stability, and performance.
What You Bring to the Table:
- Strong hands-on experience with Azure Databricks as a core data engineering platform.
- Strong proficiency in Python, PySpark, and SQL.
- Hands-on experience developing batch and streaming data pipelines.
- Experience with data modeling, transformation, and optimization within Databricks environments.
- Good understanding of Azure cloud services relevant to data engineering.
- Experience with Azure DevOps, CI/CD, and YAML-based pipeline configurations.
- Hands-on experience with Infrastructure as Code, particularly ARM/Bicep.
- Experience working in Agile engineering and delivery environments.
- Understanding of modern cloud-based data architectures and end-to-end data engineering solutions.
- Strong communication, collaboration, troubleshooting, and problem-solving skills.
You Should Possess the Ability to:
- Build scalable, high-performance, and reliable data processing solutions using Azure Databricks.
- Develop efficient PySpark and SQL-based data transformations.
- Design and manage complex batch and streaming workloads.
- Optimize data pipelines, processing performance, and resource utilization.
- Troubleshoot complex data engineering issues and improve system reliability.
- Implement automated deployment and infrastructure management practices.
- Make pragmatic technical decisions while maintaining scalability and maintainability.
- Work effectively with engineering, architecture, and business stakeholders.
- Drive continuous improvement and maintain high standards of code and solution quality.
What We Bring to the Table:
- Opportunity to work on enterprise-scale Azure and Databricks data engineering initiatives.
- Exposure to modern cloud-based data platforms and engineering practices.
- A collaborative Agile environment focused on technical excellence and innovation.
- Opportunities to work with advanced data processing, pipeline engineering, and cloud technologies.
- Continuous learning and opportunities for technical and professional growth.
- A culture focused on quality, ownership, scalability, and sustainable engineering solutions.
