We seek a talented and motivated Data Software Engineer to join our dynamic team. In this role, you will leverage your expertise in Python, Spark, Databricks, and DataOps to design, build, and optimize data pipelines and applications. You will be instrumental in enhancing our data infrastructure, ensuring the reliability, scalability, and efficiency of our data solutions within an Azure environment.
This position is within the Platform Team, which focuses on building data capabilities and enabling other teams by developing scalable and reliable data solutions. The team is dedicated to providing the foundational data infrastructure that empowers the entire organization.
Key Responsibilities
- Design, implement, and maintain scalable data pipelines using Python, Spark, and Databricks.
- Integrate DataOps practices to automate, monitor, and optimize data workflows and processes.
- Apply software engineering best practices to develop clean, maintainable, and efficient code.
- Work closely with data scientists, Data engineers, and other stakeholders to understand data requirements and translate them into technical solutions.
- Continuously monitor and improve the performance and reliability of data pipelines and applications.
- Utilize Azure services to deploy and manage data solutions, ensuring they align with organizational standards and best practices.
Requirements
- Bachelor and/or MS degree in Computer Science or relevant field.
- Strong experience with Python for data engineering and software development.
- Hands-on experience with Apache Spark for large-scale data processing.
- Hands-on experience with Databricks for building and managing big data solutions.
- Understanding and experience with DataOps practices, including CI/CD for data pipelines.
- Solid background in software engineering, with an emphasis on code quality, testing, and version control.
- Experience with Azure cloud services, particularly in deploying and managing data solutions.
- Strong analytical and problem-solving skills, with a proactive approach to identifying and addressing issues.
Preferred Qualifications
- Understanding of data governance principles and how they apply to data engineering.
- Familiarity with other big data tools and technologies is a plus.
- Relevant certifications in Azure, Databricks, or related technologies.