Consultant Data Engineer (Python/Databricks)

About the company:

Softermii (https://www.softermii.com/) is an outsourcing/outstaffing company with more than 8 years of experience in web and mobile application development with a team of more than 120 specialists from 6 countries.

We have withstood multiple challenges, such as COVID-19 pandemic and war aggression against Ukraine, we have secured sustainable development processes, saved all our projects and jobs. Even more – we’re constantly growing.     

 

Every day we add new project to our portfolio in the following areas of expertise:

• Healthcare
• Real estate
• Livestream E-commerce
• Fintech (WealthTech, Capital Trading)
• Real Time Communications (RTC)

 

In addition, we develop 2 of our own solutions that help our customers get faster applications for video or audio calls (https://www.softermii.com/solutions/vidrtc) or an application for real estate with 3D visualization

(https://www.softermii.com/solutions/apartmii)

 

Company’s mission:

Softermii: Succeed Without (L) imitations

We will turn any idea into a valuable product.

 

Our plans as a company:

We want to move from a regular outsourcing business to an ecosystem of outsourcing, education and product development companies.

 

Softermii is looking for a part-time Data Engineering Consultant / Tech Lead who will do technical interviews, assist with upcoming projects, and occasionally be hands-on with complex development tasks — including data pipeline design and solution optimization on Databricks.

 

Type of cooperation: Part-time

⚡️Your responsibilities on the project will be:

Interview and hire Data Engineers

• Supervise work of other Engineers and have hands on for the most complicated tasks from backlog, focus on unblocking other data Engineers in case of technical difficulties
• Develop and maintain scalable data pipelines using Databricks (Apache Spark) for batch and streaming use cases.
• Work with data scientists and analysts to provide reliable, performant, and well-modeled data sets for analytics and machine learning.
• Optimize and manage data workflows using Databricks Workflows and orchestrate jobs for complex data transformation tasks.
• Design and implement data ingestion frameworks to bring data from various sources (files, APIs, databases) into Delta Lake.
• Ensure data quality, lineage, and governance using tools such as Unity Catalog, Delta Live Tables, and built-in monitoring features.
• Collaborate with cross-functional teams to understand data needs and support production-grade machine learning workflows.
• Apply data engineering best practices: versioning, testing (e.g., with pytest or dbx), documentation, and CI/CD pipelines

🕹Tools we use: Jira, Confluence, Git, Figma

 

🗞Our requirements to you:

• 5+ years of experience in data engineering or big data development, with production-level work.
• Architect and develop scalable data solutions** on the Databricks platform, leveraging Apache Spark, Delta Lake, and the lakehouse architecture to support advanced analytics and machine learning initiatives.
• Design, build, and maintain production-grade data pipelines** using Python (or Scala) and SQL, ensuring efficient data ingestion, transformation, and delivery across distributed systems.
• Lead the implementation of Databricks features** such as Delta Live Tables, Unity Catalog, and Workflows to ensure secure, reliable, and automated data operations.
• Optimize Spark performance and resource utilization**, applying best practices in distributed computing, caching, and tuning for large-scale data processing.
• Integrate data from cloud-based sources** (e.g., AWS S3), ensuring data quality, lineage, and consistency throughout the pipeline lifecycle.
• Manage orchestration and automation of data workflows** using tools like Airflow or Databricks Jobs, while implementing robust CI/CD pipelines for code deployment and testing.
• Collaborate cross-functionally with data scientists, analysts, and business stakeholders** to understand data needs and deliver actionable insights through robust data infrastructure.
• Mentor and guide junior engineers**, promoting engineering best practices, code quality, and continuous learning within the team.
• Ensure adherence to data governance and security policies**, utilizing tools such as Unity Catalog for access control and compliance.
• Continuously evaluate new technologies and practices**, driving innovation and improvements in data engineering strategy and execution.
• Experience in designing, building, and maintaining data pipelines using Apache Airflow, including DAG creation, task orchestration, and workflow optimization for scalable data processing.
• Upper-Intermediate English level.

👨‍💻Who will you have the opportunity to meet during the hiring process (stages):
Call, HR, Tech interview, PM interview.

What we can offer you:

1. We have stable and highly-functioning processes – everyone has their own role and clear responsibilities, so decisions are made quickly and without unnecessary approvals. 
2. You will have enough independence to make decisions that can affect not only the project but also the work of the company.
3. We are a team of like-minded experts who create interesting products during working hours, and enjoy spending free time together.
4. Do you like to learn something new in your profession or do you want to improve your English? We will be happy to pay 50% of the cost of courses/conferences/speaking clubs.
5. Do you want an individual development plan? We will form one especially for you + you can count on mentoring from our seniors and leaders.
6. Do you have a friend who is currently looking for new job opportunities? Recommend them to us and get a bonus.
7. And what if you want to relax? Then we have 21 working days off.
8. What if you are feeling bad? You can take 5 sick leaves a year.
9. Do you want to volunteer? We will add you to a chat, where we can get a bulletproof vest, buy a pickup truck or send children's drawings to the front.
10. And we have the most empathetic HRs (who also volunteers!). So we are ready to support your well-being in various ways.

 

A little more information that you may find useful:

• our adaptation period lasts 3 months, this period of time is enough for us to understand each other better;
• there is a performance review after each year of our collaboration where we use a skills map to track your growth;
• we really have no boundaries in the truest sense of the word – we have flexible working day, remote or office is up to you.

 

Of course, we have a referral bonus system. So if you have a friend who likes our job opening, you can give us their telegram contact, and we in turn will thank you with a bonus if we start working with them!