Scala / Spark Developer - Remote Work Type

Directio Sp. z o.o. · Warsaw, Masovian Voivodeship, Poland · PLN 342K–PLN 360K/yr

Information Technology & Services · 51-200 employees

3 d ago

Remote Senior (5-10 yrs) Full-time Poland

About the role

Design, develop, and optimize scalable data processing applications and high-performance distributed systems using Scala and Apache Spark. Build and maintain ETL/ELT pipelines while collaborating with stakeholders to ensure software quality and engineering best practices.

What they look for

Scala Apache Spark Spark SQL ETL/ELT Pipelines Functional Programming Distributed Systems Performance Optimization Sbt ScalaTest Munit Big Data Data Processing Software Engineering Spark UI DataFrame API Dataset API

Requirements

Requires strong commercial experience in Scala functional programming and proven hands-on expertise with Apache Spark and distributed data pipelines. Candidates must be proficient in performance optimization and software engineering best practices for production environments.

Benefits

Flexible working conditions Private healthcare Multisport card Professional training opportunities

Full description

We are looking for a Scala/Spark Developer to join an AI-first software engineering company delivering innovative AI, data, cloud, and IoT solutions for clients across Europe and beyond. As part of the engineering team, you will design, develop, and optimize scalable data processing solutions, build high-performance distributed systems, and work with modern big data technologies on challenging international projects.

Responsibilities

Design, develop, and maintain scalable data processing applications using Scala and Apache Spark;

Build, optimize, and maintain ETL/ELT pipelines processing large volumes of data;

Develop efficient Spark applications using Spark SQL and the DataFrame/Dataset API;

Optimize Spark jobs by improving partitioning strategies, shuffle operations, caching, and join performance;

Troubleshoot and resolve performance issues using Spark UI and monitoring tools;

Collaborate with data engineers, software developers, and stakeholders to design scalable data processing solutions;

Participate in code reviews and contribute to software quality, performance, and engineering best practices.

Requirements

Strong commercial experience developing production applications in Scala, including functional programming concepts (immutability, pattern matching, higher-order functions), Scala type system, collections, sbt, and testing frameworks such as ScalaTest or munit, with the ability to write clean, maintainable, production-quality code;

Proven hands-on experience with Apache Spark, including Spark SQL, DataFrame/Dataset API, designing and maintaining ETL/ELT pipelines, performance optimization (partitioning, shuffle optimization, broadcast joins, caching), and troubleshooting using Spark UI;

Experience working in production environments with large-scale data processing and distributed data pipelines;

Strong understanding of software engineering best practices, code quality, performance optimization, and maintainability.

Nice to have

Experience with Kafka and Spark Structured Streaming;

Experience with Airflow or similar workflow orchestration tools;

Experience with cloud platforms such as AWS (EMR, S3, Glue), Azure, GCP (Dataproc), or Databricks;

Knowledge of the big data ecosystem, including Hadoop, Hive, Delta Lake, Parquet, and Avro;

Experience with SQL, data modeling, data warehouses, and data lake architectures;

Experience with Docker, Kubernetes, Git, and CI/CD pipelines;

Familiarity with Scala libraries such as Cats, ZIO, Akka, or Pekko;

Knowledge of Python or Java.

We offer

B2B salary: 28 500 -30 000 PLN + VAT;

Flexible working conditions;

Private healthcare, Multisport card, and professional training opportunities.

Please be advised that we will only contact selected candidates