Senior data platform engineer
Lusha Tel-Aviv, Tel-Aviv District, Israel
Software Development · 201-500 employees
About the role
You will manage and evolve the core data infrastructure, including Databricks, Kafka, and Kubernetes, to ensure it is fast, reliable, and cost-effective. Additionally, you will automate manual processes and integrate AI and LLM tooling into data workflows to support engineering teams.
What they look for
Requirements
The role requires 5+ years of experience in building and operating large-scale batch and streaming data pipelines in production environments. Candidates must possess deep expertise in Spark, Databricks, Python, SQL, and cloud infrastructure management on AWS.
Full description
At Lusha, we're building for builders. We build fast and AI-first, so we look for builders. By a builder, we mean someone who turns "maybe" into "done".
We're looking for a Senior Data Platform Engineer to join the Data Platform team, which owns the infrastructure every data team at Lusha runs on and sits between DevOps and the Data groups. You'll be the Databricks admin (Unity Catalog, compute, governance, cost), the Confluent (Kafka) and CDC admin, and own Airflow, Elasticsearch, the databases, the data S3 estate, and the Kubernetes cluster underneath. Your job is to make the platform fast, cheap, governed, and boring: if it's broken, you fix it, if it's manual, you automate it.
Hands-on experience using AI coding tools daily, and building or integrating LLM and agent tooling into real workflows, is a must for this role.
This role is based in Tel Aviv. We work in a hybrid model, with 3 days a week in the office.
This might be for you if:
- You enjoy designing, building, and running large scale batch and streaming data pipelines in production, and owning what you ship, including when it breaks
- You like administering and evolving core platforms: Databricks, Confluent Kafka and CDC pipelines, Airflow, and Elasticsearch
- You care about reliability, performance, and cost of a shared data platform used across R&D
- You enjoy setting the engineering standard for data work: testing, observability, cost tagging, documentation
- You like driving architecture decisions across Databricks, Kafka, Elasticsearch, and AWS
- You actively use AI tools and build self-service tooling, including MCP servers and agent interfaces, so teams can operate the platform safely
Requirements
- 5+ years hands-on building and operating large scale data pipelines, batch and streaming, in production
- Deep Spark and Databricks experience: Delta Lake, Unity Catalog, job and cluster tuning, Structured Streaming or DLT
- Streaming and CDC experience: Kafka (Confluent a plus), Debezium or equivalent, schema evolution and its failure modes
- Experience writing Airflow DAGs and operating the platform itself
- Expert Python and SQL, clean, tested, performant
- Strong data modeling and architecture judgment, with the ability to explain tradeoffs in scale, performance, and cost
- Infra literacy: comfortable with AWS (S3, IAM, networking basics), Terraform, Docker and Kubernetes, and CI/CD
- Proven experience leading cross-team technical initiatives end to end, from ambiguous ask to shipped and adopted
- Proven experience using AI coding tools daily, with LLM and agent tooling (MCP, embeddings, vector search) built into real workflows
Nice to have:
- Elasticsearch at scale: indexing pipelines, cluster operations, reindex and rollout strategies
- FinOps or cloud cost management for data platforms
- Vault or similar secrets management, and Kubernetes operations beyond deploying your own service
- Experience inheriting a system with real debt and making it boring
