JobCubby JobCubby
Sign up
Unison Group

Senior Data Engineer (Databricks and Pyspark)

Unison Group Kuala Lumpur, Kuala Lumpur, Malaysia · MYR 15K–MYR 25K/yr

Business Consulting and Services · 11-50 employees

21 h ago
data-engineer Senior (5-10 yrs) Full-time Malaysia
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.

About the role

Design, build, and optimize scalable data ingestion and transformation pipelines using Databricks, PySpark, and Delta Lake. Collaborate with cross-functional teams to deliver curated datasets while providing technical leadership to junior team members.

What they look for

Databricks Pyspark Delta Lake Python SQL Azure Data Lake ADF Azure Functions AWS Git Azure DevOps CI/CD Unity Catalog Data Engineering ETL/ELT Lakehouse Architecture

Requirements

Requires 5-8+ years of experience in data engineering with strong hands-on expertise in Databricks, PySpark, and cloud-based data platforms. Candidates must be proficient in Mandarin and possess experience with CI/CD pipelines and distributed processing.

Benefits

One month bonus

Full description

Position : Senior Data Engineer. Location : Malaysia and Singapore. Language : Mandarin Language must. Role : Full Time Perm role + one month Bonus.

We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.

Key Responsibilities

  • Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
  • Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
  • Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
  • Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
  • Optimize Spark jobs for performance, reliability, and cost efficiency.
  • Implement data validation, quality checks, and monitoring with automated alerts and retries.
  • Design secure and governed datasets using Unity Catalog and cloud security best practices.
  • Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
  • Provide technical leadership and guidance to junior team members.

 

Required Skills

  • 5–8+ years of experience in Data Engineering.
  • Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
  • Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
  • Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
  • Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
  • Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
  • Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

 

Good to Have

  • Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
  • Knowledge of Power BI or data visualization concepts.
  • Familiarity with Terraform, Scala, or PostgreSQL.
  • Experience with SharePoint development or .NET (optional)

Similar roles