Data Engineer

permanent position

Position: Data Engineer

Nexttec Technology is a leading outsourcing company that provides innovative and cutting-edge solutions to businesses worldwide. We are currently seeking a highly skilled and experienced Data Engineer to join our dynamic team on a full-time, permanent basis.

As a Data Engineer, you will be responsible for designing, building, and maintaining our data infrastructure and systems. You will work closely with our data scientists and analysts to ensure the accuracy, reliability, and efficiency of our data processes.

Key Responsibilities:

1. Develop and Maintain Data Pipelines

• Design, build, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and storage.

• Optimize data workflows for efficient processing and integration across multiple data sources.

• Work with both structured and unstructured data to ensure smooth data processing.

• Implement data ingestion with real-time streaming and batch ETL/ELT processes.

• Develop pipelines using Microsoft Fabric components including Data Pipelines, Lakehouse, Data Warehouse

2. Data Integration and Management:

• Integrate data from various internal and external sources, ensuring consistency and accuracy.

• Automate data ingestion and processing workflows to improve efficiency.

• Maintain and monitor data pipelines, resolving any failures or inefficiencies.

• Implement automated CI/CD pipelines for data workflows.

• Design and implement data ingestion pipelines from ERP systems (Oracle EBS / Fusion/ NetSuite), REST APIs and external data sources.

• Handle API pagination and rate limits, schema changes and evolving data structures.

• Implement incremental data loading strategies using timestamps, change tracking, CDC (where applicable)

• Work with existing platforms such as Google BigQuery during transition phase.

3. Lakehouse Architecture & Data Structuring

• Design and maintain Bronze / Silver / Gold layers.

• Ensure data is clean, standardized, analytics ready.

• Structure data to support downstream semantic models and reporting.

4. Data Security and Compliance

• Implement best practices for data security, ensuring compliance with regulations such as GDPR, HIPAA, and company policies.

• Work with cybersecurity teams to safeguard sensitive data from unauthorized access.

• Ensure proper data backup and disaster recovery strategies are in place.

5. Collaboration with Business and IT Teams

• Work closely with data analysts, analytics engineers, IT teams and business stakeholders to understand data requirements.

• Provide technical expertise and recommendations for data processing and storage solutions.

6. Data Performance and Optimization

• Continuously optimize data pipelines for performance, scalability, and cost efficiency.

• Troubleshoot and resolve data-related performance issues in databases and processing workflows.

• Optimize query performance for data retrieval and reporting needs.

7. Data Governance and Quality

• Ensure high data quality by implementing validation and cleansing techniques.

• Develop data monitoring processes to track data accuracy and completeness.

• Work with data architects to align data engineering solutions with governance policies.

8. Documentation and Reporting

• Document data pipeline structures, workflows, and integrations for future reference.

• Provide reports and insights on data pipeline performance and efficiency.

9. Stay Updated on Technology Trends

• Stay current with advancements in data engineering, AI, cloud computing, and big data technologies.

• Evaluate and integrate new tools and techniques to improve data processing.

10. Support Data Analytics and Business Intelligence

• Provide clean, structured, and accessible data for analysts and data scientists.

• Work with BI teams to ensure that data warehouses and reporting tools function efficiently.

Qualifications:

Education Required:

• Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field.
• Master's degree and/or cloud certifications (Azure Solutions Architect, Azure Data Engineer, Azure AI Engineer, Databricks) are a plus.

Field of Experience:

• Data Engineering: Hands-on experience in building and managing data pipelines and workflows.
• Database Management: Experience working with relational and NoSQL databases.
• Data Integration: Proficiency in integrating multiple data sources and managing data transformations. Experience implementing CI/CD pipelines (Azure DevOps)
• Cloud Computing: Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Big Data Processing: Understanding of distributed data processing frameworks and tools.

Years of Experience:

• Typically, 3 to 7 years of experience in data engineering or a related IT field.
• At least 3 years of hands-on experience in data pipeline development, ETL processes, and working with cloud-based data solutions.

Technical Skills:

• Strong hands-on experience with Microsoft Fabric on Azure
• Strong understanding of Microsoft Fabric architecture including OneLake
• Strong proficiency in SQL and Python for data processing
• Proficiency in Apache Spark (PySpark preferred)
• Strong understanding of medallion architecture (Bronze, Silver, Gold layers)
• Proven experience in building metadata-driven data pipelines
• Experience with ETL/ELT design and optimization
• Knowledge of Azure ecosystem and integration patterns
• Strong understanding of data governance, lineage, and compliance