Senior Data Engineer - Teradata/Databricks & AI/ML
NTT DATA Services · Bangalore, Karnataka, India
IT Services and IT Consulting · 10,001+ employees
20 h ago
Senior (5-10 yrs)
Full-time
India
Log in
to apply, save this posting, or score it against your profile with AI.
About the role
Build end-to-end log-driven analysis pipelines in Databricks to identify unused datasets and optimize resource usage. Develop a prioritized recommendation backlog for cost reduction and apply AI/ML models to automate data classification.
What they look for
Teradata
Databricks
AI/ML
Data Engineering
SQL
Python
Data Pipeline Construction
Log Analysis
Resource Optimization
Metadata Integration
LLM
Data Classification
Requirements
Expertise in Teradata DBQL logs, Databricks, and integrating metadata from tools like Autosys and DataStage. Proficiency in applying AI/ML or LLM-assisted analysis to detect access pattern anomalies and predict data tiers.
Full description
- Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
- Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
- Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
- Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
- Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
- Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
- Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
- Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.
