Senior Data Engineer - Teradata/Databricks & AI/ML
NTT DATA Services · Bangalore, Karnataka, India
IT Services and IT Consulting · 10,001+ employees
20 h ago
Senior (5-10 yrs)
Full-time
India
Log in
to apply, save this posting, or score it against your profile with AI.
About the role
Build end-to-end log-driven analysis pipelines in Databricks to identify unused datasets and optimize resource usage. Develop AI/ML models to automate data classification and provide cost-reduction recommendations to stakeholders.
What they look for
Teradata
Databricks
AI/ML
Data Engineering
SQL
Data Pipelines
Resource Analysis
Data Classification
LLM
Metadata Integration
Requirements
Expertise in Teradata DBQL logs, Databricks, and AI/ML models for anomaly detection and data classification. Experience integrating metadata from tools like Autosys, DataStage, and MagicWand is required.
Full description
- Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
- Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
- Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
- Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
- Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
- Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
- Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
- Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.
