Senior Databricks Solutions Architect

Overview

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, Prodapt provides consulting, business reengineering, and managed services for the largest telecom and tech enterprises building networks and digital experiences of tomorrow. A ServiceNow-invested company, Prodapt has been recognized by Gartner as a Large, Telecom-Native, Regional IT Service Provider. A “Great Place To Work® Certified™” company, Prodapt employs over 5,000 technology and domain experts across the Americas, Europe, India, Africa, & Japan. Prodapt is part of the 130-year-old business conglomerate The Jhaver Group, which employs over 32,000 people across 80+ locations globally.

This is a onsite role at Dallas, TX, in person interview is must for this role.

We are looking for a Senior Databricks Solution Architect to lead the architecture and technical execution of a Palantir Foundry to Databricks migration. The candidate will be responsible for assessing the existing Palantir application, defining the target Databricks architecture, mapping existing capabilities to Databricks services, and providing technical leadership throughout the migration.

The solution involves migration of capabilities including data ingestion and transformation, AI-based information extraction from structured and unstructured documents, large-scale data comparison, dashboards and drill-down reporting, chatbot capabilities, and business workflows/applications.

The ideal candidate should have strong expertise in Databricks, Lakehouse architecture, data engineering, AI/ML workloads, and cloud-native data platforms, along with the ability to understand and decompose existing Palantir Foundry capabilities into equivalent or improved Databricks-based solutions.

Responsibilities

1. Databricks Architecture

• Define the target-state Databricks architecture for the migrated application.
• Design scalable and secure Lakehouse architecture using Databricks.
• Define the appropriate architecture across Bronze, Silver and Gold layers.
• Design data ingestion, transformation, processing and serving patterns.
• Define standards for Delta Lake, Unity Catalog, compute, storage, workloads and environments.
• Design solutions for batch and near-real-time data processing.

2. Palantir → Databricks Migration

• Analyze the existing Palantir Foundry application, including its pipelines, datasets, applications, workflows and AI capabilities.
• Create a feature-by-feature mapping between existing Palantir capabilities and Databricks capabilities.
• Identify capabilities that can be migrated directly versus those requiring redesign.
• Define migration patterns for:• Data ingestion pipelines
• Data transformation pipelines
• Data quality and validation
• Data comparison and reconciliation
• AI/LLM-based information extraction
• Dashboards and reporting
• Chatbot capabilities
• Business workflows and actions
• Document processing

• Develop migration roadmaps, architecture decisions and technical dependencies.
• Identify opportunities to improve scalability, maintainability and performance during migration.

3. Data Engineering & Lakehouse

• Design robust data pipelines using Lakeflow / Spark Declarative Pipelines, Apache Spark and Delta Lake.
• Design ingestion from databases, APIs, files, documents and other enterprise data sources.
• Establish data quality, validation, lineage and observability mechanisms.
• Design incremental processing and change-data-capture patterns where required.
• Optimize Delta tables, Spark workloads and Databricks compute for performance and cost.

4. AI / GenAI Architecture

• Architect solutions for extracting information from structured and unstructured documents.
• Design RAG and LLM-based solutions where applicable.
• Evaluate appropriate Databricks capabilities for:• Document processing
• Information extraction
• Embeddings and vector search
• LLM inference
• Model serving
• AI agents / conversational applications

• Define approaches for migrating existing Palantir AIP-based extraction capabilities into Databricks.
• Ensure AI solutions are scalable, secure, explainable and production-ready.

5. Analytics & Application Architecture

• Design the target architecture for dashboards, drill-down reporting and analytical applications.
• Define semantic/data serving layers required by business applications.
• Architect chatbot and conversational analytics solutions.
• Design integration between Databricks and downstream applications/APIs.

6. Security & Governance

• Implement enterprise data governance using Unity Catalog.
• Define catalog, schema, table, volume and access-control strategies.
• Design RBAC and data-access policies.
• Define data lineage, auditing and governance standards.
• Ensure compliance with enterprise security and privacy requirements.

7. Technical Leadership

• Lead a team of data engineers, ML/AI engineers and developers.
• Conduct architecture reviews and technical design sessions.
• Create HLD/LLD documents, architecture diagrams and migration plans.
• Establish Databricks development standards and best practices.
• Review code and implementation approaches.
• Work closely with business stakeholders, Palantir SMEs, cloud architects and application teams.
• Troubleshoot complex architectural and performance issues.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology or related discipline.
• 14+ years of experience in data engineering/data architecture.
• 5+ years of experience working with modern cloud data platforms.
• Strong hands-on Databricks experience.
• Proven experience designing enterprise-scale data platforms.
• Experience leading technical teams and architecture initiatives.
• Strong communication and stakeholder-management skills.
• Ability to translate business requirements into scalable technical architectures.