Data Engineer, AI & Business Intelligence- San Diego

Position Summary

Event Network is seeking a mid-level Data Engineer to develop and support the pipelines, integrations, curated datasets, models, and services behind our artificial intelligence and business intelligence initiatives. This is a hands-on engineering role for someone who can turn defined business and analytical needs into secure, supportable production solutions.

You will partner closely with the Senior Business Analyst leading our AI and BI efforts and with other Information Systems resources. A central part of the role is strengthening our software delivery discipline: organizing work in Azure DevOps repos, applying legitimate Git-based version control, establishing and maintaining CI/CD pipelines, and building automated tests and deployment gates for data solutions.

What You Will Do

• Design, build, maintain, and support scheduled, incremental, and event-driven data pipelines from databases, APIs, operational applications, files, and approved Microsoft 365 sources.

• Develop transformations and curated data products for Azure Databricks, Unity Catalog, Power BI semantic models, analytics, and approved AI applications.

• Own code and deployment hygiene in Azure DevOps: organize repositories, use branches and pull requests appropriately, maintain useful commit history, apply peer-review practices, and keep production code out of personal or unmanaged locations.

• Establish and maintain CI/CD pipelines that validate, package, and promote data pipelines, notebooks, SQL, infrastructure, and configuration across development, test, and production environments.

• Build automated tests for transformation logic, pipeline behavior, schemas, data quality, reconciliations, and regression scenarios; integrate those tests into pull-request and deployment workflows.

• Implement configuration management, environment-specific settings, secret handling, approvals, rollback or recovery procedures, and auditable releases.

• Create reliable ingestion and retrieval processes for approved documents and structured data used by AI solutions, including metadata, indexing, security-aware access, evaluation data, and telemetry where appropriate.

• Implement monitoring, logging, alerts, retry and reprocessing patterns, and production support procedures for assigned pipelines, datasets, integrations, indexes, and services.

• Profile and reconcile data, detect missing or duplicate records, validate business rules and control totals, and prevent incomplete loads from being presented as successful.

• Maintain technical documentation, source-to-target mappings, deployment instructions, runbooks, and change histories as part of the definition of done.

• Communicate progress, tradeoffs, risks, data limitations, and blockers early and clearly to technical and business partners.

Required Qualifications

• Approximately 3-5 years of relevant experience in data engineering, analytics engineering, database development, or integration development; equivalent practical experience will be considered.

• Strong SQL skills, including complex queries, joins, aggregations, window functions, common table expressions, and performance troubleshooting on SQL Server or a comparable relational platform.

• Working proficiency in Python and experience building reusable, readable code for structured data, APIs, and common file formats.

• Experience building and supporting production ETL/ELT pipelines, including incremental processing, orchestration, logging, error handling, monitoring, and recovery.

• Hands-on Git experience and a clear understanding of branches, pull requests, code review, merge practices, release history, and resolving conflicts.

• Experience creating or maintaining CI/CD pipelines in Azure DevOps or a comparable platform, including automated validation and controlled promotion between environments.

• Experience designing automated unit, integration, data-quality, schema, or regression tests and incorporating them into engineering workflows.

• Understanding of development, test, and production separation; secure configuration and secrets management; and repeatable deployment practices.

• Experience preparing governed, reusable data for Power BI or a comparable BI platform and working with facts, dimensions, relationships, and semantic models.

• Ability to investigate ambiguous data problems, ask focused questions, and collaborate effectively with both business and technical stakeholders.

Preferred Experience

• Azure Data Factory, Azure Databricks, Unity Catalog, Azure Data Lake Storage, Microsoft Fabric or OneLake, and Power BI semantic models.

• Azure DevOps Repos and Pipelines, infrastructure as code such as Bicep or Terraform, Databricks Asset Bundles or comparable deployment automation, and test frameworks such as pytest.

• Azure Key Vault, Microsoft Entra ID, Azure Monitor, Application Insights, Log Analytics, or Microsoft Purview.

• Microsoft Dynamics NAV or Business Central, LS Retail, retail sales and inventory data, e-commerce, SFTP, partner-file integrations, and financial or operational reconciliation.

Section 2 | Detailed Role & Working Arrangement

This section provides internal detail on how the role will operate, how responsibilities are divided, and what production-ready delivery means in Event Network's environment. It can be used during interviewing, onboarding, goal setting, and performance discussions.

Role Purpose and Boundaries

The Data Engineer is the primary engineering implementation partner for the Senior Business Analyst responsible for AI and business intelligence. The role provides the technical capacity to investigate sources, build reusable data products, productionize approved concepts, and support them after release.

This is not primarily a Power BI report developer, business analyst, data scientist, machine-learning researcher, prompt engineer, database administrator, general IT support role, enterprise architect, or management position. The engineer may contribute in those areas, but the core accountability is production data engineering.

Working Partnership

The Senior Business Analyst will generally lead:

• Business opportunities, use cases, priorities, intended users, and expected outcomes.

• Stakeholder engagement, analytical requirements, business definitions, and initial acceptance criteria.

• AI experimentation, solution concepts, evaluation of business-facing results, and confirmation that the delivered solution addresses the intended need.

The Data Engineer will generally lead:

• Technical discovery, data-source investigation, feasibility, implementation approach, effort estimates, and identification of security, quality, dependency, or operational risks.

• Pipeline, transformation, data-model, retrieval, integration, testing, deployment, monitoring, recovery, and technical documentation work.

• Routine technical decisions about code organization, query construction, implementation patterns, validation, logging, troubleshooting, and performance improvements that do not change approved business meaning.

The two roles will collaborate on data availability, business meaning, security, quality, technical risk, expected value, and readiness for business adoption. Significant decisions involving enterprise architecture, platform selection, sensitive data, material cost, or cross-system impact will follow the appropriate Information Systems review process.

Engineering Standards and Delivery Controls

Definition of done: A solution is not complete merely because it runs once. It must be versioned, reviewed, tested, deployable, observable, documented, secure, and recoverable in proportion to its production risk.

Azure DevOps repos and Git: Place production code, notebooks, SQL, pipeline definitions, infrastructure definitions, deployment templates, and relevant configuration in approved repositories. Use meaningful commits, a documented branching approach, pull requests, reviewer approval, and traceable links between work items and changes. Avoid shared-drive code, opaque copies, and direct production edits except through a documented emergency process.

CI/CD pipelines: Build and maintain pipelines that perform automated validation, package deployable artifacts, apply environment-specific configuration, and promote changes across development, test, and production. Use approvals, checks, service connections, protected environments, and release evidence appropriate to the risk of the change.

Automated testing: Create maintainable tests for Python and SQL logic, pipeline and notebook behavior, contracts and schemas, data-quality rules, reconciliations, and representative regression scenarios. Run fast checks on pull requests and broader integration or end-to-end checks before production promotion. Failed required tests must block the release or receive a documented exception.

Environment separation: Keep development, test, and production settings distinct. Externalize configuration, store secrets in approved services, avoid hard-coded credentials or environment paths, and make deployments repeatable without manual editing.

Release and recovery: Use consistent versioning or release identification, retain deployment logs, verify post-deployment health, and document rollback, rerun, or forward-fix procedures. Production changes should be attributable to a reviewed change and reproducible from source.

Observability and support: Implement logging, alerts, freshness indicators, failure notifications, retry behavior, control totals, and runbooks. Treat recurring incidents as engineering problems requiring root-cause analysis and durable correction.

Documentation: Maintain source-to-target mappings, transformation rules, dependencies, schedules, access requirements, test coverage, deployment instructions, known limitations, and support procedures alongside the solution.

Core Delivery Responsibilities

Data pipelines and integration

• Ingest data from SQL Server, Dynamics NAV and LS Retail, point-of-sale and e-commerce systems, inventory and replenishment systems, APIs, SFTP locations, partner files, Microsoft 365 content, and other approved sources.

• Build reusable scheduled, incremental, and event-driven workflows with error handling, logging, retry, failure notification, safe reprocessing, and change-impact awareness.

• Document source mappings, schedules, dependencies, transformation rules, and operational ownership.

Transformation, modeling, and BI enablement

• Profile, clean, standardize, and combine source data into governed bronze, silver, and justified gold data products or semantic models.

• Create facts, dimensions, historical tracking, conformed identifiers, approved calculations, and reusable datasets for Power BI and other authorized consumers.

• Improve query and refresh performance, reduce duplicated report logic, implement row-level security where required, and reconcile outputs to approved sources.

• Develop a practical understanding of retail sales, transactions, products, locations, inventory movement, purchasing, pricing, replenishment, vendors, e-commerce, and operating periods.

AI data engineering

• Prepare structured data and approved SharePoint or other document content for secure AI use.

• Build ingestion and refresh processes for text, metadata, indexes, embeddings, and keyword, semantic, vector, or hybrid retrieval when the use case requires them.

• Preserve source references and permission-aware access; capture retrieval results, errors, response times, usage, and quality indicators.

• Develop representative evaluation and regression datasets, and help determine whether poor results originate in source content, transformation, indexing, retrieval, permissions, or application behavior.

• Convert successful prototypes into maintainable implementations without taking ownership of company AI strategy or foundation-model research.

Data quality and production support

• Validate counts, control totals, required fields, value ranges, relationships, timeliness, duplicates, volume changes, and source-to-target reconciliation.

• Prevent partial or failed loads from appearing successful and communicate material data limitations promptly.

• Monitor assigned services, diagnose routine failures, recover incomplete loads, confirm production changes, participate in root-cause analysis, and escalate architectural, security, vendor, or source-system issues appropriately.

Standard Work Process

Use-case definition: The Senior Business Analyst defines the problem, users, priority, intended outcome, stakeholder needs, and initial acceptance criteria.

Technical discovery: The Data Engineer evaluates sources, ownership, access, quality, history, refresh needs, security, integration options, dependencies, implementation risks, and estimated effort. Findings and unresolved questions are raised before substantial development.

Solution approach: The Data Engineer recommends ingestion and transformation methods, target structures, test strategy, validation controls, deployment path, monitoring, recovery, and documentation. The Senior Business Analyst confirms alignment with the intended outcome.

Development and validation: The Data Engineer builds the solution and its automated tests, produces sample outputs, documents assumptions, and demonstrates working results. The Senior Business Analyst validates business meaning and acceptance criteria.

Production readiness: The Data Engineer completes technical testing, code review, CI/CD setup, security controls, deployment, monitoring, and runbooks. The Senior Business Analyst confirms readiness for business validation or adoption.

Operate and improve: The Data Engineer monitors performance and quality, resolves routine defects, captures technical debt, and proposes permanent improvements. Both roles review whether the solution continues to deliver the expected business value.

Level Expectations

As a mid-level engineer, the person should independently complete clearly defined assignments, break moderately complex requirements into technical tasks, support production pipelines, troubleshoot routine issues, write readable code, test and document work, and communicate risks and blockers. Guidance is expected for enterprise-wide architecture, company-wide governance, major vendor or infrastructure decisions, highly sensitive data, and unfamiliar advanced AI patterns.

First 90 Days

First 30 days: Learn the current AI and BI priorities, Azure and Databricks environment, source systems, repositories, pipelines, reports, prototypes, security practices, and deployment processes. Identify high-risk manual or unsupported processes, begin documenting assigned data flows, and complete one limited technical assignment.

Days 31-60: Take ownership of one pipeline, dataset, integration, or retrieval process. Add or improve automated tests, validation, reconciliation, logging, and failure notifications. Demonstrate the ability to deploy, operate, and troubleshoot the assigned process using the approved repository and delivery workflow.

Days 61-90: Deliver or assume ownership of one production-ready data product. Establish or materially improve its CI/CD path, monitoring, recovery, documentation, and support procedures. Recommend the next practical engineering priorities, including technical debt and deployment controls that should be standardized across the platform.

Ongoing Success Measures

• The Senior Business Analyst spends less time on manual extraction, cleanup, reconciliation, and technical troubleshooting.

• Assigned data products run reliably, reconcile to approved sources, and surface failures or quality issues promptly.

• The team can identify who changed what, why it changed, which tests passed, where it was deployed, and how to recover it.

• Manual deployments and unmanaged code copies decline as approved repositories and CI/CD workflows become the normal way of working.

• Automated tests catch defects before production and representative regression tests protect important analytical and AI behaviors.

• Successful AI and analytics concepts move into controlled production use with security, monitoring, documentation, and clear ownership.

• Technical risks, limitations, dependencies, and blockers are communicated early enough for the team to act.

Professional Competencies

Execution: Produces working, supportable results and closes the operational details required for production.

Problem solving: Investigates technical and data issues methodically and distinguishes symptoms from root causes.

Communication: Explains progress, findings, risks, limitations, and dependencies clearly to technical and business partners.

Collaboration: Works effectively with the Senior Business Analyst, Information Systems peers, source-system owners, and stakeholders.

Business curiosity: Seeks to understand the business meaning and intended use of the data instead of treating fields as context-free values.

Reliability: Treats pipelines, datasets, retrieval services, tests, repositories, and deployment processes as production assets.

Learning mindset: Learns unfamiliar systems and evolving AI capabilities, then applies them pragmatically within established architecture and security direction.

Pay Range: $1250,000.00-$145,000.00/year