Lead Data Engineer

We are Two Circles. We are a Sports & Entertainment Marketing business. We grow audiences and revenues. We do that by knowing fans best. We work with clients to help them understand & influence what their fans are doing – the way fans spend their money, the events that fans attend, the channels fans respond to, the content fans watch and more. And we use the understanding this gives us to help our clients grow. Grow their audiences and grow their revenues - both direct to consumer and business to business revenues. Our platforms and services are trusted by over 1000 clients globally, including the English Premier League, Red Bull, UEFA, VISA, the NFL, Nike, and Amazon. We are over 1000 people, based out of 15 offices, and we deliver work for sports and entertainment businesses of all shapes and sizes all over the world. 

Two Circles is looking for a Data Engineer to join our dynamic KORE Intelligence Platform team. In this role, you'll be a key part of our Partnership Intelligence > Measurement Team that specializes in the ingestion, transformation, and storage of large-scale social and broadcast data. Working under the guidance of an experienced Data Engineers and Architects, you will gain hands-on experience designing and maintaining scalable, efficient data pipelines that support enterprise web applications, analytics, and business intelligence. 

About the Role 

We are seeking a Lead Data Engineer to join our Measurement team within the Partnership Intelligence Platform. In this role, you will provide technical leadership for the team while remaining hands-on with our data platform and pipelines. 

You will help define technical direction, lead the design and evolution of our data architecture, establish engineering best practices, and mentor other engineers. You will work closely with engineering, product, and other technical teams to turn business requirements into scalable and reliable technical solutions. 

The role requires strong experience designing, building, and operating large-scale data pipelines and distributed data systems. Our technology stack includes AWS services, Airflow, Snowflake, MS SQL Server, and .NET. We are looking for someone who is passionate about data and infrastructure, comfortable making and communicating technical decisions, and interested in continuously improving our architecture, engineering practices, and technology stack. 

Key Responsibilities 

• Provide technical leadership for the Data Engineering team, helping establish technical direction, engineering standards, and best practices while remaining hands-on with the platform. 

• Mentor and support engineers in their technical development, providing guidance on system design, troubleshooting, engineering practices, and complex technical challenges. 

• Work as part of a cross-functional product team to deliver a high-quality SaaS based data platform. 

• Lead and facilitate technical design and architecture reviews, ensuring major technical decisions consider scalability, reliability, maintainability, security, cost, and operational impact. 

• Design and implement data pipelines with a clear focus on data quality and reliability. 

• Partner with both Product Management and stakeholders, as well as both technical and non-technical team members to deliver our vision, roadmap, and data strategy in addition to innovative client solutions. 

• Function as an advocate for development best practices including technical design reviews, implementing test plans, test driven development, monitoring/alerting, peer code reviews, and documentation. 

• Contribute to our DevOps culture and participate in ownership of our designs through production operations. 

• Support the ongoing maintenance and operations of the data platform. 

Ideal Tech Stack: 

• .Net / C# 

• Python 

• AWS or similar 

• Snowflake 

• SQL 

• Airflow 

Experience 

• 8+ years of software engineering experience with a strong data focus, including designing, building, and operating production data systems in fast-paced environments. 

• Demonstrated experience providing technical leadership within an engineering team, including guiding technical direction, leading design discussions, and supporting engineers through complex technical decisions. 

• Experience designing and evolving data architectures and large-scale data pipelines, including making and communicating architectural decisions and trade-offs. 

• Experience mentoring engineers and helping raise engineering standards and technical capabilities across a team. 

• Experience leading technical initiatives that span multiple engineers, systems, or teams, from technical discovery and design through implementation and production operation. 

• Strong experience writing, debugging, optimizing, and reviewing database queries and data-processing workloads. 

• Strong experience with object-oriented programming and software engineering practices, preferably using Python and C#/.NET. 

• Strong experience building and operating cloud infrastructure and services, preferably AWS. 

• Experience with modern data platforms and orchestration technologies such as Snowflake and Airflow. 

• Experience establishing or improving engineering practices such as automated testing, CI/CD, observability, monitoring and alerting, code reviews, documentation, and production readiness. 

• Experience identifying opportunities for automation and implementing solutions that reduce repetitive operational work, improve developer experience, and increase platform reliability. 

• Experience collaborating with Product Management and technical and non-technical stakeholders and communicating complex technical topics, risks, and trade-offs clearly. 

Technical Skills 

• Strong experience designing, building, and operating data pipelines using orchestration and batch-processing technologies such as Apache Airflow. 

• Strong proficiency with Snowflake and MS SQL Server, including data modeling, query optimization, performance troubleshooting, and production operations. 

• Proficiency in Python and C#/.NET. Both are required to be successful in this role. 

• Strong experience designing and integrating with APIs, including considerations such as scalability, rate limiting, retries, idempotency, error handling, and observability. 

• Strong experience with containerization technologies such as Docker and deploying containerized workloads in cloud environments. 

• Experience with monitoring and observability tools such as New Relic, including defining appropriate metrics, alerts, and operational monitoring for data pipelines and services. 

• Experience designing effective automated testing strategies, including unit, integration, and data-quality testing, using frameworks such as PyTest. 

• Strong understanding of batch, event-driven, and streaming architectures, with the ability to evaluate their trade-offs and select appropriate patterns based on system and business requirements. 

• Strong understanding of distributed systems concepts relevant to data platforms, including concurrency, messaging, failure handling, retries, idempotency, consistency, and scalability. 

• Ability to evaluate existing architectures, identify technical risks and limitations, and propose pragmatic improvements or modernization strategies. 

• Ability to define and communicate technical patterns and engineering standards that can be consistently applied across the team. 

• Ability to communicate architecture and complex technical decisions clearly through design documents, diagrams, technical discussions, and reviews.