Job Title: Lead Data Engineer

Location: Dallas, TX

Client: Prodapt/ AT&T

Project Duration: 12 months

Job Type: Contract

Work Arrangement: Onsite role โ€“ 5 days a week

Interview: In person

ย 

Job Summary

We are seeking a highly skilled Technical Lead โ€“ ETL & Python Development to lead the design, development, and delivery of enterprise data integration and data engineering solutions. The ideal candidate will possess strong expertise in Python, ETL development, data warehousing, cloud-based data platforms, and technical leadership. The role involves collaborating with business stakeholders, solution architects, data analysts, and engineering teams to build scalable, high-performance data pipelines and modern data solutions while mentoring development teams and driving engineering best practices.

Key Responsibilities

ย 

Technical Leadership

ยท Lead a team of ETL and Python developers across multiple projects.

ยท Drive technical design, code reviews, development standards, and best practices.

ยท Provide technical mentoring and guidance to team members.

ยท Conduct solution design workshops and architecture discussions.

ยท Collaborate with architects and project managers to ensure successful project delivery.

ETL Development & Data Integration

ยท Design, develop, and maintain enterprise-scale ETL/ELT solutions.

ยท Build scalable and reusable data ingestion frameworks.

ยท Design complex data transformation and enrichment pipelines.

ยท Integrate data from multiple sources including databases, APIs, cloud storage, SaaS applications, and external systems.

ยท Ensure data quality, consistency, validation, and reconciliation.

Python Development

ยท Develop robust and scalable Python applications for data processing and automation.

ยท Build reusable libraries, frameworks, and utilities.

ยท Develop APIs and microservices supporting data integration processes.

ยท Create automated data validation, monitoring, and alerting solutions.

ยท Optimize Python code for performance and scalability.

Data Engineering & Warehousing

ยท Design and implement data warehouse solutions.

ยท Develop data models supporting reporting and analytics needs.

ยท Build batch and near real-time data processing pipelines.

ยท Support data lake and lakehouse architectures.

ยท Implement metadata management and data governance processes.

Cloud & Modern Data Platforms

ยท Design and deploy cloud-native data solutions.

ยท Work with cloud services for data ingestion, storage, processing, and analytics.

ยท Support migration from legacy ETL platforms to modern cloud-based architectures.

ยท Implement Infrastructure as Code (IaC) and automation where applicable.

Quality & Performance Optimization

ยท Perform performance tuning for ETL jobs and Python applications.

ยท Identify bottlenecks and recommend optimization strategies.

ยท Ensure adherence to security, compliance, and governance standards.

ยท Lead root cause analysis and production issue resolution.

Required Technical Skills

Python Development

ยท Strong hands-on experience with:

o Python 3.x

o Pandas

o NumPy

o SQLAlchemy

o PySpark

o FastAPI / Flask

o Object-Oriented Programming (OOP)

ยท Experience developing production-grade Python applications.

ยท Strong debugging and optimization skills.

ETL & Data Integration

Hands-on experience with one or more ETL platforms:

ยท Informatica PowerCenter

ยท Azure Data Factory (ADF)

ยท SSIS

ยท Talend

ยท DataStage

ยท AWS Glue

ยท Apache Airflow

ยท Matillion

Strong understanding of:

ยท ETL/ELT Concepts

ยท Data Integration Patterns

ยท Data Migration

ยท Data Transformation

ยท Data Validation Frameworks

Databases

Advanced SQL expertise and experience with:

ยท SQL Server

ยท Oracle

ยท PostgreSQL

ยท MySQL

ยท Snowflake

ยท Amazon Redshift

Knowledge of:

ยท Query Optimization

ยท Indexing

ยท Partitioning

ยท Performance Tuning

Cloud Skills

Hands-on experience with at least one cloud platform:

Microsoft Azure

ยท Azure Data Factory

ยท Azure Data Lake

ยท Azure SQL Database

ยท Synapse Analytics

ยท Azure Functions

AWS

ยท AWS Glue

ยท S3

ยท Lambda

ยท Redshift

ยท EMR

Google Cloud (Preferred)

ยท BigQuery

ยท Dataflow

ยท Cloud Storage

Big Data & Modern Data Technologies

Preferred experience with:

ยท Apache Spark

ยท PySpark

ยท Hadoop Ecosystem

ยท Databricks

ยท Kafka

ยท Delta Lake

ยท Lakehouse Architecture

DevOps & Automation

ยท Git / GitHub / GitLab

ยท Azure DevOps

ยท Jenkins

ยท CI/CD Pipelines

ยท Docker

ยท Kubernetes (Preferred)

ยท Terraform (Preferred)

Leadership Responsibilities

ยท Lead a team of 5-15 data engineers and developers.

ยท Conduct code reviews and technical assessments.

ยท Drive sprint planning and technical estimations.

ยท Support hiring and onboarding activities.

ยท Establish development standards and governance practices.

ยท Ensure project deliverables meet quality and performance objectives.

Required Qualifications

ยท Bachelor's Degree in Computer Science, Engineering, Information Technology, or related discipline.

ยท 12 years of overall IT experience.

ยท 5+ years of hands-on ETL development experience.

ยท 4+ years of Python development experience.

ยท 2+ years of technical leadership experience.

ยท Strong understanding of Software Development Lifecycle (SDLC) and Agile methodologies.