What you'll do:

• Team Leadership: Lead, mentor, and manage a team of data engineers, providing guidance on technical and project-related matters. Foster a collaborative team environment, ensuring effective communication and knowledge sharing.
• Data Engineering & Development: Design, develop, and optimize data pipelines and workflows using Python, PySpark, and Shell scripting. Leverage key data engineering libraries in Python, such as Pandas, Dask, and Polars, to build and maintain scalable data solutions. Apply best practices for writing clean, efficient, and optimized Python code.
• AWS Infrastructure Management: Architect, implement, and maintain infrastructure on AWS, including services such as S3, Lambda, CloudWatch, SQS, Kinesis, EventBridge, EC2, EMR, AWS Glue, Athena, CloudFormation, and Secrets Manager.  Manage and monitor APIs running on AWS API Gateway, ensuring high availability and performance. Implement logging, debugging, and monitoring practices to maintain code quality and system reliability. Track and optimize metrics and costs associated with AWS services.
• Project & Deployment Management: Oversee project management on GitLab, including version control, branch management, and code reviews. Handle deployments using serverless technologies and manage CI/CD pipelines, ensuring smooth and efficient releases. Utilize Docker containers to build, test, and deploy applications in a scalable and consistent environment.
• Database Management: Work with SQL and NoSQL databases (MongoDB) to design and manage efficient data storage and retrieval systems. Ensure data integrity, performance optimization, and security across all database systems. 
• Regular Expressions (Regex): Utilize strong understanding and proficiency in regex to extract, manipulate, and validate data efficiently within various applications and pipelines.

What makes you a great fit:

• Strong experience in Python and PySpark for data processing and analysis.
• Proficiency in Shell scripting for automation and task management.
• Expertise in SQL and NoSQL databases, particularly MongoDB.
• Strong understanding and experience with regular expressions (regex) for data extraction and manipulation.
• Familiarity with key Python libraries for data engineering, such as Pandas, Dask, and Polars.
• Experience in optimizing Python scripts using best practices.
• In-depth knowledge of AWS services, including S3, Lambda, CloudWatch, SQS, Kinesis, EventBridge, EC2, EMR, AWS Glue, Athena, CloudFormation, and Secrets Manager.
• Experience in monitoring, logging, debugging, and maintaining AWS infrastructure.
• Ability to track and optimize AWS metrics and costs effectively.
• Experience managing projects on GitLab, including code versioning, branch management, and collaboration.
• Knowledge of CI/CD pipelines and serverless deployment strategies.
• Proficiency in using Docker containers for application development and deployment.
• Proven ability to lead and mentor a small team of engineers.
• Strong communication skills, with the ability to articulate technical concepts to non-technical stakeholders.
• Experience in managing and delivering multiple projects simultaneously.
• Understanding of machine learning fundamentals and how they apply to data engineering tasks.
• AWS Certified Solutions Architect or Data Engineer certification.
• Experience with additional AWS services beyond the core stack mentioned.
• Familiarity with other big data technologies and frameworks.