Job description

What you will be doing

• Design and develop pipelines using Python, PySpark, and SQL
• Use GitLab as the versioning control system
• Utilize S3 buckets for storing large volumes of raw and processed data
• Implement and manage complex data workflows using Apache Airflow (MWAA) to orchestrate tasks
• Utilize Apache Iceberg (or similar) for managing and organizing data in the data lake
• Create and maintain data catalogs using AWS Glue Catalog to organize metadata
• AWS Athena for interactive querying
• Familiarize with data modeling techniques to support analytics and reporting requirements, as well as knowledge of the data journey stages within a data lake (Medallion Architecture)

What we are looking for

• Ideally, a degree in Information Technology, Computer Science, or a related field
• Ideally, +5 years of experience within the Data Engineering landscape
• Strong expertise in Python, PySpark, SQL, and the overall AWS data ecosystem
• Strong problem-solving and analytical skills
• Ability to explain technical concepts to non-technical users
• Proficiency to work with GitHub
• Terraform and CICD pipelines are a great ‘nice-to-have’