What makes you a great fit:

• Design, develop, and maintain scalable data pipelines and ETL processes using PySpark or Apache Spark.
• Work with large structured and unstructured datasets to process and clean data for analysis and machine learning.
• Collaborate with data engineers, analysts, and data scientists to understand data needs and deliver solutions for data ingestion, processing, and transformation.
• Optimize and troubleshoot Spark jobs for performance and cost efficiency.
• Integrate data from multiple sources and formats into reliable, unified data structures.
• Ensure data quality and reliability across all data workflows.
• Document processes, data structures, and workflow designs.
• Required Skills and Qualifications:
• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.
• 3+ years of experience working in a big data environment.
• Proficiency in PySpark and/or Apache Spark.
• Strong experience with SQL and databases (e.g., MySQL, PostgreSQL, etc.).
• Experience with distributed data storage solutions like Hadoop HDFS, Hive, or HBase.
• Familiarity with cloud data platforms (AWS, GCP, or Azure) and cloud-native big data tools.
• Strong programming skills in Python (especially with PySpark) or Scala.
• Knowledge of data modeling and data warehousing concepts.