We are seeking a highly skilled Spark Developer with strong experience in Python, AWS, and SQL to join our team. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions, ensuring data quality, scalability, and performance. This role requires a solid background in distributed computing, cloud environments, and data engineering best practices.
Compensation for NYC: 120-150000 USD Gross per year and based on interview results.
Design, develop, and maintain scalable data pipelines using Apache Spark (batch and/or streaming).
Build, optimize, and manage ETL/ELT workflows integrating multiple data sources.
Develop data solutions in Python for data transformations, automation, and orchestration.
Leverage AWS services (S3, EMR, Glue, Lambda, Redshift, Kinesis, etc.) to implement cloud-native data platforms.
Write efficient SQL queries for data extraction, transformation, and reporting.
Ensure data quality, lineage, and governance across pipelines.
Collaborate with data engineers, architects, and analysts to deliver end-to-end data solutions.
Troubleshoot performance bottlenecks and optimize Spark jobs for speed and cost-efficiency.
Must have
8+ years of experience in data engineering or backend development.
Hands-on experience with Apache Spark (PySpark) in large-scale data environments.
Strong proficiency in Python programming.
Expertise in SQL (including advanced queries, performance tuning, and optimization).
Experience working with AWS services such as S3, Glue, EMR, Lambda, Redshift, or Kinesis.
Understanding of data warehousing concepts and ETL best practices.
Strong problem-solving skills and ability to work in an agile, collaborative environment.
Nice to have
Experience with Databricks or similar Spark-based platforms.
Knowledge of streaming frameworks (Kafka, Flink).
Familiarity with CI/CD pipelines, Docker, Kubernetes, Terraform.
Exposure to data modeling (star schema, snowflake, data vault).
Experience in financial services / capital markets .
Languages
English: C1 Advanced
Seniority
Senior
...lexcellence et limpact de chaque individu. Votre nouveau poste Assurer un leadership visible en matire de sant et de scurit sur le site. Soutenir les leaders dans lapplication des bonnes pratiques SSE et dans le dveloppement de leurs comptences. Concevoir...
Security Specialist II - Physical Security 7500 GEOINT Dr, Springfield, VA 22150, USASecurity Clearance Required: Active Top Secret****************** CONTINGENT UPON AWARD **********************Duties & Responsibilities:The Security Specialist II - Physical Security...
...eTeam Inc - Secondary is seeking a travel nurse RN Stroke Coordinator for a travel nursing job in Albuquerque, New Mexico. Job Description & Requirements ~ Specialty: Stroke Coordinator ~ Discipline: RN ~ Duration: 13 weeks ~40 hours per week ~ Shift: 8...
...Summary Substitute teacher to lead classes for ages 3-15. Duties Maintain accurate and complete student records, such as attendance. Prepare materials and classrooms for class activities. Confer with parents, administrators, and specialists to become...
...Benefits ~ On-site Fitness Center access (IN only)~ Health Clinic for employee use (IN only)~ Employee Assistance Program (EAP) Benefits BOM Writer Purpose: this position exists to create and maintain accurate Bill of Materials and records in support of Jayco...