Hybrid
Full-Time
Hyderabad, Telangana
India
Skills
Amazon Web Services (AWS)
Python (Programming Language)
Data Architecture
Data Engineering
PySpark
Apache Spark
Azure Databricks
Scala
Data Services
Apache Spark Streaming
About the Role
Job Title: Senior Data Engineer
Location: Chennai or Hyderabad Office - Hybrid Role
Job Type: Full-Time
Job Description:
We are seeking a Senior Data Engineer with strong expertise in Spark Streaming, PySpark, Python, and Scala, who can design, develop, and manage large-scale data processing solutions. The ideal candidate will have hands-on experience with Databricks and a solid foundation in AWS services related to data engineering.
This role will be pivotal in building scalable and high-performance data pipelines for real-time and batch processing, enabling data-driven decision-making across the organization.
Key Responsibilities:
Design, develop, and deploy real-time and batch data pipelines using Apache Spark Streaming.
Write efficient data transformation jobs using PySpark, Python, and Scala.
Utilize Databricks for collaborative development and data pipeline orchestration.
Work with AWS data services (e.g., S3, EMR, Glue, Lambda, Kinesis, Redshift) to build end-to-end data solutions.
Optimize and monitor data workflows for performance, scalability, and reliability.
Collaborate with data scientists, analysts, and DevOps teams to deliver business-critical data applications.
Maintain data quality, integrity, and governance across systems.
Required Skills & Qualifications:
4+ years of experience in data engineering with strong programming skills in Python, PySpark, and Scala.
Proven experience with Spark Streaming for real-time data processing.
Proficiency with Databricks and its integration with cloud ecosystems.
Deep understanding of AWS data services and infrastructure.
Strong understanding of distributed systems and big data architecture.
Familiarity with CI/CD pipelines and DevOps practices in a cloud environment.
Excellent problem-solving skills and ability to work in an agile environment.
Preferred Qualifications:
AWS Certification (e.g., AWS Certified Data Analytics – Specialty, or Solutions Architect).
Experience with data lakes, data warehouses, and ELT/ETL frameworks.
Knowledge of containerization (Docker, Kubernetes) and infrastructure as code (Terraform, CloudFormation).
Location: Chennai or Hyderabad Office - Hybrid Role
Job Type: Full-Time
Job Description:
We are seeking a Senior Data Engineer with strong expertise in Spark Streaming, PySpark, Python, and Scala, who can design, develop, and manage large-scale data processing solutions. The ideal candidate will have hands-on experience with Databricks and a solid foundation in AWS services related to data engineering.
This role will be pivotal in building scalable and high-performance data pipelines for real-time and batch processing, enabling data-driven decision-making across the organization.
Key Responsibilities:
Design, develop, and deploy real-time and batch data pipelines using Apache Spark Streaming.
Write efficient data transformation jobs using PySpark, Python, and Scala.
Utilize Databricks for collaborative development and data pipeline orchestration.
Work with AWS data services (e.g., S3, EMR, Glue, Lambda, Kinesis, Redshift) to build end-to-end data solutions.
Optimize and monitor data workflows for performance, scalability, and reliability.
Collaborate with data scientists, analysts, and DevOps teams to deliver business-critical data applications.
Maintain data quality, integrity, and governance across systems.
Required Skills & Qualifications:
4+ years of experience in data engineering with strong programming skills in Python, PySpark, and Scala.
Proven experience with Spark Streaming for real-time data processing.
Proficiency with Databricks and its integration with cloud ecosystems.
Deep understanding of AWS data services and infrastructure.
Strong understanding of distributed systems and big data architecture.
Familiarity with CI/CD pipelines and DevOps practices in a cloud environment.
Excellent problem-solving skills and ability to work in an agile environment.
Preferred Qualifications:
AWS Certification (e.g., AWS Certified Data Analytics – Specialty, or Solutions Architect).
Experience with data lakes, data warehouses, and ELT/ETL frameworks.
Knowledge of containerization (Docker, Kubernetes) and infrastructure as code (Terraform, CloudFormation).