Remote
Full-Time
India
Skills
Amazon Web Services (AWS)
Python (Programming Language)
Hadoop
Hive
NoSQL
Apache Spark
Azure Databricks
Scala
Big Data
Public Cloud
About the Role
Role Summary
We are looking for a skilled Big Data Engineer with deep expertise in Databricks, Apache Spark, and Scala. The ideal candidate will be highly proficient in big data technologies, including Hadoop, Hive, Spark, and SQL/NoSQL databases. A strong programming foundation in Scala and/or Python is essential, along with hands-on experience in public cloud environments such as AWS and Databricks.
Key Responsibilities
Design, develop, and optimize big data pipelines using Databricks and Spark for real-time and batch data processing.
Work extensively with Spark components, including RDDs, DataFrames, and Datasets, to ensure efficient and scalable data transformations.
Develop, implement, and manage big data frameworks and platforms using technologies such as Hadoop, Spark, and Hive.
Optimize and tune Spark jobs for performance, ensuring data processing is reliable, scalable, and meets business requirements.
Utilize SQL and NoSQL databases to design data models that support high availability and rapid data retrieval.
Collaborate with cross-functional teams to define data requirements and drive data strategy and architecture.
Employ cloud services (AWS) and Databricks for data storage, processing, and analytics, leveraging their capabilities for scaling big data solutions.
Skills and Qualifications
Must-have experience with Databricks and Apache Spark (including RDDs, DataFrames, and Datasets).
Strong programming skills in Scala (preferred) and/or Python, with demonstrated hands-on experience in building data pipelines.
Proficiency in big data technologies: Hadoop, Spark, Hive, SQL, and NoSQL.
Experience working with public cloud platforms like AWS and utilizing their data services for scalable and secure big data processing.
Strong understanding of data processing concepts, such as data cleansing, transformation, ETL, and real-time processing.
Excellent problem-solving skills, attention to detail, and ability to work independently or collaboratively in a fast-paced environment.
We are looking for a skilled Big Data Engineer with deep expertise in Databricks, Apache Spark, and Scala. The ideal candidate will be highly proficient in big data technologies, including Hadoop, Hive, Spark, and SQL/NoSQL databases. A strong programming foundation in Scala and/or Python is essential, along with hands-on experience in public cloud environments such as AWS and Databricks.
Key Responsibilities
Design, develop, and optimize big data pipelines using Databricks and Spark for real-time and batch data processing.
Work extensively with Spark components, including RDDs, DataFrames, and Datasets, to ensure efficient and scalable data transformations.
Develop, implement, and manage big data frameworks and platforms using technologies such as Hadoop, Spark, and Hive.
Optimize and tune Spark jobs for performance, ensuring data processing is reliable, scalable, and meets business requirements.
Utilize SQL and NoSQL databases to design data models that support high availability and rapid data retrieval.
Collaborate with cross-functional teams to define data requirements and drive data strategy and architecture.
Employ cloud services (AWS) and Databricks for data storage, processing, and analytics, leveraging their capabilities for scaling big data solutions.
Skills and Qualifications
Must-have experience with Databricks and Apache Spark (including RDDs, DataFrames, and Datasets).
Strong programming skills in Scala (preferred) and/or Python, with demonstrated hands-on experience in building data pipelines.
Proficiency in big data technologies: Hadoop, Spark, Hive, SQL, and NoSQL.
Experience working with public cloud platforms like AWS and utilizing their data services for scalable and secure big data processing.
Strong understanding of data processing concepts, such as data cleansing, transformation, ETL, and real-time processing.
Excellent problem-solving skills, attention to detail, and ability to work independently or collaboratively in a fast-paced environment.