Find More Than 10K+
United Kingdom Jobs

Role and responsibilities:

• Work closely with a team of data scientists, analysts, and other stakeholders to ensure the availability, reliability, and usability of data for analytical purposes and business decision-making.
• Work on the design and development of efficient and reliable data pipelines to collect, process, and store large-scale structured and unstructured data from various sources.
• Involved in the integration and data transformation from diverse sources, ensuring compatibility and consistency, to enable effective analysis and reporting.
• Set up and maintain the infrastructure required for Big Data processing, monitoring system performance, troubleshooting issues, and optimizing resource utilization.
• Implement robust security measures to protect sensitive data, including access controls, encryption, and compliance with privacy regulations.
• Work in collaboration with data scientists and analysts to understand requirements, provide clean and structured data for analysis, and assist in deriving data-driven insights.

Required Skills and Qualifications:

• At least 5+ years experience as a Bid Data Engineer
• Strong engineering skills
• Hands on experience with Unix commands, Unix Shell scripting, Hive, Bigdata technologies.
• Strong SQL knowledge with any RDBMS/HQL/Hive.
• Demonstrated skills in Pyspark

Nice to have skill:

• Bachelor's degree in Computer Science, Information Technology, or a related field. Advanced degree preferred.
• Working knowledge on Java (good to have)
• Knowledge of Scala/Python is a plus