Senior Data Engineer
On site
Manchester, United Kingdom
£ 80,000 / year
Full Time
14-02-2025
Job Specifications
Job Title: AWS Senior Data Engineer
Location: Manchester (Trafford) – 2 days a week
Salary: £60,000 - £80,000 DOE
The client is a technology provider to the market research industry. They use cutting edge technology to help our clients gain a deeper insight into the digital lives of consumers.
We are looking for Data Engineer to join the team and play a crucial role in Data Engineer Team, ensuring the main function is developing, maintaining and improving the end-to-end data pipeline.
The Role
You will be working in the Data Engineering team whose main function is developing, maintaining and improving the end-to-end data pipeline that includes real-time data processing; extract, transform, load jobs; artificial intelligence; and data analytics on a complex and large dataset.
Your role will primarily be to perform DevOps, backend and cloud development on the data infrastructure to develop innovative solutions to effectively scale and maintain the data platform. You will be working on complex data problems in a challenging and fun environment, using some of the latest Big Data open-source technologies like Apache Spark, as well as Amazon Web Service technologies including Elastic MapReduce, Athena and Lambda to develop scalable data solutions.
Key Responsibilities:
Adhering to Company Policies and Procedures with respect to Security, Quality and Health & Safety.
Writing application code and tests that conform to standards.
Developing infrastructure automation and scheduling scripts for reliable data processing.
Continually evaluating and contribute towards using cutting-edge tools and technologies to improve the design, architecture and performance of the data platform.
Supporting the production systems running the deployed data software.
Regularly reviewing colleagues’ work and providing helpful feedback.
Working with stakeholders to fully understand requirements.
Be the subject matter expert for the data platform and supporting processes and be able to present to others to knowledge share.
Here’s what we’re looking for:
The ability to problem solve.
Knowledge of AWS or equivalent cloud technologies.
Knowledge of Serverless technologies, frameworks and best practices.
Experience using AWS CloudFormation or Terraform for infrastructure automation.
Knowledge of Scala or OO language such as Java or C#.
SQL or Python development experience.
High-quality coding and testing practices.
Willingness to learn new technologies and methodologies.
Knowledge of agile software development practices including continuous integration, automated testing and working with software engineering requirements and specifications.
Good interpersonal skills, positive attitude, willing to help other members of the team.
Experience debugging and dealing with failures on business-critical systems.
Preferred:
Exposure to Apache Spark, Apache Trino, or another big data processing system.
Knowledge of streaming data principles and best practices.
Understanding of database technologies and standards.
Experience working on large and complex datasets.
Exposure to Data Engineering practices used in Machine Learning training and inference.
Experience using Git, Jenkins and other CI/CD tools.
Benets
Work in a market leading technology company that helps research and marketing professionals achieve unique insights into the mobile and digital lives of consumers.
The client do everything they can to support our people so that they can be themselves and realise their potential. We love people who are hungry for learning and achievement!
25 days paid holiday plus bank holidays
Purchase/sale of up to 5 leave days pa – after 2 years’ service
Life insurance
Workplace pension with employer contribution
Performance based bonus scheme
Informal dress code
Cycle to work scheme
Branded company merchandise
About the Company
Based in the heart of Manchester, In Technology Group are recognised for delivering exceptional recruitment services to the technology, IT & digital sectors across the UK, Europe & US. Possessing in-depth knowledge and critical industry insights, we have access to the best active talent on the market as well as an extensive pipeline of passive candidates. Our team around us is what makes us special. We're all industry professionals, each of us driven to delivering the best possible experience. We invest in our own talent eve... Know more
Related Jobs
- Company Name
- Focus 5 Recruitment
- Job Title
- Data Engineer
- Job Description
- Data Engineer Warrington - Hybrid £50,000 - £60,000 - Depending on experience Focus 5 Recruitment are working with an exciting software business to help recruit a Data Engineer. The company have just been awarded 2 large contracts with international Mobile Network Operators. Appointed to help them source a Data Engineer, we’re looking for an experienced Data Engineer to design and optimize our client’s data pipelines and storage solutions. This is an amazing opportunity to work with a growing and ambitious software business who have contracts with some of the world’s leading mobile network companies. They are looking for candidates who can come in at a key point in their growth and develop their career as they grow. Key responsibilities for the Data Engineer – Design and build high-performance, low-latency data pipelines capable of processing large volumes of data at high speed. Develop and enhance real-time and batch data processing architectures. Manage both structured and unstructured data, ensuring efficient ingestion, transformation, and storage. Deploy scalable data storage solutions across bare metal and cloud platforms (AWS). Optimize databases, data lakes, and messaging systems for maximum throughput and minimal latency. Collaborate with DevOps and software engineering teams to maintain seamless data integration and flow. Implement monitoring, logging, and alerting systems to track data pipeline performance and integrity. Uphold data security and compliance across all environments. Data Engineer experience we’re looking for - Demonstrated expertise in designing and deploying data architectures for high-velocity, high-throughput systems. Strong proficiency in real-time data streaming technologies such as Kafka, Pulsar, and RabbitMQ. Extensive experience with high-performance databases, including PostgreSQL, ClickHouse, Cassandra, and Redis. In-depth knowledge of ETL/ELT pipelines, data transformation, and storage optimization. Skilled in working with big data frameworks like Spark, Flink, and Druid. Hands-on experience with both bare metal and AWS environments. Strong programming skills in Python, Java, and other relevant languages. Proficiency in containerization technologies (Docker, Kubernetes) and infrastructure as code. Solid understanding of data security, encryption, and compliance best practices. Preferred Qualifications - Experience working with telecom or financial systems. Background in government or defence-sector projects. This is an exclusive role with a key client. For immediate consideration and full details, please submit an application ASAP.
- Company Name
- Kraken
- Job Title
- Data Engineer
- Job Description
- Help us use technology to make a big green dent in the universe! Kraken powers some of the most innovative global developments in energy. We’re a technology company focused on creating a smart, sustainable energy system. From optimising renewable generation, creating a more intelligent grid and enabling utilities to provide excellent customer experiences, our operating system for energy is transforming the industry around the world in a way that benefits everyone. It’s a really exciting time in energy. Help us make a real impact on shaping a better, more sustainable future. Kraken Customer What we do: build the most AI-driven, innovative, forward-thinking platform for energy management. From optimising resources to delivering cost-effective, exceptional customer experiences through advanced Customer Information Systems (CIS), billing, meter data management, CRM, and AI-driven communications, Kraken is powering the next wave of innovation in the energy industry. Why we do it: future energy will not look like energy as we know it today. We need to not just think about our future, but build for it. Now. At Kraken we've developed a data platform that is used by Octopus Energy and our other clients retail energy businesses around the world. The platform empowers users with self-service data analytics and automates our data processing workflows, from simple ETL jobs to ML training and prediction. The data platform team works across the whole customer domain on anything from natural language understanding of our customer communications to processing billions of smart meter readings every day to support clients in creating customised and market leading smart energy tariffs. As the volume, scope and geographical range of our data offerings rapidly expand, we're looking for an experienced data engineer to join the team to help us build and maintain our platform, pipelines and data sources. We're passionate about building great technology to change the way customers use, and think about, energy for the good of the planet. This is a fantastic opportunity to work with us on data problems that genuinely move us closer to Net Zero, and support the energy transition. What You'll Do Build new data sources and data pipelines that deliver key data and insights to the business. Build data pipelines that power and enable the inner workings of the Kraken platform, from billing and settlement to customer facing apps. Build and maintain testing and documentation frameworks for our data sources. Work with the business to scope and deliver new data engineering projects and requirements. Maintain and build on our existing data infrastructure and tools. Support the internationalisation of our data infrastructure as we continue to grow globally. What You'll Have Python, SQL, Spark Experience in assuring data quality Experience deploying data services in a cloud environment (ideally AWS) Experience with APIs designed for high volume traffic serving data to customers or business critical applications. Experience with relational databases (Postgres/Aurora) The projects will be varied and we're looking for someone who can work autonomously and proactively to scope problems and solve and deliver pragmatic solutions. If this sounds like you then we'd love to hear from you. Are you ready for a career with us? We want to ensure you have all the tools and environment you need to unleash your potential. Need any specific accommodations? Whether you require specific accommodations or have a unique preference, let us know, and we'll do what we can to customise your interview process for comfort and maximum magic! Studies have shown that some groups of people, like women, are less likely to apply to a role unless they meet 100% of the job requirements. Whoever you are, if you like one of our jobs, we encourage you to apply as you might just be the candidate we hire. Across Octopus, we're looking for genuinely decent people who are honest and empathetic. Our people are our strongest asset and the unique skills and perspectives people bring to the team are the driving force of our success. As an equal opportunity employer, we do not discriminate on the basis of any protected attribute. Our commitment is to provide equal opportunities, an inclusive work environment, and fairness for everyone.
- Company Name
- Barclay Simpson
- Job Title
- R&D Software and Data Engineer
- Job Description
- PLEASE NOTE - SPONSORSHIP CANNOT BE PROVIDED FOR THIS POSITION R&D Software and Data Engineer Role Overview We are seeking an experienced Senior R&D Software and Data Engineer to play a key role in developing and maintaining data-driven systems. This role involves designing and optimising data pipelines, warehouses, and lakes to ensure data accuracy, accessibility, and security. The successful candidate will contribute to the evolution of digital and analytical capabilities, particularly within the field of cyber fraud prevention and financial crime mitigation. Key Responsibilities Develop and maintain robust data architectures and pipelines to enable seamless data transfer and processing, ensuring durability, consistency, and completeness. Design and implement data warehouses and data lakes that efficiently manage high-volume and high-velocity data while adhering to stringent security standards. Build and optimise data processing and analytical algorithms suitable for handling complex and large-scale datasets. Collaborate with data scientists to develop and deploy machine learning models. Provide guidance on risk mitigation and contribute to governance policies to enhance data security and compliance. Manage risk effectively and implement stronger controls across data engineering projects. Work closely with stakeholders across different areas, ensuring alignment with broader business strategies and objectives. Conduct in-depth analysis of diverse data sources, both internal and external, to develop creative and effective solutions. Communicate complex technical concepts to a wide range of audiences, including senior stakeholders. Influence decision-makers and collaborate cross-functionally to drive impactful data solutions. Key Requirements Experience within financial services, particularly in cyber fraud, financial crime, or security-related roles. Strong understanding of industry fraud and security signals, including digital identity, device authentication, voice recognition, biometrics, and behavioural profiling. Knowledge of cyber fraud attack vectors, such as device spoofing, location manipulation, identity fraud, account takeover, and fraudulent documentation. Proficiency in software development and data analytics tools, such as: Programming languages: Python, PHP, JavaScript, Java Databases: Postgres, MS SQL, Oracle, MySQL, Teradata Data analysis tools: SAS PROC SQL, Hue Database Assistant, Hadoop Familiarity with enterprise security frameworks, such as NIST Cybersecurity Framework and cyber-attack methodologies. Experience with data visualisation and MI tools, including Tableau and Power BI. Strong technical background in computer science, computing, and networks. Desirable Skills Advanced knowledge of cyber fraud tactics and countermeasures within financial services. Experience with tools such as Quantexa, ElasticSearch, SAS and related analytics platforms. Strategic thinking and business acumen, with the ability to drive digital transformation initiatives. Strong understanding of risk and control frameworks within data engineering. Location The successful candidate can work from either Knutsford or Northampton on a hybrid basis.
- Company Name
- Codeconvergence Limited (CCL)
- Job Title
- Senior Data Analyst ( investment banking )
- Job Description
- Job Description: Overall 10+ years of experience in working as data analyst within investment banking/ insurance. Familiar with Bloomberg / Statestreet data feeds Experience in understanding the functions and the data elements within Pension, Home insurance, Pet insurance Experience in drafting the reporting requirements with business stakeholders.