Big Data Lead
Remote
London, United Kingdom
Freelance
20-02-2025
Job Specifications
For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.
Knowledge in the following areas essential:
Data Engineering Experience:
Databricks: Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
AWS Cloud: Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
IBM Skills: DB2, Datastage, Tivoli Workload Scheduler, Urban Code
Programming Languages: Proficiency in Python, SQL.
Data Warehousing & ETL: Experience with modern ETL frameworks and data warehousing techniques.
DevOps & CI/CD: Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
Test automation skills
ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
Leadership & Strategy: Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
Customer Data Platform Development: Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
Data Governance & Best Practices: Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
Pipeline Automation & Optimization: Drive the automation of data pipelines and workflows to improve efficiency and reliability.
Team Management: Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
Cross Company Collaboration: Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
Cloud Management: Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
Performance Monitoring: Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.
About the Company
SoftNice is an award-winning and global IT Services and Solutions company specialised in deploying IT solutions and delivering them. We recruit for a wide range of industries including IT, Event, Healthcare Finance & Security Staff. We offer recruitment in all the forms viz. temporary, permanent and contract assignments. Softnice has been established for over 10 years, building strong relationships with our customers and candidates.Today we proudly call ourself as 360 staffing company as we are successfully delivering int... Know more
Related Jobs
- Company Name
- LSA Recruit
- Job Title
- Data Architect (DDTS)
- Job Description
- Data Architect Remote Contract Note: Need Active SC Clearence Candidates The Supplier shall provide Architecture Services to carry out the activities and outcomes defined in the SOW deliverables below. The Architecture Services will need to cover a wide range of architecture capabilities to satisfy specific project requirements and support DDTS project teams. The primary focus for the Data Architect would be the data cleansing/transformation/loading - migration activities. They will need understand the unknown touch points within the Discovery portion as are still some unknowns. Also give an informed opinion to the Discovery on how long migration will take based on their work, e.g. will we have to have a temporary data repo before moving to a new system given the tight timeline until Tranman is discontinued. Specific task requirements: Tranman data analysis for migration: Understand the asset data requirements Identify activities for migration (cleansing, restructuring etc.) Identify secure data (e.g. GDPR, transport regulations) Identify replacement for Tranman that can be implemented within the time constraint and output as an Options Assessment. Identify any IT opportunities for the service transformation, e.g.: Cost-benefit of bringing supplier databases in-house De-duplication and solution integration Automation and data analytics Standard deliverables for a Discovery Agree business resource responsibilities Review solution for Accessibility audit and any record required improvements For more info, Please share me your Updated CV at shameena@Lsarecruit.co.uk
- Company Name
- Harnham
- Job Title
- Senior Insurance Data Scientist - 12 month FTC
- Job Description
- SENIOR DATA SCIENTIST - INSURANCE - 12 MONTH FTC £75,000 FULLY REMOTE THE COMPANY This business are an exciting and unique Insurtech who are going from strength to strength as they continue their expansion. This role offers the chance to make a real impact in the business and work in a highly-motivated team offering the chance to get a lot of variety in your work. THE ROLE Develop cutting edge pricing models using Python and Machine Learning techniques Work on end-to-end ownership of these models, including work on implementation, deployment and enhancement Analyse large sets of customer data to drive insight and commercial performance Share insight with the wider team and helping to improve profitability across the business on ad hoc project work YOUR SKILLS AND EXPERIENCE: At least 3-4 years prior experience in pricing analytics within insurance Good knowledge of Python is essential Prior experience in pricing model development OR implementation Strong communication skills and desire to learn is key SALARY AND BENEFITS Up to £75,000 base salary Private medical care 26 days holiday Enhanced parental leave HOW TO APPLY Please register your interest by sending your CV to Rosie Walsh through the 'Apply' link
- Company Name
- iO Associates - UK/EU
- Job Title
- Senior Data Engineer - Matillion
- Job Description
- Senior Data Engineer - Matillion 3-month contract - potential for extension Immediate Start Outside IR35 £500 - £550pd iO Associates are seeking an experienced Data Engineer to work with one of the best analytics companies in the UK. They have nurtured a fantastic culture, with industry-leading experts in the modern data space and are seeking a Data Engineer. The ideal candidate needs to have over 8 years of commercial experience, particularly across Matillion, as well as Snowflake The role is operating Outside IR35 for an initial 3-month engagement, starting in March and paying £500 - £550 per day. As a Senior member of the team, you'll contribute as a thought leader, developing insights and sharing expertise internally and externally while mentoring others and driving innovation across the organisation We're working with a client based in the Southwest of the UK that will require occasional travel to the Bristol office (once per month). Otherwise, it will be a remote engagement. Interested in hearing more? Please get in touch with Rebecca Long on 0117 332 8100 / r.long @ ioassociates.co.uk
- Company Name
- Harnham
- Job Title
- Senior Data Scientist - Insurance
- Job Description
- SENIOR DATA SCIENTIST – INSURANCE – 12 MONTH FTC £75,000 FULLY REMOTE THE COMPANY This business are an exciting and unique Insurtech who are going from strength to strength as they continue their expansion. This role offers the chance to make a real impact in the business and work in a highly-motivated team offering the chance to get a lot of variety in your work. THE ROLE Develop cutting edge pricing models using Python and Machine Learning techniques Work on end-to-end ownership of these models, including work on implementation, deployment and enhancement Analyse large sets of customer data to drive insight and commercial performance Share insight with the wider team and helping to improve profitability across the business on ad hoc project work YOUR SKILLS AND EXPERIENCE: At least 3-4 years prior experience in pricing analytics within insurance Good knowledge of Python is essential Prior experience in pricing model development OR implementation Strong communication skills and desire to learn is key SALARY AND BENEFITS Up to £75,000 base salary Private medical care 26 days holiday Enhanced parental leave HOW TO APPLY Please register your interest by sending your CV to Rosie Walsh through the ‘Apply’ link