Data Engineer
Hybrid
England, United Kingdom
Freelance
18-03-2025
Job Specifications
Data Engineer (Security Cleared)
Location – Northern Ireland (Belfast) If based in the UK candidates must be willing to frequently travel to Northern Ireland.
Contract – 6 Months
Day rate – Flexible (Inside IR35)
Your responsibilities in the role
Design and implement efficient data ingestion pipelines to and from databases and file storage services.
Transform data from source to target to ensure data quality and consistency, based in agreed specifications.
Build, maintain and optimise data pipelines to automate data flows and enable real time data processing.
Monitor data quality and implement measures to ensure data accuracy and completeness.
Manage and maintain databases to ensure optimal performance and security.
Deploy and manage data infrastructure on cloud platforms – primarily AWS
Work closely with data analysts, data scientists, and other stakeholders to understand their data needs and deliver high-quality data solutions.
Skills and Experience
AWS focused, Ideally a AWS Certified Data Engineer.
SQL, Python, and Spark and experience with metadata-driven ETL/ELT.
AWS Glue, Databrew, S3, AWS Lambda, PostgreSQL, Quicksight.
Version Control (Gitlab), CI/CD and IaC.
Familiar with security and networking principles, especially in AWS deployment.
Experience and understanding of handling both structured and unstructured data.
Complex data migrations.
Strong problem-solving analytical skills.
Excellent communication and collaboration skills.
Understanding of GIS data models.
Further information required upon application
About the Company
Our ambition is to work with IT solutions businesses and end clients to deliver outstanding IT resource. We are passionate about being the first choice for exceptional contractors and candidates seeking new opportunities. We also attract the most talented people seeking a personally rewarding career in IT recruitment. Our teams are passionate about what they do and this shows in their work. Our consultants are dedicated to building relationships with both clients and contractors. ECS has gone from strength to strength by co... Know more
Related Jobs
- Company Name
- Project Brains
- Job Title
- Data Engineering Specialist
- Job Description
- Company Description Project Brains is a Future of Work platform that matches business needs with fractional specialists who have the expertise to deliver successful outcomes. By leveraging our vetted community of specialists, businesses can focus on their core priorities and grow ambitiously. Today we are working with a business who urgently needs senior and expert Data Engineers to migrate an existing Data Lake from SAS to Google Cloud. Qualifications - Expertise in Data Engineering: Strong skills in SAS, SQL, dbt, git, Airflow (Python), ODE Generic Export Framework. - Experience with Key Tools: Familiarity with KNIME, VS Code, DIL-Pipelines, InnovatorX (MDD Software), Iceberg/Biglake Tables.. - Industry Knowledge (Preferred but not mandatory): Understanding of fiber optic marketing. - Availability for an urgent start. Goal This is a time-sensitive project, and the right engineers will ensure a seamless migration, preventing disruptions. We are open to a streamlined hiring process for this contract need.
- Company Name
- Creo Recruitment
- Job Title
- Data QA Tester - £550 - £600 Inside IR35
- Job Description
- Key Responsibilities: - Develop Test Plans and Strategies: Create comprehensive test plans and strategies for the Salesforce project, ensuring all aspects of the program are covered. - Design and Execute Test Cases: Develop and execute detailed test cases to identify and report bugs, glitches, and other issues, ensuring thorough testing of all functionalities. - System and Functional Testing: Perform system and functional testing to ensure the solution meets business requirements and acceptance criteria, validating the effectiveness of the integration. - Issue Resolution: Collaborate with developers to resolve issues and improve the user experience, ensuring a seamless integration process. - Maintain Test Documentation: Keep detailed test documentation and record defect details. Track defects through their lifecycle, from identification to resolution and retesting, ensuring closure. - Quality Assurance: Ensure that the software meets the quality criteria established by the program team and end users, maintaining high standards of performance and reliability. - Regression Testing: Conduct regression testing to ensure new solutions or updates do not negatively impact existing core processes, maintaining system integrity. - Comprehensive Testing: Where applicable, appropriate, and feasible, ensure that testing includes performance, regression, functionality, non-functional, and integration tests, providing a holistic approach to quality assurance. - Manual and Automated Testing: Perform both manual and automated testing to ensure thorough coverage and efficiency in the testing process. - Reconciliation Testing: Designing tests and validations to ensure that migrated data is consistent across each phase of the migration pathway, from source staging to Salesforce. Key Skills: - Proven experience as a Data QA Tester or similar role, with a focus on data integration and migration testing, specifically Salesforce reports and dashboards. - Strong & Proven proficiency in Behaviour-Driven Development (BDD) techniques, tools (e.g., Cucumber, SpecFlow), and frameworks. - Extensive hands-on experience with Salesforce testing, including report validation, data validation, and dashboard testing. - Proficiency in dataset generation techniques, SQL querying, and data manipulation. - Experience working in Agile development methodologies, such as Scrum or Kanban, with a strong understanding of the testing lifecycle in Agile projects. - Ability to effectively test, validate, and verify Salesforce reports and dashboards against business requirements. - Excellent analytical skills and attention to detail, with the ability to identify data inconsistencies, anomalies, and discrepancies. - Strong communication and collaboration skills, with the ability to effectively interact with cross-functional teams and stakeholders. - Proactive mindset, with a passion for continuous learning and staying abreast of the latest trends and advancements in data QA testing. - Experience of reconciliation testing between source and target data sources.
- Company Name
- Walker Lovell
- Job Title
- Market Data Analyst
- Job Description
- We are seeking a Market Data Analyst to support a global market data team. This role is key to ensuring accurate reporting, compliance, and client support within a fast-paced financial environment. This is a 12-month fixed-term contract with the opportunity to work in a globally recognized financial environment. Key Responsibilities: Data & Compliance: Manage market data reporting, ensuring accuracy and timely submission. Client Support: Handle customer queries and support market data licensing processes. Financial Oversight: Work with finance teams on billing, reporting corrections, and debt follow-ups. Market Insights: Analyze transaction data to identify trends and business insights. Process Improvement: Contribute to audit programs and market data projects. What We’re Looking For: Experience in market data, compliance, or transaction reporting. Knowledge of exchange-traded derivatives, futures, or commodities trading is a plus. Strong analytical and communication skills. Advanced proficiency in Excel, PowerPoint, and Word. Ability to manage multiple tasks and collaborate across teams.
- Company Name
- Meritus
- Job Title
- Lead Data Engineer
- Job Description
- Lead Data Engineer | Defence Consultancy | DV Clearance Required | 12 Month Contract (Inside IR35) | Up to £800 per day | London MERITUS are working with a tech focused Defence Consultancy in London looking for a Lead Data Engineer to join on a 12 month contract working with MOD customers in the London area. If successful you will be responsible for designing, building and maintaining scalable data pipelines and integration solutions. You will work closely with business & customer stakeholders in the wider data team to ensure efficient flow of data across systems. The role is based in London & requires active DV security clearance. Main Responsibilities: Design, implement, and maintain scalable data architectures, pipelines, and ETL processes. Lead and mentor data engineers, ensuring best practices in coding, data modelling, and pipeline optimisation. Establish and enforce data quality, compliance, and security standards across data platforms. Work closely with data scientists, analysts, and business teams to understand requirements and optimise data solutions. Skills Required: Expertise in tools like Hadoop, Spark, Kafka, and cloud data services (AWS, GCP, or Azure). Strong knowledge of relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases. Proficiency in Python, Java, or Scala for data processing and automation. Experience with Apache Airflow, dbt, or other orchestration tools for efficient ETL/ELT processes. Got your attention? If you believe that you have the skills and experience for the role – then please get in touch. We also offer a referral scheme for any candidates whose details are passed to us that we successfully place. If you have any further questions then please contact Ryan Harris at MERITUS Talent.