Data engineer
On site
Croydon, United Kingdom
£ 500 / day
Freelance
10-12-2024
Job Specifications
- Sector: Public sector/Government
- Clearance required: SC clearance
- Contract length: 6 months
- Rate: 400-500/d
- IR35 status: Outside
- Onsite requirement: 2.d a week
- Role title: Data engineer
Job Description:
Build streaming data ingestion pipelines using AWS MSK and batch data ingestion pipelines using APIs
Prepare data for prescriptive & predictive modelling.
Experience working with various data formats eg, parquet, avro, json
Experience in building ETL pipelines using Lambda, Step Functions and Glue
Experience working with various AWS database services (eg, RDS, DynamoDB and Redshift)
Technical expertise with various data modelling techniques
Hands-on experience with Python and SQL
Kafka Streaming data experience to build the new data pipelines.
Supporting Skills
5 years' experience in an iterative and Agile development environment
Appreciation of Software Development Environments & Languages
Appreciation of Business Analysis Activities
Appreciation of other Scripting language, eg JavaScript, Lua, Perl, PHP, Unix
About the Company
Sanderson Government & Defence was established over 20 years ago as a bespoke solution to a unique talent problem: sourcing and placing high-calibre professional talent to work within the UK government and defence sector. MISSION We provide talent solutions for the UK government and their critical supply chains, including: Ministry of Defence and areas of National Security; Central Government and Arm’s Length Bodies; National Policing and Justice; and the broader Public Sector. EXPERTISE & SKILL Central to our success has b... Know more
Related Jobs
- Company Name
- Henderson Scott
- Job Title
- BI Developer
- Job Description
- BI Developer - Power BI We require an experienced BI Developer to work for our market leading, family run client on a 3 month contract. They are moving from Crystal Reports to Power BI. Responsibilities include but are not limited to - Interpret, maintain and simplify business reporting requirements Implement BI reports and create technical documentation in conjunction with maintaining and simplifying business reporting. Implement and maintain BI tools for integration with business data Excellent Power BI knowledge with Crystal reports being Experience in database operations and optimisation for SQL Server 2016 General support with the wider business This role requires 3-4 days per week on client site in South Oxfordshire
- Company Name
- Freshtechit
- Job Title
- Data Engineer
- Job Description
- Data Engineer- 6 month contract ( remote working) A leading travel company are seeking a Data Engineer to join on a 6 month rolling contract. Data Pipeline Management: Oversee and maintain end-to-end data pipelines, ensuring data integrity and availability for internal and external stakeholders. Reporting & Insights: Develop and manage dashboards, visualizations, and business reporting using tools like Power BI, Tableau, or similar. Conduct ad-hoc data analysis using SQL and other tools. Stakeholder Engagement: Collaborate with internal teams to gather requirements, scope projects, and deliver actionable insights. Self-Service Enablement: Train staff on using self-service reporting tools and support their data needs. Trend Analysis: Identify key trends in data and provide recommendations to drive business value. Issue Resolution: Troubleshoot data pipeline and reporting issues promptly and effectively. As a Data Engineer you will have the following experience: Data visualization and reporting tools (e.g., Power BI, Tableau). Data pipeline development and ETL tools (e.g., DBT, Apache Airflow). Advanced SQL scripting. We are seeking someone with a strong understanding of data warehousing, relational databases, data cleansing, and dimensional modeling. Please note: The contract is to start in January, we are unable to consider long notice periods. Please contact hello@freshtechit.com for more information
- Company Name
- eTeam
- Job Title
- Data Engineer
- Job Description
- Job Title: SC cleared Data Engineer (Denodo ) Clearance required: SC clearance is mandatory Duration: 6 months Location: Telford, UK Contract Inside IR35 Experience in migrating data between Denodo versions (Denodo 7 to Denodo 8). Expertise in Denodo VQL and strong understanding of Logical Data Warehousing (LDW). Problem solving and leadership skills (a support team of engineers will be provided to support the VQL engineer provided who the engineer is expected to lead and support) Ability to refactor and deploy views in Denodo 8 based on existing Denodo 7 configurations. Understanding of HMRC estate and supporting applications such as Git, Jenkins etc. Understanding of Cache and data migrations of cache.
- Company Name
- TEKsystems
- Job Title
- Junior Data Engineer
- Job Description
- Job Title: Data Engineer Job Description We are seeking a skilled Data Engineer to join our dynamic team. You will be responsible for building and maintaining data pipelines, ensuring data quality, and working with various cloud platforms and tools. This role requires excellent problem-solving skills and the ability to work collaboratively in an agile environment. Responsibilities Develop and maintain data pipelines using SQL, Python, and Spark. Utilise Azure Data Factory and Databricks for data integration and transformation. Perform data modelling and build ETL pipelines. Monitor and maintain platform performance using monitoring and alerting tools. Ensure data quality and integrity through robust quality assurance processes. Utilise GIT for version control and participate in code review processes. Collaborate effectively with team members and stakeholders. Essential Skills Proficiency in SQL, Python, and Spark. Experience with a cloud platform, preferably Azure. Knowledge of Azure Data Factory and Databricks. Expertise in data modelling and building ETL pipelines. Familiarity with platform monitoring and alerting tools. Strong data quality assurance experience . Proficiency with GIT and code review processes. Excellent communication and collaboration skills. Work Environment You will be working in a collaborative and agile environment, using cutting-edge technologies and tools. Expect a supportive team culture with a focus on continuous improvement and innovation. Our dress code is casual, promoting comfort and creativity. Location Twycross, UK Rate/Salary 360.00 - 385.00 GBP Daily Trading as TEKsystems. Allegis Group Limited, Bracknell, RG12 1RT, United Kingdom. No. 2876353. Allegis Group Limited operates as an Employment Business and Employment Agency as set out in the Conduct of Employment Agencies and Employment Businesses Regulations 2003. TEKsystems is a company within the Allegis Group network of companies (collectively referred to as "Allegis Group"). Aerotek, Aston Carter, EASi, Talentis Solutions, TEKsystems, Stamford Consultants and The Stamford Group are Allegis Group brands. If you apply, your personal data will be processed as described in the Allegis Group Online Privacy Notice available at our website. To access our Online Privacy Notice, which explains what information we may collect, use, share, and store about you, and describes your rights and choices about this, please go our website. We are part of a global network of companies and as a result, the personal data you provide will be shared within Allegis Group and transferred and processed outside the UK, Switzerland and European Economic Area subject to the protections described in the Allegis Group Online Privacy Notice. We store personal data in the UK, EEA, Switzerland and the USA. If you would like to exercise your privacy rights, please visit the "Contacting Us" section of our Online Privacy Notice on our website for details on how to contact us. To protect your privacy and security, we may take steps to verify your identity, such as a password and user ID if there is an account associated with your request, or identifying information such as your address or date of birth, before proceeding with your request. commitments under the UK Data Protection Act, EU-U.S. Privacy Shield or the Swiss-U.S. Privacy Shield.