AWS Data Engineer
On site
Telford, United Kingdom
Freelance
08-11-2024
Job Specifications
AWS Data Engineer
Role Type: Permanent
Location: Telford, UK
Hybrid – 2 days’ work from Telford office
Role : AWS Data Engineer
• Design, develop, and maintain real-time data streaming pipelines using AWS Kinesis for processing and analyzing high-velocity data streams.
• Collaborate with stakeholders to gather requirements and design efficient data streaming architectures for real-time analytics, monitoring, and alerting.
• Build and optimize Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics to process real-time data from various sources.
• Develop and implement ETL processes to transform, enrich, and aggregate streaming data for downstream consumption in data lakes or data warehouses such as AWS S3, Redshift, and DynamoDB.
• Integrate Kinesis with other AWS services like Lambda, S3, and EMR for real-time data ingestion, processing, and storage.
• Optimize streaming applications for performance and scalability, ensuring efficient resource usage and minimal latency.
• Develop and maintain robust error handling, data validation, and retry mechanisms within Kinesis pipelines to ensure data consistency and reliability.
• Implement security best practices for data streaming, including encryption in transit and at rest, access control using IAM roles and policies, and data retention policies.
• Monitor and troubleshoot AWS Kinesis streams using CloudWatch metrics, alarms, and logs, ensuring high availability and minimal downtime.
• Collaborate with DevOps teams to automate the deployment and scaling of streaming pipelines using AWS CloudFormation, Lambda, and CI/CD tools.
• Work closely with data analysts and data scientists to ensure real-time data is processed and delivered for immediate insights and reporting.
• Implement data partitioning, sharding, and key management strategies to handle high-throughput data streams and ensure scalability.
• Optimize the cost of real-time data processing by tuning the configurations of Kinesis and associated AWS services like Lambda and Firehose.
• Stay current with AWS updates and best practices for streaming data, recommending improvements to the architecture as new features and services become available.
• Provide guidance and mentorship to junior engineers in real-time data processing, ensuring best practices in streaming technologies and AWS cloud services.
• Create and maintain detailed technical documentation for data pipelines, including data flow diagrams, stream processing logic, and operational guidelines.
About the Company
Infinity Quest (IQ) with its roots from Staffing and Recruitment provides world class business solutions to our valued customers. We specialise in Professional Services, SAP, Salesforce and RPA. Our team of experts understand the challenges you face every day and provide simple technology driven solutions to your complex problems. We have over 15 years experience within the professional services & technology services industry and have catered to wide variety of clientele from SMEs to global corporations. At IQ we are dedicat... Know more
Related Jobs
- Company Name
- Revolution Technology Ltd
- Job Title
- ServiceNow Developer
- Job Description
- Our client are a global consultancy, working on a project in the defence space. They are on the lookout for a ServiceNow Developer to come in on a contract basis. Key Skills/Requirements: Must have strong ServiceNow experience Must have strong hands-on experience of developing and configuring the platform Must be able to design and roadmap complex technical solutions aligned to the latest ServiceNow best practice Must be a British Citizen (BPSS to pass on offer) Contract is running 6-12 months initially, paying up to £550p/day (Inside IR35 so engaged via Umbrella) and will be fully remote within the UK.
- Company Name
- bluewaveSELECT
- Job Title
- ServiceNow Developer, 6-12 months, Inside IR35, REMOTE, £525 a day
- Job Description
- ServiceNow Developer, 6-12 months, Inside IR35, REMOTE, £525 a day ServiceNow Developer Job Type: Contract, Inside IR35, REMOTE, £525 day Key Responsibilities: Excellent experience as a ServiceNow Developer, with architecture and complex solutions understanding Develop and customise ServiceNow applications and services. Collaborate with stakeholders to gather requirements and deliver solutions. Integrate ServiceNow with other enterprise systems. Maintain and optimize existing ServiceNow implementations. Qualifications: Proven experience as a ServiceNow Developer. Strong knowledge of JavaScript and ServiceNow Scripting. Familiarity with ITIL processes. Excellent problem-solving and communication skills. How to Apply: HIT apply now for a quick response! bluewaveSELECT is a global recruitment business that is exclusively focused on SAP. Involved in placing over 4,000 SAP professionals, our team has helped deliver some of the highest-profile SAP programmes for some of the most recognisable brands all over of the world. We are on a mission is to become the world's most trusted SAP recruitment business, delivering our unique wow factor by consistently exceeding expectations for both our customers and our network of SAP professionals. We are an official SAP partner, you will find us on the Ariba Network and we have a 5-Star approval rating on TrustPilot from fellow SAP enthusiasts. Whether this will be your first experience with us, or you are a bluewaveSELECT regular, we look forward to working with you! Disclaimer; We try and contact all applicants to every advert however due to volumes of responses with some adverts we may not be able to get back to you in some cases. If this is the case and you do not hear from us within 5 working days please take it that your application has been unsuccessful. Please note, your profile will be rejected if you do not meet language/legislation criteria. Thank you.
- Company Name
- Bangura Solutions
- Job Title
- Technical Project Manager - Infrastructure, Database Migration, Oracle on prem, SC Cleared. £665
- Job Description
- Our client within the public sector requires a proven Technical Project Manager. Preferably coming from an infrastructure background. To help deliver a major transformation programme from its Legacy technology estate. You MUSThold active SC Clearance to be eligible for this role. Key requirements: Supplier management - working with multiple suppliers Good planner, problem solving, good assurance, good around governance 10+ years Project management experience (infrastructure PM ideally) Data base migrations/Oracle on premise Essential skills and experience: Proven ability to undertake detailed analysis of technical designs and provide the business with assurance of supplier designs and proposals. Capable of leading VFM challenges of technical proposals and manage technical interdependencies within a project/programme. Bridge the gap between the technical and non-technical by speaking on behalf of technical teams and facilitate relationships with indirect stakeholders. Experience across physical and cloud-based architectures. An understanding of operating systems and oracle products preferable. Understand how governance works and what governance is required by taking responsibility for the assurance of parts of a service and know what risks need to be managed. Good project management skills including planning and change management. Make decisions characterised by managed levels of risk and complexity and recommend decisions as risk and complexity increase. Resolve technical disputes between peers and indirect stakeholders, taking into account all views and opinions. Apply strategy, using patterns, standards, policies, roadmaps and vision statements, challenging them and providing guidance. Understand trends and practices outside your team and how these will impact your work. See how your work fits into the broader strategy and historical context and consider the patterns and interactions on a larger scale. Minorities, women, LGBTQ+ candidates, and individuals with disabilities are encouraged to apply. Interviews will take place next week, so please apply immediately to be considered for this exciting contract role or call Bangura Solutions to discuss this role further.
- Company Name
- Scope AT Limited
- Job Title
- Data Centre Engineer - Server, Dell, HP, IBM, Support
- Job Description
- Looking for a Data Centre Engineer on a Contract basis. Must have own transport and be able to travel to Camberley. Break fix server engineer working on HP, Dell and IBM Servers, Server provisioning, Rack, stack and configure Servers. London/Hybrid CONTRACT By applying to this job you are sending us your CV, which may contain personal information. Please refer to our Privacy Notice to understand how we process this information. In short, in order to supply you with work finding services, we will hold and process your personal data, and only with your express permission we will share this personal data with a client (or a third party working on behalf of the client) by email or by upload to the Client/third parties vendor management system. By giving us permission to send your CV to a client, this constitutes permission to share the personal data that would be necessary to consider your application, interview you (Phone/video/face to face) and if successful hire you. Scope AT acts as an employment agency for Permanent Recruitment and an employment business for the supply of temporary workers. By applying for this job you accept the Terms and Conditions, Data Protection Policy, Privacy Notice and Disclaimers which can be found at our website.