Data Engineer
Hybrid
London, United Kingdom
Freelance
04-03-2025
Job Specifications
Data Engineer (Databricks Specialist)
6-Months
London (Hybrid)
We are seeking an experienced Data Engineer with deep expertise in Databricks to support the design, development, and optimization of a Unified Data Platform. The ideal candidate will have a strong background in data engineering, pipeline development, and system integration, with proficiency in Databricks, PySpark, Python, and SQL.
This role requires hands-on technical leadership, guiding best practices for data architecture, performance optimization, and migration from legacy systems. You’ll collaborate with internal stakeholders and vendors to ensure solutions align with business objectives and technical strategy.
Key Responsibilities:
Databricks Expertise: Act as a Subject Matter Expert (SME), driving the adoption and optimization of Databricks across the organization.
Data Pipeline Development: Design, build, and optimize scalable, efficient data pipelines using Databricks, PySpark, and Python.
Data Architecture: Contribute to the design and development of a robust, scalable Unified Data Platform, ensuring performance and efficiency.
System Integration: Work on seamless integration of Databricks with Azure Data Services and other enterprise platforms.
Legacy Migration: Lead efforts to migrate legacy data systems into Databricks, ensuring smooth transitions and minimal disruption.
Collaboration & Governance: Work with internal teams and external vendors to establish best practices, governance frameworks, and data strategy alignment.
Essential Skills & Experience:
Expert-level knowledge of Databricks, with a proven track record of building and optimizing data solutions.
Strong experience in PySpark and Python, developing and optimizing large-scale data pipelines.
Proficiency in SQL, with expertise in querying and managing complex datasets.
Azure Data Services expertise, including Azure Data Factory, Azure Synapse Analytics, and Azure Storage.
Experience in data architecture and performance optimization for cloud-based data platforms.
Ability to bridge the gap between business needs and technical solutions, ensuring scalability, reliability, and efficiency.
About the Company
Osmii believes the journey starts with the people you hire. Osmii is a firm of specialist headhunters providing worldwide resource to Agencies, Brands and Consultancies. We offer support across the Technology, Creative and Digital sectors in Europe, MEA, APAC, US and LATAM. Osmii has two key service offerings: a Specialist Headhunting partner building permanent teams internationally and a Global Services partner delivering locally compliant temporary/project based resource. We have a history of building long standing s... Know more
Related Jobs
- Company Name
- Gigged.AI
- Job Title
- Data Analyst needed to develop and maintain Customer Success Dashboards
- Job Description
- Gigged.AI is a freelance talent marketplace specialising in the IT and technology sector. We have an opportunity live at the moment with one of our clients for a Data Analyst needed to develop and maintain Customer Success Dashboards. This expected duration is 3-6 months with a rough budget of £400pd. If interested then you must submit a proposal through the Gigged.AI website for this gig. You can do so here - https://app.gigged.ai Project Overview: Our client is looking for an experienced Data Analyst to work alongside their Customer Success team. The primary responsibility of this role is to maintain and develop Customer Success dashboards that will analyse and report on revenue-generating activities. You will collaborate closely with business stakeholders to ensure the delivery of user-friendly and comprehensive executive dashboards. Expected Start Date: ASAP (March 2025) IR35: Pending final determination (expected Outside IR35) Duration: The initial contract duration will be 6 months, with working hours being 9-5 Monday-Friday. Working Practises: This is not a remote role, you will be expected full-time in the London City Centre Office. Key Responsibilities: 1. Dashboard Development: Design and build interactive and intuitive Customer Success dashboards to report on retention and revenue generating activities activities. Develop and maintain robust and scalable ETL processes to support dashboard creation and data integration. Utilise SQL and Python to query databases, perform data manipulation, and automate analysis processes. 2. Experimentation: Support experimentation efforts of our Growth & Retention organisation by analysing and reporting on A/B testing results. Provide historical data for key metrics to help establish baseline performance and experiment targets. 3. Collaboration: Work closely with Customer Success, Sales, and Marketing teams to understand reporting needs and refine dashboard functionalities. Present findings and insights to business stakeholders and executives in a clear and concise manner. 4. Quality Assurance: Ensure data accuracy, integrity, and consistency by implementing best practices in data governance and quality control. Troubleshoot and resolve any data-related issues promptly. Qualifications: Technical Skills: High proficiency in SQL. Proven experience in building dashboards in Tableau and Qliksense (or similar reporting tools). Familiarity with ETL tools and processes. Experience with A/B testing methodologies and analysis. Industry Experience: Experience working with call center data, including understanding of metrics and KPIs relevant to call center operations. Experience working with web data and understanding of digital support tools and KPIs. Professional Requirements: A track record of delivering easy-to-use dashboards for business stakeholders and executives. Strong problem-solving skills and the ability to work independently. Excellent communication skills for presenting analysis and insights to non-technical stakeholders. Ability to manage time effectively and prioritise tasks to meet project deadlines. Preferred Qualifications: Knowledge of statistical analysis and predictive modelling. Experience working in a fast-paced, dynamic environment. Examples of jobs to be done: Maintain Smart Digital Recommendations reporting and add new use cases to the existing reporting. Support the CS INTL analytics team on experiments related to revenue- generating activities. Maintain Smart Expert Recommendations reporting and provide new insights to the Growth & Retention teams on an ongoing basis. Improve the existing Retention reporting to reflect revenue. Support the shift of the clients data structure from the existing market model to the new regional model for CS data. Support Growth & Retention WBRs with insights to facilitate the improvement of revenue-generating programs.
London, United Kingdom
On site
Freelance
03-03-2025
- Company Name
- W3Global
- Job Title
- Data Solutions Architect
- Job Description
- Role: Data Architect Is it Permanent / Contract: Fine with Both Is it Onsite/Remote/Hybrid: Hybrid Start Date: Immediate Mandate Skills: Data Design Governance, Design approval and sign off. Enforce Design standards. Job Description Solution Design: Architect and design the data and technical blueprint for data solutions on the Enterprise data platform, ensuring they meet enterprise and technical requirements. Collaboration: Work closely with business, technology, and data teams to align solutions with enterprise objectives and support definition of enterprise data patterns. Implementation: Guide the technical implementation of data processing, storage, integration strategies, consumption, and platform feature to ensure optimal data flow. Performance Optimization: Ensure data solutions are optimized for scalability, performance, and security, adhering to best practices in data architecture. Governance & Compliance: Ensure that all data solutions comply with governance frameworks and regulatory requirements. Continuous Improvement: Identify opportunities and make recommendations to improve the quality and trustworthiness of data and processes. The ideal candidate should have Expertise in designing and delivering comprehensive data solutions on cloud including data modelling, management, data integration, data transformation, consumption and deployment of data. Strong background and working experience in insurance domain and understanding of the underwriting process and role of brokers/agents in business process. Knowledge of data governance and data security practices. Experience with MS Power Platform, CRM systems to support the strategic programs in the upstream. Good understanding of managing reference data and its integration with applications.
London, United Kingdom
Hybrid
Freelance
26-02-2025
- Company Name
- NHS Counter Fraud Authority
- Job Title
- Data Engineer
- Job Description
- Job Summary The NHS Counter Fraud Authority is the national body responsible for all matters relating to the prevention, detection and investigation of economic crime across the NHS. Further information about our work and annual plan for delivering this is available on our website. As a Data Engineer you will have direct oversight and responsibility for the development implementation review and establishment of data and techniques used in the Advanced Data Analytics Portfolio (DAP). You will collaborate with project managers, data scientists, counter fraud experts, stakeholders, and senior management across the NHSCFA and our wider stakeholders. The primary responsibilities will include developing, evaluating, maintaining, and managing data pipelines, processing, preparing data suitable for analysis, and developing and maintaining data models. You will support the design of analytical services, products, and models that meet the needs of NHSCFA, and guide, develop, document, and maintain data pipelines and storage solutions used in the analytics environment, ensuring that the infrastructure and methods are secure, optimized, and scalable within the Azure cloud computing environment. The post holder will be required to have a NPPV3. This is a Fixed Term contract until 31st March 2026 Potential applicants can contact Naga Balam at naga.balam@nhscfa.gov.uk for an informal chat if you have any questions regarding the role. Interviews will be held on 17/03/25 Main duties of the job Our new team are embarking on a piece of work to monitor data to identify and respond to patterns indicative of potential fraud. This will support our current work that reduces the likelihood of fraud occurring. We will bring in data science capabilities to be deployed in counter fraud activity and work closely with partners across health and government to further maximise the preventative impact of proactive counter fraud analysis. We will combine this with our range of counter fraud and domain expertise to maximise our impact using your knowledge, experience, and passion for your chosen field. At the core will be the development and use of appropriate analytical techniques that include advanced analytics and machine leaning to tackle some of the highest areas of fraud risk to the NHS and ultimately protect public funds for patient care. About Us We have offices based in Coventry and Newcastle. The NHSCFA values and respects the diversity of its employees and aims to recruit a workforce which reflects our diverse communities. We welcome applications irrespective of people's age, disability, gender, race or ethnicity, religion or belief, sexual orientation, or other personal circumstances. We have policies and procedures in place to ensure that all applicants are treated fairly and consistently at every stage of the recruitment process, including an invitation to the first stage of the selection process and consideration of reasonable adjustments for people who have a disability. If you are applying to undertake this role on a secondment basis you should have agreement to being released from your current role in principle, prior to submitting an application form. When you apply for this role, you will be redirected to our recruitment system TRAC. The NHSCFA does not hold a sponsor licence in respect of skilled worker visas and so is unable to employ candidates requiring sponsorship. We reserve the right to close this vacancy before the advertised closing date should we receive a significant number of applications. Date posted 19 February 2025 Pay scheme Agenda for change Band Band 8a Salary £53,755 to £60,504 a year Contract Fixed term Duration 12 months Working pattern Full-time, Home or remote working Reference number 076-ATH-CFA029-B Job locations NHSCFA, Cheylesmore House, Coventry CV1 2WT Job Description Job responsibilities Develop and maintain robust, efficient, secure, and reliable data pipelines using azure technologies including Fabric, Azure data factory, Synapse analytics, databricks and Lakehouses. Build data pipelines that clean, transform, and present granular and aggregate data from disparate sources. Design, build, test, automate, and maintain architectures and processing workflows for analytics and business intelligence (BI) systems which are scalable. Plan, develop and evaluate methods and processes for gathering, extracting, transforming, and cleaning data and information. To undertake development of the data warehouse, including overall design, technical development and documentation of the data warehouse, infrastructure and ETL solutions covering multiple sources of data, working alongside NHSCFA specialists and external data suppliers. Responsibilities Job description Job responsibilities Please see full Job Description and Person Specification Develop and maintain robust, efficient, secure, and reliable data pipelines using azure technologies including Fabric, Azure data factory, Synapse analytics, databricks and Lakehouses. Build data pipelines that clean, transform, and present granular and aggregate data from disparate sources. Design, build, test, automate, and maintain architectures and processing workflows for analytics and business intelligence (BI) systems which are scalable. Plan, develop and evaluate methods and processes for gathering, extracting, transforming, and cleaning data and information. To undertake development of the data warehouse, including overall design, technical development and documentation of the data warehouse, infrastructure and ETL solutions covering multiple sources of data, working alongside NHSCFA specialists and external data suppliers. Please See Full Job Description And Person Specification Person Specification Knowledge And Experience Essential Practical application expertise of managing data engineering projects and processes including data wrangling methods, models, data structures, and data formats such as JSON, XML and XSD. Machine learning for engineering practices, such as meta driven intelligent ETL and pipeline processes. Strong skills in relevant programming languages, frameworks, and platforms including, SQL, Python, Spark, R etc. A strong track record of achievement in data engineering. Experience/understanding of data lifecycle management frameworks and project management methodology. Practical implementation of CI/CD in Azure DevOps. Experience in technologies relating to Data Science, for example: statistical analysis, machine learning, or natural language processing. Experience with relational SQL and databases. Experience with Azure tech stack such as Fabric, Data factory, Synapse Analytics, Databricks and Lakehouses. Experience in building data engineering projects in Fabric. Knowledge of data warehousing and modelling concepts such as CDC/SCD. Ability to troubleshoot and solve numerical and technical problems. Extensive experience within a complex and demanding env...
Coventry, United Kingdom
Hybrid
Freelance
04-03-2025
- Company Name
- 9NEXUS
- Job Title
- Contract Data Modeler - Remote, Outside IR35
- Job Description
- We are looking for an experienced Data Architect with 8+ years in pharmaceuticals or related industries to join our team on a contract basis (Outside IR35, Remote – UK-based). Rate: £400 - £450 per day Must-Have Skills: 8+ years of Data Modelling, Metadata Management & Data Governance Hands-on experience with RDBMS, NoSQL, Data Warehousing, Data Lakes Expertise in Data Fabric/Data Mesh Architecture Proficiency with Erwin DM, ER Studio Strong understanding of Pharma Research & CMC processes Excellent stakeholder management and communication Key Responsibilities: Develop Conceptual, Logical, and Physical Data Models for Data Mesh/Fabric architectures Design and govern enterprise-wide data modelling principles and ontologies Collaborate with business and tech teams to drive data strategies and governance Ensure consistent metadata and data quality standards Industry: IT Services & Consulting Location: Remote (UK-based) Contract Type: Outside IR35 Rate: £400 - £450 per day
United Kingdom
Remote
Freelance
26-02-2025