Data Engineer, Databricks
On site
London, United Kingdom
Full Time
25-11-2024
Job Specifications
Data Engineer, Azure, DataBricks
London/hybrid
£90K + bonus and benefits
SQL, DataBricks, Azure Data Factory (ADF), Data Lakehouse
Data Engineer with extensive experience in Data Lakehousing practices, using SQL, DataBricks, and Azure Data Factory (ADF) is required to join a vibrant and growing insurance organisation with offices in the heart of the city. This is a pivotal role that will see you working collaboratively with stakeholders and wider data teams to develop key data platforms including a new Data Lakehouse.
This is a fantastic opportunity to join a forward-thinking, people-centric organisation with excellent scope for long term career development.
Key Responsibilities:
Develop and enhance the Group Data Lakehouse and other data platforms.
Collaborate with key business, data and tech teams to understand data requirements and deliver solutions.
Set and enforce data management standards, ensuring best practices across the organisation.
Conduct solution reviews and designs, providing assurance on development approaches.
Manage multiple project deliveries simultaneously with outstanding attention to detail.
Provide guidance and support to wider data teams as required
Key Skills and Experience:
Extensive experience in Data Lakehousing practices, particularly using SQL, DataBricks, and Azure Data Factory (ADF).
Strong data management background with hands-on experience.
Proven ability to manage multiple projects in a fast-paced, agile, and dynamic working environment.
Excellent attention to detail and ability to maintain high standards.
Strong problem-solving skills and ability to provide technical guidance and support.
Excellent communication skills, both written and verbal, with the ability to influence stakeholders.
Experience working within an MGA (Managing General Agent) or (re)insurance carrier is highly desirable.
For a full consultation, please send your CV to Arc IT Recruitment.
About the Company
Founded in 2000 by Marcus Freeman, ARC IT provides recruitment solutions for both contract and permanent positions across a broad range of IT and business skill sets, from graduate entry level to heads of global businesses. We have enjoyed steady growth among our loyal clients and welcome new employers every month due to our exceptional reputation in the IT recruitment industry. As well as recruiting permanent IT staff across the UK and managing a high number of contractors we also have a significant presence in Europe and... Know more
Related Jobs
- Company Name
- Auma
- Job Title
- AI Engineer, Tech Startup
- Job Description
- Role Overview As an AI Engineer, you’ll develop and implement advanced machine learning and AI solutions that power cutting-edge applications. Collaborating with interdisciplinary teams, you'll design, train, and deploy models that solve complex problems, enhance user experiences, and drive business innovation. Key Responsibilities Develop AI Models: Design, train, and deploy machine learning models tailored to solve business problems or enhance product capabilities. Data Analysis and Preparation: Clean, preprocess, and analyze large datasets to enable effective model training and evaluation. Deploy and Optimize Solutions: Build scalable AI systems, integrating models into production environments while ensuring optimal performance and reliability. Collaborate Across Teams: Work closely with data scientists, engineers, and product managers to align AI solutions with business objectives. Research and Experiment: Explore and evaluate emerging technologies, frameworks, and algorithms to enhance current systems or create new capabilities. Monitor and Improve Models: Continuously evaluate model performance and refine them based on feedback and changing requirements. Document and Communicate Findings: Create detailed documentation for processes, models, and results, sharing insights with technical and non-technical stakeholders. Key Requirements Experience: Proven experience in AI/ML development, with specific expertise in working with large language models (e.g., OpenAI GPT, LLaMA, Hugging Face Transformers). Technical Skills: Proficiency in Python and machine learning frameworks such as TensorFlow, PyTorch, or Scikit-learn. Strong foundation in data structures, algorithms, and statistics. Experience with natural language processing (NLP), computer vision, or recommendation systems. Familiarity with cloud platforms (e.g., AWS, GCP, Azure) for deploying AI solutions. Understanding of data pipeline tools like Apache Spark, Kafka, or Airflow. Soft Skills: Excellent problem-solving skills, communication abilities, and adaptability to fast-paced environments. Bonus Skills Experience with federated learning, reinforcement learning, or edge AI. Knowledge of tools like Hugging Face, OpenAI APIs, or similar. Familiarity with MLOps practices, including CI/CD pipelines for AI systems. Experience working in startup environments. What We Offer A unique opportunity to be matched with client startups aligned with your skills, career aspirations, and interests. Opportunities to work on impactful projects leveraging cutting-edge AI technologies. Access to a network of innovative startups and tech leaders. Potential for professional growth and learning in dynamic, high-impact teams.
- Company Name
- East 57th Street Partners
- Job Title
- AI/ML Engineer
- Job Description
- Job Title: AI/ML Engineer Location: 100% Remote with Quarterly Travel to Company HQ Compensation: Based on Experience and Qualifications, Subject to Base Salary and Bonus About East 57th Street Partners International East 57th Street Partners International partners with companies at the forefront of technology to provide innovative solutions in AI and machine learning. Through our Executive and Direct Hire Search Practice, we place technical leaders who drive cutting-edge development and optimization. Position Overview Our client is seeking an AI/ML Engineer to join their remote team and lead the development of machine learning models to enhance their product offerings. The AI/ML Engineer will work closely with data scientists and software developers to build, train, and deploy models that solve real-world problems. This position requires quarterly travel to the company’s headquarters for key strategic discussions and collaboration. Key Responsibilities Design, develop, and implement AI and machine learning algorithms. Collaborate with cross-functional teams to integrate AI solutions into existing products. Use advanced statistical techniques to identify patterns and build predictive models. Train machine learning models, optimize their performance, and monitor outcomes. Stay updated on the latest advancements in AI/ML technologies and implement best practices. Work with large datasets to develop scalable AI solutions. Qualifications Bachelor’s or Master’s degree in computer science, AI, machine learning, or a related field. Proven experience in AI/ML development, including model building and deployment. Proficiency in Python, TensorFlow, PyTorch, and other AI/ML frameworks. Strong problem-solving and analytical skills. Ability to work independently in a remote setting while collaborating with global teams. Compensation and Benefits Competitive compensation package based on experience and qualifications, including base salary and bonus. Access to cutting-edge AI/ML projects and professional development opportunities.
- Company Name
- TRIA
- Job Title
- Finance Data Analyst
- Job Description
- Finance Data Analyst Up to £55,000 Hybrid - Central London - 2x per week We are representing a B2B professional services business who are looking for a Finance Data Analyst to join their organisation. This is a global business, headquartered in the UK, with a great history and reputation in the market. This role will report to the Finance Manager and will join an established CFO function of approximately 15 people. As the Finance Analyst, you will be work on a mixture of reporting, financial model development and supporting their finance system (Infor Sun Systems). You will develop financial models using Excel, produce and analyse monthly management information, and develop interfaces between the finance system and other systems - eg expenses, timesheet recording etc. The company are looking to upgrade their ERP system in future, as well different finance applications, so there will be the opportunity to be a part of several important projects focused on bettering the business. We are looking for: Advanced reporting and modelling skills with Excel - eg VLOOKUP, PivotTables, VBA, and Macros Prior experience working within Finance teams Strong communication and interpersonal skills It would be a bonus if you had: Previous experience with Infor Sun Systems software Experience with Visio Data Visualizer add-in for Excel An accountancy background or relevant qualification/training If this role sounds of interest, please apply today!
- Company Name
- iO Associates - UK/EU
- Job Title
- Junior /Mid Data Engineer
- Job Description
- Junior/Mid Data Engineer - Exciting Opportunity in Burgess Hill Are you passionate about data optimization and analytics? Join My client as a Data Engineer, where you'll design and maintain data pipelines, enhance data quality, and drive business insights. This role involves working with cross-functional teams to automate processes, support data science initiatives, and contribute to digital transformation projects. If you're skilled in SQL, Azure Data Factory, big data tools, and scripting languages like Python or Java, this is your chance to make an impact in a dynamic environment. Occasional global travel may be required. Apply now and shape the future of data! Strong SQL knowledge and experience with relational and NoSQL databases. Proficiency in building and optimizing data pipelines and architectures. Experience with data pipeline/workflow management tools. Expertise in big data tools such as Hadoop, Spark, and Kafka. Proficiency in object-oriented or functional scripting languages: Python, Java, C++, Scala, etc. Experience with Azure Data Factory. Skilled in root cause analysis of data and processes. Ability to build processes supporting data transformation, data structures, metadata, and dependency management. Excellent project management and organizational skills. Familiarity with Microsoft Office, particularly Excel; Power BI is a plus. Basic knowledge of Machine Learning and SAP is advantageous. Please feel free to apply!