Lead Data Engineer (Azure)

Job description

About Fusemachines

Fusemachines is a leading AI strategy, talent, and education services provider.

Founded by Sameer Maskey Ph.D., Adjunct Associate Professor at Columbia University, Fusemachines has a core mission of democratizing AI.

With a presence in 4 countries ( Nepal, United States, Canada, Dominican Republic) and more than 400 full time employees, Fusemachines seeks to bring its global expertise in AI to transform companies around the world.

About the role

Location: Remote (Contract).

This is a full time position responsible for leading, designing, building, and maintaining the infrastructure required for data integration, storage, processing, and analytics (BI, visualization, and Advanced Analytics) using Microsoft Azure in the Media domain.

We are seeking a Lead Data Engineer with hands‑on Python, Spark experience and proven abilities to lead software development on Data and Analytics products using Agile methodology.

The candidate will lead data engineers and data scientists on the team to achieve architecture and design objectives as agreed with stakeholders.

Qualification & Experience

• Must have a full time Bachelor's degree in Computer Science or similar from a top tier school
• 4+ years of experience with Azure DevOps, Azure Cloud Platform, or other hyperscalers
• At least 4 years of experience as a data engineer with strong expertise in Azure, working on generation of big datasets using different data sources, in the Media industry
• Proven experience delivering projects and products for Data and Analytics as a data engineer
• Following certifications:
  • Microsoft Certified: Azure Fundamentals
  • Microsoft Certified: Azure Data Engineer Associate
  • Microsoft Certified: Azure Solutions Architect Expert (nice to have)
  • Databricks Certified Associate Developer for Apache Spark
  • Databricks Certified Data Engineer Associate (nice to have)

Required Skills/Competencies

• Strong programming skills in Python (must have), Scala, and proficiency in writing efficient and optimized code for data integration, storage, processing and manipulation
• Experience using Markdown to document code or automated documentation tools (e.g. PyDoc)
• Experience with Spark/PySpark (must have, Azure Databricks is a plus), DBT and Kafka, to handle large volumes of data
• Expert in designing and implementing efficient ELT/ETL processes in Azure (experience with Azure Data Factory is a plus)
• Skilled in data integration from different sources such as APIs, databases, flat files, event streaming with Azure Data Factory
• Expertise in data cleansing, transformation, and validation
• Hands‑on experience with Jupyter Notebooks and python packaging and dependency management: Poetry, PipEnv
• Proficiency with Relational Databases (Oracle, SQL Server, MySQL, Postgres, or similar) and NoSQL Databases (MongoDB or others)
• Good understanding of Data Modeling and Database Design Principles
• Strong SQL skills and writing advanced SQL queries
• Experience designing Data Warehousing solutions in Azure with Azure Synapse Analytics and/or Snowflake
• Familiarity with migration of code from SAS, R, Julia, SPSS to Python
• Proven technical leadership on prior Big Data projects
• Strong understanding of the software development lifecycle (SDLC), especially Agile methodologies
• Knowledge of SDLC tools and technologies Azure DevOps, Jira, Azure Boards, source code management (GitHub, Azure Repos, Bitbucket), CI/CD (GitHub Actions, Azure Pipelines, Jenkins)
• Understanding of DevOps principles, CI/CD, IaC, configuration management, automated testing, cost management
• Strong knowledge of Microsoft Azure services related to data and analytics (Data Factory, Databricks, Synapse, Stream Analytics, Blob Storage, Data Lake Storage, SQL Database)
• Experience in orchestration using Apache Airflow
• Strong analytical skills to identify and address technical issues, performance bottlenecks, system failures
• Proficiency in debugging and troubleshooting issues in complex analytics environments and pipelines
• Good understanding of Data Quality and Governance, with implementation of data quality checks and monitoring
• Good understanding of BI solutions including PowerBI and Tableau
• Knowledge in containers (Docker, Kubernetes, etc.) is a plus
• Effective written and verbal communication skills for collaboration across teams
• Ability to document processes, procedures, and deployment configurations
• Strong leadership skills, assertiveness
• Understanding of Azure security practices, network security groups, Azure AD, encryption, compliance
• Implementing security controls and best practices for data and analytics solutions
• Willingness to stay updated with latest Azure services, Data Engineering trends, best practices
• Well organized, able to prioritize effectively in fast paced environment
• Team player, self‑motivated, independent, detail oriented
• Commitment to agility, continuous learning, adaptability to business needs

Responsibilities

• Lead engineers on the team to meet product deliverables
• Architect, design, develop, test, optimize and maintain high performance, large‑scale data architectures prioritizing best practices, for data intake, validation, mining, engineering for delivering data products
• Support data integration (batch and real‑time), storage, processing, and infrastructure to ensure scalability, reliability, and performance of data systems
• Provide mentorship, coaching and guidance to junior data engineers and foster their professional growth
• Collaborate with cross‑functional teams (Product, Engineering, Data Scientists, Analysts, Cloud Architects, DevOps engineers) to drive discovery and requirements gathering for data management and business analytics
• Work with the product management team to understand roadmap commitments and communicate design and implementation milestones effectively
• Identify and solve code/design optimization challenges
• Learn and integrate with a variety of systems, APIs, and platforms
• Establish a full‑proof QA process for data validations and overall quality control on the product
• Continuously evaluate and implement new technologies and tools; promote reusable components
• Design, implement, and maintain data governance solutions; manage cataloging, lineage, data quality, and privacy efforts
• Work independently and collaboratively on a multi‑disciplined project team in an Agile development environment
• Deliver on projects and initiatives that impact business development, client support, and operational areas
• Contribute to continuous improvement activities
• Stay updated on market trends and emerging technologies

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

Required Skill Profession

Gestión Informática Y Gestión De Proyectos Informáticos