Assistant Manager - Data Engineering

We are seeking a highly skilled and motivated Assistant Manager with strong experience in the Databricks Domain to join our growing team.

• Design and implement scalable and optimized data pipelines on Databricks using Delta Lake, PySpark, and SQL.
  
• Develop ETL/ELT frameworks for batch and streaming data processing.
  
• Ensure data quality, governance, and observability using Unity Catalog, Great Expectations, or custom validations.
  
• Optimize Spark jobs for performance, cost, and scalability.
  
• Cloud & Infrastructure (Azure/AWS/GCP)
  
• Deploy and manage Databricks clusters, workspaces, and Jobs.
  
• Work with Terraform or ARM templates for infrastructure automation.
  
• Integrate cloud-native services like Azure Data Factory, AWS Glue, or GCP Cloud Composer.
  
• MLOps & CI/CD Automation
  
• Implement CI/CD pipelines for Databricks notebooks, workflows, and ML models.
  
• Work with MLflow for model tracking and lifecycle management.
  
• Automate data pipelines using Azure DevOps, GitHub Actions, or Jenkins.
  
• Leadership & Collaboration
  
• Lead a team of data engineers, ensuring best practices and code quality.
  
• Collaborate with data scientists, analysts, and business teams to understand requirements.
  
• Conduct performance reviews, technical mentoring, and upskilling sessions.
  

• Expertise in SQL, ETL/ELT pipelines, and data modeling.
  
• Experience with Azure, AWS, or GCP cloud platforms.
  
• Knowledge of MLOps, MLflow, and CI/CD best practices.
  
• Experience in workflow orchestration using Databricks Workflows, Airflow, or Prefect.
  
• Understanding of cost optimization, cluster tuning, and performance monitoring in Databricks.
  
• Communication and leadership experience, with experience initiating and driving projects.
  
• Strong stakeholder management and mentoring skills.
  
• Experience with data sets, Hadoop, and data modernisation tools.
  
• Development experience in at least one object-oriented language (Python, Java, etc.).
  
• BA/BS in Computer Science, Math, Physics, or other technical fields.
  

• Good to Have: Experience with data lakehouse architectures and Unity Catalog. Hands-on with Terraform, Infrastructure-as-Code (IaC), or Kubernetes. Familiarity with data governance, security, and privacy frameworks.
  

Updated Date

22-09-2025

Job ID

J_4146

Location

Chennai, Tamil Nadu, India ..+ 1

Experience

5 - 8 Years

Employee Type

Permanent