Sr. Databricks Engineer

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows using Python, PySpark, and SQL.
• Develop data ingestion, transformation, cleansing, and integration processes from multiple data sources.
• Implement and maintain Delta Lake architecture and Lakehouse solutions.
• Develop and manage Databricks notebooks, workflows, jobs, and scheduled tasks.
• Work with batch and real-time data processing solutions using Spark Structured Streaming.
• Optimize Spark jobs, SQL queries, cluster configurations, and overall data processing performance.
• Implement data quality checks, validation, monitoring, and error-handling mechanisms.
• Work with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform.
• Integrate Databricks with cloud storage, databases, data warehouses, and other enterprise applications.
• Implement data security, governance, access controls, and compliance standards.
• Collaborate with data architects, data engineers, business analysts, and other technical teams.
• Develop and maintain CI/CD pipelines for automated deployment and integration.
• Troubleshoot production issues and improve the reliability and performance of data pipelines.
• Participate in code reviews, technical discussions, and Agile development activities.

Required Technical Skills

• 8+ years of experience in Data Engineering, Big Data Development, or related technologies.
• Strong hands-on experience with Databricks development and administration.
• Expertise in Python, PySpark, and SQL.
• Strong understanding of Apache Spark architecture and distributed data processing.
• Experience developing complex ETL/ELT pipelines.
• Hands-on experience with Delta Lake and Delta Tables.
• Experience with Databricks notebooks, workflows, jobs, and cluster management.
• Strong knowledge of data warehousing concepts and data modeling.
• Experience working with structured, semi-structured, and unstructured data.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Knowledge of data integration, data transformation, and performance optimization.
• Experience with Git, version control, and CI/CD practices.

Preferred Skills

• Experience with Azure Databricks and Azure Data Factory (ADF).
• Knowledge of Databricks Unity Catalog and data governance.
• Experience with Delta Live Tables (DLT).
• Knowledge of Auto Loader and Spark Structured Streaming.
• Experience with AWS S3, Azure Data Lake Storage (ADLS), or Google Cloud Storage.
• Familiarity with Azure DevOps, Jenkins, GitHub Actions, or similar CI/CD tools.
• Experience with Terraform or infrastructure-as-code tools.
• Knowledge of MLflow and machine learning integration with Databricks.
• Experience with Snowflake, Redshift, Synapse Analytics, or other data warehouses.
• Understanding of data security, metadata management, and data lineage.