Cloud Engineer

We are seeking a specialised Cloud Engineer to design, secure, and maintain the infrastructure that powers modern data platforms. This role focuses on the unique requirements of data workloads, including large-scale storage, high-throughput networking, and compute clusters for Spark/SQL.

You will bridge DevOps and Data Engineering, ensuring data teams have robust, secure, and automated platforms to build pipelines, models, and analytics solutions. You will manage the “plumbing” of the data ecosystem, primarily working with Databricks, AWS, and Azure.

Responsibilities:

• Platform Architecture: Design and deploy scalable cloud infrastructure for data lakes and analytics using Infrastructure as Code (Terraform).
• Security and Governance: Implement identity management (IAM/Entra ID), network security (Private Link/VPC), and data governance (Unity Catalog).
• Automation: Build CI/CD pipelines to automate provisioning of compute resources and workspaces.
• Observability: Monitor cost, performance, and reliability of data workloads, ensuring efficient resource use.
• Enable Data Teams: Create self-service infrastructure patterns that let Data Engineers deploy pipelines without managing underlying servers.

Requirements

• Minimum of 5 years professional experience in cloud engineering, with exposure to data engineering and Databricks highly desirable.
• Infrastructure as Code: Terraform (modules, state, workspaces), plus CloudFormation, Bicep, or Crossplane.
• Containerisation and Orchestration: Docker, Kubernetes (EKS, AKS, or self-managed).
• CI/CD and Version Control: Git, Azure DevOps, GitHub Actions.
• Scripting and Automation: Python, Bash, CLIs (DBX, AWS, Azure), REST APIs.
• Cloud Security: Least Privilege access, RBAC, encryption, secrets management.
• Cloud Networking: VNETs/VPCs, Private Link, DNS, Firewalls
• Data Platform Architecture: Medallion architecture, Delta Lake, Data Vault.
• Observability: Cost management, logging (CloudWatch, Azure Monitor), alerting systems.
• Data Workload Understanding: Familiarity with PySpark, SQL, DBT, Spark Structured Streaming.
• Workspace administration via Terraform, Unity Catalog configuration, Cluster Policies, Instance Profiles, Private Link & VNet Injection, Jobs, Delta Live Tables, Airflow integration, MLflow, Mosaic AI, Vector

The Reference Number for this position is NG61051 which is a Permanent, hybrid role in Johannesburg offering a salary of up to R1.08mil per annum salary negotiable based on experience. E-mail Nokuthula on nokuthulag@ e-Merge.co.za or call her for a chat on 011 463 3633 to discuss this and other opportunities.

Are you ready for a change of scenery? e-Merge IT recruitment is a niche recruitment agency. We offer our candidates options so that we can successfully place the right people with the right companies, in the right roles. Check out the e-Merge IT website www.e-merge.co.za for more great positions.

Do you have a friend who is a developer or technology specialist? We pay cash for successful referrals


Posted on 17 Feb 13:24, Closing date 18 Apr