Cloud and Data Engineer

Cloud and Data Engineer

Location: KwaZulu Natal/Gauteng    

MagicOrange is a globally recognized leader in the IT Financial Management Software market, as acknowledged by Gartner. With customers and a strong presence on four continents, we are a Software as a Service (SaaS) provider in a high growth phase. Our mission is to empower individuals and organizations, making them more valuable through our innovative software solutions.

Position Summary:

The Cloud and Data Engineer is responsible for designing, implementing, and maintaining functional backend software solutions and services. This role involves leveraging cloud technologies and data engineering principles to build robust, scalable, and efficient systems that support the organization's data needs.

The successful incumbent will be a member of the MagicOrange Engineering team and operate in conjunction with the wider business, delivering measurable business results by developing high-quality software that is aligned with user needs and business goals.

Required candidate with ability to prioritize well, communicate clearly, have a consistent track record of delivery and excellent modern software engineering skills (particularly in the Data and Cloud disciplines). Must be able to work across multiple facets of the project and juggle multiple responsibilities at the same time. Strong analytic capability and the ability to create innovative solutions. A key dimension of this role is applying AI and agentic techniques to data engineering challenges: the ideal candidate will have practical experience integrating LLMs into data workflows, writing and running automated tests within Databricks, and building intelligent pipelines that go beyond traditional ETL. We are looking for a highly motivated individual who is excited to grow their career in a fast-paced, AI-first environment.

Key Responsibilities:

Data Process Optimization:

• Optimize and enhance existing data pipelines, ETL processes, and data workflows to improve performance, scalability and reliability.
• Implement best practices for data processing, storage, and retrieval.
• Implement automated testing within Databricks using frameworks such as pytest and nutter: write unit tests for notebooks and PySpark transformations, integration tests across pipeline stages, and data quality assertions, with results surfaced through CI/CD pipelines.

Algorithm Support and Development:

• Design, implement and maintain algorithms to solve technical problems related to data processing and analytics.
• Collaborate with data scientists to support and optimize machine learning algorithms and models in production.
• Assist to benchmark and ensure efficient use of computational resources for data processing and algorithm execution.
• Design and implement AI agentic workflows that automate multi-step data tasks: orchestrate LLM-powered agents to perform data discovery, anomaly investigation, and automated root-cause analysis within the data platform.
• Apply prompt engineering and model evaluation practices when integrating large language models into data pipelines; enforce responsible-AI guardrails including output validation and human-in-the-loop review for high-impact decisions.

Data Integration:

• Design, implement and maintain solutions for data integration from various sources, ensuring data consistency and integrity.
• Work on data ingestion and transformation processes to support analytics and reporting needs.
• Validate integrated data using Databricks-native testing approaches: apply Delta Live Tables expectations, Great Expectations, or equivalent data quality frameworks to enforce schema, completeness, and accuracy contracts at ingestion.

Collaboration and Support:

• Work closely with the technical product owner to understand business requirements and translate them into technical solutions.
• Provide technical support and guidance to other team members regarding data-related issues.
• Work closely with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
• Help to document key processes and services for the purposes of sharing the load and approach to technical support related issues.

Dashboarding, Reporting, and Data Interpretation:

• Work in conjunction with the BI Engineering to develop and maintain both product and internal dashboards and reports.
• Interpret data to provide meaningful insights and recommendations to stakeholders.
• Work closely with business teams to understand their and customer reporting needs and deliver tailored solutions.
• Ensure data accuracy and integrity in all reports and dashboards.

Previous Experience:

• 3+ years of experience in cloud engineering, data engineering, or a similar role within a SaaS environment.
• Demonstrable experience with Databricks: notebooks, Delta Lake, DLT pipelines, Jobs, and writing automated tests for data transformations within the platform.
• Hands-on exposure to AI/ML pipelines or agentic data workflows; experience with Azure OpenAI, Azure AI Search, or equivalent services advantageous.

Skills and Requirements:

Essential Skills:

• Strong Mathematical, Analytical, Conceptual and Problem-Solving Abilities.
• Solution Driven.
• Ability to find the root cause of problems and quickly determine effective solutions.
• Ability to anticipate risk.
• Troubleshooting, analytical and attention to details.
• Ability to prioritize and manage time effectively.
• Excellent Communication Skills.
• Proficiency in cloud platforms such as Azure, AWS or Google Cloud.
• Strong experience with data processing frameworks and tools (e.g., Databricks, Apache Spark, Hadoop, Kafka).
• Strong Expertise in SQL and experience with NoSQL databases.
• Familiarity with Git.
• Proficiency in programming languages such as Python
• Experience building AI agentic systems: multi-agent orchestration, tool/function calling, RAG pipelines, or similar autonomous workflow patterns applied to data engineering problems.
• Working knowledge of AI/LLM frameworks relevant to data engineering such as Semantic Kernel, LangChain, AutoGen, or the Azure OpenAI Service SDK; familiarity with prompt engineering and model evaluation.
• Familiarity with vector databases and embedding models (e.g. Azure AI Search, Chroma, pgvector) advantageous; understanding of retrieval-augmented generation (RAG) patterns a plus.
• Proficiency in automated testing within Databricks: pytest, nutter, Delta Live Tables expectations, or Great Expectations; ability to integrate test runs into Azure DevOps or equivalent CI/CD pipelines.

Qualifications:

• Matric
• Bachelor’s degree in Computer Science, Information Technology, or a related field. Advanced degree is a plus.

What we offer:

• Strong entrepreneurial spirit. The ability to make an impact and see the rewards of your efforts.
• Ongoing training on the latest technologies to aid automation for accountants.
• Be part of a high growth industry and product.
• A challenging career in an innovative company.
• Opportunity to influence, working in an open climate, close to decision makers at large blue-chip enterprise with the possibility to make a difference.
• A competitive remuneration package, with flexible pension options.

MagicOrange is an equal opportunity employer, committed to promoting diversity and inclusion in the workplace. We value and appreciate the diverse contributions and perspectives of all our employees.