Overview
We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. The ideal candidate will design and build scalable ETL/ELT solutions, implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.
Key Responsibilities
- Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
- Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
- Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
- Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
- Optimize Spark jobs for performance, reliability, and cost efficiency.
- Implement data validation, quality checks, and monitoring with automated alerts and retries.
- Design secure and governed datasets using Unity Catalog and cloud security best practices.
- Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
- Provide technical leadership and guidance to junior team members.
Required Skills
- 5–8+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
- Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
- Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
- Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
- Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
- Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).
Good to Have
- Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
- Knowledge of Power BI or data visualization concepts.
- Familiarity with Terraform, Scala, or PostgreSQL.
- Experience with SharePoint development or .NET (optional).
Top Skills
What We Do
Unison Consulting was launched in Singapore on September 2012, the hub of the financial industry, with innovative visions in the technocratic arena. We are a boutique next-generation Technology Company with strong business-interests in Liquidity risk, Market Risk, Credit Risk and Regulatory Compliance.
Unison provides technology consulting and services to implement Risk Management and Risk Analytics System for Financial Institutions. Our services suite comprises of Techno-Functional consulting, systems integration, Business Intelligence, information management, and custom development of IT solutions, plus project management expertise for financial institutions.
We have expertise in latest cutting edge technology to achieve better total cost of ownership. Through our qualified professionals, we assist you drive your unique risk management strategies, whether that means efficient monitoring, improving risk appetite of the financial institutions, complying with regulations, or capturing growth opportunities through innovation, this is what maximizes your decision taking potential. At Unison Consulting, we view clients as partners, and our success is only measured by the success of our partners. So we put it all on the table in order to exceed expectations.
Our staff consists of young, energetic and innovative consultants who are never afraid to challenge the conventions and push the boundaries in an effort to help our clients. For every project, no matter how large or how small, we strive to not only meet your needs, but deliver a showcase in your field
