Agilisium Consulting LLC
TITLE: Big Data Specialist (Multiple Positions)
LOCATION: Westlake Village, CA and various and unanticipated locations throughout the U.S. (Must be willing to work anywhere in the U.S. as the position may involve relocation to various and unanticipated client site locations; any relocation to be paid by employer pursuant to internal policy.)
DUTIES: Develop Databricks notebooks to implement Snaplogic pipeline data cleansing and processing. Optimize and manage data manipulation tasks. Implement and automate data flow orchestration. Design and implement modular, standardized pipelines to meet client performance expectations. Develop and execute comprehensive validation processes to ensure data integrity throughout the pipeline. Establish a standardized framework for multiple CDL layers, ensuring scalability and consistency. Create reusable code modules and maintain version control practices. Lead data integration, to drive system improvements. Develop automated processes for ingesting Google Analytics data, enabling actionable insights. EOE
REQTS: Must have a Bachelor’s degree or foreign equivalent in Computer Science, Computer Engineering, or a related field plus five (5) years of experience in the position offered, as a Software Engineer, or a related position. Must have five (5) years of experience with all of the following: Developing and optimizing data pipelines in PySpark and Databricks for large-scale data ingestion and processing, integrating GitLab version control for code management and automated deployments; Implementing advanced SQL development and optimization techniques for complex data modeling, query performance tuning, and relational warehouse management in Amazon Redshift and SQL DW; Engineering and automating ETL/ELT workflows leveraging Apache Airflow, SnapLogic, and Informatica PowerCenter, ensuring reliability, reusability, and end-to-end data integration across heterogeneous systems; Architecting and operating cloud-native big-data ecosystems on AWS, integrating services including AWS S3 for storage, AWS EC2 for compute, AWS Kinesis for real-time data streaming, and AWS Lambda for event-driven processing to enable low-latency ingestion and analytics pipelines; Designing and implementing data quality and governance frameworks within the Medallion Architecture (Bronze, Silver, and Gold layers) to ensure data reliability, lineage, and compliance with enterprise governance standards; Implementing Delta Lake optimization strategies, including Z-Ordering, OPTIMIZE, and VACUUM, to improve query performance, data compaction, and storage efficiency, while designing partition pruning and columnar data formats (Parquet) to minimize I/O overhead and accelerate analytical workloads; and Architecting and optimizing dimensional and relational data models using Star and Snowflake schemas within Databricks and Amazon Redshift, ensuring scalable performance, schema consistency, and efficient analytical processing across large datasets.
HOURS: Full-Time; Mon-Fri (40 hrs/week)
SALARY: $175,698 - $175,698 per year
Skills Required
- Bachelor's degree or foreign equivalent in Computer Science, Computer Engineering, or a related field
- Five years of experience in the position offered, as a Software Engineer, or in a related position
- Five years of experience developing and optimizing data pipelines in PySpark and Databricks
- Five years of experience integrating GitLab version control for code management and automated deployments
- Five years of experience with advanced SQL development, data modeling, query performance tuning, and relational warehouse management in Amazon Redshift and SQL DW
- Five years of experience engineering and automating ETL/ELT workflows using Apache Airflow, SnapLogic, and Informatica PowerCenter
- Five years of experience architecting and operating AWS cloud-native big-data ecosystems using S3, EC2, Kinesis, and Lambda
- Five years of experience designing data quality and governance frameworks using Medallion Architecture
- Five years of experience implementing Delta Lake optimization strategies, partition pruning, and Parquet columnar formats
- Five years of experience architecting dimensional and relational data models using Star and Snowflake schemas in Databricks and Amazon Redshift
- Must be willing to work anywhere in the United States and relocate to various client site locations
What We Do
Agilisium is the #1 Data Innovation Lab Solutions & Services Company for Cloud Customers with a vertical focus on LifeScience & Pharma Industry. We focus on providing the Top 1% of Data Professionals ( Data Engineers, Data Analysts, BI, Data Scientists, MDM experts and so one..). Agilisium provides its customers with Top Quality Cloud and Data Professionals and we are working on building a 5000 Strong & Certified, Global Workforce We possesses Global Delivery Capabilities ( USA, Canada, Mexico, Argentina, Costa Rica, Columbia, UK, Netherlands, Bulgaria, Portugal, India, Nepal, Sri Lanka ) We primarily assist our clients and partners in two ways. #1 Assist them with Talent ( Providing them with Managed and Certified Speciality Talent across the geography) - We have our AWS Cloud Certified staff working from the following locations around the world, North America (US, Canada, Mexico), Central America ( CostaRica ), South America ( Argentina ), Asia ( India, Nepal, Sri Lanka ), EU ( UK, Netherlands, Portugal, Bulgaria, Poland ) #2 Assist with Consulting, Design, Development, and Implementation of AWS Cloud Data Projects
.jpg)
