The Role
Lead Engineer responsible for administering and tuning Hadoop/NoSQL ecosystems (HDFS, Hive, YARN, Spark, Impala, HBase), managing Kafka/NiFi, automating operations, implementing security (Kerberos, Sentry), supporting data lakes, cloud/on-prem integrations, and driving capacity, backup/disaster recovery and vendor evaluations.
Summary Generated by Built In
Job Description
Job Description:
Scope of Work
POSITIONAs a Lead Engineer you will have the opportunity to engineer and administer TIAA’s big data environment. Your role will be responsible for administering our Hadoop and No-SQL ecosystem components such as HDFS, Hive, MR, Yarn, Impala, Spark, Sqoop, HBase, Sentry, Hue and Oozie. Your role will design and implement automated processes, research database technologies, communicate effectively with database administrators and application stake holders to ensure your internal clients’ needs are met.
RESPONSIBILITIES
QUALIFICATIONS
Required Experience
Additional Information
Hello,Hope you are doing well.
Job Title:Big data engineer
Duration: 6 Months
LOCATION: 1st Preference Iselin or Charlotte ONSITE And New York, NJ
Only GC/Citizens
Job Description:
Scope of Work
POSITIONAs a Lead Engineer you will have the opportunity to engineer and administer TIAA’s big data environment. Your role will be responsible for administering our Hadoop and No-SQL ecosystem components such as HDFS, Hive, MR, Yarn, Impala, Spark, Sqoop, HBase, Sentry, Hue and Oozie. Your role will design and implement automated processes, research database technologies, communicate effectively with database administrators and application stake holders to ensure your internal clients’ needs are met.
RESPONSIBILITIES
- Responsible for the implementation and on-going administration of Hadoop infrastructure including the installation, configuration and upgrading of Cloudera distribution of Hadoop
- File system, cluster monitoring, and performance tuning of Hadoop ecosystem
- Resolve issues involving map reduce, yarn, sqoop job failures; Analyze multi-tenancy job execution issues and resolve
- Design and manage backup and disaster recovery solution for Hadoop clusters
- Work on Unix operating systems to efficiently handle system administration tasks related to Hadoop clusters
- Manage the Apache Kafka and Apache NIFI environments
- Participate and manage the data lakes data movements involving Hadoop, NO-SQL databases like HBase, Cassandra and Mongodb
- Work with data delivery teams to setup new Hadoop users. Includes setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and Map Reduce access for the new users. Configure Hadoop security aspects including Kerberos setup and RBAC authorization using Apache Sentry
- Create and document best practices for Hadoop and big data environment
- Participate in new data product or new technology evaluations; manage the certification process and evaluate and implement new initiatives in technology and process improvements
- Interact with Security Engineering to design solutions, tools, testing and validation for controls
- Evaluate the database administration and operational practices, and evolve automation procedures (Using scripting languages such as Shell, Python, Chef, Puppet, CFEngine, Ruby etc.)
- Advance the cloud architecture for data stores; Work with TIAA Cloud engineering team with automation; Help operationalize Cloud usage for databases and for the Hadoop platform
- Engage vendors for feasibility of new tools, concepts and features, understand their pros and cons and prepare the team for rollout
- Analyze vendor suggestions/recommendations for applicability to TIAA’s environment and design implementation details
- Perform short and long term system/database planning and analysis as well as capacity planning
- Integrate/collaborate with application development and support teams on various IT projects
QUALIFICATIONS
Required Experience
- Bachelor’s degree; Preferably in Computer Science or Information Systems
- Ten or more years of overall IT/DBMS/Data Store experience
- Three or more years of experience in, big data, data caching, data federation and data virtualization management including experience in leveraging Hadoop
- Two or more years of expertise and in-depth knowledge of SAN, system administration, VmWare, backups, restores, data partitioning, database clustering and performance management
- Experience writing shell scripts, and automating tasks. Exposure to Chef or/and Puppet is preferred
- Experience in the implementation details of Hadoop Clusters, Impala, and HBase and other emerging data techniques
- Experience with monitoring technologies for databases
- Experience with orchestration techniques, infrastructure automation and cloud deployments
- Understating of Linux, Windows, Dockers / containers
- Familiarity with “IaaS” and “DBaaS” Service oriented concepts preferred
- Familiarity of Cloud Architecture (Public and Private clouds) – AWS , AZURE preferred
- Working knowledge of VMware and VMware vCloud Automation Center (vCAC) preferred
- Proficiency in using Microsoft Office (Word, Excel, PowerPoint) to document, present, communicate and articulate idea/s and concepts
- Strong communication skills and the ability to collaborate and work in teams with other engineers, working in a fast paced and ever changing technical environment
- Application development experience – database programming, scripting, setting up web sites and dashboards
All your information will be kept confidential according to EEO guidelines.
Skills Required
- Bachelor's degree (preferably in Computer Science or Information Systems)
- U.S. Green Card or U.S. Citizen
- Ten or more years of overall IT/DBMS/Data Store experience
- Three or more years of big data, data caching, data federation and data virtualization management leveraging Hadoop
- Two or more years experience with SAN, system administration, VMware, backups/restores, clustering and performance management
- Experience administering Hadoop ecosystem components (Cloudera Hadoop, HDFS, Hive, MapReduce, YARN, Impala, Spark, Sqoop, Oozie, Hue, Sentry)
- Experience administering NoSQL databases and data lakes (HBase, Cassandra, MongoDB) and data movement tools
- Experience administering Apache Kafka and Apache NiFi
- Experience with Hadoop security: Kerberos setup, RBAC authorization (Apache Sentry), and onboarding users
- Experience writing shell scripts and automating tasks
- Experience implementing Hadoop clusters, Impala, and HBase
- Experience with monitoring technologies for databases and performance tuning of Hadoop clusters
- Experience with orchestration, infrastructure automation and cloud deployments
- Understanding of Linux, Windows, and Docker/containers
- Exposure to Chef and/or Puppet (configuration management)
- Familiarity with IaaS and DBaaS concepts
- Familiarity with cloud architecture (AWS, Azure)
- Working knowledge of VMware and VMware vCloud Automation Center (vCAC)
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Randstad is a global leader in the HR services industry and a Dutch multinational human resource consulting firm. Founded in 1960 and headquartered in Diemen, Netherlands, it provides outsourcing, staffing, consulting, and workforce solutions. The company connects job seekers with employers across various sectors, including finance, technology, healthcare, and manufacturing, helping people secure rewarding jobs and stay relevant in the ever-changing world of work.
