Overview
Hadoop Administrator
Client: a2c/Morgan Stanley
Location: Alpharetta, GA (Onsite)
Description: Hybrid 3 days a week onsite
Databricks Operations is part of the Hadoop Support team. This team is responsible for managing the Databricks plant across Morgan Stanley. Working within an Agile delivery (Scrum/Kanban/Scrumban)/ DevOps methodology. Support the application development teams and help them in debugging and fixing an issue.
Responsibilities and activities of these team include, but not limited to
• Build, Upgrade and maintain Hadoop clusters with several nodes.
• Monitoring and troubleshooting of services running on Hadoop clusters.
• Performance tuning.
• Setup of backup and recovery.
• Troubleshooting User issues which include User on boarding, job failures.
• Installing and integrating of a new service onto the Hadoop cluster.
• Working with Vendor to discuss/apply issue bugs, patches and issues.
• Deploy and automate the implementations/fixes using Ansible scripts.
• Hands-on experience on the administration side of DataBricks preferably on Azure.
• Prior experience in a support role on admin side of Databricks on one of Azure/AWS/GCP cloud
• Excellent knowledge of git and Jenkins. Understanding of distributed systems and databases, cloud computing environments - Azure/Linux (mandatory skills)
• General familiarity with Docker and Kubernetes concepts
• Hands-on experience in Azure stack (Azure Data Lake, Azure Data Factory, Azure Databricks)
• Good understanding of other Azure services like Azure Data Lake Analytics & U-SQL, Azure SQL DW
• Demonstrated analytical and problem-solving skills, particularly those that apply to a big data environment.
• Working within an Agile delivery (Scrum/Kanban/Scrumban)/ DevOps methodology.
• Deploy Azure Databricks workspaces using IaC (terraform + azure DevOps)
Qualification:
• Experience in Cloudera Hadoop distribution CDH 6.x and CDP 7 is preferred.
• Experience with Cluster maintenance tasks like adding and removing nodes, enabling High availability, installing services, applying patches.
• Unix/Linux knowledge including the ability to understand hardware, Operating system and network settings.
• Experience with Hadoop Ecosystem components which include HDFS, YARN, Hive, Impala, Spark, Sqoop, Kafka, Flume and Solr.
• Unix Shell, perl or python scripting.
• Kerberos and LDAP integration experience.
• TLS/SSL certificates knowledge to enable encryption cross Hadoop services.
• Some development experience in Databricks on Azure/AWS/GCP
• Some developer skills in Python
• Some experience with Terraform for IaaC
Vijay Bhaskaran
(925)233-3001
vijay@flexontechnologies.com
www.linkedin.com/in/vijay-bhaskaran
www.flexontechnologies.com
https://www.flexontechnologies.com/copy-of-flex-ehr-3
Flexon Technologies is a leading end-to-end technology solutions provider to IT and Non-IT industry. We specialize in providing "Total Solutions" encompassing technology and services combined with unparalleled domain knowledge that gives our clients a distinct advantage.
We offer integrated business solutions, enabling clients to optimize their business with greater efficiency, and increased responsiveness. Our offerings are designed to cater to the entire range of clients' technology needs. We deliver end-to-end solutions that can build, manage and support our customers' IT systems across the entire value chain infrastructure, applications and business processes. The range of our offerings extends to software (including systems and application software development, implementation, maintenance and frameworks), IT architecture, network consulting, Staffing etc. These technology offerings backed by the domain solutions and knowledge to ensure maximum business alignment, allowing you to derive maximum benefits out of the IT investments
