Hadoop/Spark Admins

Critical: Hadoop Admin / Spark Admin / Linux / Windows / SAS & Python

Nice to have: AWS / Tableau

Need a great communicator. This role will be interfacing directly with Data Scientists, etc, and will need to be personable and have the ability to work issues and deal with people. It's a lot of tech, but it's a lot of internal coordinating and communicating.


There are 3 openings. These will be a contract through end of year. They need to be familiar with SAS and able to convert in Spark and Python. SAS to Python typically refers to the process of migrating data analysis, data manipulation, and statistical modeling tasks from SAS (Statistical Analysis System) to Python, a popular open-source programming language.

Locations: Scottsdale and Chicago and New York

Interview process: 30 min with hiring manager , 2nd- 1 hour onsite





Current Need:
The EWS Data Science and Analytics teams as well as the ML Ops team are fully committed, and need to augment our resources with external support to

1. Help convert legacy code-based assets to modern high-performance tools (SAS to Python)
   1. Existing data processing scripts including data movement, cleaning, and aggregation
   2. Value Testing Process. This scores a potential customers data through our models to help determine the value of EWS solutions
   3. SQL/Hive query performance tuning and enhancement

1. Develop shared toolkit to automate certain data science processes
   1. Data profiling
   2. Feature importance and effectiveness evaluation

1. Automate documentation of model development processes
2. Assist in upskilling existing team

Project Specifics

1. Code Modernization for VT, MV&P, DS, DICA and CIR teams on existing programs/processes
   1. from SAS/Hive to Python/Scala/Pyspark/SQL or other modern highly efficient technology that fits the Early Warning's current on-prem environment and set up an easy conversion path for future state in ADP/Model Factory
   2. coordinate with MLOps team to onboard new data sources that exist in SAS environment but not in Newton
   3. For new VTs, work with the relevant parties to ensure Project plans account for MLOps engagement to build the capability (other processes potentially as well key capabilities in general can be requested to be built by MLOps from scratch)
   4. Training team to ensure proper adoption/transition to the team

1. Hive code efficiency evaluation and modernization
   1. Evaluate legacy repeated Hive queries commonly used by the analytics community
   2. Upgrade the legacy code to Scala/Pyspark or other modern highly efficient technology that fits the Early Warning's current on-prem environment and set up an easy conversion path for future state in ADP/Model Factory
   3. Training team to ensure proper adoption/transition to the team

1. Analytics ToolKit / Capability (shared among all teams)
   1. When existing open-source packages not available or not fitting our modeling need, Create standard, re-usable, highly efficient procedures for end-to-end model development, validation and evaluation, for example:
      1. Data profiling tool (evaluate data missing, value ranges, outlier, categorical features etc.)
      2. Feature effectiveness triaging toolset for XGBoost or other non-transparent models
      3. Provide standard generation of outputs of various model stages that aligns with model governance documentation requirements.

1. 1. Provide a template for efficient python-based project structure that enables efficient run, test, debug and deploy pipeline.
   2. Engage with MLOps for design, code review and approval this is within MLOps roles/resp but this SOW will help to bridge the short term resource gap

1. Report Automation
   1. Replace the current SAS/VisualBasic process with automate standard report automation using the modeling outputs. Collaborate with the tech writer and analytics team to standardize template and output. This include both validation report and initial model development report (auto-inserted with template), this may depend on when we have a DR replacement
   2. Engage with MLOps for design, code review and approval this is within MLOps roles/resp but this SOW will help to bridge the short term resource gap

1. Training / Upskilling Analytics Teams
   1. Create training/onboarding materials and provide hands-on practice training environment with target adoption outcome
   2. Work with Corp Learning & Development to develop programming training path using existing platforms and tools (LinkedIn Learning and Udemy)
   3. Provide office hour and troubleshooting support
   4. Conduct regular code guidance for the team in partnership with MLOps

1. Day 1 Monitoring Script
   1. Create Day 1 model monitoring script when MLOps resource are not available