Caleb D. Email & Phone Number

New York, New York, US

• Translating business requirements by business team into Data Engineering specifications and pipelines for international Data Analytics and Data Science for a multitude of pharmaceutical solutions
• Building scalable data sets with Python and SQL on Snowflake Data Warehouse based on specifications from raw data and derive business metrics
• Understanding and Identify server APIs needed to be instrumented for data analytics aligning events for established data pipelines
• Engineering Data pipelines for Analytics on Snowflake for Rare Disease Cardiovascular products live Dashboard displaying in Tableau with reduced load times reduced to seconds from > 30 mins
• Developing deployment processes for Snowpark and SQL script process with Airflow
• Exploring and understanding sophisticated data sets, identifying and formulating correlational rules between heterogenous sources for effective analytics and reporting

New York, New York, US

• Working with a novel commercial analytics team to develop a process for data warehouse and data lake for support of reporting, analytics, and machine learning
• Translating business requirements by business team into data and engineering specifications
• Building scalable data sets based on specifications from the available raw data and deriving business metrics/insights
• Cooperating across worldwide Pizer teams to produce processes and tools for commercial analytics teams by learning about Snowpark and Snowflake deployment and CI/CD methods
• Structuring DevOps process for Snowflake SQL and Snowpark applications and processes with Git, GitHub, Github workflows and actions
• Configuring new repository for CI/CD process and code development process for version control

Charlotte, North Carolina, US

• Migrated on premises Hadoop Data Lake and energy grid IOT device data sources to downstream cloud data lake and data warehousing for the use of machine learning and analytics teams SageMaker development
• Used AWS Glue Data Catalog as a metadata store for EMR and managed metadata tables
• Deployed AWS infrastructure using various Bitbucket repositories and terraform workspaces, promoting code via pull requests
• Collaborated with security team to create compliant quality, development and production environments using KMS for encrypting data and IAM roles for AWS permissions
• Utilized on premises Bash scripts and hdfs distcp copy commands to transfer data from Hadoop to AWS S3 and use Kafka for change data capture
• Created Boto3 Utilization tool to authenticate python jobs that interacted with AWS API and check for valid http success or failure response codes

New York, NY, US

• Provisioned and managed AWS and Azure cloud data lakes for first responders and global entities
• Developed python and SQL scripts to manage data lake access for new and old users
• Managed Terraform scripts to whitelist and blacklist new users IP addresses to data lakes
• Repaired and optimized Qlik analytics visualization backend SQL queries by ~300% with indexing
• Architected complex, highly available, optimized batch and real-time data pipelines
• Worked with analytics product engineers to ensure performance, stability, availability of our MS SQL Server analytics databases

Indianapolis, Indiana, US

• Packaged, refactored and managed Machine learning models for business IT operations
• Developed terraform IAC for AWS EMR pyspark and Scala jobs
• Facilitated responsibility for Hadoop development Implementation including loading from disparate data sets, preprocessing using Hive and Pig.
• Optimized ML spark batch jobs by ~5 hours using feature extraction and filtering
• Operationalized code for machine learning model automation tool and pyspark jobs
• Performed runs for extracts for data scientist experiments

Livonia, MI, US

• Architected and built a new AWS Organizations Cloud environment with PCI, PHI, PII compliance
• Worked on creating new data lake ingesting data from on-prem and other clouds to s3 and redshift, and RDS
• Used Terraform Enterprise and GitLab to deploy IAC to various AWS accounts
• Integrated big data spark jobs with EMR and glue to create ETL jobs for around 450 GB of data daily
• Optimized EMR clusters with partitioning and parquet format to increase speeds and efficiently by 200-500%
• Created new Redshift cluster for data science using Quicksight for reporting and mobile visualization

Menlo Park, California, US

• Configured Spark streaming to receive real time data from Kafka and store to HDFS
• Use Spark streaming with Kafka and MongoDB to build continuous ETL pipeline for real time analytics
• Managed ETL jobs with UDFs in pig scripts with spark before for transformations, joins, aggregations before HDFS
• Preformed performance tuning for Spark Streaming setting right batch internal time, correct level of parallelism, selection of correct Serialization and memory tuning
• Data ingestion using Flume with source as Kafka Source and sink as HDFS
• Used Spark SQL and Data Frames API to load structured and semi structured data into Spark Clusters

Mclean, VA, US

• Created highly scalable, resilient, and performant architecture using amazon AWS cloud technologies such as simple storage service, S3, Elastic Map Reduce (EMR), Elastic Cloud Compute (EC2), Elastic Container Service.
• Deployed containerized applications using Docker, allowing for standardized service infrastructure.
• Monitored production software with logging, visualizing, and incident management software such as slunk, Kibana
• Took advantage of new Spark Avro functionality through upgrading
• Provided live demonstrations of software systems to nontechnical, executive level personnel, showing how the systems were meeting business goals and objectives.
• Spark clusters exclusively from the AWS Management Console.

Hangzhou, CN

• Building scalable distributed data solutions using Hadoop.
• Installed and configured Pig for ETL jobs and make sure Pig scripts with regular expression for data cleaning
• Used Zookeeper and Oozie for coordinating the cluster and scheduling workflows
• Used Oozie Scheduler system to automate the pipeline workflow and orchestrate the jobs that extract the data in a timely manner
• Move data from Oracle to HDFS and vice versa using Sqoop
• Imported data using Sqoop and load data from MySQL and oracle to HDFS on regular basis

Savannah, GA, US

• Deployed the application jar files into AWS instances.
• Used the image files of an instance to create instances containing Hadoop installed and running
• Developed a task execution framework on EC2 instances using SQL and DynamoDB
• Designed a cost-effective archival platform for storing big data between then using Sqoop and various ETL tools
• Extracted the data from RDBMS (Oracle, MySQL) to HDFS using Sqoop
• Used hive with spark streaming for real-time processing