Santosh K Email & Phone Number

St Louis, Missouri, United States

• Collaborated extensively with Google Cloud Platform (GCP) services, including BigQuery, Google Cloud Storage (GCS) buckets, Google Cloud Functions, Cloud Dataflow, Data Proc, and Stackdriver. • Constructed Power BI reports on Azure Analysis Services, optimizing performance for enhanced data analysis capabilities. • Leveraged Cloud Shell SDK within Google Cloud Platform (GCP) to configure services such as Data Proc, Storage, and BigQuery. • Developed Power BI reports using Azure Analysis Services to enhance data analysis capabilities and optimize performance. • Engineered intricate mappings for loading data from diverse sources into the Data Warehouse, incorporating various transformations and stages such as Joiner, Transformer, Aggregator, Update Strategy, Rank, Lookup, Filter, Sorter, Source Qualifier, and Stored Procedure transformation. • Engineered streaming applications with PySpark to ingest data from Kafka and store it in NoSQL databases like HBase and Cassandra. • Monitored YARN applications, efficiently resolving cluster-related system issues. • Devised shell scripts to parameterize Hive actions within Oozie workflows and schedule jobs. • Played a pivotal role in a team tasked with developing an initial prototype of a NiFi big data pipeline, showcasing a comprehensive end-to-end scenario of data ingestion and processing. • Utilized Apache NiFi to handle high-volume data streams, enabling ingestion, processing, and low-latency provisioning using Hadoop ecosystems like Hive, Pig, Scoop, Python, Spark, Scala, and Druid. • Developed secure data streaming solutions utilizing Apache NiFi, Apache Pulsar, and Apache Kafka to deliver highly sensitive data with low latency to multiple teams while upholding confidentiality.

Hyderabad, Telangana, India

• Employed a range of tools and services within the Google Cloud Platform, such as BigQuery, Cloud Storage, Pub/Sub, Composer, Cloud Balancing, Cloud SQL, Datapost, and Stack Drive. • Engaged in the complete data engineering life cycle, encompassing analysis, solution design, data pipeline engineering, testing, deployment, scheduling, and implementation. • Crafted Python scripts within Airflow to execute ETL operations on extracted data, leveraging Apache Spark for processing before transferring it to Mongo DB. • Configured HDFS to effectively store streamed data from Spark, enabling continuous data ingestion from Kafka. Engineered a data pipeline to capture crucial data points from customer chatbot service interactions. • Facilitated the migration of on-premise ETL processes to the Google Cloud Platform (GCP) using cloud-native technologies like BigQuery, Cloud Dataproc, Google Cloud Storage, and Composer to empower the Data Science team in training their models and enhancing Chatbot interactivity. • Orchestrated AWS EC2 instances and initiated Spark jobs on AWS Elastic Map Reduce (EMR). • Implemented automated testing scripts for the Staging and Master branches before transitioning the pipeline to the CICD pipeline. • Utilized Pyspark to operate on Glue as an ETL service. • Designed data mart and warehousing architectures employing distributed SQL principles, Presto SQL, Hive SQL, Python libraries (Pandas, NumPy, SciPy, Matplotlib), and PySpark to manage increasing data volumes. • Developed data pipelines to ingest data from the Enterprise Data Lake (utilizing MapReduce, Hadoop distribution - Hive tables/HDFS) for analytics solutions, extracting data from transactional and operational databases and loading it into target databases/data warehouses in batch and real-time.

Hyderabad, Telangana, India

• Designed and implemented an Enterprise Data Lake to accommodate diverse use cases, including analytics, processing, storage, and reporting of large and rapidly evolving data sets. • Established a robust security framework using AWS Lambda and DynamoDB to enable fine-grained access control for objects within AWS S3. • Configured and deployed Kerberos authentication principals to ensure secure network communication within the cluster, extensively testing HDFS, Hive, Pig, and MapReduce functionalities for new users. • Conducted comprehensive architecture and implementation evaluations of AWS services such as Amazon EMR, Redshift, and S3. • Developed machine learning algorithms in Python to make predictions by leveraging Kinesis Firehose and the S3 data lake. • Utilized AWS EMR to efficiently transform and migrate large volumes of data across various AWS data stores and databases, including Amazon S3 and Amazon DynamoDB. • Extracted, cleansed, and transformed data from diverse sources such as flat files, sequential files, CSV files, XML, and databases like Oracle and DB2. • Implemented AWS Step Functions to automate and orchestrate tasks related to Amazon SageMaker, including data publishing to S3, model training, and deployment for prediction. • Integrated Apache Airflow with AWS to monitor multi-stage machine learning workflows with tasks running on Amazon SageMaker. • Employed Spark SQL with Scala and Python interfaces to seamlessly convert RDD case classes to schema RDD. • Imported data from various sources, including HDFS and HBase, into Spark RDD and conducted computations using PySpark to generate desired output responses.• Developed Lambda functions with Boto3 to automate the de-registration of unused Amazon Machine Images (AMIs) across all application regions, effectively reducing EC2 resource costs. • Imported and exported databases using SQL Server Integration Services (SSIS) and Data Transformation Services (DTS Packages).

Hyderabad, Telangana, India

• Collaborated with Azure Data Factory to seamlessly integrate data from diverse sources, including on-premises databases like MySQL and Cassandra, as well as cloud storage solutions such as Blob storage and Azure SQL DB. Applied necessary transformations and efficiently loaded the processed data back into Azure Synapse. • Managed and fine-tuned resources across the cluster using Azure Kubernetes Service, while monitoring Spark cluster performance through Log Analytics and Ambari Web UI. • Enhanced query performance by transitioning log storage from Cassandra to Azure SQL Data Warehouse. • Engineered robust data pipelines utilizing Apache Flink for both stream processing and batch processing of extensive datasets. These pipelines delivered high throughput and low latency processing capabilities. • Developed data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL, while also interfacing with Cosmos DB using SQL API and Mongo API. • Leveraged Azure Logic Apps to orchestrate workflows for scheduling and automating batch jobs, integrating various services including ADF pipelines, HTTP requests, and email triggers. • Proficiently utilized Azure Data Factory, encompassing data transformations, Integration Runtimes, Azure Key Vaults, triggers, and the migration of data factory pipelines to higher environments using ARM Templates. • Created pipelines to orchestrate the loading of data from Azure Data Lake into Staging SQLDB and subsequently into Azure SQL DB. • Orchestrated the migration of large datasets to Databricks (Spark), encompassing cluster administration, data loading, and configuration of data pipelines from ADLS Gen2 to Databricks using ADF pipelines. • Developed Databricks notebooks to streamline data curation for various business use cases, including the mounting of blob storage on Databricks.

Hyderabad, Telangana, India

• Mastered the creation of Tableau dashboards for comprehensive reporting on analyzed data, ensuring effective visualization of insights. • Proficient in working with NoSQL databases, particularly HBase, for efficient data management and retrieval. Implemented a staged process for input records files, ensuring thorough cleaning and validation of data before loading it into the data warehouse. • Automated the extraction of numerous flat/excel files from diverse sources, including FTP and SFTP (Secure FTP), streamlining data acquisition processes. • Utilized Jenkins for continuous integration, seamlessly integrating code changes, while leveraging GitHub as a centralized repository for version control and collaborative development. • Executed various dataflow and control flow tasks, including loop and sequence containers, script tasks, SQL task execution, and package configuration, to streamline data processing workflows. • Developed SSIS packages to facilitate the export of data from SQL Server to Excel spreadsheets and vice versa, ensuring smooth data interchange between platforms. • Designed and implemented SSIS packages to handle the decryption, transformation, and movement of files to a data warehouse, with robust error handling and alerting mechanisms in place. These files were sourced from remote locations using FTP and SFTP protocols.