Utilizing Spark, Scala, and Python in Hadoop/Hive environments for large-scale data processing and regular expression projects.Migrating code bases to Amazon EMR and evaluating components of the Amazon ecosystem, including Redshift.Importing metadata into Hive using Scala and migrating existing tables and applications to operate effectively on Hive and the Azure cloud.Designing and implementing pipelines in Azure Data Factory (ADF) to extract, transform, and load data from diverse sources.Developing Python-based RESTful web services/APIs for revenue tracking and analysis.Executing data migrations from traditional database systems to Azure databases and designing migration strategies for traditional systems on Azure.Designing, developing, and testing dimensional data models using Star and Snowflake schema methodologies.Creating Tableau reports with complex calculations and conducting ad-hoc reporting using PowerBI.Providing day-to-day GIT support for various projects, overseeing the design and maintenance of GIT repositories and access control strategies.I am proficient in Python, Spark, Scala, Redshift, Azure, BigQuery, Hadoop, Hive, SQL, and various other tools and technologies. Let's connect to discuss how I can contribute to your data-driven initiatives.

With extensive experience in data engineering and analytics, I've successfully executed a variety of projects leveraging cloud technologies and big data frameworks. I specialize in designing and implementing end-to-end data solutions on both AWS and Azure platforms, including migration projects, ETL workflows, and data visualization.Writing scripts and indexing strategies for database migrations to AWS RedshiftUtilizing AWS Glue catalog and Data Pipeline for data ingestion and transformationDeveloping statistical models using artificial neural networks for predictive analyticsImplementing end-to-end data solutions on Azure using Databricks, ADF, and Power BICreating PySpark code for AWS Glue jobs and EMR processingHands-on experience with Python data engineering packages like Pandas, Numpy, and Scikit-LearnWriting SAM templates for serverless applications on AWSDesigning and deploying ETL workflows using Azure Data Factory and SSISCreating analytical dashboards for tracking student records and GPAsUtilizing deep learning frameworks like TensorFlow and Keras for model developmentI thrive in dynamic environments and am proficient in Hadoop, AWS services, Spark, SQL, Python, and various other tools and technologies.

Designing and deploying Hadoop clusters and implementing different big data analytic tools such as Pig, Hive, HBase, Oozie, Sqoop, Kafka, and Spark.Installing and configuring Hadoop ecosystem components like Hive, Pig, Sqoop, Flume, and Oozie on the cluster.Working with Google Cloud Platform (GCP) services like Compute Engine, Cloud Functions, Cloud DNS, Cloud Storage, and Cloud Deployment Manager.Developing and implementing data solutions using Hadoop and Spark technologies, including MapReduce, Hive, and Spark RDD.Creating high- and low-level design documents for various modules, ensuring adherence to standards and corporate guidelines.Designing and implementing data lake architectures within Oracle ERP to provide a centralized repository for diverse data types.Consuming data from Kafka using Apache Spark and developing Spark scripts using Python Shell commands.Optimizing MapReduce jobs to efficiently use HDFS by implementing various compression mechanisms.Performing data analysis using Hive queries and Pig scripts to study employee behavior.Utilizing Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.I am proficient in Informatica PowerCenter, Teradata, Oracle, Python, Business Objects, Tableau, and Unix/Linux environments. My comprehensive expertise in big data technologies and cloud platforms makes me well-equipped to drive data-driven insights and solutions.

Proficient in architecting and building multiple data pipelines, end-to-end ETL, and ELT processes for data ingestion and transformation in Azure.Strong understanding of AWS components such as EC2 and S3, with experience implementing a Continuous Delivery pipeline using Docker and GitHub.Utilized Google Cloud functions with Python to load data into BigQuery from CSV files in GCS buckets and processed and loaded bound and unbound data from Google Pub/Sub topics to BigQuery using Cloud Dataflow with Python.Developed and deployed data pipelines in cloud environments such as AWS and Azure, performing data engineering functions including extraction, transformation, loading, and integration to support enterprise data infrastructures.Designed several DAGs (Directed Acyclic Graph) to automate ETL pipelines and architect the ETL transformation layers, writing Spark jobs for data processing.Proficient in fact dimensional modeling (Star schema, Snowflake schema), transactional modeling, and SCD (Slowly Changing Dimension), with experience in devising PL/SQL Stored Procedures, Functions, Triggers, Views, and packages.Developed logistic regression models in Python to predict subscription response rates based on customer variables, leveraging past transactions, response to prior mailings, promotions, demographics, interests, and hobbies.Hands-on experience with Azure, BigQuery, GCS bucket, Google Cloud functions, Cloud Dataflow, Pub/Sub, Cloud Shell, GSUTIL, BQ command line utilities, Data Proc, and Stackdriver.Environment: Azure, AWS, BigQuery, GCS Bucket, Google Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, GSUTIL, Docker, Kubernetes, Apache Airflow, Python, Pandas, Matplotlib, Seaborn, text mining, NumPy, Scikit-learn, ETL workflows, linear regression, multivariate regression, Scala, Spark.

Created a Microsoft Access database, defining tables and entering datasets manually, while performing ER Diagram creation and executing basic SQL queries.Documented the entire process flow encompassing program development, logic, testing, implementation, and application integration.Utilized Apache Spark and Scala APIs to develop predictive analytics solutions.Worked with SQL Server Integration Services (SSIS) to integrate and analyze data from diverse heterogeneous information sources.Employed SQL and SQL Server for writing both simple and complex queries, such as identifying distinct and unique values in datasets.Designed complex ETL packages using SSIS for data extraction from staging tables to partitioned tables with incremental loading.Implemented incremental loads using various Dataflow and Control Flow Tasks within SSIS.Developed and optimized SQL queries, stored procedures, triggers, views, cursors, joins, constraints, DDL, DML, and user-defined functions to implement business logic, alongside creating clustered and non-clustered indexes.Imported data using Sqoop to load data from MySQL into HDFS on a regular basis.Delivered SQL scripts for data insertion, updating, and deletion within MS SQL database tables.Generated various ad-hoc SQL queries to fulfill customer requirements and executive management reports, encompassing report types such as tables, matrices, and sub-reports.Collaborated closely with ETL, BI, and DBA teams while working on SQL Server and Teradata, analyzing and providing solutions to data issues and other challenges encountered during OLAP model implementation.