• Day to-day responsibility includes migrating the data from legacy locations to cloud and perform transformation per data model changes between the databases.• Collaborated with cross functional teams to design, develop, test, implement and support technical solutions utilizing the programming languages like Java, Scala, Python and Open RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake.• Developed ETL pipelines in and out of data… Show more • Day to-day responsibility includes migrating the data from legacy locations to cloud and perform transformation per data model changes between the databases.• Collaborated with cross functional teams to design, develop, test, implement and support technical solutions utilizing the programming languages like Java, Scala, Python and Open RDBMS and NoSQL databases and Cloud based data warehousing services such as Redshift and Snowflake.• Developed ETL pipelines in and out of data warehouse using combination of python and Snowflake Snow SQL writing SQL queries against Snowflake.• Developed migration tool using python with selenium framework to migrate large level enterprise datasets. Later developed the migration app by integrating the APIs, AWS Lambda functions and leveraged Step functions for end-to-end automation. • Collaborated with cross-functional teams to define LDAP directory access controls and permissions.• Wrote AWS Lambda functions in python for AWS’s Lambda which invokes python scripts to perform various transformations and analytics on large datasets in EMR clusters. • Created reports with complex calculations using Tableau and Quick Sight dashboards to perform data analysis and data validation.Configured AWS SNS for event-driven architectures, enabling real-time communication and notifications.Environment: Python, Java, Scala, Spark, Shell scripting, LDAP, Kafka, SQL, NoSQL, Snowflake, Okta, AWS EMR, EC2, AWS S3, AWS Redshift, Redshift Spectrum, RDS, Lambda, AWS Glue. Show less

• Day to-day responsibility includes developing ETL Pipelines in and out of data warehouse, develop major regulatory and financial reports using advanced SQL queries in snowflake.• Implement One time Data Migration of Multistate level data from SQL server to Snowflake by using Python and Snow SQL.• Designed and implemented LDAP schema extensions to accommodate specific organizational requirements within OpenLDAP.• Involved in creating master UNIX shell scripts to call DataStage… Show more • Day to-day responsibility includes developing ETL Pipelines in and out of data warehouse, develop major regulatory and financial reports using advanced SQL queries in snowflake.• Implement One time Data Migration of Multistate level data from SQL server to Snowflake by using Python and Snow SQL.• Designed and implemented LDAP schema extensions to accommodate specific organizational requirements within OpenLDAP.• Involved in creating master UNIX shell scripts to call DataStage jobs, move files around using FTP and various Unix commands, validate the record counts in the incoming external files.• Assist service developers in locating relevant material in reference models that already exist. • Hive tables were created on HDFS to store the data processed by Apache Spark on the Cloudera Hadoop Cluster in Parquet format. • Led the implementation of data integration projects with a Data as a Service mindset, encompassing data profiling, ingestion, transformation, and quality assurance.• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.• Created DAGs to run the Airflow and designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines.• Led the development of complex data transformations using PySpark within Databricks notebooks, • Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. Show less

• Worked on implementing scalable infrastructure and platform for large amounts of data ingestion, aggregation, integration, analytics in Hadoop using Spark and Hive.• Worked on developing streamlined workflows using high-performance API services dealing with large amounts of structured and unstructured data.• Developed Spark jobs in Python to perform data transformation, creating Data Frames and Spark SQL.• Worked on processing un-structured data in JSON format to structured data… Show more • Worked on implementing scalable infrastructure and platform for large amounts of data ingestion, aggregation, integration, analytics in Hadoop using Spark and Hive.• Worked on developing streamlined workflows using high-performance API services dealing with large amounts of structured and unstructured data.• Developed Spark jobs in Python to perform data transformation, creating Data Frames and Spark SQL.• Worked on processing un-structured data in JSON format to structured data in parquet format by performing several transformations using Pyspark.• Built Scalding jobs to migrate the revenue data from Big Query to Manhattan and HDFS. Used cloud replicator to run the BQMH jobs on a GCP Hadoop cluster and replicate the data on-prem HDFS.• Developed Spark applications using spark libraries to perform ETL transformations and thereby eliminating the need for utilizing ETL tools.• Developed the end-to-end data pipeline in spark using python to ingest, transform and analyses data.• Created Hive tables using HiveQL, then loaded the data into Hive tables and analysed the data by developing Hive queries.• Used AWS EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.• Developed custom functions and UDFs in python to incorporate methods and functionality of Spark.• Developed data validation scripts in Hive and Spark and perform validation using Jupiter Notebook by spinning up the query cluster in AWS EMR.• Executed Hadoop and Spark jobs on AWS EMR using data stored in Amazon S3.• Implemented Spark RDD transformations to map business analysis and apply actions on top of transformations.Environment: GCP, Cloud SQL, Big Query, Cloud Data Proc, GCS, Cloud SQL, Cloud Composer, Informatica Power Centre, Talend for Big Data, Airflow, Hadoop, Hive, Teradata, SAS, Teradata, Spark, Python, Java, SQL Server. Show less

• Imported required modules like NumPy on Spark session, created directories for data and output. • Read train and test data into the data directory as well as into Spark variables for easy access and proceeded to train the data based on a sample submission. • Developed and implemented R and Shiny application which showcases machine learning for business forecasting. Developed predictive models using Python & R to predict customers churn and classification of customers. • Rapid… Show more • Imported required modules like NumPy on Spark session, created directories for data and output. • Read train and test data into the data directory as well as into Spark variables for easy access and proceeded to train the data based on a sample submission. • Developed and implemented R and Shiny application which showcases machine learning for business forecasting. Developed predictive models using Python & R to predict customers churn and classification of customers. • Rapid model creation in Python using pandas, NumPy and plot.ly for data visualization. These models are then implemented in SAS where they are interfaced with MSSQL databases and scheduled to update on a timely basis. • Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling • The images upon being displayed are represented as NumPy arrays, for easier data manipulation all the images are stored as NumPy arrays. • Created a validation set in order to test whether the trained model was working as intended or not. • Defined multiple helper functions that are used while running the neural network in session. Also defined placeholders and number of neurons in each layer. • Created neural networks computational graph after defining weights and biases. • After executing the program and achieving acceptable validation accuracy a submission was created that is stored in the submission directory. • Executed multiple Spark SQL queries after forming the Database to gather specific data corresponding to an image.Environment: Scala, Python, Spark, Spark ML Lib, Spark SQL, NumPy, Power BI. Show less