Pradeep P Email & Phone Number

Over 10 years of experience as GCP Data Engineer with demonstrated expertise in building and deploying data pipelines using open-source Hadoop based technologies such as Apache Spark, Hive, Hadoop, Python and PySpark.Hands on Experience in developing Spark applications using PySpark Data Frame, RDD and Spark SQL.Working with GCP cloud using in GCP Cloud storage, DataProc, Data Flow, Big Query, Cloud Composer and Cloud Pub/Sub.Expert in working with cloud PUB/SUB to replicate data real-time from source system to GCP Big Query.Good knowledge on GCP service accounts, billing projects, authorized views, datasets, GCS buckets and gsutil commands.Experienced in building and deploying Spark applications on Hortonworks Data Platform and AWS EMR.Experienced in working with AWS services such as – EMR, S3, EC2, IAM, Lambda, Cloud Formation, Cloud Watch.Worked on structured and semi structured data storage formats such as Parquet, ORC, CSV, JSON.Hands on experience on Google Cloud Platform (GCP in all the big data products Big Query, Cloud Data Proc, Google Cloud Storage, Composer (AirFlow as a service)Hands on experience working in GCP services like Big Query, Cloud Storage (GCS), cloud function, cloud dataflow, Pub/sub, Cloud Shell, GSUTIL, Big Query, Data Proc, Operations Suite (Stack driver).

• Lead Data Engineer as for developing ETL’S using informatica cloud services.• Modified existing dimension data model by adding required dimensions and facts as per business process.• Migrated previously written cron jobs to airflow/composer in GCP.• Built Scalding jobs to migrate the revenue data from BigQuery to Manhattan and HDFS. Used cloud replicator to run the BQMH jobs on a GCP Hadoop cluster and replicate the data on-prem HDFS.• Developed Spark applications using spark libraries to perform ETL transformations and thereby eliminating the need for utilizing ETL tools.• Worked on implementing scalable infrastructure and platform for large amounts of data ingestion, aggregation, integration, analytics in Hadoop using Spark and Hive.• Got involved in migrating on prem Hadoop system to using GCP (Google Cloud Platform).• Worked on developing streamlined workflows using high-performance API services dealing with large amounts of structured and unstructured data.

• Proficient in working with Azure cloud platform (DataBricks, Data Factory, HDInsight, DataLake, Blob Storage, Synapse Analytics, Azure SQL, SQL pool, Azure Serverless apps) • Involved in building an Enterprise DataLake using Data Factory and Blob storage, enabling other teams to work with more complex scenarios and ML solutions. • Designing and Developing Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and Non-relational to meet business functional requirements • Created, provisioned multiple Databricks clusters needed for batch and continuous streaming data processing and installed the required libraries for the clusters. • Have good experience in setting up separate applications and reporting data tiers across servers using Geo replication functionality and failover groups. • Extensively used data bricks notebooks for data processing and interactive analytics using Spark API’s • Extensive knowledge in data transformations, mapping, cleansing, monitoring, debugging, performance tuning and trouble-shooting clusters.

• Prepared complicated T-SQL queries and user-defined functions in SQL Server to meet business needs.• Designed and implemented code changes in existing modules - Java, python, shell-scripts for enhancement.• Developed Spark and Scala pipelines which transform the raw data from several formats to parquet files for consumption by downstream systems.• Used AWS Glue services like crawlers and ETL jobs to catalog all the parquet files and make transformations over data according to the business needs.• Worked with AWS services like S3, Glue, EMR, SNS, SQS, Lambda, EC2, RDS and Athena to process data for the downstream customers.• Created libraries and SDKs which will be helpful in making JDBC connections to hive database and query the data using Play framework and various AWS services.• Developed scripts using Spark which are used to load the data from Hive to Amazon RDS(Aurora) at a faster rate.