Dileep Reddy Email and Phone Number

• Collaborated with Business Analysts, SMEs across departments to gather business requirements, and identify workable items for further development.• Processed some simple statistic analysis of data profiling like cancel rate, var, skew, kurt of trades, and runs of each stock every day group by 1 min, 5 min, and 15 min.• Used PySpark and Pandas to calculate the moving average and RSI score of the stocks and generated them into data warehouse.• Exploring with Spark to improve the… Show more • Collaborated with Business Analysts, SMEs across departments to gather business requirements, and identify workable items for further development.• Processed some simple statistic analysis of data profiling like cancel rate, var, skew, kurt of trades, and runs of each stock every day group by 1 min, 5 min, and 15 min.• Used PySpark and Pandas to calculate the moving average and RSI score of the stocks and generated them into data warehouse.• Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL,postgreSQL,Data Frame,OpenShift, Talend,pair RDD's• Involved in integration of Hadoop cluster with spark engine to perform BATCH and GRAPHX operations. Show less

• Gathering data and business requirements from end users and management. Designed and built data solutions to migrate existing source data in Teradata and DB2 to Big Query (Google Cloud Platform).• Performing data manipulation on extracted data using Python Pandas.• Working with subject matter experts and project team to identify, define, collate, document and communicate the data migration requirements.• Designing Scoop scripts to load data from Teradata and DB2 to the Hadoop… Show more • Gathering data and business requirements from end users and management. Designed and built data solutions to migrate existing source data in Teradata and DB2 to Big Query (Google Cloud Platform).• Performing data manipulation on extracted data using Python Pandas.• Working with subject matter experts and project team to identify, define, collate, document and communicate the data migration requirements.• Designing Scoop scripts to load data from Teradata and DB2 to the Hadoop environment and Shell scripts to transfer data from Hadoop to Google Cloud Storage (GCS) and from GCS to Big Query.• Validating Scoop jobs, Shell scripts, and performing data validation to check if data is loaded correctly without any discrepancy. Performing migration and testing of static data and transaction data from one core system to another. Show less

• Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP• Strong understanding of AWS components such as EC2 and S3• Implemented a Continuous Delivery pipeline with Docker and Git Hub• Worked with g-cloud function with Python to load Data into Bigquery for on arrival csv files in GCS bucket• Process and load bound and unbound Data from Google pub/sub topic to Bigquery using cloud Dataflow with… Show more • Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP• Strong understanding of AWS components such as EC2 and S3• Implemented a Continuous Delivery pipeline with Docker and Git Hub• Worked with g-cloud function with Python to load Data into Bigquery for on arrival csv files in GCS bucket• Process and load bound and unbound Data from Google pub/sub topic to Bigquery using cloud Dataflow with Python.• Hands of experience in GCP, Big Query, GCS bucket, G-cloud function, cloud data flow, Pub/sub cloud shell, GSUTIL, BQ command line utilities, Data Proc, Stack driver• Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines• Hands on experience on architecting the ETL transformation layers and writing spark jobs to do the processing. Show less

• Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creating ETL pipelines using Pig, and Hive.• Built pipelines to move hashed and un-hashed data from XML files to Data lake.• Developed Spark scripts using Python on Azure HDInsight for Data Aggregation, Validation and verified its performance over MR jobs.• Experience with Cloud Service Providers such as Amazon AWS, Microsoft Azure, and Google GCP• Architect & implement… Show more • Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creating ETL pipelines using Pig, and Hive.• Built pipelines to move hashed and un-hashed data from XML files to Data lake.• Developed Spark scripts using Python on Azure HDInsight for Data Aggregation, Validation and verified its performance over MR jobs.• Experience with Cloud Service Providers such as Amazon AWS, Microsoft Azure, and Google GCP• Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB)• Experienced in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries and Pig scripts. Show less