Around 5 years of experience in the IT industry which includes comprehensive experience in Big Data processing using Hadoop and its ecosystem (MapReduce, Pig, Hive, Sqoop, Flume, Spark, Kafka and HBase). Solid understanding of the Hadoop Distributed File System and Big Data ecosystem.Excellent Experience in Hadoop architecture and various components such as HDFS, YARN, MapReduce, Spark, Pig, Sqoop, Hive, Impala, HBase, Kafka.Experience in design and development of custom ETL pipelines using Spark, SQL and Python.Hands-on experience on Google Cloud Platform (GCP) in all the big data products bigquery, Cloud DataProc, Google Cloud Storage, Composer (Airflow as a Service).Strong understanding of real time streaming technologies Spark and Kafka.Good Exposure on Apache Hadoop Map Reduce programming PIG Scripting and Distribute Application and HDFS.Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice - versa.Have good programming experience with Python and ScalaCan work parallel in both GCP and AWS Cloud services coherently.Knowledge of job workflow management and coordinating tools like Oozie.Hands-on experience with Amazon EC2, Amazon S3, Amazon RDS, Auto Scaling, EMR, Lambda and other services of the AWS Family.Install and configure chef server /workstation and nodes via CLI tools to AWS nodes.Created users and groups using IAM and assigned individual policies to each group.Experience in working with Github private repositories and docker repositories.Experience with Docker to create, manage, deploy and run containerized applicationsSound knowledge in various databases like MySQL & NoSQL.Experience in working with various build tools like Maven.Strong working experience using Agile methodologies including Scrum.Knowledge of some of the unix/linux commands.Experience with different file formats like Avro, parquet, ORC, Json & XML.Instantiated, created and maintained CI/CD pipelines and apple automation to environments and applications.Excellent ability to understand complex scenarios and business problems and transfer the knowledge to other team members in the most comprehensive manner.Strong communication skills, analytic skills, good team player and quick learner, organized and self-motivated.
-
Big Data EngineerIbm Jan 2023 - PresentToronto, Ontario, CanadaLed migration project of big data workflows to Google Cloud Platform (GCP), resulting in a 30% increase in data processing efficiency.Developed scripts using PySpark to push the data from GCP to the third-party vendors using API framework.Implemented real-time data processing frameworks to process and analyze terabytes of data sets, increasing data accuracy by 45%.Vast experience in identifying production bugs in the data using stack driver logs in GCP.Designed and optimized SQL queries and ETL operations.Championed data governance and security protocols on cloud platforms, to mitigate security risks.Spearheaded the adoption of innovative BigQuery solutions, improving query performance by 60%Automated routine tasks using GCP Data Fusion and enhanced data migration process to GCP using Cloud Dataflow.Leveraged GCP Dataflow to build high-throughput, fault-tolerant data pipelines.Automated data extraction and integration processes using GCP Data Fusion.Implemented various optimization techniques like Dynamic Partitions, Buckets, Map Joins, Parallel executions in Spark.Parse Json files through Spark core to extract schema for the production data using SparkSQL and Scala.Designed and executed data schemas, achieving a 10% improvement in data validation.Conducted data cleaning processes to enhance data accuracy. Created BigQuery authorized views for row level security or exposing the data to other teams. -
Data EngineerAmazon Jan 2019 - Apr 2022Hyderabad, Telangana, IndiaResponsible for building scalable distributed data solutions using Hadoop Ecosystem.Responsible for troubleshooting issues in the execution of Spark jobs by inspecting and reviewing log files.Converted ETL pipelines to Scala code base and performed data accessibility to & from S3. Develop Spark and PySpark code to extract data from various databases, apply innovative ideas around the Data Science and Advanced Analytics practices Creatively and present models to business customers and executives, utilizing a variety of formats and visualization methodologies.Experience in using Sqoop to import and export the data from Oracle DB into S3 and HIVE.Good familiarity with AWS services like DynamoDB, Redshift, Simple Storage Service(S3), Amazon ElasticSearch Services.Performed PostgreSQL DDL parsing to be Amazon Redshift compatible form in building the data warehousing.Design and develop ETL pipelines in AWS Glue to migrate data from external sources like s3, ORC/ParquetText files into AWS Redshift.Created external tables with partitions using Hive, AWS Athena and Redshift.Used Spark streaming to receive real time data from Kafka and store the stream data to S3using Scala.Understanding of data storage and retrieval techniques, ETL and databases, to include graph stores, relational databases, tuple stores, NOSQL, Hadoop, MySQL, Spark MLLIB libraries for designing recommendation Engines Analysis predicted by Statistical analysis using Spark.Implemented columnar data storage, advanced compression and massive parallel processing using Multinode Redshift feature.Involved in architecture and design of distributed time-series database platform using NOSQL technologies like Hadoop / Hbase, Zookeeper.Developed data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.Used AWS EMR to transform and move large amounts of data into and out of other AWS data sources and databases, such as Amazon Simple Storage Service (S3) and DynamoDB.
Gowtham Reddy Education Details
-
Information Technology -
Computer Science
Frequently Asked Questions about Gowtham Reddy
What company does Gowtham Reddy work for?
Gowtham Reddy works for Ibm
What is Gowtham Reddy's role at the current company?
Gowtham Reddy's current role is Big Data Engineer.
What schools did Gowtham Reddy attend?
Gowtham Reddy attended Fanshawe College, Jawaharlal Nehru Technological University.
Who are Gowtham Reddy's colleagues?
Gowtham Reddy's colleagues are Jalyna West, Yi Gu, Florence Kellermann, Larry Spencer, Partenie Marian Alexandru, Chandrashekhar Kumatkar, Monica Forbice.
Not the Gowtham Reddy you were looking for?
-
-
-
Gowtham Reddy
Toronto, On -
Gowtham R.
Data Analyst | Specializing In Bi Tools & Big Data | Turning Data Into Actionable InsightsNorth York, On
Free Chrome Extension
Find emails, phones & company data instantly
Download 750 million emails and 100 million phone numbers
Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.
Start your free trial