Ankit Kumar Email & Phone Number

Texas, United States

Buffalo, Ny

• The Apache Hadoop cluster is installed and configured using both completely distributed and pseudo-distributed operations.• Collaborated with IT and business to enhance the efficiency of system development through the use of the Agile methodology.• Adjust Hadoop XML parameters by storage and hardware specifications.• Actively engaged in all phases of the SDLC life cycle of the big data project, including requirement analysis, design, coding, testing, and production.• Utilizing… Show more • The Apache Hadoop cluster is installed and configured using both completely distributed and pseudo-distributed operations.• Collaborated with IT and business to enhance the efficiency of system development through the use of the Agile methodology.• Adjust Hadoop XML parameters by storage and hardware specifications.• Actively engaged in all phases of the SDLC life cycle of the big data project, including requirement analysis, design, coding, testing, and production.• Utilizing Sqoop, data is imported from RDBMS to HDFS as required.• Utilizing Cloudera Manager, Oozie, and Ambari Map for operational services to mitigate YARN growth.• Data from Hadoop was extracted and cross-referenced with data from files and reports to verify the Map-reduced Hive Scripts.• Capabilities Utilizing a variety of services related to Big Data, Hadoop, and NoSQL, such as Hadoop installation, scalability, and performance enhancement.• The ETL sequence was followed to conduct data validations between summary files and extract files.• Implemented numerous Teradata controls, such as the creation of tracking tables during the ETL process and the construction of contrasting landing zone tables.• Utilized Linux for big data resources, particularly for initiatives that involved regular expressions (regex), and developed Python and Spark/Scala for the Hadoop/Hive environment.• Complete the end-to-end design and development of the Apache Nifi flow, which serves as a conduit between the EBI and middleware teams and executes all of the aforementioned functions.• Employed the Cloudera Manager to consistently monitor and administer the Hadoop cluster.• Managing, updating, and configuring the Hortonworks Hadoop cluster. Show less

New City, New York, United States

• Employed Agile and Scrum methodologies during the project's development.• Assisting in the configuration of the Hadoop cluster's top-layer ecosystems, which include HBase, Oozie, Sqoop, and Hive.• Participates in each phase of the big data flow within the application, from the ingestion of upstream data to the processing and analysis of HDFS data. • To optimize ELT operations for transformation against the Hadoop file system, I implemented HIVE SQL.• Oversee the development of… Show more • Employed Agile and Scrum methodologies during the project's development.• Assisting in the configuration of the Hadoop cluster's top-layer ecosystems, which include HBase, Oozie, Sqoop, and Hive.• Participates in each phase of the big data flow within the application, from the ingestion of upstream data to the processing and analysis of HDFS data. • To optimize ELT operations for transformation against the Hadoop file system, I implemented HIVE SQL.• Oversee the development of performance optimization strategies for SQL tables, ORC, HIVE-managed, and ETL mappings.• Has experience with both NoSQL and conventional SQL databases, such as Oracle, SQL Server, HBase, and Cassandra.• Has experience with Hadoop's Hue, Pig, and Impala data analysis components, as well as Informatics Big Data Edition.• Azure Data Factory and Data Bricks were employed to construct a standardized framework for SFTP downloads or uploads.• Centralized the storage of secrets in Azure Key Vault and utilized Azure Data Factory and Data Brick notebooks to retrieve them.• Utilizing Azure Data Factory, I implemented a customized alerting system for monitoring and organized all data pipelines.• Utilized Python and Spark to execute various aggregation logics on the backend.• Utilize Azure Data Bricks and Data Factory to develop and supervise an optimal data pipeline architecture on the Microsoft Azure cloud.• I employed Pyspark and Spark SQL transformation in Azure Data Bricks to create intricate transformations that would facilitate the implementation of business principles.• Developed and implemented an ETL pipeline on the Azure cloud to facilitate the management and retrieval of customer data through APIs.• Presently, I am engaged in the development of data pipelines for Python, Pyspark, HiveSQL, and Presto.• PySpark was employed to generate scripts that transmitted data from GCP to external vendors via their API framework. Show less

India

• Ensure that the Hive and Pig queries are consistently enhanced to enhance their ability to analyze and acquire data.• Developed MapReduce, a data mining and analysis tool that operates over HDFS. It imports and stores data in R for use in Pig Script and MapReduce operations.• Proficient in the processing and analysis of data using Pyspark, as well as the manipulation of RDDs and data frames (Spark SQL).• Responsible for the configuration of HDFS and Hadoop MapReduce and the… Show more • Ensure that the Hive and Pig queries are consistently enhanced to enhance their ability to analyze and acquire data.• Developed MapReduce, a data mining and analysis tool that operates over HDFS. It imports and stores data in R for use in Pig Script and MapReduce operations.• Proficient in the processing and analysis of data using Pyspark, as well as the manipulation of RDDs and data frames (Spark SQL).• Responsible for the configuration of HDFS and Hadoop MapReduce and the establishment of numerous MapReduce tasks for data purification.• Employed the Spark execution engine in the Apache Hadoop architecture to bulk-combine intricate data sets.• Assisted in the provisioning, configuration, and construction of indexes for Elastic Search nodes. The Kibana dashboard for corporate users has been developed.• Several jobs were created to normalize data for Redshift data that was recently ingested.• Engaged in an experiment with the Amazon Web Services EC2 console.• Utilizing AWS Cloud Computing services, such as S3 and EC2, to rapidly analyze vast amounts of data.• PySpark is employed to extract, aggregate, and consolidate Adobe data within AWS Glue.• Configure Elastic Search and Kibana by utilizing Log Analysis from AWS Logs; manage data automation, dashboards, custom mapping, and searches.• AWS Terraform and Cloud Formation Templates (CFT) were employed to generate and execute stacks.• Several jobs were created to normalize data for Redshift data that was recently ingested.• Established a Lambda function to aggregate data from incoming events and store the output in S3 and Confidential Dynamo DB. This involved working on a Pyspark script to encrypt the raw data using hashing algorithm principles on client-specified columns. Show less