Big Data Engineer with 10+ years experience on gathering System Requirements, Analyzing the requirements, Designing, and developing systems. 5+ years of experience on Big Data Technologies like Hadoop. Strong expertise in ETL tools & Cloud : Azure & AWSAs a Data expert, striving primarily to understand clients’ needs and provide them exceptional results through the SDLC (Software Development Life Cycle) approach. I am determined to implement earned technical skills and knowledge for the rapid growth of the prospective organization, teams by working, managing effectively and consistently with significant experience in providing cutting-edge knowledge, services and teaming up to collaboratively implement operational structures and processes that deliver on strategic objectives through the good use of communication, data skills, technical knowledge & data analysis.My Technical Skills:Hadoop/Big Data Ecosystems: Hadoop, MapReduce, HDFS, HBase, Zookeeper, Kafka, Hive, Pig, Sqoop, Cassandra, Oozie, Storm, and Flume.ETL Tools: Informatica, Glue, snowflake.NO SQL Database: Dynamo DB, Mango DB, HBase, Cassandra.Programming: Python, Scala, SQL, Shell Scripting.Operating Systems: UNIX, LINUX, Windows.Version Control: GIT, UCD, Ansible.Cloud Computing: AWS, Azure.BI Tools: Tableau, Power BI.Development Methodologies: Agile, Waterfall.Areas of Expertise: · Product Road Mapping · Product Management · Data Analytics · Leadership · Cross-functional Team Leadership · Team Building · Project Management · Agile & Waterfall methodologies · SDLC · Jira · Service Now · Remedy · Customer Engagement · Strategic Thinking · Product Strategies · Go-to-Market Strategy · Responsiveness · Digital Engagement · Commerce · Coaching · Brand Strategy · Long-term Vision · Workload Prioritization · Timelines · Interpersonal Skills · Communication · Business Insights · User Experience (UX) · Pricing Systems · Customer Communication · Change Management · Problem Solving · Analytical Skills · Architecture · Spark · PySpark · Python · Azure · ETL development · Business Analysis · ETL tools · Tableau · Alteryx
Discover Financial Services
View- Website:
- discover.com
- Employees:
- 21509
-
Big Data EngineerDiscover Financial ServicesMemphis, Tn, Us
-
Big Data EngineerDiscover Financial Services May 2021 - PresentChicago, Illinois, United States• Designing, developing, and maintaining big data pipelines to process and analyze large datasets efficiently.• Developing and implementing data ingestion solutions using tools such as Apache NiFi, Kafka, and AWS Kinesis.• Utilizing big data technologies such as Hadoop, Spark, and AWS Cloud to develop data processing and analysis applications and migrated legacy applications from Hadoop.• Implementing and optimizing data storage solutions such as HDFS and S3.• Developing and maintaining ETL workflows using tools such as Apache Airflow and AWS Step Function.• Developing and maintaining distributed database systems such as Apache Cassandra, MongoDB, and MySQL Cluster.• Proficient in Python programming language and its data structures, libraries, and frameworks• Developed multiple projects using Python, including web applications, data analysis scripts, and automation tools.• Developed and executed SQL queries using Amazon Athena to perform ad-hoc analysis and generate insights from large datasets.• Utilized Amazon Athena to perform data transformation and manipulation tasks using SQL functions and expressions.• Worked with various data sources including Amazon S3, Amazon RDS, Amazon Redshift, and third-party data sources for querying using Amazon Athena.• Designed and developed interactive dashboards using Amazon QuickSight for data visualization and business intelligence reporting.• Utilized Amazon Quick Sight to create ad-hoc analysis reports and perform data exploration on large datasets.• Worked with various data sources including Amazon S3, Amazon RDS, Amazon Redshift, and third-party data sources for data visualization using Amazon Quick Sight.• Proficient in using EC2 instances for containerized applications using Amazon ECS or EKS.• Experienced in using EC2 instances with other AWS services such as RDS, S3, and Lambda.• Experienced in launching, configuring, and managing EC2 instances on AWS. -
Big Data EngineerCalpine Energy Solutions, Llc Jul 2019 - Apr 2021Houston, Texas, United States• Designing and implementing GCP infrastructure solutions for clients and internal teams.• Deploying and managing GCP services, including compute, storage, networking, and security components.• Strong knowledge of Apache Spark architecture and Spark RDD transformations and actions.• Hands-on experience in tuning and optimizing PySpark jobs for performance.• Ability to work with data in various formats such as JSON, CSV, Avro, Parquet, and ORC using PySpark.• Build batch and event driven data pipelines using DataFlow, Cloud Function, PubSub, Cloud SQL, Composer (Airflow), BigQuery, Data Proc and Cloud storage. • Utilized Data Proc for distributed processing of large-scale datasets in a cloud environment.• Managed and maintained Data Proc clusters for batch processing of large datasets using Apache Hadoop and Spark.• Implemented and optimized ETL workflows using Data Proc for data ingestion and transformation.• Proficient in object-oriented programming (OOP) concepts and design patterns in Python• Familiar with Python data analysis libraries like NumPy, Pandas, and SciPy for statistical analysis and machine learning tasks.• Optimized query performance in GCP BigQuery query tables using tuning features and best practices.• Utilized GCP Big Query to perform partitioning of large datasets for improved query performance.• Worked with GCP Big Query to query nested and complex data types such as JSON, Parquet, and ORC.• Designing and deploying GCP infrastructure using tools such as Google Cloud Deployment Manager, Terraform, and Ansible.• Configuring and managing GCP services such as Compute Engine, Kubernetes Engine, Cloud Storage, Cloud SQL, and Cloud Pub/Sub. -
Data EngineerFonteva Jan 2018 - Jun 2019Arlington, Virginia, United States• Implemented Snowflake data warehouse and migrated data from on-premises servers to Snowflake.• Designed and implemented ETL workflows using Snowflake's Snowpipe and bulk-loading features.• Strong knowledge of Redshift distribution styles, sort keys, and compression options.• Optimized Snowflake queries by creating materialized views and indexing tables.• Develop and maintain scalable data pipelines that will ingest, transform, and distribute data streams and batches within the AWS S3 and Snowflake using AWS Step Function, AWS Lambda, AWS Kinesis, AWS Glue, and AWS EMR. • Created batch pipelines using AWS S3, AWS Lambda, AWS Glue, AWS EMR, AWS Athena, AWS RedShift, AWS RDS, etc.,• Orchestrated pipelines and dataflow using Apache Airflow and Step Function.• Created reports and dashboards using AWS services like Lambda, Glue, Step Function, and QuickSight.• Created monitoring service using AWS CloudWatch, AWS Lambda, AWS Glue, AWS Step Function, Grafana, Sagemaker and ElasticSearch.• Created Airflow dags to extract, transform and load data into Data Warehouse. • Developed and deployed Kubernetes pods to extract, transform, and load data. • Used Docker and Kubernetes for Data Pipelines and ETL Pipelines. -
Data EngineerBajaj Allianz General Insurance Mar 2014 - Nov 2016Pune, Maharashtra, India• Installed, Configured, and Maintained Apache Hadoop clusters for application development and Hadoop tools like Hive, Pig, Zookeeper, Kafka, and Sqoop.• Integrated HDP clusters with Active Directory and enabled Kerberos for Authentication.• Implemented and maintained Snowflake's access control and security features to ensure compliance with industry standards.• Configured and maintained Snowflake's network connectivity and integration with other services.• Worked with Snowflake's semi-structured data support to process and analyze JSON and other non-tabular data formats.• Installed and Configured Sqoop to import and export the data into Hive from Relational databases.• Administering large Hadoop environments building and supporting cluster setup, performance tuning, and monitoring in an enterprise environment.• Monitoring the Hadoop cluster functioning through MCS and working on NoSQL databases including HBase. -
Etl DeveloperInfibeam Avenues Limited May 2012 - Feb 2014Gujarat, India• Experience in building and architecting multiple data pipelines end to end ETL and ELT for data ingestion and transformation in GCP and coordinate task among them.• Implemented and Managed ET solutions and automating operational processes.• Design and develop ET integration patterns using Python on Spark.• Develop framework for converting existing PowerCenter mappings and to PySpark (Python and Spark) Jobs.• Build data pipelines in airflow in GCP for ET related jobs using different airflow operators.• Worked on creating various types of indexes on different collections to get good performance in Mongo database.• Monitoring Big query, Dataproc and cloud Data flow jobs via Stack driver for all the environments.• Used Agile for the continuous model deployment.• Worked with Google data catalog and other google cloud APIs for monitoring, query and billing related analysis for big query usage.
Frequently Asked Questions about Krishna V
What company does Krishna V work for?
Krishna V works for Discover Financial Services
What is Krishna V's role at the current company?
Krishna V's current role is Big Data Engineer.
Who are Krishna V's colleagues?
Krishna V's colleagues are Carol Crego, Lucinda Gray, Ellen Payne, Autumn Camp, Destiny Hughes, Ryan Nelson, Jen Bronson.
Not the Krishna V you were looking for?
Free Chrome Extension
Find emails, phones & company data instantly
Aero Online
Your AI prospecting assistant
Select data to include:
0 records × $0.02 per record
Download 750 million emails and 100 million phone numbers
Access emails and phone numbers of over 750 million business users. Instantly download verified profiles using 20+ filters, including location, job title, company, function, and industry.
Start your free trial