Ganesh V Email & Phone Number

Addison, Tx, Us

•Developed and automated scalable data workflows using Apache Spark, Scala, and shell scripting, which reduced processing times by streamlining job management and data transformations, ensuring robust system performance.•Enhanced code deployment and operational efficiency through the implementation of CI/CD pipelines using Jenkins and UrbanCode Deploy, and managed version control with GitHub and Git Bash.•Executed big data storage and file management across Hadoop technologies and cloud environments, including migration of data operations to Azure Databricks, integration with Azure Data Lake Storage (ADLS), and optimization of data formats in Parquet.•Implemented comprehensive data compliance and security protocols, including secure file transfers via SFTP following HIPAA guidelines, to protect patient information and ensure data integrity.•Utilized Control-M and Zena for advanced job scheduling and orchestration, improving the reliability and timeliness of data processing tasks within highly regulated healthcare insurance frameworks.

Urbandale, Iowa, United States

Experienced Big Data Engineer proficient in the end-to-end flow of Big Data applications, from data ingestion to HDFS, processing, and analysis. Skilled in collaborating with team members and stakeholders to design and develop robust data environments. Demonstrated expertise in designing Restful services using Java-based APIs like JERSEY. Proficient in utilizing airflow Operational Services for dynamic batch processing and workflow scheduling. Experienced in developing customized UDFs in Python to enhance Hive and Pig Latin functionality.In my day-to-day role, I am responsible for developing ETL pipelines both into and out of data warehouses. I excel in creating major regulatory and financial reports using advanced SQL queries in Snowflake. Additionally, I have hands-on experience in creating Cassandra tables to store diverse data formats from various sources. I have successfully designed and developed data integration programs within a Hadoop environment, utilizing NoSQL data store Cassandra for efficient data access and analysis.

St Louis, Missouri, United States

Experienced Big Data Engineer skilled in analyzing large data sets and deriving valuable customer usage patterns through the development of innovative MapReduce programs using Java. I have a strong background in Spark/Scala and Python, utilizing these technologies within the Hadoop/Hive environment on both Linux and Windows platforms to leverage big data resources effectively. My expertise extends to developing and implementing regular expression (regex) projects, contributing to insightful data analysis.In data migration projects, I have showcased my proficiency by writing SQL scripts to address data mismatches and successfully loading historical data from Teradata SQL to Snowflake. Leveraging SparkSQL, I have efficiently handled JSON data, creating Schema RDDs and loading them into Hive Tables.I possess in-depth knowledge of Kafka and have designed Kafka producer clients using Confluent Kafka to produce events into Kafka topics. Additionally, I have evaluated various big data technologies and prototyped solutions, driving continuous improvements in data processing architecture.

Boise, Idaho, United States

•Analyze the existing application programs and tune SQL queries using execution plan, query analyzer, SQL Profiler and database engine tuning advisor to enhance performance.•Migrated on premise database structure to Confidential Redshift data warehouse.•Was responsible for ETL and data validation using SQL Server Integration Services.•Wrote various data normalization jobs for new data ingested into Redshift.•Implementing and Managing ETL solutions and automating operational processes.•Used Kafka functionalities like distribution, partition, replicated commit log service for messaging systems by maintaining feeds and Created applications using Kafka, which monitors consumer lag within Apache Kafka clusters.•Managed security groups on AWS, focusing on high-availability, fault-tolerance, and auto scaling using Terraform templates. Along with Continuous Integration and Continuous Deployment with AWS Lambda and AWS code pipeline.In this role, I have used Hive SQL, Presto SQL and Spark SQL for ETL jobs and using the right technology for the job to get done. Created various complex SSIS/ETL packages to Extract, Transform and Load data. Defined facts, dimensions and designed the data marts using the Ralph Kimball's Dimensional Data Mart modeling methodology using Erwin. Defined and deployed monitoring, metrics, and logging systems on AWS. Worked publishing interactive data visualizations dashboards, reports /workbooks on Tableau and SAS Visual Analytics.

Hyderabad, Telangana, India

•Involved in complete Software Development Lifecycle (SDLC).•Worked on different dataflow and control flow task, for loop container, sequence container, script task, executes SQL task and Package configuration.•Extensive use of Expressions, Variables, Row Count in SSIS packages•Created SSIS packages to pull data from SQL Server and exported to Excel Spreadsheets and vice versa.•Created batch jobs and configuration files to create automated process using SSIS.•Data validation and cleansing of staged input records was performed before loading into Data Warehouse•Automated the process of extracting the various files like flat/excel files from various sources like FTP and SFTP (Secure FTP)Database developer responsible for • Interacted with Business analysts and stake holders to gather business requirements. Transferred data from various data sources /business systems including DB2,MS EXCEL,FLAT FILES ETC. Experienced in Data modelling and Data design processing. Deploying and scheduling reports using SSRS to generate daily, weekly, monthly and quarterly reports. Loading data from various sources like OLEDB, flat files to SQL Server database Using SSIS Packages and created data mappings to load the data from source to destination.