Bala Chandar B Email and Phone Number

Features a complete ETL batch processing application leveraging blob storage, spark and is orchestrated by Airflow - DAG jobs running on Databricks. The application comprises of data sourcing from blob storage, data transformation processing layer as spark, intermediate tables are saved in hive and results are stored in snowflakes.Working closely with Data Science team and implementing the clustering/geo spatial models in large scale datasets, parallelly integrating the new application features to existing PySpark application modules.Handled application repository codes using BitBucket - SourceTree as VCS.

Added data quality checks for multiple source/target systems like Hive, Couchbase, S3, Kafka and MinIO to existing spark application framework which is delivered as API service to internal end-users.Built a custom CI/CD pipeline using concourse tool which involves git webhook push, docker image creation and spark job submission in Kubernetes cluster.Identified the opportunities and Implemented reduction of manual repeated querying activities through python using pandas (data frames) to generate excel reports.Enhanced the application to get job arguments from RDBMS instead from configuration files.

Built a pre-processing feature application comprises of shell scripting and hive queries to move the raw flat-file data from NAS to Hadoop data lake storage and then converting into consumable hive schematic data warehouse tables.Analyzing BI documents and scheduling the job runs for each source systems to perform new data sourcing. Built and implemented automated reconciliation report using hive queries for monitoring the daily/monthly incremental batch runs for a given banking source systems.