Senior Data Engineer
CurrentBuilt data pipelines using AWS firehose, S3 and loaded the JSON payload in Snowflake using snow pipeBuilding pipelines in Databricks to create Bronze, Silver& Gold layer Schema design to reduce the latency of data to < 1 minEffectively working on data which are coming from Kafka which are using messaging queues, streaming technology and batch processing to load data into SnowflakeDeveloped framework to ingest data from Bigquery to Snowflake to build single platform for any sources using sparkArchitected and developed framework to read data from intercom using API and load the data into snowflakeMigrated all pipelines from databricks to dataproc and made our framework to use open source tools Upgraded spark clusters to 3.2.0 and upgraded all other connectors to use 3.2.0 version of sparkBuilt and architected framework to export data from snowflake to multiple sources with different file format optionsOptimized snowflake usage for a 44B record table to effectively use partition and clustering the table using appropriate keysAutomated cluster restart process and drone build for frameworks to automatically restart the streaming clusters upon failure and avoid manual intervention for on-call engineers. Reduced several hours of human on-call hours