Big Data Engineer
Current๐ฃ๐ฟ๐ผ๐ท๐ฒ๐ฐ๐: ๐๐ป๐ฑ๐๐๐๐ป๐ฑ ๐๐ฎ๐ป๐ธAs a part of Innoactive Intelligence, now under the umbrella of Gupshup, I am driving the IndusInd Bank project. In this role, I architect robust data pipelines and structures to ensure efficient data processing and analysis.๐๐๐ฎ ๐๐ค๐ก๐๐จ & ๐๐๐จ๐ฅ๐ค๐ฃ๐จ๐๐๐๐ก๐๐ฉ๐๐๐จ:โพImplementing Spark SQL in production, establishing vigilant performance monitoring and proactive issue resolution systems.โพManaging serialized dataโฆ Show more ๐ฃ๐ฟ๐ผ๐ท๐ฒ๐ฐ๐: ๐๐ป๐ฑ๐๐๐๐ป๐ฑ ๐๐ฎ๐ป๐ธAs a part of Innoactive Intelligence, now under the umbrella of Gupshup, I am driving the IndusInd Bank project. In this role, I architect robust data pipelines and structures to ensure efficient data processing and analysis.๐๐๐ฎ ๐๐ค๐ก๐๐จ & ๐๐๐จ๐ฅ๐ค๐ฃ๐จ๐๐๐๐ก๐๐ฉ๐๐๐จ:โพImplementing Spark SQL in production, establishing vigilant performance monitoring and proactive issue resolution systems.โพManaging serialized data proficiently in Spark, encompassing formats like Avro, Parquet, and ORC for optimized efficiency.โพUtilizing specialized Spark serialization tools and Java serialization techniques to enhance data processing performance.โพNavigating through binary and textual data landscapes adeptly using Spark DataFrames and RDDs.โพApplying advanced Spark methodologies, including data compression and off-heap storage, for optimal performance.โพAdministering AWS EMR Spark clusters to ensure impeccable performance and high availability.โพDesigning and executing data pipelines on Amazon EMR, leveraging the capabilities of Spark, Python, and Scala.โพDevising architectures for data lakes on Amazon S3, striking a balance between performance and cost- effectiveness.โพEnhancing Spark jobs and workflows to achieve scalability and resource efficiency.โพDeveloping intricate Spark applications for complex data transformations.โพTailoring the use of ORC, Parquet, and Avro formats to suit specific use cases.โพBalancing Hive table partitioning strategies to enhance query responsiveness.โพDiagnosing common Hive challenges, fine-tuning queries for optimal performance.โพSculpting Hive performance through meticulous configuration and strategic optimization.โพMitigating Hive bottlenecks to improve overall efficiency.โพLeveraging Hive's capabilities to amplify query performance. Show less