• Designed and 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗱 inhouse 𝗗𝗮𝘁𝗮 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 for batch and streaming workloads.• Achieved a 𝟳𝟬% 𝗿𝗲𝗱𝘂𝗰𝘁𝗶𝗼𝗻 in pipeline creation 𝗲𝗳𝗳𝗼𝗿𝘁, supporting 6000+ pipelines across teams/domains.• 𝗨𝗽𝗴𝗿𝗮𝗱𝗲𝗱 𝗦𝗽𝗮𝗿𝗸 version (2.4 to 3.3) for 𝟰𝗸+ 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 achieving 𝗰𝗼𝘀𝘁 𝘀𝗮𝘃𝗶𝗻𝗴𝘀 and 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁𝘀.• 𝗟𝗲𝗱 a team of 𝟮𝟱 in 𝗟𝟭 𝗱𝗮𝘁𝗮 𝗹𝗮𝗸𝗲 𝘀𝘂𝗽𝗽𝗼𝗿𝘁, enabling… Show more • Designed and 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗱 inhouse 𝗗𝗮𝘁𝗮 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 for batch and streaming workloads.• Achieved a 𝟳𝟬% 𝗿𝗲𝗱𝘂𝗰𝘁𝗶𝗼𝗻 in pipeline creation 𝗲𝗳𝗳𝗼𝗿𝘁, supporting 6000+ pipelines across teams/domains.• 𝗨𝗽𝗴𝗿𝗮𝗱𝗲𝗱 𝗦𝗽𝗮𝗿𝗸 version (2.4 to 3.3) for 𝟰𝗸+ 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 achieving 𝗰𝗼𝘀𝘁 𝘀𝗮𝘃𝗶𝗻𝗴𝘀 and 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁𝘀.• 𝗟𝗲𝗱 a team of 𝟮𝟱 in 𝗟𝟭 𝗱𝗮𝘁𝗮 𝗹𝗮𝗸𝗲 𝘀𝘂𝗽𝗽𝗼𝗿𝘁, enabling self-sufficiency for 6000+ big data pipelines.• Empowered team for independent failure analysis and due diligence in pipeline monitoring/support.• Partnered with stakeholders to define and implement technical solutions for framework and platform needs.• Worked on 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 𝗺𝗶𝗴𝗿𝗮𝘁𝗶𝗼𝗻 (Kubernetes to Celery), upgrades, and optimization for growing pipelines.• Managed 𝗖𝗜/𝗖𝗗 components with 𝗚𝗶𝘁, 𝗠𝗮𝘃𝗲𝗻, 𝗝𝗲𝗻𝗸𝗶𝗻𝘀, and 𝗗𝗼𝗰𝗸𝗲𝗿 for pipelines and framework changes.• Established 𝗱𝗮𝘁𝗮 𝗹𝗮𝗸𝗲 𝗯𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 and conducted POCs, driving new tool adoption and improvements. Show less

• Built 𝗱𝗮𝘁𝗮 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 catering to 𝗲-𝗰𝗼𝗺𝗺𝗲𝗿𝗰𝗲 𝗳𝗿𝗮𝘂𝗱 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 use cases.• Detected collusion, frequent returns, and linked accounts in data streams (orders, refunds, cancellations).• 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗲𝗱 with UI/UX team for 𝗨𝗜 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻, ensuring data accuracy on dashboards.• Worked with 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲 team, providing data and integrating 𝗺𝗼𝗱𝗲𝗹𝘀 into 𝗳𝗿𝗮𝘂𝗱 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 pipelines.• Provided… Show more • Built 𝗱𝗮𝘁𝗮 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 catering to 𝗲-𝗰𝗼𝗺𝗺𝗲𝗿𝗰𝗲 𝗳𝗿𝗮𝘂𝗱 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 use cases.• Detected collusion, frequent returns, and linked accounts in data streams (orders, refunds, cancellations).• 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗲𝗱 with UI/UX team for 𝗨𝗜 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻, ensuring data accuracy on dashboards.• Worked with 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲 team, providing data and integrating 𝗺𝗼𝗱𝗲𝗹𝘀 into 𝗳𝗿𝗮𝘂𝗱 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 pipelines.• Provided 𝗱𝘆𝗻𝗮𝗺𝗶𝗰 𝗱𝗮𝘀𝗵𝗯𝗼𝗮𝗿𝗱𝘀 to analysts by implementing 𝗦𝗤𝗟 queries with Apache Freemarker.• 𝗠𝗶𝗴𝗿𝗮𝘁𝗲𝗱 𝗼𝗻-𝗽𝗿𝗲𝗺 Scala app to 𝗚𝗖𝗣, optimizing for 𝗰𝗼𝘀𝘁 𝘀𝗮𝘃𝗶𝗻𝗴𝘀 of $𝟰𝟱𝟬𝟬/𝗺𝗼𝗻𝘁𝗵 and 𝟮𝟱𝟬% 𝗳𝗮𝘀𝘁𝗲𝗿 execution.• 𝗠𝗶𝗴𝗿𝗮𝘁𝗲𝗱 pipelines from 𝗔𝘂𝘁𝗼𝗺𝗶𝗰 to 𝗔𝗽𝗮𝗰𝗵𝗲 𝗔𝗶𝗿𝗳𝗹𝗼𝘄, optimizing for 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 task 𝗾𝘂𝗲𝘂𝗲 𝘁𝗶𝗺𝗲.• 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 data management tasks like 𝗯𝘂𝗰𝗸𝗲𝘁 𝗰𝗿𝗲𝗮𝘁𝗶𝗼𝗻 and 𝗕𝗶𝗴𝗤𝘂𝗲𝗿𝘆 table refresh with metadata updates.• Used 𝗘𝗿𝘄𝗶𝗻 𝗗𝗮𝘁𝗮 𝗠𝗼𝗱𝗲𝗹𝗲𝗿 to streamline 𝗱𝗮𝘁𝗮 𝗺𝗼𝗱𝗲𝗹𝗶𝗻𝗴 and transformation processes. Show less

• Utilized 𝗦𝗾𝗼𝗼𝗽 and 𝗝𝗗𝗕𝗖 to 𝗶𝗻𝗴𝗲𝘀𝘁 𝗱𝗮𝘁𝗮 from RDBMS, Cosmos DB, and Cassandra 𝗶𝗻𝘁𝗼 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲.• Performed data cleaning and 𝗹𝗼𝗮𝗱𝗲𝗱 𝗱𝗮𝘁𝗮 into 𝗰𝗮𝘁𝗮𝗹𝗼𝗴 𝘇𝗼𝗻𝗲 using 𝗛𝗶𝘃𝗲 𝗦𝗤𝗟, 𝗦𝗽𝗮𝗿𝗸 𝗦𝗤𝗟, and 𝗣𝘆𝗦𝗽𝗮𝗿𝗸.• 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 pipelines, improving 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 by 𝟰𝟬% 𝘁𝗼 𝟯𝟱𝟬% with partitioning and Spark tuning.• Used 𝗔𝘂𝘁𝗼𝘀𝘆𝘀 for 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 and 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗻𝗴… Show more • Utilized 𝗦𝗾𝗼𝗼𝗽 and 𝗝𝗗𝗕𝗖 to 𝗶𝗻𝗴𝗲𝘀𝘁 𝗱𝗮𝘁𝗮 from RDBMS, Cosmos DB, and Cassandra 𝗶𝗻𝘁𝗼 𝗗𝗮𝘁𝗮 𝗟𝗮𝗸𝗲.• Performed data cleaning and 𝗹𝗼𝗮𝗱𝗲𝗱 𝗱𝗮𝘁𝗮 into 𝗰𝗮𝘁𝗮𝗹𝗼𝗴 𝘇𝗼𝗻𝗲 using 𝗛𝗶𝘃𝗲 𝗦𝗤𝗟, 𝗦𝗽𝗮𝗿𝗸 𝗦𝗤𝗟, and 𝗣𝘆𝗦𝗽𝗮𝗿𝗸.• 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 pipelines, improving 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 by 𝟰𝟬% 𝘁𝗼 𝟯𝟱𝟬% with partitioning and Spark tuning.• Used 𝗔𝘂𝘁𝗼𝘀𝘆𝘀 for 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 and 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗻𝗴 Spark 𝗷𝗼𝗯𝘀 efficiently.• Created technical design, 𝗱𝗮𝘁𝗮 𝗺𝗼𝗱𝗲𝗹𝘀, and documentation for the implemented 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀.• 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗲𝗱 data from 𝗥𝗘𝗦𝗧 𝗔𝗣𝗜𝘀 using Python libraries (requests, pycurl, urllib3).• Applied Python 𝗺𝘂𝗹𝘁𝗶𝘁𝗵𝗿𝗲𝗮𝗱𝗶𝗻𝗴 and 𝗺𝘂𝗹𝘁𝗶𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 to boost performance by several magnitudes.• 𝗪𝗼𝗿𝗸𝗲𝗱 with 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗱𝗮𝘁𝗮 𝗳𝗼𝗿𝗺𝗮𝘁𝘀: XML, YAML, JSON, HOCON, CSV, TXT, ORC, Avro, Parquet. Show less