Abhijit Bhattacharya Email & Phone Number

San Mateo, Ca, Us

Wilmington, Delaware, Us

* Mentoring engineers to data engineering career path

San Francisco, California, Us

* Leading the migration of Mural's main data warehouse from Redshift to Databricks* Spearhead the data modeling effort to design core entities & metrics, ensuring single source being delivered to stakeholders* Use of DBT to increase developer productivity and improving the underlying data models* Up level the team to follow best practices in data pipeline design, scalability & performance* Evangelize use of Databricks notebook to product manager, analytics groups

San Francisco, California, Us

* Addressed scalability, performance issues for data pipelines handling multiple of billions of records per day specially the ones with workspace and messages* Improving the core model to cater to time invariant reporting at scale and provide single source of truth to stakeholders* Leading initiatives in data quality, operational excellence and improving SLA by 30%* Led projects to collect and visualize Spark / Tez metadata to identify inefficient queries/pipelines and optimize queries resulting in half million dollar cost savings by eliminating AWS EMR clusters* Mentored members in data engineering team, presented several tech talks and guide the teams to introduce concepts like data salting to address data skew

San Francisco, California, Us

Slack data warehouse consists of hundreds of data pipelines bringing data from multitude of disparate sources like mySQL, Salesforce, Workday Financials, flat files, AWS s3 to name a few. These pipelines have complex dependencies on each other.My work involves interacting with business users, data scientists to understand and cater to their data requirements. It includes enhancing existing schema, pipelines, formalizing common dimensions to make date warehouse single source of truth, designing, deploying and tuning data pipelines which scales well with increased data volume. I mentor junior engineers, standardize ETL / Airflow design, optimize Hive/Spark SQL. To address issues like data skew, I introduced the design patterns like data salting which were extremely useful not only to solve scalability problem but to ensure operational stability.Metadata and its visualizations are close to my heart. I strongly believe that to keep pace with the needs of a growing data-driven organization, metadata can be a potent tool. I led design and development of a framework that can automate back processing (backfilling) based on data source dependencies. I paved the way to stitch up metadata coming from orchestration framework (e.g Airflow) with EMR cluster/Tez application log metadata. Visualizing task level details in Jupyter notebook using charting utilities like plotly and seaborn helped auto detect processing inefficiencies, data skew and reduce mis-utilization of cluster resources.

Sydney, Nsw, Au

I initiated the project to transition from legacy Pentaho/Kettle based ETL orchestration tool to Apache Airflow. That reduced the operational cost of supporting multiple orchestration framework and streamline data landing time of business critical metrics.Tools & Languages:AWS, Airflow, Python, Presto, Hive & Athena

Sunnyvale, Ca, Us

In the early days LinkedIn's main data warehouse was using Teradata appliance. I played lead role in moving out data transformation logic from Teradata to Hadoop based ecosystem, integrated Hadoop into traditional ETL landscape at LinkedIn. While working with workflows running across disparate platforms, I pioneered a metadata collection project to enable visualization of data dependency, generation of data & process lineage. Eventually this project (named WhereHows) became an open source initiative. Later working as part of marketing services, I was responsible for offline and real time data pipelines for clicks, impression and other user activity events for campaign performance & demographic analysis. Highlights:Initiated and shepherded a project to cater to LinkedIn's metadata management needs which eventually became open source project: WhereHowsOne of my focus area is automation in ETL systems. This included high water mark detection, easy recovery and restart-ability of ETL pipelines, ability to reprocess large volume of data.Tools & Languages: Apache Pig/Hadoop, Spark, MapReduce, mySQL, Teradata, Python & Java

Jersey City, Nj, Us

I was instrumental in starting BI initiative in the company. As data architect of DW team my responsibilities were forming the team, running proof of concepts with ETL & reporting tool vendors , manage the data model in Netezza to support existing reporting needs. I juggled with multiple projects integrating new data sets enriching data warehousing for analytical applications.Tools & Languages:SQL, Perl, IBM DataStage, Apache Pig, MapReduce, Java, MicroStrategy & TableauKey Responsibilities:• Running proof of concepts to select appropriate ETL & reporting tool• focal point for understanding data from the corporate perspective and representing the data so it can be understood uniformly throughout the company.• Data modeling and physical design of database• Scripting SQL/Shell/Perl to help in data analysis• Delivering analytical solutions using Tableau & MicroStrategy

San Mateo, California, Us

Reporting directly to VP Business Intelligence, I started BI and data analytics initiative in the company. I led efforts to set up platform infrastructure for CRM solution, catalog analysis to provide insight to Nextag merchants.Key Responsibilities:• Designed data model to store user level data in the main Oracle data warehouse. The main DW was storing 14 months of data allowing granular analysis of user behavior.• Designed and developed ETL process to consolidate data from operational systems and product catalog to provide accurate actual revenue as well as projection for coming months.

Sunnyvale, Ca, Us

Designed the database architecture of web analytical solution to unlock the richness of Yahoo’s sponsored search and display advertising business. The application became a cornerstone for account managers to understand and analyze campaign performance and compare the key metrics against industry peers.Key Responsibilities:• Responsible for dimensional modeling, star schema design to integrate search & display data in one unified platform• Implemented fine grain access control (FGAC) mechanism to enforce application security and guard sensitive data• Innovated implementation of views which can morph based upon input data selection of end users. Lead the effort to define centralize business logic , complex metric calculation in database APIs which are exposed to front end reporting• Defined unit testing methodology for database centric ETL processes, reporting APIs and automated continuous integration effort. Tools and utilities designed to unload subset of data, automating setup and execution of unit testing scripts reduced development effort drastically

Redwood City, Ca, Us

San Francisco, Ca, Us

Palo Alto, Ca, Us

Bentonville, Arkansas, Us

Mcse, Computer Science

Nanodegree, Data Structures & Algorithms

Nanodegree, Data Analyst

Bcse, Computer Science & Engineering