Azure Synapse - Spark Runtime team1. Synapse Meta Store (SYMS) :- SYMS contains metadata information. We had HMS running on SQL server. But team built SYMS & wanted to migrate due to better performance in query execution, storage optimization and better connectivity with multiple datasource not just SQL.Made changes to BBC-VHD codebase to Install the SHIM jar & its dependencies over cluster during VHD build. SHIM is contacted during query execution and on authentication query is routed to SyMS.Made changes to the meta store config parameter and ensured that the corresponding values are populated during cluster creation, like EC feature flag. These parameters are passed from RP(resource provisioning) to JS(job service) to us as key-value pair.2. Connectors :- Eventhub is data streaming platform that receive & process millions of data per second. Made changes to BBC-VHD code base to install the event hub connector & its dependency jars during VHD build. These connector enable parallelism for data transfer and thus improve performance. Also integrated other connectors like SQL, Postgres etc.3. Used Event hub to demonstrate long running spark streaming job over synapse cluster for 7 days. The job was producer/consumer one, each running on separate cluster. It was broken into smaller jobs each 1 hour long with watermarking the last data processed using Azure data flow (ADF) pipeline.4. Data Warehouse connector :- Worked on design and development of overwrite method for DW connector, including internal & external table(parquet & non-parquet).Internal table - I copied the synapse in-memory data to storage associated with cluster i..e ADLS gen2. Then used truncate & copy command to overwrite this data to DW table.External table - we got the location of actual data. Made a copy of it in ADLS gen2, logged it. Then put a lock, deleted existing data, wrote new data & released the lock.Finally cleaned the temp data.

* Worked on design and development of Samsung Smart TV Ads ecosystem which is fueled by user viewership data and runs on AWS cloud.* Worked on design and development of Campaign manager platform (which is a DSP), where an advertiser can put their ads (to be shown on Samsung smart TV) and check corresponding reports.* Worked on design and development of AD-server which serves most relevant ads to Samsung TV viewers in real time.* Worked on design and development of Tracker server, which records any ad events (like impressions, requests, servings, clicks etc) in a MySQL database table called hourly_stats.* Implemented Eden-Launcher, which is first screen tile AD and contributes nearly 40% of all ads shown. It involved writing REST APIs for Campaign Manager to support the CRUD operations of Eden Launcher ad, writing the AD-Server logic to support the Eden Launcher ad serving in real time to millions of Samsung smart TVs, made changes to Tracker server to record ad events for Eden Launcher and finally implemented the reporting side of it.* Implemented Targeting modules, which allowed advertisers to reach their specific audience. This included program targeting, channel targeting, language targeting, taste-graph targeting, Data segment targeting, location targeting, demography targeting, cross-device targeting.* Implemented Filtering module so that only the most relevant ad is served to the audience based on TV configuration, targeting set by advertiser and user viewership data.* Implemented pacer which paces up or down the priority of an AD, based on number of impressions and time left to be shown for this ad.* Implemented different product type like video ads including auto play video, Tune-in notification, Eden-sport, Eden-music* Implemented lots of background job, which run in a multi-threaded environment and cached the result to reduce the latency of Ad-serving.* Technology used :- Java, MySQL, MyBatis, NoSQL - Cassandra, AWS, GIT, REST APIs, Servlet etc

* Worked on design and development of Samsung Smart TV Ads ecosystem, which is fueled by user viewership data and runs on AWS cloud.* Worked on design and development of Campaign manager platform (which is a DSP), where an advertiser can put their ads (to be shown on Samsung smart TV) and check corresponding reports.* Worked on design and development of AD-server which serves most relevant ads to billions of Samsung TV viewers in real time.* Worked on design and development of Tracker server, which records any ad events (like impressions, requests, servings, clicks etc) in a MySQL database table called hourly_stats.* Implemented the backend of reporting pipeline on campaign manager platform, which allows the advertiser to pull report of their ads based on permutation & combination of various metrics & dimensions like date, AD type, AD-Component, impressions(billable, bonus, over served and out of bound ), clicks, requests etc.* Implemented various ad units like campaign, placement, creative, action and provided support for their CRUD operations by implementing corresponding REST APIs.* Implemented click to action feature, like click to web, click to video, click to microsite, click to APP etc* Implemented various monetization type like Reservation, Promotion, Road-Block, House ads etc.* Implemented Global Avail feature on Campaign Manager, which speaks of AD-volumes shown on a month to month basis by different countries and product types.* Worked on other features like business dashboard, What`s new, log emitter for failed ADs etc on campaign manager.* Technology used :- Java, MySQL, MyBatis, NoSQL - Cassandra, AWS, GIT, REST APIs, Servlet, etc

* My work was primarily related to big data analytics which involved machine learning algorithms like Random forest, k-mean clustering, k-nearest neighbors, logistic regression, singular value decomposition, frequent pattern growth, Monte carlo markov chain, hidden markov models, bayesian network, neural networks, naive bayes etc in a distributed environment of hadoop.* Implemented a semi-greedy algorithms for feature selection to create analytics platform called Blind Fold analytics. This involved using Random Forest algorithm to be run for n choose one (nc1) feature. That is we ran the Random Forest by taking one feature at a time and predicted the outcome. Then based on best performance we picked the top 50% of the feature as new n, we repeated it for n choose two (nc2) and so on. We repeated this till the performance graph flattened out, which gave our top k feature for predicting the model. Extrapolating it to business need, these were the top k features on which success of a product/business depends.* Technology used :- Python, Hadoop, Machine Learning Algorithms within Mahout (like Random Forest, K-Mean clustering, KNN, Neural Network etc), Natural Language Processing, MySQL, NoSQL - HBase etc.

* My work here was related to finding singular value in a multi-affine system. * Multi-Affine systems are linear & continuous in nature, which allows us to conclude between two opposite nature (positive and negative) of value for it`s parameter at two ends, there lies the zero for it in between the two ends.* Also, linearity of such system make us understand them as n-dimensional cube, given the system has n-variables.* So, by using binary search recursively, we can locate the solution of such systems in {epsilon}^n volume cube, where epsilon is a very very small value.* I modeled this system as generic one and coded to find the zeros of such systems in C++.

* My work at the mathematical institute of oxford was modelling of Ion-channels in our body, and to find the mean free escape time. * The ions like Sodium (Na+) or Potassium (K+) flows into our blood stream, and get into the cell`s mitochondria through these Ion-Channels like Sodium pump or Potassium pump.* Since the blood stream is extremely large than the pump or ion, we imagine it to be of spherical nature(taking inspiration from infinity concept) with a small opening (which is indicative of the pump).* Now, Inside this sphere, the ions are doing brownian motion and we want to find the mean escape time of ions, given that it is initially located at center of the sphere.* Also, for a more realistic model, the ion channels have negative charge, so they attract the sodium and potassium ions, thus also to include this extra electrostatic force apart from the normal brownian motion the ion is doing. * I modeled such biological systems, and coded to get the mean free escape time for 2-D (circle) and 3-D (Sphere) models in C++.