Modern Storages and Data WarehousingFTDA, FSC HSELectures and practice classes

I have managed a department of 5 people. The department specialises in providing stable work of the general infrastructure for the Analytics Direction at OZON Marketplace, developing parsers of data on competitors and organising a data warehouse. I was also a supervisor for 20 data analysts in other groups within the direction. I was responsible for all communications with DE/development/infra.Infrastructural tasks:* Moving the analytical server from Yandex Cloud Platform to OZON's DC - configuring and administering the VM, transferring code dependencies to the internal package storage, coordinating scenarios for working with external services with information security* Moving from Slack to Mattermost - auditing bots/integrations/processes, transferring them and ensuring a seamless transition to a new service* Moving from Tableau to Superset - auditing reports, setting up a pipeline for transferring data from analytical environments to Clickhouse, familiarising analysts with new instruments.* Rewriting ETL processes from using DataLake data to using DWH data* Shared code support and developing new functionality for the internal python libraryPersonal projects:* Splitting OMS (Order Management System) monolith into DWH entities and leaving DataLake - leading the project. I have worked on developing requirements and terms of reference, monitoring the development process and developing the data quality requirements and their integration with the data quality checking platform — organised seamless switching of 600+ analytical tools to new data sources and training sessions on working with DWH and designing data marts. Massive project lasting nine months* Designing and supporting ETL for key metrics - Ontime and Buyer Defect Rate* Code review and code optimisation of department employeesHiring and training:* Selection, interview and hiring of data analysts, data engineers and interns* Conducting training events and lectures

Building a workflow in a team from scratch:* Agile - organizing sprints, stand-ups and retros, controlling the processes of setting, developing and completing tasks, communicating with customers* Organizing task control in Jira, employee training* Git - employee training and organization of work with the team's shared resourcesCode culture and organization of work with data:* Maintaining code culture* Organization of data quality checks over input and output data* Creating alerts for data gaps* Carrying out code reviews* Organization of duty to support the work of critical processes during non-working hoursInfrastructure support:* Infrastructure support (Linux server/Apache Airflow)* Deployment, administrating and support of AirFlow and Operations Server* Passing a SOX audit, organizing the work of employees with the team's shared secrets* Organization of access to common analytical databases and data sources (Hadoop, Vertica, production services and master systems)* Data warehouse organization (Vertica/SQL Server/Hadoop)Hiring and training employees:* Conducting interviews* Team expansion from 8 to 17 data analysts* Recruitment, training and development of interns - interns have successfully adapted to the team and, after the end of the internship, independently lead projects and even take leadership positions* Conducting training events on working with DWH, Git, AirFlow, Hadoop and othersPersonal projects:* Supporting core data marts and Legacy-code from other teams* Designing and supporting team's key ETL processes

Competitor analysis:* Monitoring the competitors' last mile network - openings of new self-pickup points and benchmarking* Data parsing (web pages and mobile APIs) - item catalogues and prices of competitors, data on sellers on competing sites* Matching - matching product cards of OZON and competitors, matching sellers, searching for gaps in the product matrix, data to attract new sellers on OZON, matching product categories* Pricing analytics - assortment and GMV of competitors, GMV of sellers on OZON and competitor sites, search for gaps in the product matrix* Creating solutions for data parsing - creating MVP solutions for data parsing, building an alerting and monitoring systemTechnology stack:* Python (pandas, numpy, requests, asyncio, Scrapy)* Golang - development of data parsers* Headless browsers* MITM* Mobile proxies* SQL: PostgreSQL, Vertica, T-SQL* Hadoop* Docker* Scrapinghub* Apache AirflowOZON.OpenData:Developer and product-owner (concept, development, content, support, product and operational analytics)Developed and supported the productSpeaker at sales seminars and OZON Com-e OnTechnology stack:* Python (Django/Flask)* Nginx* Apache Airflow* SQL: PostgreSQL, Clickhouse* Yandex.Metrica

Geo Analytics:* Collection of metrics about the efficiency of the infrastructure - various indicators were collected (for example, Ontime, Click2Delivery, Delayed postings, GMV, IPP / IPO, number of orders) in sections to different levels of geography, logistics sections and infrastructure facilities.* Analysis of the distribution of orders/infrastructure - coverage of the population with the OZON infrastructure, parsing the infrastructure and delivery times of competitors, building benchmarking with OZON competitors* Designing data marts, building ETL processes - collecting wide data marts with data on the network of pickup points/postomats* Systematization of knowledge about own and agency network. Collection, documentation and validation of data sources.* Analysis of good points for the placement of pickup points. Parsing possible new locations with CIAN / Avito. Evaluation of received locations* Preparation of interactive reports with automatic updates. Tableau/Carto/PowerBI* Creation of tools for network development teamsTechnology stack:* Python (pandas, numpy, shapely, requests)* QGIS* Carto - administration, support, ETL design* SQL: T-SQL/PostgreSQL* Apache Airflow/crontabs* git* Integration with Confluence, JIRA, GSuite, Slack, Workplace