I work on the project of the largest retail company "X5 Retail" in RussiaWhat I did:• Accelerated the creation of reports by 75% by rewriting queries in SparkSQL, adding pre-aggregates and changing the table structure.• Accelerated Report Delivery to Clickhouse by 57%.• Optmised ETL processes which decreased the number of failures by 93.55%.• Developed new Airflow approaches and function to better work with Hive and HDFS metadata.• Conducted Technical interviews… Show more I work on the project of the largest retail company "X5 Retail" in RussiaWhat I did:• Accelerated the creation of reports by 75% by rewriting queries in SparkSQL, adding pre-aggregates and changing the table structure.• Accelerated Report Delivery to Clickhouse by 57%.• Optmised ETL processes which decreased the number of failures by 93.55%.• Developed new Airflow approaches and function to better work with Hive and HDFS metadata.• Conducted Technical interviews, Onboarded new employees, Lead Trainer for in-house Apache Airflow training.Technologies used:• Lang: Python;• ETL \ ELT: Hive, Spark Core, Spark SQL;• DB: Clickhouse;• Hadoop stack;• Hortonworks distribution (HDP);• Workflow manager: Airflow. Show less

What did:• Implemented HashiCorp Vault HA in BigData bank• Developed a CICD pipeline for testing, building and deploying Airflow DAGs.• Increased fault tolerance and speed of microservices from Docker to Openshift by 1260% by converting it from a single-thread to a multi-thread mode.• Created microservices in Java to exchange information between the front and backend of Data Science models. Technologies used:• Lang: Java;• Containerization: Docker… Show more What did:• Implemented HashiCorp Vault HA in BigData bank• Developed a CICD pipeline for testing, building and deploying Airflow DAGs.• Increased fault tolerance and speed of microservices from Docker to Openshift by 1260% by converting it from a single-thread to a multi-thread mode.• Created microservices in Java to exchange information between the front and backend of Data Science models. Technologies used:• Lang: Java;• Containerization: Docker, OpenShift;• CICD: TeamCity;• Messaging Queue: Kafka. Show less

What did:• Set up ETL for data delivery from 20+ sources, from scratch in under 6 months.• Conducted R&D projects to introduce new technologies to the team stack (GreenPlum, Airflow).• Created 30+ data marts for business clients• Maintained the health of the Hadoop cluster• Developed CICD pipelines to Hadoop servers from scratch.• Improved current Hive libraries, created new UDF functionsTechnologies used:• Lang: Python, Scala, Java, Shell;• ETL \… Show more What did:• Set up ETL for data delivery from 20+ sources, from scratch in under 6 months.• Conducted R&D projects to introduce new technologies to the team stack (GreenPlum, Airflow).• Created 30+ data marts for business clients• Maintained the health of the Hadoop cluster• Developed CICD pipelines to Hadoop servers from scratch.• Improved current Hive libraries, created new UDF functionsTechnologies used:• Lang: Python, Scala, Java, Shell;• ETL \ ELT: Hive, Spark (SQL), Streamline, NIFI;• Export \ Import of data: Sqoop, Spark, Nifi, Shell;• NoSQL: HBase;• Hadoop stack;• Hortonworks distribution (HDP / HDF);• Workflow manager: Oozie, Airflow. Show less

What did:- Reporting on fixing customer problems;- Working with documentation on the organization of communication networks of Novotelecom:- Monitoring the download and quality of channels in the IP network (Zabbix);IP network diagnostics (L2, L3) Dlink\Huawei\Eltex switches, Cisco routers;- Interaction with the divisions of the operation service of Zap-SibTranstelecom CJSC on the issues of setting up and operating the communication network equipment and eliminating… Show more What did:- Reporting on fixing customer problems;- Working with documentation on the organization of communication networks of Novotelecom:- Monitoring the download and quality of channels in the IP network (Zabbix);IP network diagnostics (L2, L3) Dlink\Huawei\Eltex switches, Cisco routers;- Interaction with the divisions of the operation service of Zap-SibTranstelecom CJSC on the issues of setting up and operating the communication network equipment and eliminating emergency situations;- Automate routine tasks using shell and python.Technologies used:- Lang: Python, Shell. Show less

What did:- Reporting on fixing customer problems;- Working with documentation on the organization of communication networks of West-Sib-Transtelecom:- Monitoring the download and quality of channels in the IP network (Cacti);IP network diagnostics (L2, L3) Dlink switches, Cisco routers;- Interaction with the divisions of the operation service of Zap-SibTranstelecom CJSC on the issues of setting up and operating the communication network equipment and eliminating emergency… Show more What did:- Reporting on fixing customer problems;- Working with documentation on the organization of communication networks of West-Sib-Transtelecom:- Monitoring the download and quality of channels in the IP network (Cacti);IP network diagnostics (L2, L3) Dlink switches, Cisco routers;- Interaction with the divisions of the operation service of Zap-SibTranstelecom CJSC on the issues of setting up and operating the communication network equipment and eliminating emergency situations.- Automate routine tasks using shell and pythonTechnologies used:- Lang: Python, Shell; Show less