Spark, Redshift, Python, Tableau, AWS Lambdas etc Data pipelines and ETL. KPI metrics and reporting via Redshift and TableauBig data investigation, analysis, machine learning and prototyping via Spark. Fuzzy entity matching via NLP, TF IDF Cosine similarity etc

• ETL, data warehousing & data mining with Redshift, Spark and S3• Built various ETL processes in Python & Redshift, automated via AWS Lambdas or Jenkins, including analytics of measured and inferred usage, alerting and performance graphing.• Pipelining data from Adobe Analytics & application event logs through Redshift for aggregations and adhoc querying through to Tableau for display of KPI dashboards • Built a Python framework for creating, packaging and deploying new Lambdas based on YAML config files. Also a web UI to present success / failure of lambdas, last execution time, log output etc in a single interface via various AWS api queries to Lambdas, Cloudwatch, Cloudwatch Metrics etc. • Architected, built and optimized Redshift implementation for Mendeley Analytics• Adhoc analyses via Spark on Databricks or Redshift Spectrum• Created a Data quality monitoring web UI to aid in anomaly detection, application failures etc• Built a Python-based abstraction layer for Mysql, Postgres, Redshift connections to handle credentials and interfacing with Pandas. Enabled analysts and data scientists more easily access various data sources by providing a common interface• Custom reporting and analytics in Python with Pandas

Development of new products (Perl, MySQL) Support of client services and external customers, integration of new clients (Perl)Produced quality code in a continuous release environment - multiple releases per week, regression testsAcquisition of domain-specific knowledge - learning the business side of securities lending

Team Lead for Data Team. Warehousing of terabyte-scale data, generating and enriching user profiles, exposing data through APIs-Architected and developed real time log processing (Storm running Python bolts via Petrel). -Created a recommendation engine and realtime API to present most promising product-related keywords for a given user (Python)-Implemented an intermediate storage layer (HBase) between long term storage (S3, MapReduce) and fast storage (MySQL, Redis)

Summary: Created Tapinteract (http://tapinteract.com/): a QR-code + NFC tag tracking platform with my fellow Director / Co-Founder. -Built Python backend with setup / control data stored in SimpleDB, stats processed and saved -using Kestrel queue and MySQL -Created Paypal IPN listener + processor in Python with bottle. Implemented a process for transparent subscription upgrades, circumventing Paypal’s 20% change limit.-Worked on Webservice layer / API / ORM in Python with bottle.-AWS configuration of EC2, S3, SES, Cloudfront etc, creating live and staging environments-Created a simple URL shortener in Python + SimpleDB-Helped to construct UI using Bootstrap, Backbone + jQuery

-Prototyped various small to medium projects using Python and AWS

1 month in India3 months in Japan1 month in Nepal

Summary: Managed UK engineering department, architected and advised on the largest projects-Managed all UK engineering (22 people), reporting directly to the CTO in the US -Provided consultation to the product team on possible features and timeframes-Advised on and worked with dev team leads to architect the largest scale projects -Coordinated transatlantic projects with dev teams in the UK and US-Worked on the design and implementation of a SOX-compliant release process-Produced a new career progression structure for developers

Summary: led development on the adserver and data warehouse; led a team focussed on improving QA, UAT, dev best practice and tools; dealt with internal and external clients-Built, tested and profiled a prototype hadoop + hive cluster (6 datanodes) which went on to form the basis of the data warehouse-Scrum master for ‘Scale’ development team, working to scale the performance of the core adserver and integrate the SM and Adviva technology stacks (Perl, Java)-Mentored new team members and junior developers-Introduced and standardized the use of a set of developer tools: JIRA, Crucible, Hudson, Nexus, Confluence-Designed & implemented a release process which ensured suitable code review, QA, UAT and business sign-off was in place prior to code going live-Consulted with internal clients on bugs and feature requests-Monitored external adserver performance with a combination of Gomez and in-house software. -Worked with external clients who reported performance-related and other technical issues, to resolve their problems in an appropriate timeframe -Produced detailed investigations into performance bottlenecks on the overall scalability of the adserving system

Summary: built and assisted with the architecting of some of the largest dev projects for the adserver, led Corp and Enterprise IT team-Architected and built a new adserver transaction processing system (Perl) which read realtime logs from the core adserver and generated aggregated financial and other statistics for business users. -Made substantial improvements to scalability and speed by creating a system which: --removed a layer of MySQL databases in favour of text files--could be run on multiple application servers in parallel--reduced the volume of inserts / updates on the main financial database--integrated Nagios monitoring in all key components-Created a distributed user profile database for the core adserver (Perl, MySQL), which removed a significant performance bottleneck-Oversaw large growth across two datacenters, which involved deploying a large volume of servers and network hardware, increasing monitoring coverage and setting up an on-call rota

Summary: dev work on both the front end UI and back end adserver, MySQL admin and tuning, server build automation, network hardware setup and deployment-Extended and overhauled the internal UI (PHP)-Refactored significant parts of the core adserver (Perl)-Created a custom monitoring framework (Perl) to complement Nagios with metric vs metric graphs to calculate application / server performance capacities and limits-Performed significant Apache, mod_perl, MySQL performance tuning-Set up MySQL master-slave, master-master replication, scripted and scheduled backups and managed general MySQL administration-Implemented server-build automation via PXE and Kickstart-Managed Netscreen (Juniper) firewalls and Foundry Loadbalancers-Built a redundant pair of Nagios monitoring servers and wrote several plugins (Perl) to monitor services and record performance data-Helped to manage the build out of a second datacenter for adserving and a new simpler and more resilient network architecture

-Extended internal UI (PHP). -Worked on overhauling some lesser backend processes (Perl)-Created Active Directory Group Policies, login scripts -Windows file server setup, desktop support