Charles Kelley Email and Phone Number

ML'S LAST MILE PROBLEM IS A SILENT KILLER. The race is on at many companies to develop great ML models, but most models never cross the finish line to capture business value.As a lead scientist in the AI Center of Excellence, I specialized in MLOps and building end-to-end applications that enhance the ML model lifecycle—drastically reducing the time from model planning to business impact.▪ VALUE LIFT - The easiest way to ensure ML projects deliver value is to always measure it. To enable this at Schwab, I created a full-stack Python application, accessible via CLI or FastAPI interface, that lets data scientists easily measure the incremental lift in financial and KPI metrics from ML models or any other ML driven business initiative that directly impacts customers.▪ FEATURE ENGINEERING - Schwab's enterprise data teams focused on operations data, largely neglecting ML use cases. To fill the gap, I developed an ML feature store using dbt on BigQuery to transform operations data into an analytical data mart that feeds a Feast access layer. Scientists can easily access thousands of proven ML model features or safely engineer their own.▪ MLOPS INFRASTRUCTURE - I was fortunate to lead an agile team of four ML engineers through architecture and development of an MLOps system using Python powered open source tooling. We used Parquet, Hive, Spark, and Dask to build out a petabyte-scale Inmon model data warehouse under a Python and Scala compute layer on a multi-petabyte Hadoop cluster. For ML model tracking, registry, and deployment we used MLFlow hosted via Cloud Foundry. DevOps was managed using python packages to encapsulate all application management runtime code so that the entire system could be updated and deployed using an internal package and container repository and any CI/CD tool (ProGet and Bamboo in our case). Lastly, all system components were documented using Sphinx or MkDocs and served via a single website for one-stop user discovery and reference.

OUR VP CALLED ME THE SWISS ARMY KNIFE because of my multifunctional skill set and track record delivering exceptional results across many analytics roles.As a senior member of the Retail Analytics team, I worked with leaders from all retail client facing lines of business providing statistical modeling and advanced analytics. There was always a huge backlog of analytical project requests addressing different products, client segments, and technical needs.The situation paired perfectly with one of my key strengths—solving complex problems by learning new technical skills and collaborating with business partners to apply them to deliver value.▪ CAUSAL INFERENCE - The corporate strategy team needed to show the incremental value of a new product roll out. I created a matched-pairs experimental design to select similar test and control clients while controlling for demographic and geographic factors. Then, I created a similarity matching model in scikit-learn wrapped with a PowerShell GUI the business operations team could use to run the test and control group management and value measurement processes independently.▪ DATA INFRASTRUCTURE - Schwab was launching its highest profile product in years and needed real-time analytics to manage the client experience. But, there was no analytical data available. So, I developed a data mart in Teradata using the Kimball data model, complete with source system ETL, to support customer journey reporting. With the data in place, I built a Tableau dashboard—beloved by business leadership—which subsequently became a best practice template for visualizing new product adoption.▪ CLUTCH ADAPTABILITY - One-time, a billing error impacted thousands of customers and Schwab's billing system was unable to correct the issue after the fact. It was a major problem, but I had a fix. I created a Python application that applied complex billing algorithms to any customer set retroactively to avoid costly per client manual calculations.

PICKING UP PENNIES IN FRONT OF A STEAMROLLER is often negatively associated with taking big risks for little reward, but for many banks it's the core business model.At Discover Bank I focused on forecasting bank deposit acquisition (marketing), and economic impacts on deposits (time-series analysis)—ensuring we always knew where the steamroller was. This let the treasury team work their magic, picking up pennies by placing excess deposits into investments that earned a premium to the yield given to depositors.▪ MARKETING MIX AND ATTRIBUTION - Discover's marketing forecast techniques were great but the data sourcing, model development, and forecast reporting processes were fragmented across systems and teams making forecast runs slow and prone to error. In response, I developed a modular pipeline where each forecast subprocesses was an idempotent stage run independently. Data was sourced via SQL stored procedures, marketing attribution and mix models were refactored into separate SAS programs, and the user interface was an Excel workbook with VBA macros that called the data update and modeling steps then aggregated the forecast in the Excel workbook for non-technical marketing analysts. The workflow reduced human error and forecast turnover time so much that the process supported Monte Carlo simulation to create forecast output confidence intervals.▪ TIME SERIES ANALYSIS - To get great time series forecasts for retail credit product portfolios, you must account for factors affecting customers along two dimensions of time. I built a dual-time dynamics model in R to solve for this. First, a parametric regression model using exogenous (macroeconomic) variables that affect all customers was built. Second, a non-parametric model of the error residuals from the first was built using endogenous (customer quality) variables specific to customer cohort vintages. The result was a much more accurate model than the traditional one dimensional time-series model it replaced.

ALL MODELS ARE WRONG BUT MANY MODELS ARE USEFUL. This slight variation of George Box's famous quote is a great heuristic for describing the corporate finance department at most banks.I worked as a financial modeling expert in the corporate finance department at Discover Bank. There was a model for everything, sometimes multiple models. None of them were perfect, but every one of them played a role in effectively managing the bank's balance sheet.▪ ENSEMBLE MODELING - Just prior to my arrival, Discover launched a new retail checking product that was growing exponentially and needed a better checking deposits forecasting model for adequate management. Quickly growing metrics are notoriously difficult to accurately forecast. So, rather than rely on a single model that had proven to be very inaccurate, I created an ensemble model by averaging a multi-variate regression model, a basic time-series trend model, and a marketing driven net acquisition model. The resulting model was much more robust against large errors in any individual assumptions—providing the stability necessary for senior business leadership trust the forecast.▪ SCENARIO ANALYSIS - Banks with more than $10 billion in deposits are subject to the Federal Reserve's Comprehensive Capital Analysis and Review (CCAR) or stress-test regime—a scenario analysis exercise where bank capital is forecast under many combinations of macroeconomic factors. Discover had ARMA and ARIMA capital forecast models to complete the CCAR requirements but each scenario was done as a manual model run with input variations and data coded by hand. I created a new self-updating data pipeline in the Teradata database, a macroeconomic scenario input API in Excel, and refactored the autoregressive models' SAS code into advanced SAS macros to run iteratively over any set of scenario input combinations without upstream data retrieval. The new process was an order of magnitude faster and had better risk scores than the one it replaced.

Streamlined working capital management, budget planning, and expense forecasting workflows and presented monthly financial results/forecasts to senior executives on an on-going basis.

Evaluated the biotech and pharmaceutical company landscape to identify strategies that consistently lead to above average ROI and assess their application in Takeda’s global reorganization.

Worked with company executives to develop and implement digital marketing strategies and develop web analytics frameworks. Created business plans, management presentations, and private placement memorandums to support multiple funding series

Led projects to implement lean production practices (Six Sigma, 5S) which resulted in a 15% increase in production efficiency in the manufacture of tablet and capsule pharmaceuticals