Reported to Sr. VP of Engineering as a member of an agile team responsible for providing evaluation and post-sales support services directly to clients, and to Dtex sales, and sales engineering staff. Primary focus was in two areas:Internal Security AssessmentExecuted one-time evaluation, on-demand, and regular quarterly security audits using proprietary Dtex data collection and reporting software Performed deep security analysis on collected user audit data with emphasis on insider threat detection and data loss risks.Analyzed user activity and infrastructure audit data to produce operational level metrics, and identify efficiency gaps and bottlenecks via correlations of user behavior and performance against many operational and environmental factors.Created metrics and evidence based report for customer presentation of findings, and recommendations for corrective and proactive measuresManaged Support administrationActed as primary contact for select clients who had purchased Managed Support contractsMonitored security alerts and weekly activity reports on an ongoing basisInvestigated unusual detected activity, and notifying client with prepared incident reporting, with associated researchResponded to customer requests for specific research and information, or creation of custom reportsMet regularly with client technical contacts to review activity and open issuesProvided product training and update news to client contacts as needed, and upon requestAdditional Shell/Python development projects:Public Filesharing IP discovery crawlerCloud-Based Malware Database and collection engineCompromised Email Address discovery and reportingCommandline Elasticsearch interface for Dtex backend software

Reported to support and QA manager as technical tools, scripting, and automation expert, as well as providing business and after-hours support to clients and internal field engineers. Developed and supported internal tools for support, QA, and engineering teams. Built and maintained basic and custom monitoring solutions using Nagios core platform.Developed automated scripting to parse incoming diagnostic crash files sent by field and lab deployed security appliancesGenerates HTML formatted tabulated report, and emails to general distribution list, including additional CC addresses as determined by associated accountsIdentifies and categorized crash file according to source (internal vs. customer), cause (panic, manual, scheduled, error), Performs first level triage for failure-based events and preliminary diagnosisGenerates system/user information and health snapshotCommunicated with Salesforce (via SOAP) to display customer contact information, hardware inventory data, and identify customer account owners, and dedicated field engineersWrote command-line interface for Redmine ticketing systemDesigned for automated ticket creation and updates, to be integrated into Nagios monitoring alertsInteractive usage provide basic issue keywords search functionsDesigned custom database check monitoring for complex core-business functionsTemplate based system allows for simple creation of new checksModular design can be easily adapted to use existing scripts, or execute custom codeWrote hourly email report generation, identifying job statuses, and provides alerts and troubleshooting documentation for abnormal results.Documented end-to-end workflow diagrams for upcoming stream processing architectureDiagrammed using yEd software for OS XTied step-by-step event driven workflow to underlying systems, storage, and database infrastructure

Reported directly to VP of Operations as senior member of 8 person operations team. Responsible for systems, application, and network operations of enterprise scale, globally distributed, cloud-based proxy and security platform. Infrastructure consisted primarily of Open BSD systems running proprietary software, Apache, and Catalina.Identify, respond to, and resolve incidents using customized Nagios + Pagerduty + Jira environmentParticipate in rotating 24/7 on-call coverageScheduled and executed software upgrades, patches, and DNS updatesOrchestrated effort to upgrade 300+ global systems to BSD 8.4 with no customer impactCoordinated with vendors, customers, and internal infrastructure team for upgrades and maintenance of network and facilitiesBuilt, installed, and deployed new cloud servers on an aggressive growth scheduleWorked with infrastructure team to build out new datacenter sites on 6 continents Developed, documented, and trained team on operational process and proceduresScripted (Borne, Perl, Python) many activities (deployment, upgrades, and troubleshooting)Administrated Jira software, created custom new workflows, automation, metrics, reporting, and data fields for operations tasksDeveloped custom monitoring and reporting plugins for NagiosScripted back-end connectivity and automation between Jira, Salesforce, Pagerduty, Nagios, and other systemsWrote web interface for generating and operational statistics collected from Nagios, system health, and bandwidth usage, to predict deployment needs and identification of chronic and long-term trendsWrote command-line interfaces for:Automating proprietary software administration tasksQuerying and reporting from internal PSQL server/hardware inventory databaseMass application installation, deployment, and upgrade, including real-time status data exported to web UI for engineering and executive staffUser management (creation, modification, key management, secure access control via Yubikey)

Refining tasks to produce simple, repeatable, and/or automated workflows to be executed by the NOC or Ops-NAN supported self-service systemsGenerate and publish metrics and reporting of tasks to internal and external resources, and make them available via on demand automationInstallation, administration and support for departmental resources:CVS and SVN repositories for internal software, tools, libraries, and configuration dataPassword Manager Pro softwareSierra Wireless WAN modem utilities (AceNet, AceManager, etc) on multiple virtual Windows-based hostsDevelopment and support for internal tools (Borne shell script, Python, SQL, SOAP, expect, etc):Interactive BladeLogic-based command-line and web interfaces for accessing remote systems and field devicesBack-end systems utilizing WSDL/SOAP procedures, SQL, and automated HTTP access to identify and correlate device and system inventory data, used by other tools to provide seamless interfacing to usersInteractive or automated scripts for mass-scale auditing, reporting, applying, and correcting configuration of NAN device populations; developed to be iterative, persistent, modular, and environment/host agnosticCentralized command-line tool for generating, archiving, and deploying system and environment configuration files, and custom NAN tools and scripts to remote Unix hostsPush/Pull based synchronization system to maintain tool and configuration file deployments across dozens of disparate remote hosts, customers, and environments with centralized repositoriesAutomation for telnet-based WAN modem polling and administrationCentralized, web-based interface to monitor, execute, administrate, and report on active, large-scale NAN tasks across all environments. Provides performance metrics, reporting, alerting, and historical data archiving.Automated NAN device analysis suite: command-line, web, or Remedy based tool to collect and present detailed, historical and real-time data for field device

Worked within the corporate NOC facility to monitor, troubleshoot, maintain, and document globally deployed, large scale, field installed WAN/NAN infrastructures (100k – 5M+ devices), and associated web services and applications in 100+ environments for 30+ separate customers.Monitoring and TroubleshootingFrom system and network level to end-user experience: Cellular WAN/NAN devices and networks, Virtual and physical servers, logs and applications, web sites and services, customer experience, internal tools, systems, and networksIdentified problems and impact from multiple data sources: Nagios-based polling and agent alerts, Email and phone engagement, externally from customers, partners, vendors, hosting, and carriers; internally from support, engineering, and automated systemsIn-depth technical diagnosis, analysis, and problem solving for all of the above componentsMaintenanceCoordinated and executed scheduled, unscheduled, expedited, and emergency change requests following ITIL workflow practicesFirmware updates, configuration changes, and data polling for upwards to millions of distributed NAN devices.Unix system and application configuration changes, software updates, installations, migrations, and maintenanceOracle database retention and upgrade proceduresLarge scale execution, coordination, and testing of disaster recovery (DR) and fail over procedures, environment and server migrations, and other mass activities in coordination with customers, multiple internal organizations, and third partiesDocumentationCompiled change steps, problems and solutions into executable runbook procedures to communicate and share with team and organization peers.Worked with internal resources from engineering and support to maintain, revise, and update documentation libraries (Wiki, Sharepoint, Confluence)Ticket management: Creation, prioritization and triage, assignment, time tracking, statusing, internal/external updates, support/management escalation

Professional Services EngineerDelivered on-site, remote, and on-call services for all NetApp NAS and SAN enterprise storage and data management applications on a self managed, customer focused schedule at business and datacenter sites throughout Northern CaliforniaPerformed pre and post sales service functions, working directly with sales and the clients as facilitator, technical liaison and consultant, provided provisioning, architecture and deployment functionsProvided installation services to customers, including physical site (power, networking, cooling, space) evaluation, racking, network configuration, software installation, and decommissionAssisted customers with software and OS upgradesProvided a variety of supplemental services, including data migration, redundancy, data protection, automation, integration, disaster planning, customer education, and documentationMaintained on-call and after hours 24/7 emergency support for clientsMajor enterprise customers included: extensive on site support for Cisco and Apple, Microsoft, Tivo, eBay, Logitech, Rambus, Stanford University, Juniper Networks, and a variety of smaller scale clients

Performed day-to-day monitoring of 1000+ server enterprise environment, including web servers, java applications, ,search engines, and custom product indexers using Nagios softwareWorked with Engineering and QA to deploy and monitor new code to production and test environmentsCoordinated with management to develop a 24/7 schedule that complies with CA non-exempt overtime statutes.Worked with ProactiveNet sales team to install and deploy a comprehensive evaluation of their monitoring, trending, and reporting softwareDeveloped UNIX and Windows scripting to automate code deployment and other routine maintenance activities, reducing downtime due to operator error, and decreasing duration of said activities by 50%Developed custom monitoring and tools for Nagios, and assisted with scaling and maintenance of that software and it’s hardware platformsAssisted with major effort to document and implement Sarbanes-Oxley (SOX) compliance

 Transitioned from role as NOC team member 08/2004 Managed process, procedure and documentation for NOC operations Acted as liaison between remote technical teams for implementing new technologies and process Developed training and efficiency improvement planning Offered support and enhancement for NOC tools Provided oversight of new client deployment and launch activities Oversaw day-to-day NOC incidents and outage management Orchestrated major NOC facilities move from Sunnyvale, CA to Plano, TX. Developed training material for new NOC employees Worked with IT to deploy Hyperion reporting suite  Contributed to a major effort to document migration timeline for shifting to a new agile business model Lead initiative to develop and deliver recommendations for implementing a full-time Process Engineering team Consulted with Reporting team to help provide and interprets metrics for operational and business data

Worked in operation center environment to monitor international network of over 1000 Windows and Unix servers and network devices across 6 datacenter locations for over 50 customersUsed Netcool with ISM web and TCP functionality, Sysedge, Keynote, custom URL transaction monitoring, and native SNMP to maintain total awareness of operational network infrastructure, databases, applications and web content.Supported monitoring engineers in creating, configuring and maintaining real-time server diagnostics using Netcool's ISM and Sysedge toolsOffered first level support for all of the above using established and continuously updated procedures and documentation to provide customers with immediate resolutions to issues as they were detected or found to be imminentProvided in depth support for web servers and back-end applications, including system level support and log troubleshooting for Apache, IIS, ATG and WeblogicAssisted 2nd and 3rd tier support members with technical issues that required escalation outside of the operation centerActed as 1st tier support contact for customers via phone, email, and Remedy system, creating tickets for support and changes initiated externallyMaintained ownership of customer facing tickets throughout the life of an issue, ensuring frequent updates to both the customer and internal management, engaged proper escalation as time and situations warranted, and provided detailed notes and instructions to preserve integrity of operation center issue management capabilitiesPerformed and assisted with code pushes and releases to customer staging and production serversAuthored, revised, and appended procedural and technical documentation libraries to account for changes to the operations environment and to improve workflow efficiency

Transitioned to NOC systems and monitoring administration Apr. 2001Built and administrated 12 NT, UNIX, and Redhat machinesManaged monitoring server-side applications on above machines (SiteScope and Big Brother)Installed and maintained Big Brother client on production servers (500+)Acted as NOC liaison to inform of new features and capabilities, received feedback to improve monitoring interfaceCreated and maintained centralized documentation of server inventory and monitoring proceduresEvaluated new products to improve NOC’s ability to monitor problems with sites and serversCreated scripting for NOC to automate escalation to engineeringServed as liaison between west coast brands, including WhoWhere, Matchmaker, and Quote.com to provide monitoring coverage of all aspects of applications, servers, sites, and escalationWorked with each team as individual projects arose to ensure monitoring was fundamental and functional part of their frameworks at the onsetSeamlessly migrated monitoring server farm between data centers as part of major facility moveWorked additional time to provide support as acting PSA for Quote.com: performed weekly maintenance of NT machines, participated in weekly 24/7 on-call rotation, built, configured, racked, and wired IBM and Sun machines and cabinets as part of ongoing datacenter move.

Assisted with building NOC infrastructure, writing documentation and procedures, running and staffing NOC. Additional sysadmin responsibilities included shell scripting and automation, building NT 4.0 QA domain, desktop support.