Experience

Scripps Networks Interactive // ULive

Lead Cloud Operations/DevOps Engineer • Nov 2015 — Jan 2017

I joined ULive in order to transition and take lead ownership of DevOps & CloudOps from the VP of Engineering.

  • Acted as owner and primary point-of-contact for all infrastructure & operations issues
  • Responsible for systems uptime, alerting (PagerDuty), monitoring/metrics/log aggregation (DataDog/ELK), and all escalated systems incidents
  • Participated in weekly “Tech Leads Council,” reviewing and improving cross-functional DevOps cohesion and process
  • Led weekly Ops/DevOps retrospectives
  • Ran daily engineering standup meetings
  • Led/participated in design, architecture, and implementation surrounding all major engineering initiatives across all engineering teams
  • Worked with product managers of each respective team to understand requirements, ensure timely delivery of requisite systems/infrastructure, and meet release deadlines
  • Groomed, prioritized, and maintained Operations agile board
  • Led and built proof-of-concept initiatives for building out or prospecting new technologies, including scoping and timeboxing said initiatives to maintain critical work throughput
  • Built, tested, documented, and delivered infrastructure, toolchains, and pipelines to support data engineers, developers, and product managers
  • Identified and paid down significant technical debt across the infrastructure and disparate systems
  • Test-driven Chef (and infrastructure) development; authored a substantial body of cookbooks & systems automation
  • Infrastructure automation tooling (primarily written in Ruby)
  • Hacked and rolled Ansible into an AWS Lambda listener service, to operate as an event-driven orchestration/automation API
  • Spearheaded investigation and resolution of a bug our data engineers discovered in Amazon’s EMR (Hadoop) offering by coordinating efforts between our data engineering team and Amazon’s support engineers
  • Exposed a programmatic/scripted point of entry to hook our own proprietary automation into Amazon’s EMR provisioning

Doctorsoft Corporation

Lead DevOps Engineer • February 2015 — November 2015

Senior Technical Consultant • November 2015 — November 2016

I rejoined Doctorsoft in order to help them scale out and revamp their systems and operations during a period where they doubled their customer base. I maintained an advisory/senior consultancy with them once that objective was accomplished.

  • 24/7 on-call technical lead
  • Worked directly with CEO to coordinate development, operations, QA, and customer support in meeting critical business objectives and deadlines
  • Evolutionary Database Design basics/safe refactoring (Liquibase)
  • Introduced Scrum & Kanban practices to harmonize resource constraints and business objectives with maintenance release and feature request delivery in timely, predictable iteration cycles
  • Blocked strict waterfall approaches to development cycles
  • Scoped and paid down tech debt
  • Managed JIRA cases, groomed agile boards, and regulated sprint deadlines for development, QA, and Ops teams
  • Specced, built, tuned, and migrated production systems away from RDS to a highly available MariaDB cluster with synchronous replication and automated backups
  • Implemented Zabbix for monitoring, instrumenting relevant services for all hosts throughout the infrastructure
  • Built dashboards to count meaningful application, user, and business metrics
  • Authored Ansible repository to begin the process of creating a self-documenting configuration as code
  • Authored Terraform repository to create immutable, self-documenting, stateful infrastructure as code
  • Migrated the entire infrastructure into a multi-tiered VPC with zero downtime, retiring older, more expensive systems
  • Led Ops & Development in upgrades & patches for production software stack and java libraries shipped with the application

Connexity, Inc.

DevOps Engineer II • May 2014 — February 2015

I joined Connexity as their first DevOps hire in order to help them establish DevOps practices, automation, and useful development/configuration/deployment toolchains.

  • Led scaling efforts by collaborating with software engineering team to double the size of the production infrastructure (webheads, firewalls, load balancers, back-end & big data infrastructure) before/during their highest recorded revenue quarter
  • Maintained, scaled, and administered highly available front-end & back-end web and big-data systems, serving over 200K QPS in a very diverse and technologically colorful stack - Hadoop/HDFS, MapReduce/YARN, Hive, Nginx, Redis, ZooKeeper, Linux, PFsense, Ruby, Scala
  • Contributed extensively to company wiki/documentation
  • Generated weekly monitoring reports to reduce unactionable alerts and false-positives, thereby increasing actionable alerts and cutting down total alerting noise by over 80%
  • Utilized Ansible to manage release process, A/B testing, and fault tolerance/rollback during deployments
  • Designed and proposed application & infrastructure management and deployment pipelines using Rundeck, Ansible, Chef, and Jenkins, with a focus on the concept of infrastructure as code & deployment automation

InVitae

Compute Platform Engineer • September 2013 — December 2013

I joined Invitae with the hope of participating in an exciting early-stage startup.

  • Responsible for day to day IT & systems operations
  • Administered a large, auto-scaled AWS infrastructure (primarily Linux)
  • On call for systems uptime & emergencies
  • Collaborated with software developers and engineers to “Puppetize” various elements within the infrastructure & software stack
  • Specced, built, and delivered necessary development and production environments to various teams across the organization (data warehouse engineers, Python engineers, web developers, lab automation engineers)
  • Helped engineers and developers troubleshoot and remediate troublesome environments or application failures

Doctorsoft Corporation

DevOps Administrator • May 2012 — December 2012

Lead Database & Systems Engineer • December 2012 — May 2014

Doctorsoft was looking for a SysAdmin to help them scale and achieve production readiness. I quickly began to collaborate with both the development team and management, adapting to occupy a unique and challenging role.

  • 24/7/365 on-call technical lead
  • Transitioned infrastructure & operations from startup/development phase into production operations; this included rebuilding infrastructure, tuning the full production stack (front-end/back-end) and instating Ops policies/procedures
  • Conceived and lead the concept/design/spec for HIPAA compliant, RESTful connector enabling interoperability & integration of billing and patient health information between Doctorsoft, clients, and 3rd party practice management software; managed developers to get it built and released by deadline
  • Prepared official company Information Security Policy manual; consulted management on compliance, risk analysis, and disaster recovery
  • Implemented centralized logging for correlation of application failures to system events and metrics
  • Led developers in identifying and remedying connection leaks, memory leaks, and performance hindrances
  • Worked directly with management to formulate customer support policy and procedure
  • Worked directly with external partners and clients to integrate 3rd party services
  • Acted as point of contact between management, clients, and software developers on customer software deployments and feature requests

Alchemy Communications

NOC Technician/Systems Administrator • June 2010 — July 2012

I began my career at Alchemy as a NOC Technician with basic monitoring, security, server installation, and network troubleshooting duties. I grew to assume the role of SysAdmin, helping manage our core/flagship NOC.

  • Systems Administration (primarily Linux, some Windows Server/Windows SQL)
  • “Go-Live” preparation and operational readiness for new customers
  • SAS 70 Compliance/PCI Auditing and procedural documentation
  • High-availability best practices: architecture, design, troubleshooting, monitoring, alerting, uptime
  • Information systems security and forensics (network & server-side)
  • Troubleshooting and resolving network & systems failures in high-pressure production
  • Disaster recovery & post-disaster data recovery
  • NOC policy/procedure revision & wiki documentation
  • Managing relations between technical management & customers

Courses & Certifications

HackReactor

Software Engineering Immersive (Onsite - Los Angeles) • January 2017

"The Hack Reactor immersive is focused on building autonomous, fully capable software engineers."

Reference link: Curriculum Overview

BigNerdRanch

Ruby On The Server • July 2016

"Learn how to create Rack middleware, Sinatra applications and complete Ruby on Rails applications."

Reference link: Curriculum Overview

Amazon Web Services

AWS Certified SysOps Administrator - Associate • December 2016

Pending examination June 2017

Reference link: Curriculum Overview



AWS Certified DevOps Engineer - Professional • December 2016

Pending examination July 2017

Reference link: Curriculum Overview

Projects

Lambsible

Lambsible is an AWS Lambda-based implementation (albeit hacky) of Ansible. It is guided by 'serverless' principles and is a proof-of-concept exploration of a push/API/event-driven model of exposing and utilizing the Ansible API - inspired by and based on a post by José Coelho.

Source material: Ansible in AWS Lambda

xtendEMR

xtendEMR is a simple toolkit for hooking into EMR's bootstrap process in order to include custom scripts, configurations, and packages without having to bootstrap your own Hadoop cluster. Initially, it was intended for addressing a bug in the EMR stack that our organization identified for AWS. AWS later patched the bug based on our work with their support engineers and architects. By then, this little toolkit had taken on a life of its own, so we continued to use it.

Cvrcle.io

An "MVP-first" project spanning all of one week, Cvrcle was an app my team and I built as an exploration of crowd-sourced trip and travel itineraries. Although we all owned full-stack features, I spent much of my time building out the back-end/API, the relational data model, and all of the object-relational mapping and validation. I also implemented a simple asynchronous authentication scheme by using Auth0 with Facebook sign-on, and managed the users's profile/session/authentication statefully with Redux & some simple thunks. Core functionality has been implemented, and we are currently waiting for FaceBook's approval before we can implement the final piece, which requires a subset of specific permissions for the Graph API.

Stack: React, Redux, ObjectionJS + Knex, Node+Express, Semantic-UI, Amazon RDS, Terraform, Ansible

Stelth.io (In Development) - "Digital dead-drops made dead easy."

For our HackReactor "thesis" project, we have decided to build a mobile-first web application that ties Augmented Reality, digital information exchange, and geolocation together. It is comprised of our own API, an online web client for creating and tracking digital dead-drops, and a native iOS app with some simple A.R. for finding and retreiving "Stelth-drops."

Stack & Tooling: React, ImmutableJS, React-Native, React-Native-Motion-Manager, Redux, Redux Sagas/Observables, XCode (for building & deploying the mobile app), ObjectionJS + Knex, Node+Express, Semantic-UI, Amazon RDS, Circle-CI, Terraform, Ansible

Skills

Technical

AWS - Route53 DNS, EC2, Elastic BeanStalk, Elastic Load Balancing, RDS, ElastiCache, CloudFormation, VPC, S3, IAM, Glacier, Elastic MapReduce, RedShift, Lambda, Kinesis (FireHose), SNS, SQS, API Gateway

Scripting/Langues - BASH, Ruby, JavaScript

Infrastructure/configuration as code – Chef, Ansible, Terraform, Ruby SparkleFormation

Centralized logging - GrayLog2, Elasticsearch + Logstash + Kibana, rsyslogd

Monitoring & Metrics – DataDog, LogicMonitor, Zabbix, Nagios, Zenoss, Beats (formerly PacketBeat), Splunk, PagerDuty, AirBreak

Continuous Integration & Deployment – RunDeck, Travis-CI + Docker, Cucumber, Jenkins, Heaven, Capistrano

Databases - MySQL/MariaDB/Percona XtraDB, MongoDB, ElasticSearch, Hadoop ecosystem

Java - Performance troubleshooting, JVM/garbage collection tuning, connection pooling, session persistence, Hibernate/ORM

Soft Skills/Concepts I'm a Fan of

  • “Servant Leadership”
  • DevOps as an organizational aptitude & movement
  • Agile principles & their methodological variants - scrum, kaizen, kanban, scrumfall, scrumban, etc.
  • “The Three Ways (of DevOps)”
  • Project management & separations of concerns
  • Operational agility
  • Build-or-buy analyses
  • High-value metric identification & analysis
  • Managed risk strategy & tactics
  • Eliyahu Goldratt’s theory of constraints as applied to lean production
  • OODA Loops/Boyd Cycles
  • Boyd’s axiom of time-based competition

Recognition

Boy Scouts of America

Eagle Scout Award • 2005

The Eagle is the highest recognition that Scouting offers to Scouts. It is earned through the advancement program, and only a small percentage of boys who begin in Scouting receive this honor. The wearer of the Eagle award is the epitome of Scouting's best efforts, beliefs, and leadership.

Additional Links

  • CtrlAltDevOps.com - Ramblings, musings, and useful DevOps nuggets, authored by yours truly

Reading List (2016/2017)