2013-11-26 Tuesday - Deployment Optimization

A nice little nugget of a problem was handed to me today: identify ways to help an operations team reduce their system maintenance / deployment window [for production system updates] that has somehow grown to require a xx-hour window, and achieve zero downtime (or as close as possible).

The environemnt is complicated in the extreme: highly regulated industry, compliance requirements, clustered servers, high availability, PCI security zones, 3rd party software/service providers, cloud service providers/integrations (SaaS and PaaS), frequent commercial software upgrades/patches, vendor constraints on database schema changes, disaster recovery dependencies, a legion of upstream and downstream data integration dependencies.

For the last year I've been carefully planting seeds of certain ideas in various conversations with key stakeholders within an organization - to begin the gradual introduction of concepts and practices such as DevOps, Continuous Deployment, and Continuous Operations. Now that a sufficient level of pain has been experienced, there is a broad consensus and acceptance that there needs to be change.

"He was not in a hurry, 'hurry' being one human concept he had failed to grok at all. He was sensitively aware of the key importance of correct timing in all acts — but with the Martian approach: correct timing was accomplished by waiting."
- Stranger in a Strange Land, by Robert E. Heinlein

I have some ideas, but as a good researcher, first order of business is to review current directions, trends, peer articles.  This posting will be a place for me to share some of the information that may be of interest to others:

Zero Downtime, Instant Deployment and Rollback
http://www.ebaytechblog.com/2013/11/21/zero-downtime-instant-deployment-and-rollback/

Jevgeni Kabanov (ZeroTurnaround)
Pragmatic Continuous Delivery, at W-JAX 2012
http://vimeo.com/79959315

Continuous Operations for Zero Downtime Deployments
http://www.virtualizationpractice.com/continuous-operations-for-zero-downtime-deployments-22680/

The Virtualization Practice
http://www.virtualizationpractice.com/

Deploying the Netflix API
http://techblog.netflix.com/2013/08/deploying-netflix-api.html

http://slideshare.net/netflix

http://netflix.github.com 

http://github.com/netflix

Cloud Architecture Tutorial
Constructing Cloud Architecture the Netflix Way
Gluecon May 23rd, 2012, by Adrian Cockroft
http://www.slideshare.net/adrianco/netflix-architecture-tutorial-at-gluecon

Cassandra in the Netflix Architecture, Denis Sheahan
CassandraEU London March 28th, 2012
http://www.slideshare.net/acunu/cassandra-eu-2012-netflixs-cassandra-architecture-and-open-source-efforts

Patterns for Continuous Delivery, Reactive, High Availability, DevOps and Cloud Native Open Source with Netflix OSS
Adrian Cockroft + Ben Christensen, YOW! Workshop Dec'2013
https://speakerdeck.com/adrianco/patterns-for-continuous-delivery-reactive-high-availability-devops-and-cloud-native-open-source-with-netflixoss

Best Practices for Zero Risk, Zero Downtime Database Maintenance
http://www.oracle.com/us/products/database/311390-133499.pdf

VMware vSphere High Availability 5.0 Deployment Best Practices
http://www.vmware.com/files/pdf/techpaper/vmw-vsphere-high-availability.pdf

Free Ebook: Continuous Delivery — What It Is and How to Get Started
http://info.puppetlabs.com/download-free-continuous-delivery-ebook.html

The Phoenix Project, A Novel About IT, DevOps & Helping Your Business Win
http://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592/

How Draw Something Scaled to 50 million New Users, in 50 Days, with Zero Downtime
http://www.infoq.com/presentations/games-scalability-omgpop

I Ain't Afraid of No Downtime: Scaling Continuous Deployment, by Cody Powell
http://www.codypowell.com/taods/2012/04/i-aint-afraid-of-no-downtime-scaling-continuous-deployment.html

Mandi Walls free ebook, Building a DevOps Culture [Kindle]
http://www.amazon.com/Building-DevOps-Culture-Mandi-Walls-ebook/dp/B00CBM1WFC

Daily Dose of DevOps: 27 People to Follow on Twitter
http://puppetlabs.com/blog/daily-dose-devops-27-people-follow

Selected QCON 2013 San Francisco presentations:

Adopting Continuous Delivery, Adjusting your Architecture
Rachel Laycock, ThoughtWorks
http://qconsf.com/system/files/presentation-slides/Adopting%20Continuous.pdf

 Build Your Own PaaS the Netflix Way
Sudhir Tonse, Manager, Cloud Platform Infrastructure, Netflix
http://qconsf.com/system/files/presentation-slides/BuildYourOwnPaaSTheNetflixWay-QConSF.pdf

Facebook Infrastructure
Pedro Canahuati, Director, Infrastructure Operations
http://qconsf.com/system/files/presentation-slides/ScalingtheOperationsOrganizationatFacebook.pdf

Tools:

Liquidbase:

• V3.0.7 was released on October 24, 2013

• Improved checksum performance

• CORE-1509: Significantly decreased memory usage, especially with large sql files

• CORE-1533: Performance improvements in dropAll

ZeroTurnAround's LiveRebel:

 log4j2:

• "Log4j 2 can automatically reload its configuration upon modification"

• "Log4j 2 contains next-generation Asynchronous Loggers based on the LMAX Disruptor library. In multi-threaded scenarios Asynchronous Loggers have 10 times higher throughput and orders of magnitude lower latency than Log4j 1.x"

• Note the performance benchmark results recently posted on takipiblog.com

Puppet Labs:

• PuppetConf 2014, September 23-24 – San Francisco

2013-11-24 Sunday - Book Review: Python In Practice

I recently had an opportunity to read Python in Practice, by Mark Summerfield as a reviewer for Pearson Education.

Two noteworthy points:
- The code samples are based on Python 3.x
- The author's web site provides sample code download

The very first thing I noticed about this book is that the sample bits of code are very concise - without extraneous fluff.

This book is an excellent addition to any software craftsman's library that needs a quick dive into practical hands-on Python coding.  Each page has a feeling of being written from the perspective of someone who has lived in the trenches producing production-quality python code for over a decade. This book has the feeling of a friendly mentor's guiding hand.

To help you understand the impression that this book left in my mind, let me share another word that arose unbidden in my mind as I made my way through the pages of this book: In Japanese, the phrase "Sensei" is often translated as "teacher", but also has a literal meaning of "One Who Has Gone Before". Mark's transmission of his depth of experience to the printed word is given in that same spirit of a Sensei teaching lessons gleaned from years of practice.

Another unexpected pleasure discovered within the pages of Mark's book: An emphasis on patterns. The first three chapters provide an excellent overview of the python-esque approach to implementing the various patterns that were described in the book Design Patterns: Elements of Reusable Object-Oriented Software.

I also found Chapter 5. Extending Python to be very interesting for focusing the discussion on Python's ctypes and Cython - techniques which are often needed to solve high performance problems.

One minor criticism is the author's choice in focusing on XML-RPC style in Chapter 6. High Level Networking [versus RESTful web service based integration strategies]


Chapter 1. Creational Design Patterns

1.1 Abstract Factory Pattern
1.2 Builder Pattern
1.3 Factory Method Pattern
1.4 Prototype Method Pattern
1.5 Singleton Pattern

Chapter 2. Structural Design Patterns

2.1 Adapter Pattern
2.2 Bridge Pattern
2.3 Composite Pattern
2.4 Decorator Pattern
2.5 Facade Pattern
2.6 Flyweight Pattern
2.7 Prox Pattern

Chapter 3. Behavioral Design Patterns

3.1 Chain of Responsibility Pattern
3.2 Command Pattern
3.3. Interpreter Pattern
3.4 Iterator Pattern
3.5 Mediator Pattern
3.6 Memento Pattern
3.7 Observer Pattern
3.8 State Pattern
3.9 Strategy Pattern
3.10 Template Method Pattern
3.11 Visitor Pattern
3.12 Case Study: An Image Package

Chapter 4. High-Level Concurrency

4.1 CPU-Bound Concurrency
4.2 I/O Bound Concurrency
4.3 Case Study: A Concurrent GUI Application

Chapter 5. Extending Python

5.1 Accessing C Libraries with ctypes
5.2 Using Cython
5.3 Case Study: An Accelerated Image Package

Chapter 6. High-Level Networking

6.1 Writing XML-RPC Applications
6.2 Writing RPyC Applications

Chapter 7. Graphical User Interfaces with Tkinter

7.1 Introduction to Tkinter
7.2 Creating Diaglogs with Tkinter
7.3 Creating Main-Window Applications with TKinter

Chapter 8. OpenGL 3D Graphics

8.1 A Perspective Scene
8.2 An Orthographic Game

2013-11-21 Thursday - XML/XSD validation in Python and Installing PIP (Python Package Manager)

I'm working on developing a small utility to help automate the documentation and generation of some Enterprise Integration Architecture Context Diagrams from various spreadsheets that list the inventory of different systems and their integration (transforming the data into an XML canonical form, and then using that canonical form as the basis for generating graphviz .dot files).  This way, I will have a reusable tool that can be used across multiple projects - and simply either ask for the data in the canonical form - or limit my future effort to just getting it into the canonical form.

So I found myself needing to incorporate a few 3rd party Python packages into my solution, so I decided to spend some time this evening to finally install a decent package manager in my development environment.

First, install setuptools
https://pypi.python.org/pypi/setuptools#using-setuptools-and-easyinstall


https://pypi.python.org/pypi/pip
http://www.pip-installer.org/en/latest/installing.html

http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows

http://dubroy.com/blog/so-you-want-to-install-a-python-package/

Unofficial Windows Binaries for Python Extension Packages
http://www.lfd.uci.edu/~gohlke/pythonlibs/

http://dubroy.com/blog/so-you-want-to-install-a-python-package/



My original need was to install a package to help develop some code that requires XSD/XML manipulation.

Initially, I thought would be a good fit for my need - but the pip install process seems have some issues (at least in my environment)
http://lxml.de/installation.html
http://lxml.de/validation.html#xmlschema
http://stackoverflow.com/questions/17819884/xml-xsd-feed-validation-against-a-schema
http://code.activestate.com/recipes/578503-validate-xml-with-schemalocation/

A second possible package for consideration (although it doesn't appear to have any development since 2006...):
https://pypi.python.org/pypi/pyxsd

The xlrd package provides support for reading Excel spreadsheets:
https://pypi.python.org/pypi/xlrd
http://www.python-excel.org/ 

2013-11-07 Thursday - Suggested SOA reading list

A colleague recently asked what resources I might recommend for them to begin learning about SOA (he's recently been dropped into a large enterprise organization that has a complex existing SOA environment)

Here's the reading list I prepared...

First, check InfoQ for articles and conference presentations:
http://www.infoq.com/soa/

Next, check YouTube for recent videos:
http://www.youtube.com/results?filters=year&search_query=service+oriented+architecture+soa&lclk=year

There are 10 IBM SOA Tutorials linked to here:

http://www.ibm.com/developerworks/webservices/library/ws-top10/

TechTarget.com has some

http://searchsoa.techtarget.com/tutorial/SOA-Tutorials

This author’s tutorials may be of interest
http://tutorials.jenkov.com/soa/index.html

The best resource for quickly getting up to speed (with a holistic and comprehensive approach) would be to read Jim Bean’s book:

http://www.amazon.com/SOA-Web-Services-Interface-Design/dp/0123748917/

(there is a kindle version available too)

Next, I would direct to you read anything on the web that is SOA-focused by Arnon Rotem-Gal-Oz, as well as his book:

http://www.amazon.com/SOA-Patterns-Arnon-Rotem-Gal-Oz/dp/1933988266/

(he is one of the most clear-headed people when it comes to real-world, practical SOA.

Thomas Erl is one of my favorite writers on SOA – because he is meticulous in building a body of knowledge (some might criticize his writing for be repetitive, but I see it as reinforcing the concepts)

I would start with his SOA Design patterns book:

http://www.amazon.com/Design-Patterns-Prentice-Service-Oriented-Computing/dp/0136135161/

And perhaps then his SOA with REST: Principles, Patterns & Constraints for Building Enterprise Solutions with REST

http://www.amazon.com/SOA-REST-Principles-Constraints-Enterprise/dp/0137012519/

If you think you still need more foundational information, then go back to his  Service Oriented Architect: Concepts, Technology, and Design

http://www.amazon.com/Service-Oriented-Architecture-SOA-Concepts-Technology/dp/0131858580/

Or his SOA: Principles of Service Design

http://www.amazon.com/SOA-Principles-Service-Thomas-Erl/dp/0132344823/

His most recent book, which I reviewed for Prentice Hall, is also worth reading: Cloud Computing: Concepts, Technology & Architecture

http://www.amazon.com/Cloud-Computing-Concepts-Technology-Architecture-ebook/dp/B00CM9V7Q8/

For a more technical deep dive discussion of SOA service contract versioning:

http://www.amazon.com/Web-Service-Contract-Design-Versioning/dp/013613517X/

I also reviewed Robert Daigneau’s book: Service Design Patterns: Fundamental Design Solutions for SOAP/WSDL and RESTful Web Services, which has a very light touch on RESTful service – but has some concise writing on service design patterns that may be of interest

http://www.amazon.com/Service-Design-Patterns-Fundamental-Solutions/dp/032154420X/