2018-12-04 Tuesday - AWS Lambda Functions

This posting is to to provide a link to the notes that I've begun assembling in my github repository for AWS Lambda Functions (i.e. Serverless Functions):

https://github.com/intltechventures/Lab.Cloud.AWS/tree/master/aws.Lambda

2018-11-20 Tuesday - Machine Learning Technical Debt and Anti-Patterns

This posting is a placeholder for links to interesting articles/papers that touch on technical debit and anti-patterns in Machine Learning algorithms/models/solutions.

Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

"Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns."

A  Convex Framework for Fair Regression

https://arxiv.org/abs/1706.02409

https://arxiv.org/pdf/1706.02409.pdf

"The widespread use of machine learning to make consequential decisions about individual citizens (including in domains such as...criminal sentencing has been accompanied by increased reports of instances in which the algorithms and models employed can be unfair or discriminatory in a variety of ways ...we introduce a rich family of fairness metrics for regression models that take the form of a fairness regularizer and apply them to the standard loss functions for linear and logistic regression." 

"Since these loss functions and our fairness regularizer are convex, the combined objective functions obtained from our framework are also convex, and thus permit efficient optimization. Furthermore, our family of fairness metrics covers the spectrum from the type of group fairness that is common in classification formulations (where e.g. false arrests in one racial group can be “compensated” for by false arrests in another racial group) to much stronger notions of individual fairness (where such cancellations are forbidden, and every injustice is charged to the model). Intermediate fairness notions are also covered. Our framework also permits one to either forbid the use of a "protected” variable (such as race), by demanding that a single model be learned across all groups, or to build different group-dependent models." 

"Most importantly, by varying the weight on the fairness regularizer, our framework permits us to compute the entire “Pareto curve” or efficient frontier of the trade-off between predictive accuracy and fairness.  Such curves are especially important to examine and understand in a domain-specific manner: since demanding fairness of models will always come at a cost of reduced predictive accuracy , it behooves practitioners working with fairness-sensitive data sets to understand just how mild or severe this trade-off is in their particular arena, permitting them to make informed modeling and policy decisions""...in this work we have studied a variety of fairness regularizers for regression problems, and applied them to data sets in which fairness is not subservient to generalization, but is instead a first-order consideration. Our empirical study has demonstrated that the choice of fairness regularizer (group, individual, hybrid, or other) and the particular data set can have qualitative effects on the trade-off between accuracy and fairness...." 

"The Communities and Crime dataset, from the UCI repository is a dataset which includes many features deemed relevant to violent crime rates (such as the percentage of the community’s population in an urban area, the community’s racial makeup, law enforcement involvement and racial makeup of that law enforcement in a community, amount a community’s law enforcement allocated to drug units) for different communities. This data is provided to train regression models based on this data to predict the amount of violent crime (murder, rape, robbery, and assault) in a given community...." 

"The COMPAS dataset The COMPAS dataset contains data from Broward County, Florida originally compiled by ProPublica in which the goal is to predict whether a convicted individual would commit a violent crime in the following two years or not. ..." 

2018-11-20 Tuesday - Conference on Neural Information Processing Systems (NIPS)

The Thirty-second Conference on Neural Information Processing Systems (NIPS) will be held Dec 2-8, 2018 - at the Palais des Congrès de Montréal, Montréal, Canada 

https://nips.cc/Conferences/2018/Schedule



Electronic Proceedings of the Neural Information Processing Systems Conference
https://papers.nips.cc/


Interesting papers from the 2017 NIPS conference:
https://papers.nips.cc/book/advances-in-neural-information-processing-systems-30-2017

Hunt For The Unique, Stable, Sparse And Fast Feature Learning On Graphs
https://papers.nips.cc/paper/6614-hunt-for-the-unique-stable-sparse-and-fast-feature-learning-on-graphs

"For the purpose of learning on graphs, we hunt for a graph feature representation that exhibit certain uniqueness, stability and sparsity properties while also being amenable to fast computation. This leads to the discovery of family of graph spectral distances (denoted as FGSD) and their based graph feature representations, which we prove to possess most of these desired properties. To both evaluate the quality of graph features produced by FGSD and demonstrate their utility, we apply them to the graph classification problem. Through extensive experiments, we show that a simple SVM based classification algorithm, driven with our powerful FGSD based graph features, significantly outperforms all the more sophisticated state-of-art algorithms on the unlabeled node datasets in terms of both accuracy and speed; it also yields very competitive results on the labeled datasets - despite the fact it does not utilize any node label information."

Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
https://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent

"We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones. Causes of failures include software bugs, network asynchrony, biases in local datasets, as well as attackers trying to compromise the entire system. Assuming a set of   workers, up to   being Byzantine, we ask how resilient can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient aggregation rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the aggregation rule capturing the basic requirements to guarantee convergence despite   Byzantine workers. We propose \emph{Krum}, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD. We also report on experimental evaluations of Krum."

One-Shot Imitation Learning
https://papers.nips.cc/paper/6709-one-shot-imitation-learning

"Imitation learning has been commonly applied to solve different tasks in isolation. This usually requires either careful feature engineering, or a significant number of samples. This is far from what we desire: ideally, robots should be able to learn from very few demonstrations of any given task, and instantly generalize to new situations of the same task, without requiring task-specific engineering. In this paper, we propose a meta-learning framework for achieving such capability, which we call one-shot imitation learning. Specifically, we consider the setting where there is a very large (maybe infinite) set of tasks, and each task has many instantiations. For example, a task could be to stack all blocks on a table into a single tower, another task could be to place all blocks on a table into two-block towers, etc. In each case, different instances of the task would consist of different sets of blocks with different initial states. At training time, our algorithm is presented with pairs of demonstrations for a subset of all tasks. A neural net is trained that takes as input one demonstration and the current state (which initially is the initial state of the other demonstration of the pair), and outputs an action with the goal that the resulting sequence of states and actions matches as closely as possible with the second demonstration. At test time, a demonstration of a single instance of a new task is presented, and the neural net is expected to perform well on new instances of this new task. Our experiments show that the use of soft attention allows the model to generalize to conditions and tasks unseen in the training data. We anticipate that by training this model on a much greater variety of tasks and settings, we will obtain a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks."

DPSCREEN: Dynamic Personalized Screening
https://papers.nips.cc/paper/6731-dpscreen-dynamic-personalized-screening

"Screening is important for the diagnosis and treatment of a wide variety of diseases. A good screening policy should be personalized to the disease, to the features of the patient and to the dynamic history of the patient (including the history of screening). The growth of electronic health records data has led to the development of many models to predict the onset and progression of different diseases. However, there has been limited work to address the personalized screening for these different diseases. In this work, we develop the first framework to construct screening policies for a large class of disease models. The disease is modeled as a finite state stochastic process with an absorbing disease state. The patient observes an external information process (for instance, self-examinations, discovering comorbidities, etc.) which can trigger the patient to arrive at the clinician earlier than scheduled screenings. The clinician carries out the tests; based on the test results and the external information it schedules the next arrival. Computing the exactly optimal screening policy that balances the delay in the detection against the frequency of screenings is computationally intractable; this paper provides a computationally tractable construction of an approximately optimal policy. As an illustration, we make use of a large breast cancer data set. The constructed policy screens patients more or less often according to their initial risk -- it is personalized to the features of the patient -- and according to the results of previous screens – it is personalized to the history of the patient. In comparison with existing clinical policies, the constructed policy leads to large reductions (28-68 %) in the number of screens performed while achieving the same expected delays in disease detection."

Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
https://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent

"We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones. Causes of failures include software bugs, network asynchrony, biases in local datasets, as well as attackers trying to compromise the entire system. Assuming a set of n workers, up to f being Byzantine, we ask how resilient can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient aggregation rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the aggregation rule capturing the basic requirements to guarantee convergence despite f Byzantine workers. We propose \emph{Krum}, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD. We also report on experimental evaluations of Krum."

2018-11-11 Sunday - Suggested Books for QA/Test Teams

While doing some research for recommendations on elevating the maturity of a QA Test Team - I happened across five books in particular that I think would be beneficial to any software testing team:

"A Friendly Introduction to Software Testing", 
by Bill Laboon - a lecturer in the Computer Science department at the University of Pittsburgh, where he focuses on teaching how to develop high-quality software in a real-world environment.
https://www.amazon.com/Friendly-Introduction-Software-Testing/dp/152347737

"Complete Guide to Test Automation: Techniques, Practices, and Patterns for Building and Maintaining Effective Software Projects", 
by Arnon Axelrod - a test automation expert, working as a senior consultant, architect, trainer, and lead of the test automation team at Sela Group
https://www.amazon.com/Complete-Guide-Test-Automation-Maintaining/dp/1484238311/

"Mastering Selenium WebDriver 3.0: Boost the performance and reliability of your automated checks by mastering Selenium WebDriver, 2nd Edition", 
by Mark Collin - a professional software tester who has been working in the software testing industry since 2001.
https://www.amazon.com/Mastering-Selenium-WebDriver-3-0-performance/dp/1788299671

"Software Testing: A Craftsman’s Approach, Fourth Edition", 
by Paul C. Jorgensen, PhD
https://www.amazon.com/Software-Testing-Craftsmans-Approach-Fourth/dp/1466560681/

"Lessons Learned in Software Testing: A Context-Driven Approach", 
by Cem Kaner, JD, PhD, - a professor of computer sciences at Florida Institute of Technology;
James Bach - founder and principal consultant of Satisfice, Inc.,a software testing and quality assurance company;
Bret Ptetichord - who works as an independent consultant, http://pettichord.com/
https://www.amazon.com/Lessons-Learned-Software-Testing-Context-Driven/dp/0471081124/

2018-10-27 Saturday - 10 Lessons from Sailing, for IT

Recently I had some work done on my mast - and I took some time this morning to ponder some of the lessons I've learned from sailing (either through direct experience, or by studying the historical accounts of the voyages of others)...and how those might apply to the world of IT...

1) Preventive Maintenance: On a sailboat, preventative maintenance is a lot like taking care of Technical Debt in IT. Avoid or ignore it at your peril. A mast failure, when sailing in storm conditions - much like a performance/load failure for your IT systems. It all comes crashing down.

2) Keeping a Weather Eye: Means to monitor the horizon for developing weather conditions. Within the world of IT that can mean observing closely which projects/areas of IT are gaining/losing funding, organizational restructuring, and whenever new external third-party IT vendors are being introduced into the organization.

3) Sometimes You Must Tack: On a sailboat, you must often tack (change course) to reach your target destination - because the wind changes direction, or you are trying to make way against a current that is not in your favor. Your career, much like the track of a sailboat - will rarely run on a straight path for very long. Often, you may find that you need to sail away from your destination, or even sail backwards. In your IT career, this can take the form of willingly taking what might appear as a demotion (lower job title, less pay) - so that you can make "headway" in gaining new skills and experience - as well as gaining important connections.

4) Faster Doesn't Always Win (or Finish) The Race. If you focus only on speed in sailing - you are much more likely to experience catastrophic failure of essential equipment - or, your crew will become exhausted and burned-out to the point of making catastrophically bad decisions. Same goes for your IT staff.

5) Compasses Can Lie. On a boat, the accuracy of your compass can be skewed by magnetic declination and/or magnetic deviation. So too, in IT - While you may have your 3 or 5 year IT Roadmap all planned out - and you may have your Key Performance Indicators (KPIs) established to help you measure your progress and alignment with your plan - your "compass" can be skewed by underlying market forces, or competitors bypassing your strategy entirely by focusing on something that is even more fundamentally important to your customers.

6) You Need A Course Plotted Out. Leaving the harbor on a boat for a leisurely afternoon sail is fine - but if you expect to cross great expanses of open ocean - you will have to contend with weather patterns, shipping traffic, and currents - and you need to plan for enough provisions, fuel, and electrical power management to keep your crew healthy, happy, well fed - and all of your essential systems operational. Often in IT, we see a pattern of constantly changing priorities - and poorly charted courses - with apparently no thought to the sustainability of making any voyage - nor any intent in reaching a destination. You'll soon find your "crew" abandoning ship as soon as the first habitable "island" appears on the horizon.

7) Things Break, Have Contingency Plans. Before embarking on a long voyage, the prudent skipper has already charted a primary course (and alternative courses to safe harbors), laid on sufficient provisions, and has gone over the game plan for making the voyage with the crew. But preparations for making a voyage don't end there. The prudent skipper has also planned for various emergencies that may arise - emergency pumps, emergency locator beacons, a life raft, emergency provisions, an emergency watermaker, signaling devices, solar battery chargers for emergency communication gear, etc. Within IT, having third-party relationships established - to provide supplemental staffing in case key team members are lost could be part of your contingency plan - or ensuring that you have active-passive fail-over strategies for your cloud-hosted infrastructure. Or implementing abstraction layers over a cloud-vendor's specific implementation of a particular service - so that you may more easily swap out another cloud service provider's service - without requiring rip-and-replace throughout your application architecture.

8) Expertise, Competence, Passion - Beats Size, Every Day. Given the choice to crew aboard a small ship, or a huge mega yacht - it all comes down to who else in onboard. The size of the ship - or the number of crew - or the fancy amenities offered onboard are of little concern - what matters is whether the selected crew members brings the right mix of expertise, competence - and passion. Without those three things, a ship can breakdown in the middle of a voyage and become a floating prison - or a floating hell. With those three things - even a small ship with a good crew can make a faster passage than a huge crew who forever fights among themselves - shirking their duties - and finding fault rather than pitching in and doing the tasks at hand that need doing. Ships that must make long voyages must be run as efficiently as possible, that means you cannot afford to have dead-weight onboard - consuming resources, but not adding to the operation of the ship. There are many analogies in this - in the way many IT organizations are often mismanaged.

9) Selection of Crew and Crew Cohesion. On a long voyage - the fit, temperament, and cohesiveness of the crew is vital and of paramount importance to the success of the endeavor. For a master treatise on leadership, the importance of crew selection, and the importance of leadership in maintaining crew cohesion - you can do no better than to read Ernest Henry Shackleton's own written account of the 1914 voyage of the Endurance to the Antartic, "South: The Story of Shackleton's 1914-1917 Expedition"

10) Sometimes Your Ship Sinks. No matter how diligently you plan, no matter how much care and attention you give to the maintenance of your ship - sometimes catastrophically bad things will happen - after which, events unfold in ways that you cannot manage (for example, the rate of water ingress may exceed the capacity of your pumps to keep up with the inflow). That's when your planning and approach to voyage preparation may spell the difference between hoping for rescue, or rescuing yourself. I always prefer to have a sailing dinghy on deck, in lieu of a life raft (for several excellent historical accounts of why this is my preferred choice, please see: The Voyage of Bounty's Launch; Dougal Robertson: Survive The Savage Sea; 438 Days: An Extraordinary True Story of Survival at Sea; Adrift: Seventy-six Days Lost at Sea; Once Is Enough). Within the world of IT, a Death March Project is a comparable event - but so too are much slower scenarios of your own personal "ship" sinking - such as being pigeon holed into roles (for years, or decades) that don't allow you to be exposed to new technologies, learn, grow, and gain in responsibility.

2018-10-23 Tuesday - Camtasia Tips & Tutorials

I watched an interesting tutorial on Youtube recently, and I was impressed by some of the tricks and techniques that were used to make the video more engaging (like zooming in and out of different parts of a screen). 

This post is a placeholder for links to interesting resources, collections of tips, documentation, and tutorials on becoming a more proficient user of Camtasia...

Zooming:

• https://www.techsmith.com/blog/how-to-zoom-in-on-a-video/
• "Make sure to set your project dimensions (i.e. the size of the video you will produce) smaller than the media you’re working with. Any videos or images you choose will look best when displayed at their natural size, that is, 100% scale."
• "Add zooms to a video in Camtasia by changing the scale (think: relative size) of a clip over a period of time. As the scale of the clip increases or decreases, it gives the effect of getting closer to or farther away from the subject."
• "Create a zoom effect by adding an animation to a clip on the Camtasia timeline. Animations allow you to change the properties of a clip over a period of time. To add an animation go to the Animations tab and drag a Custom animation to a clip."
• "The next step is to edit the clip’s properties at the end of the animation. First, place the playhead to the right of the animation." 
• "To create the effect of zooming in on the clip, go to the Properties panel and increase the scale by dragging the slider to the right. If you want to zoom out, decrease the scale by dragging the slider to the left."

• Youtube: How to Zoom In with Camtasia for Screenshare Tutorials
• Youtube: Cool "Pop Out" Zoom Effect in Camtasia Studio 

 Animations & Effects:

• https://www.techsmith.com/tutorial-camtasia-animations-effects.html

 Trim Videos:

• https://www.techsmith.com/blog/trim-videos/

Alternative Tools:

• http://gilisoft.com/product-video-recorder.htm  
• https://atomisystems.com/activepresenter/
• https://atomisystems.com/download/
• https://elearningsupporter.com/2017/07/21/activepresenter-6-reviews/

2018-10-11 Thursday - Proposed Talk for O'Reilly AI Conference 2019

A proposed talk I've submitted for the O'Reilly Artificial Intelligence Conference in New York, April 15-18, 2019

https://conferences.oreilly.com/artificial-intelligence/ai-ny

JAGS vs Edward for Markov Chain Monte Carlo (MCMC) Simulation

An AI neophyte's experience exploring the capabilities and features  of JAGS vs Edward for Markov Chain Monte Carlo Simulation.  Lessons learned. A discussion of my approach to tackling the various learning materials I found useful to gaining a level of working knowledge.

This presentation will compare and contrast the utility of JAGS vs Edward for implementing Markov chain Monte Carlo (MCMC) simulation.

In statistics, Markov chain Monte Carlo (MCMC) methods comprise a class of algorithms for sampling from a probability distribution. By constructing a Markov chain that has the desired distribution as its equilibrium distribution, one can obtain a sample of the desired distribution by observing the chain after a number of steps. The more steps there are, the more closely the distribution of the sample matches the actual desired distribution.

Edward is a Python library for probabilistic modeling, inference, and criticism. It is a testbed for fast experimentation and research with probabilistic models, ranging from classical hierarchical models on small data sets to complex deep probabilistic models on large data sets. Edward fuses three fields: Bayesian statistics and machine learning, deep learning, and probabilistic programming.

JAGS is Just Another Gibbs Sampler. It is a program for analysis of Bayesian hierarchical models using Markov Chain Monte Carlo (MCMC) simulation.

2018-10-09 Tuesday - Early Access: Model Based Machine Learning (book)

http://mbmlbook.com/
by John Winn and Christopher Bishop, with Thomas Diethe, John Guiver and Yordan Zaykov

2018-10-07 Sunday - Serverless Architectures on AWS, Second Edition

https://www.manning.com/books/serverless-architectures-on-aws-second-edition

"Serverless Architectures on AWS, Second Edition teaches you how to design, secure and manage serverless backends that can power the most demanding web and mobile apps! This new edition has been fully-updated to include new chapters on current best practices and innovative use-cases, along with full coverage of the latest changes to AWS."

some additional links that may be of interest:

• https://twitter.com/sbarski
• https://github.com/sbarski/serverless-architectures-aws
• https://github.com/sbarski?tab=repositories
• https://github.com/pmuens/awesome-serverless

Some of the talks from this year's from Serverlessconf San Francisco 2018

https://acloud.guru/series/serverlessconf-sf-2018

https://medium.com/lumigo/takeaways-from-the-serverlessconf-sf-2018-7167838394d5

2018-10-06 Saturday - Evaluating Open Broadcaster Software

This weekend I'm evaluating OBS (Open Broadcaster Software, GPLv2 license) - as a possible alternative to TechSmith's Camtasia commercial product.

• https://obsproject.com/ 
• https://obsproject.com/docs/
• https://obsproject.com/wiki/
• https://obsproject.com/wiki/OBS-Studio-Overview
• https://obsproject.com/wiki/OBS-Studio-Quickstar
• https://obsproject.com/forum/
• https://github.com/obsproect/obs-studio 
• https://github.com/obsproject/obs-studio/blob/master/README.rst
• Capabilities:
• "High performance real time video/audio capturing and mixing. Create scenes made up of multiple sources including window captures, images, text, browser windows, webcams, capture cards and more."
• "Set up an unlimited number of scenes you can switch between seamlessly via custom transitions."
• "Intuitive audio mixer with per-source filters such as noise gate, noise suppression, and gain. Take full control with VST plugin support."
• "Powerful and easy to use configuration options. Add new Sources, duplicate existing ones, and adjust their properties effortlessly."
• "Streamlined Settings panel gives you access to a wide array of configuration options to tweak every aspect of your broadcast or recording."
• "Choose from a number of different and customizable transitions for when you switch between your scenes or add your own stinger video files."
• "Set hotkeys for nearly every sort of action, such as switching between scenes, starting/stopping streams or recordings, muting audio sources, push to talk, and more."
• "Studio Mode lets you preview your scenes and sources before pushing them live. Adjust your scenes and sources or create new ones and ensure they're perfect before your viewers ever see them."
• "Get a high level view of your production using the Multiview. Monitor 8 different scenes and easily cue or transition to any of them with merely a single or double click."
• "OBS Studio is equipped with a powerful API, enabling plugins and scripts to provide further customization and functionality specific to your needs."
• "Utilize native plugins for high performance integrations or scripts written with Lua or Python that interface with existing sources."

I've been using Camtasia for some client work - and while I think it does a decent job - the UI /UX  isn't always intuitively obvious (or, I just need to spend more time reading the documentation and going through their tutorials...) - so I'm hopeful that OBS may be a viable alternative.

These are the Recording configuration settings that I've settled on for the moment (which reduced the recording size from ~6MB to 837 kb)

So far I'm very pleased with the ease-of-use - it took me about 5 minutes to become productive with the tool.

2018-10-04 Thursday - Computation Modeling, Two Suggested Books

Based on the reviews, table of contents, etc. - I've ordered the two following books:


Computational Modeling of Cognition and Behavior
by Simon Farrell (Author), Stephan Lewandowsky
https://www.amazon.com/Computational-Modeling-Cognition-Behavior-Farrell/dp/110710999X/

"Computational modeling is now ubiquitous in psychology, and researchers who are not modelers may find it increasingly difficult to follow the theoretical developments in their field. This book presents an integrated framework for the development and application of models in psychology and related disciplines. Researchers and students are given the knowledge and tools to interpret models published in their area, as well as to develop, fit, and test their own models. Both the development of models and key features of any model are covered, as are the applications of models in a variety of domains across the behavioural sciences. A number of chapters are devoted to fitting models using maximum likelihood and Bayesian estimation, including fitting hierarchical and mixture models. Model comparison is described as a core philosophy of scientific inference, and the use of models to understand theories and advance scientific discourse is explained."

Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan 2nd Edition
https://www.amazon.com/Doing-Bayesian-Data-Analysis-Tutorial/dp/0124058884/

"Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan, Second Edition provides an accessible approach for conducting Bayesian data analysis, as material is explained clearly with concrete examples. Included are step-by-step instructions on how to carry out Bayesian data analyses in the popular and free software R and WinBugs, as well as new programs in JAGS and Stan. The new programs are designed to be much easier to use than the scripts in the first edition. In particular, there are now compact high-level scripts that make it easy to run the programs on your own data sets."

2018-09-30 Sunday - Business Functions vs. Business Capabilities

My view is that a Business Capability is the higher, stable, abstract concept of what a business does (e.g. Marketing, Sales, etc.) - independent of the organization's structure, processes, people, or domains.

The Business Functions are the implementation decomposition/breakdown of the Business Capabilities into processes/activities/roles in how a Business Capability is delivered. 

A great article: Rethinking the Function of Business Functions

https://hbr.org/2013/02/strength-focus-and-corporate-c

• "Business units come and go, but finance, HR, IT, marketing, legal, and R&D are forever. Nonetheless, many CEOs and top executives struggle with their functional organizations, and some question whether the established functional model is still relevant. In their view, functional priorities are all too often in conflict with — or not fully supportive of — the strategic needs of the business."
• "The challenge for the functional model today is that companies don’t need to build generic functional strengths. They need to build more specific, bespoke capabilities that are part of the inherent identity of the company, and hard for anyone else to duplicate."
• "The most distinctive, differentiating capabilities are almost always cross-functional"

Another great article: Leveraging business capabilities for strategic planning

https://www.cio.com/article/3250677/it-strategy/leveraging-business-capabilities-for-strategic-plann...

• "Business capabilities are an integrated set of processes, technologies, and deep expertise that are manifested as a functional capacity to capture or deliver value to the organization. They outline “what” a business does, as opposed to “how” a business does it. They are the definition of your organizational skills, best represented in a landscape map that allows you to evaluate the full spectrum of capabilities against each other."
• "Business capability maps are not just about technology; these tools are designed to improve an organization's holistic ability to improve a business outcome, and in many cases, it is not the technology that is the constraint, but rather a process, skill, or policy issue."
• "...often challenging in an organization that thinks and acts in functional silos, but a capability-driven approach will bridge that gap."

Gartner is often engagd to help businesses define their Business Capabilities (L0, L1, L2 - for example) 

https://www.gartner.com/webinar/2680616

• "This webinar provides practical advice for mid-market IT leaders on using business capability modeling to bring clarity and insight to their IT strategy"

https://www.gartner.com/webinar/3680517

• "Business capability modeling is used to represent how enterprises deliver strategic value to customers. These models represent the future-state capabilities of the business and illustrate how current capabilities need to change to support strategic opportunities and threats"

https://www.gartner.com/doc/3534851/best-practices-creating-highimpact-business

• "Focus on Future-State Business Capabilities First and Foremost"
• "Build Your Business Capability Model Based on Your Business Strategy, Not Generic Reference Models or Templates"
• "Don't Confuse a Business Capability Model With a Business Process or Other Type of Model"
• "Label Each Capability With a "Verb-Noun" Combination to Emphasize Action"
• "Limit Business Capabilities to Eight to 10 Capabilities at Each Level"
• "Use Capability Labels to Convey the Value Delivered to External Stakeholders"

https://www.gartner.com/it-glossary/business-capability-modeling

• "Business capability modeling is a technique for the representation of an organization’s business anchor model, independent of the organization’s structure, processes, people or domains. "

https://eam-initiative.org/pages/yg7qjgxv78f5/Business-Capability-Business-Capability-Map

• "A business capability is an abstraction of a business function, answering the question what a company needs to be able to do to be successful."

TOGAF 9.2

• "Capability: An ability that an organization, person, or system possesses. For example, Enterprise Architecture, marketing, customer contact, or outbound telemarketing.:"
• "Business Function: Delivers business capabilities closely aligned to an organization, but not necessarily explicitly governed by the organization."

ArchiMate 3.0.1
http://pubs.opengroup.org/architecture/archimate3-doc/chap08.html

• "A business function offers functionality that may be useful for one or more business processes. It groups behavior based on, for example, required skills, resources, (application) support, etc. Typically, the business processes of an organization are defined based on the products and services that the organization offers, while the business functions are the basis for, for example, the assignment of resources to tasks and the application support."
• "A business function is a collection of business behavior based on a chosen set of criteria (typically required business resources and/or competencies), closely aligned to an organization, but not necessarily explicitly governed by the organization."
• "Just like a business process, a business function also describes internal behavior performed by a business role.However, while a business process groups behavior based on a sequence or flow of activities that is needed to realize a product or service, a business function typically groups behavior based on required business resources, skills, competencies, knowledge, etc."

https://en.wikipedia.org/wiki/Capability_management_in_business

• "Capability management is the approach to the management of an organization, typically a business organization or firm, based on the "theory of the firm" as a collection of capabilities that may be exercised to earn revenues in the marketplace and compete with other firms in the industry. "Capability Management" seeks to manage the stock of capabilities within the firm to ensure its position in the industry and its ongoing profitability and survival."
• "A process is how the capability is executed."
• "Dave Ulrich makes a distinction between capabilities and competencies: individuals have [Functional] competencies while organizations have [business] capabilities."

https://en.wikipedia.org/wiki/Business_architecture

• "The Business Capabilities view describes the primary business functions of an enterprise and the pieces of the organization that perform those functions"

https://cio-wiki.org/home/loc/home?page=business-capability

• "Business capability is the expression or the articulation of the capacity, materials and expertise an organization needs in order to perform core functions."
• "Capability is an attribute of a system - so it is used largely as an abstraction of a system. The capability is used in the context of assessment - capacity to realize an outcome, existing or intended, or potential - which is in stark contrast to the use of the process. With the capability view, being more abstract, no detail is provided as to the means by which the transformation occurs, nevertheless, the system has external interfaces which deal with inputs and outputs. With the process view, being more logic and detailed, the process shows the receiving processes, the distributing processes, and the transforming processes."
• "What vs. How: Capabilities are WHAT abilities/ competencies an organization has/needs. Processes are HOW an organization does something."

https://www.architectureandgovernance.com/strategy-planning/business-capability-models-might-missing...

• "The business capabilities are the top layer of the business architecture. They belong to a business domain and are governed by the business principles and outcomes of the organization. The capabilities are realized by a combination of business process, people, and technology and are, therefore, at a higher level than a business process and sits in the conceptual layer."
• "Capability represents what the organization can do."
• "Function represents what the organization is doing with that capability."
• "Process identifies how the organization is performing the function."
• "Organization Unit identifies the department responsible for performing the process."

https://searchmicroservices.techtarget.com/definition/business-capability

• "Business capabilities are sometimes confused with other concepts in business process management such as business processes and business functions. Business processes describe the methods an organization employs in order to provide and leverage business capabilities. Business functions describes the roles that individuals and units within the business play in regards to meeting business objectives."
• "While functions and roles tend to change rapidly as new employees enter the business, business capabilities remain relatively stable. High-level business capabilities include concepts such as sales and supply chain management that can be met by a number of various business processes, which in turn can incorporate a variety of business roles. Business capabilities can also be broken down into more granular levels. Supply chain management, for example, could be split into product flow, information flow, and finances flow."

Capability Attributes

https://www.omg.org/news/member-news/OMG-Putting-Cap-To-Work.pdf

• "Each capability is unique. A capability is a fundamental element of the organization and as such is clearly different from other capabilities. A capability might be applied throughout the organization and can be applied in different ways to affect different outcomes but it is still a single capability."
• "Capabilities are stable. Well-defined capabilities rarely change. They provide a much more stable view of organizations than do projects, processes, applications, or even strategies. Capabilities only change when there is a significant shift in the underlying business model or mission which might occur through a business transformation initiative or in conjunction with a new merger or acquisition. This stability is a major part of their appeal."
• "Capabilities are abstracted from the organizational model. Capability models are not just a simple restatement of the enterprise’s organizational model. They are organizationally neutral which means that changes in the organizational structure don’t require a change in the capability model. In simple organizations, the capability model may indeed look similar to the corporate organizational structure; however, in more-complex organizations capabilities both common to and unique to organizational units are found."

Enterprise Architecture at Work, Fourth Edition, Springer, (2017) by Marc Lankhorst et al
5.8.2 Business Behavior Concepts, page-91

"...business processes of an organisation are defined based on the products and services that the organization offers, while the business functions are the basis for the assignment of resources to tasks and for the application support."

A few additional definitions from various Google searches: 

• "Business capability is the expression or the articulation of the capacity, materials and expertise an organization needs in order to perform core functions. ... Businessfunctions describes the roles that individuals and units within the business play in regards to meeting business objectives."

• "Business capability mapping focuses on what a business does and provides a crisper view of an organizations objectives by eliminating the inherent complexity of discussing the how or the who."

• "Defining Business Capability. A business capability is what a company needs to be able to do to execute its business strategy. Another way to think about capabilities is as a collection of people, process, and technology gathered for a specific purpose."

• "A Capability is a higher-level solution behavior that typically spans multiple Agile Release Trains (ARTs). They are sized and split into multiple features so that they can be implemented in a single PI. SAFe describes a hierarchy of artifacts that define functional system behavior:"

• "A business function is defined as any set of activities performed by the department that is initiated by an event, transform information, materials or business commitments, and procedures an output (e.g. order fulfillment, invoicing, cash management, manufactured batch, customer response tracking, regulatory "

• "Business Capability modeling is a technique for the representation of an organization's business anchor model independent of the organization's structure, processes, people or domains. ... Business Capability mapping allows companies to clearly see what a business does to reach its objectives."

• "Note that business functions are distinct from capabilities. Capabilities represent the current or desired abilities of an organization, realized by its people, processes, information, and technology. They are focused on specific business outcomes, and are used for strategic planning purposes. In contrast, business functions describe the work actually done by the organization; they are often explicitly managed, and are more closely aligned to the organization structure. Each capability occurs only once in a capability map, whereas in a functional decomposition of the enterprise the same sub-function can occur multiple times."

2019-12-07 Update:

I happened upon this blog post by Aaron Rouse post today:  

• "How to recognise a Capability"
• https://classiq.co.uk/2019/10/31/how-to-recognise-a-capability/

2018-12-19 Update:

I recently noted another interesting article that somewhat touches on this topic...

• Ability, Capability, Capacity and Competence (and Reality)
• "A capability on the other hand is a feature, ability, faculty or process that can be developed or improved. A capability is a collaborative process that can be deployed and through which individual competencies and abilities can be applied and exploited. A capability can refer to an ability that exists in an individual but can be improved upon."
• "Within a business setting a Business Capability is the articulation of the capacity, materials and expertise an organisation needs in order to perform its core functions. Defining the Business Capability does not mean that the organisation can deploy it. It must have the capacity to do so."
• "A Capacity is the ability that exists at present whilst capability refers to the higher level of ability that could be demonstrated under the right conditions."

2019-03-08 Update

An interesting write-up from December 2012 that I missed in my original research (tip of the hat to Gerben Wierda for his comment on a LinkedIn post that brought this to my attention):

• https://ea.rna.nl/2012/12/27/missing-from-archimate-business-capability/            

A more recent Deloitte article, mentioned in the LinkedIn post:

• https://www2.deloitte.com/us/en/pages/strategy/articles/strategic-capabilities-bridging-strategy-and-impact.html                     

2020-02-16 Update

• https://bizzdesign.com/blog/archimate-3-0-capability-mapping/
• "A capability defines what the business does or what it can do, not how it does that or who is doing it. They are different from business processes, functions, services, organization units, or IT systems, although these may all contribute to a capability. The same capability may be implemented in different ways, e.g. manually, IT-supported or fully automated." 

2020-05-05 Update

• Arron Rouse recently posted a link on LinkedIn to remind folks of his previously published article: How to recognise a Capability

• I found it noteworthy - in the AirBnB CEO's letter today to employees - he specifically called out mapping to capabilities: "Map all reductions to our future business strategy and the capabilities we will need."
• https://news.airbnb.com/a-message-from-co-founder-and-ceo-brian-chesky/?mod=article_inline

   

I've also started assembling links to Business Capability Model exemplars here:
https://github.com/intltechventures/Lab.Architecture/blob/master/Business/Capabilities.md

 

2020-05-14 Update

• https://acrossandahead.com/en/blog-en/on-what-constitutes-a-capability/  

 

2024-03-27 Update

OMG Business Architecture Core Metamodel (BACM) v1.0 (beta, October 2023)

https://www.omg.org/spec/BACM

"Its primary concept is the business capability - an abstraction representing what a business does but not how." 

 2024-06-26 Update

• Dan KisKis, PhD
• His Medium.com page
  
• Why your Capabilities should be based on Business Objects
  

 

2018-09-23 Sunday - Book Recommendation: Functional Programming for Mortals

A book, by Sam Halliday, that I've added to my stack for October:

Functional Programming for Mortals
https://leanpub.com/fpmortals

read online (free)
https://leanpub.com/fpmortals/read

2018-09-23 Sunday - Daniel Abadi new posting, NewSQL database systems

I happened upon Daniel Abadi's recent blog posting:

NewSQL database systems are failing to guarantee consistency, and I blame Spanner 

...and think it worth your attention.

I've also added Daniel to the list of blogs that I find interesting (lower-right of my blog page)

2018-09-11 Tuesday - Ideas to Reignite Passion in Your Work, Career, Field of Endeavor

1) Envision: Write a bit of fiction - imagine where you want to be, what you want to be doing - [x] months or years in the future. Then prepare the outline of steps, actions, investments you will need to execute - and the relationships, skills, experience, and knowledge you will need to acquire. Then, as Zig Ziglar once said: "Plan your work, work your plan".

2) Expand: Attend meet-ups in your area (or form one, if one doesn't exist) on topics of interest relevant to your field.

3) Bridge: Offer to help a non-profit - in ways that will leverage the skills you want to acquire or hone.

4) Explore: Make a list of the people you respect and admire in your field - then reach out and ask them if you can interview them for a profile article you wish to submit to a news site, or professional journal, related to your industry. Probe for what drives them, Ask 'why' questions.

5) Speculate: Find a struggling start-up that you think is worthy - and offer to provide some of your time and expertise (ideally in exchange for some form of an equity stake - but if not, then offer on a Pro Bono basis).

6) Plant Seeds: Connect with educators and guidance counselors in your local high schools and community colleges - offering to give talks on your field/profession, as a career choice.

7) Fertilize Your Mind: Read inspiring biographies of inventors, builders, creators, makers.

8) Change your Latitude: For me, it took a month of sailing along Baja Mexico - and it was like a light was switched "on" again.

9) Change your Attitude: Get your blood flowing on a regular basis with some kind of exercise. Even better, couple that with learning a new physical skill.

10) Change your Altitude: Get up into the mountains - see the valleys and views from a different perspective. It will open your senses - and open your mind.

11) Write for Yourself: Write a journal, with one clear intention: In what way have I expanded my knowledge/expertise today?

12) Write for Others: Submit ideas for articles, on speculation, to professional journals or industry news sites - topics that would be interesting to explore/learn - and when accepted, you'll be forced to dive into the material.

13) Exercise Your Idea Muscle: If you feel stuck, stagnant, powerless - it probably has more to do with your attitude - than lack of opportunity - than you might think. Invest an hour a day to practice writing business cases for new business ideas (or changes/improvements to your existing company) that you would implement - if you had the budget, time, and resources. Submit those you connect strongly with to your management team (or, consider as the basis for launching your own company).

14) Reflect: Consciously set aside time to reflect and explore how each part of the work that you do goes into the assembly of the whole - and how what you do affects the final product or service, who it affects, how it affects them, and the value that you add and create. Consider how what you do can be done better, faster, more efficiently, at lower cost. Practice #Mindfulness

15) Network with Intent, Outward. Find ways to continually expand your network of professional connections - related to your field. Seek ways to introduce, connect, help others - and don't worry if/when there might be some pay-off. Practice #Selflessness

16) Network with Intent, Within: Organize a weekly lunchtime brown bag session with your company - seeking to invite people from different divisions, departments, groups - with a rotating responsibility for picking a topic for discussion. Use this time to focus on how such a diverse group can generate a sustained synergy for the creation of new ideas to remove roadblocks to the development of new products/services - eliminate bureaucracy and waste - streamline operations - and reduce costs.

17) Apply Shu-Ha-Ri: Return to the source - immerse yourself in the study of the basics and fundamentals of your field, detach from your assumptions of how things must be - breaking from tradition, seek to transcend by evolving new ideas that escape from the staid boundaries of what has been.

18) Transpose Your Frame of Reference with New Coordinates: If you feel you have been in a particular industry for too long - spending time seeding new problems/ideas may be just the ticket to re-ignite your passion. Try allocating some time to reading recent research papers in completely different fields to expose your mind to different contexts. This is my favorite site for such inspiration: https://arxiv.org/

19) Play: Give yourself permission to explore useless things - just for fun. in "Surely You're Joking, Mr. Feynman!", Richard P. Feynman tells of reaching a point of burnout - and how he recovered his zest for his work:

Then I had another thought: Physics disgusts me a little bit now, but I used to enjoy doing physics. Why did I enjoy it? I used to play with it. I used to do whatever I felt like doing — it didn’t have to do with whether it was important for the development of nuclear physics, but whether it was interesting and amusing for me to play with. …

So I got this new attitude. Now that I am burned out and I’ll never accomplish anything, I’ve got this nice position at the university teaching classes which I rather enjoy, and just like I read the Arabian Nights for pleasure, I’m going to play with physics, whenever I want to, without worrying about any importance whatsoever.

Within a week I was in the cafeteria and some guy, fooling around, throws a plate in the air. As the plate went up in the air I saw it wobble, and I noticed the red medallion of Cornell on the plate going around. It was pretty obvious to me that the medallion went around faster than the wobbling. …

I went on to work out equations of wobbles. Then I thought about how electron orbits start to move in relativity. Then there’s the Dirac Equation in electrodynamics. And then quantum electrodynamics. And before I knew it (it was a very short time) I was “playing’’ — working, really — with the same old problem that I loved so much, that I had stopped working on when I went to Los Alamos: my thesis-type problems; all those old-fashioned, wonderful things.

It was effortless. It was easy to play with these things. It was like uncorking a bottle: Everything flowed out effortlessly. I almost tried to resist it! There was no importance to what I was doing, but ultimately there was. The diagrams and the whole business that I got the Nobel Prize for came from that piddling around with the wobbling plate.  

2018-08-31 Friday - The Greater Foundation, Opportunities for Community Service

In looking for a community service program to support - I happened upon The Greater Foundation, founded by Russell Okung - they are looking for mentors in the LA and Seattle area. 

 "Feed someone a fish, and you feed them for a day. Teach them to fish, and you feed them for a lifetime. Teach them to start a fishing company, and they can feed the entire world." -The Greater Foundation 

https://begreater.org/get-involved/#be-a-mentor

2018-08-28 Tuesday - Published Updated High Level Design (HLD) Template

https://github.com/intltechventures/Consulting.Project.Tools/blob/master/templates/HighLevelDesign.md
 

Rationale: Numerous times, in my travels as a consultant, I've encountered organizations in which there is no established standard/template for what should be included in an HLD. Quite often (even in organizations that ostensibly have a template), the artifacts that architects and engineers produce devolve into a Wild Wild West of anything-goes. Consistency engenders repeatability - which can help reduce variability in quality - and thus, a worthy goal.

Therefore, the intended purpose in sharing this...at least as a starting point for further customization by a client organization.

STATUS: DRAFT

The goal of an HLD is to facilitate communication and coordination, both within internal teams and organizations - as well as serving as a tool for communication and coordination with external partners.

The HLD provides a consistent format for teams to assemble details - which supports a goal of being able to achieve some level of reuse of design artifacts - and is intended to provide sufficient information to Program/Project Management to be able to plan, estimate, and coordinate large-scale development efforts. Additionally, an HLD servces as an effective mechanism to support Design Review and Architecture Governance efforts.

The ideal place for an HLD to live is in a Wiki or in a Markdown file (i.e. in the git repository for the project)

The target level of detail is to scope the effort, provide sufficient input to the estimation process, and to clearly articulate the How of an approach.

The What should be ideally defined in a collection of User Stories, Use Cases, Feature Requests, etc. - or, minimally, in a Business Requirements Document (BRD).

Investing effort in the creation of the HLD is intended for larger-scoped projects/epics - that have some non-trivial number of unknowns, technical complexity, a high number of coordination points, or extensive external integration/coordination requirements.