2018-11-20 Tuesday - Machine Learning Technical Debt and Anti-Patterns

This posting is a placeholder for links to interesting articles/papers that touch on technical debit and anti-patterns in Machine Learning algorithms/models/solutions.

Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

"Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns."

A  Convex Framework for Fair Regression

https://arxiv.org/abs/1706.02409

https://arxiv.org/pdf/1706.02409.pdf

"The widespread use of machine learning to make consequential decisions about individual citizens (including in domains such as...criminal sentencing has been accompanied by increased reports of instances in which the algorithms and models employed can be unfair or discriminatory in a variety of ways ...we introduce a rich family of fairness metrics for regression models that take the form of a fairness regularizer and apply them to the standard loss functions for linear and logistic regression." 

"Since these loss functions and our fairness regularizer are convex, the combined objective functions obtained from our framework are also convex, and thus permit efficient optimization. Furthermore, our family of fairness metrics covers the spectrum from the type of group fairness that is common in classification formulations (where e.g. false arrests in one racial group can be “compensated” for by false arrests in another racial group) to much stronger notions of individual fairness (where such cancellations are forbidden, and every injustice is charged to the model). Intermediate fairness notions are also covered. Our framework also permits one to either forbid the use of a "protected” variable (such as race), by demanding that a single model be learned across all groups, or to build different group-dependent models." 

"Most importantly, by varying the weight on the fairness regularizer, our framework permits us to compute the entire “Pareto curve” or efficient frontier of the trade-off between predictive accuracy and fairness.  Such curves are especially important to examine and understand in a domain-specific manner: since demanding fairness of models will always come at a cost of reduced predictive accuracy , it behooves practitioners working with fairness-sensitive data sets to understand just how mild or severe this trade-off is in their particular arena, permitting them to make informed modeling and policy decisions""...in this work we have studied a variety of fairness regularizers for regression problems, and applied them to data sets in which fairness is not subservient to generalization, but is instead a first-order consideration. Our empirical study has demonstrated that the choice of fairness regularizer (group, individual, hybrid, or other) and the particular data set can have qualitative effects on the trade-off between accuracy and fairness...." 

"The Communities and Crime dataset, from the UCI repository is a dataset which includes many features deemed relevant to violent crime rates (such as the percentage of the community’s population in an urban area, the community’s racial makeup, law enforcement involvement and racial makeup of that law enforcement in a community, amount a community’s law enforcement allocated to drug units) for different communities. This data is provided to train regression models based on this data to predict the amount of violent crime (murder, rape, robbery, and assault) in a given community...." 

"The COMPAS dataset The COMPAS dataset contains data from Broward County, Florida originally compiled by ProPublica in which the goal is to predict whether a convicted individual would commit a violent crime in the following two years or not. ..." 

2018-11-20 Tuesday - Conference on Neural Information Processing Systems (NIPS)

The Thirty-second Conference on Neural Information Processing Systems (NIPS) will be held Dec 2-8, 2018 - at the Palais des Congrès de Montréal, Montréal, Canada 

https://nips.cc/Conferences/2018/Schedule



Electronic Proceedings of the Neural Information Processing Systems Conference
https://papers.nips.cc/


Interesting papers from the 2017 NIPS conference:
https://papers.nips.cc/book/advances-in-neural-information-processing-systems-30-2017

Hunt For The Unique, Stable, Sparse And Fast Feature Learning On Graphs
https://papers.nips.cc/paper/6614-hunt-for-the-unique-stable-sparse-and-fast-feature-learning-on-graphs

"For the purpose of learning on graphs, we hunt for a graph feature representation that exhibit certain uniqueness, stability and sparsity properties while also being amenable to fast computation. This leads to the discovery of family of graph spectral distances (denoted as FGSD) and their based graph feature representations, which we prove to possess most of these desired properties. To both evaluate the quality of graph features produced by FGSD and demonstrate their utility, we apply them to the graph classification problem. Through extensive experiments, we show that a simple SVM based classification algorithm, driven with our powerful FGSD based graph features, significantly outperforms all the more sophisticated state-of-art algorithms on the unlabeled node datasets in terms of both accuracy and speed; it also yields very competitive results on the labeled datasets - despite the fact it does not utilize any node label information."

Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
https://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent

"We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones. Causes of failures include software bugs, network asynchrony, biases in local datasets, as well as attackers trying to compromise the entire system. Assuming a set of   workers, up to   being Byzantine, we ask how resilient can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient aggregation rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the aggregation rule capturing the basic requirements to guarantee convergence despite   Byzantine workers. We propose \emph{Krum}, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD. We also report on experimental evaluations of Krum."

One-Shot Imitation Learning
https://papers.nips.cc/paper/6709-one-shot-imitation-learning

"Imitation learning has been commonly applied to solve different tasks in isolation. This usually requires either careful feature engineering, or a significant number of samples. This is far from what we desire: ideally, robots should be able to learn from very few demonstrations of any given task, and instantly generalize to new situations of the same task, without requiring task-specific engineering. In this paper, we propose a meta-learning framework for achieving such capability, which we call one-shot imitation learning. Specifically, we consider the setting where there is a very large (maybe infinite) set of tasks, and each task has many instantiations. For example, a task could be to stack all blocks on a table into a single tower, another task could be to place all blocks on a table into two-block towers, etc. In each case, different instances of the task would consist of different sets of blocks with different initial states. At training time, our algorithm is presented with pairs of demonstrations for a subset of all tasks. A neural net is trained that takes as input one demonstration and the current state (which initially is the initial state of the other demonstration of the pair), and outputs an action with the goal that the resulting sequence of states and actions matches as closely as possible with the second demonstration. At test time, a demonstration of a single instance of a new task is presented, and the neural net is expected to perform well on new instances of this new task. Our experiments show that the use of soft attention allows the model to generalize to conditions and tasks unseen in the training data. We anticipate that by training this model on a much greater variety of tasks and settings, we will obtain a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks."

DPSCREEN: Dynamic Personalized Screening
https://papers.nips.cc/paper/6731-dpscreen-dynamic-personalized-screening

"Screening is important for the diagnosis and treatment of a wide variety of diseases. A good screening policy should be personalized to the disease, to the features of the patient and to the dynamic history of the patient (including the history of screening). The growth of electronic health records data has led to the development of many models to predict the onset and progression of different diseases. However, there has been limited work to address the personalized screening for these different diseases. In this work, we develop the first framework to construct screening policies for a large class of disease models. The disease is modeled as a finite state stochastic process with an absorbing disease state. The patient observes an external information process (for instance, self-examinations, discovering comorbidities, etc.) which can trigger the patient to arrive at the clinician earlier than scheduled screenings. The clinician carries out the tests; based on the test results and the external information it schedules the next arrival. Computing the exactly optimal screening policy that balances the delay in the detection against the frequency of screenings is computationally intractable; this paper provides a computationally tractable construction of an approximately optimal policy. As an illustration, we make use of a large breast cancer data set. The constructed policy screens patients more or less often according to their initial risk -- it is personalized to the features of the patient -- and according to the results of previous screens – it is personalized to the history of the patient. In comparison with existing clinical policies, the constructed policy leads to large reductions (28-68 %) in the number of screens performed while achieving the same expected delays in disease detection."

Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
https://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent

"We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones. Causes of failures include software bugs, network asynchrony, biases in local datasets, as well as attackers trying to compromise the entire system. Assuming a set of n workers, up to f being Byzantine, we ask how resilient can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient aggregation rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the aggregation rule capturing the basic requirements to guarantee convergence despite f Byzantine workers. We propose \emph{Krum}, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD. We also report on experimental evaluations of Krum."

2018-11-11 Sunday - Suggested Books for QA/Test Teams

While doing some research for recommendations on elevating the maturity of a QA Test Team - I happened across five books in particular that I think would be beneficial to any software testing team:

"A Friendly Introduction to Software Testing", 
by Bill Laboon - a lecturer in the Computer Science department at the University of Pittsburgh, where he focuses on teaching how to develop high-quality software in a real-world environment.
https://www.amazon.com/Friendly-Introduction-Software-Testing/dp/152347737

"Complete Guide to Test Automation: Techniques, Practices, and Patterns for Building and Maintaining Effective Software Projects", 
by Arnon Axelrod - a test automation expert, working as a senior consultant, architect, trainer, and lead of the test automation team at Sela Group
https://www.amazon.com/Complete-Guide-Test-Automation-Maintaining/dp/1484238311/

"Mastering Selenium WebDriver 3.0: Boost the performance and reliability of your automated checks by mastering Selenium WebDriver, 2nd Edition", 
by Mark Collin - a professional software tester who has been working in the software testing industry since 2001.
https://www.amazon.com/Mastering-Selenium-WebDriver-3-0-performance/dp/1788299671

"Software Testing: A Craftsman’s Approach, Fourth Edition", 
by Paul C. Jorgensen, PhD
https://www.amazon.com/Software-Testing-Craftsmans-Approach-Fourth/dp/1466560681/

"Lessons Learned in Software Testing: A Context-Driven Approach", 
by Cem Kaner, JD, PhD, - a professor of computer sciences at Florida Institute of Technology;
James Bach - founder and principal consultant of Satisfice, Inc.,a software testing and quality assurance company;
Bret Ptetichord - who works as an independent consultant, http://pettichord.com/
https://www.amazon.com/Lessons-Learned-Software-Testing-Context-Driven/dp/0471081124/