Work-in-progress notes here:
https://github.com/intltechventures/Lab.JavaScript/blob/master/RuleEngines.md
Saturday, February 23, 2019
2019-02-23 Saturday - Genetic Algorithms for Automated Generation of Test Cases
I'm doing some background reading and research on potential use of Genetic Algorithms to perform Automated Generation of Test Cases (and, Test Data).
My work-in-progress research notes are here:
https://github.com/intltechventures/Lab.Architecture/blob/master/Testing/GeneticAlgorithms.md
I've tried posting this URL on LinkedIn - but they truncate it, so posting here for convenience.
My work-in-progress research notes are here:
https://github.com/intltechventures/Lab.Architecture/blob/master/Testing/GeneticAlgorithms.md
I've tried posting this URL on LinkedIn - but they truncate it, so posting here for convenience.
Saturday, February 16, 2019
2019-02-16 Saturday - ShiftLeft for code analysis to patch vulnerabilities
ShiftLeft raises $20 million for code analysis software that automatically patches vulnerabilities
https://venturebeat.com/2019/02/12/shiftleft-raises-20-million-for-code-analysis-software-that-automatically-patches-vulnerabilities/
https://venturebeat.com/2019/02/12/shiftleft-raises-20-million-for-code-analysis-software-that-automatically-patches-vulnerabilities/
- "Gupta says that ShiftLeft achieved 75 percent on Open Web Application Security Project (OWASP) Benchmark for Security Automation — the highest score ever recorded — and that it enables the average developer to analyze 500,000 lines of code in less than 10 minutes,"
- "ShiftLeft’s core security-as-a-service offering obviates the need for traditional code-checking security software, Gupta claims, because it conducts a study of apps’ and services’ code and dependencies before runtime and creates an “agent” that protects against exploits as they crop up. It supports virtual machines, cloud infrastructure, containers, and other environments, and automatically identifies external data leakages and shields new versions of programs as they’re deployed."
Thursday, February 14, 2019
2019-02-14 Thursday - Professional Development Advice for an Architect
[This list was assembled for a colleague in September 2018, I will revisit this in the near future and amend some of the suggested resources]
Below, I’ve assembled a suggested “short-list” of what I think are the essential topics - in which someone involved in Architecture might benefit – over the long-term – by investing the time to become familiar with the material outlined. This is a journey – it will take time – and it will be challenging. There is a plethora of articles, web sites, and books out there – many badly written, often with bad/conflicting suggestions/conclusions/practices. Hopefully this may be useful in providing some guidance on where you may get the most benefit for effort invested, with an eye to a long career arc.
Below, I’ve assembled a suggested “short-list” of what I think are the essential topics - in which someone involved in Architecture might benefit – over the long-term – by investing the time to become familiar with the material outlined. This is a journey – it will take time – and it will be challenging. There is a plethora of articles, web sites, and books out there – many badly written, often with bad/conflicting suggestions/conclusions/practices. Hopefully this may be useful in providing some guidance on where you may get the most benefit for effort invested, with an eye to a long career arc.
The topics are roughly in order of suggested reading…
Suggested resources to
help you stay up-to-date on news and emerging trends in the world of
Architecture and Distributed Systems:
Suggested Useful References to Know
The Open Group ArchiMate 3.x Specification
The Open Group TOGAF Standard, version 9.2
The Open Group IT4IT Reference Architecture
W3C Standards
IETF RFCs
Essential Concepts to Know, When Evaluating Architecture/Design Decisions/Choices
Suggested resources for learning to formally document architectures
Mastering ArchiMate, 3rd Edition
Archi – Open Source Archimate Modeling Tool
Sparx Enterprise Architect
Documenting Software Architectures: Views and Beyond
Software Architecture in Practice, 3rd Edition
Designing Software Architectures: A Practical Approach
UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd edition
-
While UML may have
fallen out of favor in certain camps , it is very useful technique for
capturing complex interactions, and doing quick diagrams that have a
high density of information encoded in
just a few diagram elements.
Learning UML 2.0: A Pragmatic Introduction to UML
Suggested resources: These are good books to provide a good introduction/foundation – to the world of Enterprise/System/Solution Architecture
Clean Architecture: A Craftsman’s Guide to Software Structure and Design
Building Evolutionary Architectures: Supporting Constant Change
Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions
Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services
Domain-Driven Design: Tackling Complexity in the Heart of Software
Good books to develop a deeper understanding of Patterns
Patterns of Enterprise Application Architecture
Domain-Driven Design Reference: Definitions and Pattern Summaries
Refactoring to Patterns
Refactoring: Improving the Design of Existing Code, 2nd Edition
-
(available Nov 2018)
For API, SOA, Web Services based architectures….
Building Microservices: Designing Fine-Grained Systems (2nd Edition, October 2019)
SOA Design Patterns (used copy, as low as $2.83)
For Data-Intensive, Large-Scale, Streaming, Distributed type Architecture problems…
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Streaming Systems: The What, Where, When, and How of Large-Scale data Processing
Learning Spark: Lightning-Fast Big Data Analysis
Spark: The Definitive Guide: Big Data Processing Made Simple
High Performance Spark: Best Practices for Scaling and Optimizing Apache Spark
For a deeper dive into the foundations of Patterns, but at a more granular/code level:
Head First Design Patterns: A Brain-Friendly Guide
Design Patterns: Elements of Reusable Object-Oriented Software
These are good books to
deepen your understanding of the concept of Programmer Craftsmanship
–and the application team level concerns in designing and developing
great software
Clean Code: A Handbook of Agile Software Craftsmanship
The Pragmatic Programmer: From Journeyman to Master
Code Complete: A Practical Handbook of Software Construction, 2nd Edition
The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition, 2nd Edition
Algorithms, 4th Edition
-
a good intro book…
Introduction to Algorithms, 3rd Edition
-
a more robust, deeper dive into the topic of algorithms
Suggested learning resources for Cloud, Infrastructure, Operations, DevOps, Performance, Site Reliability
Cloud Native Infrastructure: Patterns for Scalable Infrastructure and Applications in a Dynamic Environment
Infrastructure as Code: Managing Servers in the Cloud
Site Reliability Engineering: How Google Runs Production Systems
o
(free to read online)
Kubernetes: Up and running: Dive into the Future of Infrastructure
Kubernetes Cookbook: Building Cloud Native Applications
Suggested learning resources for AI, Machine/Deep Learning
The Current State of Machine Learning 3.0
Machine Learning Crash Course: Google’s fast-paced, practical introduction to machine learning
100 Days of ML Code
MIT Open Courseware, Artificial Intelligence
Coursera: Machine Learning Specialization
Coursera: Machine Learning
Deep Learning Tutorial Lessons
Deep Learning with Python
-
(I’ve done some of the technical edit review for Manning – a great book)
Deep Learning: A Practitioner’s Approach
Grokking Deep Learning
-
(available in January 2019, I’ve done some early technical edit/review for Manning on this – a really good book)
Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent System
-
Note: new version coming out May 2019 – but this is very good edition
Deep Learning
-
(free to read online)
Artificial Intelligence: A Modern Approach, 3rd Edition
Machine Learning
-
(free to read online)
Bayesian Reasoning and Machine Learning
-
(free to read online)
Gaussian Processes for Machine Learning
-
(free to read online)
Suggested resources to help you stay up-to-date on news and emerging trends in AI, Machine/Deep Learning
A great resource for periodically checking new / leading edge academic papers, related to Computer Science Research
My personal Architecture Review Checklist tool
https://github.com/intltechventures/Consulting.Project.Tools/blob/master/templates/Architecture_Review_Checklists.xlsx
Wednesday, February 13, 2019
2019-02-13 Wednesday - Homomorphic Encryption
An interesting discussion arose in the last week, which introduced me to the concepts of using Homomorphic Encryption in machine learning solutions.
https://en.wikipedia.org/wiki/Homomorphic_encryption
IBM's Blindfolded Calculator
A very casual introduction to Fully Homomorphic Encryption
Encrypt your Machine Learning
How Practical is Homomorphic Encryption for Machine Learning?
A FULLY HOMOMORPHIC ENCRYPTION SCHEME
A brief survey of Fully Homomorphic Encryption, computing on encrypted data
Homomorphic EncryptionShai Halevi (IBM Research)April 2017
Fully Homomorphic Encryption: Cryptography’s Holy Grail
Cryptology ePrint Archive: Report 2015/1192
A Guide to Fully Homomorphic Encryption
https://en.wikipedia.org/wiki/Homomorphic_encryption
IBM's Blindfolded Calculator
A very casual introduction to Fully Homomorphic Encryption
Encrypt your Machine Learning
How Practical is Homomorphic Encryption for Machine Learning?
A FULLY HOMOMORPHIC ENCRYPTION SCHEME
- A DISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
- Craig Gentry, September 2009
A brief survey of Fully Homomorphic Encryption, computing on encrypted data
Homomorphic EncryptionShai Halevi (IBM Research)April 2017
"Fully homomorphic encryption (FHE) has been called the “Swiss Army knife of cryptog-raphy”, since it provides a single tool that can be uniformly applied to many cryptographicapplications. In this tutorial we study FHE and describe its different properties, relations withother concepts in cryptography, and constructions. We briefly discuss the three generations ofFHE constructions since Gentry’s breakthrough result in 2009, and cover in detail the third-generation scheme of Gentry, Sahai, and Waters (GSW)"
Fully Homomorphic Encryption: Cryptography’s Holy Grail
Cryptology ePrint Archive: Report 2015/1192
A Guide to Fully Homomorphic Encryption
Monday, February 11, 2019
2019-02-11 Monday - CVE-2019-5736: runc container breakout
An important security vulnerability to know about, if you are using Docker container based technologies:
CVE-2019-5736: runc container breakout (all versions)
https://seclists.org/oss-sec/2019/q1/119
Doomsday Docker security hole uncovered
https://www.zdnet.com/article/doomsday-docker-security-hole-uncovered/
CVE-2019-5736: runc container breakout (all versions)
https://seclists.org/oss-sec/2019/q1/119
Doomsday Docker security hole uncovered
https://www.zdnet.com/article/doomsday-docker-security-hole-uncovered/
Saturday, February 09, 2019
2019-02-09 Saturday - Git Status/Activity Tooling
An example of of a tool to produce git stats (like svnstat):
Another possible alternative:
- https://github.com/soundasleep/statgit2
- Example of types of reports produced:
Another possible alternative:
Wednesday, February 06, 2019
2019-02-06 Wednesday - Interesting Kaggle Competition, $50K Prize
LANL Earthquake Prediction
Can you predict upcoming laboratory earthquakes?https://www.kaggle.com/c/LANL-Earthquake-Prediction
$50,000 Prize Money
"Forecasting earthquakes is one of the most important problems in Earth science because of their devastating consequences. Current scientific studies related to earthquake forecasting focus on three key points: when the event will occur, where it will occur, and how large it will be."
"In this competition, you will address when the earthquake will take place. Specifically, you’ll predict the time remaining before laboratory earthquakes occur from real-time seismic data."
"If this challenge is solved and the physics are ultimately shown to scale from the laboratory to the field, researchers will have the potential to improve earthquake hazard assessments that could save lives and billions of dollars in infrastructure."
"This challenge is hosted by Los Alamos National Laboratory which enhances national security by ensuring the safety of the U.S. nuclear stockpile, developing technologies to reduce threats from weapons of mass destruction, and solving problems related to energy, environment, infrastructure, health, and global security concerns."
2019-02-06 Wednesday - An Interesting ML Paper
Go-Explore: A New Type of Algorithm for Hard-exploration Problems
https://www.youtube.com/watch?v=SWcuTgk2di8
Go-Explore: a New Approach for Hard-Exploration Problems
https://arxiv.org/abs/1901.10995
https://www.youtube.com/watch?v=SWcuTgk2di8
Go-Explore: a New Approach for Hard-Exploration Problems
https://arxiv.org/abs/1901.10995
"A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to improve performance on hard-exploration domains. To address this shortfall, we introduce a new algorithm called Go-Explore. It exploits the following principles: (1) remember previously visited states, (2) first return to a promising state (without exploration), then explore from it, and (3) solve simulated environments through any available means (including by introducing determinism), then robustify via imitation learning. The combined effect of these principles is a dramatic performance improvement on hard-exploration problems. On Montezuma's Revenge, Go-Explore scores a mean of over 43k points, almost 4 times the previous state of the art. Go-Explore can also harness human-provided domain knowledge and, when augmented with it, scores a mean of over 650k points on Montezuma's Revenge. Its max performance of nearly 18 million surpasses the human world record, meeting even the strictest definition of "superhuman" performance. On Pitfall, Go-Explore with domain knowledge is the first algorithm to score above zero. Its mean score of almost 60k points exceeds expert human performance. Because Go-Explore produces high-performing demonstrations automatically and cheaply, it also outperforms imitation learning work where humans provide solution demonstrations. Go-Explore opens up many new research directions into improving it and weaving its insights into current RL algorithms. It may also enable progress on previously unsolvable hard-exploration problems in many domains, especially those that harness a simulator during training (e.g. robotics). "
2019-02-06 Wednesday - Open Access Books
A wealth of open access, scientist authored, peer reviewed, academic level books - covering many different disciplines.
3,900 Open Access Books
Subscribe to:
Posts (Atom)