[Paper Review] Dissection of a Bug Dataset - Anatomy of 395 Patches from Defects4J
Short Introduction to This PaperThis paper deeply studied 395 patches of the Defects4J dataset. Quantitative properties (patch size and spreading) were automatically extracted, whereas qualitative ones (repair actions and patterns) were manually extracted using a thematic analysisbased approach. It found that:1) the median size of Defects4J patches is four lines, and almost 30% of the patches contain only addition of lines;2) 92% of the patches change only one file, and 38% has no spreading at all;3) the top-3 most applied repair actions are addition of method calls, conditionals, and assignments, occurring in 77% of the patches;4) nine repair patterns were found for 95% of the patches, where the most prevalent, appearing in 43% of the patches, is on conditional blocks.These results are useful for researchers to perform advanced analysis on their techniques’ results based on Defects4J. Moreover, this set of properties can be used to characterize and compare different bug datasetsHighlights of This PaperThe anatomy of the patches in Defects4J containing an extensive set of patch properties, consolidated into a JSON file and augmented with a web user-interface to facilitate explorationA bug dataset dissection methodology to extract valuable quantitative and qualitative properties regarding patches from bug datasets. The methodology is based on diff and advanced patch analysis and combines automated and manual thematic analysisA taxonomy of repair actions and patterns, resulted from manual analysis of patches according our methodologyKey InfomationResearch questions:What is the size distribution of Defects4J patches?To what extent are Defects4J patches spread in source code?What is the composition of Defects4J patches in terms of repair actions (additions, removals and modifications) over code elements (e.g. conditions and method calls)?What repair patterns can be found in Defects4J using a manual thematic analysis?Data Collection: For each bug, first produced diff views between the buggy program version and its associated fixed version. These views served as source for data extraction and analysisRepair Patterns in the Defects4J Patches:Conditional BlockExpression FixWraps-with / Unwraps-fromSingle LineWrong ReferenceMissing Null-CheckCopy / PasteConstant ChangeCode MovingRelevant Future WorksMaybe some supplement for Defects4J's documentCharacterization and comparison between different bug datasetsURLDissection of a Bug Dataset: Anatomy of 395 Patches from Defects4J
[Experiment] Docker Monitoring And Metrics Exporting Solutions
BackgroundThis post introduces 2 basic solutions about docker monitoring. And what's more, we want to dig into metrics exporting methods. In order to do some data analysis, we want to extract metrics in a usable/simple format (e.g. CSV). It's easy to query some metrics, but the tools paid less attention to export multiple metrics joined by timestamp(this can be some scenarios about data analysis, or dev daily report), we are going to build a small tools for this.[...]
[Notes] Chaos Engineering, Building Confidence in System Behavior through Experiments
Part I IntruductionChaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.-- Principles of ChaosUsing Chaos Engineering may be as simple as manually running kill -9 on a box inside of your staging environment to simulate failure of a service. Or, it can be as sophisticated as automatically designing and carrying out experiments in a production enviroment against a small but statistically significant fraction of live traffic.The History of Chaos Engineering at Netflix: started in 2008Chaos Monkey: ball rolling, gaining notoriety for turning off services in the production environmentChaos Kong: transferred those benefits from the small scale to the very largeFailure Injection Testing (FIT): the foundation for tackling the space in between[...]
[Paper Review] Data Poisoning: Lightweight Soft Fault Injection for Python
Short Introduction to This PaperThis paper introduces and explores the idea of data poisoning, a light-weight peer-architecture technique to inject faults into Python programs. This method requires very small modification to the original program, which facilitates evaluation of sensitivity of systems that are prototyped or modeled in Python. Actually this paper doesn't show much detail about the implementation, but the types of data poisoning it declares are very interesting.Highlights of This PaperData poisoning's symbolic expressionDifferent types of data poisoningKey InfomationTypes of data poisoning: deterministic effect poisoning, intermittent effect poisoning (need define the lifetime of poisoned data), infectious/non-infectious poisoningRelevant Future WorksOnly doing data poisoning is not enough, we should analysis the system's behaviour under different types of perturbationURLData Poisoning: Lightweight Soft Fault Injection for Python
[Paper Review] DICE Fault Injection Tool
Short Introduction to This PaperThis paper describes the motivation, innovation, design, running example and future development of a Fault Inject Tool (FIT). This tool enables controlled causing of cloud platform issues such as resource stress and service or VM outages, the purpose being to observe the subsequent effect on deployed applications.Highlights of This PaperThe DICE FIT will address the need to generate various cloud agnostic faults at the VM Admin and Cloud Admin levels. So greater flexibility and the ability to generate multiple faults, relatively lightweightKey InfomationDesign: To access the VM level and issue commands the DICE FIT uses SSH to connect to the Virtual Machines and issue the commands. By using JSCH, the tool is able to connect to any VM that has SSH enabled and issue commands as a pre-defined user. This allows greater flexibility of commands as well as the installation of tools and dependences.Relevant Future WorksContainerised environments will also be considered as future FIT targets to help understand the effect on microservices when injecting faults to the underlying host as well as the integrity of the containerised deploymentThe CACTOS project will expand the tool functionality by initiating a specific application level fault to trigger optimisation algorithmsURLDICE Fault Injection Tool(Paper)DICE-Fault-Injection-Tool(Github Project)
[Experiment] Using byte-monkey to do short-circuit testing
What is byte-monkeyByte-Monkey is a small Java library for testing failure scenarios in JVM applications - it works by instrumenting application code on the fly to deliberately introduce faults like exceptions and latency. Original blogpost here.Original git repoForked one to do short-circuit testingHow to use[...]
[Paper Review] Automating Failure Testing Research at Internet Scale
Short Introduction to This PaperThis paper describes how to adapt and implement a research prototype called lineage-driven fault injection (LDFI, another paper view see here) to automate failure testing at Netflix. Along the way, the authors describe the challenges that arose adapting the LDFI model to the complex and dynamic realities of the Netflix architecture. They show how they implemented the adapted algorithm as a service atop the existing tracing and fault injection infrastructure, and present preliminary results.Highlights of This PaperThe way of how they adapted LDFI to automate failure testing at Netflix is worth learning, including defining request classes, learning mappings, replay mechanisms.Key InfomationMore explanations about LDFI: It begins with a correct outcome, and asks why the system produced it. This recursive process of asking why questions yields a lineage graph that characterizes all of the computations and data that contributed to the outcome. By doing so, it reveals the system’s implicit redundancy by capturing the various alternative computations that could produce the good result.About the solutions in the SAT problem: It does not necessarily indicate a bug, but rather a hypothesis that must be tested via fault injectionImplementationTraining: the service collects production traces from the tracing infrastructure and uses them to build a classifier that determines, given a user request, to which request class it belongs. We found that the most predictive features include service URI, device type, and a variety of query string parameters including parameters passed to the Falcor data platformModel enrichment: the service also uses production traces generated by experiments (fault injection exercises that test prior hypotheses produced by LDFI) to update its internal model of alternatives. Intuitively, if an experiment failed to produce a user-visible error, then the call graph generated by that execution is evidence of an alternative computation not yet represented in the model, so it must be added. Doing so will effectively prune the space of future hypothesesExperiments: finally, the service occasionally "installs" a new set of experiments on Zuul. This requires providing Zuul with an up-to-date classifier and the current set of failure hypotheses for each active request class. Zuul will then (for a small fraction of user requests) consult the classifier and appropriately decorate user requests with fault metadataRelevant Future WorksThey used the single-label classifier for the first release of the LDFI service, but are continuing to investigate the multi-label formulationURLAutomating Failure Testing Research at Internet Scale
[Paper Review] Lineage-driven Fault Injection
Short Introduction to This PaperThis paper proposes a novel approach for discovering bugs in fault-tolerant data management systems: lineage-driven fault injection. A lineage-driven fault injector reasons backwards from correct system outcomes to determine whether failures in the execution could have prevented the outcome. The authors present MOLLY, a prototype of lineage-driven fault injection that exploits a novel combination of data lineage techniques from the database literature and state-of-the-art satisfiability testing. If fault-tolerance bugs exist for a particular configuration, MOLLY finds them rapidly, in many cases using an order of magnitude fewer executions than random fault injection. Otherwise, MOLLY certifies that the code is bug-free for that configuration.Highlights of This PaperA novel top-down strategy for discovering bugs in distributed data management systems: lineage-driven fault injection (LDFI)MOLLY, an implementation of LDFI. Like fault injection, MOLLY finds bugs in large-scale, complex distributed systems quickly, in many cases using an order of magnitude fewer executions than a random fault injectorKey InfomationFigure 1: The MOLLY system performs forward/backward alternation: forward steps simulate concrete distributed executions with failures to produce lineage-enriched outputs, while backward steps use lineage to search for failure combinations that could falsify the outputs. MOLLY alternates between forward and backward steps until it has exhausted the possible falsifiers.LDFI is inspired by the database literature notion of data lineage, which allows it to directly connect system outcomes to the data and messages that led to them. LDFI uses data lineage to reason backwards (from effects to causes) about whether a given correct outcome could have failed to occur due to some combination of faults. Rather than generating faults at random (or using application-specific heuristics), a lineage-driven fault injector chooses only those failures that could have affected a known good outcome, exercising fault-tolerance code at increasing levels of complexity. Injecting faults in this targeted way allows LDFI to provide completeness guarantees like those achievable with formal methods such as model checking, which have typically been used to verify small protocols in a bottom-up fashion. When bugs are encountered, LDFI’s topdown approach provides—in addition to a counterexample trace fine-grained data lineage visualizations to help programmers understand the root cause of the bad outcome and consider possible remediation strategiesLike formal methods, MOLLY finds all of the bugs that could be triggered by failures: when a MOLLY execution completes without counterexamples it certifies that no fault-tolerance bugs exist for a given configuration. MOLLY integrates naturally with root-cause debugging by converting counterexamples into data lineage visualizations.MOLLY quickly identifies 7 critical bugs in 14 fault-tolerant systems, for the remaining 7 systems, it provides a guarantee that no invariant violations exist up to a bounded execution depth, an assurance that state-of-the-art fault injectors cannot provideLimitationsWhile MOLLY can be used to find bugs in fundamentally non-deterministic protocols like anti-entropy or randomized consensus, certifying such protocols as bug-free will require additional researchAn abstraction that factors apart partial failure and asynchrony is fundamentally incomplete, because these algorithms are required to (attempt to) distinguish between delay and failureRelevant Future WorksVerifying recovery protocols by modeling crash-recovery failuresMany of the why-not provenance techniques discussed in the paper could assist in implementing LDFIRelaxing the synchronicity abstractionExplore using the backwards reasoning approach of LDFI to assist in fault-tolerant program synthesisURLLineage-driven Fault Injection
[Paper Review] Exception Handling Analysis and Transformation Using Fault Injection - Study of Resilience Against Unanticipated Exceptions
Short Introduction to This PaperThis paper aims at analyzing and improving how software handles unanticipated exceptions. The first objective is to set up contracts about exception handling and a way to assess them automatically. The second one is to improve the resilience capabilities of software by transforming the source code. The authors devise an algorithm, called short-circuit testing, which injects exceptions during test suite execution so as to simulate unanticipated errors. It is a kind of fault-injection techniques dedicated to exceptionhandling. This algorithm collects data that is used for verifying two formal contracts that capture two resilience properties w.r.t. exceptions: the source-independence and pure-resilience contracts. Then the team propose a code modification technique, called “catch-stretching” which allows error-recovery code (of the form of catch blocks) to be more resilient.Highlights of This PaperThis work shows that it is possible to reason on software resilience by injecting exceptions during test suite executionDefinition of two contracts for exception handling: source independence contract, pure resilience contractAn algorithm and four predicates to verify whether a try-catch satisfies those contractsA source code transformation to improve the resilience against exceptionsAn empirical evaluation on 9 open sources applications with one test suite each showing that there exists resilient try-catch blocks in practiceKey InfomationSource-independent: A try-catch is source-independent if the catch block proceeds equivalently, whatever the source of the caught exception is in the try blockPure Resilience: A try-catch is purely resilient if the system state is equivalent at the end of the try-catch execution whether or not an exception occurs in the try blockShort-circuit testing consists of dynamically injecting exceptions during the test suite execution in order to analyze the resilience of try-catch blocksCatch Stretching: Replacing the type of the caught exceptions so that they catch more exceptions than before. For instance, replacing catch(FileNotFoundException e) by catch(IOExceptione). The extreme of catch stretching is to parametrize the catch with the most generic type of exceptions(e.g. Throwable in Java, Exception in .NET)Relevant Future WorksFurther exploring how to improve the resilience of software applications: the scope of try blocks can be automatically adapted while still satisfying the test suiteThe purely resilient catch blocks could probably be used elsewhere because they have a real recovery powerThe resilience oracle has not to be only a test suite, but for example metamorphic relations or production tracesAutomated refactoring of the relevant test suiteQuestionsHow to do catch stretching when there is a try with multiple catch blocks? And maybe the original test suites are not enough to verify new catch blocksURLException Handling Analysis and Transformation Using Fault Injection - Study of Resilience Against Unanticipated Exceptions
[Paper Review] Correctness Attraction - A Study of Stability of Software Behavior Under Runtime Perturbation
Short Introduction to This PaperMany perturbations do not break the correctness in programs. This phenomenon is called "correctness attraction". This paper devises a novel protocol, which considers a systematic exploration of the perturbation space as well as perfect oracles to determine the correctness of the output. To this extent, a qualitative manual analysis enables it to set up the first taxonomy ever of the reasons behind correctness attraction.Highlights of This PaperA protocol called ATTRACT to study the stability of programs under perturbationExplored the perturbability of 10 subjects for two perturbation models, PONE for integers and PBOOL for booleans.Original taxonomy of the causes behind correctness attraction: natural randomization, potential aternative executions, fixed point effect, extra resources, relaxed problem, nullified perturbation, overfit to input dataKey InfomationAn execution perturbation is a runtime change of the value of one variable in a statement or an expression. An execution perturbation has 3 characteristics: time: when the change occurs (e.g. at the second and the fourth iteration of a loop condition), location: where in the code (e.g. on variable ‘i’ at line 42) and perturbation model: what is this change, according to the type of the location (e.g. +1 on an integer value)The perturbation space for an input is composed of of all possible unique perturbed executions according to a perturbation modelCorrectness attraction is the phenomenon by which the correctness of an output is not impacted by execution perturbation. Correctness attraction means that one can perturb an execution while keeping the output correct according to a perfect oracleRelevant Future WorksThe taxonomy is probably not complete since the benchmark does not reflect the diversity of softwareURLCorrectness Attraction - A Study of Stability of Software Behavior Under Runtime Perturbation